Publications
CONFERENCE (INTERNATIONAL) Model Merging as an Alternative to Fine-Tuning on Combined Data for a Multi-Domain Dense Retriever
Taiga Sasaki (University of Hyogo), Takehiro Yamamoto (University of Hyogo), Hiroaki Ohshima (University of Hyogo), Sumio Fujita
The 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR 2026)
July 24, 2026
This study aims to construct a single dense retriever that achieves high effectiveness across multiple domains. A common approach is fine-tuning on combined training data covering all domains. However, this can lead to lower effectiveness due to imbalanced training data and interference across heterogeneous domains. To address this, we investigate model merging via simple weight averaging to integrate individually fine-tuned domain-specific dense retrievers into a unified model. We evaluate the approach in an incremental setting where the number of target domains increases from two to five. Experiments using two base dense retrievers with different retrieval effectiveness show that, when a new domain is added, the merged model tends to improve retrieval effectiveness on the newly added domain while often continuing to outperform the base retriever on previously covered domains. Moreover, while fine-tuning on combined data can be competitive when the number of target domains is small, merged models tend to achieve higher macro-average nDCG@10 across the target domains as the number of target domains increases. This suggests that model merging is more robust to domain imbalance and task conflicts across domains.
Paper :
Model Merging as an Alternative to Fine-Tuning on Combined Data for a Multi-Domain Dense Retriever
(external link)