Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR
Abstract
Model merging is a scalable alternative to multi-task training that combines the capabilities of multiple specialised models into a single model. This is particularly attractive for large speech foundation models, which are typically adapted through domain-specific fine-tuning, resulting in multiple customised checkpoints, for which repeating full fine-tuning when new data becomes available is computationally prohibitive. In this work, we study model merging for multi-domain ASR and benchmark 11 merging algorithms for 10 European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance. We further propose BoostedTSV-M, a new merging algorithm based on TSV-M that mitigates rank collapse via singular-value boosting and improves numerical stability. Overall, our approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper makes a meaningful contribution to the field of machine learning by advancing the methodology of model merging in ASR, demonstrating its effectiveness through rigorous experimentation and providing a toolkit for future research. The work addresses a critical challenge in the deployment of ASR systems, making it a valuable addition to the literature.
Methodology
The paper introduces a novel approach to model merging specifically tailored for Automatic Speech Recognition (ASR) in multi-domain settings. The proposed BoostedTSV-M method enhances the existing TSV-M algorithm by addressing rank collapse through singular-value boosting, which is a significant methodological advancement. The paper systematically evaluates eleven merging algorithms across ten distinct European Portuguese domains, providing a comprehensive framework for assessing the performance of these methods. The introduction of MergeWhisper as a toolkit for implementing these merging methods further strengthens the methodology, allowing for reproducibility and extension by other researchers.
Results & evaluation
The experiments are well-structured, utilizing a diverse set of datasets that include both in-domain (ID) and out-of-domain (OOD) evaluations. The paper reports detailed results on word error rates (WER) and character error rates (CER), comparing the performance of the proposed merging methods against full fine-tuning and individual fine-tuning approaches. The findings demonstrate that while full fine-tuning yields the best in-domain performance, it can degrade out-of-domain performance, highlighting the trade-offs involved in model merging. The experimental results are robust and support the claims made regarding the effectiveness of the proposed methods.
Reproducibility
The paper provides sufficient details regarding the implementation, including the choice of the Whisper Large-v3 model, training configurations, and evaluation metrics. However, the absence of a publicly available code repository or demo URL limits the reproducibility of the results. The authors mention that the code will be available post-review, which is a positive step towards enhancing reproducibility.
Limitations
One limitation noted in the paper is the trade-off between in-domain specialization and out-of-domain robustness, which suggests that while the proposed methods improve generalization across domains, they may not achieve the same level of performance as dedicated fine-tuning for each domain. Additionally, the focus on European Portuguese may limit the generalizability of the findings to other languages or dialects.
Broader impact
The findings of this research have significant implications for the deployment of ASR systems in multi-domain environments, particularly in scenarios where computational resources are constrained. The ability to merge models effectively can lead to more efficient and scalable ASR solutions, potentially benefiting applications in multilingual settings and low-resource languages. The introduction of a merging toolkit also encourages further research and experimentation in this area. This paper makes a meaningful contribution to the field of machine learning by advancing the methodology of model merging in ASR, demonstrating its effectiveness through rigorous experimentation and providing a toolkit for future research. The work addresses a critical challenge in the deployment of ASR systems, making it a valuable addition to the literature.
Author of this paper? Send a correction.