Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs
Abstract
Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
Methodology
The paper proposes EAVA, a two-stage fine-tuning strategy for Speech-LLMs. The core insight is that standard fine-tuning often fails to adequately adapt the speech encoder to new acoustic domains because the LLM dominates the gradient flow. Stage 1 ("Encoder Awakening") freezes the entire pre-trained model and trains only lightweight adapters (Residual Adapters) inserted into the encoder layers. Stage 2 jointly fine-tunes the encoder, adapters, projector, and LLM LoRA. The methodology is sound, leveraging the parameter efficiency of adapters to preserve pre-trained knowledge while injecting domain-specific acoustic features. The use of CKA analysis to visualize layer-wise representation shifts provides a strong theoretical grounding for why this approach works (preserving lower layers, adapting upper layers).
Results & evaluation
The experiments are comprehensive, covering three distinct domain-shifted datasets (OGI Kids, MyST, CORAAL) and two different Speech-LLM backbones (Canary-Qwen, Phi-4-Multimodal). The paper reports state-of-the-art results, outperforming vanilla fine-tuning, multi-stage alignment, and previous SOTA methods. Ablation studies effectively isolate the contribution of the two-stage process and the choice of adapter architecture. The inclusion of cross-dataset transfer analysis and loss function variations (CTC vs. CE) adds significant depth to the evaluation.
Reproducibility
The paper provides a public GitHub repository link. It specifies model architectures, dataset splits, hyperparameters (learning rates, adapter dimensions), and training schedules. The use of open-source backbones (Canary-Qwen, Phi-4) and standard datasets enhances reproducibility.
Limitations
The method relies on the availability of a pre-trained Speech-LLM backbone. The gains, while consistent, are moderate in some cases (e.g., MyST). The paper does not extensively explore the computational overhead of the two-stage training compared to single-stage fine-tuning, though the parameter efficiency is highlighted. The "awakening" stage requires a separate training phase, which may be a barrier for rapid deployment scenarios.
Broader impact
This work addresses a critical bottleneck in deploying Speech-LLMs in real-world, diverse acoustic environments. By providing a parameter-efficient and effective domain adaptation method, it lowers the barrier for adapting large speech models to specific populations (children, dialect speakers) or noisy environments. The insights into encoder representation shifts via CKA are valuable for the broader community working on multimodal model adaptation. Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
Author of this paper? Send a correction.