Two-Stage Acoustic Adaptation with Gated Cross-Attention Adapters for LLM-Based Multi-Talker Speech Recognition
Abstract
Large Language Models (LLMs) are strong decoders for Serialized Output Training (SOT) in two-talker Automatic Speech Recognition (ASR), yet their performance degrades substantially in challenging conditions such as three-talker mixtures. A key limitation is that current systems inject acoustic evidence only through a projected prefix, which can be lossy and imperfectly aligned with the LLM input space, providing insufficient fine-grained grounding during decoding. Addressing this limitation is crucial for robust multi-talker ASR, especially in three-talker mixtures. This paper improves LLM-based multi-talker ASR by explicitly injecting talker-aware acoustic evidence into the decoder. We first revisit Connectionist Temporal Classification (CTC)-derived prefix prompting and compare three variants with increasing acoustic content. The CTC information is obtained using the serialized CTC proposed in our previous works. While acoustic-enriched prompts outperform the SOT-only baseline, prefix-only conditioning remains inadequate for three-talker mixtures. We therefore propose a lightweight gated residual cross-attention adapter and design a two-stage acoustic adaptation framework based on low-rank updates (LoRA). In Stage 1, we insert gated cross-attention adapters after the self-attention sub-layer to stably inject acoustic embeddings as external memory. In Stage 2, we refine both the cross-attention adapters and the pretrained LLM's self-attention projections using parameter-efficient LoRA, improving robustness for large backbones under limited data; the learned updates are merged into the base weights for inference. Experiments on Libri2Mix/Libri3Mix under clean and noisy conditions show consistent gains, with particularly large improvements in three-talker settings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant advancement in multi-talker ASR by introducing a two-stage acoustic adaptation framework that enhances the integration of acoustic evidence into LLMs. The innovative methodology and promising experimental results position it as a valuable contribution to the field of audio processing and machine learning.
Methodology
The paper proposes a two-stage acoustic adaptation framework that integrates gated cross-attention adapters into a large language model (LLM) for multi-talker automatic speech recognition (ASR). The methodology is innovative in addressing the limitations of prefix-only conditioning by dynamically injecting talker-aware acoustic evidence during decoding. The use of low-rank updates (LoRA) for parameter-efficient adaptation is particularly noteworthy, as it enhances the model's robustness under limited data conditions. The systematic exploration of CTC-derived prefix prompting variants adds depth to the methodology, although the paper could benefit from a clearer description of the experimental setup and hyperparameter tuning processes.
Results & evaluation
The experiments are well-structured, utilizing the Libri2Mix and Libri3Mix datasets to evaluate the proposed methods under both clean and noisy conditions. The results demonstrate consistent performance gains, particularly in three-talker scenarios, validating the effectiveness of the proposed gated cross-attention adapters. However, the paper lacks detailed statistical analysis of the results and comparisons with state-of-the-art methods beyond the baseline systems, which would strengthen the claims of improvement.
Reproducibility
The paper provides a comprehensive overview of the model architecture and training procedures, but it lacks specific implementation details that would facilitate reproducibility, such as exact hyperparameter settings and training schedules. The absence of a publicly available code repository further hinders reproducibility efforts.
Limitations
A notable limitation is the reliance on the performance of the gated cross-attention mechanism, which, while effective, still falls short of the robustness offered by the serialized CTC approach in certain scenarios. Additionally, the paper does not address the potential computational overhead introduced by the proposed adaptations, which may limit practical deployment in real-time systems.
Broader impact
The advancements presented in this paper have significant implications for multi-talker ASR systems, particularly in applications such as conference transcription, voice assistants, and accessibility technologies. By improving the ability of LLMs to handle overlapping speech, the research could enhance communication tools for diverse user groups, including those with hearing impairments. This paper presents a significant advancement in multi-talker ASR by introducing a two-stage acoustic adaptation framework that enhances the integration of acoustic evidence into LLMs. The innovative methodology and promising experimental results position it as a valuable contribution to the field of audio processing and machine learning.
Author of this paper? Send a correction.