Grounded in Sound: Reinforcement Learning with a Frozen Acoustic Judge to Curb ASR Insertion Hallucinations
Abstract
When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
Methodology
The paper proposes a reinforcement learning (RL) post-training framework for ASR that augments the standard Word Error Rate (WER) reward with an "acoustic-fidelity" reward. This reward is generated by a frozen, non-autoregressive wav2vec2-CTC model that scores the acoustic consistency of the policy's generated transcriptions against the input audio. The core innovation is the use of an independent, frozen judge to penalize hallucinations (specifically insertions) during training, thereby forcing the policy to ground its outputs in acoustic evidence rather than relying solely on language priors. The method is applied to a 7B speech LLM (Qwen2-Audio) using Group Relative Policy Optimization (GRPO). The approach is conceptually sound, addressing a known failure mode in LLM-based ASR where models generate fluent but ungrounded text when acoustic signals are weak. The use of a character-level CTC judge is a practical choice, avoiding the need for a complex language model in the reward function.
Results & evaluation
The experimental setup is rigorous, featuring a six-tier difficulty gradient ranging from clean LibriSpeech to noisy conditions and real-world AMI meeting speech (both close-talking and far-field). The evaluation focuses on insertion errors, which is the specific target of the proposed reward. The results show a significant reduction in insertion errors (28.3% on AMI-IHM, 22.3% on AMI-SDM) without a significant increase in WER on other tiers. The paper includes four prespecified analyses: a counterfactual probe with unintelligible audio, a comparison with CTC rescoring, a judge-free selective ASR analysis, and a cost-sensitive WER analysis. These analyses provide strong evidence that the model is learning to suppress insertions based on acoustic content rather than just energy detection. The use of paired bootstrap confidence intervals adds statistical rigor.
Reproducibility
The paper provides detailed implementation specifics, including the exact models used (Qwen2-Audio-7B, wav2vec2-large-960h), hyperparameters (LoRA rank, learning rate, batch size), and the specific reward formulation. The data splits and augmentation strategies are clearly described. However, the code is not explicitly linked in the provided text, and the specific version of vLLM and the LoRA merging process are mentioned as implementation details that might be tricky to reproduce exactly without the authors' code. The "frozen judge" setup is straightforward to implement given the open-source nature of the components.
Limitations
The primary limitation is the scope of the evaluation, which is limited to one policy model (Qwen2-Audio-7B) and one judge model (wav2vec2-CTC). The paper acknowledges that judge swaps (e.g., with HuBERT or Whisper) were not performed. Additionally, the method does not address deletion errors, which can increase as a trade-off for reducing insertions, although the paper notes this cost is low. The quadratic normalization in the CTC score is an implementation detail that the authors acknowledge as a potential source of bias, though they argue it is mitigated by group normalization. The lack of a no-RL baseline in the counterfactual analysis limits the ability to attribute all grounding behavior to the RL process itself.
Broader impact
This work has significant implications for the deployment of LLM-based ASR systems in real-world, noisy environments. By providing a training-time mechanism to curb hallucinations without adding inference-time latency or complexity, it offers a practical path to more reliable speech recognition. The concept of using an independent acoustic judge for RL rewards could be extended to other multimodal tasks where grounding in sensory input is critical. It also highlights the importance of reward design in RL for generative models, moving beyond simple text-matching metrics to include modality-specific fidelity checks. The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
Author of this paper? Send a correction.