Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition
Abstract
As pretrained large language models replace task-specific decoders in speech recognition, a critical question arises: do their text-derived priors make recognition fairer or more biased across demographic groups? We evaluate nine models spanning three architectural generations (CTC with no language model, encoder-decoder with an implicit LM, and LLM-based with an explicit pretrained decoder) on about 43,000 utterances across five demographic axes (ethnicity, accent, gender, age, first language) using Common Voice 24 and Meta's Fair-Speech, a controlled-prompt dataset that eliminates vocabulary confounds. On clean audio, three findings challenge assumptions: LLM decoders do not amplify racial bias (Granite-8B has the best ethnicity fairness, max/min WER = 2.28); Whisper exhibits pathological hallucination on Indian-accented speech with a non-monotonic insertion-rate spike to 9.62% at large-v3; and audio compression predicts accent fairness more than LLM scale. We then stress-test these findings under 12 acoustic degradation conditions (noise, reverberation, silence injection, chunk masking) across both datasets, totaling 216 inference runs. Severe degradation paradoxically compresses fairness gaps as all groups converge to high WER, but silence injection amplifies Whisper's accent bias up to 4.64x by triggering demographic-selective hallucination. Under masking, Whisper enters catastrophic repetition loops (86% of 51,797 insertions) while explicit-LLM decoders produce 38x fewer insertions with near-zero repetition; high-compression audio encoding (Q-former) reintroduces repetition pathology even in LLM decoders. These results suggest that audio encoder design, not LLM scaling, is the primary lever for equitable and robust speech recognition.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the first systematic study isolating how the degree of language model integration affects ASR fairness across multiple demographic axes and under various acoustic conditions. The comprehensive evaluation of model performance and the nuanced insights into bias and fairness in speech recognition systems represent a significant advancement in understanding the implications of integrating LLMs into ASR technologies.
Methodology
The paper presents a systematic approach to evaluating the impact of large language model (LLM) decoders on speech recognition fairness across various demographic axes. The methodology is robust, involving the evaluation of nine models across three architectural generations and utilizing two diverse datasets. The authors employ a variety of acoustic degradation conditions to assess model performance, which is a significant strength. The use of metrics such as word error rate (WER) and max/min ratio (MMR) for fairness assessment is appropriate and well-justified. However, the paper could benefit from a clearer explanation of the statistical methods used for significance testing.
Results & evaluation
The experiments are comprehensive, covering a large number of utterances (43,000) and multiple demographic axes (ethnicity, accent, gender, age, first language). The stress-testing under 12 acoustic degradation conditions adds depth to the analysis, revealing nuanced insights into model performance. The findings regarding the non-amplification of racial bias by LLM decoders and the specific hallucination issues with Whisper on Indian-accented speech are particularly noteworthy. However, the paper lacks detailed visualizations of results, which could enhance understanding and interpretation of the data.
Reproducibility
The authors commit to reproducibility by providing model weights, inference parameters, and preprocessing steps in the appendix. However, the lack of a public repository or demo URL limits accessibility for other researchers who may wish to replicate the study or build upon its findings. Clear documentation of the experimental setup is essential for reproducibility, and while the paper mentions this, it could be more explicit in detailing the steps taken.
Limitations
The study is limited to English read and prompted speech, which may not generalize to multilingual or spontaneous ASR systems. Additionally, the authors acknowledge that training data differences could confound results, particularly regarding the decoder's contributions. The synthetic nature of perturbation conditions may not fully capture real-world scenarios, and the reliance on self-reported demographic labels could introduce biases.
Broader impact
This research has significant implications for the deployment of speech recognition systems in diverse populations. By highlighting the potential biases inherent in LLM decoders and the architectural factors that influence fairness, the study encourages the development of more equitable ASR systems. The findings could inform best practices for model selection and deployment in applications that serve varied demographic groups. The main contribution of this paper is the first systematic study isolating how the degree of language model integration affects ASR fairness across multiple demographic axes and under various acoustic conditions. The comprehensive evaluation of model performance and the nuanced insights into bias and fairness in speech recognition systems represent a significant advancement in understanding the implications of integrating LLMs into ASR technologies.
Author of this paper? Send a correction.