Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
Abstract
This work presents the first systematic investigation of speech bias in multilingual MLLMs. We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite, spanning English, Chinese, and Korean, balanced by gender and accent, and totaling 70.8 hours ($\approx$4,249 minutes) of speech with 11,200 questions. Using four complementary metrics (accuracy, entropy, APES, and Fleiss' $κ$), we evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations. Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases. Moreover, architectural design and reasoning strategy substantially affect robustness across languages. Overall, this study establishes a unified framework for assessing fairness and robustness in speech-integrated LLMs, bridging the gap between text- and speech-based evaluation. The resources can be found at https://github.com/ntunlplab/BiasInEar.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents the first systematic study of speech bias and robustness in multilingual MLLMs. It contributes significantly to the understanding of how demographic and structural factors influence model performance, establishing a framework for future research in fairness and robustness in speech technologies.
Methodology
The paper introduces a novel dataset, BiasInEar, which is a significant contribution to the field of multilingual speech bias assessment. The methodology involves a systematic evaluation of multilingual MLLMs using a diverse set of metrics (accuracy, entropy, APES, and Fleiss' Kappa) to analyze the robustness of these models across various linguistic, demographic, and structural factors. The use of TTS for audio generation and the detailed approach to question rewriting are particularly noteworthy, as they address challenges in creating a reliable speech dataset. The framework established for assessing fairness and robustness in speech-integrated LLMs is a substantial methodological advancement.
Results & evaluation
The experiments are comprehensive, evaluating nine representative models across multiple languages and demographic factors. The dataset's size (70.8 hours of speech and 11,200 questions) allows for robust statistical analysis. The findings reveal critical insights into how demographic factors like gender and accent influence model performance, with a clear emphasis on the sensitivity of models to option order. The results are well-presented, with appropriate use of figures and tables to illustrate findings, which enhances the clarity and impact of the experimental outcomes.
Reproducibility
The paper provides detailed implementation details, including the use of specific APIs for model evaluation and the generation of audio samples. However, the reliance on proprietary APIs and TTS systems may pose challenges for full reproducibility. The authors do mention computational constraints, which could limit the ability of other researchers to replicate the experiments exactly. Nonetheless, the clear documentation of the methodology and dataset construction steps aids in reproducibility.
Limitations
The study acknowledges limitations related to the use of TTS-generated speech, which may not fully capture the nuances of real-world speaker variability. Additionally, the dataset construction could be seen as limited by the number of voice variants synthesized for each condition, which may affect the generalizability of the findings. The evaluation is also confined to nine models, which may not represent the full spectrum of existing MLLMs.
Broader impact
This work has significant implications for the development of fairer and more robust AI systems, particularly in multilingual and multimodal contexts. By highlighting the biases present in speech-integrated LLMs, the study encourages further research into mitigating these biases, which is crucial for applications in education, conversational AI, and automated systems that interact with diverse populations. The establishment of a benchmark dataset like BiasInEar can serve as a foundation for future studies in this area. This paper presents the first systematic study of speech bias and robustness in multilingual MLLMs. It contributes significantly to the understanding of how demographic and structural factors influence model performance, establishing a framework for future research in fairness and robustness in speech technologies.
Author of this paper? Send a correction.