When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration
Abstract
When audio and text conflict, speech-enabled language models follow the text 10 times more often than when arbitrating between two text sources, even when explicitly instructed to trust the audio. Using ALME, a benchmark of 57,602 controlled audio-text conflict stimuli across 8 languages, we find that Gemini 2.0 Flash exhibits 16.6\% text dominance under audio-text conflict versus 1.6\% under text-text conflict with identical reliability cues. This gap is not explained by audio quality: audio-only accuracy (97.2\%) exceeds cascade accuracy (93.9\%), indicating audio embeddings preserve more information than text transcripts. We propose that text dominance reflects an asymmetry not in information content but in arbitration accessibility: how easily the model can reason over competing representations. This framework explains otherwise puzzling findings. Forcing transcription before answering increases text dominance (19\% to 33\%), sacrificing audio's information advantage without improving accessibility. Framing text as ``deliberately corrupted'' reduces text dominance by 80\%. A fine-tuning ablation provides interventional evidence: training only the audio projection layer increases text dominance (+26.5\%), while LoRA on the language model halves it ($-$23.9\%), localizing text dominance to the LLM's reasoning rather than the audio encoder. Experiments across four state-of-the-art audio-LLMs and 8 languages show consistent trends with substantial cross-linguistic and cross-model variation, establishing modality arbitration as a distinct reliability dimension not captured by standard speech benchmarks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive study on modality arbitration in audio-LLMs, revealing critical insights into how these models prioritize conflicting audio and text inputs. The innovative methodology and extensive experimental evaluation contribute significantly to the understanding of multimodal interactions in machine learning, particularly in the context of speech recognition and processing.
Methodology
The methodology is robust, introducing a novel benchmark (ALME) with a large dataset (57,602 stimuli) that effectively captures the dynamics of modality arbitration in audio-LLMs. The use of controlled semantic conflicts to measure the Text Dominance Ratio (TDR) is innovative and allows for clear quantification of model behavior under conflicting inputs. The experiments are well-structured, comparing multiple models and languages, which adds depth to the analysis. However, the reliance on specific prompt structures may limit generalizability.
Results & evaluation
The experiments are comprehensive, covering various models and languages, and include a fine-tuning ablation study that provides insights into the mechanisms behind text dominance. The results demonstrate significant findings, such as the 10% arbitration gap and the effects of prompt framing on TDR. The cross-linguistic analysis is particularly valuable, revealing how different languages affect model behavior, which is crucial for multilingual applications.
Reproducibility
The paper provides sufficient detail regarding the experimental setup, including the dataset and evaluation protocols, which supports reproducibility. The use of a publicly available GitHub repository for the benchmark enhances the potential for others to replicate the study. However, the specific models and their versions used could be better documented to ensure full reproducibility.
Limitations
The study has several limitations, including its focus on lexical and semantic conflicts without addressing prosodic or non-speech audio conflicts. The human validation process involved a limited number of native speakers, which may affect the validity of the stimuli. Additionally, the findings may not generalize to real-world applications where audio and text inputs are more complex. The potential overlap in training data for the models evaluated could also confound results.
Broader impact
The findings have significant implications for the design and deployment of audio-LLMs in real-world applications, particularly in multilingual contexts. Understanding how models arbitrate between audio and text can inform better user interfaces and improve the reliability of speech-enabled systems. The work highlights the need for careful consideration of modality arbitration in AI systems, which could lead to more effective and inclusive technologies. The paper presents a comprehensive study on modality arbitration in audio-LLMs, revealing critical insights into how these models prioritize conflicting audio and text inputs. The innovative methodology and extensive experimental evaluation contribute significantly to the understanding of multimodal interactions in machine learning, particularly in the context of speech recognition and processing.
Author of this paper? Send a correction.