Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
Abstract
Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of VoxParadox, a benchmark that effectively isolates and evaluates the paralinguistic understanding of Audio LLMs, alongside innovative methods to enhance model performance in this domain. The work is significant as it addresses a critical gap in the capabilities of current Audio LLMs and proposes actionable solutions that could lead to more robust multimodal systems.
Methodology
The paper introduces VoxParadox, a novel adversarial benchmark designed to evaluate the paralinguistic understanding of Audio LLMs by creating controlled linguistic-acoustic contradictions. The methodology is robust, employing a systematic approach to generate adversarial examples and utilizing layer-wise probing to diagnose model limitations. The proposed Prompt-Conditioned Layer Mixer (PCLM) is a significant innovation that adaptively combines information from multiple audio layers based on the input prompt, addressing identified bottlenecks in model performance.
Results & evaluation
The experiments are comprehensive, evaluating a diverse set of Audio LLMs against the VoxParadox benchmark. The results demonstrate a clear performance gap in paralinguistic tasks, with models showing a tendency to rely on transcript-implied answers rather than acoustic evidence. The paper provides detailed metrics, including ground truth accuracy and adversarial-label agreement, which effectively illustrate the models' weaknesses and the improvements achieved through the proposed methods.
Reproducibility
The paper includes sufficient detail regarding the experimental setup, data generation pipeline, and evaluation metrics, which supports reproducibility. However, the implementation specifics of the PCLM and DPO methods could benefit from additional clarity to ensure that other researchers can replicate the results accurately.
Limitations
The authors acknowledge that PCLM is a post-hoc solution and that the degradation of paralinguistic information in deeper layers and at the encoder-LLM interface presents inherent limitations. Additionally, while VoxParadox serves as a controlled stress test, it may not fully capture the complexities of naturalistic speech scenarios. The reliance on TTS-generated audio also raises questions about the generalizability of the findings.
Broader impact
The research has significant implications for improving speech-based interfaces and accessibility technologies, enhancing the ability of Audio LLMs to interpret non-verbal cues accurately. However, the potential for misuse in profiling and surveillance contexts necessitates careful consideration of ethical implications and the establishment of safeguards in deployment. The main contribution of this paper is the introduction of VoxParadox, a benchmark that effectively isolates and evaluates the paralinguistic understanding of Audio LLMs, alongside innovative methods to enhance model performance in this domain. The work is significant as it addresses a critical gap in the capabilities of current Audio LLMs and proposes actionable solutions that could lead to more robust multimodal systems.
Author of this paper? Send a correction.