Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
Abstract
Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Methodology
The paper employs a rigorous diagnostic methodology to localize the source of "acoustic underuse" in audio-LLMs. It systematically compares ASR-supervised encoders (Whisper) against reconstruction-based codecs (EnCodec, DAC-VAE, WavTokenizer) within a controlled shared-LM pipeline (Qwen3.5-4B). The core methodological strength lies in the multi-stage information tracing: using linear probes, geometric distance ratios, and LogitLens analyses to track task-relevant information from the encoder through the projector and LM layers to the final output head. The causal intervention of fine-tuning only the choice-letter rows of the LM head ($W_U$) is a clever and effective way to isolate readout failure from representation loss.
Results & evaluation
The experiments are extensive, covering ASR, emotion recognition, and sound captioning across multiple datasets (LibriSpeech, IEMOCAP, FSD50K, NSynth, etc.). The results clearly demonstrate that while Whisper encoders perform best on end-to-end tasks, they do not necessarily contain more recoverable acoustic information than reconstruction-based codecs at the latent level. The finding that linear probe accuracy at the final LM layer significantly exceeds MCQA accuracy (by up to 83 points) provides strong evidence for a readout bottleneck. The recovery of 30-42 points in MCQA accuracy by updating only ~10-26k scalars in the LM head is a compelling result.
Reproducibility
The paper provides detailed implementation specifics, including model checkpoints, projector architecture, training hyperparameters, and dataset splits. The use of standard open-source components (Whisper, EnCodec, Qwen) and clear descriptions of the probing and intervention protocols make the work highly reproducible.
Limitations
The analysis is primarily focused on MCQA-format classification tasks. While the authors acknowledge this, the generalization of the "readout bottleneck" finding to free-form generative tasks (like open-ended captioning or dialogue) is not fully established. The LM scale is limited to 4B parameters, so it is unclear if the bottleneck persists or shifts in larger models. The study excludes self-supervised encoders (like WavLM or HuBERT), which are known to retain strong paralinguistic features, limiting the scope of the encoder comparison.
Broader impact
This work provides a crucial diagnostic framework for the audio-LLM community. By identifying that the failure often lies in the LM's ability to map internal acoustic representations to text tokens (readout) rather than in the encoder's ability to capture them, it shifts the focus of future research toward better alignment strategies, label-aware training, or readout regularization. It challenges the common assumption that swapping ASR encoders for codec-based ones is a sufficient solution for improving non-lexical audio understanding. The paper identifies that acoustic underuse in audio-LLMs is primarily a readout alignment failure rather than an encoder-side information loss, demonstrating that task-relevant acoustic structure remains recoverable in the LM's final hidden states even when the model fails to verbalize it correctly. Through a combination of encoder replacement experiments, layer-wise probing, and targeted LM head interventions, the study provides a detailed localization of this bottleneck, offering actionable insights for improving the acoustic grounding of language models.
Author of this paper? Send a correction.