Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different set
Abstract
Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16.7% a guess would give. Adding a fixed bias to the attention logits of a hundred heads, under a tenth of the model's and with no training, redirects the description to whichever speaker we choose, on 90.7% to 99.0% of trials. Those heads are largely not specific to audio. Rank the text-only language model an audio model was built from, or a released model of the same family, on a written version of the task, take its top hundred heads, and carry them over unchanged: they redirect the audio model on 80.8% to 95.0% of trials, with nothing about audio entering the selection. The audio and text head sets share 66 to 74 of 100 where chance would give about 20, and the shared part alone reproduces almost all of the steering. What that does not show is that sharing is what makes the heads work: an equal-sized draw from the same discovered hundred does nearly as well, and none of our three models separates the two explanations. A second finding concerns how such heads are found. Ranking heads by how much attention they place on the segment asked about, as an established score does, or by how much of their attention moves with the question, as a per-head normalised variant does, gives top hundreds that share 69, 37 and 4 heads across our three models. In Ultravox, where they share 4, the established score's heads leave output the judge cannot place on any segment on 69.7% of trials, against 40.0% with no intervention and 1.0% for the normalised variant. That is one arm of six; on the other five the established score steers above a random draw.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
Methodology
The paper proposes a rigorous mechanistic analysis of audio language models (ALMs) using attention head interventions. The core method involves identifying "tracking heads" that attend to specific speaker segments and applying a fixed bias to their attention logits to steer the model's output. The novelty lies in the comparison between heads selected via audio-specific metrics (mass vs. selectivity) and heads selected purely from the text-only backbone. The authors demonstrate that text-derived head rankings can effectively steer audio models, suggesting that speaker tracking mechanisms are inherited from the language backbone rather than being modality-specific. The methodology is sound, utilizing causal interventions (biasing logits) rather than mere correlation, and includes careful controls for random head selection and layer matching.
Results & evaluation
The experiments are conducted on three distinct ALMs (Qwen2-Audio, Ultravox, SALMONN) using a custom dataset of 6-speaker recordings. The evaluation metrics include accuracy of speaker attribution and "unattributable rate" (abstention). The results show that the proposed intervention significantly improves attribution accuracy (from ~10% to >90%) and that text-derived heads are largely sufficient for this task. The comparison between mass and selectivity rankings reveals that the standard "mass" score can lead to unattributable outputs in some models, highlighting the importance of the selectivity metric. The inclusion of random controls and layer-matched baselines strengthens the causal claims.
Reproducibility
The paper provides detailed descriptions of the intervention (bias magnitude, token spans) and the scoring metrics. However, the specific implementation details for the "lexical judge" and the exact prompt templates are not fully detailed in the provided text, which might hinder exact reproduction. The use of 4-bit quantization is noted as a potential threat to fidelity, which is a good transparency point. The code and data availability are not explicitly stated in the provided text, but the methodology is described with enough precision for a skilled researcher to attempt reproduction.
Limitations
The primary limitation is the reliance on a lexical judge for evaluation, which may not capture all nuances of speaker attribution. The paper acknowledges that the "sharing" of heads between text and audio models does not prove causality of the sharing itself, as random draws from the same set also work. The analysis is limited to three models, and the generalizability to other architectures or tasks is not fully established. The use of a fixed bias magnitude ($B=10^4$) for comparability might not reflect natural model behavior, though $B=5$ is also tested.
Broader impact
This work has significant implications for the interpretability of multimodal models, suggesting that high-level reasoning tasks like speaker tracking may rely on pre-existing linguistic structures rather than modality-specific features. This could inform the design of more efficient multimodal models and provide insights into how attention mechanisms generalize across modalities. It also highlights the limitations of standard attention scoring metrics (mass) for causal analysis, advocating for more nuanced metrics like selectivity. The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
Author of this paper? Send a correction.