Audio Token Attention Is Predictable Before the Language Model Runs
Abstract
A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at $ρ\geq .69$ on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a label-free, linear-predictive method for pruning audio tokens in LALMs before the language model runs, significantly improving efficiency and context window capacity. The technical contribution is strong, combining a novel insight about attention predictability with a practical, two-stage pruning algorithm that outperforms existing baselines across multiple models and tasks.
Methodology
The paper proposes "Triage," a two-stage audio token pruning method for Large Audio Language Models (LALMs). The core innovation is the discovery that the attention an audio token will receive across the entire language model (LM) is linearly predictable from the encoder output alone, before the LM runs. The authors fit a simple linear map (the "prior") in closed form using ridge regression on unlabelled data. This prior is used in Stage 1 to prune tokens before the LM prefill, significantly reducing context window usage. Stage 2, applied to multiple-choice tasks, refines the ranking using attention observed at layer 2, combining the prior's prediction with observed attention via a "precision fusion" mechanism. The method is label-free for calibration, using self-consistency with the model's own full-audio outputs to set compression budgets. This approach is distinct from existing methods that rely on acoustic energy, position, or mid-prefill attention, which the paper demonstrates are weak predictors of final attention for audio tokens.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors test the method on 13 different LALMs from 5 families, demonstrating the generality of the linear prior (achieving Spearman correlation >= 0.69 on 11/13 models). They evaluate on both transcription (LibriSpeech, FLEURS, TEDLIUM) and multiple-choice (MMSU, DREAM, AudioMarathon-RACE) benchmarks. The results show that Triage outperforms strong baselines like DART, FastV, and HeadRouter, particularly at aggressive compression ratios. A key strength is the efficiency analysis, showing that Triage allows 4x more concurrent streams on a single GPU and extends the effective context window from ~22 to ~62 minutes for Qwen2.5-Omni-3B. The ablation studies convincingly show that both the prior and the stage-2 refinement contribute to performance.
Reproducibility
The paper provides high reproducibility. The method relies on a closed-form linear fit, which is computationally trivial and easy to implement. The authors specify the hyperparameters (e.g., ridge regression lambda=10) and the calibration procedure in detail. The project page is provided, and the use of standard benchmarks and open-source models (Qwen, Voxtral, Phi-4) facilitates replication. The "screen" for identifying models where the prior fails is also described clearly.
Limitations
The primary limitation is that the linear prior fails on two specific models (Qwen-Audio variants), although the authors provide a diagnostic screen to identify such cases. The method is designed for pruning before the LM runs, so it does not address KV-cache eviction during the decoding phase, which the authors note as future work. The performance gains on multiple-choice tasks are modest in absolute terms (e.g., +0.043 accuracy over DART), though statistically significant.
Broader impact
This work has significant practical impact for deploying audio LLMs in resource-constrained environments. By enabling longer audio inputs within fixed context windows and increasing throughput, Triage makes real-time or long-form audio understanding more feasible. The finding that attention is linearly predictable from encoder outputs is a valuable insight that could inform other efficiency techniques in multimodal models. The paper introduces a label-free, linear-predictive method for pruning audio tokens in LALMs before the language model runs, significantly improving efficiency and context window capacity. The technical contribution is strong, combining a novel insight about attention predictability with a practical, two-stage pruning algorithm that outperforms existing baselines across multiple models and tasks.
Author of this paper? Send a correction.