AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Abstract
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
Methodology
The paper proposes AudioTETRIS, a training-free token compression framework for Large Audio Language Models (LALMs). The core innovation is the derivation of a "phoneme-transition-aware" acoustic prior from the temporal derivative of the audio encoder's hidden states. The authors hypothesize that regions of rapid spectral transition in the encoder space correspond to phonetic transitions, which carry high information density. This prior is fused with a depth-adaptive attention score from the LLM decoder to guide progressive token pruning. The methodology is sound, leveraging existing model components (encoder hidden states, decoder attention) without requiring fine-tuning. The validation of the acoustic prior via correlation with forced alignments and kurtosis analysis is a strong methodological addition that grounds the heuristic in linguistic reality.
Results & evaluation
The experiments are comprehensive, evaluating three distinct backbones (Qwen2-Audio-7B, Phi-4-Multimodal, Audio Flamingo 3) across four diverse tasks (ASR, Emotion Recognition, Audio QA, Translation). The results demonstrate that AudioTETRIS retains near-lossless performance at moderate compression rates and significantly outperforms baselines (SpeechPrune, SparseVLM, LTBM) at aggressive compression rates (75% token removal). The ablation studies effectively isolate the contributions of the acoustic prior and the attention score, confirming their complementary nature. The efficiency analysis provides concrete FLOP and latency savings, validating the practical utility of the method.
Reproducibility
The paper provides high reproducibility standards. It includes detailed algorithmic summaries, specifies the exact layers for pruning and prior computation, and describes the calibration procedure for the crossover layer. The authors commit to releasing the full implementation and evaluation scripts. The use of standard, publicly available datasets and checkpoints further enhances reproducibility.
Limitations
The method requires a one-time calibration pass per backbone to determine the crossover layer, which limits its "zero-shot" applicability to unseen architectures. The acoustic prior assumes specific geometric properties of encoder hidden states that may not hold for all encoder types (e.g., HTSAT-style event encoders). The evaluation is limited to English-source benchmarks, and performance on noisy or multilingual audio remains untested. The progressive pruning within the decoder incurs a slight TFLOP overhead compared to single-shot prefill pruning, though this is offset by accuracy gains.
Broader impact
This work addresses a critical bottleneck in deploying LALMs: the quadratic scaling of self-attention with long audio token sequences. By providing a training-free, plug-and-play solution that preserves fine-grained acoustic details, AudioTETRIS enables more efficient and cost-effective deployment of speech-text models. The insight that encoder representation geometry can serve as a structural prior for compression is broadly applicable to other multimodal settings where modality-specific encoders produce redundant token sequences. AudioTETRIS introduces a training-free token compression framework for LALMs that leverages encoder-state dynamics as a phoneme-transition prior to guide attention-based pruning. The paper demonstrates that preserving acoustically rich transition regions is crucial for maintaining high accuracy under aggressive compression, outperforming existing baselines across multiple backbones and tasks while significantly reducing inference compute.
Author of this paper? Send a correction.