Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
Abstract
Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
Methodology
The paper proposes "Macer," a training-free token compression method for OmniLLMs (Omnimodal Large Language Models). The core theoretical contribution is the identification of a structural flaw in existing shared-decoder compression methods: the coupling of cross-modal capacity allocation and within-modality token selection via a single shared attention score. The authors argue that raw shared attention inherently favors audio tokens due to architectural priors, leading to suboptimal video retention. Macer addresses this by decoupling these two decisions. It first explicitly allocates a budget for audio and video tokens based on a user-defined ratio. Then, it performs modality-local ranking using "Allocation-Normalized Modality Scoring" (ANMS), which normalizes attention scores within each modality to remove the cross-modal mass bias. Furthermore, it leverages the observation that audio and video saliency become readable at different shallow layers, applying pruning at modality-specific depths ($L_a$ for audio, $L_v$ for video). A lightweight temporal coverage term is added for video to prevent temporal clustering. The approach is inference-time only, requiring no training or model modification.
Results & evaluation
The evaluation is conducted on three OmniLLM backbones: Qwen2.5-Omni-7B, Qwen2.5-Omni-3B, and OmniVinci-9B. Benchmarks cover audio-grounded, audio-video joint, visual-dominant, and video-centric tasks. The results claim significant performance preservation at low retention ratios (e.g., 98.7% of full performance at 25% retention on Qwen2.5-Omni-7B). The paper compares Macer against ToMe-style merging and random dropping, demonstrating that explicit budget allocation outperforms implicit budgeting induced by shared ranking. The ablation studies support the three key observations: (1) shared attention is not neutral, (2) budgeting is the primary driver of cross-modal balance, and (3) modality-specific readout layers are beneficial. The results are strong and directly validate the proposed mechanism.
Reproducibility
The paper provides detailed mathematical formulations for the ANMS scores and the budget allocation logic. It specifies the hyperparameters used (readout layers, budget ratios). Since the method is training-free and applied to open-source models (Qwen2.5-Omni, OmniVinci), reproduction is feasible for researchers with access to the necessary compute. The paper mentions an appendix with additional details, which is standard for reproducibility.
Limitations
The method relies on specific architectural assumptions of shared-decoder OmniLLMs where audio and video tokens are interleaved and processed by the same attention mechanism. Its applicability to encoder-decoder architectures or models with separate modality-specific encoders that feed into a unified decoder might require adaptation. The "training-free" nature means it cannot learn task-specific compression patterns that a trained compressor might, potentially capping its maximum efficiency compared to learned methods. The paper notes that the temporal coverage term is lightweight but does not fully address complex long-range video dependencies if the budget is extremely tight.
Broader impact
This work addresses a critical bottleneck in deploying OmniLLMs: inference cost and latency. By enabling efficient compression without retraining, it lowers the barrier to using these models in resource-constrained environments. The insight into modality bias in shared attention mechanisms is broadly applicable to the design and analysis of multimodal models, potentially influencing future model architectures to be more modality-agnostic or to handle allocation explicitly. The paper makes a significant contribution to multimodal efficiency by identifying and correcting a fundamental mis-specification in how shared-attention mechanisms handle token compression, proposing a theoretically grounded, training-free solution that significantly improves the efficiency-accuracy trade-off for OmniLLMs.
Author of this paper? Send a correction.