HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Abstract
Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of HeadRouter, a dynamic head-weight routing mechanism for audio token pruning in large audio language models, which significantly enhances performance and efficiency in processing diverse audio tasks. This work represents a meaningful advancement in the field of audio language models, addressing critical challenges in token management and model efficiency while maintaining high performance across various audio tasks.
Methodology
The proposed HeadRouter method introduces a novel dynamic head-weight routing mechanism that adapts to the varying importance of attention heads in large audio language models (LALMs). This approach is innovative in its use of entropy-based selectivity scores and Gaussian soft mixing to create task-specific head-weight profiles. The training-free nature of the method allows it to be easily integrated into existing models without additional training overhead, which is a significant advantage for practical applications.
Results & evaluation
The experiments conducted on the AudioMarathon and MMAU-Pro benchmarks demonstrate the effectiveness of HeadRouter in outperforming existing token pruning methods across various audio tasks. The results indicate that the method not only maintains performance while aggressively pruning tokens but also adapts well to different audio contexts, showcasing its robustness. The comparative analysis with state-of-the-art methods further validates the proposed approach's superiority in managing token importance dynamically.
Reproducibility
The paper provides a clear description of the methodology, including the routing mechanism and evaluation setup, which supports reproducibility. However, the lack of publicly available code or detailed implementation guidelines may hinder full reproducibility for other researchers.
Limitations
One limitation is the reliance on pre-calibrated head-weight profiles, which may not generalize across all audio tasks or models. Additionally, while the method shows promise in reducing computational costs, the paper does not explore the implications of using HeadRouter in real-time applications or its impact on latency in practical deployments.
Broader impact
The implications of this research extend to various applications in audio processing, including speech recognition, music analysis, and multimodal systems. By improving the efficiency of LALMs, this work could facilitate more widespread adoption of advanced audio understanding technologies in real-time applications, enhancing user experiences in voice-interactive systems. The main contribution of this paper is the introduction of HeadRouter, a dynamic head-weight routing mechanism for audio token pruning in large audio language models, which significantly enhances performance and efficiency in processing diverse audio tasks. This work represents a meaningful advancement in the field of audio language models, addressing critical challenges in token management and model efficiency while maintaining high performance across various audio tasks.
Author of this paper? Send a correction.