AudioKV: KV Cache Eviction in Efficient Large Audio Language Models

★ Yuxuan Wang , Peize He, Xiyan Gui ...Full author list in the original paper.

Huazhong University of Science and Technology +3 · Huazhong University of Science and Technology +3

Affiliations: Huazhong University of Science and Technology, Shanghai Jiao Tong University, HKUST (GZ), Xidian University

Paper

Abstract

Large Audio-Language Models (LALMs) have set new benchmarks in speech processing, yet their deployment is hindered by the memory footprint of the Key-Value (KV) cache during long-context inference. While general KV cache compression techniques excel in LLMs, they often fail in the audio domain by overlooking the intrinsic temporal continuity of acoustic signals. To bridge this gap, we propose AudioKV, a novel framework that robustly prioritizes audio-critical attention heads through a hardware-friendly semantic-acoustic alignment mechanism. Specifically, we identify these modality-specialized heads by analyzing attention scores in ASR tasks and dynamically allocate KV cache budgets preferentially to them. Furthermore, we introduce Spectral Score Smoothing (SSS), an FFT-based global filtering strategy designed to suppress high-frequency noise and recover smooth global trends from importance scores, ensuring more balanced token selection with unprecedented precision. Extensive evaluations across multiple LALMs, including Qwen and Gemma series, demonstrate that AudioKV significantly outperforms baselines while enhancing computational efficiency. Notably, at a 40% compression ratio, AudioKV maintains near-full accuracy on Qwen3-Omni-30B with only a 0.45% drop, whereas traditional methods suffer from catastrophic performance degradation and repetition. Our code will be released after acceptance.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The main contribution of this paper is the introduction of AudioKV, a novel framework for efficient KV cache management in audio-language models, which significantly enhances performance while reducing memory usage. This work addresses a critical bottleneck in deploying LALMs and offers a robust solution that combines innovative methodologies with thorough experimental validation, marking a meaningful advancement in the field of machine learning for audio processing.

Methodology

The methodology presented in the paper is innovative, focusing on the unique challenges of Key-Value (KV) cache management in Large Audio-Language Models (LALMs). The authors propose a dual approach that combines audio-aware head allocation with Spectral Score Smoothing (SSS) to enhance the efficiency of KV cache usage. The identification of audio-critical attention heads through attention score analysis is a significant contribution, as it allows for a more nuanced allocation of resources compared to traditional uniform methods. The SSS technique, which employs FFT-based filtering to stabilize importance scores, is particularly noteworthy for its potential to improve performance in dynamic audio contexts.

Results & evaluation

The experiments are comprehensive and demonstrate the effectiveness of AudioKV across multiple benchmarks, including Automatic Speech Recognition (ASR) and Speech Translation (ST). The results show that AudioKV outperforms existing methods significantly, especially at high compression ratios where other methods fail. The use of diverse datasets and models strengthens the validity of the findings, and the detailed performance metrics provide a clear picture of the advantages of the proposed method.

Reproducibility

The paper mentions that the code will be released after acceptance, which is a positive step towards reproducibility. However, the absence of a public demo or project URL limits immediate access to the implementation details. The methodology is described in sufficient detail to allow for replication, but the lack of a publicly available codebase at this time is a drawback.

Limitations

One limitation noted in the paper is the potential for repetition and degeneration in output under high KV cache compression ratios, which could affect the quality of generated text. Additionally, while the method shows promise, its applicability to other modalities beyond audio is not explored, which may limit its generalizability.

Broader impact

The implications of this work are significant for the deployment of LALMs in real-world applications, particularly in resource-constrained environments where efficient memory usage is critical. The techniques developed could lead to advancements in speech recognition and multimodal interactions, potentially enhancing user experiences in various applications such as virtual assistants, transcription services, and interactive audio systems. The main contribution of this paper is the introduction of AudioKV, a novel framework for efficient KV cache management in audio-language models, which significantly enhances performance while reducing memory usage. This work addresses a critical bottleneck in deploying LALMs and offers a robust solution that combines innovative methodologies with thorough experimental validation, marking a meaningful advancement in the field of machine learning for audio processing.

Author of this paper? Send a correction.