Extending Audio Context for Long-Form Understanding in Large Audio-Language Models
Abstract
Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, audio-only extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training and improving robustness for long-context audio understanding. Our experiments on SALMONN and Qwen2-Audio show that Partial YaRN outperforms the original models across wide range of settings, and VLAT training strategy provides substantial improvement, achieving strong performance on long audio of unseen lengths.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Partial YaRN and VLAT, which effectively extend the audio context windows of LALMs, improving their performance on long-form audio understanding tasks. This work represents a meaningful advancement in the field of audio-language models, providing a robust framework for future research and applications.
Methodology
The paper introduces two novel methods, Partial YaRN and Virtual Longform Audio Training (VLAT), which effectively extend the audio context windows of Large Audio-Language Models (LALMs). Partial YaRN is a training-free approach that modifies only the audio token positions, preserving the text capabilities of the base model. VLAT enhances this by simulating diverse audio lengths during training, allowing for better generalization. The methodology is well-structured, with clear definitions and justifications for the proposed techniques, although it could benefit from more extensive comparisons with existing methods.
Results & evaluation
The experiments conducted on SALMONN and Qwen2-Audio demonstrate the effectiveness of the proposed methods across various settings. The results show that Partial YaRN consistently outperforms the original models, and VLAT significantly improves performance on long audio inputs. The dataset construction and evaluation metrics are clearly defined, although the focus on multiple-choice question answering (MCQA) limits the generalizability of the findings to other tasks.
Reproducibility
The paper provides sufficient details about the implementation of the proposed methods, including hyperparameter settings and the experimental setup. However, the lack of a comprehensive description of the dataset construction process and the absence of a demo URL may hinder full reproducibility. The GitHub repository for the code is a positive aspect, but additional documentation would enhance clarity.
Limitations
The paper acknowledges that its evaluation is limited to MCQA tasks, which may not capture the full spectrum of capabilities of the LALMs. Additionally, the hyperparameter tuning for Partial YaRN introduces a computational burden, and the performance of the methods is model-dependent, suggesting that further exploration is needed for broader applicability.
Broader impact
The proposed methods have significant implications for improving long-form audio understanding in LALMs, which can enhance applications in areas like speech recognition, audio transcription, and multimedia content analysis. By addressing the limitations of existing models, this work contributes to the advancement of multimodal AI systems. The main contribution of this paper is the introduction of Partial YaRN and VLAT, which effectively extend the audio context windows of LALMs, improving their performance on long-form audio understanding tasks. This work represents a meaningful advancement in the field of audio-language models, providing a robust framework for future research and applications.
Author of this paper? Send a correction.