When Scaling Fails: Mitigating Audio Perception Decay of LALMs via Multi-Step Perception-Aware Reasoning
Abstract
Test-Time Scaling has shown notable efficacy in addressing complex problems through scaling inference compute. However, within Large Audio-Language Models (LALMs), an unintuitive phenomenon exists: post-training models for structured reasoning trajectories results in marginal or even negative gains compared to post-training for direct answering. To investigate it, we introduce CAFE, an evaluation framework designed to precisely quantify audio reasoning errors. Evaluation results reveal LALMs struggle with perception during reasoning and encounter a critical bottleneck: reasoning performance suffers from audio perception decay as reasoning length extends. To address it, we propose MPAR$^2$, a paradigm that encourages dynamic perceptual reasoning and decomposes complex questions into perception-rich sub-problems. Leveraging reinforcement learning, MPAR$^2$ improves perception performance on CAFE from 31.74% to 63.51% and effectively mitigates perception decay, concurrently enhancing reasoning capabilities to achieve a significant 74.59% accuracy on the MMAU benchmark. Further analysis demonstrates that MPAR$^2$ reinforces LALMs to attend to audio input and dynamically adapts reasoning budget to match task complexity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a comprehensive evaluation framework and a novel training paradigm to address the challenges of audio perception decay in LALMs. This work significantly advances the understanding of reasoning in audio contexts and proposes actionable solutions to improve model performance, marking a valuable addition to the field of machine learning.
Methodology
The paper introduces a novel framework, CAFE, for evaluating audio reasoning errors in Large Audio-Language Models (LALMs) and proposes MPAR², a two-stage training strategy that combines supervised learning and reinforcement learning to enhance audio perception and reasoning. The methodology is well-structured, addressing the identified issue of audio perception decay during extended reasoning processes. The detailed description of the training phases and the innovative reward mechanisms (perception reward, stepwise perception-reasoning reward, and review-enhanced accuracy reward) provide a comprehensive approach to tackling the problem.
Results & evaluation
The experiments are robust, utilizing multiple benchmarks (MMAU and MMAR) to validate the effectiveness of MPAR². The results demonstrate significant improvements in perception accuracy and reasoning performance, with clear metrics defined for evaluation. The paper includes a thorough analysis of the results, showing how the proposed methods outperform existing models and effectively mitigate perception decay.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including model configurations, training procedures, and evaluation metrics. However, the absence of a demo URL and the reliance on external datasets may hinder full reproducibility. The provided GitHub link for the framework is a positive aspect, as it allows for further exploration of the methodology.
Limitations
While the paper presents a strong framework, it does not address potential scalability issues or the computational costs associated with the proposed methods. Additionally, the focus on audio reasoning may limit the generalizability of the findings to other modalities or tasks. The paper could benefit from a more extensive discussion on the implications of the findings for future research.
Broader impact
The research has the potential to significantly impact the field of audio processing and multimodal learning by enhancing the reasoning capabilities of LALMs. Improved audio perception could lead to advancements in applications such as automated transcription, audio-based question answering, and interactive AI systems that require a nuanced understanding of audio content. The main contribution of this paper is the introduction of a comprehensive evaluation framework and a novel training paradigm to address the challenges of audio perception decay in LALMs. This work significantly advances the understanding of reasoning in audio contexts and proposes actionable solutions to improve model performance, marking a valuable addition to the field of machine learning.
Author of this paper? Send a correction.