AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives
Abstract
Although Large Audio-Language Models (LALMs) deliver state-of-the-art (SOTA) performance, they frequently suffer from hallucinations, e.g. generating text not grounded in the audio input. We analyze these grounding failures and identify a distinct taxonomy: Event Omission, False Event Identity, Temporal Relation Error, and Quantitative Temporal Error. To address this, we introduce the AHA (Audio Hallucination Alignment) framework. By leveraging counterfactual hard negative mining, our pipeline constructs a high-quality preference dataset that forces models to distinguish strict acoustic evidence from linguistically plausible fabrications. Additionally, we establish AHA-Eval, a diagnostic benchmark designed to rigorously test these fine-grained temporal reasoning capabilities. We apply this data to align Qwen2.5-Omni. The resulting model, Qwen-Audio-AHA, achieves a 13.7% improvement on AHA-Eval. Crucially, this benefit generalizes beyond our diagnostic set. Our model shows substantial gains on public benchmarks, including 1.3% on MMAU-Test and 1.6% on MMAR, outperforming latest SOTA methods. The model and dataset are open-sourced at https://github.com/LLM-VLM-GSL/AHA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the AHA framework, which effectively reduces hallucinations in LALMs by employing a novel taxonomy of errors and counterfactual hard negative mining. This work not only enhances the performance of audio-language models but also sets a precedent for future research in the domain, emphasizing the importance of rigorous evaluation and alignment strategies.
Methodology
The paper introduces the AHA framework, which innovatively addresses the issue of hallucinations in Large Audio-Language Models (LALMs) through a systematic taxonomy of errors and the use of counterfactual hard negative mining. The methodology is well-structured, combining dataset construction with a diagnostic benchmark (AHA-Eval) to rigorously evaluate the model's performance. The approach of leveraging negative examples to enhance model alignment is particularly noteworthy, as it challenges the traditional reliance on positive reinforcement alone.
Results & evaluation
The experimental results demonstrate significant improvements in the Qwen-Audio-AHA model, with a 13.7% reduction in hallucination rates on the AHA-Eval benchmark and consistent performance gains on public datasets. The comprehensive evaluation across multiple benchmarks and the clear presentation of results enhance the credibility of the findings. However, the paper could benefit from more detailed comparisons with other state-of-the-art models to contextualize the improvements.
Reproducibility
The implementation details are adequately provided, including hyperparameters and training configurations. The open-sourcing of the model and dataset is a strong point for reproducibility, allowing other researchers to validate and build upon the work. However, more explicit instructions on the setup and execution of experiments would further enhance reproducibility.
Limitations
The paper acknowledges limitations related to the evaluation protocols, particularly the rigid sensitivity of current models in judging semantic equivalence. This suggests that the reported hallucination rates might be inflated due to false positives. Additionally, the focus on fine-grained reasoning might overlook broader contextual understanding, which could limit the applicability of the findings in more complex scenarios.
Broader impact
The proposed AHA framework has the potential to significantly advance the field of audio-language understanding by providing a robust mechanism to mitigate hallucinations, thus improving the reliability of LALMs in real-world applications such as multimedia forensics and acoustic monitoring. The emphasis on fine-grained reasoning could also inspire further research into more nuanced evaluation metrics for multimodal models. The main contribution of this paper is the introduction of the AHA framework, which effectively reduces hallucinations in LALMs by employing a novel taxonomy of errors and counterfactual hard negative mining. This work not only enhances the performance of audio-language models but also sets a precedent for future research in the domain, emphasizing the importance of rigorous evaluation and alignment strategies.
Author of this paper? Send a correction.