AudioDER: A Deduplication-Enhanced Reasoning Dataset for Post-Training Large Audio-Language Models
Abstract
Large Audio-Language Models (LALMs) have shown strong performance on a wide range of audio understanding tasks, yet they still struggle with complex audio reasoning. A practical way to improve such capabilities is post-training, whose effectiveness critically depends on the quality and diversity of training data. However, existing audio-language datasets often contain substantial redundancy, where many samples are highly similar in acoustic content and thus provide overlapping supervisory signals. Such redundancy not only increases annotation cost, but also limits corpus diversity and reduces the effectiveness of post-training. To address this issue, we propose a redundancy-aware data construction pipeline for building reasoning-oriented supervision for LALMs. Specifically, we first perform acoustic similarity-based deduplication across raw audio datasets to improve corpus diversity. We then integrate existing audio captions and question-answer pairs into a unified multiple-choice format. Based on these unified annotations, we leverage Qwen3-30B to generate chain-of-thought (CoT) rationales for reasoning-oriented supervision. Based on this pipeline, we construct AudioDER, a reasoning-oriented post-training dataset containing approximately 191k samples spanning sound, speech, and music. Each sample consists of an audio clip, a multiple-choice question, four answer candidates, an audio caption, and a CoT rationale. Extensive experiments show that post-training on AudioDER consistently improves the performance of Qwen2-Audio-7B-Instruct on multiple audio reasoning benchmarks, including MMAU-mini, MMSU, and MMAR. We hope AudioDER can serve as a valuable resource for advancing audio reasoning research and the development of more capable LALMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AudioDER, a reasoning-oriented dataset designed to enhance the post-training of large audio-language models through a novel redundancy-aware construction pipeline. This work significantly advances the field by addressing the challenges of dataset redundancy and providing a comprehensive resource for improving audio reasoning capabilities in LALMs.
Methodology
The proposed methodology is robust, focusing on a redundancy-aware data construction pipeline that effectively enhances the quality and diversity of training data for LALMs. The multi-stage process, which includes acoustic similarity-based deduplication, integration of existing annotations, and generation of CoT rationales, is well-structured and addresses key challenges in audio reasoning. The use of Qwen3-30B for rationale generation is particularly innovative, as it combines language understanding with audio processing to create a comprehensive dataset. The methodology is clearly articulated, with a logical flow from data collection to final dataset construction.
Results & evaluation
The experimental evaluation is thorough, demonstrating the effectiveness of the AudioDER dataset through extensive post-training experiments on multiple audio reasoning benchmarks. The results show consistent improvements in performance across various models, indicating that the dataset is not only well-constructed but also impactful in enhancing reasoning capabilities. The benchmarks chosen (MMAU-mini, MMSU, and MMAR) are relevant and challenging, providing a solid basis for evaluating the dataset's effectiveness.
Reproducibility
The paper provides sufficient implementation details, including the architecture used (Qwen2-Audio-7B-Instruct), training parameters, and the experimental setup. However, the lack of a publicly available demo or interactive component limits the ease of reproducibility for external researchers. The open-source nature of the dataset is a positive aspect that encourages further exploration and validation by the community.
Limitations
One limitation is the reliance on existing datasets for annotations, which may introduce biases inherent in those sources. Additionally, while the redundancy filtering process is beneficial, it may inadvertently remove samples that could contribute valuable diversity. The paper does not address potential scalability issues related to the dataset size or the computational resources required for post-training on larger models.
Broader impact
The AudioDER dataset has significant potential for advancing research in audio reasoning and LALMs. By providing a high-quality, structured dataset, it can facilitate the development of more capable audio understanding systems, which could have applications in various fields such as accessibility, education, and entertainment. The emphasis on reducing redundancy also highlights a critical area for improvement in dataset construction practices across machine learning. The main contribution of this paper is the introduction of AudioDER, a reasoning-oriented dataset designed to enhance the post-training of large audio-language models through a novel redundancy-aware construction pipeline. This work significantly advances the field by addressing the challenges of dataset redundancy and providing a comprehensive resource for improving audio reasoning capabilities in LALMs.
Author of this paper? Send a correction.