Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering
Abstract
DCASE~2026 Task~5 introduces Audio-Dependent Question Answering (ADQA), which tests whether large audio-language models answer from the audio rather than from textual priors. An Audio-Dependency Filtering (ADF) pipeline combines silent-audio probing, per-option perplexity, a large language model (LLM) commonsense check, and human review to remove items solvable from text alone. The 3000 items that pass form the ADQA-Bench evaluation set, spanning music, speech, and environmental audio. The inaugural edition draws 14 teams and 36 submissions across two tracks defined by total parameter count (up to 100B and under 10B). A Chung-Ang University ensemble of MOSS-Audio-8B-Thinking and Qwen3-Omni-30B reaches the top overall accuracy at \pct{58.33}, and a MOSS-only configuration from the same team leads the sub-10B track at \pct{57.30}. Across the 30 submissions with a comparable development score, evaluation accuracy falls by 11.91 percentage points (pp) on average (median 10.91\,pp) on the hidden evaluation split, which is designed to be harder than the development split. The most common building blocks are: the MOSS-Audio-8B-Thinking backbone (13 of 36 submissions), Low-Rank Adaptation (LoRA) fine-tuning on AudioMCQ-StrongAC, and preference or reinforcement-learning objectives -- Group Relative Policy Optimization (GRPO) in five teams, Group reward-Decoupled Normalization Policy Optimization (GDPO) in two. At test time, prompt engineering is near-universal, and majority or choice-permutation voting is common. Every system misses the same set of 233 evaluation items.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
Methodology
The paper introduces Audio-Dependent Question Answering (ADQA), a novel benchmark designed to mitigate textual priors in Large Audio-Language Models (LALMs). The core methodological contribution is the Audio-Dependency Filtering (ADF) pipeline, a multi-stage process combining silent-audio probing, perplexity filtering, LLM commonsense checks, and human verification to curate a dataset where answers cannot be derived from text alone. The paper also provides a comprehensive analysis of 36 submissions, identifying common techniques such as LoRA fine-tuning, GRPO/GDPO reinforcement learning, and choice-permutation ensembling. While the benchmark creation is methodologically sound, the paper itself is a challenge summary rather than a proposal of a new algorithmic architecture, focusing on empirical observation of existing methods.
Results & evaluation
The evaluation is rigorous in scope, covering 3000 items across music, speech, and environmental audio. The paper highlights a significant performance gap between development and hidden evaluation sets (average drop of ~12 pp), effectively demonstrating the difficulty of the benchmark and the prevalence of overfitting to development artifacts. The analysis of universal failures (233 items missed by all systems) provides valuable insight into current model limitations. The use of multiple tracks (parameter size) allows for a nuanced comparison of efficiency versus performance. However, the lack of ablation studies isolating specific components (e.g., the acoustic tagger vs. ensemble) limits the depth of causal claims regarding what drives success.
Reproducibility
The paper provides detailed descriptions of the ADF pipeline and the training data (AudioMCQ-StrongAC). It lists the base models and techniques used by top teams, offering high transparency. However, as a challenge summary, it does not provide a single unified codebase. Reproducibility depends on the individual team submissions, most of which are open-source. The specific hyperparameters and exact filtering thresholds for the ADF pipeline are described conceptually but may require checking individual team papers for full implementation details.
Limitations
The paper acknowledges that the evaluation set, while filtered, may still contain subtle biases. The performance drop from dev to eval suggests that the development set, while harder than previous benchmarks, is not perfectly representative of the generalization capability required for the hidden test set. Furthermore, the reliance on multiple-choice questions inherently limits the assessment of open-ended audio reasoning capabilities. The "universal failure" set indicates that current LALM backbones have fundamental gaps in audio grounding that no amount of prompt engineering or RL can fully bridge.
Broader impact
This work significantly advances the field of multimodal AI by establishing a more rigorous standard for evaluating audio-language models. By explicitly targeting and removing textual shortcuts, ADQA-Bench pushes the community to develop models that truly understand audio signals rather than exploiting dataset artifacts. This has implications for improving the reliability of audio assistants, accessibility tools, and automated content analysis systems. The findings also highlight the need for better audio encoders and pre-training strategies that integrate acoustic features more deeply with language models. This paper presents the DCASE 2026 Task 5 challenge, introducing the ADQA-Bench, a rigorously filtered benchmark for Audio-Dependent Question Answering that effectively exposes and mitigates textual shortcutting in Large Audio-Language Models, thereby setting a new standard for evaluating genuine audio understanding capabilities.
Author of this paper? Send a correction.