Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks
Abstract
Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
Methodology
The paper proposes Audio-R1, a reinforcement learning post-training framework that leverages self-supervised audio tasks to generate verifiable rewards. The core innovation is the design of three specific tasks—Identity Detection, Region Inpainting, and Temporal Ordering—that require the model to rely on acoustic evidence rather than linguistic priors. By using Group Relative Policy Optimization (GRPO) with rule-based verifiers, the method avoids the need for human-annotated reasoning traces or learned reward models. The approach effectively bridges the gap between self-supervised representation learning and reasoning-oriented post-training, offering a scalable solution to the data scarcity problem in audio reasoning.
Results & evaluation
The experiments are comprehensive, evaluating four different LALM backbones across three distinct benchmarks (MMAR, MMAU-mini, MMAU-Pro). The ablation studies are particularly strong, demonstrating that multi-task training yields complementary benefits over single-task training. The comparison between SFT and RL is a key strength, showing that while SFT improves task accuracy, it degrades reasoning trace quality, whereas Audio-R1 improves both. The use of an LLM-as-judge to evaluate reasoning quality provides valuable qualitative insights, although it remains a proxy metric.
Reproducibility
The paper provides detailed descriptions of the task construction, reward functions, and training hyperparameters. The inclusion of prompt templates and judge rubrics in the appendix enhances reproducibility. However, the specific implementation details of the audio processing pipeline and the exact composition of the 24K source recordings could be more transparent. The use of standard libraries and frameworks (implied by the use of GRPO and LoRA) suggests reasonable reproducibility for researchers with similar resources.
Limitations
The primary limitation is the reliance on self-supervised tasks that may not fully capture the complexity of open-ended audio understanding. The paper acknowledges that the tasks are designed for specific capabilities (correspondence, completion, ordering) and may not generalize to all forms of audio reasoning. Additionally, the use of an LLM-as-judge for reasoning quality evaluation is a proxy and does not establish causal links between reasoning traces and accuracy gains. The method is also conditioned on the pretrained backbone's capabilities, meaning it cannot develop reasoning skills absent from pretraining.
Broader impact
This work has significant implications for the development of large audio-language models, providing a scalable and efficient method for improving audio reasoning without the need for extensive human annotation. The approach could be extended to other modalities and tasks, potentially leading to more general-purpose multimodal models. The emphasis on verifiable rewards and self-supervised tasks aligns with broader trends in AI research towards more autonomous and scalable learning methods. Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
Author of this paper? Send a correction.