Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
Abstract
Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Methodology
The paper proposes "AudioRubrics," a reinforcement learning framework designed to enhance the reasoning capabilities of Large Audio-Language Models (LALMs). The core innovation lies in replacing static, hand-crafted process rewards with self-evolving, audio-grounded rubrics. The methodology involves three key steps: (1) Initializing rubrics from raw audio waveforms using an audio-capable model to ensure grounding; (2) Evolving these rubrics during training by eliciting new criteria from the model's own rollouts, pruning non-discriminative ones (those with zero variance in judgments across a group), and re-weighting the survivors; (3) Combining these evolving rubric rewards with outcome accuracy and an overthinking penalty (to prevent verbose, hallucinated reasoning) within a Group Relative Policy Optimization (GRPO) framework. The approach addresses the saturation of static rewards and the lack of audio grounding in text-based process rewards. The technical design is sound, leveraging the heterogeneity of rollouts to generate adaptive evaluation criteria.
Results & evaluation
The authors evaluate AudioRubrics on three benchmarks: MMAU, MMAR, and MMSU. They compare against a wide range of proprietary (GPT-4o-Audio, Gemini) and open-source baselines (Qwen2-Audio, Phi-4-Multimodal, etc.), as well as other RL-based reasoning methods. The results indicate substantial improvements over baselines, particularly in multi-step reasoning tasks. The paper includes analysis showing that the rubrics evolve to become more sophisticated as training progresses, and that the gains scale with the capability of the rubric generator. The inclusion of an overthinking penalty is shown to stabilize reasoning length. The evaluation is comprehensive and covers diverse audio domains (speech, sound, music).
Reproducibility
The paper provides a GitHub repository link and a Hugging Face collection for model weights and datasets. The methodology is described in detail, including the GRPO objective and the specific steps for rubric elicitation and filtering. However, the exact hyperparameters for the rubric generation model, the specific taxonomy of initial dimensions, and the precise implementation details of the "negative criteria" elicitation might require careful reading of the appendix (not fully provided here, but implied). The code availability significantly aids reproducibility.
Limitations
The approach relies heavily on the quality of the rubric generator/judge model. If this model is biased or lacks audio understanding, the evolving rubrics may propagate errors. The method introduces additional computational overhead during training due to the continuous generation and evaluation of rubrics. The paper mentions that static criteria saturate, but does not extensively discuss the potential for "rubric collapse" where the model optimizes for a narrow set of evolving criteria that do not generalize. The reliance on an external judge model for rubric generation and judging introduces a dependency that might not be available in all settings.
Broader impact
This work contributes to the development of more robust and interpretable audio reasoning systems. By grounding reasoning in audio evidence and providing dense process-level feedback, it reduces hallucinations and improves the reliability of AI systems in critical applications like healthcare and multimedia analysis. The self-evolving nature of the rewards could inspire similar adaptive reward mechanisms in other multimodal reasoning tasks. AudioRubrics introduces a novel reinforcement learning framework that utilizes self-evolving, audio-grounded rubrics to provide dense, adaptive process-level rewards for audio reasoning, significantly outperforming static reward baselines on multiple benchmarks.
Author of this paper? Send a correction.