AudioRSI: Recursively Self-Improving Audio Reasoners
Abstract
Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
Methodology
The paper proposes AudioRSI, a framework where an Audio LLM acts as both actor and critic within a GRPO (Group Relative Policy Optimization) loop. The core innovation is the "recursive" aspect: the critic is periodically synchronized with the actor's weights, allowing the reward signal to improve as the model improves. The methodology is sound, leveraging the unique advantage of audio modality where the critic can verify acoustic claims against the raw waveform, unlike text-only self-rewarding models. The use of pairwise comparison for process-level rewards is a solid adaptation of DPO-style thinking to RLVR (Reinforcement Learning with Verifiable Rewards).
Results & evaluation
The experiments are conducted on Qwen2.5-Omni-7B using AVQA for training and four OOD benchmarks (MMAU, MMSU, MMAU-Pro, MMAR) for evaluation. The results show consistent improvements over baseline RL methods (Ke-Omni-R) and strong performance against proprietary models like Gemini 2.5 Pro on specific splits. The ablation on synchronization period $T$ is crucial and well-executed, demonstrating the trade-off between stability and reward freshness. The claim that the model develops structured reasoning behaviors is supported by LLM-as-Judge and keyword analysis, though these are somewhat proxy metrics.
Reproducibility
The paper provides detailed algorithmic descriptions and hyperparameters in the appendix. The reliance on specific base models (Qwen2.5-Omni) and datasets (AVQA) makes reproduction feasible for groups with similar compute resources. The code is not explicitly linked in the provided text, which is a minor gap for immediate reproducibility.
Limitations
The approach is computationally expensive due to the dual role of the model (generating and critiquing). The "guarded handoff" mechanism, while practical, adds complexity. The evaluation relies heavily on LLM-as-Judge for reasoning quality, which may introduce bias if the judge model shares biases with the actor. The gains are modest in some domains (e.g., Music), suggesting the method may be more effective for domains with clearer acoustic evidence.
Broader impact
This work pushes the boundary of self-improving AI systems in multimodal domains. It demonstrates that models can provide meaningful process-level supervision for their own reasoning, reducing reliance on external, static reward models. This has implications for scaling audio reasoning capabilities without massive human annotation of reasoning steps. AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
Author of this paper? Send a correction.