SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
Abstract
Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
Methodology
The paper introduces SpeechGym, a novel audio-native agentic environment designed for training voice agents via reinforcement learning. The core methodological contribution is the creation of an end-to-end trainable loop for multi-turn dialogue and tool use in speech, which is claimed to be a first. The environment integrates two omni-modal models (Qwen3-Omni-30B-A3B) – a frozen user model and a trainable agent model – allowing for native audio interaction without external ASR/TTS or proprietary APIs. This local setup is crucial for enabling gradient flow and affordable on-policy RL. The problem is formulated as a POMDP, where the agent decides between structured tool calls (textual) and variable-length speech responses. Keeping tool calls textual is a sensible design choice, isolating perceptual errors from reasoning errors. The environment inherits tasks, tools, databases, and success checks from the `tau2` text benchmark, ensuring modality is the primary variable. A new "Banking" domain is introduced, specifically stressing high-stakes numeric slots, which is a valuable addition for evaluating robustness to perceptual errors. The reward mechanism is based on `tau2`'s outcome check, a product of binary components, ensuring strict task completion. To address the severe sparsity of this reward in audio settings, the paper proposes a per-turn process shaping, crediting successful tool executions and penalizing failures. This is a classical reward shaping technique adapted effectively to the agentic setting. For efficient rollout collection, the authors leverage `vLLM-Omni`, serving both models and the LoRA adapter, which significantly speeds up training epochs and eliminates API costs, making online speech RL practical. The overall methodology is well-conceived, addressing key challenges in audio agent training.
Results & evaluation
The experimental evaluation is comprehensive and well-structured, addressing key questions about the text-to-audio gap, training efficacy, and transferability. The initial diagnosis of the text-to-audio gap is insightful, identifying perceptual errors (mis-heard slot values) and behavioral issues (over-action, dead loops) as dominant failures, rather than reasoning deficits. This qualitative and quantitative analysis
Author of this paper? Send a correction.