Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
Abstract
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns how, when, and whether the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
Methodology
The paper proposes a comprehensive framework for real-time voice agents, structured around three layers: Think (foundation post-training), Act (agentic tool use), and Speak/Coordinate (conversational policy). The "Think" layer introduces a sophisticated post-training pipeline combining Core-Cocktail SFT with a novel "M$^2$-OPD" (Multimodality and Multi-Teacher On-Policy Distillation) strategy. This involves using a Text Teacher and a frozen Audio Reference to supervise student-generated trajectories, effectively transferring text-based reasoning capabilities to native audio models while preserving audio-specific nuances. The "Act" layer is particularly strong, introducing self-evolving executable environments for Group Relative Policy Optimization (GRPO). By using code agents to build, validate, and evolve tasks based on model performance (difficulty gating), the system learns robust tool use, state verification, and grounded progress communication. The "Speak/Coordinate" layer formalizes the decision-making process for when to speak, act, or remain silent, addressing critical issues in full-duplex interaction like background speech handling and turn-taking.
Results & evaluation
The evaluation is extensive, covering intelligence (audio reasoning, multilingual ASR, long-context), action (tool use, retrieval), interaction (persona, empathy, full-duplex behavior), and safety. Key results include a significant improvement in task success on the $\tau$-Voice benchmark (78.4% to 82.0%) and a dramatic reduction in response rate to background speech on Full-Duplex-Bench (73.0% to 13.0%), indicating much better conversational etiquette. The model also shows strong gains in multilingual audio understanding (BBA) and safety metrics. The inclusion of a "Voice Harness" prototype for persistent tasks adds depth to the system-level contribution.
Reproducibility
The paper provides detailed descriptions of the training pipeline, data scales (approx. 1M hours), and evaluation protocols. However, as a technical report from a major industry lab, specific hyperparameters, exact dataset compositions, and code are not fully released, limiting independent reproduction. The reliance on in-house benchmarks (Long-AMC, WebSearch1K, VoiceChat) also restricts external verification.
Limitations
The primary limitation is the lack of open-source code and data, making it difficult for the community to replicate the results. The evaluation relies heavily on in-house benchmarks and automatic judges (e.g., GPT-4o-mini, Qwen-Plus), which may introduce bias. The "Voice Harness" is presented as a prototype and is not fully integrated into the core model evaluation. Additionally, the paper focuses on a specific adaptation of $\tau$-Voice (half-duplex S2T) rather than the official full-duplex S2S protocol, which may limit direct comparability with other full-duplex systems.
Broader impact
This work significantly advances the state-of-the-art in real-time voice assistants by integrating robust agentic capabilities with natural conversational policies. The M$^2$-OPD technique and self-evolving environment for GRPO are valuable contributions to the broader field of multimodal LLM post-training. The focus on safety and reliable interaction (e.g., handling background speech, refusing unsupported requests) is crucial for the deployment of such systems in real-world scenarios. The paper presents a comprehensive framework for reliable agentic voice interaction, combining novel on-policy distillation techniques with self-evolving executable environments for tool use. It demonstrates significant improvements in task success, multilingual understanding, and conversational etiquette, particularly in handling background speech and full-duplex interactions, marking a substantial step forward in the development of practical, safety-conscious voice agents.
Author of this paper? Send a correction.