SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
Abstract
Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $τ$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $τ$-Voice with an acceptable increase in the delegation rate.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SALMONN-duo introduces an adaptive dual-system voice agent that balances real-time responsiveness with deep reasoning via knowledge-boundary-aware training and cost-aware reinforcement learning. The paper presents a rigorous methodology for training a full-duplex frontend to delegate tasks to an asynchronous backend only when necessary, demonstrating superior performance-cost trade-offs and improved safety in complex, multi-turn, environment-grounded tasks compared to existing full-duplex voice agents.
Methodology
The paper proposes SALMONN-duo, a dual-system architecture for full-duplex voice agents. The core methodological contribution is the separation of a lightweight, always-on full-duplex frontend (System 1) from a powerful, asynchronous backend agent (System 2). The novelty lies in the "adaptive delegation" mechanism, where System 1 learns to decide when to answer directly versus when to delegate to System 2. This is achieved through two key training strategies: (1) Knowledge-Boundary-Aware Supervised Fine-Tuning (SFT), which uses the model's own rollout correctness to label delegation targets, and (2) Cost-Aware Group Relative Policy Optimization (GRPO), which introduces a reward function that balances task success, safety (hallucination/policy violation), and the cost of unnecessary delegation. The design is practical, addressing the latency-cost trade-off inherent in real-time voice interactions. The use of a delegation token within the response stream rather than a separate head is a specific architectural choice validated in the paper.
Results & evaluation
The experimental evaluation is comprehensive, covering single-turn QA, multi-turn daily conversations, and complex environment-grounded tasks (using a customized $\tau$-Voice benchmark). The paper compares against recent full-duplex and half-duplex baselines (Moshi, VoiceChat, etc.). Results show that SALMONN-duo achieves a better performance-cost trade-off, with lower invocation rates on easier tasks and higher accuracy on complex reasoning tasks compared to baselines that either always invoke or rarely invoke tools. The inclusion of safety rewards in the RL phase demonstrates a reduction in hallucinations and policy violations. The evaluation of interruption handling (barge-in) is a strong point, showing the model can maintain conversational context while waiting for backend responses.
Reproducibility
The paper provides detailed descriptions of the data generation pipeline, including the roles of user, assistant, and supervisor simulators. It specifies model architectures (Llama-3.1-8B, CosyVoice2), training hyperparameters (learning rates, batch sizes, GPU counts), and reward weights. However, the reliance on proprietary models (GPT-4o, GPT-5.2, OpenAI TTS) for simulation and evaluation limits full open-source reproducibility. The code and specific weights are not explicitly linked in the provided text, though the paper claims to be open-source work.
Limitations
The system relies on a strong proprietary backend (GPT-5.2) for the "System 2" component, which may not be accessible to all researchers or deployable in privacy-sensitive on-device scenarios. The evaluation assumes ground-truth dialogue history for the backend in some settings, which may overestimate performance in noisy real-world ASR conditions (though ASR sensitivity is analyzed). The "cost" is defined primarily as the number of backend invocations, not necessarily computational cost or latency in a strict sense, though latency is simulated.
Broader impact
This work is significant for the development of practical, low-latency voice assistants that can handle complex tasks without sacrificing conversational fluidity. The dual-process approach offers a scalable path for integrating heavy reasoning capabilities into real-time speech interfaces. It addresses a critical gap in current voice agents: the inability to efficiently manage the trade-off between responsiveness and deep reasoning/tool use. SALMONN-duo introduces an adaptive dual-system voice agent that balances real-time responsiveness with deep reasoning via knowledge-boundary-aware training and cost-aware reinforcement learning. The paper presents a rigorous methodology for training a full-duplex frontend to delegate tasks to an asynchronous backend only when necessary, demonstrating superior performance-cost trade-offs and improved safety in complex, multi-turn, environment-grounded tasks compared to existing full-duplex voice agents.
Author of this paper? Send a correction.