JoyAI-Talker: Full-Duplex Speech Interactive Large Model Built for Empathetic Voice Agents
Abstract
We present JoyAI-Talker, a full-duplex speech dialogue system that delivers robust foundation model capabilities while empowering empathetic interaction and voice agent intelligence. JoyAI-Talker adopts a modular Thinker-Talker architecture and further implements a unified speech-text joint training pipeline to mitigate the common "cognitive degradation" bottleneck, thereby largely preserving the model's core textual reasoning, STEM, and logical capabilities while extending them to speech-based interaction. For expressive speech synthesis, the Talker module employs a text-controllable generation paradigm that enables natural-language instructions to flexibly control vocal attributes and localized paralinguistic events, such as laughter and sighs, supporting more expressive and fine-grained speech responses. To enhance conversational empathy, we introduce the Persona-Adaptive Empathetic Response (PAER) framework. PAER employs a hierarchical cognitive pipeline to extract non-verbal speaker cues, such as gender, age, and emotional state, from raw input audio, incorporate them into the Thinker's CoT reasoning, and generate context-adaptive responses that align semantically appropriate text with fine-grained control over utterance-level expressiveness and localized paralinguistic events, including sighs, speaking rate, and volume. We further integrate Joy-Duplex, a state-driven, plug-and-play full-duplex framework that functions as an efficient gating engine for real-time turn control. Extensive evaluations show that JoyAI-Talker achieves highly competitive performance on foundational T2T and S2T benchmarks. In full-duplex evaluation, the system reaches a high response rate of 0.88 under user interruptions while maintaining an extremely low false-trigger rate under background speech, demonstrating its readiness for fluid and natural speech dialogue.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
Methodology
The paper proposes JoyAI-Talker, a full-duplex speech dialogue system built on a decoupled "Thinker-Talker" architecture. The Thinker is a 48.9B parameter MoE LLM (JoyAI-LLM Flash) adapted for speech via a projector, while the Talker is a speech generation model based on JoyVoice. Key methodological contributions include: 1) A unified speech-text joint training pipeline starting from mid-training to mitigate "cognitive degradation" in LLMs; 2) The Persona-Adaptive Empathetic Response (PAER) framework, which uses Chain-of-Thought reasoning to extract speaker attributes (gender, age, emotion) from audio and inject them into the response generation; 3) A state-driven full-duplex framework (Joy-Duplex) using a lightweight 1.7B decoder to predict interleaved text and state tokens (e.g., `<|accept|>`, `<|reject|>`) for turn-taking control, rather than relying on energy-based VAD. The approach is technically sound and addresses significant pain points in current LSLM (Large Speech Language Model) systems, particularly the trade-off between reasoning capability and acoustic alignment, and the fragility of full-duplex interaction. However, the core components (MoE LLMs, DiT-based TTS, CoT for empathy) are incremental combinations of existing state-of-the-art techniques rather than fundamentally new algorithmic breakthroughs.
Results & evaluation
The evaluation is extensive and covers multiple dimensions: T2T benchmarks (to verify reasoning retention), S2T benchmarks (ASR, translation, QA), empathy perception (AIR-Bench, MER2025), empathetic dialogue generation (EchoMind), and full-duplex interaction (Full-Duplex-Bench v1.5). The results show competitive performance on T2T and S2T tasks, demonstrating that the joint training did not catastrophically forget textual capabilities. The full-duplex evaluation highlights a strong balance between interruption response (0.88) and low false-trigger rates, which is a practical and significant achievement. The inclusion of a commercial baseline (Gemini 3.1 Live) adds credibility. However, the paper relies heavily on LLM-as-a-Judge for empathy metrics, which can be biased. The specific architectural novelty of the "Thinker-Talker" interface is standard in many recent systems (e.g., Qwen2-Audio, Moshi), though the specific implementation of the state-token gating is a notable engineering contribution.
Reproducibility
The paper provides detailed descriptions of the training stages (Mid-training, Context Extension, SFT, DPO), the model architecture (MoE, MLA), and the data construction strategies (e.g., voice conversion for Talker adaptation). The use of standard frameworks (Megatron-Core) and public benchmarks enhances reproducibility. However, the specific "JoyAI-LLM Flash" backbone and the internal data mixes are proprietary to JD.com, making exact replication impossible without access to their internal models and data. The description of the Joy-Duplex state machine is clear enough for implementation.
Limitations
The decoupled architecture introduces latency overhead compared to unified end-to-end models, as noted by the authors. The reliance on a separate 1.7B model for duplex control adds computational complexity. The empathy framework relies on the Thinker's ability to correctly infer attributes via CoT, which can fail if the audio is noisy or ambiguous, potentially leading to hallucinated persona traits. The "cognitive degradation" mitigation is empirical; while results are good, the theoretical guarantee is absent. Furthermore, the system's performance is heavily dependent on the quality of the ASR transcription passed to the Thinker, creating a potential error propagation bottleneck in the decoupled design.
Broader impact
This work contributes to the development of more natural and robust voice assistants, which has significant implications for human-computer interaction, accessibility, and companion AI. The focus on empathy and full-duplex interaction moves the field closer to seamless human-like dialogue. However, the deployment of such systems raises concerns about privacy (continuous audio processing), bias in empathetic responses (stereotyping based on inferred gender/age), and the potential for deceptive anthropomorphism. The open-weight nature of the underlying components (if released) could accelerate research in this area. [The paper presents JoyAI-Talker, a modular full-duplex speech dialogue system that integrates a large MoE language model with an expressive speech generator, utilizing a novel state-driven gating mechanism and a persona-adaptive empathy framework to achieve robust, low-latency, and emotionally intelligent voice interactions while preserving the reasoning capabilities of the underlying LLM.]
Author of this paper? Send a correction.