Realtime-Venus: A full-duplex interaction system with asynchronous delegation
Abstract
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
Methodology
The paper proposes Realtime-Venus, a system built on the MiniCPM-o 4.5 architecture, featuring two 9B models (Omni and Audio) for full-duplex interaction. The core methodological contribution is the "Dual-Loop" runtime: a foreground loop for latency-sensitive perception and speech generation, and a background "Harness" loop for asynchronous tool execution and reasoning. The models are trained to emit private ` ` tokens, which are intercepted by the harness to execute tasks without interrupting the user-facing speech stream. The training data pipeline is a significant contribution, synthesizing 2.8M samples that explicitly couple conversational control (backchannels vs. interruptions) with delegation workflows on a shared causal timeline. The inclusion of a training-free long-video memory module using motion-compensated prediction for gating and MaxSim for retrieval is a practical engineering addition, though less novel than the core interaction framework.
Results & evaluation
The evaluation is extensive, covering streaming video understanding (StreamingBench, OVO-Bench), audio understanding (MMAU, MMAU-Pro), and specifically full-duplex metrics (Full-Duplex-Bench v1.5). The results show Realtime-Venus-Omni leading on 6/8 video benchmarks and Realtime-Venus-Audio performing competitively on audio tasks. Notably, the full-duplex metrics show high continuation rates (97% under backchannels), demonstrating the system's ability to distinguish between non-interruptive and interruptive speech. However, the comparison is limited to a specific set of online models, and the "delegation" success is measured via an internal benchmark, limiting external validation of the tool-use component.
Reproducibility
The paper provides detailed descriptions of the architecture, data pipeline, and training recipe. However, as a system paper from a large industry lab (Ant Group), specific hyperparameters, exact data construction scripts, and the code for the "Harness" runtime are not fully detailed in a way that would allow independent reproduction without significant reverse engineering. The reliance on the open-source MiniCPM-o 4.5 base model helps, but the proprietary data synthesis pipeline is a black box.
Limitations
The primary limitation is the lack of open-source code and data for the specific Realtime-Venus models and the Harness framework. The evaluation of the asynchronous delegation component relies on an internal benchmark, making it difficult for the community to verify the quality of tool-use integration. Additionally, the paper does not provide ablation studies isolating the contribution of the memory module vs. the base model improvements in the main video benchmarks.
Broader impact
This work is significant for the development of real-time AI assistants that can handle complex, multi-step tasks while maintaining natural conversational flow. The dual-loop architecture addresses a critical bottleneck in current LLM agents: the latency of tool execution. By decoupling perception/speech from reasoning/tools, it enables more robust and human-like interaction patterns, particularly in multimodal environments. Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
Author of this paper? Send a correction.