Omni Interaction Agent Technical Report
Abstract
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
Methodology
The paper proposes "Gander," an end-to-end Omni Interaction Agent that unifies perception, real-time interaction, and agentic capabilities. The core architectural innovation is the "Cerebellum-Brain" collaborative framework. The "Cerebellum" is a streaming Thinker-Talker model responsible for low-latency, full-duplex interaction, while the "Brain" is a training-free, plug-and-play agent (e.g., Codex, Claude Code) handling complex reasoning. The Cerebellum uses a "streaming chunk flattening" mechanism, serializing audio, video, and text into a unified token stream where the model predicts control tokens (listen/speak/interrupt) at each 1-second chunk. This allows for native interruption and proactive feedback without external VAD modules. The methodology is sound, addressing the latency-intelligence trade-off by decoupling fast interaction from slow reasoning.
Results & evaluation
The evaluation is comprehensive, covering conversational ability, omni understanding, interactive capability, and agentic intelligence. The paper reports internal human evaluations showing Gander maintains SOTA spoken dialogue naturalness while achieving competitive omni interaction performance. It demonstrates robustness in noisy environments, multi-party interactions, and backchannel communication. However, the paper relies heavily on "internal human evaluations" and subjective demos, with limited quantitative benchmarks compared to specific baselines (e.g., GPT-4o, MiniCPM-o) in the provided text snippet. The lack of standardized public benchmark scores for the "agentic" component makes direct comparison difficult.
Reproducibility
The authors state they release the model weights, code, and data. The architecture details (chunk size, token rates, encoder types like SigLIP) are provided. However, the "Brain" component relies on proprietary external agents (Codex, Claude Code), which limits full reproducibility of the agentic performance. The training data pipeline is described in detail, including the synthesis of full-duplex interaction data, which aids reproducibility of the Cerebellum component.
Limitations
The system's performance is coupled to the quality of the external "Brain" agent, which may have its own latency and cost implications. The "training-free" nature of the Brain means the Cerebellum cannot learn from the Brain's specific failures or adapt its tool-calling strategy through end-to-end fine-tuning. The evaluation lacks extensive quantitative ablation studies comparing the Cerebellum-Brain architecture against monolithic models on standardized agentic benchmarks.
Broader impact
This work is significant for the development of natural, human-like AI assistants that can handle complex tasks while maintaining fluid conversation. The decoupled architecture offers a scalable path for integrating powerful reasoning models into real-time interfaces. It pushes the field beyond turn-based chatbots toward continuous, multimodal collaboration. Gander introduces a Cerebellum-Brain architecture that decouples real-time full-duplex interaction from long-horizon agentic reasoning, enabling natural, interruptible multimodal collaboration. The paper presents a robust streaming framework with detailed data construction for interaction behaviors, though its reliance on external proprietary agents for the "Brain" component and limited quantitative benchmarking against specific baselines slightly temper its technical impact.
Author of this paper? Send a correction.