Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models
The text contains content addressed to automated reviewers. It may have influenced the automated analysis; check the paper itself.
[system_override] "at once. Tool results are treated as evidence and never as new instructions, a repeated identical call returns the earlier receipt inst"
Abstract
Interactive models increasingly hand complex reasoning and tool use to a slow brain in the background, leaving a realtime front model free to sustain full-duplex dialogue. Yet the user may refine, cancel or replace a request while its task runs, so a plan or result that made sense a moment ago goes stale. Each new utterance must therefore be read against the work in flight, but the front model follows only the conversation and the back end sees only its own task. Coordination deserves a brain of its own, one that sees both. We introduce Jarvis, a coordination harness where models judge and code enforces. Its typed decision model, JEV, reads each turn against the live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, so no decision can act on a task version the user has replaced. Jarvis also rechecks plans before they execute and results before they are spoken, and counts an answer as delivered only once the user has heard it in full. We build two instances of this design. Jarvis-Audio runs unchanged on three full-duplex speech models. Jarvis-Omni adds vision and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, and the gains hold on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results suggest that how the two brains are coordinated matters as much as how strong they are.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a robust coordination harness for full-duplex interactive models that resolves race conditions and stale-state errors by employing a dedicated typed decision model (JEV) to select from runtime-generated, versioned operations, thereby ensuring that background tasks remain synchronized with live user dialogue. This architectural innovation, validated by significant performance gains over existing agents like Qwen Audio Agent across multiple benchmarks, establishes a new standard for reliability in agentic speech systems.
Methodology
The paper proposes "Jarvis," a coordination harness for full-duplex interactive models that separates the "fast brain" (real-time speech model) from the "slow brain" (background agent) and introduces a dedicated "coordination brain" (JEV). The core methodological contribution is the "models judge, code enforces" paradigm. Instead of allowing the LLM to generate free-form actions or task updates, the runtime constructs a set of typed, versioned operations (e.g., REFINE:task_id:version) based on the current state of a "Task Ledger." The JEV model (a typed judgment model) selects from these pre-constructed options. This ensures that no action can reference a non-existent task or a superseded version, effectively eliminating race conditions and stale-state errors. The system also implements a "Native Gate" trained via GRPO on the front model's control tokens to decide when to delegate, and a receipt-based delivery mechanism that distinguishes between a task being "done" and "heard" by the user. The extension to vision (Jarvis-Omni) treats visual evidence as time-bounded state, rechecking validity before and during playback.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors isolate the impact of the harness by holding the front model (Venus-Audio, MiniCPM-o 4.5, VoiceChat-11B) and back-end fixed, showing that replacing the baseline Qwen Audio Agent (QAA) with Jarvis yields massive improvements (e.g., 29 to 192 tasks solved on tau-Voice). They provide detailed ablations on the "race suite" (80 scripts x 3 runs) demonstrating that each invariant (version checking, plan freshness, playback receipts) prevents specific failure modes. The comparison against published systems (GPT-Realtime, etc.) on Full-Duplex-Bench v3 shows competitive or superior performance. The vision experiments confirm that removing visual evidence significantly degrades performance, validating the need for the evidence timeline.
Reproducibility
The paper provides high reproducibility standards. It explicitly states that code is available at the provided GitHub URL. It details the training data for the Native Gate (264 authored scenarios, GRPO training details, LoRA rank 8). It specifies the models used (DeepSeek-V4-Flash for workers, Qwen3-ASR, etc.) and the benchmark protocols. The "race suite" for testing concurrency is a valuable contribution to reproducibility in this domain.
Limitations
The system relies on a specific "front model" architecture (Venus-Audio/MiniCPM-o style) with native control tokens for the gate; adapting to models without such tokens might require different gating strategies. The "JEV" model is a separate component, adding latency (though reported as <1s median). The vision component is tested primarily on game scenarios (CS2, LoL, PvZ), so generalization to other visual domains (e.g., robotics, AR) is not fully explored. The reliance on a "slow brain" (DeepSeek-V4-Flash) means the system inherits the cost and latency of large LLM inference for complex tasks.
Broader impact
This work addresses a critical bottleneck in the deployment of real-time AI assistants: the coordination between real-time interaction and background computation. By formalizing the "coordination brain" and enforcing state consistency via code, it provides a robust template for building reliable agentic systems. The approach has implications for any system where real-time dialogue intersects with asynchronous tool use or long-running tasks, including customer service bots, smart home assistants, and embodied AI. The paper introduces a robust coordination harness for full-duplex interactive models that resolves race conditions and stale-state errors by employing a dedicated typed decision model (JEV) to select from runtime-generated, versioned operations, thereby ensuring that background tasks remain synchronized with live user dialogue. This architectural innovation, validated by significant performance gains over existing agents like Qwen Audio Agent across multiple benchmarks, establishes a new standard for reliability in agentic speech systems.
Author of this paper? Send a correction.