RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can
Abstract
Recent Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Methodology
The paper proposes RUBATO, a multistream positional encoding mechanism that assigns independent "clocks" to different data streams (real-time audio vs. internal thinking) in full-duplex speech models. The core innovation is decoupling the positional indices of asynchronous streams to prevent temporal distortion when the decoding rate of the thinking stream varies. The authors derive three variants: TM-RoPE (subspace partitioning), Key-Clock (conditional rotation based on key token clock), and Full-Clock (rotating all dimensions by the key's clock). The methodology is mathematically sound, leveraging the properties of Rotary Position Embeddings (RoPE) to allow zero-shot interpolation of inference speeds. The implementation strategy, which expands hidden dimensions during training to fit standard FlashAttention kernels and uses separate KV caches during inference, is a clever engineering solution that avoids custom Triton kernels, enhancing practicality.
Results & evaluation
The experiments are rigorous, evaluating the model on Spoken GSM8K and BIG-Bench Audio, including a novel "Overlapped" scenario where the model must handle interruptions while reasoning. The results demonstrate that RUBATO-Full maintains high accuracy (48-51%) across varying decoding budgets, whereas baselines collapse to near-zero accuracy at interpolated speeds. The paper provides strong evidence for the utility of the method in reducing time-to-answer by leveraging faster hardware without retraining. The inclusion of UTMOS and Speaker Similarity metrics for filler audio quality adds depth to the evaluation, confirming that the architectural changes do not degrade acoustic quality.
Reproducibility
The paper provides a detailed reproducibility statement, including closed-form equations for the attention mechanisms and clear descriptions of the training data synthesis process. The use of standard backbones (Gemma-4) and tokenizers (NanoCodec) aids reproducibility. However, the specific hyperparameters for the training mixture and the exact construction of the synthetic datasets are deferred to the appendix, which is not fully visible in the provided text, though the main text describes the logic clearly.
Limitations
The primary limitation is the reliance on synthetic data for training, as no suitable full-duplex spoken reasoning corpora exist. The evaluation is limited to mathematical and logical reasoning tasks (GSM8K, BIG-Bench), which may not generalize to all conversational domains. Additionally, the method assumes a specific interleaving strategy (SCoT), and its applicability to other full-duplex architectures (e.g., concatenation-based) is not explored. The "Full-Clock" variant, while best performing, requires duplicating the head dimension, which increases memory usage.
Broader impact
This work has significant implications for the deployment of real-time AI agents. By decoupling reasoning speed from real-time constraints, it enables more efficient use of heterogeneous hardware. The concept of independent clocks for asynchronous streams is a generalizable architectural pattern that could apply to other multimodal or multi-agent systems requiring variable-rate processing. It addresses a critical bottleneck in making LLMs truly interactive and responsive in speech-based interfaces. The paper introduces a novel decoupled clock mechanism for positional encoding in full-duplex speech models, enabling zero-shot adaptation to variable inference speeds. This technical contribution is highly significant as it solves a fundamental temporal alignment problem in real-time AI agents, allowing for faster reasoning without sacrificing conversational latency or accuracy.
Author of this paper? Send a correction.