Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
Abstract
We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[Wan-Streamer introduces a native-streaming, end-to-end foundation model that unifies text, audio, and video perception and generation within a single causal Transformer, achieving sub-second full-duplex interaction latency through a novel thinker-performer inference architecture and strictly causal multimodal components.]
Methodology
The paper proposes Wan-Streamer, a native-streaming, end-to-end foundation model designed for real-time, low-latency, full-duplex audio-visual interaction. The core architectural innovation lies in replacing cascaded pipelines (ASR, LLM, TTS, Avatar Renderer) with a single Transformer that processes interleaved text, audio, and video tokens using block-causal attention. The methodology introduces strictly causal audio and video Variational Autoencoders (VAEs) to enable streaming latent coding, and employs a "thinker-performer" inference paradigm. In this setup, the "thinker" handles perception, language reasoning, and KV-cache updates, while the "performer" handles the computationally expensive flow-matching solver for audio-visual latent generation. This separation allows for overlapping computation, achieving significant latency reductions. The training strategy involves a three-stage process: independent-task pretraining, end-to-end interaction training on duplex data, and distillation for low-latency streaming using self-forcing and distribution matching. This approach fundamentally shifts the design philosophy from post-hoc synchronization to native causal alignment of all modalities.
Results & evaluation
The authors evaluate Wan-Streamer primarily on latency metrics and qualitative naturalness. They report a model-side response latency of approximately 200 ms and a total interaction latency of ~550 ms (including 350 ms network latency), which is competitive with specialized speech-only systems and significantly faster than cascaded video-generation pipelines. The paper compares these metrics against public systems like Doubao and various digital human platforms, highlighting the advantage of a unified model in avoiding module-boundary delays. Qualitative assessments focus on "naturalness," specifically the model's ability to maintain idle behaviors, responsive non-verbal feedback during listening, and seamless interruption handling. While the latency results are impressive and well-documented, the paper lacks extensive quantitative benchmarks on generation quality (e.g., FID, CLIP scores) or dialogue performance (e.g., BLEU, BERTScore) compared to state-of-the-art specialized models. The evaluation relies heavily on the claim that the unified model preserves identity and synchronization better than cascaded systems, which is supported by the architecture but would benefit from more rigorous ablation studies on specific failure modes.
Reproducibility
The paper provides a detailed description of the architecture, including the causal VAEs, block-causal attention mechanism, and the thinker-performer scheduling logic. The training stages are clearly delineated. However, as an arXiv preprint (v0.1), the code and pre-trained weights are not publicly available, which limits immediate reproducibility. The specific hyperparameters for the flow-matching solver and the exact composition of the "broad mixture" of training data are not fully specified, which are critical for reproducing the distillation results. The reliance on proprietary infrastructure (Alibaba Group) for the "thinker-performer" deployment optimization (CUDA graphs, custom kernels) also poses a barrier to exact replication without access to similar engineering resources.
Limitations
The current implementation operates at a low resolution (192p), which the authors acknowledge as a proof-of-concept limitation, though they claim scaling is straightforward. The model's performance on complex, long-horizon interactions is not extensively tested; the "self-forcing" distillation helps mitigate drift, but long-term consistency remains a challenge for any autoregressive streaming system. Additionally, the computational cost of running a single large Transformer for all modalities is high, necessitating the specialized thinker-performer split and potentially limiting deployment to high-end GPU clusters rather than edge devices. The paper does not provide a detailed analysis of the model's failure modes in terms of hallucination or synchronization errors under high-noise conditions.
Broader impact
Wan-Streamer represents a significant step towards more natural and immersive human-computer interaction. By enabling real-time, full-duplex audio-visual communication, it has broad applications in embodied AI, virtual assistants, telepresence, and interactive entertainment. The reduction in latency and error accumulation compared to cascaded systems addresses a major bottleneck in current multimodal AI. However, the ability to generate realistic audio-visual avatars also raises concerns regarding deepfakes and misinformation, necessitating robust safety measures and watermarking. The shift towards unified, end-to-end models may also concentrate computational resources and capabilities within a few large organizations, potentially impacting the diversity of the open-source AI ecosystem. [Wan-Streamer introduces a native-streaming, end-to-end foundation model that unifies text, audio, and video perception and generation within a single causal Transformer, achieving sub-second full-duplex interaction latency through a novel thinker-performer inference architecture and strictly causal multimodal components.]
Author of this paper? Send a correction.