Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
Abstract
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce Ex-Omni-2D, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured Visual Thought Plan (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal Streaming Student whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at 400times720/720times400, providing a practical quality--efficiency operating point.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Ex-Omni-2D, an omni-modal dialogue framework that generates coordinated text, speech, and video responses by decoupling visual intent planning (VTP) and acoustic timing (speech units) to enable efficient training from heterogeneous data sources. The work is a significant engineering achievement in integrating large language models with speech and video generation for interactive dialogue. Its primary value lies in the system-level design: the VTP interface and the shared acoustic-temporal interface allow for modular training and inference. While not introducing a new foundational model, it effectively demonstrates how to overcome data scarcity in omni-modal dialogue by leveraging structured intermediate representations. The "Prefix Streaming" technique is a notable contribution to the specific problem of autoregressive video generation stability. The paper is well-written, thoroughly evaluated, and addresses a timely and important problem in multimodal AI.
Methodology
The paper proposes Ex-Omni-2D, a framework for omni-modal dialogue that generates text, speech, and video. The core methodological contribution is the "Visual Thought Plan" (VTP), a structured textual representation of visual intent (scene, emotion, motion) generated by the LLM, and a "native multi-codebook speech unit" interface. The architecture factors the problem: the LLM generates VTP and text; a Speech Generator produces acoustic units; and a Video Generator (based on Wan2.1) synthesizes video conditioned on the reference image, VTP, and frame-aligned speech units. A key technical detail is the "Prefix Streaming" mechanism for the video generator, which attempts to mitigate cumulative degradation in autoregressive video generation by carrying a clean latent from the previous chunk. The approach is a sophisticated integration of existing components (LLM, TTS, Video Diffusion) rather than a fundamental architectural breakthrough in any single modality. The factorization to avoid paired dialogue-video data is a pragmatic engineering solution to a data scarcity problem.
Results & evaluation
The evaluation covers audio quality (PQ, CU, SIM), video quality (SC, IQ, DD), and synchronization (Sync-C). The paper demonstrates that the Teacher model achieves high quality but is slow, while the Streaming Student offers a trade-off. The ablation studies on VTP and speech conditioning are valuable, showing that the VTP improves subject consistency and synchronization compared to a neutral plan. However, the video quality metrics (SC ~94%, IQ ~67%) are competitive but not state-of-the-art for dedicated video generators, which is expected given the constraints of dialogue-native generation. The audio quality is strong, leveraging recent TTS advances. The evaluation protocol is rigorous, using established benchmarks like VoiceBench and OmniCharacter.
Reproducibility
The paper provides detailed descriptions of the architecture, training stages, and hyperparameters. The use of open-source backbones (Qwen3, Wan2.1) facilitates reproduction. The specific "Prefix Streaming" mechanism and data construction pipelines are described in sufficient detail for replication. The project page link suggests code or at least detailed visualizations may be available, though the code repository itself is not explicitly linked in the text provided.
Limitations
The authors acknowledge several limitations: the speaker similarity is not perfect, the VTP is not independently sufficient for video control (requiring acoustic conditioning), and there is a measurable trade-off between language capability and visual planning in the shared LLM channel. The Streaming Student still has significant latency (first video chunk >3s) and is not real-time. The quality-efficiency trade-off means the fast model is significantly lower quality than the teacher.
Broader impact
This work contributes to the field of embodied AI and virtual humans by providing a framework for generating coherent, multi-modal responses. It addresses the challenge of generating visually present dialogue agents, which has applications in customer service, education, and entertainment. The factorization strategy offers a blueprint for training complex multi-modal systems where paired data is scarce. The paper introduces Ex-Omni-2D, an omni-modal dialogue framework that generates coordinated text, speech, and video responses by decoupling visual intent planning (VTP) and acoustic timing (speech units) to enable efficient training from heterogeneous data sources. The work is a significant engineering achievement in integrating large language models with speech and video generation for interactive dialogue. Its primary value lies in the system-level design: the VTP interface and the shared acoustic-temporal interface allow for modular training and inference. While not introducing a new foundational model, it effectively demonstrates how to overcome data scarcity in omni-modal dialogue by leveraging structured intermediate representations. The "Prefix Streaming" technique is a notable contribution to the specific problem of autoregressive video generation stability. The paper is well-written, thoroughly evaluated, and addresses a timely and important problem in multimodal AI.
Author of this paper? Send a correction.