EchoWM: Open and Enterable Omnimodal World Models
Abstract
We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera--character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and speech over long-horizon generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
Methodology
The paper proposes "EchoWM," an omnimodal world model that jointly generates 720p video, environmental sound, music, and speech conditioned on a continuous 6-DoF camera trajectory. The core methodological contribution lies in the "camera intent" interface, which unifies first-person and third-person navigation by mapping discrete commands and continuous poses to a shared relative trajectory. The authors introduce a "complementary data engine" combining internal gameplay, internet gameplay, UE simulation, and web video, processed through separate audio-visual and geometry paths. A key technical detail is the use of Unified Camera Positional Encoding (UCPE) to inject relative camera geometry into the video transformer's attention mechanism, avoiding the need for separate camera encoders. The training strategy is progressive: Audio-Visual Continued Pretraining (AV-CPT), Action-SFT (freezing the backbone to learn trajectory conditioning), Joint-FT, and Autoregressive Post-Training (using Self-Gradient Forcing) for long-horizon generation. The approach is technically sound, addressing the distributional mismatch between audio-visual quality and control precision by curating specific data mixtures for each training stage.
Results & evaluation
The paper claims state-of-the-art performance on public benchmarks (WBench Navigation, SANA-WM-Bench). It emphasizes the ability to maintain synchronized audio and video over long horizons. However, the provided text is truncated, limiting the assessment of the full experimental results, ablation studies, and quantitative metrics (e.g., FAD, CLAP score, UTMOS, or human evaluation scores). The reliance on "public world-model benchmarks" suggests a focus on visual and control consistency, but the specific metrics for audio quality and synchronization fidelity are not detailed in the excerpt. The claim of "strong trajectory following" is supported by the controlled data engine, but the evaluation of speech synchronization specifically is less clear from the text.
Reproducibility
The paper provides detailed descriptions of the data processing pipeline, including metric pose recovery using ViPE, VGGT-Omega, and MoGe-2. It specifies the use of Qwen3-Omni and Gemini-3-Pro for annotation. The training stages and loss functions are mathematically defined. However, the exact hyperparameters, the specific versions of the base models (e.g., the pretrained joint audio-visual diffusion transformer), and the code for the data engine are not included in the text. The reliance on proprietary internal gameplay data and specific annotation pipelines may hinder exact reproducibility without significant effort.
Limitations
The model is limited to 720p resolution, which is modest compared to recent high-fidelity video generators. The interaction is restricted to camera/navigation intent; semantic actor actions (attacking, jumping) are not modeled. The audio generation, while synchronized, is conditioned on visual trajectory rather than direct acoustic modeling, which may limit the fidelity of complex acoustic environments. The reliance on recovered camera poses from internet video introduces potential noise, although quality filtering is applied. The truncation of the paper prevents a full assessment of failure modes or edge cases in long-horizon generation.
Broader impact
EchoWM contributes to the field of generative AI by enabling more immersive and interactive multimedia experiences. The ability to generate synchronized audio-visual content in response to navigation could impact gaming, virtual reality, and simulation. However, the generation of realistic speech and environmental sounds raises concerns about deepfakes and misinformation, particularly if the technology is misused to create deceptive audio-visual content. The paper does not explicitly discuss safety mitigations. EchoWM presents a comprehensive framework for omnimodal world modeling by unifying camera-centric navigation with joint audio-visual generation through a novel data engine and progressive training curriculum. The technical contribution of mapping heterogeneous control signals to a shared metric-scale trajectory and integrating this via UCPE is a significant step towards more controllable and interactive generative media, although the reliance on recovered geometry and the limitation to 720p resolution remain notable constraints.
Author of this paper? Send a correction.