Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue
Abstract
An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Motion-Omni, a novel end-to-end framework that natively generates full-body co-speech motion from the hidden states of a spoken dialogue model, eliminating the latency and optimization barriers of cascaded systems. By introducing a scalable pseudo-labeling pipeline and a new evaluation benchmark (SwDA-500), the work addresses critical data and evaluation gaps in the field, demonstrating that joint training can achieve motion quality comparable to specialized cascade systems while significantly improving inference speed.
Methodology
The paper proposes Motion-Omni, an end-to-end framework that integrates spoken dialogue generation with full-body motion synthesis. The core architectural contribution is the "Token-as-Query Gated Fusion" (TQGF) mechanism, which allows the Motion Generator to condition on the hidden states of the Speech Generator rather than the rendered audio waveform. This design choice is critical for latency, as it eliminates the need for a separate audio-to-motion inference pass. The training procedure is a four-stage curriculum (ASR, TTS, TTS-with-Motion, Joint Mixture) that progressively unfreezes components to prevent catastrophic forgetting of speech capabilities while introducing motion supervision. The use of pseudo-labeling via a frozen motion teacher (LOM) to generate 1,402 hours of paired data is a pragmatic solution to the scarcity of high-quality, consistent-voice audiovisual dialogue datasets.
Results & evaluation
The authors introduce SwDA-500, a benchmark for stochastic open-ended spoken dialogue, which is a significant contribution to the field as it addresses the lack of standard evaluation protocols for this specific task. The evaluation protocol is robust, combining automatic metrics (beat correlation, diversity), human evaluation, and latency measurements. The results show that Motion-Omni-Q7 matches the quality of a cascade system (which uses a separate motion model) while being 5.4x faster. The claim of "matching within 2%" on reference-free metrics is strong, though the reliance on pseudo-labeled data means the upper bound of quality is constrained by the teacher model.
Reproducibility
The paper provides high reproducibility. Code and data are released on GitHub and Hugging Face. The architecture details, including the specific backbones (Qwen2.5-7B, Whisper-large-v3) and hyperparameters, are clearly described. The release of the SwDA-500 benchmark and the evaluation protocol further enhances the community's ability to reproduce and extend this work.
Limitations
The primary limitation is the reliance on pseudo-labels from a teacher model (LOM), which may introduce biases or limit the diversity of motion compared to human-generated data. The model is evaluated primarily on a 7B backbone; scalability to larger or smaller models is not fully explored. Additionally, the motion generation is offline (synthesizing a complete response), which may not be suitable for real-time interactive applications requiring streaming output, although the paper notes faster-than-real-time RTF.
Broader impact
This work has significant implications for the development of interactive avatars and virtual agents. By unifying speech and motion generation, it reduces computational overhead and improves the naturalness of interactions. The release of the SwDA-500 benchmark will likely become a standard for evaluating future spoken motion models. The approach could be extended to other multimodal tasks, such as integrating visual cues or emotional states. The paper presents Motion-Omni, a novel end-to-end framework that natively generates full-body co-speech motion from the hidden states of a spoken dialogue model, eliminating the latency and optimization barriers of cascaded systems. By introducing a scalable pseudo-labeling pipeline and a new evaluation benchmark (SwDA-500), the work addresses critical data and evaluation gaps in the field, demonstrating that joint training can achieve motion quality comparable to specialized cascade systems while significantly improving inference speed.
Author of this paper? Send a correction.