SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning
Abstract
Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is never observed during training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion--beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.
Methodology
The paper proposes SingDance, a video diffusion framework that addresses the compositional zero-shot problem of generating singing-and-dancing videos. The core methodological contribution is the formulation of vocal articulation as a semantic "role" (source vs. listener) rather than a simple binary lip-sync instruction. This allows the model to distinguish between a subject producing vocals and a subject listening to off-screen vocals. The architecture utilizes hard-compact routing to select relevant audio tokens (speech from Wav2Vec 2.0, music from MuQ) and injects them via frame-wise joint audio injection into a Video DiT backbone (Wan2.2). A key innovation is the staged training strategy with asymmetric supervision: role control is learned from speech data, and music-conditioned dance is learned from instrumental/dancing-only data, without ever seeing paired singing-and-dancing data during training. The zero-shot composition at inference (Song/Source) combines these separately learned capabilities. The approach is technically sound and elegantly solves the conditioning conflict between speech and music in a unified framework.
Results & evaluation
The experimental evaluation is comprehensive. The authors curate specific test sets (SingDance-50, Dance-100) to evaluate the proposed capabilities. They compare against strong baselines including MusicInfuser and Wan-S2V. The results demonstrate strong motion-beat alignment and visual fidelity. Crucially, the paired vocal-role switching experiment effectively verifies that the model can switch between singing and listening while maintaining consistent body motion and audio conditioning, a capability that baseline models lacking explicit role conditioning fail to achieve robustly. The lip synchronization metrics are competitive with specialized speech-driven models, despite the parameter efficiency. The ablation studies support the necessity of the music pathway for beat alignment.
Reproducibility
The paper provides detailed implementation details, including the backbone (Wan2.2-TI2V-5B), training hardware (64 A100s), hyperparameters (learning rate, batch size), and specific audio encoders (Wav2Vec 2.0, MuQ). The data sources are described (HuMoSet, MA-Data, proprietary collections), though the proprietary nature of some data may limit full reproducibility of the training set. The project page URL is provided, which likely contains code or weights, enhancing reproducibility potential.
Limitations
The current model is limited to short, single-person clips with a binary, clip-level vocal role. It does not handle within-clip role transitions, duets, or multiple vocal roles. The reliance on hard-compact routing means the model must be explicitly told the role; it does not automatically infer the role from the audio content, which is a design choice but also a limitation for fully autonomous generation. The use of proprietary data for training is a constraint for independent verification.
Broader impact
This work significantly advances the field of multimodal video generation, particularly in the niche of music and speech-conditioned human animation. By enabling compositional zero-shot singing-and-dancing, it lowers the barrier for creating high-quality, personalized performance videos. The role-aware conditioning framework could be extended to other interactive scenarios. However, the ability to generate realistic singing and dancing videos also raises concerns regarding deepfake misuse, although the explicit role conditioning might offer some avenue for detection or control. SingDance presents a novel and effective framework for compositional zero-shot singing-and-dancing video generation by introducing role-aware audio conditioning and asymmetric staged training, achieving strong performance in motion-beat alignment and lip synchronization while maintaining high visual fidelity.
Author of this paper? Send a correction.