TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation
Abstract
Existing music-driven dance generation approaches have achieved strong realism and effective audio-motion alignment. However, they generally lack semantic controllability, making it difficult to guide specific movements through natural language descriptions. This limitation primarily stems from the absence of large-scale datasets that jointly align music, text, and motion for supervised learning of text-conditioned control. To address this challenge, we propose TeMuDance, a framework that enables text-based control for music-conditioned dance generation without requiring any manually annotated music-text-motion triplet dataset. TeMuDance introduces a motion-centred bridging paradigm that leverages motion as a shared semantic anchor to align disjoint music-dance and text-motion datasets within a unified embedding space, enabling cross-modal retrieval of missing modalities for end-to-end training. A lightweight text control branch is then trained on top of a frozen music-to-dance diffusion backbone, preserving rhythmic fidelity while enabling fine-grained semantic guidance. To further suppress noise inherent in the retrieved supervision, we design a dual-stream fine-tuning strategy with confidence-based filtering. We also propose a novel task-aligned metric that quantifies whether textual prompts induce the intended kinematic attributes under music conditioning. Extensive experiments demonstrate that TeMuDance achieves competitive dance quality while substantially improving text-conditioned control over existing methods.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of TeMuDance, a framework that enables fine-grained textual control for music-driven 3D dance generation without requiring paired music-text-motion triplets. This innovative approach addresses significant limitations in the field and demonstrates strong potential for practical applications in animation and interactive media.
Methodology
The TeMuDance framework introduces a novel approach to music-driven dance generation by leveraging a motion-centered bridging paradigm that aligns disjoint datasets without requiring manually annotated triplets. This is a significant advancement, as it allows for the integration of textual control into dance generation, which has been a notable limitation in previous works. The use of a frozen music-to-dance diffusion backbone with a lightweight text control branch is innovative, as it preserves rhythmic fidelity while enabling semantic guidance. The dual-stream fine-tuning strategy and the introduction of the Kinematic Primitive Success (KPS) metric further enhance the methodology by addressing the challenges of noise in retrieved supervision and providing a robust evaluation of text-driven control.
Results & evaluation
The experiments are comprehensive, demonstrating the effectiveness of TeMuDance against state-of-the-art methods in both music-driven dance generation and text-controllable generation. The paper employs a variety of metrics, including FID, BAS, and the newly proposed KPS, to evaluate motion quality, diversity, and controllability. The results indicate that TeMuDance achieves competitive performance, particularly in motion quality and physical plausibility, which are crucial for practical applications in animation and gaming.
Reproducibility
The implementation details are well-documented, including specifics on datasets, training protocols, and evaluation metrics. However, the absence of a publicly available code repository or demo limits the reproducibility of the results. Providing access to the trained models and code would significantly enhance the ability of other researchers to validate and build upon this work.
Limitations
One limitation is the reliance on disjoint datasets, which may introduce challenges in generalization to unseen data. Additionally, while the KPS metric is innovative, its effectiveness in capturing all aspects of text-driven control may require further validation. The paper also does not address potential biases in the datasets used, which could impact the generated dance quality and diversity.
Broader impact
The ability to generate dance movements that are semantically controlled by text descriptions has significant implications for fields such as animation, virtual reality, and gaming. This research could lead to more intuitive and expressive character animations, enhancing user experience and engagement. The framework could also inspire future research in multimodal generative models, particularly those that integrate audio, text, and motion. The main contribution of this paper is the introduction of TeMuDance, a framework that enables fine-grained textual control for music-driven 3D dance generation without requiring paired music-text-motion triplets. This innovative approach addresses significant limitations in the field and demonstrates strong potential for practical applications in animation and interactive media.
Author of this paper? Send a correction.