TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation
Abstract
Unified audio-visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio-video synthesis to music-dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music-dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music-dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio-visual models, such as Veo 3 and Sora 2, produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music-dance models that explicitly optimize rhythmic and kinetic coherence.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents TMD-Bench, a novel benchmark for music-dance co-generation, and introduces RhyJAM, a unified model that achieves competitive rhythmic alignment and unimodal fidelity. This work significantly advances the evaluation and generation of synchronized audio-visual content, addressing a critical gap in existing methodologies.
Methodology
The paper introduces TMD-Bench, a comprehensive evaluation framework specifically designed for music-dance co-generation, which is a significant advancement in the field. The methodology is well-structured, integrating both low-level physical metrics and high-level perceptual assessments through a multi-level evaluation pipeline. The authors also develop a unified model, RhyJAM, that generates music and dance in a coherent manner, addressing the critical challenge of rhythmic alignment. The use of a fine-grained Music Captioner for structured music semantics is a novel aspect that enhances the evaluation process.
Results & evaluation
The experiments are robust, utilizing a curated dataset of 10,000 rhythm-aligned music-dance pairs. The evaluation metrics are comprehensive, covering unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The results demonstrate that RhyJAM outperforms existing models in rhythmic alignment while maintaining competitive unimodal fidelity. The comparative analysis against various baselines, including closed-source and open-source models, provides a clear picture of the model's strengths and weaknesses.
Reproducibility
The paper provides detailed implementation details, including training configurations and dataset processing methods, which enhance reproducibility. However, the absence of a publicly available demo or project URL limits the ability for others to directly replicate the results.
Limitations
One limitation is the reliance on existing commercial models for baseline comparisons, which may not fully represent the capabilities of open-source alternatives. Additionally, while the evaluation framework is comprehensive, the subjective nature of some assessments may introduce variability in human judgments.
Broader impact
The proposed benchmark and model have the potential to significantly impact the fields of generative audio and video synthesis, particularly in applications related to virtual production and interactive media. By addressing the intricate coupling between music and dance, this work paves the way for more sophisticated generative models that can enhance user experiences in entertainment and education. The paper presents TMD-Bench, a novel benchmark for music-dance co-generation, and introduces RhyJAM, a unified model that achieves competitive rhythmic alignment and unimodal fidelity. This work significantly advances the evaluation and generation of synchronized audio-visual content, addressing a critical gap in existing methodologies.
Author of this paper? Send a correction.