Dance to Music Generation leveraging Pre-training with Unpaired data and Contrastive Alignment
Abstract
Dance-to-music generation is a promising task for applications such as choreography support and automatic accompaniment, where temporal coordination between body movement and sound is essential. In particular, using human joint positions as the motion representation is attractive because they explicitly capture body dynamics while being lightweight, privacy-preserving, and easy to integrate with motion capture and pose-estimation pipelines. A central challenge in this setting, however, is the scarcity of high-quality paired dance-music data, since collecting accurately synchronized pairs is costly and often constrained by copyright and performance rights. This makes it difficult to train end-to-end models solely from paired data. To address this issue, we propose a dance-conditioned music generation framework that efficiently exploits both unpaired and paired data. Our method combines pretrained unimodal encoders for motion and music, beat-guided contrastive pretraining to align their feature spaces, and a ControlNet-style conditioning module on top of a pretrained text-to-audio diffusion model. Experiments on AIST++ demonstrate that the proposed techniques improve both dance-music alignment and audio quality, as confirmed by quantitative and qualitative evaluations. Compared to a state-of-the-art method, our approach achieves superior dance alignment performance and competitive audio quality. Code is available at https://github.com/kmraven/AudioLDM-ControlNet .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.
Methodology
The paper proposes a two-stage framework for dance-to-music generation. Stage 1 involves beat-guided contrastive pretraining to align features from a pretrained MotionBERT encoder (for dance) and MERT encoder (for music). Stage 2 uses these aligned motion features to condition a frozen AudioLDM backbone via a ControlNet-style adapter. The methodology is technically sound and follows established patterns in multimodal generation (contrastive alignment + diffusion conditioning). The use of MotionBERT for 2D keypoints is a strong choice for capturing kinematic structure. However, the novelty is moderate; the core idea of using contrastive learning to align modalities (like CLIP) and then conditioning a diffusion model (like ControlNet) is a well-known paradigm in computer vision and has been applied to audio (e.g., AudioLDM). The specific application to dance-to-music with beat-guided alignment is a valid contribution but represents an incremental adaptation of existing techniques rather than a fundamental methodological breakthrough. The "beat-guided" aspect is a specific engineering detail rather than a new theoretical framework.
Results & evaluation
The experiments are conducted on the AIST++ dataset, which is the standard benchmark for this task. The evaluation includes both objective metrics (BHS, BCS, F1, TD, FAD, CLAP, BAS) and subjective user studies (MOS). The results show that the proposed method outperforms the baseline (Li et al.) in dance-alignment metrics (BAS and MOS), which is the primary goal. However, the subjective sound quality MOS is slightly lower than the baseline, although objective audio quality metrics (FAD, CLAP) are better. The ablation studies effectively demonstrate the contribution of the contrastive pretraining and the MotionBERT encoder. The comparison with Li et al. is fair as they use the same dataset split. One limitation is the small test set (36 pairs), which can lead to high variance in metrics, though the authors acknowledge this by using both objective and subjective measures. The results are consistent with the claims.
Reproducibility
The paper provides sufficient detail for reproduction. It specifies the datasets (AIST++), preprocessing steps (resampling, normalization), model architectures (MotionBERT, MERT, AudioLDM, ControlNet), and training hyperparameters (learning rates, batch sizes, epochs). The code is available on GitHub. The use of standard pretrained models (AudioLDM, MotionBERT, MERT) facilitates reproducibility. The specific beat extraction method (Librosa) and contrastive loss details are clearly described.
Limitations
The paper acknowledges the small scale of paired dance-music data. The subjective sound quality is slightly inferior to the baseline, which the authors attribute to the backbone (AudioLDM vs MusicGen). The reliance on 2D keypoints may lose some 3D motion information, although MotionBERT is designed to handle this. The beat extraction is heuristic-based (velocity smoothing), which might not be robust to all dance styles. The evaluation is limited to AIST++, and generalizability to other datasets or styles is not tested. The subjective study has a small number of participants (35) and samples (10 per participant), which may not fully capture perceptual nuances.
Broader impact
This work contributes to the field of multimodal generation, specifically the intersection of dance and music. It has potential applications in choreography support, automatic accompaniment, and entertainment. The use of joint positions is privacy-preserving compared to video-based methods. The framework could be extended to other motion-to-audio tasks. The open-source code promotes further research in this area. The paper presents a competent application of contrastive alignment and ControlNet-style conditioning to the dance-to-music generation task, achieving state-of-the-art alignment performance on AIST++. While the methodological novelty is incremental, the technical execution is solid, and the results demonstrate the effectiveness of combining unimodal pretraining with cross-modal alignment for this specific, data-scarce problem.
Author of this paper? Send a correction.