MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding
Abstract
Music is both an auditory and an embodied phenomenon, closely linked to human motion and naturally expressed through dance. However, most existing audio representations neglect this embodied dimension, limiting their ability to capture rhythmic and structural cues that drive movement. We propose MotionBeat, a framework for motion-aligned music representation learning. MotionBeat is trained with two newly proposed objectives: the Embodied Contrastive Loss (ECL), an enhanced InfoNCE formulation with tempo-aware and beat-jitter negatives to achieve fine-grained rhythmic discrimination, and the Structural Rhythm Alignment Loss (SRAL), which ensures rhythm consistency by aligning music accents with corresponding motion events. Architecturally, MotionBeat introduces bar-equivariant phase rotations to capture cyclic rhythmic patterns and contact-guided attention to emphasize motion events synchronized with musical accents. Experiments show that MotionBeat outperforms state-of-the-art audio encoders in music-to-dance generation and transfers effectively to beat tracking, music tagging, genre and instrument classification, emotion recognition, and audio-visual retrieval. Our project demo page: https://motionbeat2025.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MotionBeat presents a novel framework for motion-aligned music representation learning that integrates embodied aspects of music with advanced contrastive learning techniques. This work significantly contributes to the field by addressing the gap between audio representations and human motion, paving the way for more intuitive and synchronized interactions in music-related applications.
Methodology
The methodology presented in MotionBeat is innovative, introducing two novel loss functions—Embodied Contrastive Loss (ECL) and Structural Rhythm Alignment Loss (SRAL)—that specifically target the alignment of music and motion. The architectural innovations, such as bar-equivariant phase rotations and contact-guided attention, are well-conceived to capture the cyclic nature of rhythm and enhance the model's ability to focus on motion events. The approach effectively combines contrastive learning with structural alignment, which is a significant advancement over traditional methods that do not consider the embodied aspect of music.
Results & evaluation
The experimental evaluation is comprehensive, covering a wide range of tasks including music-to-dance generation, beat tracking, music tagging, genre classification, instrument classification, emotion recognition, and audio-visual retrieval. The results demonstrate that MotionBeat consistently outperforms state-of-the-art audio encoders, providing strong evidence of its effectiveness. The use of ablation studies to analyze the contributions of ECL and SRAL further strengthens the findings, showcasing the importance of each component in achieving superior performance.
Reproducibility
The paper provides sufficient implementation details, including architecture specifications, training protocols, and datasets used, which are crucial for reproducibility. However, the absence of a public code repository limits the ability of other researchers to directly replicate the results, which is a common expectation in contemporary ML research.
Limitations
One limitation is the reliance on a specific dataset (AIST++), which may affect the generalizability of the findings to other types of music or motion data. Additionally, while the model shows strong performance in various tasks, the paper does not extensively discuss potential challenges or failures in scenarios where music and motion may not align as expected.
Broader impact
The implications of this work extend beyond academic research; it has potential applications in fields such as interactive entertainment, dance choreography, and rehabilitation, where understanding the relationship between music and movement can enhance user experiences. The framework could also inspire further research into embodied learning in other multimodal contexts. MotionBeat presents a novel framework for motion-aligned music representation learning that integrates embodied aspects of music with advanced contrastive learning techniques. This work significantly contributes to the field by addressing the gap between audio representations and human motion, paving the way for more intuitive and synchronized interactions in music-related applications.
Author of this paper? Send a correction.