TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba
Abstract
Music-to-dance generation has broad applications in virtual reality, dance education, and digital character animation. However, the limited coverage of existing 3D dance datasets confines current models to a narrow subset of music styles and choreographic patterns, resulting in poor generalization to real-world music. Consequently, generated dances often become overly simplistic and repetitive, substantially degrading expressiveness and realism. To tackle this problem, we present TokenDance, a two-stage music-to-dance generation framework that explicitly addresses this limitation through dual-modality tokenization and efficient token-level generation. In the first stage, we discretize both dance and music using Finite Scalar Quantization, where dance motions are factorized into upper and lower-body components with kinematic-dynamic constraints, and music is decomposed into semantic and acoustic features with dedicated codebooks to capture choreography-specific structures. In the second stage, we introduce a Local-Global-Local token-to-token generator built on a Bidirectional Mamba backbone, enabling coherent motion synthesis, strong music-dance alignment, and efficient non-autoregressive inference. Extensive experiments demonstrate that TokenDance achieves overall state-of-the-art (SOTA) performance in both generation quality and inference speed, highlighting its effectiveness and practical value for real-world music-to-dance applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TokenDance presents a novel two-stage music-to-dance generation framework that effectively addresses the limitations of existing methods through dual-modality tokenization and a Local-Global-Local generator architecture. The comprehensive evaluation demonstrates its state-of-the-art performance and practical value, marking a significant contribution to the field of generative models in the context of dance and music.
Methodology
The methodology presented in TokenDance is innovative, particularly in its use of dual-modality tokenization and the Local-Global-Local architecture built on the Bidirectional Mamba backbone. The approach of discretizing both music and dance into reusable tokens through Finite Scalar Quantization (FSQ) is a significant advancement that addresses the limitations of existing methods that rely on continuous representations. The separation of upper and lower body movements during dance tokenization enhances the model's ability to generate diverse and coherent dance motions. The integration of genre-aware modeling and the non-autoregressive inference mechanism further contribute to the framework's efficiency and effectiveness.
Results & evaluation
The experiments conducted are extensive and demonstrate the effectiveness of TokenDance across multiple datasets, including AIST++, FineDance, and PopDanceSet. The quantitative results show that TokenDance achieves state-of-the-art performance in terms of motion quality and diversity, as evidenced by metrics such as FID and DIV. Additionally, the qualitative evaluations and user studies indicate that the generated dances are perceived as more synchronized and creative compared to existing methods. The thorough evaluation across different metrics and datasets strengthens the claims made by the authors regarding the framework's superiority.
Reproducibility
The paper provides detailed implementation details, including the architecture configurations, training procedures, and evaluation metrics used. However, the lack of a publicly available code repository or demo URL limits the reproducibility of the results. Future work should include making the code and models accessible to facilitate further research and validation by the community.
Limitations
One limitation of the study is the reliance on existing datasets, which may still impose constraints on the diversity of dance styles and music genres that the model can generalize to. The authors acknowledge the limited scale of current 3D dance datasets, which could affect the model's performance in real-world applications. Additionally, while the model shows promise, the subjective nature of dance and music may lead to variability in user preferences that are not fully captured in the quantitative metrics.
Broader impact
The potential applications of TokenDance are significant, particularly in fields such as virtual reality, dance education, and digital character animation. By improving the expressiveness and realism of generated dance motions, this framework could enhance user experiences in interactive environments and contribute to advancements in AI-generated content for entertainment and education. The ability to generate diverse and coherent dance movements based on music could also open new avenues for creative expression in the arts. TokenDance presents a novel two-stage music-to-dance generation framework that effectively addresses the limitations of existing methods through dual-modality tokenization and a Local-Global-Local generator architecture. The comprehensive evaluation demonstrates its state-of-the-art performance and practical value, marking a significant contribution to the field of generative models in the context of dance and music.
Author of this paper? Send a correction.