DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching
Abstract
Generating full-length, high-quality songs is challenging, as it requires maintaining long-term coherence both across text and music modalities and within the music modality itself. Existing non-autoregressive (NAR) frameworks, while capable of producing high-quality songs, often struggle with the alignment between lyrics and vocal. Concurrently, catering to diverse musical preferences necessitates reinforcement learning from human feedback (RLHF). However, existing methods often rely on merging multiple models during multi-preference optimization, which results in significant performance degradation. To address these challenges, we introduce DiffRhythm 2, an end-to-end framework designed for high-fidelity, controllable song generation. To tackle the lyric alignment problem, DiffRhythm 2 employs a semi-autoregressive architecture based on block flow matching. This design enables faithful alignment of lyrics to singing vocals without relying on external labels and constraints, all while preserving the high generation quality and efficiency of NAR models. To make this framework computationally tractable for long sequences, we implement a music variational autoencoder (VAE) that achieves a low frame rate of 5 Hz while still enabling high-fidelity audio reconstruction. In addition, to overcome the limitations of multi-preference optimization in RLHF, we propose cross-pair preference optimization. This method effectively mitigates the performance drop typically associated with model merging, allowing for more robust optimization across diverse human preferences. We further enhance musicality and structural coherence by introducing stochastic block representation alignment loss.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DiffRhythm 2, a novel semi-autoregressive framework for high-fidelity song generation that effectively addresses lyric alignment and multi-preference optimization challenges. The comprehensive analysis of its technical contributions, methodology, and experimental results underscores its significance in advancing the state of the art in music generation.
Methodology
The paper introduces DiffRhythm 2, a semi-autoregressive framework that leverages block flow matching for song generation. This approach addresses significant challenges in lyric alignment and multi-preference optimization, which have been inadequately handled in prior works. The integration of a music variational autoencoder (VAE) with a low frame rate enhances the efficiency of long-sequence generation. The stochastic block REPA loss is a novel contribution that improves musicality and structural coherence, while cross-pair preference optimization effectively mitigates performance degradation associated with multi-preference alignment. Overall, the methodology is well-structured and presents a significant advancement in the field of music generation.
Results & evaluation
The experimental setup is robust, utilizing a large-scale dataset of 1.4 million songs across various genres. The evaluation metrics are comprehensive, encompassing both subjective and objective assessments. The results demonstrate that DiffRhythm 2 outperforms existing models in several key areas, including lyric accuracy and music quality. The ablation studies further validate the importance of the proposed methods, showing significant performance drops when key components are removed. However, comparisons with commercial systems reveal that while DiffRhythm 2 excels among open-source models, it still lags behind in overall performance.
Reproducibility
The authors commit to reproducibility by releasing inference code and model checkpoints, which is crucial for the community to validate and build upon their work. However, the paper could benefit from more detailed descriptions of the training process and hyperparameter settings to facilitate easier replication of results.
Limitations
The paper acknowledges limitations, particularly regarding the fidelity of audio reconstruction due to the low frame rate of the VAE. Additionally, the challenge of improving vocal modeling without sacrificing creativity remains a significant hurdle. The authors also note that open-source models generally do not match the performance of commercial systems, indicating a need for further advancements.
Broader impact
The implications of this work are substantial, as it addresses the growing demand for high-quality, automated music generation tools. The potential for misuse in creating disinformation or deepfake audio is a concern that the authors recognize, and they express a commitment to responsible advancement in the field. This research could pave the way for innovative applications in music production, entertainment, and personalized content creation. The main contribution of this paper is the introduction of DiffRhythm 2, a novel semi-autoregressive framework for high-fidelity song generation that effectively addresses lyric alignment and multi-preference optimization challenges. The comprehensive analysis of its technical contributions, methodology, and experimental results underscores its significance in advancing the state of the art in music generation.
Author of this paper? Send a correction.