MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation
Abstract
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
Methodology
The paper proposes MPEcho, a framework for Controllable Cover Song Generation (CSG) that integrates explicit phoneme-level conditioning into the SongEcho architecture. The core methodological contribution is the adaptation of Singing Voice Synthesis (SVS) techniques—specifically a length regulator and phoneme encoder—into a diffusion-based full-song generation model. This addresses the limitation of previous models (SongEcho) that relied on coarse V/UV tags, leading to high phoneme error rates (PER). The authors also introduce Phonsa, a Whisper-based automatic transcription model optimized for singing voices, to provide the necessary high-precision phoneme-level annotations. The methodology involves a multi-condition guidance strategy using Adaptive Projection Guidance (APG) to disentangle text, lyrics, melody, and phoneme conditions. The approach is technically sound, leveraging established SVS priors to solve a specific controllability issue in music generation. While the integration of SVS components into LTS/CSG is not entirely new in concept, the specific implementation of a length regulator for end-to-end cover generation and the dedicated alignment model (Phonsa) for this domain represent a solid engineering contribution.
Results & evaluation
The experimental section is comprehensive, including objective metrics (Audiobox, CLAP, RPA, RCA, PER) and subjective evaluations (MOS). The results demonstrate that MPEcho significantly reduces PER compared to SongEcho (0.1865 vs 0.4562) while maintaining or improving musical quality metrics. Ablation studies effectively isolate the contributions of melody conditioning, phoneme conditioning, and the specific guidance strategies. The evaluation of Phonsa against MFA shows substantial improvements in alignment accuracy. However, the reliance on a Chinese-only dataset limits the generalizability of the findings. The objective metrics, particularly CLAP, show low scores across the board, which the authors attribute to domain mismatch, a valid but somewhat concerning observation for a generative model's performance assessment. The subjective evaluation adds valuable human-centric validation, confirming the objective trends.
Reproducibility
The paper provides detailed implementation details, including dataset sizes, training hardware, hyperparameters, and model architectures. The code and weights are made available via a GitHub page, which significantly enhances reproducibility. The description of the Phonsa model and the MPEcho architecture is sufficiently detailed for replication. The use of standard datasets (M4Singer, Opencpop, GTsinger) for Phonsa training and a curated internal dataset for MPEcho allows for verification, although the internal dataset's exact composition is less transparent than public benchmarks.
Limitations
The primary limitation is the restriction to single-singer scenarios and Mandarin language. The model's performance on multi-singer tracks or languages with different phonetic structures is untested. The reliance on Phonsa for alignment means that errors in Phonsa will propagate to MPEcho. Additionally, the internal dataset used for training MPEcho is not publicly available, which hinders direct comparison with other CSG models trained on different data. The low CLAP scores suggest that the generated audio may not always align well with text prompts in a broad semantic sense, although this is common in specialized music generation tasks.
Broader impact
This work contributes to the field of generative AI for music by improving the controllability of cover song generation. Accurate lyric rendering is crucial for user acceptance and creative utility in AI music tools. The development of Phonsa also advances the field of automatic speech recognition and alignment for singing, which has broader applications in music information retrieval and education. However, the ease of generating high-quality covers raises ethical concerns regarding copyright and artist likeness, which should be addressed in future deployments. MPEcho introduces a novel integration of SVS-style phoneme-level control into end-to-end cover song generation, significantly improving lyrical accuracy while maintaining melodic fidelity. This work demonstrates the value of cross-domain priors (SVS to CSG) and provides a robust framework for controllable music generation, advancing the state-of-the-art in both technical methodology and practical application.
Author of this paper? Send a correction.