MeloDISinger: Melody-Aware & Duration-Preserving Singing Voice Editing with Audio Infilling
Abstract
Text-based singing voice editing (SVE) aims to revise sung lyrics while preserving the original melody, total duration, and non-edited regions. In this paper, we propose MeloDISinger, a flow-matching-based SVE model for melody-aware and duration-preserving editing. Its core module, MeloDRP, predicts fixed-budget duration ratios, enabling explicit span-wise duration control. For melody-aware duration allocation, MeloDRP fuses phonetic cues with pseudo-MIDI melodic context through cross-attention, while temporal-overlap supervision encourages soft phoneme--note correspondences. We further use a flow-matching mel decoder for audio infilling to synthesize edited regions while preserving surrounding context. In addition, we introduce a duration-aware edited-lyric generation pipeline using WhisperX and an LLM to construct feasible evaluation scenarios. Experiments demonstrate state-of-the-art performance in both objective and subjective evaluations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[This paper presents MeloDISinger, a novel flow-matching-based singing voice editing model that introduces melody-aware duration ratio prediction to ensure strict temporal synchronization and high-quality audio infilling, achieving state-of-the-art performance in both objective and subjective evaluations.]
Methodology
The paper proposes MeloDISinger, a flow-matching-based architecture for text-based Singing Voice Editing (SVE). The core technical novelty lies in the "MeloDRP" (Melody-aware Duration Ratio Predictor) module. Unlike previous methods that predict absolute durations or reuse original phoneme durations (which fails when phoneme counts change), MeloDRP predicts duration *ratios* within a fixed budget for each edit span. This ensures strict total duration preservation, a critical constraint for synchronization with accompaniment. The method fuses phonetic cues with pseudo-MIDI melodic context via cross-attention to inform these ratios, addressing the strong link between melody and rhythm in singing. The audio generation uses a flow-matching mel decoder with an infilling strategy, conditioning on the predicted durations, pitch, and original context to seamlessly replace edited regions. The use of pseudo-MIDI derived from F0 rather than score annotations is a pragmatic and effective choice for real-world singing voice editing where pitch deviations are common.
Results & evaluation
The evaluation is comprehensive, covering six distinct editing scenarios (insertion, deletion, mixed, and three types of replacement based on phoneme/syllable matching). The authors construct a novel, duration-aware evaluation dataset using WhisperX and an LLM to ensure temporal feasibility, addressing a significant gap in prior SVE benchmarks where generated edits often violated timing constraints. Objective metrics (WER, CER, Duration Consistency, F0 Pearson Correlation) and subjective MOS scores demonstrate state-of-the-art performance against baselines like EditSinger and Vevo2. The ablation studies effectively isolate the contributions of melody conditioning, guided-attention loss, and duration ratio prediction. The results clearly show that explicit duration ratio prediction significantly outperforms methods that do not account for the fixed budget, particularly in complex replacement scenarios.
Reproducibility
The paper provides detailed implementation details, including model architectures (Transformer layers, hidden sizes), training hyperparameters (Adam optimizer, learning rate schedule), and preprocessing steps (MFA alignment, g2p-en, Parselmouth for F0). The dataset (GTSinger-En) is publicly available. However, the code is not explicitly linked in the provided text (only a demo page is listed), and the baseline "EditSinger" was reproduced from the paper rather than using a public repository, which may introduce slight implementation variances. The use of proprietary LLMs (Gemini-2.5-flash) for data generation limits full reproducibility of the evaluation dataset construction, though the pipeline is described in detail.
Limitations
The method relies on accurate pseudo-MIDI extraction from F0; poor F0 estimation or highly vibrato-heavy sections could degrade the melodic context input to the duration predictor. The assumption that a fixed budget can be strictly allocated via ratios may struggle with extreme lyrical changes where the semantic content requires significantly different rhythmic phrasing than the original, potentially leading to unnatural "speech-like" timing if the melody conditioning is insufficient. The evaluation is limited to English singing voices (GTSinger-En), and the generalizability to other languages or singing styles (e.g., rap, which has different rhythmic constraints) is not demonstrated. Additionally, the reliance on WhisperX for alignment introduces potential errors in onset/offset detection, which could affect the syllable capacity calculation.
Broader impact
This work advances the field of audio generation and music production tools by providing a robust solution for precise singing voice editing. It enables more natural and efficient post-production workflows for musicians and producers. The proposed evaluation pipeline offers a new standard for assessing temporal fidelity in SVE systems. However, the technology also raises ethical concerns regarding the potential for deepfake singing voices and the misappropriation of artists' vocal styles, necessitating responsible use guidelines. [This paper presents MeloDISinger, a novel flow-matching-based singing voice editing model that introduces melody-aware duration ratio prediction to ensure strict temporal synchronization and high-quality audio infilling, achieving state-of-the-art performance in both objective and subjective evaluations.]
Author of this paper? Send a correction.