CLASVS: Continuous-Latent Autoregression for Melody-Preserving Lyric Editing in Singing Voice Synthesis
Abstract
Reference-conditioned melody-preserving lyric editing replaces words while retaining a performance's timing, singer identity, and naturalness. Continuous-latent autoregression avoids finite codebooks and offers stepwise generation with learned stopping. Editing creates a conflict absent from ordinary reconstruction: training pairs reference cues with original lyrics, whereas inference asks revised lyrics to override source-lyric-correlated cues; one source-following patch can propagate through AR history. We introduce CLASVS. Its State-Control-Transition (SCT) routing keeps target-lyric and reference-melody controls persistent, returns semantic feedback on phonetic progress to the causal planner, and confines the previous latent patch to the local Transition. Progressive State-Control Grounding (PSCG) learns this contract through paired-edit-free, content-consistent Mandarin reconstruction. On two Mandarin benchmarks, CLASVS improves all four operations over discrete-AR Vevo2 and reduces macro-PER by 46.2%, while maintaining melody, singer similarity, and perceptual quality. Together, these results establish a strong continuous-AR operating point for score-annotation-free lyric edits and a basis for broader stepwise control. Audio demonstrations are available on our project page: https://piedpiperg.github.io/Liyric-SVS/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
Methodology
The paper proposes CLASVS, a continuous-latent autoregressive model for melody-preserving lyric editing in singing voice synthesis. The core methodological contribution is the State-Control-Transition (SCT) routing mechanism, which attempts to resolve the conflict between preserving reference melody/timing and generating new lyrics. It separates the control signals (target lyrics, reference melody) from the generative state (semantic feedback, previous latent patch). The training strategy, Progressive State-Control Grounding (PSCG), relies on content-consistent reconstruction (training on the same audio/transcript) rather than paired counterfactual edits, which is a significant theoretical challenge. The use of a frozen causal semantic encoder for phonetic progress feedback is a novel architectural choice for maintaining lyric alignment in an autoregressive setting. However, the reliance on existing components (AudioVAE, Qwen3-0.6B, Flow-DiT) limits the novelty of the underlying generative primitives, focusing innovation primarily on the routing and training protocol.
Results & evaluation
The evaluation is comprehensive, comparing CLASVS against discrete-AR (Vevo2) and continuous-NAR (YingMusic-Singer-Plus) baselines. The paper reports significant improvements in macro-Phone Error Rate (PER) and perceptual metrics (MOS) over Vevo2. The inclusion of a "target-preference margin" metric to audit whether the model actually follows target lyrics rather than reverting to source lyrics is a strong methodological addition to the evaluation suite. The ablation studies effectively isolate the contributions of the SCT components. However, the evaluation is restricted to Mandarin, and the comparison with YingMusic+ shows mixed results (CLASVS is better at deletion/insertion, but YingMusic+ is better at substitution), suggesting that the autoregressive approach may have limitations in handling large lexical changes compared to parallel generation.
Reproducibility
The paper provides detailed descriptions of the architecture, training stages, and hyperparameters. It mentions the release of code and checkpoints upon publication, which supports reproducibility. The use of standard datasets (Emilia, curated singing) and public baselines facilitates comparison. The specific "CLA-LyricEdit-320" benchmark is described in detail, allowing for independent verification.
Limitations
The primary limitation is the lack of paired counterfactual training data, which forces the model to generalize from reconstruction tasks. This may limit its ability to handle complex edits or non-Mandarin languages. The autoregressive nature of the model results in slower inference speeds compared to NAR baselines like YingMusic+. The evaluation is limited to Mandarin, and cross-lingual generalization is not tested. Additionally, the reliance on automatic melody extraction (chroma-derived tokens) may introduce errors that propagate through the generation process.
Broader impact
This work contributes to the field of controllable singing voice synthesis, enabling more natural and flexible lyric editing without the need for manual score annotations. This has applications in music production, karaoke, and accessibility. However, the ability to edit lyrics while preserving singer identity raises ethical concerns regarding voice impersonation and copyright, which the authors acknowledge and address with proposed safety measures. CLASVS introduces a novel continuous-latent autoregressive framework with a State-Control-Transition routing mechanism that effectively addresses the conflict between melody preservation and lyric editing in singing voice synthesis, achieving state-of-the-art performance in Mandarin lyric editing tasks.
Author of this paper? Send a correction.