AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
Abstract
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: https://any-edit.github.io/. AnyEditBench: https://huggingface.co/datasets/Ririiiii/anyeditbench.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
Methodology
The paper proposes a unified, separation-free framework for speech and singing voice editing that decouples semantic editing from acoustic rendering. The core innovation lies in the two-stage architecture: an autoregressive (AR) editor that operates in a clean latent space to predict prosody and content-style tokens, followed by a flow-matching transformer (FMT) that renders the final mel-spectrogram conditioned on the source acoustic context. This design elegantly avoids the artifacts associated with source separation while preserving environmental consistency. The introduction of a noise-robust prosody tokenizer via teacher-student distillation is a strong technical contribution, enabling stable prosodic conditioning even from degraded inputs. Additionally, the use of Direct Preference Optimization (DPO) on mined real-to-real singing pairs (leveraging song structure repetitions) is a creative and effective strategy for improving singing quality without relying on synthetic targets. The unmodified-area-aware supervision mechanism is a practical and effective solution to the common problem of prosody drift in localized editing.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors introduce AnyEditBench, a valuable new benchmark covering speech and singing in clean, simulated, and in-the-wild conditions, which fills a gap in the literature. The evaluation includes both objective metrics (WER/CER, speaker similarity, F0 correlation, background similarity, RT60/SNR error) and subjective evaluations (MOS, Fooling Rate). The results demonstrate consistent superiority over strong baselines, including Vevo2, Step-Audio-EditX, and SSR-Speech, particularly in noisy and reverberant environments. Ablation studies convincingly validate the necessity of each component, such as the noise-robust tokenizer, the staged training, and the DPO alignment. The comparison with SSR-Speech is handled well, with a clear explanation of why SSR-Speech's higher scores on some metrics are not directly comparable due to its full-utterance regeneration nature.
Reproducibility
The paper provides sufficient details for reproduction, including model architectures, training strategies, data construction pipelines, and evaluation metrics. The release of AnyEditBench on Hugging Face is a significant contribution to reproducibility and community benchmarking. However, the specific implementation details of the AR editor and FMT are somewhat high-level, and the exact hyperparameters for the DPO training and the teacher-student distillation are not fully detailed in the main text (though likely in the appendix). The use of standard components like Qwen2.5 and LLaMA-2 blocks aids in reproducibility.
Limitations
The primary limitation is the reliance on a two-stage cascaded architecture, which may introduce latency and potential error propagation from the AR stage to the FMT stage. The model's performance on extremely low SNR conditions (e.g., below 0 dB) is not extensively explored. Additionally, the framework is currently limited to Chinese and English, and its generalization to other languages or more complex polyphonic backgrounds remains to be seen. The DPO mining strategy, while clever, is specific to popular music structures and may not generalize to all types of singing or non-musical vocal performances.
Broader impact
This work has significant potential impact on audio post-production, content creation, and accessibility tools. By enabling high-quality, localized editing of real-world recordings without the need for source separation, it simplifies the workflow for creators and reduces computational overhead. The unified handling of speech and singing opens up new possibilities for music production and interactive media. The release of a comprehensive benchmark will facilitate future research in this area and enable fair comparison of different approaches. AnyEdit introduces a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments by decoupling semantic editing from acoustic rendering. The paper makes significant technical contributions, including a noise-robust prosody tokenizer, an instruction-guided autoregressive editor with unmodified-area-aware supervision, and a flow-matching acoustic model for in-context rendering, along with the release of AnyEditBench, a comprehensive benchmark for real-world vocal editing. The methodology is well-motivated, the experiments are thorough, and the results demonstrate state-of-the-art performance, making this a strong and impactful contribution to the field of audio processing.
Author of this paper? Send a correction.