CosyEdit2: Speech-Editing-Oriented Reinforcement Learning Unlocks Better Zero-Shot TTS
Abstract
Speech editing and zero-shot Text-to-Speech (TTS) share a similar generative foundation conditioned on speech prompts, yet speech editing demands far stricter local acoustic consistency with surrounding unedited content. While prior work has shown that Supervised Fine-Tuning (SFT) enables TTS models to acquire functional editing capability, this approach remains fundamentally bottlenecked by imperfect paired editing data and coarse-grained optimization signals. To address these limitations, we propose CosyEdit2, a speech editing model built on a two-stage post-training framework that progresses from supervised editing initialization to editing-oriented Group Relative Policy Optimization (GRPO) over target-speech-free data. Extensive experiments demonstrate that CosyEdit2 not only substantially advances speech editing performance, but also unlocks better zero-shot TTS capability, revealing a deeper mutual relationship between the two tasks. Audio samples are available at https://cjy1018.github.io/CosyEdit2.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents CosyEdit2, a novel framework that enhances speech editing and zero-shot TTS through innovative reinforcement learning techniques and a well-structured methodology. The contributions are significant, addressing key limitations in the field and paving the way for future advancements in audio processing technologies.
Methodology
The paper introduces CosyEdit2, a two-stage post-training framework that innovatively combines supervised fine-tuning with reinforcement learning (GRPO) to enhance speech editing capabilities while also improving zero-shot TTS performance. The methodology is well-structured, addressing the limitations of previous approaches by eliminating the need for imperfect paired data and optimizing through editing-specific rewards. The architecture leverages a unified text-speech language model and a conditional flow-matching model, showcasing a novel integration of LLMs with audio processing.
Results & evaluation
The experiments are extensive, utilizing multiple benchmarks for both speech editing and zero-shot TTS. The results demonstrate significant improvements over existing models, particularly in terms of acoustic consistency and editing fidelity. The use of both objective and subjective evaluation metrics strengthens the findings, providing a comprehensive assessment of the model's performance.
Reproducibility
The paper provides detailed training and evaluation setups, including data sources, training parameters, and model architectures, which facilitate reproducibility. However, access to the datasets used for training and evaluation may be a limiting factor for complete reproducibility.
Limitations
The authors acknowledge limitations in the design space of the reward formulation and the language coverage of the framework, which is currently constrained to a few languages. Additionally, broader acoustic editing capabilities remain unexplored, suggesting areas for future research.
Broader impact
The advancements in speech editing and zero-shot TTS have significant implications for applications in accessibility, multimedia production, and human-computer interaction. However, the potential for misuse in voice impersonation and misinformation propagation raises ethical concerns that need to be addressed through responsible deployment practices. The paper presents CosyEdit2, a novel framework that enhances speech editing and zero-shot TTS through innovative reinforcement learning techniques and a well-structured methodology. The contributions are significant, addressing key limitations in the field and paving the way for future advancements in audio processing technologies.
Author of this paper? Send a correction.