EditVoice: Variable-Length Non-Autoregressive Zero-Shot TTS and Speech Editing with Edit Flows
Abstract
Recent non-autoregressive (NAR) zero-shot text-to-speech (TTS) models generate in parallel but typically require the target sequence length to be specified before generation. We introduce EditVoice, to our knowledge the first variable-length NAR zero-shot TTS model, which uses Edit Flows to jointly update speech content and sequence length through insertions, deletions, and substitutions. EditVoice adopts speech-infilling training, which unifies zero-shot TTS and text-based speech editing and allows both prefix and suffix speech prompt placements at inference. We introduce Complementary Prompt Sampling (CPS) to leverage the complementary Edit Flow predictions induced by the two prompt placements. We further find that EditVoice can edit source and model-generated speech beyond its training sources. We use this generalization for end-to-end editing and training-free post-generation refinement. With the Edit Flow model trained on 10K h of GigaSpeech, EditVoice demonstrates competitive zero-shot TTS performance on Seed-TTS Eval EN and LibriSpeech-PC and speech editing performance on RealEdit.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces EditVoice, a variable-length non-autoregressive zero-shot TTS model using Edit Flows and Complementary Prompt Sampling, achieving competitive performance in TTS and speech editing with high inference efficiency.
Methodology
The paper proposes EditVoice, a non-autoregressive (NAR) zero-shot TTS model based on Edit Flows. The core methodological contribution is the application of Edit Flows to speech token sequences, enabling variable-length generation through insertions, deletions, and substitutions without pre-specifying sequence length. The authors introduce a speech-infilling training objective that unifies TTS and speech editing. A key heuristic contribution is Complementary Prompt Sampling (CPS), which leverages the complementary predictions of prefix and suffix prompt placements to improve generation quality, along with a local consistency rule to prevent artifacts. The method also exploits the model's ability to generalize to editing plausible (non-noise) token sequences for post-generation refinement.
Results & evaluation
Experiments are conducted on Seed-TTS Eval EN and LibriSpeech-PC for TTS, and RealEdit for speech editing. The model is trained on 10K hours of GigaSpeech. Results show competitive WER and speaker similarity compared to autoregressive baselines like CosyVoice2 and other NAR models. The 16-NFE variant achieves a low RTF of 0.0989. Ablations confirm the effectiveness of CPS and post-generation refinement. Subjective evaluations (NMOS, SMOS) are included.
Reproducibility
The paper provides detailed architectural specifications (14-layer LLaMA-style Transformer, Conformer text encoder) and training hyperparameters. It specifies the use of frozen S3Tokenizer2 and CosyVoice2 acoustic decoder. However, the code repository is not explicitly linked in the text provided (only a demo page), which may limit immediate reproducibility compared to papers with open-source code.
Limitations
The model relies on a specific tokenizer (S3Tokenizer2) and acoustic decoder (CosyVoice2), which may limit generalizability to other speech tokenization schemes. The CPS heuristic, while effective, is a manual rule-based approach that may not scale optimally to all conditions. The training data size (10K hours) is significantly smaller than some state-of-the-art systems (1M+ hours), though the paper argues for efficiency.
Broader impact
This work contributes to the efficiency of NAR TTS by removing the need for length prediction, a common bottleneck. The unification of TTS and editing in a single model is a step towards more flexible speech manipulation tools. The use of Edit Flows for speech is a novel application of this generative framework. The paper introduces EditVoice, a variable-length non-autoregressive zero-shot TTS model using Edit Flows and Complementary Prompt Sampling, achieving competitive performance in TTS and speech editing with high inference efficiency.
Author of this paper? Send a correction.