Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing
Abstract
Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
Methodology
The paper proposes SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing. The core methodological contribution is HiCoDD, a hierarchical diffusion process that respects the Residual Vector Quantization (RVQ) structure of neural codecs (like EnCodec). Instead of flattening all codebooks into a single sequence, HiCoDD generates codebooks sequentially from coarse to fine, conditioning the diffusion on previously committed (clean) lower-level codebooks. This addresses the issue of "leakage" and ensures that the hierarchical dependency of the codec is preserved during generation. The paper also introduces span-localized classifier-free guidance (CFG) adapted for categorical scores, which randomizes phonemes only within the edited span to preserve surrounding context. A duration predictor is included to handle variable-length edits. The methodology is technically sound, well-motivated by the limitations of autoregressive models in non-causal generation tasks, and correctly applies discrete diffusion theory (DWDSE) to the hierarchical token space.
Results & evaluation
The evaluation is conducted on the RealEdit benchmark, comparing SIEDD against VoiceCraft, SSR-Speech, and a TTS baseline. Metrics include WER, Speaker Similarity (SIM), Mel Cepstral Distortion (MCD), F0/Energy distance, and UTMOS. SIEDD achieves state-of-the-art results in overall editing performance (lowest WER, highest SIM, lowest MCD) and significantly outperforms baselines in speech inpainting, particularly in multi-gap scenarios where autoregressive models struggle with error propagation. The ablation studies effectively isolate the contributions of the hierarchical structure, duration predictor, and localized CFG. The results are consistent with the claims, showing that modeling the codec hierarchy improves context preservation. The use of standard, widely accepted metrics strengthens the validity of the claims.
Reproducibility
The paper provides a GitHub link to the code and configuration files. The appendix details the architecture, training hyperparameters, and dataset construction for the duration predictor. The training setup (550k steps on two A6000 GPUs) is specified. The use of standard datasets (LibriTTS, GigaSpeech) and tokenizers (EnCodec) aids reproducibility. The detailed ablation study configurations further support reproducibility.
Limitations
The method relies on the quality of the underlying neural codec (EnCodec); errors in tokenization may propagate. The diffusion process, while effective, is computationally more expensive than autoregressive inference, although inference time is not the primary focus of this work. The duration predictor, while better than heuristics, may still struggle with extreme prosodic variations or non-speech content. The evaluation is limited to English speech.
Broader impact
This work contributes to the field of generative audio, specifically in robust speech editing and restoration. Applications include audio repair, content creation, and accessibility. The ability to edit speech with high fidelity and speaker consistency has significant implications for media production and communication. However, as with all speech editing technologies, there are potential risks regarding misinformation and deepfakes, necessitating responsible use and detection methods. The paper presents SIEDD, a novel discrete diffusion framework that leverages the hierarchical structure of neural codecs for superior text-guided speech inpainting and editing. By introducing HiCoDD, which enforces coarse-to-fine generation and leakage-free conditioning, the authors address key limitations of existing autoregressive and naive diffusion approaches, achieving state-of-the-art performance on the RealEdit benchmark.
Author of this paper? Send a correction.