SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows
Abstract
Existing training-based speech emotion editing methods often require substantial task-specific training and can be unstable. This motivates us to investigate whether the pretrained generative dynamics of large-scale text-to-speech (TTS) models can be directly manipulated for training-free emotion editing. To answer this question, we probe the editability of pretrained flow-matching and hybrid TTS models by constructing a controlled test set and systematically diagnosing editing effects along the generative trajectory. Our analysis reveals that pretrained TTS models are substantially editable in emotion, but such editability is architecture- and trajectory-dependent and can be disrupted by early flow-matching steps, while cross-speaker emotion transport carries additional acoustic attributes beyond emotion. To address these limitations, we propose SEmoEdit, the first training-free framework that formulates emotion editing as dynamic velocity transport between source and target emotions, enabling robust, flow-based speech emotion editing directly within pretrained TTS models. SEmoEdit unifies three core operations: emotion replacement, emotion erasure, and continuous emotion interpolation, requiring neither parameter updates nor task-specific optimization. To systematically evaluate these capabilities, we introduce SEmoEditBench, a dataset comprising 600 editing cases, and conduct extensive experiments across state-of-the-art (SOTA) models and backbones. Our results show that SEmoEdit is highly effective and broadly applicable, outperforming existing training-based and activation-steering methods. Ultimately, this work reveals that pretrained speech flows possess rich, latent emotion-editing capabilities, providing useful guidance for real applications. Code, benchmark, and Audio samples are available at https://github.com/imxtx/SEmoEdit.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel training-free framework for speech emotion editing by leveraging dynamic velocity transport in flow-matching TTS models, supported by a systematic analysis of editability and a new benchmark. The technical contribution is strong, offering a robust alternative to training-based and static steering methods, with comprehensive experiments validating its effectiveness across multiple state-of-the-art backbones.
Methodology
The paper proposes SEmoEdit, a training-free framework for speech emotion editing based on flow-matching TTS models. The core contribution is formulating emotion editing as dynamic velocity transport. Instead of using static activation steering vectors (which are often unstable and require paired data to derive), the method dynamically computes the velocity difference between source and target conditions at each step of the ODE integration. This approach is theoretically grounded in the properties of Conditional Flow Matching (CFM). The method also introduces "emotion bridging" to handle interpolation artifacts, where intermediate states are not well-formed speech samples. The inclusion of a systematic probing study (Sec. 2) to diagnose *when* and *how* editing works along the trajectory is a strong methodological addition, providing empirical justification for the design choices (e.g., skipping early steps to preserve timing).
Results & evaluation
The authors introduce SEmoEditBench, a 600-case benchmark, which is a valuable contribution to the field as standardized benchmarks for speech editing are scarce. Experiments are conducted on three SOTA backbones (F5-TTS, CosyVoice 2, IndexTTS 2), demonstrating broad applicability. The evaluation metrics are comprehensive, covering effectiveness (TEP, SES), preservation (WER, S-SIM), quality (UTMOS), and subjective evaluation (MOS). The results show that SEmoEdit outperforms training-based and activation-steering baselines. The ablation studies on trajectory timing and attribute factorization are insightful and support the paper's claims.
Reproducibility
The paper provides a GitHub link with code, benchmark, and audio samples. The methodology is described in sufficient detail (equations for velocity transport, noise coupling, and bridging) to allow reproduction. The use of standard open-source TTS models (F5-TTS, CosyVoice 2) further enhances reproducibility.
Limitations
The method relies on the quality of the underlying TTS model's velocity field; if the base model is poor at emotion generation, the editing will suffer. The "emotion bridging" step for interpolation adds computational overhead and complexity. The benchmark, while useful, is relatively small (600 cases) compared to large-scale speech datasets. The paper focuses on English and Chinese (implied by ESD/IEMOCAP/RAVDESS/CREMA-D usage), and generalization to other languages is not explicitly tested.
Broader impact
This work has significant implications for the controllability of generative speech models. By demonstrating that pretrained models contain latent editing capabilities that can be unlocked via inference-time manipulation, it opens the door to more flexible and efficient speech editing pipelines without the need for fine-tuning. This is particularly relevant for applications in dubbing, accessibility, and personalized voice assistants where emotional nuance is important. The insights into trajectory-dependent editability can guide the design of future controllable TTS systems. The paper presents a novel training-free framework for speech emotion editing by leveraging dynamic velocity transport in flow-matching TTS models, supported by a systematic analysis of editability and a new benchmark. The technical contribution is strong, offering a robust alternative to training-based and static steering methods, with comprehensive experiments validating its effectiveness across multiple state-of-the-art backbones.
Author of this paper? Send a correction.