TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion
Abstract
Traditional emotional voice conversion (EVC) conditions generation on explicit target emotions like labels or references, defining the target affective state but omitting the direction or nature of the transition. We introduce instruction-guided relative emotional voice conversion, a task where natural-language instructions specify source-conditioned affective transformations (e.g., "make the speech slightly calmer" or "sound noticeably more confident") instead of fixed targets. To support this task, we construct TRACE-Instruct, a dataset of relative emotion instructions covering categorical transitions, intensity modifications, and open-ended affective changes. We propose TRACE-EVC, a zero-shot framework built around Emo-Compass, a module that models each conversion as a source-anchored rectified flow. Rather than conditioning on an explicit target, it predicts the direction and degree of the affective change. Experiments demonstrate that TRACE-EVC accurately follows relative emotion instructions while preserving speaker identity, linguistic content, and speech quality, and remains competitive with conventional EVC systems on standard categorical emotion conversion.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
Methodology
The paper introduces a novel task formulation: instruction-guided relative emotional voice conversion (EVC). Unlike traditional EVC which maps to absolute emotional states (labels or references), this work conditions generation on natural language instructions describing the *change* in emotion relative to the source (e.g., "make it calmer"). The core technical innovation is "Emo-Compass," a module that models this transformation as a source-anchored rectified flow. Instead of predicting an absolute emotion embedding from noise, it predicts the velocity vector (displacement) in a continuous affective space (combining emotion2vec, VAD, and prosodic features) from the source anchor. This is a sophisticated application of continuous normalizing flows/rectified flows to affective computing. The synthesis module adapts DurFlex-EVC to use these continuous embeddings. The approach is theoretically sound and addresses a genuine gap in fine-grained, directional control of speech emotion.
Results & evaluation
The authors evaluate on two datasets: ESD (inter-emotion) and MEAD (intra-emotion intensity). They compare against strong baselines including label-based (SGEVC, DurFlex-EVC) and reference-based (ZEST, VEVO) systems. Results show TRACE-EVC achieves competitive or superior performance in emotion similarity (EECS), classification accuracy, and speaker similarity (SECS) compared to baselines, while maintaining high speech quality (UTMOS/NISQA). Crucially, they introduce a human evaluation metric for "Instruction Following" (IF), which is essential for this task. The ablation studies on Emo-Compass components (VAD, prosody) are thorough and validate the design choices. The inclusion of unseen-speaker zero-shot evaluation on MEAD strengthens the claim of generalization.
Reproducibility
The paper provides detailed implementation details, including model architectures (6-layer Transformer for Emo-Compass), training hyperparameters (AdamW, LR 1e-4), and datasets. They release code and demos. The dataset construction process (using LLMs to generate instructions from paired speech and VAD differences) is described, though the specific prompts and filtering rules could be more detailed for perfect reproducibility. However, the core code release mitigates this.
Limitations
The reliance on LLMs for dataset construction introduces potential biases or hallucinations in the instruction generation, although rule-based filtering helps. The evaluation is limited to English. The "zero-shot" claim relies on the disentanglement capabilities of the underlying DurFlex-EVC backbone; if the source speaker's emotion is very strong, the relative control might struggle to override it completely, a common issue in EVC. The comparison with other prompt-based EVC methods is limited by the lack of public implementations, which is a fair point but leaves a gap in direct comparison with similar prompt-based approaches.
Broader impact
This work enables more natural and intuitive human-computer interaction in speech synthesis, allowing users to specify nuanced emotional adjustments via natural language. It has applications in audiobooks, dubbing, and assistive communication. The dataset TRACE-Instruct is a valuable resource for the community. Potential misuse includes generating deceptive emotional speech, though this is a general risk in EVC. This paper presents a significant methodological advance in emotional voice conversion by shifting from absolute target conditioning to relative, instruction-guided transformation using source-anchored rectified flows, offering a more intuitive and flexible control interface for expressive speech synthesis.
Author of this paper? Send a correction.