A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
Abstract
While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
Methodology
The paper employs a rigorous geometric analysis framework to compare two distinct activation steering sites (SLM and CFM) within a hybrid Text-to-Speech (TTS) architecture. The methodology is sound and well-structured, utilizing linear probing for discriminability and Local Intrinsic Dimensionality (LID) to characterize the manifold structure of emotion representations. The extraction of steering vectors via mean subtraction and the application of weighted summation for mixed emotions are standard but effectively applied in this context. The core methodological contribution lies in the systematic correlation between geometric properties (LID, discriminability gaps) and steering performance (proportional control, speaker fidelity), providing a mechanistic explanation for why certain steering sites work better than others. The analysis of joint steering interference is particularly insightful, attributing degradation to distribution shift and speaker entanglement.
Results & evaluation
The experimental setup is comprehensive, covering three major emotion datasets (ESD, CREMA-D, RAVDESS) and evaluating both emotion control metrics (E-SIM, TEP, Proportional Control, H-Rt) and speech quality metrics (S-SIM, WER). The results clearly demonstrate the trade-offs: SLM offers better proportional control and speaker preservation, while CFM offers higher intensity but suffers from speaker entanglement. The use of objective metrics like WER and S-SIM alongside emotion-specific embeddings adds robustness. However, the reliance on objective metrics for emotion perception (E-SIM, TEP) rather than human subjective evaluation (MOS/MUSHRA) is a limitation, though common in early-stage steering papers. The ablation on steering strength and the comparison of single vs. joint steering provide strong empirical evidence for the geometric claims.
Reproducibility
The paper provides sufficient detail for reproduction, including the backbone model (CosyVoice2), specific layers for steering (SLM layers 14, 17; CFM every 5th layer), and the datasets used. The mathematical definitions for LID and steering vector extraction are clear. However, the code is not explicitly linked in the text provided, and some hyperparameters for the linear probes and LID estimation (e.g., K for nearest neighbors) are referenced via citations rather than detailed in the main text, which might slightly hinder immediate reproducibility without accessing the cited works or supplementary material.
Limitations
The primary limitation is the lack of human subjective evaluation for emotion perception and speech quality, relying solely on proxy metrics. The study is confined to a single hybrid TTS architecture (CosyVoice2), limiting the generalizability of the geometric findings to other architectures (e.g., end-to-end diffusion TTS or different hybrid designs). The joint steering analysis is limited to in-distribution data; out-of-distribution generalization of the interference effects is not explored. Additionally, the LID analysis, while insightful, is computationally expensive and sensitive to the choice of K, which is not fully ablated.
Broader impact
This work significantly advances the understanding of controllable speech generation by linking representation geometry to steering efficacy. It provides practical guidelines for developers of hybrid TTS systems, suggesting that SLM is a superior site for precise emotional mixing, while CFM is better for intensity but risky for speaker identity. The findings on speaker-emotion entanglement in flow-matching modules have broader implications for interpretability and control in generative models. By highlighting the geometric properties of latent spaces, the paper contributes to the growing field of mechanistic interpretability in audio generation. This paper presents the first comparative geometric analysis of activation steering in hybrid TTS models, revealing that Speech Language Models offer cleaner, more disentangled emotion subspaces than Conditional Flow-Matching modules, thereby providing crucial insights for designing effective and interpretable emotion control mechanisms in speech synthesis.
Author of this paper? Send a correction.