Steerspeech: Activation Steering For Emotion Control In Generated Speech
Abstract
Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SteerSpeech introduces a lightweight, multi-expert supervised activation steering framework for fine-grained emotion control in TTS models. The paper demonstrates that by learning a low-rank transform on steering vectors and using a two-pass replay pipeline with straight-through estimators, it is possible to achieve monotonic emotion intensity control while preserving speaker identity and linguistic content, outperforming naive steering baselines and generalizing to unseen and accented speakers.
Methodology
The paper proposes SteerSpeech, a lightweight framework for inference-time emotion control in Text-to-Speech (TTS) models. The core method involves learning a low-rank residual transform ($T_e$) that refines raw activation steering vectors (derived from the difference between target emotion and neutral mean activations). The key technical contribution is the training objective, which uses a multi-expert setup (emotion, speaker, and transcription experts) to enforce monotonic emotion intensity with respect to steering strength while penalizing speaker drift and transcription errors. To handle the non-differentiability of discrete speech token sampling in autoregressive TTS models, the authors introduce a two-pass generation-and-replay pipeline using a straight-through estimator (STE). This allows gradients from the frozen expert models to flow back to the lightweight transform without updating the large TTS backbone. The approach is computationally efficient (only ~33K parameters per emotion) and plug-and-play.
Results & evaluation
The experiments are conducted on the Qwen3-TTS-0.6B model using the Emotional Speech Dataset (ESD). The evaluation covers three generalization scenarios: seen speakers, unseen speakers, and accented speakers (cross-corpus). Objective metrics include emotion confidence, top-1 accuracy, speaker drift, WER, and UTMOS. Subjective evaluations involve two blinded listening studies with 55 participants. The results show significant improvements over baselines (Emotion Reference and Naive Difference-of-Means), particularly in maintaining speaker identity while increasing emotion intensity. The paper demonstrates that the method generalizes well to unseen and accented speakers, which is a strong point. The ablation studies confirm the contribution of each loss term.
Reproducibility
The paper provides sufficient detail on the architecture of the transform, the training procedure, and the specific hyperparameters (learning rate, batch size, steering strengths). However, the code and specific implementation details of the "straight-through estimator" integration with the specific TTS backbone (Qwen3-TTS) are not fully detailed in the text, and no code repository is provided. The reliance on specific frozen expert models (emotion2vec, WavLM, Whisper) is clear, but the exact version and preprocessing steps might require trial-and-error for full reproduction.
Limitations
The method is tested primarily on a single TTS backbone (Qwen3-TTS-0.6B). It is unclear how well the method generalizes to other TTS architectures (e.g., non-autoregressive or flow-based models). The emotion control is limited to a few discrete emotions (Angry, Happy, Sad) in the main experiments, and the "continuous" control is approximated by discrete steering strengths. The subjective evaluation, while positive, is limited to a small number of participants (55) and a specific emotion (Anger) for the detailed intensity study. The method requires paired emotional and neutral utterances for training the transform, which may not always be available.
Broader impact
This work contributes to the field of controllable speech synthesis by providing a lightweight, training-free (for the backbone) method for fine-grained emotion control. It addresses a practical limitation of current TTS systems where emotion control is often coarse or requires expensive fine-tuning. The approach could be extended to other attributes (accent, style) and other generative models. The use of activation steering is a growing area of interest in interpretability and control, and this paper provides a solid application to speech. SteerSpeech introduces a lightweight, multi-expert supervised activation steering framework for fine-grained emotion control in TTS models. The paper demonstrates that by learning a low-rank transform on steering vectors and using a two-pass replay pipeline with straight-through estimators, it is possible to achieve monotonic emotion intensity control while preserving speaker identity and linguistic content, outperforming naive steering baselines and generalizing to unseen and accented speakers.
Author of this paper? Send a correction.