HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech
Abstract
Recently, Large Language Model (LLM)-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised Fine-Tuning paradigm often converges to statistically averaged prosody, limiting emotional expressiveness. While preference-driven optimization offers a promising alternative, existing approaches suffer from two structural mismatches: information conflict, where content and emotion in a shared latent space produce conflicting gradients, leading to reward hacking and semantic degradation; and scale gap, where sparse sentence-level rewards struggle to guide dense frame-level generation. To overcome these challenges, we propose HPRO, a hierarchical progressive reward optimization framework. Within HPRO, we introduce the HD-Emo codec as a novel differentiable reward model to resolve the information conflict. It extracts speech into distinct content and style preference tokens, structurally isolating emotional optimization from semantic content. Building upon this structured preference space, HPRO bridges the scale gap by progressively aligning frame-, word- and sentence-level objectives. Experiments demonstrate that HPRO significantly enhances emotional expressiveness, while effectively preserving linguistic intelligibility. The code and audio samples are publicly available at https://xxh333.github.io/hpro-demo/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[HPRO introduces a hierarchical progressive reward optimization framework with a novel HD-Emo codec that disentangles content and style in speech tokens, effectively resolving information conflict and scale gap issues in emotional TTS.] This paper presents a significant technical advancement in emotional TTS by addressing the fundamental challenges of gradient conflict and credit assignment in preference-based optimization. The proposed HD-Emo codec provides a structured latent space that allows for independent optimization of semantic and emotional attributes, leading to superior performance in both naturalness and emotional expressiveness while maintaining high intelligibility. The progressive optimization strategy further stabilizes training and enhances the model's ability to capture multi-scale emotional nuances.
Methodology
The paper proposes HPRO, a framework addressing two specific structural mismatches in preference-driven emotional TTS: information conflict (content vs. emotion) and scale gap (sparse rewards vs. dense generation). The core technical contribution is the HD-Emo codec, a differentiable reward model that disentangles speech into content and style preference tokens using Finite Scalar Quantization (FSQ). This allows for separate supervision: ASR for content and hierarchical emotional objectives (SER, wVAD) for style. The optimization is progressive, moving from frame-level alignment to word-level and finally sentence-level rewards. This approach is methodologically sound and addresses a genuine pain point in current LLM-based TTS systems where emotional intensity often degrades intelligibility. The use of a differentiable reward model to bypass policy gradient instability is a strong technical choice, aligning with recent trends in differentiable RL for discrete generation.
Results & evaluation
The experimental setup includes comparisons against strong baselines like CosyVoice2/3, IndexTTS2, and HD-PPT. The evaluation covers both subjective metrics (MOS-N, MOS-E) and objective metrics (WER, wVAD-CCC, EMO-SIM, DNSMOS). The results show HPRO achieving the best MOS-N and competitive MOS-E, with significant improvements in WER and emotional similarity metrics compared to baselines. The ablation studies effectively demonstrate the contribution of each component (frame, word, sentence levels) and the necessity of the disentanglement. The inclusion of a simulated DiffRO baseline highlights the advantage of the hierarchical approach. However, the reliance on external models (Whisper, emotion2vec) for evaluation introduces some dependency, though the authors note this prevents metric optimization bias.
Reproducibility
The paper provides detailed implementation details, including dataset splits, model architectures (Conformer, Qwen2.5-0.5B), and training hyperparameters. The code and audio samples are made publicly available via a GitHub Pages demo. The use of standard tools (MFA, Whisper) and open-source backbones enhances reproducibility. The specific architecture of the HD-Emo codec is described in sufficient detail for replication.
Limitations
The method relies heavily on pre-trained models (Whisper, emotion2vec, Wav2vec2) for supervision, which may limit its generalizability if these models have biases or fail on out-of-distribution data. The progressive training strategy, while effective, adds complexity to the training pipeline. The performance gain in emotional expressiveness comes with a slight trade-off in fine-grained word-level prosody (as noted in the ablation), which might be noticeable in critical applications. Additionally, the evaluation is limited to specific datasets (LibriSpeech, LSSED, EmoVoice-DB), and generalization to other languages or highly diverse emotional spectra is not thoroughly explored.
Broader impact
This work contributes to the field of affective computing and speech synthesis, enabling more natural and expressive human-computer interaction. By mitigating the trade-off between emotion and intelligibility, it has potential applications in virtual assistants, audiobooks, and entertainment. The hierarchical reward framework could also be adapted for other controllable generation tasks where multiple, potentially conflicting, objectives need to be balanced. [HPRO introduces a hierarchical progressive reward optimization framework with a novel HD-Emo codec that disentangles content and style in speech tokens, effectively resolving information conflict and scale gap issues in emotional TTS.] This paper presents a significant technical advancement in emotional TTS by addressing the fundamental challenges of gradient conflict and credit assignment in preference-based optimization. The proposed HD-Emo codec provides a structured latent space that allows for independent optimization of semantic and emotional attributes, leading to superior performance in both naturalness and emotional expressiveness while maintaining high intelligibility. The progressive optimization strategy further stabilizes training and enhances the model's ability to capture multi-scale emotional nuances.
Author of this paper? Send a correction.