Iterative Self-Learning for Expressive Text-to-Speech Synthesis
Abstract
Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a robust iterative self-learning framework for expressive TTS that effectively leverages unlabeled data to improve pseudo-label quality and synthesis performance in low-resource settings, demonstrating significant gains in both objective metrics and subjective listener preference.
Methodology
The paper proposes an Iterative Self-Learning (ISL) framework to address the scarcity of expressive labels in Text-to-Speech (TTS). The core methodological contribution is the integration of "Invert-Classify" into an iterative loop. Invert-Classify is a classifier-free technique that recovers discrete expressive labels (emotion, prominence) by optimizing input embeddings via gradient descent on a frozen generative model's loss function. The authors adapt this for flow-matching models (Matcha-TTS) by fixing stochastic sampling parameters during inversion to ensure convergence. The iterative process involves pseudo-labeling unlabeled data, retraining on the combined dataset, and repeating. The methodology is sound and addresses a specific gap in semi-supervised TTS, moving beyond simple pseudo-labeling to a refinement loop. However, the novelty is somewhat limited by the reliance on existing inversion techniques and the standard IPL protocol, adapted specifically for expressive TTS.
Results & evaluation
The evaluation is comprehensive, covering two distinct tasks (word-level prominence and utterance-level emotion) and multiple low-resource data splits (0.5% to 20%). The authors employ rigorous metrics including TTSDS2, emotion recognition F1, and subjective A/B listening tests using Bradley-Terry models. The results demonstrate that ISL improves pseudo-label accuracy and downstream synthesis quality compared to single-pass baselines, particularly in low-resource regimes. The analysis of training duration (3 vs 10 vs 50 epochs) provides valuable insight into the stability of the self-learning loop, identifying a "saturation" failure mode with excessive retraining. The use of human listening tests adds significant weight to the claims of perceptual improvement.
Reproducibility
The paper provides detailed implementation details, including hyperparameters for optimization, inversion steps, and dataset splits. The use of standard backbones (Matcha-TTS, HiFi-GAN) and public datasets (ESD, Naver-Prosody) enhances reproducibility. The description of the inversion process, including the fixing of timesteps and noise tensors, is sufficiently detailed for replication.
Limitations
The primary limitation is the reliance on a single backbone architecture (Matcha-TTS), although the authors acknowledge this. The method is sensitive to the amount of seed data; too little prevents bootstrapping, while too much yields diminishing returns. The approach assumes that the generative model can effectively invert to recover labels, which may not hold for all expressive traits or model architectures. Furthermore, the computational cost of iterative gradient-based inversion is non-trivial.
Broader impact
This work contributes to the democratization of expressive TTS by reducing the dependency on expensive, manually annotated expressive datasets. It enables the use of large-scale unlabeled speech corpora for training controllable TTS systems, which has broad implications for accessible speech synthesis, virtual assistants, and content creation. The insights into iterative self-learning dynamics are also applicable to other semi-supervised speech tasks. This paper presents a robust iterative self-learning framework for expressive TTS that effectively leverages unlabeled data to improve pseudo-label quality and synthesis performance in low-resource settings, demonstrating significant gains in both objective metrics and subjective listener preference.
Author of this paper? Send a correction.