Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens
Abstract
Objective evaluation of synthesized speech is critical for advancing speech generation systems, yet existing metrics for intelligibility and prosody remain limited in scope and weakly correlated with human perception. Word Error Rate (WER) provides only a coarse text-based measure of intelligibility, while F0-RMSE and related pitch-based metrics offer a narrow, reference-dependent view of prosody. To address these limitations, we propose TTScore, a targeted and reference-free evaluation framework based on conditional prediction of discrete speech tokens. TTScore employs two sequence-to-sequence predictors conditioned on input text: TTScore-int, which measures intelligibility through content tokens, and TTScore-pro, which evaluates prosody through prosody tokens. For each synthesized utterance, the predictors compute the likelihood of the corresponding token sequences, yielding interpretable scores that capture alignment with intended linguistic content and prosodic structure. Experiments on the SOMOS, VoiceMOS, and TTSArena benchmarks demonstrate that TTScore-int and TTScore-pro provide reliable, aspect-specific evaluation and achieve stronger correlations with human judgments of overall quality than existing intelligibility and prosody-focused metrics.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of TTScore, a novel evaluation framework for synthesized speech that provides targeted assessments of intelligibility and prosody through conditional prediction of discrete speech tokens. This work significantly advances the field by addressing the limitations of existing metrics and aligning more closely with human perceptions of speech quality.
Methodology
The paper introduces TTScore, a novel evaluation framework that utilizes conditional prediction of discrete speech tokens to assess intelligibility and prosody in synthesized speech. The methodology is well-structured, employing two distinct sequence-to-sequence models tailored for intelligibility (TTScore-int) and prosody (TTScore-pro). This targeted approach addresses the limitations of existing metrics, such as WER and F0-RMSE, by providing reference-free evaluations that align more closely with human perception. The use of discrete speech tokens derived from advanced models like HuBERT and FACodec adds robustness to the evaluation process.
Results & evaluation
The experiments are comprehensive, utilizing multiple benchmarks (SOMOS, VoiceMOS, TTSArena) to validate the effectiveness of the proposed metrics. The paper reports strong correlations between TTScore metrics and human judgments of speech quality, outperforming traditional metrics. The evaluation setup is rigorous, comparing TTScore against established baselines and demonstrating its reliability across diverse datasets. However, the paper could benefit from a more detailed analysis of the statistical significance of the results.
Reproducibility
The authors provide a GitHub repository with code and pre-trained models, enhancing the reproducibility of their work. Implementation details are sufficiently described, including model architectures and training procedures. However, the paper lacks specific hyperparameter settings and training configurations that could further aid in reproducing the results.
Limitations
One limitation is the reliance on existing datasets for evaluation, which may not encompass all variations of synthesized speech. Additionally, while TTScore shows improved correlations with human judgments, it may still be sensitive to the quality of the underlying speech synthesis systems. The paper does not address potential biases in the datasets used for training and evaluation.
Broader impact
The proposed evaluation framework has significant implications for the field of speech synthesis, offering a more nuanced understanding of intelligibility and prosody. This can lead to improved speech generation systems, enhancing applications in assistive technologies, human-computer interaction, and language learning. The methodology could also inspire further research into targeted evaluation metrics in other domains of machine learning. The main contribution of this paper is the introduction of TTScore, a novel evaluation framework for synthesized speech that provides targeted assessments of intelligibility and prosody through conditional prediction of discrete speech tokens. This work significantly advances the field by addressing the limitations of existing metrics and aligning more closely with human perceptions of speech quality.
Author of this paper? Send a correction.