Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Abstract
Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions. Neither class reliably captures a breadth of linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a linguistically grounded, dimension-level meta-evaluation benchmark for TTS, revealing significant gaps in the diagnostic capabilities of current MOS predictors and Audio-LLM judges, thereby providing a crucial foundation for more interpretable and targeted speech quality assessment.
Methodology
The paper proposes a novel meta-evaluation framework for Text-to-Speech (TTS) systems, moving beyond holistic Mean Opinion Score (MOS) predictors. The core methodological contribution is a linguistically grounded annotation schema that deconstructs "naturalness" into 10 distinct perceptual dimensions across word, prosodic, and paralinguistic levels. The authors construct a dataset of 860 utterances with controlled errors introduced via LLM-altered IPA, acoustic manipulation, and API-enforced emotion tags. They then benchmark four neural MOS predictors and four Audio-LLM judges against this human-annotated ground truth. The methodology is rigorous in its construction of a diagnostic benchmark, though the error generation methods (particularly acoustic manipulation) represent upper-bound cases that may not reflect natural system failures. The approach to evaluating Audio-LLMs via varied prompting conditions (schema-guided vs. isolated) is a significant methodological step for understanding model capabilities.
Results & evaluation
The experimental design is comprehensive, covering a wide range of current state-of-the-art evaluators. The results clearly demonstrate that MOS predictors collapse onto acoustic signal quality rather than linguistic naturalness, and that Audio-LLM judges exhibit selective, prompt-dependent detection capabilities. The use of Krippendorff's alpha to assess inter-annotator reliability adds robustness to the human ground truth. However, the dataset size (860 samples) is relatively small for such a high-dimensional evaluation space, and the reliance on synthetic error injection limits the generalizability of the findings to real-world TTS system outputs. The correlation analysis between automated scores and human labels is well-executed but reveals a significant gap in current technology's ability to diagnose specific linguistic errors.
Reproducibility
The paper states that the dataset, annotation schema, and evaluation code are publicly released, which strongly supports reproducibility. The detailed description of the error generation pipeline, including specific tools (Praat, Parselmouth) and model versions (Cartesia Sonic-3, GPT-5), allows for replication of the dataset construction. The clear definition of the 10 dimensions and the binary rating scale facilitates consistent annotation by future researchers.
Limitations
The primary limitation is the reliance on synthetic errors. Acoustic manipulations (e.g., F0 contour changes, duration scaling) create extreme cases that may not be representative of the subtle, complex errors found in production TTS systems. Additionally, the dataset is limited to English, restricting the generalizability of the schema to other languages with different phonological and prosodic structures. The small sample size per dimension (ranging from 46 to 128) may limit the statistical power of the findings for less common error types. The evaluation of Audio-LLMs is also constrained by the specific prompts tested, and the "black box" nature of these models means the exact reasoning behind their scores remains opaque.
Broader impact
This paper has significant implications for the development and deployment of TTS systems. By providing a diagnostic benchmark, it enables developers to identify specific weaknesses in their models rather than relying on opaque holistic scores. This can lead to more targeted improvements in TTS technology. The release of the dataset and schema also contributes to the broader ML community by providing a standardized resource for evaluating speech quality along linguistically grounded dimensions. This work highlights the current limitations of automated evaluators and underscores the need for more sophisticated models that can understand and diagnose linguistic nuances in speech. This paper introduces a linguistically grounded, dimension-level meta-evaluation benchmark for TTS, revealing significant gaps in the diagnostic capabilities of current MOS predictors and Audio-LLM judges, thereby providing a crucial foundation for more interpretable and targeted speech quality assessment.
Author of this paper? Send a correction.