Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations
Abstract
Mean opinion score (MOS) prediction models are widely used as proxy metrics in text-to-speech (TTS) research, yet their ability to capture quality differences beyond acoustic fidelity remains unclear. We investigate this via controlled perturbations on speech: acoustic degradation, prosodic errors, and manipulation of speaker-specific characteristics such as pitch and speaking rate. We obtained MOS predictions for these speech samples from both human listeners and the model, and analyzed the differences in their perceptual characteristics. Results show that most models track acoustic degradation well, while all are insensitive to prosodic errors despite large subjective score drops. For speaker characteristics, models exhibit a double dissociation: strong mean fundamental frequency (F0) biases absent in human ratings, yet insensitivity to speaking rate and F0 variability that humans notice. These findings highlight limitations of scalar MOS prediction beyond acoustic fidelity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper provides a critical diagnostic analysis of automatic MOS prediction models, demonstrating their insensitivity to prosodic errors and misalignment with human perception regarding speaker characteristics, thereby highlighting the need for more perceptually aligned evaluation metrics in speech synthesis research.
Methodology
The paper employs a rigorous diagnostic methodology to probe the internal representations and output sensitivities of existing automatic MOS prediction models (UTMOS, DNSMOS, NISQA, SHEET-MB, SHEET-BV). By applying controlled, independent perturbations to speech signals—specifically acoustic degradation, prosodic (accent) errors, and speaker characteristics (pitch/rate)—the authors isolate specific quality dimensions. This "ablation-style" evaluation of the *evaluator* rather than the *generator* is a sophisticated approach to understanding model failure modes. The use of Japanese pitch-accent language provides a critical test case for prosodic sensitivity that is often overlooked in English-centric benchmarks. The methodology is sound, well-controlled, and directly addresses the gap between scalar model outputs and multidimensional human perception.
Results & evaluation
The experimental design is robust. Group A confirms that models generally track acoustic fidelity well (with some exceptions like SHEET-MB on MP3 compression), validating the baseline utility of these models. Group B provides the most striking result: a complete dissociation between human sensitivity to prosodic errors (large MOS drops) and model insensitivity (negligible score changes). This is a significant finding. Group C reveals a "double dissociation" in speaker characteristics: models are biased by mean F0 (likely due to training data distributions) while ignoring F0 variability and speaking rate, which humans notice. The statistical reporting (SRCC, Pearson r, confidence intervals) is adequate. The comparison across six distinct models allows for a comprehensive view of the field's current state.
Reproducibility
The paper provides sufficient detail for reproduction. It specifies the datasets (JVS, NANSY-TTS internal data), the perturbation methods (SiFi-GAN for pitch, WORLD for rate), and the models evaluated (using the VERSA toolkit). The specific conditions (e.g., clipping levels, SNR values, accent swap probabilities) are clearly defined. However, the use of an "internal dataset" for the TTS generation in Group B limits full independent reproduction of that specific subset, though the methodology is clear. The VERSA toolkit ensures standard inference, enhancing reproducibility for the model evaluation part.
Limitations
The primary limitation is the reliance on a single language (Japanese) for the prosodic analysis. While Japanese is a pitch-accent language, it is unclear if these findings generalize to tonal languages (e.g., Mandarin) or non-tonal languages with different prosodic structures (e.g., stress-accent languages like English). Additionally, the study focuses on *naturalness* and *quality* as perceived by humans, but does not explore whether these model biases correlate with other metrics like intelligibility or speaker similarity. The sample size for the subjective evaluation (15 listeners) is on the lower end for robust statistical power, though acceptable for a diagnostic study.
Broader impact
This work has significant implications for the TTS and speech processing communities. It challenges the uncritical use of automatic MOS predictors as proxies for human evaluation, particularly in contexts where prosody and speaker identity are crucial (e.g., expressive TTS, voice conversion). It highlights that current SSL-based models may encode spurious correlations (like mean F0) rather than perceptually relevant features. This could guide future model training, suggesting the need for explicit prosodic supervision or multi-objective loss functions that account for speaker variability. It serves as a cautionary tale for relying solely on scalar metrics in AI research. This paper provides a critical diagnostic analysis of automatic MOS prediction models, demonstrating their insensitivity to prosodic errors and misalignment with human perception regarding speaker characteristics, thereby highlighting the need for more perceptually aligned evaluation metrics in speech synthesis research.
Author of this paper? Send a correction.