SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level
Abstract
The evaluation of intelligibility for TTS has reached a bottleneck, as existing assessments heavily rely on word-by-word accuracy metrics such as WER, which fail to capture the complexity of real-world speech or reflect human comprehension needs. To address this, we propose Spoken-Passage Multiple-Choice Question Answering, a novel subjective approach evaluating the accuracy of key information in synthesized speech, and release SP-MCQA-Eval, an 8.76-hour news-style benchmark dataset for SP-MCQA evaluation. Our experiments reveal that low WER does not necessarily guarantee high key-information accuracy, exposing a gap between traditional metrics and practical intelligibility. SP-MCQA shows that even state-of-the-art (SOTA) models still lack robust text normalization and phonetic accuracy. This work underscores the urgent need for high-level, more life-like evaluation criteria now that many systems already excel at WER yet may fall short on real-world intelligibility.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces SP-MCQA, a novel framework for evaluating the intelligibility of TTS systems beyond traditional metrics. It significantly advances the field by addressing critical gaps in current evaluation methods and providing a comprehensive dataset for future research.
Methodology
The proposed methodology, SP-MCQA, introduces a novel subjective evaluation framework that shifts the focus from traditional word-level accuracy metrics to assessing key information accuracy in synthesized speech. This approach is innovative as it addresses the limitations of existing metrics like WER and MOS, which do not adequately reflect human comprehension needs. The detailed construction of the SP-MCQA-Eval dataset, with its emphasis on challenging real-world speech scenarios, is commendable and provides a robust foundation for evaluating TTS systems.
Results & evaluation
The experiments conducted are thorough and well-structured, utilizing a diverse set of state-of-the-art TTS models. The evaluation metrics employed, including both subjective and objective assessments, provide a comprehensive view of model performance. The results reveal significant insights into the limitations of current TTS systems, particularly regarding phonetic accuracy and text normalization, which are critical for real-world applications.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including the selection of models and the evaluation process. However, the absence of a publicly available code repository or demo limits the reproducibility of the results. Future work should focus on making the methodology and datasets accessible to encourage further research in this area.
Limitations
The primary limitation identified is the substantial manual effort required for human evaluation, which could hinder scalability. Additionally, the dataset may not encompass all possible speech variations, potentially limiting the generalizability of the findings. The reliance on human annotators also introduces variability that could affect the consistency of results.
Broader impact
This research has significant implications for the development of more intelligent TTS systems that prioritize human comprehension over mere word accuracy. By highlighting the importance of key information accuracy, it encourages the design of TTS systems that are better suited for real-world applications, potentially enhancing user experience in various domains, including education, media, and accessibility. The paper introduces SP-MCQA, a novel framework for evaluating the intelligibility of TTS systems beyond traditional metrics. It significantly advances the field by addressing critical gaps in current evaluation methods and providing a comprehensive dataset for future research.
Author of this paper? Send a correction.