TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis
Abstract
While generative text-to-speech (TTS) models approach human-level quality, monolithic metrics fail to diagnose fine-grained acoustic artifacts or explain perceptual collapse. To address this, we propose TTS-PRISM, a multi-dimensional diagnostic framework for Mandarin. First, we establish a 12-dimensional schema spanning stability to advanced expressiveness. Second, we design a targeted synthesis pipeline with adversarial perturbations and expert anchors to build a high-quality diagnostic dataset. Third, schema-driven instruction tuning embeds explicit scoring criteria and reasoning into an efficient end-to-end model. Experiments on a 1,600-sample Gold Test Set show TTS-PRISM outperforms generalist models in human alignment. Profiling six TTS paradigms establishes intuitive diagnostic flags that reveal fine-grained capability differences. TTS-PRISM is open-source, with code and checkpoints at https://github.com/xiaomi-research/tts-prism.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of TTS-PRISM, a multi-dimensional diagnostic framework for Mandarin speech synthesis that significantly enhances the evaluation of TTS models. This work represents a substantial advancement in the field, providing a structured approach to diagnose and improve TTS systems, which is crucial for achieving higher quality and more expressive speech synthesis.
Methodology
The authors present a comprehensive and innovative methodology through TTS-PRISM, which integrates a multi-dimensional diagnostic framework specifically tailored for Mandarin speech synthesis. The establishment of a 12-dimensional schema is a significant advancement, allowing for a nuanced evaluation of TTS outputs that goes beyond traditional metrics like MOS. The targeted synthesis pipeline, which employs adversarial perturbations and expert anchors, enhances the quality of the diagnostic dataset, making it robust for fine-grained analysis. The schema-driven instruction tuning approach is particularly noteworthy, as it embeds explicit scoring criteria into the model, promoting interpretability and reducing ambiguity in evaluations.
Results & evaluation
The experimental setup is rigorous, utilizing a 1,600-sample Gold Test Set to benchmark TTS-PRISM against leading models. The results indicate that TTS-PRISM outperforms generalist models in human alignment, demonstrating its effectiveness in diagnosing fine-grained acoustic artifacts. The use of multiple evaluation metrics, including LCC, SRCC, and MSE, provides a comprehensive assessment of model performance. The ablation studies further validate the contributions of each component, confirming the importance of negative samples and instruction tuning in achieving high alignment scores.
Reproducibility
The paper includes a clear description of the methodology and the experimental setup, which aids in reproducibility. The authors have made their framework open-source, providing access to code and checkpoints, which is a positive step towards enabling other researchers to replicate and build upon their work. However, detailed hyperparameter settings and training configurations would enhance reproducibility further.
Limitations
One of the identified limitations is the performance gap in Pronunciation Accuracy, attributed to the inherent intelligibility tolerance of ASR-pretrained models. This suggests that while TTS-PRISM excels in many areas, there are still challenges in achieving optimal performance in specific dimensions of speech synthesis. Additionally, the focus on Mandarin may limit the generalizability of the framework to other languages without further adaptation.
Broader impact
The TTS-PRISM framework has significant implications for the field of speech synthesis, particularly in enhancing the interpretability and diagnostic capabilities of TTS systems. By providing a structured approach to evaluate fine-grained acoustic features, it can facilitate advancements in TTS technologies, leading to more human-like speech generation. The open-source nature of the project encourages collaboration and innovation within the research community, potentially influencing future developments in multilingual TTS systems. The main contribution of this paper is the introduction of TTS-PRISM, a multi-dimensional diagnostic framework for Mandarin speech synthesis that significantly enhances the evaluation of TTS models. This work represents a substantial advancement in the field, providing a structured approach to diagnose and improve TTS systems, which is crucial for achieving higher quality and more expressive speech synthesis.
Author of this paper? Send a correction.