PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech
Abstract
Standard text-to-speech (TTS) evaluation measures intelligibility (WER, CER) and overall naturalness (MOS, UTMOS) but does not quantify accent. A synthesiser may score well on all four yet sound non-native on features that are phonemic in the target language. For Indic languages, these features include retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant (letter zha). We present PSP, the Phoneme Substitution Profile, an interpretable, per-phonological-dimension accent benchmark for Indic TTS. PSP decomposes accent into six complementary dimensions: retroflex collapse rate (RR), aspiration fidelity (AF), vowel-length fidelity (LF), Tamil-zha fidelity (ZF), Frechet Audio Distance (FAD), and prosodic signature divergence (PSD). The first four are measured via forced alignment plus native-speaker-centroid acoustic probes over Wav2Vec2-XLS-R layer-9 embeddings; the latter two are corpus-level distributional distances. In this v1 we benchmark four commercial and open-source systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) on Hindi, Telugu, and Tamil pilot sets, with a fifth system (Praxy Voice) included on all three languages, plus an R5->R6 case study on Telugu. Three findings: (i) retroflex collapse grows monotonically with phonological difficulty Hindi < Telugu < Tamil (~1%, ~40%, ~68%); (ii) PSP ordering diverges from WER ordering -- commercial WER-leaders do not uniformly lead on retroflex or prosodic fidelity; (iii) no single system is Pareto-optimal across all six dimensions. We release native reference centroids (500 clips per language), 1000-clip embeddings for FAD, 500-clip prosodic feature matrices for PSD, 300-utterance golden sets per language, scoring code under MIT, and centroids under CC-BY. Formal MOS-correlation is deferred to v2; v1 reports five internal-consistency signals plus a native-audio sanity check.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel accent evaluation benchmark for Indic TTS systems, offering a detailed and interpretable framework that enhances the understanding of accent fidelity in synthesized speech. The innovative methodology and significant findings position this work as a valuable contribution to the field of machine learning and speech synthesis.
Methodology
The paper proposes a novel framework, the Phoneme Substitution Profile (PSP), which quantitatively evaluates accent fidelity in Indic languages for TTS systems. The methodology is robust, utilizing a combination of acoustic probes and distributional metrics to capture phonological dimensions of accent. The use of Wav2Vec2 embeddings for forced alignment and the construction of native speaker centroids are particularly innovative, allowing for a detailed analysis of accent features that are often overlooked in traditional TTS evaluations. The six dimensions of accent fidelity (RR, AF, LF, ZF, FAD, PSD) provide a comprehensive approach to understanding TTS performance across different systems and languages.
Results & evaluation
The experiments benchmark four commercial and open-source TTS systems across Hindi, Telugu, and Tamil, showcasing the effectiveness of the PSP framework. The findings reveal significant insights into the performance of these systems, particularly the divergence between traditional intelligibility metrics (WER) and the proposed accent fidelity metrics. The detailed analysis of results across different languages highlights the varying challenges posed by phonological complexity, making the evaluation both thorough and insightful.
Reproducibility
The authors have made a commendable effort to ensure reproducibility by releasing the scoring code and native speaker centroids under open-source licenses. However, the reliance on specific aligners and the current limitations in the quality of these tools may affect the reproducibility of results, particularly for Telugu and Tamil. Future versions of the benchmark are expected to address these issues, enhancing the overall reproducibility.
Limitations
The paper acknowledges several limitations, including the dependency on forced alignment accuracy, which varies by language, and the potential noise floor in per-phoneme scores. The authors also note that the current version of the PSP does not include formal MOS calibration, which is essential for validating the proposed metrics against human judgment. Additionally, the limited size of pilot sets may affect the statistical significance of some findings.
Broader impact
The PSP framework has the potential to significantly impact the development of TTS systems for Indic languages, providing a much-needed tool for developers to optimize accent fidelity. By focusing on specific phonological features, the framework can help improve the naturalness and intelligibility of synthesized speech, making it more accessible to native speakers. This work also opens avenues for further research into accent evaluation in other languages and dialects, contributing to the broader field of speech synthesis. The paper presents a novel accent evaluation benchmark for Indic TTS systems, offering a detailed and interpretable framework that enhances the understanding of accent fidelity in synthesized speech. The innovative methodology and significant findings position this work as a valuable contribution to the field of machine learning and speech synthesis.
Author of this paper? Send a correction.