Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Abstract
Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel multidimensional pairwise evaluation framework for TTS systems in Indian languages, significantly advancing the methodology for evaluating speech synthesis quality. The comprehensive analysis of TTS systems across multiple perceptual dimensions and the large-scale data collection provide valuable insights that can inform future developments in the field.
Methodology
The paper presents a well-structured controlled multidimensional pairwise evaluation framework tailored for multilingual TTS systems. The methodology includes a comprehensive benchmark construction with a diverse set of sentences across 10 Indic languages, addressing real-world linguistic phenomena such as code-mixing. The use of a multi-stage rater recruitment and training protocol enhances the reliability of human evaluations. The incorporation of perceptual dimensions and the Bradley-Terry model for ranking adds depth to the evaluation process, making it a significant advancement over traditional methods.
Results & evaluation
The experiments are extensive, involving over 120K pairwise comparisons from 1900+ native raters. The evaluation of 7 state-of-the-art TTS systems across multiple languages and perceptual dimensions provides a robust analysis of system performance. The results, including the construction of a multilingual leaderboard and insights into perceptual drivers of preference, are well-documented and demonstrate the effectiveness of the proposed evaluation framework.
Reproducibility
While the paper outlines the methodology and evaluation process in detail, the lack of a publicly available dataset or code repository limits reproducibility. The authors mention that the benchmark and preference data will be released, which is a positive step towards enabling future research but currently hinders immediate reproducibility.
Limitations
The study primarily focuses on TTS systems within the context of Indian languages, which may limit the generalizability of the findings to other languages or regions. Additionally, while the evaluation framework is robust, the reliance on subjective human judgments introduces variability that may affect the consistency of results across different contexts.
Broader impact
The findings have significant implications for the development of TTS systems in multilingual contexts, particularly in regions with high linguistic diversity like India. The proposed evaluation framework can serve as a model for future research in TTS and other areas of speech synthesis, potentially leading to improved accessibility and user experience in voice-driven applications. The paper introduces a novel multidimensional pairwise evaluation framework for TTS systems in Indian languages, significantly advancing the methodology for evaluating speech synthesis quality. The comprehensive analysis of TTS systems across multiple perceptual dimensions and the large-scale data collection provide valuable insights that can inform future developments in the field.
Author of this paper? Send a correction.