PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors
Abstract
Existing mean opinion score (MOS) prediction models typically predict utterance-level naturalness MOS and can be insensitive to localized pitch-accent errors. We propose Pitch-Accent-focused Speech Quality Assessment (PASQA), which explicitly targets pitch-accent correctness. To train our model, we construct a controlled Japanese accent-error dataset by changing accent patterns using an accent-controllable text-to-speech system, and compute a pseudo accent-quality score from the accent-error rate. PASQA builds on self-supervised representations and employs mora-conditioned fusion, ranking loss, an auxiliary accent-error localization task, and speaker-invariant training. Experiments show that conventional models fail to preserve the ordering by accent-error severity, whereas PASQA achieves high ordering accuracy on both seen and unseen speakers. Further, PASQA shows stronger agreement with human accent-correctness judgments. The code is available at https://github.com/lycorp-jp/PASQA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
PASQA introduces a specialized speech quality assessment model for Japanese pitch accents, leveraging synthetic data and multi-task learning to achieve superior sensitivity to prosodic errors compared to general MOS predictors. This work addresses a critical niche in TTS evaluation, providing a robust, reproducible, and human-aligned metric for accent correctness that can drive improvements in synthetic speech naturalness.
Methodology
The paper proposes PASQA, a model specifically designed to assess pitch-accent correctness in Japanese speech, addressing a gap where standard MOS predictors are insensitive to localized prosodic errors. The methodology is sound and well-structured. Key innovations include the use of mora-conditioned fusion (incorporating linguistic structure into SSL features), a ranking loss (Bradley-Terry) to enforce ordinal relationships between error severities, an auxiliary frame-level error detection head for localized supervision, and a Gradient Reversal Layer (GRL) for speaker-invariant training. The use of self-supervised representations (wav2vec 2.0) as a backbone is appropriate for this task. The integration of linguistic tokens via cross-attention is a strong design choice for a language-specific prosodic task.
Results & evaluation
The experimental setup is rigorous. The authors construct a large-scale synthetic dataset with controlled accent errors, which is a significant contribution in itself, allowing for precise evaluation of sensitivity to error severity. The evaluation metrics (Order Accuracy, SRCC, KTAU) are well-chosen to test the specific hypothesis that PASQA preserves severity ordering better than conventional models. The results clearly demonstrate that PASQA outperforms both traditional acoustic feature-based models and general SSL-based MOS predictors (like UTMOS, DNSMOS, NISQA) on this specific task. The ablation study effectively isolates the contribution of each component (mora fusion, ranking loss, frame head, GRL). The inclusion of an out-of-domain (OOD) evaluation on GPT-4o-mini-TTS speech adds robustness to the claims.
Reproducibility
The paper provides sufficient detail for reproduction. It specifies the backbone (wav2vec 2.0), the TTS system used for data generation (NANSY-TTS), the loss functions, hyperparameters (learning rate, batch size, loss weights), and the dataset construction pipeline. The code is made available on GitHub, which significantly enhances reproducibility. The synthetic nature of the training data allows other researchers to replicate the data generation process exactly.
Limitations
The primary limitation is the reliance on synthetic data for training. While the controlled errors are well-defined, there may be a domain gap between synthetic accent errors and natural speech variations or errors produced by other TTS systems. The model is specifically tailored for Japanese pitch accent; while the methodology could be extended, the paper does not demonstrate cross-lingual applicability. Additionally, the "pseudo" accent-quality scores are derived from error rates, which assumes a linear or monotonic relationship between error count and perceived quality, which might not perfectly capture human perceptual nuances (e.g., context-dependent tolerance).
Broader impact
This work has significant implications for the development and evaluation of Text-to-Speech systems, particularly for languages with complex prosodic systems like Japanese. By providing a tool that aligns better with human judgments of accent correctness, it enables more efficient iteration cycles for TTS developers. It also contributes to the broader field of speech quality assessment by highlighting the need for task-specific metrics beyond general naturalness. The open-source release promotes further research in prosodic quality evaluation. PASQA introduces a specialized speech quality assessment model for Japanese pitch accents, leveraging synthetic data and multi-task learning to achieve superior sensitivity to prosodic errors compared to general MOS predictors. This work addresses a critical niche in TTS evaluation, providing a robust, reproducible, and human-aligned metric for accent correctness that can drive improvements in synthetic speech naturalness.
Author of this paper? Send a correction.