Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis
Abstract
Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
Methodology
The paper proposes a two-stage approach to evaluating live-streaming TTS. First, it distills a proprietary LLM (Gemini) into an open-weight model (Qwen3-Omni) to create a cost-effective pairwise judge (LPJ). The core methodological contribution is the identification and mitigation of "verdict coupling," where judges lazily align all dimension scores with a single overall preference. To solve this, the authors propose D-LPJ, which removes the overall verdict target, masks uncertain dimensions during SFT, and introduces a "span-local GRPO" strategy. This reinforcement learning technique applies normalized advantages strictly to the rationale spans of specific dimensions, preventing cross-dimension reward cancellation. The methodology is sound, leveraging standard distillation techniques but applying them to a specific failure mode in multi-dimensional evaluation. The span-local GRPO is a clever application of RLHF principles to structured output generation.
Results & evaluation
The experiments are rigorous for an industry-adjacent paper. The authors construct a curated dataset of 1,043 human-annotated pairs, controlling for lexical confounding by using matched transcripts. They evaluate on multiple test sets (T1-T5) covering different TTS models and human recordings. Key results show that the distilled model (LPJ) outperforms a single call of the teacher model (Gemini) in point accuracy, which is a strong claim. The decoupled model (D-LPJ) successfully produces varied dimension judgments, with 86.10% agreement on high-confidence sets. The Best-of-8 selection experiment demonstrates practical utility, with the judge's top choice falling within the human top-3 in 85.29% of cases. The inclusion of position bias diagnostics and ablation studies (e.g., removing stage-1 SFT) adds depth to the evaluation.
Reproducibility
Reproducibility is moderate. The paper provides detailed hyperparameters (LoRA ranks, learning rates, batch sizes) and describes the data construction process. However, the reliance on proprietary models (Gemini for distillation, Qwen3-Omni as base) and internal datasets (TaoLive data) limits full external reproducibility. The code is not explicitly linked, and the specific "span-local GRPO" implementation details, while described mathematically, would require significant effort to replicate without the exact codebase. The use of internal human annotations for the test sets is a strength for validity but a barrier for independent verification.
Limitations
The primary limitation is the dependence on proprietary teacher models for distillation, which may not be accessible to all researchers. The evaluation is heavily focused on live-streaming TTS, which is a specific niche; generalizability to other speech domains (e.g., audiobooks, conversational agents) is not tested. The "span-local GRPO" is a novel idea but its computational cost and stability compared to standard GRPO are not deeply analyzed. The paper acknowledges that the decoupled model is currently limited to core dimensions (C1-C4) and does not fully address the sparse conditional dimensions (L1-L3) in the decoupled setting.
Broader impact
This work has significant impact on the field of speech synthesis evaluation. By providing a scalable, cost-effective alternative to expensive LLM-based judges, it enables more frequent and fine-grained evaluation of TTS systems, which is crucial for RLHF-based optimization. The concept of "verdict coupling" and its mitigation via span-local rewards could be applied to other multi-dimensional evaluation tasks in NLP and audio. The Best-of-N selection framework demonstrated here is directly applicable to production TTS pipelines. The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
Author of this paper? Send a correction.