SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation
Abstract
Generative speech technologies are progressing rapidly, but evaluating the perceptual quality of synthetic speech remains a core challenge. Existing methods typically rely on scalar scores or binary decisions, which lack interpretability and generalization across tasks and languages. We present SpeechLLM-as-Judges, a new paradigm for enabling large language models (LLMs) to conduct structured and explanation-based speech quality evaluation. To support this direction, we introduce SpeechEval, a large-scale dataset containing 32,207 multilingual speech clips and 128,754 annotations spanning four tasks: quality assessment, pairwise comparison, improvement suggestion, and deepfake detection. Based on this resource, we develop SQ-LLM, a speech-quality-aware LLM trained with chain-of-thought reasoning and reward optimization to improve capability. Experimental results show that SQ-LLM delivers strong performance across tasks and languages, revealing the potential of this paradigm for advancing speech quality evaluation. Relevant resources will be open-sourced.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive framework for interpretable and generalizable speech quality evaluation using LLMs, addressing key challenges in the field and demonstrating strong experimental results across multiple tasks and languages.
Methodology
The paper introduces a novel approach called SpeechLLM-as-Judges, which leverages large language models (LLMs) for structured and interpretable speech quality evaluation. The methodology includes the development of a large-scale dataset (SpeechEval) with diverse tasks and languages, and the creation of SQ-LLM, a speech-quality-aware LLM trained using chain-of-thought reasoning and reward optimization. This dual-stage training enhances the model's interpretability and generalization capabilities across various evaluation tasks, a significant improvement over traditional scalar scoring methods.
Results & evaluation
The experimental setup is robust, featuring comprehensive evaluations across multiple tasks, including quality assessment, comparison, improvement suggestion, and deepfake detection. The results demonstrate that SQ-LLM outperforms existing models in terms of accuracy and interpretability, with strong correlations to human judgments. The inclusion of multilingual data and diverse evaluation metrics adds to the rigor of the experiments, showcasing the model's versatility.
Reproducibility
The paper provides detailed implementation details, including dataset construction, model architecture, and training protocols. However, the absence of a publicly accessible code repository limits full reproducibility. The authors mention that relevant resources will be open-sourced, which is promising but not yet realized.
Limitations
The study acknowledges limitations, such as the focus on only four languages and a fixed set of tasks. Future work could expand to include more languages and additional evaluation scenarios, like emotional expressiveness. The model's performance in certain languages is also noted to be less robust, indicating areas for improvement.
Broader impact
This research has significant implications for the field of speech technology, particularly in enhancing the quality evaluation of synthetic speech. The ability to provide interpretable and actionable feedback could advance the development of more reliable speech generation systems, impacting applications in voice assistants, conversational AI, and content creation. The paper presents a comprehensive framework for interpretable and generalizable speech quality evaluation using LLMs, addressing key challenges in the field and demonstrating strong experimental results across multiple tasks and languages.
Author of this paper? Send a correction.