Few-Shot and Pseudo-Label Guided Speech Quality Evaluation with Large Language Models
Abstract
In this paper, we introduce GatherMOS, a novel framework that leverages large language models (LLM) as meta-evaluators to aggregate diverse signals into quality predictions. GatherMOS integrates lightweight acoustic descriptors with pseudo-labels from DNSMOS and VQScore, enabling the LLM to reason over heterogeneous inputs and infer perceptual mean opinion scores (MOS). We further explore both zero-shot and few-shot in-context learning setups, showing that zero-shot GatherMOS maintains stable performance across diverse conditions, while few-shot guidance yields large gains when support samples match the test conditions. Experiments on the VoiceBank-DEMAND dataset demonstrate that GatherMOS consistently outperforms DNSMOS, VQScore, naive score averaging, and even learning-based models such as CNN-BLSTM and MOS-SSL when trained under limited labeled-data conditions. These results highlight the potential of LLM-based aggregation as a practical strategy for non-intrusive speech quality evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of GatherMOS, a novel framework that utilizes large language models for aggregating diverse acoustic signals and pseudo-labels to improve speech quality evaluation. This research represents a significant step forward in the integration of LLMs into audio processing tasks, showcasing the potential for enhanced performance in non-intrusive quality assessment systems.
Methodology
The proposed GatherMOS framework innovatively combines large language models (LLMs) with acoustic descriptors and pseudo-labels to enhance speech quality evaluation. The integration of lightweight acoustic features with pseudo-labels from existing models (DNSMOS and VQScore) allows the LLM to leverage multimodal inputs effectively. The methodology is well-structured, employing both zero-shot and few-shot learning paradigms, which is a significant advancement in the field of non-intrusive speech quality assessment. The use of structured textual prompts for LLM input is particularly noteworthy, as it facilitates the reasoning process of the model.
Results & evaluation
The experiments are comprehensive, utilizing the VoiceBank-DEMAND dataset to evaluate the performance of GatherMOS against several baselines, including traditional metrics and learning-based models. The results demonstrate that GatherMOS consistently outperforms these baselines, particularly in limited data scenarios, which is a critical aspect of the research. The evaluation metrics (SRCC and LCC) are appropriate for assessing the correlation between predicted and human-rated scores, providing a clear picture of the model's effectiveness.
Reproducibility
The paper provides a detailed description of the experimental setup, including the choice of datasets and the methodology for evaluating performance. However, there are no specific URLs or repositories provided for code or data access, which could hinder reproducibility. Including such resources would enhance the transparency and usability of the research.
Limitations
One limitation identified is the potential for domain bias when using few-shot examples that do not represent the broader distribution of test samples. This can lead to overfitting and reduced generalization capabilities. Additionally, while the framework shows promise, the reliance on pseudo-labels may limit the model's performance in scenarios where these labels are not reliable.
Broader impact
The implications of this research are significant for the field of speech processing, particularly in applications requiring non-intrusive quality assessment, such as telecommunications and hearing aids. By reducing the reliance on large annotated datasets, GatherMOS could facilitate the development of more scalable and efficient evaluation systems. The integration of LLMs into this domain also opens avenues for future research in multimodal learning and audio quality assessment. The main contribution of this paper is the introduction of GatherMOS, a novel framework that utilizes large language models for aggregating diverse acoustic signals and pseudo-labels to improve speech quality evaluation. This research represents a significant step forward in the integration of LLMs into audio processing tasks, showcasing the potential for enhanced performance in non-intrusive quality assessment systems.
Author of this paper? Send a correction.