SpeechQualityLLM: LLM-Based Multimodal Assessment of Speech Quality
Abstract
Objective speech quality assessment is central to telephony, VoIP, and streaming systems, where large volumes of degraded audio must be monitored and optimized at scale. Classical metrics such as PESQ and POLQA approximate human mean opinion scores (MOS) but require carefully controlled conditions and expensive listening tests, while learning-based models such as NISQA regress MOS and multiple perceptual dimensions from waveforms or spectrograms, achieving high correlation with subjective ratings yet remaining rigid: they do not support interactive, natural-language queries and do not natively provide textual rationales. In this work, we introduce SpeechQualityLLM, a multimodal speech quality question-answering (QA) system that couples an audio encoder with a language model and is trained on the NISQA corpus using template-based question-answer pairs covering overall MOS and four perceptual dimensions (noisiness, coloration, discontinuity, and loudness) in both single-ended (degraded only) and double-ended (degraded plus clean reference) setups. Instead of directly regressing scores, our system is supervised to generate textual answers from which numeric predictions are parsed and evaluated with standard regression and ranking metrics; on held-out NISQA clips, the double-ended model attains a MOS mean absolute error (MAE) of 0.41 with Pearson correlation of 0.86, with competitive performance on dimension-wise tasks. Beyond these quantitative gains, it offers a flexible natural-language interface in which the language model acts as an audio quality expert: practitioners can query arbitrary aspects of degradations, prompt the model to emulate different listener profiles to capture human variability and produce diverse but plausible judgments rather than a single deterministic score, and thereby reduce reliance on large-scale crowdsourced tests and their monetary cost.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SpeechQualityLLM, a multimodal speech quality assessment system that leverages a language model to provide interactive, interpretable evaluations of audio quality. This work represents a significant advancement in the field by integrating natural language processing with audio quality assessment, offering a novel approach that enhances both flexibility and user engagement in evaluating speech quality.
Methodology
The methodology presented in the paper is innovative as it combines audio encoding with a language model to create a multimodal QA system for speech quality assessment. The use of template-based question-answer pairs to train the model on the NISQA corpus is a thoughtful approach that allows for flexibility in querying and understanding audio quality metrics. The model's ability to generate textual answers rather than fixed scores enhances interpretability and user interaction, which is a significant advancement over traditional methods.
Results & evaluation
The experiments conducted on held-out NISQA clips demonstrate solid performance, with a mean absolute error of 0.41 and a Pearson correlation of 0.86, indicating a strong correlation with human ratings. The competitive performance on dimension-wise tasks further validates the effectiveness of the proposed system. However, the paper could benefit from a more extensive evaluation across different datasets and real-world scenarios to fully establish its robustness.
Reproducibility
The authors provide a GitHub repository with code, model weights, and experimental results, which is a positive aspect for reproducibility. However, the paper could enhance reproducibility by including more detailed descriptions of hyperparameters, training procedures, and evaluation metrics used in the experiments.
Limitations
One limitation is the reliance on the NISQA corpus, which may not encompass all possible audio quality scenarios encountered in real-world applications. Additionally, while the model offers flexibility in querying, it may still struggle with edge cases or highly subjective audio quality assessments that require nuanced human judgment.
Broader impact
The potential applications of SpeechQualityLLM are significant, particularly in telephony, VoIP, and streaming services, where audio quality is paramount. By reducing reliance on costly human evaluations and enabling interactive queries, this system could streamline quality assessment processes and improve user experiences across various platforms. The main contribution of this paper is the introduction of SpeechQualityLLM, a multimodal speech quality assessment system that leverages a language model to provide interactive, interpretable evaluations of audio quality. This work represents a significant advancement in the field by integrating natural language processing with audio quality assessment, offering a novel approach that enhances both flexibility and user engagement in evaluating speech quality.
Author of this paper? Send a correction.