DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors
Abstract
Human subjective evaluation is the gold standard to evaluate speech quality optimized for human perception. Perceptual objective metrics serve as a proxy for subjective scores. The conventional and widely used metrics require a reference clean speech signal, which is unavailable in real recordings. The no-reference approaches correlate poorly with human ratings and are not widely adopted in the research community. One of the biggest use cases of these perceptual objective metrics is to evaluate noise suppression algorithms. This paper introduces a multi-stage self-teaching based perceptual objective metric that is designed to evaluate noise suppressors. The proposed method generalizes well in challenging test conditions with a high correlation to human ratings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DNSMOS, a robust and innovative metric for evaluating noise suppression methods in speech quality, which significantly improves upon existing objective metrics by addressing the challenges of label noise and generalization across diverse audio conditions. The comprehensive methodology and strong experimental validation position DNSMOS as a valuable tool for researchers and practitioners in the field.
Methodology
The paper introduces DNSMOS, a novel multi-stage self-teaching model for evaluating noise suppression methods in speech quality. The methodology is well-structured, leveraging a CNN architecture trained on human-rated data, and incorporates a self-teaching mechanism to mitigate label noise. This approach is innovative in its application to speech quality metrics, particularly in the context of noisy labels and generalization across diverse audio impairments. The choice of features (log power Mel spectrogram) is appropriate given the task, and the model architecture is optimized for performance without excessive complexity.
Results & evaluation
The experiments are robust, utilizing a large dataset of 600 noisy speech clips and over 120,000 associated MOS scores. The evaluation metrics (PCC and SRCC) are well-chosen to assess the correlation between DNSMOS and human ratings. The results demonstrate that DNSMOS outperforms traditional metrics like PESQ and POLQA, indicating its effectiveness in accurately ranking noise suppression methods. The generalizability tests across different datasets further validate the model's robustness.
Reproducibility
The paper provides sufficient detail on the model architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of a publicly available code repository limits the ease of replication by other researchers. The paper does mention the availability of DNSMOS as an Azure service, which is a positive step towards accessibility.
Limitations
One limitation noted is the inherent noise in human ratings, which can affect the training process. Additionally, the model's performance on non-English audio and emotional content is less robust, indicating a need for further training on diverse datasets. The reliance on human ratings, while valuable, introduces variability that could impact the consistency of results.
Broader impact
The development of DNSMOS has significant implications for the field of speech enhancement and audio quality assessment. By providing a reliable, non-intrusive metric for evaluating noise suppression methods, it can facilitate advancements in audio processing technologies, benefiting applications in telecommunications, voice recognition, and hearing aids. The potential for integration into Azure services also suggests a pathway for widespread adoption in industry. The main contribution of this paper is the introduction of DNSMOS, a robust and innovative metric for evaluating noise suppression methods in speech quality, which significantly improves upon existing objective metrics by addressing the challenges of label noise and generalization across diverse audio conditions. The comprehensive methodology and strong experimental validation position DNSMOS as a valuable tool for researchers and practitioners in the field.
Curator note
Reddy et al., Microsoft; non-intrusive automatic MOS for noise-suppressed speech; standard in speech enhancement
Author of this paper? Send a correction.