ReFESS-QI: Reference-Free Evaluation For Speech Separation With Joint Quality And Intelligibility Scoring
Abstract
Source separation is a crucial pre-processing step for various speech processing tasks, such as automatic speech recognition (ASR). Traditionally, the evaluation metrics for speech separation rely on the matched reference audios and corresponding transcriptions to assess audio quality and intelligibility. However, they cannot be used to evaluate real-world mixtures for which no reference exists. This paper introduces a text-free reference-free evaluation framework based on self-supervised learning (SSL) representations. The proposed framework utilize the mixture and separated tracks to predict jointly audio quality, through the Scale Invariant Signal to Noise Ratio (SI-SNR) metric, and speech intelligibility through the Word Error Rate (WER) metric. We conducted experiments on the WHAMR! dataset, which shows a WER estimation with a mean absolute error (MAE) of 17% and a Pearson correlation coefficient (PCC) of 0.77; and SI-SNR estimation with an MAE of 1.38 and PCC of 0.95. We further demonstrate the robustness of our estimator by using various SSL representations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a reference-free evaluation framework for speech separation that effectively predicts audio quality and intelligibility metrics using self-supervised learning. This work significantly advances the field by addressing the limitations of traditional evaluation methods and providing a robust solution for real-world applications.
Methodology
The proposed methodology introduces a novel reference-free evaluation framework for speech separation that leverages self-supervised learning (SSL) representations. The architecture is designed to predict joint audio quality and intelligibility metrics (SI-SNR and WER) without relying on matched references, which is a significant advancement in the field. The use of multiple speech separation systems to create a diverse training dataset enhances the robustness of the approach. The model architecture, which includes a transformer encoder and multi-output regression, is well-structured and addresses the limitations of existing methods.
Results & evaluation
The experiments conducted on the WHAMR! dataset provide a solid basis for evaluating the proposed framework. The reported results, including MAE and PCC for both WER and SI-SNR estimations, demonstrate that the model outperforms existing baselines. The use of various SSL representations and the evaluation on a real-world dataset (REAL-M) further validate the effectiveness of the approach. However, the paper could benefit from additional comparisons with more baseline methods to strengthen its claims.
Reproducibility
The paper outlines the training setup, including the architecture, loss functions, and data processing pipeline, which is crucial for reproducibility. However, the lack of a publicly available code repository or demo limits the ability for others to replicate the results fully. Providing access to the trained models and datasets would enhance reproducibility.
Limitations
The paper acknowledges several limitations, including the potential lack of generalization due to the use of the same dataset for training and testing. The uniform distribution of scores in the training data may not align well with out-of-domain data, which could affect performance. Additionally, the reliance on large GPUs for inference poses practical challenges for deployment.
Broader impact
The proposed framework has significant implications for real-world applications in speech processing, particularly in automatic speech recognition and other related fields. By enabling reference-free evaluation, it opens up new avenues for assessing speech separation systems in diverse environments, potentially improving the robustness and usability of these technologies in everyday applications. The main contribution of this paper is the introduction of a reference-free evaluation framework for speech separation that effectively predicts audio quality and intelligibility metrics using self-supervised learning. This work significantly advances the field by addressing the limitations of traditional evaluation methods and providing a robust solution for real-world applications.
Author of this paper? Send a correction.