Robust Generative Audio Quality Assessment: Disentangling Quality from Spurious Correlations
Abstract
The rapid proliferation of AI-Generated Content (AIGC) has necessitated robust metrics for perceptual quality assessment. However, automatic Mean Opinion Score (MOS) prediction models are often compromised by data scarcity, predisposing them to learn spurious correlations-- such as dataset-specific acoustic signatures-- rather than generalized quality features. To address this, we leverage domain adversarial training (DAT) to disentangle true quality perception from these nuisance factors. Unlike prior works that rely on static domain priors, we systematically investigate domain definition strategies ranging from explicit metadata-driven labels to implicit data-driven clusters. Our findings reveal that there is no "one-size-fits-all" domain definition; instead, the optimal strategy is highly dependent on the specific MOS aspect being evaluated. Experimental results demonstrate that our aspect-specific domain strategy effectively mitigates acoustic biases, significantly improving correlation with human ratings and achieving superior generalization on unseen generative scenarios.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel approach to audio quality assessment by leveraging domain adversarial training to disentangle quality perception from spurious correlations, significantly enhancing the reliability of automatic MOS prediction models. The comprehensive methodology and rigorous experimental validation contribute to its significance in the field, addressing a pressing challenge in evaluating AI-generated audio content.
Methodology
The paper introduces a robust framework for Mean Opinion Score (MOS) prediction using Domain Adversarial Training (DAT) to mitigate spurious correlations in audio quality assessment. The methodology is well-structured, employing three distinct domain definition strategies: explicit metadata-driven labels, implicit K-means clustering, and random assignment, which are systematically analyzed for their effectiveness. The use of a pre-trained SSL feature extractor and a MultiGauss backbone for quality prediction adds depth to the approach, ensuring that the model captures intrinsic quality features while remaining invariant to domain-specific biases.
Results & evaluation
The experiments are comprehensive, utilizing the AES-Natural dataset with a well-defined split protocol for training, validation, and evaluation. The results demonstrate significant improvements in correlation with human ratings across various aspects of audio quality, showcasing the effectiveness of the proposed domain strategies. The statistical significance of the results is validated through rigorous testing, including t-tests, which strengthens the findings.
Reproducibility
The paper provides sufficient details regarding the model architecture, training setup, and evaluation metrics, which facilitates reproducibility. The inclusion of a GitHub repository for code access further enhances the potential for others to replicate the study.
Limitations
While the study presents a robust framework, it may be limited by the dataset used, which could affect the generalizability of the findings. Additionally, the reliance on specific domain definitions may not universally apply to all audio quality assessment scenarios, suggesting a need for further exploration of domain strategies across diverse datasets.
Broader impact
The implications of this research are significant, as it addresses a critical challenge in the evaluation of AI-generated audio content, which is increasingly relevant in the context of content creation and multimedia applications. The findings could influence the development of more reliable audio quality assessment tools, potentially impacting industries such as entertainment, broadcasting, and AI content generation. The paper presents a novel approach to audio quality assessment by leveraging domain adversarial training to disentangle quality perception from spurious correlations, significantly enhancing the reliability of automatic MOS prediction models. The comprehensive methodology and rigorous experimental validation contribute to its significance in the field, addressing a pressing challenge in evaluating AI-generated audio content.
Author of this paper? Send a correction.