Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms
Abstract
We propose the Fréchet Audio Distance (FAD), a novel, reference-free evaluation metric for music enhancement algorithms. We demonstrate how typical evaluation metrics for speech enhancement and blind source separation can fail to accurately measure the perceived effect of a wide variety of distortions. As an alternative, we propose adapting the Fréchet Inception Distance (FID) metric used to evaluate generative image models to the audio domain. FAD is validated using a wide variety of artificial distortions and is compared to the signal based metrics signal to distortion ratio (SDR), cosine distance and magnitude L2 distance. We show that, with a correlation coefficient of 0.52, FAD correlates more closely with human perception than either SDR, cosine distance or magnitude L2 distance, with correlation coefficients of 0.39, -0.15 and -0.01 respectively.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents the Fréchet Audio Distance (FAD), a significant advancement in evaluating music enhancement algorithms. By addressing the shortcomings of existing metrics and providing a robust, perceptually relevant alternative, this work has the potential to enhance the quality of audio processing across various applications.
Methodology
The paper introduces the Fréchet Audio Distance (FAD) as a novel reference-free metric for evaluating music enhancement algorithms, adapting the Fréchet Inception Distance (FID) from image processing to audio. The methodology is sound, utilizing embeddings from a pretrained VGGish model to compute multivariate Gaussians for both enhanced and clean audio, allowing for a robust statistical comparison. The paper effectively highlights the limitations of existing metrics like SDR and cosine distance, demonstrating how FAD correlates better with human perception of audio quality.
Results & evaluation
The authors conduct a thorough experimental evaluation using a diverse set of artificial distortions, validating the FAD metric against traditional metrics and human evaluations. The dataset used, Magnatagatune, is substantial, and the experiments are well-structured, providing clear comparisons across various distortion types. The correlation coefficients reported strengthen the argument for FAD's effectiveness, although further details on the human evaluation process could enhance transparency.
Reproducibility
The paper provides sufficient detail on the methodology and experimental setup, including the use of the VGGish model and the specific parameters for distortions. However, the absence of a direct link to the code or a demo limits reproducibility. Including a GitHub repository with the implementation would greatly benefit the community.
Limitations
While FAD shows promise, the paper acknowledges that it may not capture all possible distortions and is limited to the embeddings generated from the VGGish model. The reliance on a fixed embedding window size may overlook long-term temporal changes in music. Additionally, the human evaluation is limited in scope, using only a subset of distortions and configurations.
Broader impact
The introduction of FAD could significantly influence the evaluation of music enhancement algorithms, providing a more perceptually relevant metric. This could lead to improved audio quality in applications ranging from mobile recordings to music streaming services. The potential for FAD to be adapted for other audio domains also suggests wide applicability in audio processing and enhancement research. The paper presents the Fréchet Audio Distance (FAD), a significant advancement in evaluating music enhancement algorithms. By addressing the shortcomings of existing metrics and providing a robust, perceptually relevant alternative, this work has the potential to enhance the quality of audio processing across various applications.
Curator note
Kilgour et al., Google; audio equivalent of FID; standard for evaluating audio/music generation quality
Related later work
- Standard metric in AudioLDM, MusicGen, AudioCraft evaluations 2020
- CLAP score as replacement in newer works 2022
Author of this paper? Send a correction.