Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias
Abstract
Audio deepfake detection systems are increasingly deployed in high-stakes security applications, yet their fairness across demographic groups remains critically underexamined. Prior work measures gender disparity but does not investigate where it comes from or how to fix it systematically. We present the first diagnosis-first framework that identifies bias source before applying targeted mitigation, evaluated on two models, AASIST and Wav2Vec2+ResNet18, on ASVSpoof5. Our diagnosis shows that bias does not stem from imbalanced training data but from acoustic representation differences, gender leakage in learned features, and structural evaluation asymmetry. We test mitigation strategies across in-processing, post-processing and combined families, including novel methods introduced in this work. Adjusting the decision threshold separately per gender reduces unfairness by 54% to 75% at no cost to detection accuracy, and our new epoch-level fairness regularisation method outperforms existing per-batch approaches. Adversarial debiasing succeeds only when gender leakage is localised, and fails when it is diffuse, an outcome correctly predicted by our diagnosis before training. No single method fully closes the fairness gap, confirming that bias sources must be identified before fixes are applied and that fairer benchmark design is equally important
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a pioneering diagnosis-first framework for addressing gender bias in audio deepfake detection systems, significantly advancing the understanding and mitigation of bias in machine learning applications. The comprehensive methodology and rigorous experimental evaluation contribute valuable insights to the field, highlighting the importance of systematic bias diagnosis before applying mitigation strategies.
Methodology
The paper introduces a systematic diagnosis-first framework for identifying and mitigating gender bias in audio deepfake detection. This approach is innovative as it emphasizes understanding the sources of bias before applying mitigation strategies, which is a significant departure from existing methods that often apply fixes without thorough diagnosis. The methodology is well-structured, detailing a comprehensive evaluation of bias sources at data, model, and decision levels, and it introduces novel mitigation techniques such as EAFR, SGFS, and GNEA. Each method is clearly defined, and the rationale for their implementation is well-articulated.
Results & evaluation
The experimental setup is robust, utilizing the ASVSpoof5 dataset, which is appropriate for the study's focus on gender fairness in audio deepfake detection. The paper conducts extensive experiments across multiple models (AASIST and Wav2Vec2+ResNet18) and evaluates various mitigation strategies, providing a thorough analysis of their effectiveness. The results are presented clearly, with a focus on multiple fairness metrics, which enhances the credibility of the findings. However, the reliance on a single dataset may limit the generalizability of the results.
Reproducibility
The paper provides sufficient detail regarding the experimental setup, model architectures, and evaluation protocols, which supports reproducibility. However, the absence of publicly available code or a project repository limits the ability for others to reproduce the findings directly. Including a demo or project URL would enhance the reproducibility aspect significantly.
Limitations
The study is limited to a single dataset (ASVSpoof5) and focuses on binary gender labels, which may not capture the full spectrum of gender representation. Additionally, while the paper identifies multiple sources of bias, it acknowledges that no single method completely closes the fairness gap, indicating that further research is needed to address these issues comprehensively.
Broader impact
The implications of this work are significant, particularly in high-stakes applications such as security and identity verification, where fairness and bias in detection systems can have profound societal impacts. By addressing gender bias in audio deepfake detection, the paper contributes to the broader discourse on fairness in AI systems, emphasizing the need for equitable treatment across demographic groups. This paper presents a pioneering diagnosis-first framework for addressing gender bias in audio deepfake detection systems, significantly advancing the understanding and mitigation of bias in machine learning applications. The comprehensive methodology and rigorous experimental evaluation contribute valuable insights to the field, highlighting the importance of systematic bias diagnosis before applying mitigation strategies.
Author of this paper? Send a correction.