From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection
Abstract
With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel approach to audio-visual deepfake detection by addressing the unique challenges of singing scenarios and introducing a new benchmark dataset. The technical contributions, particularly in methodology and experimental validation, position this work as a meaningful advancement in the field of machine learning and deepfake detection.
Methodology
The methodology introduces the T-AVFD framework, which effectively combines facial authenticity patterns with audio-visual consistency to address the unique challenges posed by singing deepfakes. The use of a contrastive learning approach with multi-granularity text prompts enhances the robustness of the model against domain shifts. The integration of a differential weight learning mechanism is particularly innovative, allowing the model to adaptively weigh contributions from different modalities based on their reliability.
Results & evaluation
The experiments are comprehensive, covering multiple datasets and scenarios, including the newly introduced SHDF dataset. The results demonstrate significant improvements over existing methods, validating the effectiveness of the proposed approach. The robustness tests under various perturbations further strengthen the findings, showcasing the model's practical applicability.
Reproducibility
The paper provides detailed implementation details, including training configurations and dataset descriptions, which support reproducibility. However, the reliance on specific generative models and the absence of publicly available code for some components may hinder full reproducibility.
Limitations
While the proposed framework shows strong performance, its reliance on the quality of the training data and the specific generative models used for dataset creation may limit its generalizability. Additionally, the model's performance on unseen types of deepfakes beyond singing and talking remains to be explored.
Broader impact
This research has significant implications for the detection of deepfakes in various applications, including media verification, security, and combating misinformation. The introduction of a singing-focused dataset also opens avenues for future research in audio-visual synthesis and detection. This paper presents a novel approach to audio-visual deepfake detection by addressing the unique challenges of singing scenarios and introducing a new benchmark dataset. The technical contributions, particularly in methodology and experimental validation, position this work as a meaningful advancement in the field of machine learning and deepfake detection.
Author of this paper? Send a correction.