SAVe: Self-Supervised Audio-visual Deepfake Detection Exploiting Visual Artifacts and Audio-visual Misalignment
Abstract
Multimodal deepfakes can exhibit subtle visual artifacts and cross-modal inconsistencies, which remain challenging to detect, especially when detectors are trained primarily on curated synthetic forgeries. Such synthetic dependence can introduce dataset and generator bias, limiting scalability and robustness to unseen manipulations. We propose SAVe, a self-supervised audio-visual deepfake detection framework that learns entirely on authentic videos. SAVe generates on-the-fly, identity-preserving, region-aware self-blended pseudo-manipulations to emulate tampering artifacts, enabling the model to learn complementary visual cues across multiple facial granularities. To capture cross-modal evidence, SAVe also models lip-speech synchronization via an audio-visual alignment component that detects temporal misalignment patterns characteristic of audio-visual forgeries. Experiments on FakeAVCeleb and AV-LipSync-TIMIT demonstrate competitive in-domain performance and strong cross-dataset generalization, highlighting self-supervised learning as a scalable paradigm for multimodal deepfake detection.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SAVe, a self-supervised audio-visual deepfake detection framework that effectively learns to identify manipulations without relying on synthetic training data. This innovative approach enhances the robustness and scalability of deepfake detection systems, addressing critical challenges in the field.
Methodology
The proposed SAVe framework introduces a novel self-supervised learning approach for deepfake detection that leverages authentic videos to generate pseudo-manipulations. This methodology is significant as it addresses the limitations of existing supervised models that rely on synthetic datasets, which can introduce biases and limit generalization. The framework's multi-branch architecture effectively captures both visual artifacts and audio-visual synchronization inconsistencies, enhancing detection robustness across various manipulation techniques. The use of region-aware self-blending for generating pseudo-manipulations is particularly innovative, allowing the model to learn nuanced forgery cues across different facial regions.
Results & evaluation
The experiments are well-structured, utilizing two established benchmarks (FakeAVCeleb and AV-LipSync-TIMIT) to evaluate the model's performance. The results demonstrate strong in-domain accuracy and impressive cross-dataset generalization, indicating that the self-supervised approach can adapt to unseen manipulation techniques effectively. The paper provides comprehensive metrics, including AUC and Average Precision, which are critical for assessing the performance of deepfake detection systems. The results suggest that SAVe outperforms existing methods, reinforcing the effectiveness of the proposed framework.
Reproducibility
The paper includes detailed implementation details, including the training process, data preparation, and evaluation metrics, which would facilitate reproducibility. However, the absence of a publicly available code repository or demo URL limits the ability for independent verification of results. The methodology is described sufficiently to allow for replication, but access to the actual implementation would enhance reproducibility.
Limitations
One notable limitation is the reliance on authentic videos for training, which may not encompass the full range of potential manipulations present in real-world scenarios. Additionally, while the model shows strong performance on the evaluated datasets, its effectiveness on more diverse or complex datasets remains to be tested. The paper also acknowledges that purely visual cues may fail in scenarios where audio manipulation is present without corresponding visual artifacts.
Broader impact
The implications of this research are significant, given the increasing prevalence of deepfake technology and its potential misuse in misinformation and identity fraud. By providing a scalable and robust detection framework, SAVe contributes to the development of tools that can help mitigate the risks associated with deepfakes. The self-supervised approach also opens avenues for further research in other domains where labeled data is scarce, promoting broader applications of machine learning in real-world scenarios. The main contribution of this paper is the introduction of SAVe, a self-supervised audio-visual deepfake detection framework that effectively learns to identify manipulations without relying on synthetic training data. This innovative approach enhances the robustness and scalability of deepfake detection systems, addressing critical challenges in the field.
Author of this paper? Send a correction.