Leveraging large multimodal models for audio-video deepfake detection: a pilot study
Abstract
Audio-visual deepfake detection (AVD) is increasingly important as modern generators can fabricate convincing speech and video. Most current multimodal detectors are small, task-specific models: they work well on curated tests but scale poorly and generalize weakly across domains. We introduce AV-LMMDetect, a supervised fine-tuned (SFT) large multimodal model that casts AVD as a prompted yes/no classification - "Is this video real or fake?". Built on Qwen 2.5 Omni, it jointly analyzes audio and visual streams for deepfake detection and is trained in two stages: lightweight LoRA alignment followed by audio-visual encoder full fine-tuning. On FakeAVCeleb and Mavos-DD, AV-LMMDetect matches or surpasses prior methods and sets a new state of the art on Mavos-DD datasets.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AV-LMMDetect, a large multimodal model for audio-visual deepfake detection that utilizes a novel two-stage training approach, achieving state-of-the-art results and demonstrating the potential of large models in enhancing detection capabilities. The methodology and results present a meaningful advancement in the field, addressing critical challenges in deepfake detection and setting a foundation for future research.
Methodology
The paper introduces AV-LMMDetect, a novel supervised fine-tuned large multimodal model for audio-visual deepfake detection, employing a two-stage training strategy that combines lightweight LoRA alignment with full fine-tuning of audio-visual encoders. This methodology is innovative as it reformulates the detection task into a binary question-answering format, which is a fresh approach in the context of deepfake detection. The use of a large multimodal model also enhances the model's ability to capture cross-modal inconsistencies, a significant improvement over traditional methods that often rely on smaller, task-specific models.
Results & evaluation
The experimental results demonstrate that AV-LMMDetect achieves state-of-the-art performance on the MAVOS-DD dataset and competitive results on FakeAVCeleb. The paper provides thorough evaluations across multiple scenarios, including in-domain and open-set conditions, showcasing the model's robustness and generalization capabilities. The use of standard binary classification metrics (accuracy, AUC, and mAP) adds rigor to the evaluation process, although the paper could benefit from more detailed comparisons with additional baseline methods.
Reproducibility
The paper lacks explicit details regarding the implementation and hyperparameter settings, which are crucial for reproducibility. While the methodology is described, the absence of a dedicated section on experimental setup and code availability limits the ability of other researchers to replicate the results. Including a link to a code repository or supplementary materials would significantly enhance reproducibility.
Limitations
One limitation of the study is the reliance on two specific datasets, which may not fully represent the diversity of deepfake techniques in real-world applications. Additionally, while the model shows strong performance, the paper does not address potential biases in the datasets or the model's performance across different demographic groups, which is critical for ethical considerations in deployment.
Broader impact
The implications of this research are significant, as robust audio-visual deepfake detection is crucial for maintaining media integrity and public trust in an era of increasing misinformation. The proposed model could be applied in various domains, including social media, journalism, and law enforcement, to identify and mitigate the risks posed by deepfake technology. The main contribution of this paper is the introduction of AV-LMMDetect, a large multimodal model for audio-visual deepfake detection that utilizes a novel two-stage training approach, achieving state-of-the-art results and demonstrating the potential of large models in enhancing detection capabilities. The methodology and results present a meaningful advancement in the field, addressing critical challenges in deepfake detection and setting a foundation for future research.
Author of this paper? Send a correction.