DeepAgent: A Dual Stream Multi Agent Fusion for Robust Multimodal Deepfake Detection
Abstract
The increasing use of synthetic media, particularly deepfakes, is an emerging challenge for digital content verification. Although recent studies use both audio and visual information, most integrate these cues within a single model, which remains vulnerable to modality mismatches, noise, and manipulation. To address this gap, we propose DeepAgent, an advanced multi-agent collaboration framework that simultaneously incorporates both visual and audio modalities for the effective detection of deepfakes. DeepAgent consists of two complementary agents. Agent-1 examines each video with a streamlined AlexNet-based CNN to identify the symbols of deepfake manipulation, while Agent-2 detects audio-visual inconsistencies by combining acoustic features, audio transcriptions from Whisper, and frame-reading sequences of images through EasyOCR. Their decisions are fused through a Random Forest meta-classifier that improves final performance by taking advantage of the different decision boundaries learned by each agent. This study evaluates the proposed framework using three benchmark datasets to demonstrate both component-level and fused performance. Agent-1 achieves a test accuracy of 94.35% on the combined Celeb-DF and FakeAVCeleb datasets. On the FakeAVCeleb dataset, Agent-2 and the final meta-classifier attain accuracies of 93.69% and 81.56%, respectively. In addition, cross-dataset validation on DeepFakeTIMIT confirms the robustness of the meta-classifier, which achieves a final accuracy of 97.49%, and indicates a strong capability across diverse datasets. These findings confirm that hierarchy-based fusion enhances robustness by mitigating the weaknesses of individual modalities and demonstrate the effectiveness of a multi-agent approach in addressing diverse types of manipulations in deepfakes.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DeepAgent, a dual-agent framework for deepfake detection that effectively integrates visual and audio modalities to enhance robustness and accuracy. This work significantly advances the field by addressing existing limitations in deepfake detection methodologies and demonstrating strong performance across multiple datasets.
Methodology
The proposed methodology in DeepAgent is innovative, utilizing a dual-agent framework that effectively combines visual and audio modalities for deepfake detection. Agent-1 employs a lightweight CNN based on AlexNet for visual analysis, while Agent-2 integrates audio features, speech transcripts, and OCR outputs to assess audio-visual consistency. The use of a Random Forest meta-classifier for decision-level fusion is a notable strength, allowing for the integration of complementary information from both agents. The approach addresses critical gaps in existing deepfake detection methods by explicitly modeling semantic and lexical consistency between audio and visual content, which is a significant advancement in the field.
Results & evaluation
The experimental evaluation is thorough, utilizing three benchmark datasets (Celeb-DF, FakeAVCeleb, and DeepFakeTIMIT) to assess the performance of both agents and the meta-classifier. The reported accuracies are impressive, particularly the 97.49% accuracy achieved on the DeepFakeTIMIT dataset by the meta-classifier. The cross-dataset validation demonstrates robustness, which is crucial for real-world applications. However, the paper could benefit from additional comparative analysis with more recent state-of-the-art methods to further validate its effectiveness.
Reproducibility
The implementation details are well-documented, including the architecture of both agents, the training process, and the datasets used. However, the absence of a publicly available code repository limits reproducibility. Providing access to the code and trained models would enhance the ability of other researchers to replicate the results and build upon this work.
Limitations
While the dual-agent approach is innovative, it may introduce complexity in terms of computational efficiency and resource requirements. The reliance on a Random Forest meta-classifier, while effective, may not leverage the full potential of deep learning techniques for final classification. Additionally, the paper does not address potential biases in the datasets used, which could affect the generalizability of the results.
Broader impact
The implications of this research are significant, particularly in the context of digital content verification and security. As deepfakes become more prevalent, effective detection methods like DeepAgent can contribute to safeguarding against misinformation and manipulation in various domains, including social media, journalism, and digital forensics. The multi-agent approach could also inspire future research into collaborative frameworks for other complex detection tasks. The main contribution of this paper is the introduction of DeepAgent, a dual-agent framework for deepfake detection that effectively integrates visual and audio modalities to enhance robustness and accuracy. This work significantly advances the field by addressing existing limitations in deepfake detection methodologies and demonstrating strong performance across multiple datasets.
Author of this paper? Send a correction.