MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection
Abstract
Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces MADBench, a novel benchmark for component-aware audio deepfake detection that disentangles speech and environmental audio manipulation, providing rigorous evaluation protocols and insightful findings on detector behavior and cross-component interference.
Methodology
The paper proposes MADBench, a novel benchmark for audio deepfake detection that disentangles speech and environmental audio components. The methodology involves a rigorous pipeline: source video separation using MossFormer2, generation of fake speech via TTS/VC models (F5-TTS, SeedVC, etc.), and generation of fake environmental audio via Text-to-Audio and Video-to-Audio models (AudioLDM2, MMAudio, etc.). A key methodological contribution is the introduction of a "scene-consistency" axis (matched vs. mismatched environmental audio) and the evaluation of cross-component interference. The approach is technically sound, addressing a genuine gap in existing benchmarks which often conflate audio modalities or focus solely on speech. However, the novelty is somewhat tempered by the fact that the core components (separation, TTS, TTA) are established techniques; the innovation lies primarily in the specific combination and the evaluation protocol rather than a new algorithmic breakthrough.
Results & evaluation
The experimental evaluation is comprehensive. The authors benchmark a wide range of models: pretrained AV detectors (AVH-Align, etc.), frozen AV encoders (ImageBind, PE-AV, CAV-MAE), and zero-shot Omni models (Qwen2.5-Omni, MiniCPM-o). They evaluate across multiple protocols (Binary, 4-way, Component-level, Scene-consistency). The results are insightful: they find that environmental audio manipulation is easier to detect than speech, and that fake environmental audio can obscure speech detection (cross-component interference). They also find that frozen AV encoders perform surprisingly well, often better than task-specific detectors, and that video input does not consistently improve manipulation detection but helps with scene consistency. The analysis is thorough, including ablation studies on input modality and scene consistency. The statistical reporting (AUC, EER) is appropriate.
Reproducibility
The paper provides detailed descriptions of the dataset construction, including the source dataset (AVSpeech), the separation model (MossFormer2), the specific generation models used, and the quality control steps. The split strategy (7:1:2) and the balancing of generation settings are described. However, the paper does not provide a public link to the dataset or code in the text provided (URLs are "none"). The use of proprietary or hard-to-access models (e.g., specific checkpoints of F5-TTS, SeedVC, etc.) might pose reproducibility challenges for the exact generation pipeline, although the models themselves are generally available. The detailed protocol allows for reasonable reproducibility of the evaluation framework.
Limitations
The paper acknowledges that the visual stream is held fixed, which limits the study to audio manipulation detection in authentic video. It does not address scenarios where both audio and video are manipulated. The dataset size (1,892 clips) is relatively small for deep learning benchmarks, which might limit the generalizability of the findings, especially for complex multimodal models. The reliance on automatic scene classification (Qwen2.5-VL) for taxonomy construction introduces potential noise, although manual verification was performed. The performance of zero-shot models is limited, which is expected but highlights the current gap in multimodal reasoning for forensic tasks.
Broader impact
This paper has significant broader impact. By providing a benchmark that reflects a realistic attack scenario (independent manipulation of speech and background), it enables more robust development of deepfake detection systems. The findings that environmental audio manipulation is easier to detect and can interfere with speech detection have practical implications for security and forensics. The benchmark will likely spur research into more robust, component-aware detection models. It also highlights the limitations of current zero-shot multimodal models in forensic reasoning, guiding future research directions. The paper introduces MADBench, a novel benchmark for component-aware audio deepfake detection that disentangles speech and environmental audio manipulation, providing rigorous evaluation protocols and insightful findings on detector behavior and cross-component interference.
Author of this paper? Send a correction.