HQ-MPSD: A Multilingual Artifact-Controlled Benchmark for Partial Deepfake Speech Detection
Abstract
Detecting partial deepfake speech is challenging because manipulations occur only in short regions while the surrounding audio remains authentic. However, existing detection methods are fundamentally limited by the quality of available datasets, many of which rely on outdated synthesis systems and generation procedures that introduce dataset-specific artifacts rather than realistic manipulation cues. To address this gap, we introduce HQ-MPSD, a high-quality multilingual partial deepfake speech dataset. HQ-MPSD is constructed using linguistically coherent splice points derived from fine-grained forced alignment, preserving prosodic and semantic continuity and minimizing audible and visual boundary artifacts. The dataset contains 350.8 hours of speech across eight languages and 550 speakers, with background effects added to better reflect real-world acoustic conditions. MOS evaluations and spectrogram analysis confirm the high perceptual naturalness of the samples. We benchmark state-of-the-art detection models through cross-language and cross-dataset evaluations, and all models experience performance drops exceeding 80% on HQ-MPSD. These results demonstrate that HQ-MPSD exposes significant generalization challenges once low-level artifacts are removed and multilingual and acoustic diversity are introduced, providing a more realistic and demanding benchmark for partial deepfake detection. The dataset can be found at: https://zenodo.org/records/17929533.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces HQ-MPSD, a high-quality multilingual dataset for partial deepfake speech detection, addressing critical gaps in existing datasets and providing a rigorous benchmark for evaluating detection models. The comprehensive methodology and experimental evaluations demonstrate significant contributions to the field, paving the way for advancements in robust detection systems.
Methodology
The methodology for constructing the HQ-MPSD dataset is robust and innovative. It employs a three-stage process for generating partial deepfake speech that emphasizes linguistic coherence and acoustic fidelity. The use of fine-grained forced alignment for splice points and the normalization of loudness and spectral characteristics are noteworthy techniques that enhance the quality of the dataset. Additionally, the incorporation of background effects to simulate real-world conditions is a significant improvement over existing datasets. The careful design choices made to minimize artifacts and ensure natural transitions contribute to the dataset's overall quality and applicability for training detection models.
Results & evaluation
The experiments conducted using HQ-MPSD are comprehensive and well-structured. The cross-language and cross-dataset evaluations provide valuable insights into the generalization capabilities of state-of-the-art detection models. The performance drop observed in existing models when tested on HQ-MPSD highlights the dataset's effectiveness in revealing the limitations of current methodologies. The use of metrics such as Equal Error Rate (EER) and Area Under the Curve (AUC) for evaluation is appropriate and provides a clear understanding of model performance.
Reproducibility
The paper provides sufficient detail regarding the dataset generation process and experimental setup, which aids in reproducibility. However, the lack of a publicly available code repository limits the ability for others to fully replicate the experiments. The dataset itself is accessible, which is a positive aspect for researchers looking to build upon this work.
Limitations
While the dataset is a significant advancement, it may still have limitations regarding the diversity of accents and dialects within the eight languages represented. Additionally, the reliance on forced alignment may introduce its own biases, particularly if the alignment tools are not perfectly accurate. The paper does not address potential ethical concerns related to the misuse of deepfake technology, which is an important consideration in this field.
Broader impact
The development of HQ-MPSD has the potential to significantly advance the field of deepfake detection by providing a high-quality, multilingual benchmark that can improve the robustness of detection models. The dataset's design encourages the exploration of genuine manipulation cues rather than superficial artifacts, which can lead to more effective solutions in real-world applications. This work is particularly relevant in the context of misinformation and security, where the ability to detect partial deepfake speech can have substantial societal implications. The paper introduces HQ-MPSD, a high-quality multilingual dataset for partial deepfake speech detection, addressing critical gaps in existing datasets and providing a rigorous benchmark for evaluating detection models. The comprehensive methodology and experimental evaluations demonstrate significant contributions to the field, paving the way for advancements in robust detection systems.
Author of this paper? Send a correction.