Bridging the SEA Gap: An Initial Benchmark for Neural Audio Codec-Synthesized Speech Deepfakes in South-East Asian Languages
Abstract
Codecfakes (CFs) are a type of speech deepfakes generated through Audio Language Models (ALMs), with Neural Audio Codecs (NACs) forming the core mechanism for speech encoding and generation. CFs exhibit distributional characteristics that differ from vocoder-based deepfakes, causing detectors trained on vocoder data to generalize poorly to CFs detection. Although this has led to the development of CF detection benchmarks, existing resources are largely confined to English -- and to a limited extent Chinese -- leaving South-East Asian (SEA) languages unexplored. To bridge this gap, we introduce SEA-CF, the first large-scale benchmark for CF detection spanning multiple SEA languages, diverse speaker profiles, and a wide range of NAC architectures. SEA-CF is constructed by synthesizing publicly available real speech corpora. Our experiments show that state-of-the-art (SOTA) CF detectors trained on English-centric datasets fail to generalize to SEA speech due to language-specific phonetic structures, tonal variations, and rich prosodic diversity. We further conduct a comprehensive zero-shot and fine-tuned evaluation of recent SOTA ALMs on SEA-CF. Fine-tuning the ALMs improves performance, however, these are very large being impractical for real-world application due to their scale, particularly in low-resource and latency-constrained settings. To address this limitation, we propose a novel small-ALM, GARUDA tailored for CF detection, which delivers strong performance while remaining lightweight. Extensive evaluations demonstrate that the proposed Small-ALM outperforms strong end-to-end and ALM-based baselines, establishing a new, practical direction for robust CF detection in SEA languages and beyond.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces SEA-CF, the first large-scale benchmark for CF detection in SEA languages, and proposes GARUDA, a lightweight Small-ALM that outperforms existing models while addressing practical deployment challenges. The technical contributions are significant, with a strong focus on methodology and experimental validation, positioning this work as a valuable asset in the field of audio deepfake detection.
Methodology
The methodology presented in the paper is robust, introducing the SEA-CF benchmark for CF detection in SEA languages, which is a significant advancement given the lack of resources in this area. The authors propose GARUDA, a lightweight Small-ALM that effectively combines dual-encoder architectures to capture semantic and prosodic features, which is innovative. The use of JS divergence as a loss function for aligning representations is a novel approach that enhances the model's performance. Overall, the methodology is well-structured and addresses practical deployment challenges.
Results & evaluation
The experimental evaluation is comprehensive, utilizing both zero-shot and fine-tuned settings to assess the performance of GARUDA against existing SOTA models. The results demonstrate significant improvements over baselines, with rigorous statistical testing (McNemar’s test) validating the findings. The paper effectively highlights the necessity of in-domain training and the limitations of existing models when applied to SEA languages, underscoring the importance of the proposed SEA-CF benchmark.
Reproducibility
The paper provides sufficient details regarding the dataset construction, model architecture, and training procedures, which enhances reproducibility. The authors mention the use of publicly available datasets and provide a project URL for accessing the SEA-CF benchmark, which is crucial for other researchers looking to replicate or build upon this work.
Limitations
While the paper makes significant contributions, it acknowledges limitations such as the incomplete coverage of all SEA languages and the current restriction of evaluations to available benchmarks. Future work is needed to expand the dataset and improve generalization across diverse generators.
Broader impact
The work has substantial implications for enhancing security against audio deepfakes in low-resource language contexts, addressing a critical gap in the current landscape of speech technology. By focusing on SEA languages, the research promotes inclusivity and provides tools that can be vital for protecting vulnerable communities against audio fraud. This paper introduces SEA-CF, the first large-scale benchmark for CF detection in SEA languages, and proposes GARUDA, a lightweight Small-ALM that outperforms existing models while addressing practical deployment challenges. The technical contributions are significant, with a strong focus on methodology and experimental validation, positioning this work as a valuable asset in the field of audio deepfake detection.
Author of this paper? Send a correction.