Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection
Abstract
Speech deepfake detection (SDD) focuses on identifying whether a given speech signal is genuine or has been synthetically generated. Existing audio large language model (LLM)-based methods excel in content understanding; however, their predictions are often biased toward semantically correlated cues, which results in fine-grained acoustic artifacts being overlooked during the decisionmaking process. Consequently, fake speech with natural semantics can bypass detectors despite harboring subtle acoustic anomalies; this suggests that the challenge stems not from the absence of acoustic data, but from its inadequate accessibility when semantic-dominant reasoning prevails. To address this issue, we investigate SDD within the audio LLM paradigm and introduce SDD with Auditory Perception-enhanced Audio Large Language Model (SDD-APALLM), an acoustically enhanced framework designed to explicitly expose fine-grained time-frequency evidence as accessible acoustic cues. By combining raw audio with structured spectrograms, the proposed framework empowers audio LLMs to more effectively capture subtle acoustic inconsistencies without compromising their semantic understanding. Experimental results indicate consistent gains in detection accuracy and robustness, especially in cases where semantic cues are misleading. Further analysis reveals that these improvements stem from a coordinated utilization of semantic and acoustic information, as opposed to simple modality aggregation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SDD-APALLM, a novel framework that enhances speech deepfake detection by explicitly exposing fine-grained acoustic evidence, thereby improving model robustness and interpretability. This work addresses a significant gap in the current methodologies for audio LLMs, providing a promising direction for future research in the field of audio processing and deepfake detection.
Methodology
The proposed methodology, SDD-APALLM, innovatively enhances the accessibility of fine-grained acoustic evidence by integrating structured time-frequency representations alongside raw audio inputs. This approach effectively shifts the focus from semantic plausibility to acoustically grounded evidence, addressing a critical limitation in existing audio LLM-based speech deepfake detection methods. The use of Constant-Q Transform (CQT) to create visual tokens that highlight spectral structures linked to speech synthesis artifacts is particularly noteworthy, as it provides a clear mechanism for improving model interpretability and robustness.
Results & evaluation
The experiments are comprehensive, involving both in-domain and cross-domain evaluations across multiple datasets (ASVspoof2019 LA and ASVspoof2021 LA). The results demonstrate significant improvements in detection accuracy and robustness when utilizing the proposed framework, particularly under conditions where traditional models struggle. The ablation studies effectively illustrate the contributions of different modalities and reinforce the claim that explicit acoustic evidence enhances performance.
Reproducibility
The paper provides detailed implementation information, including model architecture, training objectives, and hyperparameters, which supports reproducibility. However, the absence of a publicly accessible code repository or demo limits the ease with which other researchers can replicate the findings.
Limitations
One limitation is the reliance on specific datasets, which may not fully capture the diversity of real-world audio deepfakes. Additionally, while the approach improves robustness, it may still be susceptible to novel spoofing techniques that exploit different acoustic characteristics not covered in the training data.
Broader impact
The implications of this research extend to various applications in security and trustworthiness of speech-based systems, such as voice authentication and content verification. By improving the detection of speech deepfakes, this work contributes to safeguarding against misinformation and enhancing the integrity of audio communications. The main contribution of this paper is the introduction of SDD-APALLM, a novel framework that enhances speech deepfake detection by explicitly exposing fine-grained acoustic evidence, thereby improving model robustness and interpretability. This work addresses a significant gap in the current methodologies for audio LLMs, providing a promising direction for future research in the field of audio processing and deepfake detection.
Author of this paper? Send a correction.