Large Audio Language Models for Spoofing-Aware Speaker Verification
Abstract
Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV). Existing defenses mainly address this threat through binary countermeasures (CMs) for deepfake detection or spoofing-aware speaker verification (SASV), where current systems are dominated by modular ASV-CM fusion and cascaded pipelines. Although large audio language models (LALMs) have shown promise on related audio tasks, including CM and ASV, their use for SASV remains unexplored, despite their capacity to produce natural-language rationales for auditing and robustness beyond discriminative predictions. This work systematically evaluates LALMs for SASV against conventional pipelines under zero-shot prompting, supervised adaptation, reasoning-oriented training, and reinforcement-learning-based optimization. Our results show that pretrained LALMs are near chance in the zero-shot setting, confirming that they are not natively suited to SASV, but that task-specific adaptation closes this gap. We further find that competitive SASV performance can be achieved through several distinct routes. These findings position LALMs as a promising and auditable foundation for unified SASV, while clarifying where conventional cascade systems still lead.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
Methodology
The paper proposes a novel adaptation of Large Audio Language Models (LALMs) for Spoofing-Aware Speaker Verification (SASV), a task that traditionally relies on cascaded or fused discriminative pipelines (ASV + CM). The authors formulate SASV as a three-way classification problem (target, non-target, spoof) using LALMs like SALMONN and Qwen-Audio. Key methodological contributions include: 1) A composite loss function combining cross-entropy, Additive Angular Margin (AAM) for speaker discrimination, and binary cross-entropy for spoof detection to balance the competing objectives. 2) Hard-sample mining to focus training on difficult pairs. 3) An exploration of Chain-of-Thought (CoT) reasoning and Group Relative Policy Optimization (GRPO) to generate interpretable rationales for decisions. The approach is technically sound, leveraging parameter-efficient fine-tuning (LoRA) and addressing the specific challenge of balancing speaker identity verification with anti-spoofing detection within a generative framework.
Results & evaluation
The evaluation is conducted on the ASVspoof 5 dataset, a challenging benchmark for SASV. The results demonstrate that while zero-shot LALMs perform near chance, supervised fine-tuning (SFT) significantly improves performance. The proposed method with composite losses and hard mining achieves competitive results against strong baselines (ECAPA-TDNN + WavLM/W2V2-AASIST fusion), particularly in terms of overall accuracy and min a-DCF. However, the conventional baselines still outperform the LALM in EER, which the authors attribute to the lack of calibrated continuous scores in the token-based LALM output. The ablation studies effectively isolate the contribution of each component (AAM head, spoof head, hard mining). The reasoning component, while interesting, did not yield performance gains over hard-label SFT, which is a significant finding but limits the technical impact of that specific branch.
Reproducibility
The paper provides sufficient detail regarding model architectures (SALMONN-7B, Qwen2-Audio-7B), training hyperparameters (LoRA rank, learning rate, batch size), and data processing (resampling, augmentation). The dataset (ASVspoof 5 + VoxCeleb) is standard. However, the code and pre-trained checkpoints are not mentioned as available, and the specific cold-start reasoning generation process using Step-Audio R1 is described but the model weights are not provided, making full reproduction of the CoT component difficult. The evaluation on a stratified sample of 20K pairs rather than the full set is noted, which aids speed but requires careful interpretation of metrics.
Limitations
The primary limitation is the lack of performance superiority over established discriminative baselines in key metrics like EER. The LALM approach is computationally more expensive and slower than conventional pipelines. The reasoning traces, while interpretable, are not validated for correctness or utility in a human-evaluation setting, and the authors admit they are prone to hallucination. The "reasoning" capability does not translate to better hard-label accuracy, questioning the practical value of the CoT component for the core task. The reliance on external models (Step-Audio R1) for data generation introduces a dependency that is not fully analyzed.
Broader impact
This work opens a new direction for using LALMs in security-critical audio applications, offering the potential for interpretable decision-making which is crucial for auditing voice authentication systems. It highlights the gap between general audio understanding and specific security tasks. The findings help guide future research on whether reasoning capabilities can be effectively integrated into security systems or if discriminative models remain superior for raw performance. It contributes to the broader understanding of LALM capabilities and limitations in specialized domains. This paper provides the first systematic evaluation of Large Audio Language Models for Spoofing-Aware Speaker Verification, demonstrating that while they can be adapted to compete with conventional pipelines, they currently lag in threshold-based metrics and require significant task-specific adaptation to overcome their lack of native security awareness.
Author of this paper? Send a correction.