Positive-Incentive Noise Predictor for Adversarial Purification in Speaker Verification
Abstract
Modern automatic speaker verification (ASV) systems are vulnerable to adversarial perturbations. Diffusion-based purification has recently shown strong effectiveness against such perturbations, but its reverse denoising process requires iterative sampling and leads to high inference latency. We find that the forward noising process provides most of the robustness gain. Motivated by this observation, we reformulate adversarial purification as a learnable noising problem, and propose the Positive-Incentive Noise Predictor (PnP), the first framework that explicitly introduces positive-incentive noise (π-noise) into the purification task. PnP learns input-adaptive π-noise and mixes it with the input to improve the robustness of downstream ASV systems. Experiments on four advanced ASV backbones show that PnP effectively defends against adversarial attacks while preserving performance on natural speech. Compared with representative purification baselines, the proposed framework provides a competitive balance among defense effectiveness, impact on genuine utterances, and inference efficiency under white-box, black-box, and defender-aware adaptive attacks, with a real-time factor as low as 0.014. Moreover, PnP can be cascaded with a diffusion denoiser to further improve the perceptual quality of purified utterances. Code and purified audio examples are available at https://eurecom-asp.github.io/pnp/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
Methodology
The paper proposes a novel paradigm shift in adversarial purification for Automatic Speaker Verification (ASV). Instead of relying on the computationally expensive reverse denoising process of diffusion models, the authors hypothesize that the forward noising process provides the majority of the robustness gain. They introduce the Positive-Incentive Noise Predictor (PnP), which learns an input-adaptive noise pattern ($\pi$-noise) that is task-beneficial (i.e., it preserves speaker identity while suppressing adversarial perturbations). The methodology involves training a U-Net based noise predictor using a variational lower bound of mutual information, instantiated as a hinge loss on ASV similarity scores. The framework includes variants like PnP-Gaussian (simple additive) and PnP-Diff (diffusion-style schedule). The approach is theoretically grounded in information theory and practically motivated by the inefficiency of current diffusion-based purifiers.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors test on four state-of-the-art ASV backbones (ECAPA-TDNN, CAM++, ResNet, SimAMResNet) and under three attack settings: white-box (MI-FGSM, PGD), black-box (FAKEBOB), and defender-aware adaptive attacks. They compare against strong baselines including DAP, AudioPure, and neural codecs. Key findings include: 1) PnP-Diff achieves state-of-the-art robustness with a very low Real-Time Factor (RTF) of 0.014, significantly faster than iterative diffusion methods. 2) The forward-process-only hypothesis is validated, showing minimal performance drop compared to full diffusion pipelines. 3) Cascading PnP with a diffusion denoiser improves perceptual quality (WB-PESQ, SI-SDR) without significantly compromising robustness. The ablation studies on hyperparameters and purification steps add depth to the analysis.
Reproducibility
The paper provides detailed descriptions of the architecture, loss functions, and training procedures. The code and purified audio examples are available via the provided URL, which greatly enhances reproducibility. The datasets (VoxCeleb, LibriSpeech) are standard and accessible. The use of open-source toolkits (WeSpeaker, torchattacks) further supports reproducibility.
Limitations
The primary limitation is that PnP-Gaussian, while fast, degrades audio quality significantly (low WB-PESQ, high WER), making it less suitable for applications where perceptual quality is critical. The PnP-Diff variant is better balanced but still introduces some distortion compared to the clean signal. The method is specifically tailored for ASV; its generalizability to other audio tasks (e.g., speech recognition, emotion recognition) is not explored in depth, although the core idea might transfer. The adaptive attack evaluation is limited to gradient-based attacks through the purifier; more complex adaptive attacks (e.g., black-box adaptive) are not fully explored.
Broader impact
This work has significant implications for the security of biometric systems. By providing a lightweight, effective defense against adversarial attacks, it enhances the trustworthiness of ASV systems in real-world applications. The insight that forward noising can be optimized for robustness opens new avenues for efficient adversarial defense in other domains using generative models. However, the dual-use nature of adversarial attacks and defenses means that improved defenses may also spur more sophisticated attacks, necessitating continuous research. The paper presents a significant and well-executed contribution to adversarial robustness in speaker verification by reformulating diffusion-based purification as a learnable forward noising problem, achieving a superior balance between defense effectiveness, inference efficiency, and audio quality.
Author of this paper? Send a correction.