Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models
Abstract
End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a principled, psychoacoustically aligned latent smoothing defense for full-duplex speech agents, significantly reducing adversarial attack success rates while maintaining utility, though its certified guarantees are conservative and its institutional provenance is obscure.
Methodology
The paper proposes Psychoacoustically Aligned Latent Smoothing (PALS), a defense mechanism for full-duplex speech-to-speech (S2S) models. The core innovation lies in applying randomized smoothing at the latent interface of the Residual Vector Quantization (RVQ) codec, using anisotropic Gaussian noise shaped by local codebook covariance. This is coupled with an input-space augmentation that matches the psychoacoustic masking threshold, derived via a maximum-entropy argument. The training objective incorporates a KL consistency term (similar to TRADES) to ensure robustness without requiring adversarial examples during training. The theoretical contribution includes a certified robustness radius for the smoothed operator, though the authors correctly note that the certificate is a lower bound and the empirical robustness of the deployed model (PALS-Train) is the primary metric of interest. The methodology is sound, leveraging the geometric properties of the quantizer to define the noise distribution, which is a clever and non-trivial design choice.
Results & evaluation
The experiments are comprehensive, evaluating three distinct attack types (semantic hijacking, response suppression/muting, and jailbreaking) against a Moshi-style agent. The paper reports significant reductions in attack success rates (e.g., hijacking drops from 91.7% to 8.3%) while maintaining clean quality metrics (UTMOS, WER, Turn-Taking F1). Ablations clearly demonstrate the contribution of each component (anisotropic covariance, input-space dual, KL consistency). The inclusion of over-the-air (OTA) tests with reverberation and babble noise adds practical relevance. However, the evaluation relies on a specific "Moshi-style" agent architecture; generalizability to other S2S architectures (e.g., those without RVQ or with different codec designs) is not tested.
Reproducibility
The paper provides detailed hyperparameters, training objectives, and attack configurations. It mentions a released `certify.py` harness. However, the code and model checkpoints are not explicitly linked in the provided text (no GitHub URL found), and the "Moshi-style" agent is described rather than provided as a specific open-source artifact. The reliance on specific proprietary or large-scale datasets (CANDOR, Seamless Interaction) may limit immediate reproducibility for smaller groups, though the methodology is clearly specified.
Limitations
The certified robustness radius is relatively small (0.616 latent units, corresponding to -26 dB input budget), which is far below the attack budgets tested empirically. The authors acknowledge this, positioning the certificate as a "floor" rather than a guarantee against strong attacks. The defense is specific to RVQ-based codecs; its applicability to continuous latent spaces or other quantization schemes is unclear. The "People Make Things" affiliation is obscure, raising questions about the scale of resources behind the work, though the technical depth suggests significant effort.
Broader impact
As full-duplex voice agents become prevalent, securing them against imperceptible audio attacks is critical. This work provides a strong baseline for latent-space defenses in speech, potentially influencing future designs of secure speech interfaces. The psychoacoustic alignment of the defense with the attack constraint is a principled approach that could be adapted to other sensory modalities. The paper introduces a principled, psychoacoustically aligned latent smoothing defense for full-duplex speech agents, significantly reducing adversarial attack success rates while maintaining utility, though its certified guarantees are conservative and its institutional provenance is obscure.
Author of this paper? Send a correction.