Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection
Abstract
Modern Large audio-language models (LALMs) power intelligent voice interactions by tightly integrating audio and text. This integration, however, expands the attack surface beyond text and introduces vulnerabilities in the continuous, high-dimensional audio channel. While prior work studied audio jailbreaks, the security risks of malicious audio injection and downstream behavior manipulation remain underexamined. In this work, we reveal a previously overlooked threat, auditory prompt injection, under realistic constraints of audio data-only access and strong perceptual stealth. To systematically analyze this threat, we propose \textit{AudioHijack}, a general framework that generates context-agnostic and imperceptible adversarial audio to hijack LALMs. \textit{AudioHijack} employs sampling-based gradient estimation for end-to-end optimization across diverse models, bypassing non-differentiable audio tokenization. Through attention supervision and multi-context training, it steers model attention toward adversarial audio and generalizes to unseen user contexts. We also design a convolutional blending method that modulates perturbations into natural reverberation, making them highly imperceptible to users. Extensive experiments on 13 state-of-the-art LALMs show consistent hijacking across 6 misbehavior categories, achieving average success rates of 79\%-96\% on unseen user contexts with high acoustic fidelity. Real-world studies demonstrate that commercial voice agents from Mistral AI and Microsoft Azure can be induced to execute unauthorized actions on behalf of users. These findings expose critical vulnerabilities in LALMs and highlight the urgent need for dedicated defense.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel framework for auditory prompt injection attacks against LALMs, revealing critical vulnerabilities and proposing effective methodologies for adversarial audio generation. The comprehensive analysis of the technical contributions, methodology, and significance underscores the importance of addressing security in multimodal AI systems.
Methodology
The proposed methodology, AudioHijack, is innovative in its approach to crafting context-agnostic and imperceptible adversarial audio for hijacking large audio-language models (LALMs). It effectively addresses the challenges of gradient obstruction during audio tokenization and context sensitivity through sampling-based gradient estimation and attention-guided context generalization techniques. The convolutional perturbation blending method enhances perceptual stealth, making the adversarial audio less detectable. The combination of these methods represents a significant advancement in the field of adversarial machine learning, particularly in the context of audio inputs.
Results & evaluation
The experiments conducted on 13 state-of-the-art LALMs demonstrate the robustness and effectiveness of the AudioHijack framework. The reported success rates of 79%-96% across various misbehavior categories and unseen user contexts provide strong evidence of the framework's efficacy. The inclusion of real-world tests on commercial voice agents further validates the practical implications of the findings. However, the paper could benefit from a more detailed analysis of the experimental setup, including the specific metrics used for evaluating success rates and the diversity of the datasets employed.
Reproducibility
The paper provides a clear link to the project repository, which includes code and data for reproducing the experiments. However, the paper could improve its reproducibility by including more detailed descriptions of the experimental parameters and configurations used during training and evaluation. Additionally, providing access to the datasets or detailed descriptions of them would enhance reproducibility.
Limitations
While the paper presents a comprehensive framework, it does not fully address potential countermeasures or defenses against the proposed attacks. The focus on hijacking LALMs without a thorough examination of mitigation strategies may limit the practical applicability of the findings. Furthermore, the reliance on specific audio carriers and the potential for variability in real-world scenarios could affect the generalizability of the results.
Broader impact
The findings of this research have significant implications for the security of audio-language models and voice-activated systems. By exposing vulnerabilities in LALMs, the work highlights the need for improved defenses against adversarial attacks in voice interfaces, which are increasingly prevalent in everyday technology. The research could inform the development of more robust models and security protocols, ultimately contributing to safer human-computer interactions. This paper presents a novel framework for auditory prompt injection attacks against LALMs, revealing critical vulnerabilities and proposing effective methodologies for adversarial audio generation. The comprehensive analysis of the technical contributions, methodology, and significance underscores the importance of addressing security in multimodal AI systems.
Author of this paper? Send a correction.