Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio
Abstract
Modern text-to-speech (TTS) systems, particularly those built on Large Audio-Language Models (LALMs), generate high-fidelity speech that faithfully reproduces input text and mimics specified speaker identities. While prior misuse studies have focused on speaker impersonation, this work explores a distinct content-centric threat: exploiting TTS systems to produce speech containing harmful content. Realizing such threats poses two core challenges: (1) LALM safety alignment frequently rejects harmful prompts, yet existing jailbreak attacks are ill-suited for TTS because these systems are designed to faithfully vocalize any input text, and (2) real-world deployment pipelines often employ input/output filters that block harmful text and audio. We present HARMGEN, a suite of five attacks organized into two families that address these challenges. The first family employs semantic obfuscation techniques (Concat, Shuffle) that conceal harmful content within text. The second leverages audio-modality exploits (Read, Spell, Phoneme) that inject harmful content through auxiliary audio channels while maintaining benign textual prompts. Through evaluation across five commercial LALMs-based TTS systems and three datasets spanning two languages, we demonstrate that our attacks substantially reduce refusal rates and increase the toxicity of generated speech. We further assess both reactive countermeasures deployed by audio-streaming platforms and proactive defenses implemented by TTS providers. Our analysis reveals critical vulnerabilities: deepfake detectors underperform on high-fidelity audio; reactive moderation can be circumvented by adversarial perturbations; while proactive moderation detects 57-93% of attacks. Our work highlights a previously underexplored content-centric misuse vector for TTS and underscore the need for robust cross-modal safeguards throughout training and deployment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper makes a substantial contribution by exploring a previously underexamined misuse vector for TTS systems, presenting innovative attack methodologies, and demonstrating their effectiveness across various platforms. The findings highlight urgent security concerns that need to be addressed in the development and deployment of TTS technologies.
Methodology
The paper introduces HARMGEN, a novel suite of attacks that creatively combines semantic obfuscation and audio-modality exploits to bypass TTS safety mechanisms. The methodology is well-structured, addressing two significant challenges in the misuse of TTS systems. The authors provide a clear description of the attack families and their operational principles, which demonstrates a thoughtful approach to the problem. However, the complexity of the attacks may require further elaboration for full comprehension.
Results & evaluation
The experiments are comprehensive, involving five commercial LALMs-based TTS systems and three datasets across two languages. The evaluation metrics are relevant, and the results show a substantial reduction in refusal rates and an increase in toxicity, indicating that the proposed attacks are effective. However, a deeper analysis of the datasets used and the specific metrics for measuring toxicity would enhance the robustness of the findings.
Reproducibility
The paper lacks detailed implementation information, which raises concerns about reproducibility. While the methodology is described, specifics regarding the experimental setup, including parameters and configurations used in the attacks, are not sufficiently detailed. This could hinder other researchers from replicating the results.
Limitations
The study primarily focuses on the effectiveness of the attacks without a thorough exploration of the ethical implications or potential countermeasures in depth. Additionally, the performance of the attacks may vary with different TTS systems, and the paper does not address this variability extensively.
Broader impact
The implications of this research are significant, as it highlights a critical vulnerability in TTS systems that could be exploited for malicious purposes. The findings underscore the need for enhanced safeguards in TTS deployment and raise awareness about the potential for harmful content generation, which is increasingly relevant in today's digital landscape. The work could influence future research directions in TTS safety and security. This paper makes a substantial contribution by exploring a previously underexamined misuse vector for TTS systems, presenting innovative attack methodologies, and demonstrating their effectiveness across various platforms. The findings highlight urgent security concerns that need to be addressed in the development and deployment of TTS technologies.
Author of this paper? Send a correction.