RedVox: Safety and Fairness Gaps in Speech Models Across Languages
Abstract
Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox, a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox, we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant contribution to the field of AI safety by introducing RedVox, the first multilingual speech safety and fairness benchmark built on naturalistic human voices. It effectively highlights the urgent need for improved safety evaluations in non-English settings and under realistic interaction conditions, providing both a valuable resource and critical insights into the vulnerabilities of current speech models.
Methodology
The paper introduces RedVox, a novel benchmark for evaluating safety and fairness in speech-capable models across five languages. The methodology is robust in its design: it moves beyond synthetic voices to use naturalistic human recordings, addressing a critical gap in current evaluation practices. The dual-request type design (Speech vs. Audio) effectively isolates the impact of paralinguistic features and modality on model vulnerability. The use of an LLM-as-a-judge (GPT-5.5) for evaluation, validated against human annotators with high inter-annotator agreement, provides a scalable and reliable assessment framework. The inclusion of a sociotechnical analysis of the data collection process adds significant depth, highlighting the ethical and psychological challenges of red-teaming with speech data.
Results & evaluation
The experimental evaluation is comprehensive, covering eight state-of-the-art models (both open-weight and proprietary) across five languages. The results clearly demonstrate that safety vulnerabilities persist under naturalistic conditions and are exacerbated in non-English languages and with spoken inputs. The analysis of "safe-by-accident" responses adds nuance to the safety metrics, revealing that some models appear safe only due to poor comprehension. The statistical analysis, including Spearman's correlation for ranking preservation and Chi-squared tests for distribution robustness, strengthens the validity of the findings. The comparison between open and proprietary models highlights a concerning trend where open models often exhibit higher vulnerability rates.
Reproducibility
The paper provides a clear description of the data collection protocol, including participant consent, data filtering (VAD), and quality control measures. The code is available under an Apache 2.0 license, and the dataset is released under a gated-access model, which is appropriate for sensitive content. The detailed experimental settings, including model versions and inference parameters, enhance reproducibility. The validation of the LLM-judge against human annotations further supports the reliability of the evaluation pipeline.
Limitations
The study is limited to five high-resource Indo-European languages, which may not generalize to typologically distinct or low-resource languages. The focus on naturalistic, non-adversarial requests means that the benchmark may not capture the full extent of vulnerabilities under deliberate jailbreaking attempts. The single-turn evaluation setup does not account for multi-turn dynamics or context-dependent safety behaviors. Additionally, the reliance on an LLM-judge, while validated, still introduces potential biases inherent in the judge model itself.
Broader impact
RedVox addresses a critical need for multilingual and naturalistic safety evaluation in speech AI. By exposing significant gaps in current model safety, particularly for non-English speakers, the work has important implications for equitable AI deployment. The documentation of the sociotechnical challenges in speech data collection provides valuable insights for future research in AI ethics and safety. The benchmark serves as a foundational resource for the community to develop more robust and fair speech models. This paper presents a significant contribution to the field of AI safety by introducing RedVox, the first multilingual speech safety and fairness benchmark built on naturalistic human voices. It effectively highlights the urgent need for improved safety evaluations in non-English settings and under realistic interaction conditions, providing both a valuable resource and critical insights into the vulnerabilities of current speech models.
Author of this paper? Send a correction.