Protecting Bystander Privacy via Selective Hearing in LALMs
Abstract
Large audio language models (LALMs) are increasingly deployed in real-world settings where they inevitably capture speech from unintended nearby bystanders, raising privacy risks that existing benchmarks and defences largely overlook. We introduce SH-Bench, the first benchmark designed to evaluate selective hearing: a model's ability to attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. SH-Bench contains 3,968 multi-speaker audio mixtures spanning both real-world and synthetic scenarios, paired with 77k multiple-choice questions that probe models under general and selective operating modes. We propose Selective Efficacy (SE), a unified metric capturing both multi-speaker comprehension and bystander-privacy protection. Our evaluation of state-of-the-art open-source and proprietary LALMs reveals substantial privacy leakage, with strong audio understanding failing to translate into selective protection of bystander privacy. To mitigate this gap, we introduce Bystander Privacy Fine-Tuning (BPFT), a training pipeline that teaches models to refuse bystander-related queries without degrading main-speaker comprehension. BPFT yields substantial gains which improve SE by up to 15.9% over Gemini 2.5 Pro, demonstrating that selective hearing is learnable but far from achieved in current LALMs. SH-Bench and BPFT provide the first systematic framework for measuring and improving bystander privacy in audio foundation models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a pioneering approach to evaluating and improving bystander privacy in large audio language models through the introduction of SH-Bench and BPFT. This work is significant as it not only highlights an overlooked aspect of AI deployment but also provides a systematic framework for addressing privacy concerns, marking a crucial step towards responsible AI development in audio processing.
Methodology
The paper introduces a novel benchmark (SH-Bench) specifically designed to evaluate the selective hearing capabilities of LALMs, focusing on bystander privacy. The methodology includes a comprehensive dataset of multi-speaker audio mixtures and a unified metric (Selective Efficacy) to assess both comprehension and privacy protection. The introduction of the Bystander Privacy Fine-Tuning (BPFT) training pipeline is a significant methodological advancement that addresses the identified privacy gaps in existing models.
Results & evaluation
The experiments are well-structured, utilizing a diverse set of models and rigorous evaluation metrics. The results demonstrate the effectiveness of BPFT in improving bystander privacy protection while maintaining comprehension abilities. The evaluation of various models under different operational modes provides a thorough understanding of the current state of LALMs in terms of privacy risks.
Reproducibility
The paper provides a clear description of the data collection and evaluation processes, along with links to the GitHub repository for code and implementation details. However, specific hyperparameters and training configurations for the models used in experiments could be better detailed to enhance reproducibility.
Limitations
The paper acknowledges limitations in focusing solely on single main speaker scenarios and does not explore more complex situations like group discussions. Additionally, while BPFT shows improvements, it may lead to slight degradation in main speaker comprehension, which could be a concern in practical applications.
Broader impact
The work has significant implications for the deployment of LALMs in real-world applications, particularly in contexts where privacy is a concern. By addressing the privacy of bystanders, the research contributes to the ethical deployment of AI technologies in everyday environments, potentially influencing future standards and practices in the field. The paper presents a pioneering approach to evaluating and improving bystander privacy in large audio language models through the introduction of SH-Bench and BPFT. This work is significant as it not only highlights an overlooked aspect of AI deployment but also provides a systematic framework for addressing privacy concerns, marking a crucial step towards responsible AI development in audio processing.
Author of this paper? Send a correction.