Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions
Abstract
While recent Spoken Language Models (SLMs) have been actively deployed in real-world scenarios, they lack the capability to discern Third-Party Interruptions (TPI) from the primary user's ongoing flow, leaving them vulnerable to contextual failures. To bridge this gap, we introduce TPI-Train, a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling, and TPI-Bench, a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. Experiments demonstrate that our dataset design mitigates semantic shortcut learning-a critical pitfall where models exploit semantic context while neglecting acoustic signals essential for discerning speaker changes. We believe our work establishes a foundational resource for overcoming text-dominated unimodal reliance in SLMs, paving the way for more robust multi-party spoken interaction. The code for the framework is publicly available at https://tpi-va.github.io
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper establishes a foundational framework for TPI-aware voice assistants, addressing a critical gap in current Spoken Language Models. It innovatively combines dataset construction with evaluation methodologies to enhance the robustness of multi-party spoken interactions, paving the way for more effective voice assistant technologies.
Methodology
The authors present a comprehensive framework that addresses the critical issue of Third-Party Interruptions (TPI) in Spoken Language Models (SLMs). They introduce TPI-Train, a dataset of 88K instances designed to prioritize acoustic cues for interruption handling, and TPI-Bench, an evaluation framework for measuring interruption-handling strategies. The methodology is well-structured, integrating speaker-aware hard negatives to mitigate semantic shortcut learning, which is a significant advancement in the field. The proposed two-stage response strategy classification (actionable vs. ignorable) is innovative and allows for flexibility in user interactions.
Results & evaluation
The experiments conducted are rigorous, demonstrating the effectiveness of the proposed methods through quantitative metrics such as Response Strategy Following (RSF) and Overall Helpfulness (OH). The use of both synthetic and real-world test sets enhances the validity of the findings. The ablation studies provide insights into the contributions of different components of the training data, showcasing the robustness of the proposed model against existing baselines.
Reproducibility
The paper provides clear details on the dataset construction, evaluation metrics, and experimental setup, which supports reproducibility. The code for the framework is publicly available, which further enhances the potential for other researchers to replicate the study and build upon it.
Limitations
While the framework offers a significant step forward, it acknowledges that the response strategies are inherently subjective and user-dependent. The study focuses on turn-based interactions, which may limit its applicability in full-duplex settings where the assistant can continuously listen and respond. Additionally, the reliance on LLMs for generating responses may introduce biases based on the training data used.
Broader impact
The proposed framework has the potential to significantly improve the user experience with voice assistants by enabling them to handle interruptions more effectively. This advancement could lead to more natural and engaging interactions in multi-party settings, which are increasingly relevant in real-world applications. The work lays a foundation for future research in multi-party dialogue systems and could influence the design of more sophisticated conversational agents. The paper establishes a foundational framework for TPI-aware voice assistants, addressing a critical gap in current Spoken Language Models. It innovatively combines dataset construction with evaluation methodologies to enhance the robustness of multi-party spoken interactions, paving the way for more effective voice assistant technologies.
Author of this paper? Send a correction.