Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering
Abstract
Full-duplex spoken language models (FD-SLMs) enable seamless speech interaction by allowing models to listen and speak simultaneously, yet the internal mechanism by which they coordinate listening and speaking remains underexplored. We analyze the predictive behavior encoded in FD-SLM hidden representations and find that they exhibit stream-specific predictive patterns: during listening, they preferentially predict the incoming user stream, whereas during speaking, they preferentially predict the model output stream. Building on this observation, we show that FD-SLMs dynamically modulate their internal predictive focus between two states: a generative state aligned with model output generation and a perceptive state aligned with incoming user input. However, this modulation can lag behind abrupt changes in conversational context. During user interruptions, the model remains transiently biased toward the generative state before transitioning into the perceptive state, causing it to miss the beginning of the incoming input. We term this delayed internal transition state inertia. To quantify its downstream impact, we introduce the Zero-Buffer Benchmark (ZBB), a diagnostic benchmark for evaluating immediate interruption comprehension when user speech begins abruptly. We evaluate this setting using response correctness and initial-word occurrence rate (IWOR). Finally, we mitigate state inertia through activation steering with a perception vector, a training-free intervention with little additional computational overhead. Across multiple state-of-the-art FD-SLMs, activation steering substantially improves interruption handling; for example, on PersonaPlex, it improves correctness from 28% to 45% and IWOR from 40% to 72% without any fine-tuning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a substantial contribution to the understanding and improvement of full-duplex spoken language models, introducing innovative methodologies and benchmarks that enhance the robustness of conversational AI systems. The technical contributions, particularly in analyzing state transitions and mitigating inertia, are significant advancements in the field.
Methodology
The paper introduces a novel framework for analyzing the internal mechanisms of full-duplex spoken language models (FD-SLMs) through the lens of hidden representations. It identifies the phenomenon of state inertia, where models struggle to transition between generative and perceptive states during interruptions. The proposed activation steering method, which utilizes a perception vector to mitigate this inertia, is innovative and practical, as it requires no fine-tuning and operates at inference time. The introduction of the Zero-Buffer Benchmark (ZBB) is a significant methodological advancement, providing a targeted evaluation of interruption handling in FD-SLMs.
Results & evaluation
The experiments are well-structured, utilizing multiple state-of-the-art FD-SLMs and evaluating their performance on the ZBB. The results demonstrate significant improvements in interruption handling metrics (correctness and IWOR) when using the activation steering method. The paper provides clear comparisons across different conditions, effectively showcasing the impact of the proposed interventions. However, more extensive testing across diverse datasets could strengthen the findings.
Reproducibility
The paper outlines the experimental setup and methodology in detail, allowing for reproducibility of results. However, the lack of publicly available models and datasets may hinder full reproducibility for external researchers. The use of human evaluations for the ZBB adds robustness but also introduces variability that could be challenging to replicate.
Limitations
The paper acknowledges limitations, including reliance on an energy-based onset detector for identifying interruptions, which may not perform well in noisy environments. Additionally, the evaluation is constrained by the limited availability of open-source FD-SLMs, which could affect the generalizability of the findings. The logit-lens-based affinity scores are also noted as potentially noisy for individual examples.
Broader impact
The findings have significant implications for the development of more effective conversational agents, particularly in applications requiring real-time interaction, such as virtual assistants and customer service bots. By improving interruption handling, the proposed methods could enhance user experience and engagement in spoken dialogue systems. The introduction of the ZBB could also pave the way for further research in evaluating conversational AI performance under challenging conditions. The paper makes a substantial contribution to the understanding and improvement of full-duplex spoken language models, introducing innovative methodologies and benchmarks that enhance the robustness of conversational AI systems. The technical contributions, particularly in analyzing state transitions and mitigating inertia, are significant advancements in the field.
Author of this paper? Send a correction.