SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
Abstract
Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This prevents the model from interacting during the user's turn and can lead to high response latency while it waits to think. Consequently, thinking after receiving the full input is not suitable for speech-to-speech interaction, where real-time, low-latency exchange is important. We address this by noting that humans naturally "think while listening." In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to the user input. SHANKS streams the input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses this unspoken reasoning to decide whether to interrupt the user and to make tool calls to complete the task. We demonstrate that SHANKS enhances real-time user-SLM interaction in two scenarios: (1) when the user is presenting a step-by-step solution to a math problem, SHANKS can listen, reason, and interrupt when the user makes a mistake, achieving 37.1% higher interruption accuracy than a baseline that interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can complete 56.9% of the tool calls before the user finishes their turn. Overall, SHANKS moves toward models that keep thinking throughout the conversation, not only after a turn ends. Animated illustrations of Shanks can be found at https://d223302.github.io/SHANKS/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SHANKS, a framework that allows spoken language models to think while listening, significantly enhancing real-time user interaction. This work presents a meaningful advancement in the field of machine learning, particularly in the development of more responsive and intelligent conversational agents.
Methodology
The methodology proposed in SHANKS is innovative, focusing on enabling spoken language models (SLMs) to generate unspoken chain-of-thought reasoning while a user is still speaking. This is achieved through a chunked input processing approach, allowing the model to listen and think simultaneously. The framework is well-structured, detailing the inference process and training data preparation, which includes generating thinking chunks based on user speech. The authors also provide a clear distinction between their work and related efforts, such as the concurrent work by Chiang et al. (2025), which focuses on thinking while speaking rather than listening. However, the methodology could benefit from more extensive details on the training process and the specific algorithms used for generating thinking chunks.
Results & evaluation
The experiments are robust, demonstrating the effectiveness of SHANKS in two distinct scenarios: interrupting a user during a math problem-solving session and making tool calls in a dialogue setting. The results show significant improvements in interruption accuracy and tool call completion rates compared to baseline models. The use of metrics such as interruption ratio, valid interrupt ratio, and call accuracy provides a comprehensive evaluation of the model's performance. However, the paper could enhance its experimental evaluation by including more diverse datasets and a broader range of use cases to validate the generalizability of the results.
Reproducibility
The paper provides a reasonable level of detail regarding the implementation of SHANKS, including the architecture used and the training data preparation. However, it lacks specific information on hyperparameters and the training environment, which could hinder reproducibility. Including a link to a code repository or supplementary materials would greatly improve the reproducibility of the findings.
Limitations
The paper acknowledges several limitations, such as the requirement for structured user speech and the fixed chunk size, which can introduce latency. Additionally, the model's performance may vary depending on the user's speech patterns and the complexity of the task. The authors also note that the thinking process may not always yield useful tokens, which could lead to inefficiencies. Addressing these limitations in future work could enhance the model's applicability in real-world scenarios.
Broader impact
The potential applications of SHANKS are significant, particularly in educational settings where real-time feedback is crucial, and in interactive dialogue systems that require low-latency responses. By enabling models to think while listening, SHANKS could lead to more natural and effective human-computer interactions. This work may also inspire further research into real-time reasoning in other domains, such as robotics and virtual assistants. The main contribution of this paper is the introduction of SHANKS, a framework that allows spoken language models to think while listening, significantly enhancing real-time user interaction. This work presents a meaningful advancement in the field of machine learning, particularly in the development of more responsive and intelligent conversational agents.
Author of this paper? Send a correction.