VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model
Abstract
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
Methodology
The paper proposes VoxTurn, a 0.8B speech LLM based on Qwen3-ASR-0.6B, adapted for turn-taking decisions. The core methodological innovation is the "one-pass multi-question" inference mechanism using a block-isolating attention mask. This allows the model to answer multiple related questions (e.g., end-of-turn, interruption, backchannel, invalid input) in a single forward pass, mathematically equivalent to asking them sequentially but significantly more efficient. The training strategy is also notable: instead of using segmented utterances with static labels, the authors train on real conversational pauses, sampling decision times uniformly from within the pause and deriving labels from the timing of subsequent speech. This addresses the distribution mismatch between training (segmented) and deployment (streaming/real-time) scenarios. The use of a linear head on hidden states for multiple-choice scoring rather than text generation is a standard but effective adaptation for classification tasks in LLMs.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors evaluate on eot-bench (14 languages) and TurnBench (two-party conversations). Key results include 11.5% false cut-offs in English at 300ms latency, comparable to commercial detectors, and the best open-model performance on the 14-language mean (19.9%). The ablation studies clearly demonstrate the impact of real pause data versus segmented data, showing that the data strategy is the primary driver of performance gains. The efficiency analysis shows a 3.7x speedup for multi-question inference compared to separate passes. The integration tests with Pipecat and LiveKit frameworks provide strong evidence of practical deployability, showing reduced cut-off rates and lower latency in real-world agent setups.
Reproducibility
The paper provides high reproducibility. The authors release the model weights, code, and framework plugins on Hugging Face. They provide detailed training hyperparameters, data mixture descriptions, and the specific attention mask implementation details. The use of open-source benchmarks (eot-bench, TurnBench) and the release of an interactive demo further enhance reproducibility and verification.
Limitations
The model is evaluated primarily on English and a set of 14 languages, but the performance gap with commercial detectors in some languages (Hindi, Korean) remains. The reliance on a specific backbone (Qwen3-ASR) may limit generalizability to other architectures. The "zero-shot" performance on TurnBench is strong, but the model was trained on data that overlaps conceptually with the task, so true generalization to unseen conversational dynamics is hard to assess fully. The latency budget of 300ms is tight, and performance may degrade if hardware constraints are stricter.
Broader impact
This work has significant impact on the development of real-time voice agents. By providing an open-source model that matches commercial performance in turn-taking, it lowers the barrier to entry for building high-quality conversational AI. The multi-question approach is a valuable architectural pattern that can be applied to other sequential decision-making tasks in speech. The focus on real pause data sets a new standard for training data curation in this domain. VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
Author of this paper? Send a correction.