FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection
Abstract
Recent advances in AudioLLMs have enabled spoken dialogue systems to move beyond turn-based interaction toward real-time full-duplex communication, where the agent must decide when to speak, yield, or interrupt while the user is still talking. Existing full-duplex approaches either rely on voice activity cues, which lack semantic understanding, or on ASR-based modules, which introduce latency and degrade under overlapping speech and noise. Moreover, available datasets rarely capture realistic interaction dynamics, limiting evaluation and deployment. To mitigate the problem, we propose \textbf{FastTurn}, a unified framework for low-latency and robust turn detection. To advance latency while maintaining performance, FastTurn combines streaming CTC decoding with acoustic features, enabling early decisions from partial observations while preserving semantic cues. We also release a test set based on real human dialogue, capturing authentic turn transitions, overlapping speech, backchannels, pauses, pitch variation, and environmental noise. Experiments show FastTurn achieves higher decision accuracy with lower interruption latency than representative baselines and remains robust under challenging acoustic conditions, demonstrating its effectiveness for practical full-duplex dialogue systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FastTurn presents a unified framework for low-latency and robust turn detection in full-duplex dialogue systems. The technical contributions, particularly in integrating acoustic and semantic cues, represent a meaningful advancement in the field of audio processing and dialogue systems, with potential applications in various real-time communication scenarios.
Methodology
The methodology presented in FastTurn is innovative, combining streaming CTC decoding with acoustic features to enhance turn detection in full-duplex dialogue systems. The architecture is well-structured, comprising three main components that progressively integrate semantic and acoustic cues. The use of a four-stage training pipeline is commendable, as it stabilizes the optimization process and aligns speech and text modalities effectively. However, the reliance on CTC for initial transcription raises concerns about potential error propagation in noisy environments.
Results & evaluation
The experiments are thorough, utilizing a diverse set of datasets and a comprehensive evaluation framework. The introduction of a new test set with realistic human dialogue scenarios is a significant contribution, allowing for better assessment of the model's performance in practical applications. The results demonstrate that FastTurn outperforms existing baselines in terms of accuracy and latency, underscoring its effectiveness. However, the paper could benefit from additional comparisons with more recent models in the field to contextualize its performance.
Reproducibility
The paper provides sufficient details regarding the model architecture, training strategy, and evaluation metrics, which aids in reproducibility. However, the absence of publicly available code or a demo could hinder independent verification of results. Clear instructions for reproducing the experiments would enhance the paper's impact.
Limitations
One limitation is the potential sensitivity of the model to CTC errors, especially in overlapping speech scenarios. Additionally, while the model shows robustness in various conditions, the performance on English datasets did not meet expectations, indicating a need for further optimization. The paper also does not address the computational resources required for training and inference, which could be a barrier for broader adoption.
Broader impact
The FastTurn framework has significant implications for real-time spoken dialogue systems, particularly in applications requiring low-latency interaction, such as virtual assistants and customer service bots. By improving turn detection, it can enhance user experience and facilitate more natural conversations. The release of the new dataset also opens avenues for future research in dialogue systems, potentially leading to advancements in multimodal interaction technologies. FastTurn presents a unified framework for low-latency and robust turn detection in full-duplex dialogue systems. The technical contributions, particularly in integrating acoustic and semantic cues, represent a meaningful advancement in the field of audio processing and dialogue systems, with potential applications in various real-time communication scenarios.
Author of this paper? Send a correction.