Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
Abstract
Full-duplex spoken dialogue systems enable simultaneous listening and speaking, but their audio-only perception often fails under background noise and overlapping speech, leading to incoherent responses. Recent audio-visual dialogue approaches show that incorporating visual cues such as lip movements improve robustness under audio corruption. However, existing approaches often adapt the large speech dialogue model itself to process visual input, requiring costly multimodal training. We propose AV-STE, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM. The downstream dialogue model remains entirely frozen, preserving its pretrained conversational capabilities. When integrated with frozen Moshi, AV-STE improves average GPT-4o-judged response coherence from 1.42 to 1.91 under same-dataset speaker interference while largely preserving turn-taking behavior. Gains also transfer to out-of-domain Seamless Interaction.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
Methodology
The paper proposes AV-STE, a modular front-end for enhancing semantic speech tokens in full-duplex dialogue systems. The core contribution is the decoupling of robustness enhancement from the large language model (LLM) backbone. By keeping the speech LLM (Moshi) frozen and only enhancing the semantic token stream using a streaming audio-visual encoder (modified AV-HuBERT) and a noise-adaptive fusion module, the authors address a significant practical limitation of existing multimodal dialogue systems that require costly retraining. The introduction of "Soft Token Cross-Attention" to handle uncertainty in noisy tokenizer outputs and "Noise-Adaptive Modulation" based on entropy are technically sound and well-motivated innovations. The adaptation of bidirectional AV-HuBERT to a causal, streaming architecture with bounded lookahead is a necessary and well-executed engineering contribution for real-time applications.
Results & evaluation
The experimental setup is rigorous, covering in-domain (LRS3) and out-of-domain (Seamless Interaction) datasets, with various noise types (non-speech, speaker interference). The evaluation metrics are appropriate for the task: semantic token accuracy for the front-end and GPT-4o-judged coherence plus turn-taking metrics (TOR, latency) for the downstream dialogue. The results show substantial improvements in token recovery (e.g., from 12.61% to 65.60% under same-dataset interference) and downstream coherence (1.42 to 1.91). The comparison against a non-streaming baseline (PASE) and a different dialogue model (PersonaPlex) provides useful context, although the primary comparison is against the frozen Moshi baseline. The qualitative examples strongly support the quantitative findings.
Reproducibility
The paper provides a GitHub link to the code and models. The training configuration, dataset preprocessing, and model architecture details are described in sufficient detail for reproduction. The use of standard datasets (LRS3, AudioSet, Seamless Interaction) and open-source components (Moshi, AV-HuBERT) further enhances reproducibility.
Limitations
The paper acknowledges that AV-STE assumes reliable visual input and does not handle visual occlusion or missing video. There is a noted domain shift issue where clean-token accuracy drops on out-of-domain data (Seamless Interaction), indicating sensitivity to domain characteristics. The reliance on GPT-4o for evaluation, while common, introduces a potential bias, though it is a standard practice in current LLM evaluation. The method is specific to the Moshi/Mimi tokenization scheme, limiting immediate applicability to other speech LLM architectures without adaptation.
Broader impact
This work has significant implications for the deployment of robust spoken dialogue systems in real-world, noisy environments. By providing a plug-and-play module that enhances robustness without retraining the expensive LLM backbone, it lowers the barrier to entry for audio-visual dialogue systems. The focus on full-duplex interaction is timely and relevant to the next generation of conversational AI. The approach could be extended to other modalities or tokenization schemes, potentially influencing the design of future multimodal speech interfaces. The paper presents a novel and effective modular approach to enhancing the robustness of full-duplex spoken dialogue systems by leveraging audio-visual cues to restore corrupted semantic tokens without retraining the underlying speech LLM. The technical contributions, including the streaming adaptation of AV-HuBERT and the noise-adaptive fusion mechanism, are well-validated through comprehensive experiments on in-domain and out-of-domain datasets, demonstrating significant improvements in both token recovery and downstream dialogue coherence.
Author of this paper? Send a correction.