Full-Duplex Interaction with Speech, Expressions and Gestures
Abstract
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
Methodology
The paper proposes ELLSA 2, a unified end-to-end framework that integrates full-duplex speech interaction with streaming gesture and facial expression generation. The core methodological contribution is the "contextual semantic call" mechanism, which decouples the decision of *whether* to perform a semantic gesture from *how* to realize it. This is achieved via a binary classification head on the LLM backbone's hidden states, which triggers the retrieval of a curated exemplar template to guide the autoregressive gesture generator. The architecture uses a chunk-wise autoregressive synthesizer that interleaves speech, listening, and motion tokens, allowing non-verbal behaviors to persist during verbal silence (listening states). The use of RVQ-VAE for motion tokenization and a curriculum training strategy to integrate modalities is technically sound and well-motivated.
Results & evaluation
The experimental setup is robust, introducing a new human-verified benchmark (138 clips) specifically for dialog-aware gesture generation, covering full-duplex behaviors and semantic gestures. The evaluation employs both objective metrics (prosodic alignment, activity ratio) and subjective human evaluation, along with MLLM-as-a-judge metrics. The results demonstrate that ELLSA 2 outperforms cascaded baselines and remains competitive with state-of-the-art full-duplex speech models. The semantic template guidance shows a significant improvement in semantic gesture quality (human score 2.014 to 2.711). The correlation analysis between MLLM judges and human annotators (0.853/0.919) provides strong evidence for the utility of the proposed automatic metrics.
Reproducibility
The paper provides extensive implementation details, including tokenizer specifications, training hyperparameters, and data curation pipelines. The authors commit to releasing code, data, and model checkpoints upon acceptance. The detailed description of the semantic exemplar library construction and the evaluation protocols enhances reproducibility.
Limitations
The system relies on a curated library of 3,227 exemplars for semantic gestures, which may limit the diversity of semantic actions compared to fully generative approaches. The evaluation, while improved, is still based on a relatively small test set (138 clips). The model's performance on long-horizon coherence of gestures over extended conversations is not deeply analyzed.
Broader impact
This work advances the field of human-computer interaction by moving beyond text or speech-only interfaces to fully embodied, multimodal agents. The proposed evaluation metrics and benchmark will be valuable for the community working on co-speech gesture generation and full-duplex dialogue systems. The integration of semantic understanding with physical gesture generation has implications for robotics, virtual avatars, and immersive communication. ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
Author of this paper? Send a correction.