DuoGesture: Neuro-Inspired and Biomechanically Informed Dual-Stream Co-Speech Gesture Generation
Abstract
Co-speech gesture generation requires both semantic expressivity and biomechanically plausible rhythmic motion. Existing holistic gesture models mix lexically grounded semantic gestures with frequent prosody-aligned beat gestures. This limits semantic grounding, speech-motion alignment, and kinematic smoothness. We propose \emph{DuoGesture}, a neuro-inspired and biomechanically informed dual-stream approach that decomposes co-speech gesture synthesis into coupled semantic and beat streams. The two streams are coordinated by a \emph{Semantic Variational Information Bottleneck}, a stochastic frame-level gate that learns when semantic gestures should override rhythmic beat motion. The semantic stream is controlled by \emph{Motion-Grounded Semantic Conditioning}, which replaces purely linguistic word embeddings with motion-language representations to provide motion-aligned semantic priors for long-tailed lexical triggers of gestures. The beat stream is further regularised by an \emph{Inertial Beat Prior}, an anthropometry-weighted arm-chain module that reduces jitter and improves rhythmic consistency without constraining semantic frames. Objective evaluations and subjective experiments show that DuoGesture outperforms strong holistic baselines, while component ablations confirm the complementary roles of semantic grounding, stochastic stream selection, and biomechanical regularisation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DuoGesture proposes a novel dual-stream framework for co-speech gesture generation that effectively separates and integrates semantic and rhythmic motion. This work significantly advances the state of the art in gesture synthesis, addressing key limitations of previous models and demonstrating strong empirical results across various evaluation metrics.
Methodology
The methodology presented in DuoGesture is innovative, utilizing a dual-stream approach that separates semantic and beat gestures, which is a significant departure from previous holistic models. The introduction of Motion-Grounded Semantic Conditioning (MGSC), Semantic Variational Information Bottleneck (S-VIB), and Inertial Beat Prior (IBP) provides a robust framework for addressing the limitations of existing gesture generation models. Each component is well-justified, and the integration of biomechanical principles into the model enhances its realism and applicability in real-world scenarios.
Results & evaluation
The experiments are thorough, utilizing the BEAT2 dataset, which is a well-established benchmark in the field. The paper presents both objective and subjective evaluations, demonstrating that DuoGesture outperforms existing models across multiple metrics. The ablation studies effectively highlight the contributions of each component, providing strong evidence for the proposed approach's efficacy. The user study further validates the model's performance in a perceptually meaningful way, indicating that the generated gestures are perceived as more natural and aligned with speech.
Reproducibility
The paper provides detailed implementation information, including architecture specifics, training protocols, and evaluation metrics. However, the lack of a publicly available code repository limits the ease of reproducibility. While the methodology is clear, the absence of shared code could hinder other researchers from validating the results independently.
Limitations
The authors acknowledge limitations regarding the generalizability of DuoGesture across different languages, cultures, and interaction settings. The reliance on a pretrained text-to-motion representation may introduce biases, and the biomechanical prior may not fully capture the complexities of all types of gestures. Future work is needed to address these issues and expand the model's applicability.
Broader impact
The implications of this research are significant for the fields of human-computer interaction, robotics, and animation. By improving the realism and expressiveness of co-speech gestures, DuoGesture could enhance the effectiveness of virtual avatars and embodied agents, leading to more natural and engaging interactions. The findings could also inform future research on gesture generation and multimodal communication. DuoGesture proposes a novel dual-stream framework for co-speech gesture generation that effectively separates and integrates semantic and rhythmic motion. This work significantly advances the state of the art in gesture synthesis, addressing key limitations of previous models and demonstrating strong empirical results across various evaluation metrics.
Author of this paper? Send a correction.