Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Prediction
Abstract
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Gelina represents a significant advancement in the field of multimodal synthesis, offering a unified framework that effectively integrates speech and gesture generation. The innovative methodology, combined with strong experimental validation, positions this work as a valuable contribution to the ongoing research in human behavior synthesis and multimodal communication.
Methodology
The methodology presented in Gelina is innovative, introducing an interleaved token autoregressive architecture that allows for simultaneous synthesis of speech and gestures. This approach addresses the limitations of traditional sequential synthesis methods, enhancing synchrony and prosody alignment. The use of a conditional flow-matching decoder to improve gesture quality further demonstrates a sophisticated understanding of multimodal generation. The training strategy that leverages large unimodal datasets for generalization under sparse paired data is particularly commendable, showcasing a clever adaptation to existing data limitations.
Results & evaluation
The experimental evaluation is robust, utilizing both objective metrics and subjective user studies to assess the performance of Gelina. The choice of datasets, including GigaSpeech and BEAT2, is appropriate for the tasks at hand, and the comparative analysis against strong unimodal baselines highlights the effectiveness of the proposed model. The results indicate that Gelina achieves competitive performance in both speech quality and gesture generation, which is a significant achievement given the complexity of joint synthesis.
Reproducibility
The paper provides detailed implementation specifics, including architecture choices, training procedures, and evaluation metrics, which enhance reproducibility. However, the absence of a publicly available code repository limits the ability for others to fully replicate the results without additional effort.
Limitations
The primary limitations identified include the model's focus on body gestures, excluding finer details such as finger movements and facial expressions. Additionally, the speech quality is constrained by the tokenizer used, which could be improved in future iterations. The reliance on large-scale datasets may also pose challenges in terms of accessibility for researchers with limited resources.
Broader impact
The implications of Gelina extend to various applications in human-computer interaction, such as embodied conversational agents and social robotics, where natural multimodal communication is essential. The ability to generate synchronized speech and gestures could significantly enhance user experience and engagement in these domains. Gelina represents a significant advancement in the field of multimodal synthesis, offering a unified framework that effectively integrates speech and gesture generation. The innovative methodology, combined with strong experimental validation, positions this work as a valuable contribution to the ongoing research in human behavior synthesis and multimodal communication.
Author of this paper? Send a correction.