Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
Abstract
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel framework for object-grounded co-speech gesture generation using a causal latent diffusion model. It makes significant contributions through the introduction of a new dataset (SceneGes), novel temporal masking techniques for speech-gesture alignment, and a gated fusion module for incorporating 3D object geometry, resulting in more realistic and physically consistent gestures.
Methodology
The paper proposes a three-stage framework for co-speech gesture generation. Stage 1 introduces a Causal Variational Autoencoder (CausalVAE) that encodes gesture primitives into a temporally ordered latent space, avoiding the quantization errors of VQ-VAEs and the entanglement of bidirectional latents. Stage 2 employs an autoregressive diffusion model in this latent space, utilizing novel temporal cross-attention masking (Audio Window and Text Span) to enforce precise speech-gesture synchronization. Stage 3 introduces a Gated Object Fusion module using Basis Point Sets (BPS) to condition the frozen base model on 3D object geometry, enabling object-grounded generation without retraining the core gesture prior. The methodology is sound, with a clear separation of concerns between learning the gesture prior and injecting scene awareness. The use of causal latents for autoregressive generation is a strong architectural choice for streaming and long-horizon tasks.
Results & evaluation
The experiments are comprehensive, covering quantitative comparisons on BEAT2 and Embody3D, ablation studies on the CausalVAE and masking strategies, and specific evaluations for object awareness. The introduction of the SceneGes dataset is a significant contribution, providing the first synthetic 3D dataset for this specific task. The evaluation metrics include standard FGD and BC, along with new metrics for diversity and collision penetration. The results show improvements in realism, synchronization, and diversity over baselines like LoM and GestureLSM. The object-awareness ablation clearly demonstrates the reduction in body-object collisions. However, the reliance on a synthetic dataset (SceneGes) for the primary object-grounded claims is a potential weakness, as generalization to real-world scenes is not directly tested.
Reproducibility
The paper provides detailed implementation specifics, including network architectures, loss functions, hyperparameters, and dataset construction pipelines. The release of the SceneGes dataset and the project page with video samples enhances reproducibility. However, the code repository is not explicitly linked in the provided text (only the project page), which may limit immediate reproducibility for practitioners. The use of specific proprietary models for dataset generation (Gemini, Veo 3) introduces dependencies that may be hard to replicate exactly.
Limitations
The primary limitation is the reliance on a synthetic dataset (SceneGes) for object-grounded generation, which may not fully capture the complexity and variability of real-world environments. The model is evaluated primarily on upper-body gestures, and the handling of full-body dynamics in complex scenes is limited. The computational cost of the diffusion process, even in latent space, may be a barrier for real-time applications, although the causal structure helps. The paper does not extensively discuss the failure modes of the object fusion module in cases of significant occlusion or dynamic object movement.
Broader impact
This work has significant implications for the development of embodied AI agents, virtual avatars, and interactive entertainment. By enabling gestures that are aware of posture and surrounding objects, the model can produce more natural and physically plausible interactions in virtual environments. The introduction of the SceneGes dataset and new evaluation metrics will benefit the broader community working on human motion synthesis and human-scene interaction. The causal latent formulation also opens up possibilities for real-time streaming gesture generation and interactive control. The paper presents a novel framework for object-grounded co-speech gesture generation using a causal latent diffusion model. It makes significant contributions through the introduction of a new dataset (SceneGes), novel temporal masking techniques for speech-gesture alignment, and a gated fusion module for incorporating 3D object geometry, resulting in more realistic and physically consistent gestures.
Author of this paper? Send a correction.