ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Cheng Luo, Bizhu Wu, Bing Li ...Full author list in the original paper.

King Abdullah University of Science and Technology +3 · arXiv

Affiliations: King Abdullah University of Science and Technology, Shenzhen University, University of Nottingham, University of Nottingham Ningbo China

Paper Demo

Abstract

In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The main contribution of this paper is the introduction of a new task and dataset for generating reactive listener motions from speaker utterances, coupled with a robust generative framework that integrates multiple modalities. This work represents a meaningful advancement in the understanding and modeling of nonverbal communication, with significant implications for various applications in AI and human-computer interaction.

Methodology

The paper introduces a novel task of generating reactive listener motions based on speaker utterances, which is a significant step in the intersection of natural language processing and computer vision. The methodology involves the creation of a large-scale dataset, ReactMotionNet, that captures the one-to-many nature of listener behavior, allowing for more nuanced modeling of human reactions. The authors propose a generative framework that integrates multiple modalities (text, audio, emotion, and motion) and employs preference-based objectives, which is a sophisticated approach that enhances the diversity and appropriateness of generated motions. The preference-oriented evaluation protocols are particularly noteworthy as they address the shortcomings of conventional metrics.

Results & evaluation

The experiments demonstrate that ReactMotion outperforms existing baselines, including retrieval methods and cascaded LLM-based pipelines. The extensive evaluation across various metrics, including qualitative assessments of generated motions, provides strong evidence of the framework's effectiveness. However, the paper would benefit from a more detailed analysis of the quantitative results, including statistical significance testing, to further substantiate the claims made.

Reproducibility

The paper provides sufficient methodological details to allow for reproducibility, including descriptions of the dataset, model architecture, and training procedures. However, the lack of a publicly available code repository limits the ease with which other researchers can replicate the results. The authors should consider releasing their code to enhance reproducibility and foster further research in this area.

Limitations

One limitation is the reliance on subjective evaluations for assessing the appropriateness of listener motions, which can introduce variability in the results. Additionally, the dataset may not fully capture the vast range of human reactions in diverse contexts, potentially limiting the generalizability of the model. The authors could also explore the implications of cultural differences in nonverbal communication, which may affect the model's performance across different populations.

Broader impact

The potential applications of this research are significant, particularly in fields such as virtual reality, human-computer interaction, and robotics. By enabling more naturalistic interactions between humans and machines, the work could enhance user experience in various applications, from virtual assistants to social robots. Moreover, the framework could serve as a foundation for future research on multimodal interaction and emotional AI, contributing to the development of more empathetic and responsive systems. The main contribution of this paper is the introduction of a new task and dataset for generating reactive listener motions from speaker utterances, coupled with a robust generative framework that integrates multiple modalities. This work represents a meaningful advancement in the understanding and modeling of nonverbal communication, with significant implications for various applications in AI and human-computer interaction.

Author of this paper? Send a correction.