SentiAvatar: Towards Expressive and Interactive Digital Humans
Abstract
We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SentiAvatar, a framework that advances the creation of expressive interactive digital humans through innovative methodologies and a comprehensive dataset. This work represents a significant step forward in the field of machine learning and computer graphics, with the potential to influence future research and applications in multimodal interaction and digital character animation.
Methodology
The methodology presented in SentiAvatar is innovative, particularly in its approach to addressing the challenges of creating expressive digital humans. The construction of the SuSuInterActs dataset, which includes synchronized speech, full-body motion, and facial expressions, is a significant contribution to the field, as it provides a large-scale, high-quality multimodal dataset that can be utilized for further research. The proposed audio-aware plan-then-infill architecture effectively separates semantic planning from prosody-driven interpolation, which is a novel approach that enhances the realism and expressiveness of generated motions. This decoupling allows for more flexibility and better alignment with the nuances of human speech, showcasing a thoughtful integration of audio and motion data.
Results & evaluation
The experiments conducted demonstrate the effectiveness of the SentiAvatar framework, achieving state-of-the-art results on both the SuSuInterActs and BEATv2 benchmarks. The reported metrics (R@1 43.64% for SuSuInterActs and FGD 4.941, BC 8.078 for BEATv2) indicate a substantial improvement over existing baselines, which underscores the technical advancements made by the authors. The ability to produce 6 seconds of output in just 0.3 seconds with unlimited multi-turn streaming is particularly impressive and highlights the practical applicability of the framework in real-time scenarios.
Reproducibility
The paper mentions that the source code, model, and dataset are available online, which is a positive aspect for reproducibility. However, the paper could benefit from more detailed implementation guidelines and descriptions of the training process to ensure that other researchers can replicate the results without ambiguity.
Limitations
One limitation of the study is the reliance on a single character (SuSu) for the dataset, which may limit the generalizability of the findings to other characters or scenarios. Additionally, while the framework shows promise for real-time applications, the performance in more complex, varied interactions beyond the dataset's scope remains to be evaluated.
Broader impact
The potential applications of SentiAvatar are vast, ranging from gaming and virtual reality to telepresence and social robotics. By enabling more expressive and interactive digital humans, this research could significantly enhance user experiences in various domains, including entertainment, education, and mental health support. The main contribution of this paper is the introduction of SentiAvatar, a framework that advances the creation of expressive interactive digital humans through innovative methodologies and a comprehensive dataset. This work represents a significant step forward in the field of machine learning and computer graphics, with the potential to influence future research and applications in multimodal interaction and digital character animation.
Author of this paper? Send a correction.