Beyond Monologue: Interactive Talking-Listening Avatar Generation with Conversational Audio Context-Aware Kernels
Abstract
Audio-driven human video generation has achieved remarkable success in monologue scenarios, largely driven by advancements in powerful video generation foundation models. Moving beyond monologues, authentic human communication is inherently a full-duplex interactive process, requiring virtual agents not only to articulate their own speech but also to react naturally to incoming conversational audio. Most existing methods simply extend conventional audio-driven paradigms to listening scenarios. However, relying on strict frame-to-frame alignment renders the model's response to long-range conversational dynamics rigid, whereas directly introducing global attention catastrophically degrades lip synchronization. Recognizing the unique temporal Scale Discrepancy between talking and listening behaviors, we introduce a multi-head Gaussian kernel to explicitly inject this physical intuition into the model as a progressive temporal inductive bias. Building upon this, we construct a full-duplex interactive virtual agent capable of simultaneously processing dual-stream audio inputs for both talking and listening. Furthermore, we introduce a rigorously cleaned Talking-Listening dataset VoxHear featuring perfectly decoupled speech and background audio tracks. Extensive experiments demonstrate that our approach successfully fuses strong temporal alignment with deep contextual semantics, setting a new state-of-the-art for generating highly natural and responsive full-duplex interactive digital humans. The project page is available at https://warmcongee.github.io/beyond-monologue/ .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a unified framework for generating interactive talking-listening avatars, significantly advancing the field of audio-driven video generation. The innovative use of multi-head Gaussian kernels and the creation of the VoxHear dataset represent substantial contributions, addressing key challenges in achieving natural and responsive digital human interactions.
Methodology
The paper presents a novel methodology for generating interactive talking-listening avatars using a dual-stream audio input approach. The introduction of a multi-head Gaussian kernel to manage the temporal discrepancies between talking and listening behaviors is innovative and effectively addresses the limitations of existing methods. The progressive temporal inductive bias allows for improved lip synchronization while maintaining contextual awareness, which is a significant advancement in the field of audio-driven video generation. The architecture is well-structured, and the two-stage training process is logically sound, focusing first on speaking and then integrating listening capabilities.
Results & evaluation
The authors conducted extensive experiments using their newly created VoxHear dataset, which is a substantial contribution to the field. The dataset's design, emphasizing cleanly decoupled audio tracks, enhances the reliability of the results. The experiments demonstrate clear improvements over existing methods across multiple metrics, including lip synchronization and identity preservation. The use of both objective metrics and qualitative assessments (user studies) strengthens the evaluation of the proposed method's efficacy.
Reproducibility
The paper provides detailed implementation information, including the training process, architecture specifications, and the datasets used. However, the lack of a publicly available code repository limits reproducibility. While the methodology is well-documented, sharing the code would significantly enhance the ability of other researchers to replicate and build upon this work.
Limitations
One limitation is the reliance on a large, high-quality dataset, which may not be readily available for all researchers. Additionally, while the paper addresses the trade-off between lip synchronization and contextual understanding, the complexity of the model may pose challenges for real-time applications. The paper could also discuss potential ethical implications of creating highly realistic avatars in interactive settings.
Broader impact
This research has significant implications for the development of interactive digital avatars in various applications, including virtual assistants, gaming, and telecommunication. The ability to generate avatars that can both speak and respond naturally to audio inputs could enhance user experiences in virtual environments, making interactions more lifelike and engaging. The advancements in audio-visual synchronization and contextual understanding could also pave the way for future innovations in human-computer interaction. The paper introduces a unified framework for generating interactive talking-listening avatars, significantly advancing the field of audio-driven video generation. The innovative use of multi-head Gaussian kernels and the creation of the VoxHear dataset represent substantial contributions, addressing key challenges in achieving natural and responsive digital human interactions.
Author of this paper? Send a correction.