Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation
Abstract
Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Hallo-Live introduces a novel framework for real-time, text-driven audio-video avatar generation, significantly improving efficiency and quality through innovative methodologies. The combination of asynchronous dual-stream diffusion with preference-guided distillation represents a meaningful advancement in the field, with implications for various applications in interactive media and communication.
Methodology
The methodology presented in Hallo-Live is innovative, particularly with the introduction of Future-Expanding Attention and Human-Centric Preference-Guided DMD (HP-DMD). The Future-Expanding Attention mechanism addresses the critical issue of articulation lag in causal generation by allowing video blocks to access future phonetic cues, which is a significant improvement over traditional causal masking methods. The HP-DMD approach enhances the distillation process by reweighting training samples based on human-centric metrics, which is a novel contribution that emphasizes the importance of visual fidelity, speech naturalness, and audio-visual synchronization. The two-stage training pipeline is well-structured and effectively integrates these novel components.
Results & evaluation
The experimental evaluation is robust, with comprehensive benchmarking against existing models such as Ovi and others. The authors provide clear quantitative metrics, including FPS, latency, and various quality scores (VideoAlign, Sync Confidence, AudioBox), demonstrating significant improvements in efficiency and quality. The qualitative results further support the claims of robust generalization across different scenarios, showcasing the model's versatility. However, the paper could benefit from a more detailed discussion of the datasets used and the specific training conditions.
Reproducibility
The paper provides a GitHub link for code and models, which is a positive aspect for reproducibility. However, the implementation details could be more explicit, particularly regarding the training data preparation and the specific configurations used during training. Clearer documentation of hyperparameters and training procedures would enhance reproducibility.
Limitations
Some limitations include the reliance on high-performance hardware (NVIDIA H200 GPUs), which may not be accessible to all researchers. Additionally, while the model shows strong performance in various scenarios, the paper does not extensively discuss potential edge cases or failure modes in generation, which could be important for practical applications. The paper could also explore the implications of the model's performance in real-world settings beyond the controlled experimental conditions.
Broader impact
The potential applications of Hallo-Live are significant, particularly in the fields of virtual reality, gaming, and telecommunication, where real-time avatar generation can enhance user interaction and engagement. The ability to generate high-fidelity audio-visual content in real-time could also have implications for education, entertainment, and remote communication. The focus on human-centric metrics suggests a commitment to improving user experience, which is increasingly important in AI-driven applications. Hallo-Live introduces a novel framework for real-time, text-driven audio-video avatar generation, significantly improving efficiency and quality through innovative methodologies. The combination of asynchronous dual-stream diffusion with preference-guided distillation represents a meaningful advancement in the field, with implications for various applications in interactive media and communication.
Author of this paper? Send a correction.