HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos
Abstract
Most voice-cloning systems are built for audio alone: they copy the reference recording's delivery and acoustics, so a voice cloned on a phone sounds like a phone recording whether the speaker is pictured in a studio, on a street, or in a cathedral. Speech for a talking video should sound like the pictured person speaking in the pictured place. We present HeyGen Voice, a scene-aware voice-cloning model that separates who is speaking from how and where: it keeps the speaker's voice from an everyday recording while the target image sets the delivery and acoustics. Prior image-conditioned TTS does half of this job: it either adds scene acoustics without keeping a specific speaker or infers a voice from a face. HeyGen Voice is a multimodal autoregressive transformer conditioned on the image, speaker and scene captions, and the voice recording, trained in five stages that move from multilingual TTS pretraining through simulated and real cross-scene adaptation to reinforcement and preference optimization. Two clone policies trade scene adaptation against fidelity to the original recording, and an audio anchor holds long scripts in one voice and one acoustic environment. Listeners prefer HeyGen Voice over leading commercial systems across a third-party arena, production votes, and talking-video studies: it ranks first (Elo 1201) on the public Artificial Analysis controlled-voice TTS leaderboard (October 2026); in a blind test, users cloning their own voice chose it over ElevenLabs v4 and Cartesia Sonic 3.6 in 62% and 54% of votes; and volunteers rated talking videos of their own digital twin about 10 to 30 points higher (out of 100) when voiced by HeyGen Voice than by leading commercial alternatives. Try HeyGen Voice and hear samples at https://heygen.com/heygen-voice.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
Methodology
The paper proposes "HeyGen Voice," a multimodal autoregressive transformer for scene-aware voice cloning. The core methodological contribution is the decoupling of speaker identity (timbre, accent) from scene acoustics (reverberation, background noise) and delivery style. This is achieved through a conditioning prefix that combines pooled reference-voice tokens, image tokens (from a frozen VLM), and explicit text captions for speaker and background. A key technical innovation is the "cross-scene training curriculum," which includes a simulated adaptation stage using reference perturbations to remove the shortcut of relying on the reference recording's acoustics, followed by fine-tuning on real cross-scene pairs. The use of GRPO for intelligibility/speaker similarity and DPO with a low-rank adapter for expressiveness is a robust application of modern alignment techniques to TTS. The "clone policies" (voice-clone vs. full-clone) provide a practical mechanism for users to control the trade-off between fidelity to the source recording and adaptation to the target scene.
Results & evaluation
The evaluation is extensive and multi-faceted, covering objective benchmarks (cross-scene similarity, multilingual CER), subjective human evaluations (paired listening tests in 5 languages), production user votes, third-party arena rankings (Artificial Analysis), and video-based studies. The cross-scene benchmark (331 cases) is a valuable contribution to the field, as it specifically tests the ability to adapt voice to new environments, a task often ignored in standard TTS benchmarks. The human studies are well-designed, including blind tests with users cloning their own voices and volunteer studies rating talking videos. The results show consistent superiority over leading commercial systems (ElevenLabs, Cartesia, Inworld) in scene alignment and user preference, particularly in the "voice-clone" mode. The inclusion of a "corrected" multilingual benchmark addresses known issues with transcription errors in standard datasets.
Reproducibility
Reproducibility is limited by the proprietary nature of the system. While the architecture is described in detail, the specific data pipelines (especially the "cross-scene pair mining" and the 5,000+ GPU training infrastructure) are not open-sourced. The paper references internal infrastructure (HELIOS) and data reports (Avatar V) that are not publicly available. The benchmarks (cross-scene, multilingual) are described but not released as public datasets with code. The use of specific commercial APIs for baselines limits the ability to reproduce the exact comparisons without access to those services.
Limitations
The primary limitation is the lack of open-source code and data, which hinders independent verification and community adoption. The reliance on a frozen VLM for image understanding may limit the model's ability to handle complex or abstract scene descriptions not well-captured by the VLM. The "captioning" step at inference (where an LLM writes captions from the image) introduces an additional point of failure and latency. The paper does not extensively discuss the computational cost of the multimodal conditioning compared to standard TTS, although it mentions an RTF of 0.1. The evaluation, while strong, is largely conducted by the authors or their platform users, which may introduce bias, although the third-party arena ranking helps mitigate this.
Broader impact
This work has significant implications for the creation of digital humans and talking avatars, where audio-visual coherence is crucial. By enabling voice cloning that adapts to the visual scene, it enhances the realism and immersion of AI-generated video content. The techniques developed for cross-scene adaptation and consistency in long-form synthesis are applicable to other multimodal generation tasks. The focus on user-controlled trade-offs (clone policies) sets a precedent for more flexible and user-centric AI audio tools. HeyGen Voice introduces a scene-aware voice cloning framework that decouples speaker identity from acoustic environment using multimodal conditioning and a cross-scene training curriculum. The paper demonstrates state-of-the-art performance in both objective cross-scene benchmarks and subjective human evaluations, establishing a new standard for audio-visual coherence in talking video generation.
Author of this paper? Send a correction.