VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing
Abstract
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a replacement-aware distillation framework for two-step video dubbing, effectively resolving the conflict between source mouth reconstruction and target audio synchronization through masked supervision. The technical contribution is significant, combining a novel training objective with efficient distillation to achieve state-of-the-art performance in lip synchronization and visual quality, supported by a comprehensive evaluation and the release of a new benchmark.
Methodology
The paper proposes a two-stage approach for video dubbing: a replacement-aware teacher training strategy and a two-step distillation pipeline. The core methodological contribution is the "replacement-aware cross-audio supervision," which addresses the conflict between source mouth reconstruction and target audio synchronization by masking out source-mouth reconstruction losses during training on unpaired audio. This is a logical and effective solution to a known problem in video editing models. The distillation process uses matched trajectory regression and distribution matching to compress a 48-step teacher into a 2-step student, which is a standard but well-executed application of recent diffusion distillation techniques. The architecture relies on Wan-1.3B with VACE conditioning, which is a solid baseline.
Results & evaluation
The experimental evaluation is comprehensive, covering multiple benchmarks (HDTF, TalkVid-Bench, VidDub-Bench) and metrics (Sync-C, Sync-D, FVD, SSIM, PSNR, VSR). The paper demonstrates state-of-the-art performance in cross-audio synchronization while maintaining high visual quality. The ablation studies clearly show the benefit of the masking strategy. The comparison with both feed-forward and multi-step diffusion methods is fair and informative. The release of a curated dataset and benchmark (VidDub-Bench) adds significant value to the community.
Reproducibility
The authors commit to releasing the training dataset, benchmark, preprocessing outputs, and unified evaluation code. The GitHub repository is provided. The detailed description of the training pipeline, loss functions, and distillation steps suggests high reproducibility, assuming the released code is complete and well-documented.
Limitations
The model is designed for offline dubbing, requiring the full source video and target speech beforehand, which limits its applicability to real-time or streaming scenarios. The reliance on a specific base model (Wan-1.3B) may limit generalizability to other architectures. The evaluation, while extensive, is primarily quantitative; subjective human evaluation (MOS) would further strengthen the claims about visual quality and naturalness.
Broader impact
This work has significant potential for applications in video localization, accessibility, and content creation. By enabling high-quality, fast (2-step) video dubbing, it can reduce the cost and time associated with traditional dubbing processes. The open-source release of the dataset and code will likely accelerate research in audio-visual synchronization and video editing. The paper introduces a replacement-aware distillation framework for two-step video dubbing, effectively resolving the conflict between source mouth reconstruction and target audio synchronization through masked supervision. The technical contribution is significant, combining a novel training objective with efficient distillation to achieve state-of-the-art performance in lip synchronization and visual quality, supported by a comprehensive evaluation and the release of a new benchmark.
Author of this paper? Send a correction.