Hierarchical Codec Diffusion for Video-to-Speech Generation
Abstract
Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of HiCoDiT, a Hierarchical Codec Diffusion Transformer that leverages the hierarchical structure of speech tokens for improved video-to-speech generation. This work represents a substantial advancement in the field by addressing the limitations of existing methods and providing a robust framework for future research and applications in multimodal audio-visual synthesis.
Methodology
The proposed methodology, HiCoDiT, introduces a novel hierarchical codec diffusion transformer that effectively utilizes the hierarchical structure of speech tokens to improve video-to-speech generation. By incorporating low-level and high-level blocks for token generation, the model captures both speaker-aware semantics and prosodic details, which is a significant advancement over existing methods that treat speech as a flat sequence. The dual-scale adaptive instance layer normalization is particularly innovative, allowing for better conditioning of speech generation based on visual features.
Results & evaluation
The experiments are extensive, utilizing well-known datasets such as VoxCeleb2, LRS2, and LRS3. The paper provides a comprehensive evaluation with both subjective (MOS, A/B testing) and objective metrics (WER, DNSMOS, MCD), demonstrating that HiCoDiT outperforms state-of-the-art methods in several key areas, including naturalness and synchronization. The ablation studies further validate the importance of the hierarchical modeling and dual-scale AdaLN in enhancing performance.
Reproducibility
The paper includes sufficient implementation details, such as the training procedure, model architecture, and hyperparameters, which support reproducibility. The availability of the code and demo enhances the likelihood that other researchers can replicate the results.
Limitations
While the paper demonstrates strong performance, it does not address potential limitations related to the diversity of the training data, which may affect the model's generalization capabilities. Additionally, the reliance on specific visual features for conditioning may limit applicability in scenarios where such features are not easily extractable.
Broader impact
The implications of this work are significant, particularly for applications in assistive communication, dubbing, and other areas where video-to-speech generation can enhance user experience. The hierarchical approach could pave the way for more nuanced and expressive speech synthesis systems, potentially benefiting a wide range of industries. The main contribution of this paper is the introduction of HiCoDiT, a Hierarchical Codec Diffusion Transformer that leverages the hierarchical structure of speech tokens for improved video-to-speech generation. This work represents a substantial advancement in the field by addressing the limitations of existing methods and providing a robust framework for future research and applications in multimodal audio-visual synthesis.
Author of this paper? Send a correction.