SMC-ITA: Sequential Monte Carlo Inference-Time Alignment for Video-to-Audio Generation
Abstract
Video-to-audio (V2A) generation must jointly satisfy audiovisual alignment, semantic consistency, temporal synchronization, and perceptual quality. While prior work has mainly focused on model architecture, multimodal conditioning, and training objectives, inference-time alignment for V2A remains underexplored. In this paper, we study inference-time alignment for flow-matching-based V2A generation and formulate it as a search problem. We propose Sequential Monte Carlo Inference-Time Alignment (SMC-ITA), which combines lookahead-based reward estimation and sequential Monte Carlo resampling to reallocate computation adaptively using multi-dimensional cross-modal rewards. SMC-ITA improves over naive single-trajectory sampling, achieving a 55.67% relative reduction in DeSync, a 20.23% improvement in IB-score, and a 15.44% improvement in Audio Quality. Under matched NFE budgets, it also achieves the best overall trade-off among the compared search baselines, outperforming Best-of-N and Beam Search. Ablation studies further show that lookahead improves the reliability of intermediate reward estimates and that systematic resampling is a strong practical default for V2A inference-time alignment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SMC-ITA, a novel method for inference-time alignment in video-to-audio generation that significantly improves the quality and synchronization of generated audio through adaptive computation and multi-dimensional reward optimization. This work stands out for its innovative approach to a relatively underexplored area in multimodal generation, providing a strong foundation for future research and applications in the field.
Methodology
The proposed Sequential Monte Carlo Inference-Time Alignment (SMC-ITA) method addresses the challenge of inference-time alignment in video-to-audio generation by framing it as a search problem. It innovatively combines lookahead-based reward estimation with sequential Monte Carlo resampling, allowing for adaptive computation reallocation based on multi-dimensional cross-modal rewards. This approach effectively mitigates the noise in early-stage reward signals, which is a significant improvement over traditional single-trajectory sampling methods. The methodology is well-structured, with clear definitions of the reward functions and a robust framework for trajectory evaluation and resampling.
Results & evaluation
The experiments are comprehensive, utilizing a well-defined dataset (VGGSound) and comparing SMC-ITA against several strong baselines, including naive sampling, Best-of-N, and Beam Search. The results demonstrate significant improvements across multiple metrics, including a 55.67% reduction in DeSync and enhancements in audio quality and alignment scores. The ablation studies further validate the contributions of individual components, such as the lookahead strategy and the resampling method, reinforcing the robustness of the findings.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including the model architecture, evaluation metrics, and specific configurations used for each method. However, the absence of a publicly available code repository or demo limits the reproducibility of the results. Future work should consider sharing the implementation to facilitate further research and validation by the community.
Limitations
One limitation is the reliance on a fixed NFE budget, which may not be optimal for all scenarios. The paper also does not explore the scalability of SMC-ITA beyond the tested configurations, nor does it address potential computational overhead introduced by the lookahead strategy. Additionally, while the subjective evaluation shows promise, the sample size for human evaluation is relatively small, which may affect the generalizability of the findings.
Broader impact
The advancements in inference-time alignment for video-to-audio generation have significant implications for multimedia applications, including content creation, accessibility for the hearing impaired, and automated media production. By improving the alignment and quality of generated audio, this research can enhance user experiences in various domains, from entertainment to education. The main contribution of this paper is the introduction of SMC-ITA, a novel method for inference-time alignment in video-to-audio generation that significantly improves the quality and synchronization of generated audio through adaptive computation and multi-dimensional reward optimization. This work stands out for its innovative approach to a relatively underexplored area in multimodal generation, providing a strong foundation for future research and applications in the field.
Author of this paper? Send a correction.