Audio-Visual Intelligence in Large Foundation Models

You Qin, Kai Liu, Shengqiong Wu ...Full author list in the original paper.

University of Toronto +7 · arXiv

Affiliations: University of Toronto, National University of Singapore, Queen Mary University of London, The Hong Kong University of Science and Technology, The University of Texas at Dallas, University of Oxford, University of Rochester, Microsoft Research

Paper

Abstract

Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper serves as a foundational reference for future research on large-scale Audio-Visual Intelligence, effectively synthesizing existing knowledge and proposing a structured framework for the field. Its comprehensive nature and focus on methodological synthesis position it as a significant contribution to the understanding and advancement of multimodal AI systems.

Methodology

The paper provides a comprehensive survey of Audio-Visual Intelligence (AVI) within the context of large foundation models, establishing a unified taxonomy that organizes diverse tasks and synthesizes methodological foundations. The authors effectively categorize tasks into understanding, generation, and interaction, and detail the foundational techniques necessary for AVI, including representation learning, cross-modal fusion, and generative modeling. The methodology is well-structured, providing a clear framework for future research.

Results & evaluation

While the paper is primarily a survey and does not present original experimental results, it does curate existing datasets, benchmarks, and evaluation metrics across various task families. This curation is crucial for establishing a baseline for future work and promoting reproducibility in the field. However, the lack of new experimental results limits the paper's impact in demonstrating the efficacy of the proposed methodologies.

Reproducibility

The paper outlines the methodologies and frameworks clearly, which aids in reproducibility. However, as it is a survey, it does not provide specific implementation details or code, which could enhance reproducibility further. The authors mention that they will publicly release summarized resources, which is a positive step toward transparency.

Limitations

The paper acknowledges the fragmentation in the literature and the inconsistencies in evaluation practices, which complicate systematic comparisons. It also highlights open challenges in synchronization, spatial reasoning, and safety that remain unaddressed in current research. Additionally, the lack of new experimental validation of the proposed frameworks is a notable limitation.

Broader impact

The survey addresses a rapidly evolving field with significant implications for various applications, including assistive technologies, education, robotics, and entertainment. By consolidating knowledge and establishing a coherent framework, the paper can guide future research directions and foster collaboration across subfields. The emphasis on safety and governance also highlights the importance of ethical considerations in the deployment of AVI systems. The paper serves as a foundational reference for future research on large-scale Audio-Visual Intelligence, effectively synthesizing existing knowledge and proposing a structured framework for the field. Its comprehensive nature and focus on methodological synthesis position it as a significant contribution to the understanding and advancement of multimodal AI systems.

Author of this paper? Send a correction.