OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Abstract
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Methodology
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
Results & evaluation
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
Reproducibility
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
Limitations
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
Broader impact
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Author of this paper? Send a correction.