OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Abstract
We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
OmniSeek introduces an agentic framework for Omni-LLMs that enables active, multi-turn audio-visual evidence seeking through a novel tool-use protocol and a three-phase training strategy featuring an Audio-Visual Necessity reward. The paper makes a substantial contribution to multimodal reasoning by addressing the dilution of fine-grained evidence in long contexts, demonstrating that dynamic, modality-decoupled retrieval significantly improves performance on complex audio-visual benchmarks compared to single-pass or text-only reasoning approaches.
Methodology
The paper proposes OmniSeek, an agentic framework that transforms Omni-LLMs into active multi-turn reasoning agents. The core methodological contribution is the decoupling of perception and reasoning via a tool-use protocol (`think` -> `tool_call` -> `observe`). Unlike prior works that rely on single-pass encoding or text-only Chain-of-Thought (CoT), OmniSeek allows the model to dynamically retrieve specific audio or video segments at higher resolution/fps based on its evolving reasoning state. The training strategy is robust, employing a three-phase pipeline: (1) SFT on a newly synthesized dataset (OmniTraj-170K) to instill the tool-use format; (2) RL with verifiable rewards (accuracy, format, tool-use) for broad exploration; and (3) Hard-example refinement using a novel "Audio-Visual Necessity" (AVN) reward. The AVN reward is a significant technical contribution, using attention masking to compute counterfactual log-likelihoods, thereby penalizing trajectories that rely on single-modality shortcuts. This ensures the model genuinely integrates both modalities rather than hallucinating grounding.
Results & evaluation
The experimental evaluation is extensive, covering 10 omni-modal benchmarks and 4 general video understanding benchmarks. OmniSeek demonstrates state-of-the-art or highly competitive performance among open-source models, particularly on long-form and complex reasoning tasks (e.g., MMOU, LVOmni). Ablation studies clearly isolate the contributions of each training phase and the AVN reward, showing that the multi-turn tool-use paradigm significantly outperforms single-turn text-only CoT baselines. The analysis of tool-call frequency vs. accuracy provides insightful evidence that the model learns adaptive reasoning strategies, using more tools for complex cross-modal correlations.
Reproducibility
The paper provides high reproducibility. It details the three-stage data engine for OmniTraj-170K, including specific prompts for visual annotation and QA generation. The training hyperparameters, hardware requirements (H200 GPUs), and optimization settings for each phase are explicitly listed in the appendix. The definition of the AVN reward is mathematically precise, allowing for straightforward implementation. While the base model (Qwen3-Omni-30B) is proprietary, the framework and training procedure are sufficiently detailed for replication on similar Omni-LLMs.
Limitations
The primary limitation is the computational cost of the multi-turn inference process, which involves multiple forward passes and dynamic tool invocations, potentially increasing latency compared to single-pass models. Additionally, the reliance on a large, synthesized dataset (OmniTraj-170K) raises questions about the generalizability of the learned tool-use behaviors to out-of-distribution video content not covered by the FineVideo source. The paper also notes an "alignment tax" during the SFT phase, where performance temporarily drops, indicating sensitivity to the training data distribution.
Broader impact
This work has significant implications for the development of agentic multimodal models. By demonstrating that active evidence acquisition improves reasoning over passive ingestion, it provides a blueprint for building more robust and interpretable AI systems that can handle long, complex audio-visual streams. The AVN objective is a valuable technique for preventing modality shortcuts, which could be applied to other multimodal reasoning tasks. The framework also highlights the potential of RL with verifiable rewards in shaping complex, multi-step behaviors in LLMs. OmniSeek introduces an agentic framework for Omni-LLMs that enables active, multi-turn audio-visual evidence seeking through a novel tool-use protocol and a three-phase training strategy featuring an Audio-Visual Necessity reward. The paper makes a substantial contribution to multimodal reasoning by addressing the dilution of fine-grained evidence in long contexts, demonstrating that dynamic, modality-decoupled retrieval significantly improves performance on complex audio-visual benchmarks compared to single-pass or text-only reasoning approaches.
Author of this paper? Send a correction.