Native Active Perception as Reasoning for Omni-Modal Understanding
Abstract
Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
OmniAgent introduces a novel native active perception framework for omni-modal video understanding, leveraging POMDP-based iterative reasoning and entropy-steered reinforcement learning to achieve state-of-the-art performance with significantly reduced computational overhead compared to passive models.
Methodology
The paper proposes OmniAgent, a native omni-modal agent that formulates video understanding as a Partially Observable Markov Decision Process (POMDP). The core innovation lies in the Observation-Thought-Action (OTA) cycle, which allows the model to actively query video frames, audio, and clips, distilling them into a persistent textual memory. This decouples reasoning complexity from video duration. The training methodology is robust, featuring Agentic Supervised Fine-Ting (SFT) with best-of-N trajectory synthesis and dual-stage quality control (outcome verification and rationality audit). Crucially, the paper introduces TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), an RL objective that addresses "Advantage Homogenization" in multi-turn agentic reasoning by using turn-level entropy to steer credit assignment toward pivotal discovery turns. This is a significant methodological contribution to the field of agentic LLMs, particularly in handling long-horizon, multi-modal reasoning where standard RL algorithms struggle with sparse or uniform reward signals.
Results & evaluation
The empirical evaluation is comprehensive, covering ten benchmarks across video understanding, audio-visual reasoning, and temporal grounding. OmniAgent-7B achieves state-of-the-art performance among open-source models, notably outperforming the significantly larger Qwen2.5-VL-72B on LVBench (50.5% vs 47.3%). The paper provides strong ablation studies demonstrating the efficacy of Agentic SFT over standard SFT, and TAURA over vanilla GRPO. The analysis of "Advantage Homogenization" is well-supported by entropy analysis, showing that critical reasoning steps exhibit higher entropy. The test-time scaling analysis further validates the active perception paradigm, showing performance improvements with increased reasoning turns until saturation. The results are statistically significant and demonstrate clear advantages in both accuracy and computational efficiency (fewer frames processed).
Reproducibility
The paper provides extensive implementation details, including the architecture of the environment (Ray/Verl), specific ffmpeg parameters for media extraction, and hyperparameters for both SFT and RL stages. The code and models are released on GitHub and Hugging Face, ensuring high reproducibility. The detailed description of the "Dual-Stage Quality Control" and the specific reward functions allows for independent verification of the training pipeline.
Limitations
The primary limitation is the latency overhead introduced by the sequential interaction loop. While the model is more parameter-efficient, the time taken to execute multiple OTA cycles is significantly higher than single-shot inference. The paper acknowledges this and suggests parallelized exploration as future work. Additionally, the reliance on a teacher model for SFT trajectory synthesis and rationality audit introduces potential bias or cost, although the paper mitigates this with automated verification where possible. The performance gain, while significant, is relative to other open-source models; closing the gap with proprietary models like Gemini-2.5-Pro remains a challenge.
Broader impact
This work has significant implications for making long-form video understanding computationally feasible and scalable. By decoupling reasoning from raw data volume, it enables the analysis of hour-long videos on modest hardware. The active perception paradigm aligns more closely with human cognitive processes, potentially leading to more interpretable and robust AI systems. The TAURA method could be generalized to other multi-turn agentic tasks beyond video, such as complex web browsing or scientific discovery, where identifying critical information steps is paramount. OmniAgent introduces a novel native active perception framework for omni-modal video understanding, leveraging POMDP-based iterative reasoning and entropy-steered reinforcement learning to achieve state-of-the-art performance with significantly reduced computational overhead compared to passive models.
Author of this paper? Send a correction.