Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
Abstract
Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of OmniClean, a visually debiased evaluation framework for omni-modal language models, and the demonstration of its effectiveness through a comprehensive staged post-training approach. This work significantly enhances the interpretability of model performance and sets a new standard for evaluating multi-modal capabilities in machine learning.
Methodology
The paper introduces a novel evaluation framework, OmniClean, which effectively filters out visually solvable queries to assess the true omni-modal capabilities of language models. This methodology is significant as it addresses the prevalent issue of visual shortcuts in existing benchmarks, providing a more accurate measure of model performance across audio-visual-language tasks. The staged post-training approach, OmniBoost, combines mixed bi-modal supervised fine-tuning, mixed-modality reinforcement learning, and self-distillation, showcasing a comprehensive strategy for enhancing model performance.
Results & evaluation
The experiments are well-structured, utilizing a large dataset of 16,968 queries, from which 8,551 were retained after visual-only probing. The results demonstrate that the proposed methods yield meaningful improvements in model performance, particularly in the context of the cleaned evaluation view. The comparison against existing models, including larger counterparts, highlights the effectiveness of the proposed training stages.
Reproducibility
The paper provides detailed descriptions of the experimental setup, including data sources, training protocols, and evaluation metrics. However, the lack of access to the actual model weights and code may hinder full reproducibility. The authors do release the OmniClean dataset, which aids in replicating the evaluation process.
Limitations
One limitation is the reliance on the specific model architecture (Qwen2.5-Omni-3B) for the experiments, which may not generalize to other architectures or larger models. Additionally, while the visually debiased evaluation is a significant improvement, it does not eliminate all forms of bias in the benchmarks. The paper also acknowledges that the self-distillation results are profile-dependent, indicating variability in effectiveness across different datasets.
Broader impact
The findings have substantial implications for the development of omni-modal language models, as they provide a clearer understanding of model capabilities and limitations. By addressing visual leakage in evaluations, the work encourages the design of more robust benchmarks that can better assess true multi-modal integration. This could lead to advancements in applications requiring comprehensive understanding across audio, visual, and textual modalities. The main contribution of this paper is the introduction of OmniClean, a visually debiased evaluation framework for omni-modal language models, and the demonstration of its effectiveness through a comprehensive staged post-training approach. This work significantly enhances the interpretability of model performance and sets a new standard for evaluating multi-modal capabilities in machine learning.
Author of this paper? Send a correction.