Training-Free Speech-Centric Omni Understanding with Frozen VLMs
Abstract
Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a training-free framework for converting frozen VLMs into speech-centric omni models by routing Whisper transcripts through the language interface. This approach demonstrates competitive performance on audio-visual benchmarks while preserving the VLM's original visual and reasoning capabilities, offering a cost-effective alternative to native omni training.
Methodology
The paper proposes Training-Free Omni (TFO), a framework that converts frozen Vision-Language Models (VLMs) into speech-centric omni models without architectural changes or multimodal re-alignment. The core methodology relies on using Whisper to extract confidence-filtered, timestamped transcripts from audio inputs, which are then routed through the VLM's existing language interface. This approach effectively decouples the audio processing from the visual reasoning backbone, preserving the VLM's original capabilities while adding speech understanding. The method is simple, modular, and avoids the high computational cost and potential capability drift associated with native omni training.
Results & evaluation
The experimental evaluation is extensive, covering 56 benchmarks and 21 languages across five matched model comparisons (Qwen2.5, MiniCPM, VILA, Qwen3). The results demonstrate that TFO is competitive with native omni models on audio-visual understanding tasks, particularly those relying on spoken content and temporal alignment. Furthermore, TFO generally preserves stronger image/video understanding, coding, and mathematical reasoning capabilities compared to native omni checkpoints, validating the hypothesis that freezing the backbone prevents capability degradation. However, the paper acknowledges limitations in handling non-speech acoustic cues like music or environmental sounds.
Reproducibility
The paper provides a clear description of the pipeline and references a GitHub repository (OmniEvalKit) for code and evaluation tools. The use of standard components like Whisper and CosyVoice3 enhances reproducibility. The confidence threshold (0.65) is specified, and the prompt formats are described, allowing for reasonable reproduction of the results.
Limitations
The primary limitation is the reliance on ASR accuracy; TFO cannot capture non-speech acoustic information (e.g., tone, music, environmental sounds) that requires rich acoustic representations rather than linguistic transcripts. Additionally, the approach introduces ASR latency, which may be a bottleneck for real-time applications. The method is also limited to speech-centric tasks and does not address general audio understanding.
Broader impact
This work offers a practical and cost-effective alternative to training native omni models, which is significant for the community as VLM backbones continue to evolve. By demonstrating that modular audio-to-language routing can achieve competitive performance while preserving backbone capabilities, the paper provides a valuable control for evaluating the necessity of native omni training. It also highlights the trade-offs between acquiring new modalities and preserving existing ones, guiding future research in multimodal model design. The paper introduces a training-free framework for converting frozen VLMs into speech-centric omni models by routing Whisper transcripts through the language interface. This approach demonstrates competitive performance on audio-visual benchmarks while preserving the VLM's original visual and reasoning capabilities, offering a cost-effective alternative to native omni training.
Author of this paper? Send a correction.