Listen, Do Not Copy: Internalizing Audio-Grounded Scaffold Context for Robust Omni-Model Speech Understanding
Abstract
Omni models transcribe clean, single-speaker speech well, but their accuracy drops sharply when speakers overlap and the scene is noisy, exactly where knowing who said what matters most. A natural fix is a short scene description. We show why this is risky: answer-bearing text lets the model copy instead of listen, so the score rises although nothing has been heard; a silent test exposes this shortcut at once. We call this failure mode perception bypass and address it with Audio-Grounded Scaffold Context (AGSC). AGSC links three steps: first, we build clues from audio to guide listening without giving the answer; second, answer-overlap and silence tests probe them for leakage and audio dependence; finally, those clues scaffold training but vanish at test time, yielding no-clue capability. Across three heterogeneous Omni models, training on AGSC lowers no-clue capped mean permutation word error rate (mpWER) on overlapping, noisy speech from 25%-71% to 9%-15%. For streaming control, we formulate a joint GDPO task in which the model learns when to use a clue and how to produce a speaker-attributed transcript from separately normalized format, gate, and transcript rewards. After internalization, AGSC adds almost no inference overhead.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
Methodology
The paper proposes Audio-Grounded Scaffold Context (AGSC), a framework designed to mitigate "perception bypass" in Omni models—where models rely on textual context rather than audio signals. The methodology involves synthesizing "clues" (partial, shuffled words, time windows) from audio using off-the-shelf front-ends (diarization, ASR, VAD). These clues are used during Supervised Fine-Tuning (SFT) and Group Reward-Decoupled Policy Optimization (GDPO). The core innovation lies in the training protocol: using a "silent test" (silencing the audio input) to detect if the model is copying text, and ensuring that performance gains persist even when the scaffold clues are removed at inference time (internalization). The GDPO component introduces a novel reward structure with separate normalization for format, gate, and transcript rewards, and a conditional reward mechanism to prevent gate collapse. While the concept of using scaffolds is not entirely new, the specific application to preventing perception bypass in Omni models and the rigorous "silent test" evaluation metric constitute a meaningful methodological contribution to robust multimodal training.
Results & evaluation
The authors evaluate their approach on three heterogeneous Omni models (Qwen3-Omni, MiniCPM-o, Ming-flash-omni) using a custom benchmark, Context-Speech Bench (CSB), which includes overlapping and noisy speech scenarios. The results show significant improvements in mean permutation Word Error Rate (mpWER) for the smaller models (Ming, MiniCPM) after SFT, reducing errors from ~25-68% to ~8-15%. The paper provides a thorough ablation study, including the "silent test" to prove that gains are not due to text copying. The GDPO experiments demonstrate the ability to learn gating strategies, although the full-chain joint training shows some instability (gate collapse) which is addressed with a secondary supervised step. The evaluation is comprehensive, covering both offline transcription and streaming control, and the use of multiple models strengthens the generalizability claims.
Reproducibility
The paper states that code, benchmark manifests, and result evidence will be open-sourced upon acceptance. The methodology describes the pipeline components (Pyannote, Silero, Sepformer) and the training hyperparameters (LoRA ranks, learning rates). However, the reliance on specific versions of large base models (Qwen3, MiniCPM, Ming) and the complex synthesis pipeline for clues may pose reproducibility challenges. The description of the GDPO reward normalization and the specific "silent test" protocol is detailed enough to allow replication by researchers familiar with RLHF techniques.
Limitations
The paper acknowledges that the external detector for gating has lower performance than the learned gate but notes the learned gate can suffer from collapse. The improvements are less pronounced for the strongest baseline model (Qwen3), suggesting diminishing returns for already high-performing systems. The "clue" generation relies on imperfect off-the-shelf tools, which might introduce noise or errors that the model must learn to ignore, potentially limiting the ceiling of performance. Furthermore, the "silent test" is an aggregate control and may not catch all forms of shortcut learning. The paper also notes that unconditional clue usage at test time can hurt performance, highlighting the necessity of the learned gate.
Broader impact
This work addresses a critical failure mode in multimodal AI systems, enhancing their reliability in real-world scenarios like meetings and call centers where audio is often noisy or overlapping. By preventing "perception bypass," the method ensures that models are genuinely listening, which is crucial for trust and safety in automated transcription and assistance systems. The Context-Speech Bench provides a valuable resource for evaluating robustness in challenging acoustic conditions. This paper introduces AGSC, a training framework that uses audio-grounded scaffolds and a novel "silent test" evaluation to prevent perception bypass in Omni models, significantly improving robustness in overlapping and noisy speech scenarios through internalized learning.
Author of this paper? Send a correction.