Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR
Abstract
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $τ$ that bounds the decoder's lookahead. We show that $τ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $τ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($τ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
Methodology
The paper proposes a two-stage training recipe for streaming ASR using Delayed Streams Modeling (DSM). Stage 1 is standard Supervised Fine-Tuning (SFT) to reproduce the Voxtral Realtime baseline. Stage 2 introduces a Reinforcement Learning (RL) post-training phase using Group Relative Policy Optimization (GRPO). The core methodological contribution is the definition of a new latency metric, AWED (Aligned Word Emission Delay), which measures the time difference between the acoustic end of a word and its emission by the model. This metric is used to construct a joint reward function that balances transcription accuracy (WER) and latency. The use of GRPO is appropriate for the discrete, non-differentiable nature of emission timing decisions. The method effectively decouples the optimization of "what to say" and "when to say it" by leveraging the existing token prediction head, avoiding architectural changes.
Results & evaluation
The experiments are conducted on a large-scale dataset of ~175k hours of English speech. Evaluation is performed on eight diverse benchmarks from the HuggingFace Open ASR Leaderboard. The results demonstrate consistent improvements in the accuracy-latency Pareto frontier compared to the SFT baseline and the original Voxtral Realtime model. Specifically, the model achieves a 30.8% relative WER reduction at tight latency budgets (80ms) and maintains accuracy gains at higher latency budgets while reducing median AWED. Ablation studies on training duration and reward weight provide insight into the hyperparameter sensitivity, confirming the robustness of the chosen configuration.
Reproducibility
The paper provides sufficient detail for reproducibility, including the specific base model (Voxtral-Mini-4B-Realtime), the use of LoRA for parameter efficiency, the data sources, and the evaluation protocol. The definition of AWED is clearly specified. However, the specific implementation details of the GRPO loop (e.g., group size, clipping epsilon) are not fully detailed in the text provided, though standard GRPO practices are implied. The reliance on a specific proprietary checkpoint (Voxtral) may limit reproducibility for those without access to that specific model version, but the method is generalizable.
Limitations
The study is limited to English ASR. The evaluation relies on a specific base model (Voxtral), so the generalizability of the AWED+GRPO approach to other streaming ASR architectures (e.g., RNN-T based or other LALMs) is not tested. The latency metric AWED, while novel, is specific to the DSM paradigm and may not directly apply to other streaming approaches. The paper does not discuss computational overhead of the RL training phase compared to SFT.
Broader impact
This work addresses a critical practical issue in streaming ASR: the gap between nominal structural delay and perceived user latency. By providing a metric (AWED) and an optimization method (GRPO with joint reward) to close this gap, the paper offers a path to more responsive voice assistants and real-time transcription systems. The approach of using RL to optimize non-differentiable aspects of generation (timing) is a valuable technique that could be extended to other tasks involving sequential decision-making with latency constraints. The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
Author of this paper? Send a correction.