Towards Quantifying Benchmark Optimization in ASR Models
Abstract
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper provides a critical diagnostic analysis of benchmark contamination in Automatic Speech Recognition models, introducing novel behavioral probes to quantify how models prioritize reference transcripts over acoustic evidence. The work is highly relevant to the current landscape of LLM-audio integration and evaluation, as it addresses a fundamental validity issue in model assessment. The technical contribution is strong, offering mechanistic insights and causal manipulation techniques that advance the field's understanding of model generalization versus benchmark overfitting. While not a new generative architecture or a primary quality metric, it serves as a high-value evaluation study that will likely influence future benchmark design and model training practices.
Author of this paper? Send a correction.