Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
Abstract
Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation. However, high agreement with human ratings does not guarantee that their verdicts are grounded in the audio. A judge may instead rely on specialist labels or reference data supplied by the evaluation protocol itself, taking a shortcut in place of listening to the audio. In this paper, we audit such protocol-level ``shortcuts'' in LALM judges across three common deployment protocols: feature-blueprint judging, where the audio is replaced by a structured text description of acoustic features, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, we find that several LALMs rely on protocol-level shortcuts. For example, in feature-blueprint judging, incorrect specialist labels reduce five judges' emotion accuracy to 0.10 or below, and in concatenated A/B comparisons, Qwen3-Omni-Thinking often picks the same slot regardless of order swaps. These results indicate that aggregate agreement can overstate the validity of LALM judges unless the model and the evaluation protocol are assessed jointly, and that each model-protocol pair should be evaluated with a matched shortcut probe.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a significant contribution to the field of audio evaluation by systematically auditing and exposing protocol-level shortcuts in LALM judges, demonstrating that model validity is contingent on the specific evaluation protocol used.
Methodology
The paper proposes a rigorous auditing framework for Large Audio-Language Models (LALMs) acting as judges. It identifies and quantifies three specific "protocol-level shortcuts": specialist-copying in feature-blueprint judging, reference-label following in reference-conditioned judging, and positional locking in pairwise A/B comparisons. The methodology is sound, employing counter-factual probes (e.g., inserting wrong specialist labels, swapping audio order, changing reference placement) to isolate these biases. The approach of treating the judge as a measurement protocol and testing its stability under irrelevant perturbations is a strong conceptual contribution to the evaluation of AI systems.
Results & evaluation
The experiments are comprehensive, covering six diverse LALMs (closed and open-weight) across four speech attributes (emotion, naturalness, language, speaker similarity). The results clearly demonstrate that high aggregate agreement with human ratings can be misleading. For instance, the paper shows that models like GPT-Audio and Qwen3-Omni-Thinking exhibit severe positional biases in A/B testing, while others like Gemini-3-Flash heavily rely on embedded specialist labels even when audio is present. The separation of protocol-level effects from capability-dependent effects (e.g., judges ignoring wrong labels only when they can independently infer the attribute from audio) adds significant depth to the analysis.
Reproducibility
The paper provides detailed descriptions of the protocols, the specific models used, the datasets (RAVDESS, BVCC, FLEURS, VoxCeleb1), and the metrics (accuracy, reference-follow rate, position-lock rate). The definition of the "wrong" labels (fixed permutation) and the prompt structures allows for high reproducibility. The inclusion of seed sweeps and best-of-N checks further supports the robustness of the findings.
Limitations
The study focuses on a limited set of six models and four attributes. While the findings are likely generalizable, the specific magnitude of shortcuts may vary with model size and training data. Additionally, the "feature-blueprint" protocol relies on external specialist classifiers, which introduces a dependency on the quality of those specialists, although the paper attempts to control for this by using "wrong" labels. The study does not explore mitigation strategies, only diagnosis.
Broader impact
This work has significant implications for the reliability of automated evaluation in speech processing. As LALMs become standard tools for benchmarking speech synthesis and recognition systems, understanding their biases is crucial. The findings warn against uncritical reliance on LALM-as-a-judge metrics and provide a checklist of probes that researchers should use to validate their evaluation protocols. This promotes more trustworthy and transparent AI evaluation practices. The paper makes a significant contribution to the field of audio evaluation by systematically auditing and exposing protocol-level shortcuts in LALM judges, demonstrating that model validity is contingent on the specific evaluation protocol used.
Author of this paper? Send a correction.