The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?
Abstract
Current speech LLMs largely perform implicit ASR: on tasks solvable from a transcript, they are behaviorally and mechanistically equivalent to simple Whisper$\to$LLM cascades. We show this through matched-backbone testing across four speech LLMs and six tasks, controlling for the LLM backbone for the first time. Ultravox is statistically indistinguishable from its matched cascade ($κ{=}0.93$); logit lens reveals literal text emerging in hidden states; LEACE concept erasure confirms text representations are causally necessary in both architectures tested, collapsing accuracy to near-zero. Qwen2-Audio genuinely diverges, revealing cascade equivalence is architecture-dependent, not universal. For most deployed use cases, current speech LLMs are expensive cascades, and under noise, they are worse ones, with clean-condition advantages reversing by up to 7.6% at 0 dB.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a systematic investigation into the equivalence of speech LLMs and ASR-LLM cascades, revealing critical insights into their operational similarities and differences. This work contributes to the understanding of model architectures in speech processing and provides a framework for evaluating their effectiveness in various contexts.
Methodology
The paper introduces a novel methodology termed "matched-backbone behavioral testing" that effectively isolates architectural effects from backbone effects in evaluating speech LLMs. This approach is significant as it allows for a more accurate comparison between end-to-end speech models and traditional ASR-LLM cascades, addressing a critical gap in prior research. The use of per-example behavioral metrics rather than aggregate accuracy is a strong methodological choice that enhances the robustness of the findings.
Results & evaluation
The experiments are well-structured, involving a comprehensive evaluation of four speech LLMs and their corresponding ASR-LLM cascades across six tasks. The use of various behavioral metrics, including Cohen's kappa and conditional error overlap, provides a nuanced understanding of model performance. The inclusion of noise perturbation conditions adds practical relevance to the findings, highlighting the robustness of the models under real-world conditions.
Reproducibility
While the methodology is detailed, the paper lacks explicit information on the availability of the datasets and code used for the experiments, which raises concerns about reproducibility. The absence of a project URL or demo links further complicates the ability for other researchers to replicate the study.
Limitations
The paper acknowledges limitations, such as the potential underestimation of acoustic surplus due to the use of TTS-synthesized speech and the focus on only two of the four evaluated models for mechanistic analysis. These factors may limit the generalizability of the findings across different architectures and real-world applications.
Broader impact
The implications of this research are significant for both academic and industrial applications in speech processing. By demonstrating that current speech LLMs often behave like expensive cascades, the findings encourage practitioners to reconsider their model choices based on task requirements and operational conditions. The insights gained could lead to more efficient model designs and deployment strategies in speech technology. The paper presents a systematic investigation into the equivalence of speech LLMs and ASR-LLM cascades, revealing critical insights into their operational similarities and differences. This work contributes to the understanding of model architectures in speech processing and provides a framework for evaluating their effectiveness in various contexts.
Author of this paper? Send a correction.