Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems
Abstract
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
Methodology
The paper introduces Output Divergence Rate (ODR), a metric designed to quantify the semantic impact of speech enhancement (SE) on downstream Large Language Model (LLM) intent classification. The methodology is sound, utilizing a cascaded pipeline of ASR (Whisper large-v3 and wav2vec2-large) followed by LLM classification (Gemini 2.5 Flash Lite). The approach correctly identifies a critical gap in current evaluation practices: standard audio quality metrics (PESQ, STOI) and transcription metrics (WER) do not correlate well with semantic task performance. The use of a closed-set intent classification task (SLURP) provides a controlled environment to measure divergence, although it limits the scope to specific command structures. The decomposition of divergence into "corrections" vs. "regressions" using ground truth labels is a rigorous addition that validates the metric's sensitivity to actual harm rather than random noise.
Results & evaluation
The experiments are comprehensive, covering five distinct enhancement conditions (Noisy, MetricGAN+, Echo, Echo+AEC, Dereverb) across 2,974 clips. The results are striking: MetricGAN+ doubles the ODR compared to unenhanced noisy speech despite improving PESQ, and unmitigated echo causes catastrophic failure (ODR 0.836) due to speaker substitution, a failure mode WER cannot capture. The replication across two architecturally distinct ASR models (attention-based vs. CTC) strengthens the findings, showing that the issue is not model-specific. The analysis of within-condition vs. pooled correlations is particularly valuable, demonstrating that while metrics separate conditions, they fail to predict per-clip divergence, which is crucial for deployment monitoring.
Reproducibility
High. The authors release the full pipeline on GitHub. The use of standard, publicly available datasets (SLURP, DNS Challenge) and models (Whisper, MetricGAN+, Gemini API) ensures that the results are reproducible by other researchers. The prompt used for LLM classification is provided in the appendix, and hyperparameters for ASR and SE models are specified.
Limitations
The study relies on simulated degradations (DNS Challenge) rather than real-world field recordings, which may differ in acoustic complexity. The LLM evaluation is limited to a single model family (Gemini) for the primary results, with only partial replication on a larger model (Gemini Pro). The task is restricted to English and closed-set intent classification, so findings may not generalize to open-ended generation or other languages. ODR requires a clean reference, limiting its use as an online monitor.
Broader impact
This paper has significant implications for the deployment of spoken AI systems. It challenges the common assumption that improving audio quality metrics (like PESQ) leads to better user experience in LLM-based assistants. It provides a new diagnostic tool (ODR) for developers to evaluate SE front-ends specifically for their impact on semantic understanding. The finding that standard metrics are poor predictors of LLM failure urges the field to adopt task-specific evaluation protocols for voice interfaces. The paper introduces Output Divergence Rate (ODR) to measure the semantic impact of speech enhancement on LLM-based voice systems, revealing that standard audio quality metrics are insufficient predictors of downstream task performance. Through rigorous benchmarking on the SLURP dataset using multiple ASR architectures and LLM classifiers, the study demonstrates that perceptual improvements in speech enhancement can paradoxically increase semantic divergence, particularly through failure modes like speaker substitution in echo scenarios that WER cannot capture. This work provides a critical new evaluation framework for the integration of speech processing and large language models, highlighting the need for task-specific metrics in the design of robust spoken AI pipelines.
Author of this paper? Send a correction.