Where Speech Enhancement Hurts Recognition: An Inference Time Polar Projection Diagnosis
Abstract
Speech enhancement (SE) can substantially improve perceptual quality, yet enhanced speech does not necessarily improve automatic speech recognition (ASR). Existing remedies, such as retraining the enhancer jointly with recognizer or interpolating enhanced speech with the noisy input, can mitigate this mismatch, but common explanations such as artifacts and over-suppression remain qualitative and do not localize which enhancement component harms recognition. We propose inference time polar projection, a diagnosis for STFT domain enhancement. Given a mask $M=Ae^{jφ}$, polar projection forms $M_{α,γ}=A^αe^{jγφ}$, where $α$ controls magnitude strength and $γ$ controls phase correction. Sweeping these controls on frozen SE and ASR models turns ASR degradation into measurable magnitude and phase effects. Our projection analysis shows that magnitude strength is the operative axis, while estimated phase correction provides no recognition benefit. The optimal magnitude strength is recognizer dependent: waveform-input wav2vec2.0 favors strong correction, whereas log-Mel-input, noise-robust Whisper prefers weaker correction. Finally, the projection provides a simple mitigation for any SE front end in the STFT mask domain, without retraining either the enhancer or the recognizer, making it directly useful for voice assistants and agents that rely on enhanced speech.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
Methodology
The paper introduces "Inference Time Polar Projection," a diagnostic framework that factorizes STFT-domain complex masks into independent magnitude and phase controls. By parameterizing the mask as $M_{\alpha,\gamma} = A^\alpha e^{j\gamma\phi}$, the authors decouple the effects of magnitude scaling ($\alpha$) and phase correction ($\gamma$) on Automatic Speech Recognition (ASR) performance. This approach is methodologically sound and offers a rigorous way to dissect the black-box nature of Speech Enhancement (SE) models. The derivation of the effective mask for non-multiplicative models (like MossFormerGAN-SE) is a clever technical addition that extends the applicability of the diagnosis. However, the core concept of sweeping mask parameters is not entirely new; it builds on existing work in mask exponentiation and observation adding, though the specific disentanglement of magnitude vs. phase for ASR diagnosis is a distinct contribution.
Results & evaluation
The experimental evaluation is comprehensive and well-structured. The authors test on the VoiceBank+DEMAND benchmark, which is standard for this domain, using six different SE models (both discriminative and generative) and two distinct ASR backbones (Whisper and wav2vec 2.0). The results clearly demonstrate the SE-ASR mismatch and validate the polar projection hypothesis: magnitude strength is the primary driver of ASR performance, while phase correction provides negligible benefit. The analysis of recognizer dependence (wav2vec 2.0 favoring strong suppression vs. Whisper preferring mild correction) is insightful and aligns with the known robustness differences between waveform-based and feature-based models. The statistical significance testing (paired bootstrap) adds rigor to the claims.
Reproducibility
The paper provides sufficient detail for reproduction. It specifies the datasets (VoiceBank+DEMAND), the specific SE models used (with references to ClearerVoice-Studio implementations), and the ASR models. The mathematical formulation of the polar projection is clear. However, the code for the "effective mask" calculation for MossFormerGAN-SE is described conceptually but not provided in the text, which might require some inference to implement exactly. The use of standard benchmarks and publicly available models enhances reproducibility.
Limitations
The primary limitation is the scope of the evaluation. The experiments are conducted on a single dataset (VoiceBank+DEMAND), which, while standard, may not generalize to more complex real-world conditions with heavy reverberation or diverse noise types not present in the test set. The authors acknowledge this in the limitations section. Additionally, the diagnosis is limited to STFT-domain enhancements; while they extend it to effective masks for other architectures, it may not apply to pure time-domain models that do not operate in the spectral domain. The phase result, while robust for the tested models, is specific to the estimated phase of those enhancers and does not rule out the potential utility of explicitly modeled phase in other contexts.
Broader impact
This work has significant implications for the deployment of SE systems in voice assistants and agents. By showing that phase correction is largely unnecessary for ASR and that magnitude strength is recognizer-dependent, it suggests that SE systems can be simplified or calibrated specifically for machine listening, potentially reducing computational cost and improving robustness. It shifts the paradigm from optimizing for perceptual quality (PESQ, SI-SDR) to optimizing for recognition utility, which is crucial for the next generation of speech-centric AI applications. The paper presents a rigorous diagnostic framework for understanding the mismatch between speech enhancement quality and automatic speech recognition performance, demonstrating that magnitude strength is the critical factor and that phase correction is largely superfluous for recognition tasks.
Author of this paper? Send a correction.