Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition
Abstract
The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and frame-level attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel framework for robust whispered speech recognition by integrating self-supervised uncertainty perception into an Audio-LLM, achieving state-of-the-art results and significantly reducing hallucinations.
Methodology
The paper proposes a "Whisper-Aware LLM" framework that integrates an Uncertainty Perception Module (UPM) into an Audio-LLM (Qwen2-Audio). The core innovation lies in using self-supervised tasks—F0 contour prediction and masked spectrum reconstruction—to quantify signal uncertainty. This uncertainty is then operationalized via a "Confidence-Fused Decoding" mechanism that injects global instruction embeddings and frame-level attention biases into the LLM decoder. The methodology is technically sound and addresses a specific, well-defined problem (whispered speech recognition) with a plausible mechanism for handling signal ambiguity. The use of physics-informed auxiliary tasks (F0 absence) to drive uncertainty estimation is a novel angle compared to standard confidence calibration methods. However, the integration of uncertainty into the attention mechanism via additive bias is a relatively simple modification, and the "self-awareness" claim is somewhat marketing-heavy for what is essentially a multi-task learning setup with a specific decoding strategy.
Results & evaluation
The experimental evaluation is comprehensive, covering whispered speech (AISHELL6-Whisper, wTIMIT), general ASR (AISHELL-1, LibriSpeech), and a novel "Noise Hallucination Set" to evaluate reliability. The results show significant improvements in CER/WER on whispered speech and a drastic reduction in hallucination rates (from >25% to 4.5%). The ablation studies support the contribution of the UPM and the decoding mechanism. The inclusion of a hallucination metric is a strong point, addressing a critical gap in current ASR evaluations. The comparison against strong baselines (Whisper-v3, Qwen2-Audio, Seed-ASR) is appropriate. The 17% relative CER reduction is a substantial empirical gain.
Reproducibility
The paper provides detailed implementation details, including the base model (Qwen2-Audio), UPM architecture (1D-CNN + Transformer), training stages, and hyperparameters (LoRA rank, learning rates). The dataset construction is described, though the "pure noise" collection is less defined. The three-stage training strategy is clearly outlined. However, the code is not released, and some details regarding the "Noise Hallucination Set" creation (source of non-speech sounds) are vague. Reproducibility is moderate to high, assuming access to the base Qwen2-Audio weights and the described datasets.
Limitations
The paper does not extensively discuss the computational overhead of the UPM or the decoding mechanism during inference. The generalization of the UPM to other types of speech degradation (e.g., heavy noise, reverberation) beyond whispering is not thoroughly explored, although some general ASR results are provided. The reliance on F0 prediction for uncertainty might be less effective for non-tonal languages or speech with very low pitch, though the masked spectrum task helps mitigate this. The "hallucination" metric, while useful, is specific to the authors' curated dataset; broader standard metrics for hallucination in ASR are still evolving.
Broader impact
This work has significant implications for robust speech recognition systems, particularly in low-resource or noisy environments where whispered speech might occur (e.g., libraries, hospitals, late-night settings). By addressing the accuracy-reliability trade-off, it contributes to the development of more trustworthy AI systems. The methodology of using self-supervised uncertainty perception for decoding control could be adapted to other modalities or tasks requiring robustness to signal degradation. The paper presents a novel framework for robust whispered speech recognition by integrating self-supervised uncertainty perception into an Audio-LLM, achieving state-of-the-art results and significantly reducing hallucinations.
Author of this paper? Send a correction.