Reducing Linguistic Hallucination in LM-Based Speech Enhancement via Noise-Invariant Acoustic-Semantic Distillation
Abstract
Language model (LM)-based speech enhancement (SE) can generate natural-sounding speech, but under severe noise it often suffers from unreliable conditioning, leading to perceptually plausible yet linguistically incorrect outputs. To address this issue, we propose L3-SE, a noise-invariant acoustic-semantic distillation framework for reducing linguistic hallucination in LM-based SE. The proposed method learns a noise-invariant conditioning encoder from noisy speech by jointly distilling two complementary clean-speech targets: an acoustic target for reconstruction fidelity and a semantic target for linguistic consistency. The resulting noise-invariant acoustic-semantic representations are used to condition a decoder-only autoregressive language model, which predicts clean acoustic tokens that are decoded into enhanced speech. To support high-quality generation, we further employ a high-fidelity codec built on learnable weighted WavLM layer representations as the discrete acoustic interface. By improving the reliability of conditioning under adverse conditions, the proposed framework substantially reduces hallucination and improves content faithfulness. Experiments show that the proposed method consistently outperforms prior LM-based speech enhancement baselines on linguistic consistency metrics, with especially clear gains under low-SNR and reverberant conditions, while maintaining competitive perceptual quality. Audio samples are available at https://max1wz.github.io/L3-SE-Demo-Page/. The complete source code will be released after the manuscript is accepted.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents L3-SE, a novel framework for reducing linguistic hallucination in LM-based speech enhancement through noise-invariant acoustic-semantic distillation, demonstrating significant improvements in linguistic consistency and perceptual quality under challenging conditions.
Methodology
The proposed L3-SE framework introduces a novel approach to speech enhancement by utilizing a noise-invariant acoustic-semantic distillation strategy. This dual-target distillation method, which leverages both acoustic fidelity and semantic consistency, is innovative in addressing the issue of linguistic hallucination in generative speech models. The architecture effectively combines a shared backbone with task-specific heads, allowing for robust conditioning that enhances the model's performance under noisy conditions. The integration of a high-fidelity codec further supports the quality of the generated speech, making the methodology both comprehensive and well-structured.
Results & evaluation
The experiments are thorough, utilizing a variety of datasets and evaluation metrics that cover perceptual quality, linguistic consistency, and speaker preservation. The results demonstrate that L3-SE outperforms existing baselines, particularly in challenging conditions such as low-SNR and reverberation. The use of both objective and subjective metrics strengthens the evaluation, providing a well-rounded assessment of the framework's capabilities.
Reproducibility
The paper mentions that the complete source code will be released upon acceptance, which is a positive aspect for reproducibility. However, the implementation details are somewhat dense, and while they provide a comprehensive overview of the training process, clearer guidelines or supplementary materials could enhance reproducibility further.
Limitations
One limitation is the reliance on specific datasets for training and evaluation, which may affect generalizability to other speech enhancement scenarios. Additionally, while the framework shows improvements in linguistic consistency, the perceptual quality metrics could still be further optimized to match or exceed the best-performing models in all conditions.
Broader impact
The proposed framework has significant implications for applications in speech recognition, communication technologies, and assistive devices, where clarity and accuracy in speech are crucial. By addressing linguistic hallucination effectively, it could enhance user experience in various real-world applications, making it a valuable contribution to the field of audio processing and machine learning. The paper presents L3-SE, a novel framework for reducing linguistic hallucination in LM-based speech enhancement through noise-invariant acoustic-semantic distillation, demonstrating significant improvements in linguistic consistency and perceptual quality under challenging conditions.
Author of this paper? Send a correction.