UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations
Abstract
Universal speech enhancement (USE) aims to restore speech signals from diverse distortions across multiple sampling rates. We propose UniPASE, an extension of the low-hallucination PASE framework tailored for USE. At its core is DeWavLM-Omni, a unified representation-level enhancement module fine-tuned from WavLM via knowledge distillation on a large-scale supervised multi-distortion dataset. This module directly converts degraded waveforms into clean and linguistically faithful phonetic representations, ensuring robust enhancement with minimal linguistic hallucination. Based on these enhanced phonetic representations, an Adapter generates enhanced acoustic representations containing rich acoustic details, which a neural Vocoder uses to reconstruct corresponding high-fidelity 16-kHz waveforms. A PostNet then converts the waveforms to 48~kHz before resampling them to their original rates, enabling seamless handling of inputs and outputs at multiple sampling rates. Experimental results on several evaluation datasets, covering sub-tasks and full tasks, demonstrate that UniPASE achieves superior or competitive performance compared with existing state-of-the-art models. The proposed model also serves as the backbone of our submission to the URGENT 2026 Challenge, which achieved 1st place in the objective evaluation. The source code and audio demos are available at https://github.com/xiaobin-rong/unipase/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of UniPASE, a generative model that effectively enhances speech across multiple distortions and sampling rates while minimizing hallucinations. This work significantly advances the field of universal speech enhancement by integrating innovative methodologies and demonstrating superior performance against existing state-of-the-art models.
Methodology
The methodology presented in UniPASE is robust and innovative, extending the low-hallucination PASE framework to a universal speech enhancement context. The introduction of DeWavLM-Omni, which utilizes knowledge distillation for phonetic representation enhancement, is a significant advancement. The dual-stream approach, combining phonetic and acoustic representations, effectively addresses the challenges of linguistic and acoustic hallucinations. The explicit acoustic enhancement stage via an Adapter, along with the PostNet for flexible sampling rates, showcases a comprehensive design that addresses multiple distortions and enhances fidelity.
Results & evaluation
The experiments are thorough, utilizing a diverse set of evaluation datasets that cover various speech enhancement tasks. The performance metrics reported, including DNSMOS, UTMOS, and speaker similarity, demonstrate that UniPASE achieves competitive results against state-of-the-art models. The model's performance in the URGENT 2025 Challenge, where it ranked first, further validates its effectiveness. The comprehensive evaluation across different metrics and datasets indicates a rigorous approach to assessing the model's capabilities.
Reproducibility
The paper provides detailed implementation details, including configurations for each module and the training setup. The availability of source code and audio demos on GitHub enhances reproducibility. However, the reliance on specific datasets and configurations may require careful attention from other researchers attempting to replicate the results.
Limitations
While the paper presents a strong model, it may still face challenges in real-world applications where distortions are unpredictable. The performance under extreme noise conditions or in highly variable environments has not been extensively tested. Additionally, the model's complexity may pose challenges for deployment in resource-constrained settings.
Broader impact
The advancements in speech enhancement presented in this paper have significant implications for various applications, including telecommunications, virtual assistants, and accessibility technologies. By improving the fidelity and robustness of speech signals, UniPASE can enhance user experiences in noisy environments and contribute to more effective communication technologies. The main contribution of this paper is the introduction of UniPASE, a generative model that effectively enhances speech across multiple distortions and sampling rates while minimizing hallucinations. This work significantly advances the field of universal speech enhancement by integrating innovative methodologies and demonstrating superior performance against existing state-of-the-art models.
Author of this paper? Send a correction.