StuPASE: Towards Low-Hallucination Studio-Quality Generative Speech Enhancement
Abstract
Achieving high perceptual quality without hallucination remains a challenge in generative speech enhancement (SE). A representative approach, PASE, is robust to hallucination but has limited perceptual quality under adverse conditions. We propose StuPASE, built upon PASE to achieve studio-level quality while retaining its low-hallucination property. First, we show that finetuning PASE with dry targets rather than targets containing simulated early reflections substantially improves dereverberation. Second, to address performance limitations under strong additive noise, we replace the GAN-based generative module in PASE with a flow-matching module, enabling studio-quality generation even under highly challenging conditions. Experiments demonstrate that StuPASE consistently produces perceptually high-quality speech while maintaining low hallucination, outperforming state-of-the-art SE methods. Audio demos are available at: https://xiaobin-rong.github.io/stupase_demo/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of StuPASE, a generative speech enhancement framework that achieves studio-quality output while maintaining low hallucination through innovative methodologies. The technical contributions, particularly the integration of flow-matching and dry-target finetuning, represent a meaningful advancement in the field of generative audio processing, addressing critical challenges in speech enhancement.
Methodology
The paper introduces a novel approach to generative speech enhancement (SE) by building on the existing PASE framework. The authors effectively leverage dry-target finetuning to improve dereverberation and replace the GAN-based generative module with a flow-matching module, which enhances the quality of generated speech under adverse conditions. The methodology is well-structured, with clear explanations of the components involved, including the semantic enhancement module and the acoustic enhancement module. The use of flow-matching for high-fidelity generation is a significant advancement over traditional GAN approaches, which are known to struggle with artifacts and noise suppression.
Results & evaluation
The experiments are comprehensive, utilizing a substantial dataset of around 2,000 hours of clean speech and various noise types. The evaluation metrics are robust, encompassing both objective measures (DNSMOS, UTMOS, etc.) and subjective assessments (Q-MOS, S-MOS). The results demonstrate that StuPASE consistently outperforms state-of-the-art methods, showcasing significant improvements in perceptual quality and linguistic integrity. The ablation studies further validate the contributions of dry-target finetuning and the flow-matching module, reinforcing the effectiveness of the proposed approach.
Reproducibility
The paper provides detailed implementation details, including training protocols, dataset descriptions, and evaluation metrics. However, the lack of a publicly available code repository limits reproducibility. While the methodology is described thoroughly, access to the actual implementation would enhance the ability of other researchers to replicate the results.
Limitations
One limitation is the reliance on specific datasets, which may not generalize across all speech enhancement scenarios. Additionally, while the paper demonstrates improvements in perceptual quality, it does not extensively explore the computational efficiency of the new model compared to its predecessors. The subjective evaluation is based on a limited number of samples, which may not capture the full variability of real-world applications.
Broader impact
The advancements presented in this paper have significant implications for applications in speech enhancement, particularly in environments with challenging acoustic conditions. The ability to produce high-quality speech with low hallucination can benefit various fields, including telecommunications, assistive technologies, and media production. The findings could pave the way for further research into generative models in audio processing, potentially influencing future developments in AI-driven audio technologies. The main contribution of this paper is the introduction of StuPASE, a generative speech enhancement framework that achieves studio-quality output while maintaining low hallucination through innovative methodologies. The technical contributions, particularly the integration of flow-matching and dry-target finetuning, represent a meaningful advancement in the field of generative audio processing, addressing critical challenges in speech enhancement.
Author of this paper? Send a correction.