MAGE: A Coarse-to-Fine Speech Enhancer with Masked Generative Model
Abstract
Speech enhancement remains challenging due to the trade-off between efficiency and perceptual quality. In this paper, we introduce MAGE, a Masked Audio Generative Enhancer that advances generative speech enhancement through a compact and robust design. Unlike prior masked generative models with random masking, MAGE employs a scarcity-aware coarse-to-fine masking strategy that prioritizes frequent tokens in early steps and rare tokens in later refinements, improving efficiency and generalization. We also propose a lightweight corrector module that further stabilizes inference by detecting low-confidence predictions and re-masking them for refinement. Built on BigCodec and finetuned from Qwen2.5-0.5B, MAGE is reduced to 200M parameters through selective layer retention. Experiments on DNS Challenge and noisy LibriSpeech show that MAGE achieves state-of-the-art perceptual quality and significantly reduces word error rate for downstream recognition, outperforming larger baselines. Audio examples are available at https://hieugiaosu.github.io/MAGE/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MAGE, a novel Masked Audio Generative Enhancer that utilizes a scarcity-aware coarse-to-fine masking strategy and a lightweight corrector module, achieving state-of-the-art performance in speech enhancement with a significantly reduced model size. This work represents a meaningful advancement in the field of generative speech enhancement, balancing efficiency and perceptual quality, and setting a foundation for future research in practical applications.
Methodology
The methodology presented in this paper is innovative, particularly with the introduction of the scarcity-aware coarse-to-fine masking strategy. This approach addresses the limitations of traditional masked generative models by prioritizing token frequencies, which enhances both efficiency and generalization. The inclusion of a lightweight corrector module for low-confidence predictions is a significant advancement, allowing for iterative refinement of predictions. The architecture is built upon established models like BigCodec and Qwen2.5-0.5B, yet the selective layer retention to achieve a compact model size of 200M parameters is a notable achievement in balancing performance and efficiency.
Results & evaluation
The experimental evaluation is robust, utilizing well-established benchmarks such as the DNS Challenge and noisy LibriSpeech. The results demonstrate that MAGE outperforms larger baselines in terms of perceptual quality and word error rate, which is critical for downstream applications like speech recognition. The paper effectively compares MAGE against both discriminative and generative models, providing comprehensive metrics that highlight its advantages. However, the reliance on simulated distortions raises questions about its real-world applicability.
Reproducibility
The paper provides sufficient details regarding the implementation, including model architecture, training parameters, and evaluation metrics. However, the lack of a publicly available code repository limits full reproducibility. The authors should consider releasing their code to facilitate further research and validation of their findings.
Limitations
While MAGE shows strong results, its performance may be limited by its training on simulated data, which could affect generalization to real-world scenarios. Additionally, the evaluation metrics focus primarily on perceptual quality and WER, potentially overlooking other important aspects of speech enhancement like latency and computational efficiency in practical applications.
Broader impact
The advancements presented in this paper could have significant implications for real-world applications in speech enhancement, particularly in environments with background noise or reverberation. The compact design of MAGE makes it suitable for deployment in resource-constrained settings, which is crucial for applications like mobile devices and real-time communication systems. The potential for future extensions to multilingual and streaming scenarios further enhances its relevance in diverse applications. The main contribution of this paper is the introduction of MAGE, a novel Masked Audio Generative Enhancer that utilizes a scarcity-aware coarse-to-fine masking strategy and a lightweight corrector module, achieving state-of-the-art performance in speech enhancement with a significantly reduced model size. This work represents a meaningful advancement in the field of generative speech enhancement, balancing efficiency and perceptual quality, and setting a foundation for future research in practical applications.
Author of this paper? Send a correction.