RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
Abstract
We present RT-SEMamba, a fully causal speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.75x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents RT-SEMamba, a causal Mamba-based speech enhancement model with progressive knowledge distillation, offering a competitive quality-latency trade-off for real-time applications.
Methodology
The paper proposes RT-SEMamba, a fully causal speech enhancement model utilizing Time-Frequency Mamba (TF-Mamba) blocks. The core methodological contribution lies in adapting the selective state-space model (SSM) architecture for strict real-time streaming, specifically by ensuring causality in the temporal dimension while maintaining bidirectional frequency modeling. The authors introduce a progressive knowledge distillation (KD) strategy to compress an 8-layer teacher into a 1-layer student. This involves distilling both complex spectral outputs (magnitude, phase, complex spectrum) and intermediate feature representations. The methodology is technically sound and addresses a relevant gap in applying Mamba architectures to streaming audio, where memory efficiency is critical. However, the novelty is somewhat incremental; adapting existing SSM blocks for causality and applying standard KD techniques are well-established practices in the field, though their specific combination and tuning for SE are valuable.
Results & evaluation
The experiments are conducted on the VoiceBank-DEMAND dataset, a standard but relatively small benchmark for SE. The results show that the distilled 1-layer student achieves 3.18 PESQ, outperforming a naive 1-layer baseline (3.06) and approaching the 8-layer teacher (3.32). The paper provides a detailed analysis of the quality-latency trade-off, demonstrating that KD effectively shifts the Pareto frontier. The inclusion of hybrid Mamba-Transformer ablations adds depth to the architectural analysis. However, the evaluation is limited to objective metrics (PESQ, CSIG, etc.) on a single dataset. The lack of subjective listening tests or evaluation on larger, more diverse benchmarks (like DNS Challenge datasets) limits the generalizability of the claims. The comparison with prior works is fair, but the performance gains, while consistent, are modest in absolute terms.
Reproducibility
The paper provides sufficient architectural details, including layer configurations, loss functions, and training hyperparameters (e.g., ramp-up steps). The code is promised to be released on GitHub, which enhances reproducibility. The dataset and evaluation protocol are standard, facilitating independent verification. The description of the streaming inference setup, including buffer management, is clear enough for implementation.
Limitations
The primary limitation is the reliance on the VoiceBank-DEMAND dataset, which may not reflect performance on more challenging, real-world noise conditions or larger-scale training data. The model's performance on unseen, complex acoustic environments is not thoroughly tested. Additionally, the "progressive" nature of the KD is described but the specific scheduling or adaptive mechanisms are standard, limiting the perceived innovation in the distillation strategy itself. The paper does not discuss potential failure modes of the Mamba blocks in handling very long-term dependencies compared to Transformers in the streaming context.
Broader impact
This work contributes to the development of efficient, real-time audio processing systems, which are crucial for accessibility technologies (hearing aids), immersive communications (AR/VR), and edge computing applications. By demonstrating that Mamba-based models can achieve competitive performance with lower latency and memory footprint than Transformers, it encourages the exploration of alternative sequence modeling architectures in audio AI. The open-source release will further facilitate research in this direction. The paper presents RT-SEMamba, a causal Mamba-based speech enhancement model with progressive knowledge distillation, offering a competitive quality-latency trade-off for real-time applications.
Author of this paper? Send a correction.