IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention
Abstract
Achieving a balance between lightweight design and high performance remains a significant challenge for speech enhancement (SE) tasks on resource-constrained devices. Existing state-of-the-art methods, such as MUSE, have established a strong baseline with only 0.51M parameters by introducing a Multi-path Enhanced Taylor (MET) transformer and Deformable Embedding (DE). However, an in-depth analysis reveals that MUSE still suffers from efficiency bottlenecks: the MET module relies on a complex "approximate-compensate" mechanism to mitigate the limitations of Taylor-expansion-based attention, while the offset calculation for deformable embedding introduces additional computational burden. This paper proposes IMSE, a systematically optimized and ultra-lightweight network. We introduce two core innovations: 1) Replacing the MET module with Amplitude-Aware Linear Attention (MALA). MALA fundamentally rectifies the "amplitude-ignoring" problem in linear attention by explicitly preserving the norm information of query vectors in the attention calculation, achieving efficient global modeling without an auxiliary compensation branch. 2) Replacing the DE module with Inception Depthwise Convolution (IDConv). IDConv borrows the Inception concept, decomposing large-kernel operations into efficient parallel branches (square, horizontal, and vertical strips), thereby capturing spectrogram features with extremely low parameter redundancy. Extensive experiments on the VoiceBank+DEMAND dataset demonstrate that, compared to the MUSE baseline, IMSE significantly reduces the parameter count by 16.8\% (from 0.513M to 0.427M) while achieving competitive performance comparable to the state-of-the-art on the PESQ metric (3.373). This study sets a new benchmark for the trade-off between model size and speech quality in ultra-lightweight speech enhancement.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents IMSE, an ultra-lightweight speech enhancement network that optimizes existing models by introducing MALA and IDConv, achieving a notable balance between model size and performance. This work contributes to the ongoing efforts to develop efficient deep learning models for speech enhancement, addressing critical challenges in the field.
Methodology
The paper introduces two significant innovations: Amplitude-Aware Linear Attention (MALA) and Inception Depthwise Convolution (IDConv). MALA addresses the limitations of traditional linear attention by reintroducing amplitude information, which is crucial for maintaining the sharpness of attention distributions. This is a notable improvement over existing methods, as it allows for efficient global modeling without the need for additional compensation branches. IDConv, on the other hand, leverages the Inception architecture to efficiently capture spectrogram features through a multi-branch approach, which reduces parameter redundancy while maintaining performance. The methodology is well-structured and clearly articulated, demonstrating a solid understanding of the challenges in speech enhancement.
Results & evaluation
The experiments are conducted on the VoiceBank+DEMAND dataset, a well-established benchmark in the field of speech enhancement. The results show that IMSE achieves a competitive PESQ score of 3.373 with a reduced parameter count of 0.427M, outperforming the baseline MUSE model and other lightweight models. The ablation studies provide valuable insights into the effectiveness of the proposed components, confirming the contributions of both MALA and IDConv. However, the paper could benefit from more extensive comparisons with a broader range of state-of-the-art methods to further validate the claims.
Reproducibility
The paper provides a GitHub link to the open-source code, which is a positive aspect for reproducibility. However, it lacks detailed implementation specifics, such as the exact training configurations and hyperparameters used, which could hinder replication efforts by other researchers.
Limitations
While the proposed model shows promising results, it is still limited to the VoiceBank+DEMAND dataset. The generalizability of the model to other datasets or real-world scenarios remains to be tested. Additionally, the paper does not address potential limitations in terms of computational efficiency during inference on actual low-power devices, which is crucial for deployment.
Broader impact
The advancements presented in this paper have significant implications for real-time speech enhancement applications, particularly in resource-constrained environments such as mobile devices and hearing aids. By improving the efficiency of speech enhancement models, this research could enhance communication technologies, making them more accessible and effective in noisy environments. The paper presents IMSE, an ultra-lightweight speech enhancement network that optimizes existing models by introducing MALA and IDConv, achieving a notable balance between model size and performance. This work contributes to the ongoing efforts to develop efficient deep learning models for speech enhancement, addressing critical challenges in the field.
Author of this paper? Send a correction.