TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation
Abstract
Transformers with global attention capture long-range dependencies but can miss the fine-grained local continuity crucial for speech separation. We propose TF-MossFormer, a time-frequency transformer that combines local and global attention to jointly model short- and long-range contexts for monaural speech separation. At its core is a content-aware sliding-window attention mechanism that dynamically adapts receptive fields for stronger local interactions, avoiding the rigidity of static convolutions. Unlike time-domain chunk-based methods, TF-MossFormer leverages the 2D spectrogram to model structure along both time and frequency axes. Convolutional gating between attention layers further improves feature selection and information flow. TF-MossFormer achieves SI-SDRi of 22.6, 24.0, and 24.4 dB on WSJ0-2Mix with 5.9M, 16.9M, and 25.4M parameters, respectively, outperforming prior approaches.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
Methodology
The paper proposes TF-MossFormer, a time-frequency domain architecture for monaural speech separation. The core innovation lies in integrating a content-aware sliding-window local attention mechanism with global self-attention, augmented by convolutional gating. This hybrid approach aims to balance the computational efficiency and local feature preservation of sliding windows with the long-range dependency modeling of global attention. The architecture follows a dual-path (time-frequency) design, processing spectro-temporal features through stacked blocks. While the combination of local and global attention is not entirely new in the broader transformer literature (e.g., Longformer, BigBird), its specific adaptation to the 2D spectrogram structure with convolutional gating for speech separation is a targeted and reasonable engineering contribution. The use of Conv-SwiGLU and RMSGroupNorm indicates a solid understanding of modern transformer stabilization techniques.
Results & evaluation
The experimental evaluation is conducted on the standard WSJ0-2Mix benchmark. The authors report results for three model scales (Small, Medium, Large) with parameter counts ranging from 5.9M to 25.4M. The results demonstrate state-of-the-art performance (SI-SDRi up to 24.4 dB) compared to prior works like TF-LocoFormer, TF-GridNet, and SepTDA2. The ablation studies provide valuable insights into the hyperparameters (window sizes) and the architectural choices (order of local/global attention, presence of gating). The comparison with SPMamba highlights efficiency gains. However, the evaluation is limited to a single dataset (WSJ0-2Mix), which restricts the generalizability claims. The computational complexity (MACs) is reported, showing competitive efficiency, which is a strong point.
Reproducibility
The paper provides sufficient implementation details, including STFT parameters (16ms window, 8ms hop), optimizer settings (AdamW, learning rate schedule), and loss function (SI-SDR). The architecture diagrams and block descriptions are clear. The use of the ESPnet pipeline is mentioned, which aids reproducibility for those familiar with the framework. However, the specific code repository is not linked in the text provided, and some hyperparameters for the ablation studies (e.g., exact dimension $D$ for the medium/large models) are referenced but not explicitly detailed in the text snippets provided (though implied by parameter counts).
Limitations
The primary limitation is the lack of evaluation on other benchmarks (e.g., LibriMix, WHAM!, or real-world noisy mixtures), which limits the assessment of robustness. The "content-aware" nature of the sliding window is mentioned but the mechanism for dynamic adaptation is not deeply elaborated in the provided text, raising questions about whether it is truly dynamic or just a fixed window with a specific size. The focus on WSJ0-2Mix, while standard, does not demonstrate performance on more challenging, diverse datasets.
Broader impact
This work contributes to the field of audio signal processing, specifically speech separation, which has applications in hearing aids, voice assistants, and audio post-production. By improving the efficiency and quality of TF-domain separation, it supports the development of more accessible and effective speech processing tools. The hybrid attention mechanism could inspire similar architectures in other sequential data domains. TF-MossFormer achieves state-of-the-art performance on WSJ0-2Mix by effectively combining sliding-window local attention with global self-attention and convolutional gating, offering a robust and efficient solution for monaural speech separation.
Author of this paper? Send a correction.