TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation
Abstract
Recent advances in speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[This paper presents TF-MoE, a sparse Mixture-of-Experts framework that enhances speech separation performance by adding capacity without increasing computational cost through dual-dimension expert routing.] The authors effectively demonstrate that integrating sparse MoE layers into a Conformer-based speech separation model allows for significant performance gains (+1.3 dB over a strong Conformer baseline) while maintaining a lightweight footprint suitable for edge deployment. The technical contribution is solid, leveraging established MoE principles in a novel audio-specific architecture, and the experimental results on Libri2Mix are compelling for the low-compute regime. However, the novelty is incremental relative to the broader MoE literature, and the evaluation is somewhat narrow in scope.
Methodology
The paper proposes TF-MoE, a sparse Mixture-of-Experts framework for speech separation. The core innovation lies in applying sparse expert routing in both time and frequency dimensions within a Conformer-based backbone. Specifically, it replaces standard Feed-Forward Networks (FFNs) in Time-Conformer and Frequency-Conformer blocks with MoE-FFNs. The authors argue that this allows for capacity scaling without increasing computational cost (GMACs), as only a subset of experts (top-1) is activated per token. The approach is technically sound and leverages established MoE mechanisms (gating, load balancing loss) in a novel domain-specific context (audio spectrograms). However, the novelty is somewhat incremental; applying MoE to replace FFNs in sequence modeling tasks is a known pattern (e.g., Switch Transformer, GLaM), and adapting it to speech separation is a logical but not radically new architectural shift. The specific contribution of "dual-dimension" routing is the key differentiator, offering a more granular specialization than temporal-only MoE.
Results & evaluation
The experimental evaluation is conducted on the Libri2Mix dataset, a standard benchmark for speech separation. The primary metric is Signal-to-Distortion Ratio (SDR). The results show that the proposed TF-MoE outperforms the baseline BSRNN by +3.8 dB SDR and the TF-Conformer backbone by +1.3 dB SDR, all while maintaining a very low computational cost of ~4.1 GMACs/s. This is a significant performance gain for such a constrained compute budget. The ablation studies are thorough, validating the contributions of the Conformer backbone, the MoE mechanism, and the number of experts. The interpretability analysis (visualizing gating policies) adds value by showing that experts do specialize by frequency bands and temporal segments, which supports the architectural design choices. However, the comparison is limited to models with similarly low computational costs. Comparisons against high-performance, high-compute models (like TF-GridNet) are mentioned only to highlight the efficiency gap, not to show competitive performance at equal compute, which is a common limitation in efficiency-focused papers.
Reproducibility
The paper provides sufficient detail regarding the architecture (Conformer blocks, MoE structure, gating mechanism), hyperparameters (hidden dimension 32, 6 blocks, 12 experts, top-1 routing), and training setup (AdamW, SI-SNR loss, Libri2Mix). The computational cost analysis is explicit. However, the code is not linked, and some implementation details of the "mel-band-splitting" and the specific Conformer variant (macaron style) could be clearer. The use of generative AI for polishing is disclosed, which is good practice but does not affect reproducibility of the scientific content.
Limitations
The paper does not report results on other speech separation benchmarks (e.g., WSJ0-2mix, Libri3Mix) which would strengthen the generalizability claims. The performance drop when increasing experts to 24 suggests that training stability with MoE in this specific audio context can be challenging, a limitation not fully explored. The comparison is strictly limited to low-compute models; it does not demonstrate if TF-MoE can scale up to compete with larger models if compute constraints were relaxed. The real-time factor (RTF) is mentioned as a metric but specific values are not provided in the text snippets, only implied to be efficient.
Broader impact
This work has significant potential impact for deploying speech separation on edge devices (smartphones, IoT devices) where power and compute are limited. By decoupling parameter count from computational cost, it offers a pathway to higher quality audio processing in resource-constrained environments. This aligns with the broader trend of efficient AI and on-device machine learning. [This paper presents TF-MoE, a sparse Mixture-of-Experts framework that enhances speech separation performance by adding capacity without increasing computational cost through dual-dimension expert routing.] The authors effectively demonstrate that integrating sparse MoE layers into a Conformer-based speech separation model allows for significant performance gains (+1.3 dB over a strong Conformer baseline) while maintaining a lightweight footprint suitable for edge deployment. The technical contribution is solid, leveraging established MoE principles in a novel audio-specific architecture, and the experimental results on Libri2Mix are compelling for the low-compute regime. However, the novelty is incremental relative to the broader MoE literature, and the evaluation is somewhat narrow in scope.
Author of this paper? Send a correction.