REDnet: Recursive Encoder and Decoder for Speech Separation under Unknown Number of Speakers and Variable Number of Microphones
Abstract
We propose $\textit{recursive encoder and decoder}$ (RED) for building a single deep neural network (DNN) model that can separate multi-speaker mixtures containing unknown numbers of speakers and variable numbers of microphones arranged in an unknown geometry, a task that has not been studied yet. The decoder of RED recursively detects whether there are active speakers left and separates one speaker at a time. It is designed to be trained in an end-to-end fashion to improve separation performance. The encoder of RED recursively encodes each microphone channel of the input mixture, sequentially incorporating spatial cues. Combining both, the DNN can be trained to separate mixtures not only with unknown numbers of speakers but also with variable numbers of microphones, achieving state-of-the-art performance on multiple public datasets.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
Methodology
The paper proposes REDnet, a recursive encoder-decoder architecture for speech separation. The core novelty lies in handling two variable dimensions simultaneously: the number of microphones (via a recursive encoder that aggregates channel features sequentially) and the number of speakers (via a recursive decoder that peels off one speaker at a time using a gating mechanism and a speaker detector). The methodology is technically sound, building upon established blocks (TF-GridNet) and concepts (Permutation Invariant Training, recursive separation). The recursive encoder design effectively addresses the variable microphone input problem by avoiding fixed-size pooling or explicit pairwise attention, offering a linear complexity solution. The recursive decoder's use of a gating mechanism to update the residual representation is a logical extension of prior recursive separation works, adapted for end-to-end training with specific loss functions to encourage orthogonality between separated speakers and residuals. While the combination is novel, the individual components are incremental improvements on existing paradigms (TAC for variable mics, OR-PIT/SepTDA for unknown speakers).
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors decompose the problem to validate the Recursive Encoder (RE) and Recursive Decoder (RD) separately before presenting the full REDnet. They use standard public datasets (WSJ0, WHAM!, CHiME-4) and compare against strong baselines (TAC, FlexIO, MUSE, SR-CorrNet). The results show state-of-the-art performance on multiple metrics (SI-SDR, SDR, PESQ, STOI) across various configurations (variable mics, unknown speakers, noisy/reverberant conditions). The ablation studies on the RDnet component are particularly helpful in isolating the contributions of the gating mechanism, speaker interaction module, and loss functions. The evaluation on real-recorded signals (DNSMOS) adds practical value. The comparison with FlexIO is fair, noting that FlexIO assumes known speaker counts, whereas REDnet does not, making REDnet's performance gains even more significant.
Reproducibility
The paper provides detailed architectural descriptions, including hyperparameters for the TF-GridNet blocks, optimizer settings, and loss weights. The dataset construction and preprocessing steps are clearly defined. The use of standard datasets and metrics facilitates reproduction. However, the specific implementation details of the "modified TF-GridNet blocks" (e.g., exact SwiGLU placement, kernel sizes) are described but might require careful reading to implement exactly. The code is not linked, which is a minor hindrance to immediate reproducibility, but the description is sufficient for a competent researcher.
Limitations
The primary limitation is that the model's performance depends on the accuracy of the speaker detector in the recursive decoder. If the detector fails to stop (false positive) or stops too early (false negative), performance degrades. The paper reports high counting accuracy, but this is on controlled datasets. In highly noisy or reverberant real-world scenarios, detector robustness might be a challenge. Additionally, the recursive nature of the decoder implies that errors can propagate if the gating mechanism does not perfectly isolate the speaker, although the residual update mechanism mitigates this. The model assumes that the number of microphones is known at inference time (to process the input), which is standard, but the "unknown geometry" aspect is handled by the channel-agnostic aggregation, which is a strength but relies on the assumption that spatial cues are sufficiently captured by the recursive encoding.
Broader impact
This work contributes to the development of more robust and flexible audio processing systems, which are crucial for real-world applications like hearing aids, smart speakers, and teleconferencing systems where the number of speakers and microphones can vary. By unifying these capabilities into a single model, it reduces the need for multiple specialized models and simplifies deployment. The recursive approach is generalizable to other source separation tasks (e.g., music, sound event detection) and potentially to other modalities where variable input/output dimensions are a challenge. REDnet is the first single DNN model to jointly handle speech separation with unknown numbers of speakers and variable numbers of microphones in unknown geometries, achieving state-of-the-art performance through recursive encoding and decoding mechanisms. The paper presents a significant technical contribution to the field of audio source separation by addressing two critical practical challenges simultaneously: variable microphone arrays and unknown speaker counts. The proposed REDnet architecture is innovative in its use of recursive mechanisms for both encoding spatial information from variable microphone inputs and decoding separated sources from a residual mixture. The recursive encoder effectively aggregates channel-specific features without the computational overhead of pairwise attention or the information loss of global averaging. The recursive decoder, enhanced with a gating mechanism and speaker interaction modules, allows for end-to-end training of a variable-output system. The comprehensive experimental evaluation on multiple public datasets demonstrates that this unified approach outperforms existing specialized models, highlighting its potential for real-world deployment in dynamic acoustic environments.
Author of this paper? Send a correction.