Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation
Abstract
Speech separation in realistic acoustic environments remains challenging because overlapping speakers, background noise, and reverberation must be resolved simultaneously. Although recent time-frequency (TF) domain models have shown strong performance, most still rely on late-split architectures, where speaker disentanglement is deferred to the final stage, creating an information bottleneck and weakening discriminability under adverse conditions. To address this issue, we propose SR-CorrNet, an asymmetric encoder-decoder framework that introduces the separation-reconstruction (SepRe) strategy into a TF dual-path backbone. The encoder performs coarse separation from mixture observations, while the weight-shared decoder progressively reconstructs speaker-discriminative features with cross-speaker interaction, enabling stage-wise refinement. To complement this architecture, we formulate speech separation as a structured correlation-to-filter problem: spatio-spectro-temporal correlations computed from the observations are used as input features, and the corresponding deep filters are estimated to recover target signals. We further incorporate an attractor-based dynamic split module to adapt the number of output streams to the actual speaker configuration. Experimental results on WSJ0-2/3/4/5Mix, WHAMR!, and LibriCSS demonstrate consistent improvements across anechoic, noisy-reverberant, and real-recorded conditions in both single- and multi-channel settings, highlighting the effectiveness of TF-domain SepRe with correlation-based filter estimation for speech separation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of SR-CorrNet, a novel asymmetric encoder-decoder framework that improves speech separation in complex acoustic environments by leveraging spatio-spectro-temporal correlations and a dynamic split module. This research significantly advances the state-of-the-art in speech separation, providing a robust solution for real-world applications.
Methodology
The proposed SR-CorrNet framework introduces a novel asymmetric encoder-decoder architecture that effectively addresses the limitations of late-split designs in speech separation tasks. By employing a separation-reconstruction strategy and a correlation-to-filter paradigm, the methodology enhances speaker discrimination and robustness in challenging acoustic environments. The incorporation of spatio-spectro-temporal correlations as input features is a significant advancement, allowing the model to leverage temporal and spatial dependencies more effectively. The dynamic split module further enhances the model's adaptability to varying speaker counts, which is crucial for real-world applications.
Results & evaluation
The experiments conducted on multiple datasets (WSJ0-2/3/4/5Mix, WHAMR!, and LibriCSS) demonstrate the effectiveness of the proposed method across different conditions, including anechoic, noisy-reverberant, and real-recorded environments. The results show consistent improvements over existing models, indicating the robustness and generalizability of SR-CorrNet. The use of objective metrics like SI-SNRi and SDRi provides a solid basis for evaluating performance, although subjective evaluations could further strengthen the findings.
Reproducibility
The paper provides detailed descriptions of the model architecture, training procedures, and datasets used, which are essential for reproducibility. However, the absence of a public code repository or demo URL limits the ability for other researchers to replicate the experiments directly. Clearer documentation or supplementary materials could enhance reproducibility.
Limitations
One limitation of the study is the lack of subjective evaluation metrics, such as human listening tests, which could provide insights into the perceptual quality of the separated audio. Additionally, while the dynamic split module shows promise, its performance in highly variable acoustic environments needs further validation. The model's complexity may also pose challenges in real-time applications.
Broader impact
The advancements in speech separation technology have significant implications for various applications, including automatic speech recognition, hearing aids, and communication systems in noisy environments. The ability to effectively separate overlapping speech can enhance the user experience in real-world scenarios, making this research highly relevant to both academia and industry. The main contribution of this work is the introduction of SR-CorrNet, a novel asymmetric encoder-decoder framework that improves speech separation in complex acoustic environments by leveraging spatio-spectro-temporal correlations and a dynamic split module. This research significantly advances the state-of-the-art in speech separation, providing a robust solution for real-world applications.
Author of this paper? Send a correction.