CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction
Abstract
This paper targets a new scenario that integrates speech separation with speech compression, aiming to disentangle multiple speakers while producing discrete representations for efficient transmission or storage, with applications in online meetings and dialogue archiving. To address this scenario, we propose CodeSep, a codec-driven model that jointly performs speech separation and low-bitrate compression. CodeSep comprises a residual vector quantizer (RVQ)-based plain neural speech codec, a base-token disentanglement (BTD) module, and parallel auxiliary-token serial prediction (ATSP) modules. The BTD module disentangles mixed-speech mel-spectrograms into base tokens for each speaker, which are then refined by ATSP modules to serially predict auxiliary tokens, and finally, all tokens are decoded to reconstruct separated waveforms through the codec decoder. During training, the codec's RVQ provides supervision with permutation-invariant and teacher-forcing-based cross-entropy losses. As only base tokens are transmitted or stored, CodeSep achieves low-bitrate compression. Experimental results show that CodeSep attains satisfactory separation performance at only 1 kbps compared with baseline methods.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of CodeSep, a codec-driven model that effectively combines speech separation and low-bitrate compression, showcasing significant advancements in both methodology and practical applications. The innovative approach and comprehensive evaluation position this work as a valuable addition to the field of audio processing and machine learning.
Methodology
The proposed CodeSep model introduces a novel architecture that integrates speech separation and low-bitrate compression through a codec-driven approach. The use of a residual vector quantizer (RVQ) for token representation and the innovative base-token disentanglement (BTD) and auxiliary-token serial prediction (ATSP) modules are significant contributions. The methodology effectively addresses the dual challenge of separating mixed speech while maintaining low bitrate for efficient transmission, which is a critical need in real-world applications like online meetings. The training strategy, including permutation-invariant cross-entropy loss for BTD and teacher-forcing for ATSP, demonstrates a thoughtful approach to tackle the inherent challenges of the task.
Results & evaluation
The experiments utilize the Libri2Mix-clean dataset, which is appropriate for the task, and the evaluation metrics are well-chosen to assess both objective and subjective performance. The results indicate that CodeSep outperforms baseline methods significantly, achieving high-quality speech separation at a remarkably low bitrate of 1 kbps. The inclusion of both objective metrics (UTMOS, DNSMOS) and subjective assessments (NMOS, SMOS) strengthens the validity of the findings, providing a comprehensive evaluation of the model's performance.
Reproducibility
The paper provides sufficient details on the architecture, training procedures, and evaluation metrics, which would allow other researchers to replicate the experiments. However, the absence of a public code repository limits full reproducibility. The demo URL offers some audio samples, but a complete code release would enhance the reproducibility of the findings.
Limitations
One limitation is the focus on a two-speaker scenario, which may not generalize well to cases with more speakers or different acoustic environments. Additionally, while the results are promising, the performance at higher bitrates could be further explored to understand the model's scalability and robustness in various conditions.
Broader impact
The integration of speech separation and compression has significant implications for applications in telecommunications, online conferencing, and archiving of spoken dialogues. By achieving high-quality separation at low bitrates, CodeSep can facilitate more efficient use of bandwidth and storage, which is increasingly important in the context of growing digital communication needs. The main contribution of this paper is the introduction of CodeSep, a codec-driven model that effectively combines speech separation and low-bitrate compression, showcasing significant advancements in both methodology and practical applications. The innovative approach and comprehensive evaluation position this work as a valuable addition to the field of audio processing and machine learning.
Author of this paper? Send a correction.