Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model
Abstract
Extracting individual elements from music mixtures is a valuable tool for music production and practice. While neural networks optimized to mask or transform mixture spectrograms into the individual source(s) have been the leading approach, the source overlap and correlation in music signals poses an inherent challenge. Also, accessing all sources in the mixture is crucial to train these systems, while complicated. Attempts to address these challenges in a generative fashion exist, however, the separation performance and inference efficiency remain limited. In this work, we study the potential of diffusion models to advance toward bridging this gap, focusing on generative singing voice separation relying only on corresponding pairs of isolated vocals and mixtures for training. To align with creative workflows, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio. This enables efficient optimization and faster inference. Our system is trained using only open data. We outperform existing generative separation systems, and level the compared non-generative systems on a list of signal quality measures and on interference removal. We provide a noise robustness study on the latent encoder, providing insights on its potential for the task. We release a modular toolkit for further research on the topic.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of an efficient and effective generative model for singing voice separation using latent diffusion, which significantly advances the state of the art in music source separation. The combination of innovative methodology and rigorous evaluation positions this work as a valuable addition to the field of audio processing and machine learning.
Methodology
The paper introduces a novel approach to singing voice separation using latent diffusion models (LDM), which operate in a compact latent space rather than directly in the audio domain. This method leverages the strengths of denoising diffusion probabilistic models (DDPM) while addressing the challenges of source overlap and the need for extensive training data. The use of a pre-trained neural audio codec (EnCodec) for generating latent representations is particularly innovative, as it allows for efficient training and inference. The methodology is well-structured, detailing the diffusion process, the architecture of the U-Net generator, and the conditioning mechanism that guides the separation process.
Results & evaluation
The authors conduct a thorough evaluation of their model against both generative and non-generative baselines using objective metrics such as log-spectral distance (LSD), Mel-spectrogram Mean Absolute Error (Mel-MAE), and perceptual evaluation metrics like PESQ. The results indicate that the proposed system outperforms existing generative models and matches the performance of non-generative systems on several metrics, demonstrating its effectiveness in real-world applications. Additionally, the perceptual tests provide valuable insights into the quality of the generated vocals, highlighting the importance of user-centered evaluations in audio processing.
Reproducibility
The paper includes a modular Python toolkit released on GitHub, which facilitates reproducibility. The authors provide detailed information about the experimental setup, including the training process, data augmentation techniques, and the architecture of the model. However, the reliance on a specific pre-trained codec (EnCodec) may limit reproducibility for those without access to the same resources.
Limitations
While the paper presents promising results, it acknowledges the presence of high-frequency artifacts and reconstruction errors that can affect the output quality. The authors suggest that fine-tuning the latent encoder with more vocal data could mitigate these issues. Additionally, the model's performance may vary based on the quality and diversity of the training data, which could limit its generalizability to other musical contexts.
Broader impact
The proposed method has significant implications for music production and education, as it provides a more accessible tool for vocal separation that can be utilized by musicians and audio engineers. By reducing the computational demands and training data requirements, this approach could democratize access to high-quality audio processing tools, fostering creativity and innovation in the music industry. The main contribution of this paper is the development of an efficient and effective generative model for singing voice separation using latent diffusion, which significantly advances the state of the art in music source separation. The combination of innovative methodology and rigorous evaluation positions this work as a valuable addition to the field of audio processing and machine learning.
Author of this paper? Send a correction.