Soloist: A Generative Model for Blind Voice Separation in the Wild
Abstract
Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
Methodology
The paper proposes Soloist, a generative model for blind voice separation that frames the task as autoregressive generation of audio-codec sequences. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on synthetic mixtures with isolated targets and, crucially, on real recordings with only speaker timeline annotations (who speaks when), which helps the model learn count inference and activity patterns without waveform targets; and (2) Reinforcement Learning (RL) on real recordings using a "model-judge" reward. The RL component is particularly novel, employing a multi-step verification process where an audio-language model (Qwen3-Omni) describes candidates independently, and a text model (Qwen3.6) ranks them based on these descriptions, with a consistency check (Kendall's tau) under shuffled/reversed orders to filter out order-biased or unstable feedback. This addresses the lack of ground-truth isolated sources in real-world data. The architecture leverages a Qwen3-4B backbone with a local transformer for codec code prediction, integrating continuous and discrete audio features.
Results & evaluation
The evaluation is comprehensive, covering overlapping speech (LibriMix, WildTV), noisy environments (ViW), and singing vocals (SongYT, MIR-1K, MUSDB18-HQ). The paper reports strong results in PI-WER and DNSMOS, outperforming traditional separators (SepFormer, MossFormer) and query-based extractors. The inclusion of a full-duplex conversation pipeline demonstrates practical utility, showing how the model can be integrated into a larger system for continuous speaker track assembly. The judge validation section provides evidence that the RL reward signal correlates with objective quality metrics (PI-WER) on known-source mixtures, which is a critical sanity check for the RL approach.
Reproducibility
The paper provides detailed descriptions of the data construction, training objectives, and RL hyperparameters. However, the reliance on specific, potentially proprietary or large-scale models (Qwen3-Omni, MOSS-Audio-Tokenizer, MOSS-Transcribe-Diarize) and the complexity of the judge pipeline make exact reproduction difficult without access to these specific components and the in-house datasets (e.g., the "in-house song recordings" and specific Pixabay subsets). The code is not explicitly linked in the provided text, though the use of open-source base models (Qwen3) suggests a path to partial reproduction.
Limitations
The method is computationally expensive due to the autoregressive generation of codec codes and the multi-step RL judge process. The "blind" separation relies on the model's ability to infer the correct number of speakers, which can fail in highly overlapping or noisy scenarios. The RL reward, while robust to order bias, is still based on an LLM's judgment of audio descriptions, which may not perfectly align with human perceptual quality or subtle acoustic artifacts. The singing separation uses teacher-generated pseudo-labels, which limits the fidelity to the true studio vocal.
Broader impact
This work bridges the gap between traditional signal processing for speech separation and modern generative AI techniques. By using LLMs as judges for RL in audio tasks, it offers a template for training audio models on unpaired, real-world data. The application to full-duplex conversation processing highlights its potential for real-time or near-real-time conversational AI systems that require robust handling of overlapping speech and noise. Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
Author of this paper? Send a correction.