Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors
Abstract
Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. We take a different approach. Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the \textit{Reference Shortcut}. During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics while generating rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper introduces ScenA, a flow-matching framework for multi-speaker audio scene generation that overcomes the "Reference Shortcut" via high-noise-biased training, enabling robust speaker binding and rich ambient audio generation from minimal natural language and reference inputs.]
Methodology
The paper proposes ScenA, a novel framework for multi-speaker audio scene generation that conditions a pre-trained text-to-audio flow-matching model on multiple reference voices and free-form natural language prompts. The core methodological innovation lies in the "Reference Shortcut" diagnosis and its mitigation. The authors identify that standard flow-matching training schedules allow the model to bypass text-based speaker binding by relying on acoustic similarity between the noisy target and the clean reference latents. To counter this, they introduce a high-noise-biased timestep distribution (Beta+Uniform mixture) that forces the model to rely on the text prompt for identity assignment during the critical early denoising steps. The architecture is notably minimalist, using concatenated reference latents with lightweight identity-aware positional encodings, avoiding complex identity encoders or structured supervision tags. This approach leverages the inherent capabilities of large-scale in-the-wild audio foundation models to generate ambient textures, overlapping speech, and paralinguistic events jointly with dialogue, a significant departure from traditional speech-only TTS pipelines.
Results & evaluation
The evaluation is rigorous and well-designed, focusing on both speaker binding fidelity and audio quality. The authors utilize the CoVoMix2-Dialogue benchmark, creating specific subsets (CoVoMix2-Dialogue-20s and CoVoMix2-Dialogue-WildRef) to test performance on studio-clean versus in-the-wild references. They compare ScenA against state-of-the-art multi-speaker dialogue TTS systems (MOSS-TTS, VibeVoice, ZipVoice, Dia). Results show ScenA outperforms baselines on binding-aware metrics (cpWER, cpSIM, ACC), particularly demonstrating robustness when references are noisy or from the wild, where baselines fail significantly. The inclusion of a "Reference Shortcut Probe" provides strong empirical evidence for their hypothesis, showing that the model can identify speakers from noisy targets at low noise levels, validating the need for the high-noise training bias. Human preference tests further support the qualitative superiority of the generated scenes.
Reproducibility
The paper provides substantial implementation details, including the backbone architecture (LTX-2 audio stream), training hyperparameters (AdamW, batch size, learning rate schedule), and the specific mathematical formulation of the timestep distribution. The dataset construction pipeline is described in detail, including the use of diarization and captioning models to create the training data. The code is not explicitly linked in the text provided (only a project page URL), but the methodological description is sufficient for reproduction by researchers familiar with flow-matching and diffusion transformers. The ablation studies on positional encodings and augmentation strategies add to the reproducibility and robustness of the claims.
Limitations
The authors acknowledge several limitations. The generation duration is capped at 20 seconds due to the backbone's constraints, although they note this can be extended with modest fine-tuning. The number of supported speakers is limited to $K_{max}=3$ in the current configuration, constrained by the linear growth of the self-attention sequence with the number of references. The reliance on a pre-trained foundation model means the quality is bound by the underlying model's capabilities and potential biases in the in-the-wild training data. Additionally, the "Reference Shortcut" phenomenon, while solved for this specific setup, highlights a general fragility in reference-conditioned generation that may require similar careful schedule design in other modalities.
Broader impact
This work significantly advances the field of generative audio by demonstrating that complex, structured multi-speaker interactions can be generated using minimal, natural language conditioning on top of general-purpose audio models. This reduces the need for complex, brittle pipeline architectures in dialogue TTS. The ability to generate realistic, ambient-rich conversational audio has applications in virtual reality, gaming, and accessible media creation. However, the ease of cloning voices and generating realistic dialogue raises concerns about deepfakes and misinformation, necessitating responsible use guidelines and watermarking techniques, which are not discussed in the paper. [The paper introduces ScenA, a flow-matching framework for multi-speaker audio scene generation that overcomes the "Reference Shortcut" via high-noise-biased training, enabling robust speaker binding and rich ambient audio generation from minimal natural language and reference inputs.]
Author of this paper? Send a correction.