Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning
Abstract
We propose diarization-conditioned spoken language models (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Output Training, which risks catastrophic forgetting, we condition the acoustic encoder on diarization masks to extract target-speaker representations, keeping the decoder frozen. We instantiate this as Dixtral, integrating a Diarization Conditioned Whisper (DiCoW) encoder into the Voxtral SLM. On AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, Dixtral outperforms Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 on speaker-attributed transcription by 29.0%, 19.8%, and 16.0% absolute cpWER respectively. On a novel long-form multi-speaker QA benchmark, zero-shot Dixtral matches Gemini on far-field content understanding, and when fine-tuned surpasses both Gemini and Voxtral operating on close-talk across all tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a compelling and effective method for grounding spoken LLMs in multi-speaker audio through encoder-side diarization conditioning, achieving state-of-the-art performance on transcription and novel capabilities in multi-speaker reasoning and QA.
Methodology
The paper proposes a novel architectural strategy for extending Spoken Large Language Models (SLMs) to multi-speaker scenarios by conditioning the acoustic encoder on diarization masks, rather than adapting the decoder. This approach, instantiated as Dixtral, integrates a Diarization Conditioned Whisper (DiCoW) encoder with a frozen Voxtral decoder. The core innovation lies in the "Diarization Conditioning" mechanism, which uses frame-level speaker activity probabilities (STNO masks) to modulate internal representations via learnable affine transformations (FDDT). This allows the model to extract target-speaker representations while keeping the LLM decoder frozen, thereby avoiding catastrophic forgetting of reasoning capabilities associated with Serialized Output Training (SOT) and vocabulary expansion. The methodology is theoretically sound, offering a computationally efficient alternative ($O(S N^2)$ vs $O((SN)^2)$) for multi-speaker decoding.
Results & evaluation
The evaluation is comprehensive, covering four standard multi-speaker ASR datasets (AMI, NOTSOFAR-1, LibriSpeechMix, Mixer6) and a novel long-form QA/Summarization benchmark (NSF-QA). Dixtral demonstrates significant improvements over strong baselines, including Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2, with absolute cpWER reductions of 16-29%. The inclusion of a paralinguistic QA task (emotion/gender) is particularly strong, as it tests the model's ability to utilize audio features beyond text, which cascaded systems cannot do. The results are robust, showing that zero-shot Dixtral matches Gemini on content QA and surpasses it when fine-tuned. The out-of-domain performance on Mixer6 further validates generalization.
Reproducibility
The authors provide open-source code and a new dataset (NSF-QA). Training details are well-specified, including hardware constraints (8x A5000), optimization settings, and data chunking strategies. The use of established backbones (Whisper, Ministral, DiariZen) and clear integration points (FDDT, MLP adapter) ensures high reproducibility. The release of the benchmark dataset is a significant contribution to reproducibility in this niche.
Limitations
The performance is inherently dependent on the quality of the external diarization system (DiariZen). Errors in diarization will propagate directly to the transcription and reasoning tasks. The paper acknowledges this but does not extensively analyze the sensitivity to diarization errors. Additionally, the current implementation requires separate inference passes for each target speaker, which, while more efficient than joint decoding, still scales linearly with the number of speakers. The fine-tuning for QA/Summarization slightly degrades pure ASR performance, indicating a trade-off that requires careful multi-task optimization in future work.
Broader impact
This work significantly advances the field of spoken language understanding by enabling end-to-end, multi-speaker reasoning in far-field audio. It bridges the gap between modular ASR pipelines and unified SLMs, offering a path towards more robust and capable voice assistants and meeting transcription tools. The ability to handle paralinguistic information (emotion, gender) in a multi-speaker context opens new avenues for affective computing and human-computer interaction. The paper presents a compelling and effective method for grounding spoken LLMs in multi-speaker audio through encoder-side diarization conditioning, achieving state-of-the-art performance on transcription and novel capabilities in multi-speaker reasoning and QA.
Author of this paper? Send a correction.