Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Abstract
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
Methodology
The paper proposes "Cocktail-Talker," a framework for multi-speaker spoken dialog in noisy environments. The core methodological contribution is the introduction of three discrete action tokens (`<|respond|>`, `<|listen|>`, `<|ignore|>`) to model turn-taking behavior, trained via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The approach leverages a pre-existing speech LLM backbone (Qwen2.5-Omni) and freezes the audio encoder and TTS decoder, focusing training on the language model head and LoRA adapters. A significant portion of the work involves "Cocktail-DialogGen," an LLM-based pipeline to synthesize training data by simulating multi-speaker conversations with varying acoustic conditions (SNR, background noise) and semantic constraints (anonymous vs. named speakers). The use of GRPO with a reward function based on action accuracy and format integrity is a novel application of RLHF techniques to the specific problem of turn-taking decision-making in speech LLMs.
Results & evaluation
The evaluation is comprehensive, covering both seen and unseen environments. The paper compares Cocktail-Talker against several strong baselines, including Moshi, PersonaPlex, Step-Audio2, Kimi-Audio, Qwen2.5-Omni, and Qwen3-Omni. The metrics include binary response accuracy (Respond/Silent) and response quality (METEOR, ROUGE-L, BERTScore, SentCos). The results show that Cocktail-Talker significantly outperforms baselines in decision accuracy (macro F1 ~0.93) and response quality. The analysis breaks down performance by SNR, number of speakers, anonymity, and context length, providing robust insights into model behavior. The inclusion of "unseen environments" tests generalization, which is crucial for real-world applicability. The ablation studies (SFT vs. SFT+GRPO) demonstrate the benefit of the reinforcement learning stage.
Reproducibility
The paper provides a GitHub link to the code. The data generation pipeline is described in detail, including the use of specific LLMs (Gemini 3 Pro, Qwen3-TTS) and noise datasets (DEMAND, Freesound). However, the reliance on proprietary models (Gemini, Qwen) for data generation and the specific random seeds or exact prompts used in the LLM-based data synthesis might pose reproducibility challenges for other researchers. The hyperparameters for SFT and GRPO are provided, which aids in reproducing the model training.
Limitations
The paper acknowledges several limitations. The system is not streaming, which limits its real-time applicability. It assumes fixed turn boundaries in the simulation, which may not reflect the fluid nature of real-world interruptions. The model relies heavily on the simulated data, and while it generalizes to unseen environments, the distribution shift between synthetic and real-world cocktail party scenarios remains a potential issue. The performance degrades with increased speaker count and anonymity, indicating room for improvement in complex social reasoning.
Broader impact
This work contributes to the development of more natural and robust speech assistants that can operate in real-world social settings. By enabling selective participation, it reduces the annoyance of assistants interrupting irrelevant conversations or responding to noise. This has implications for the deployment of AI agents in homes, public spaces, and social robotics. However, the potential for misuse in surveillance or eavesdropping scenarios should be considered, although the model is designed to ignore irrelevant speech. Cocktail-Talker introduces a novel framework for multi-speaker spoken dialog by modeling turn-taking as a discrete action selection problem, trained with GRPO on synthetically generated data, achieving state-of-the-art performance in noisy, multi-speaker environments. The paper makes a significant technical contribution by addressing the under-explored "speaking side" of multi-speaker conversation, moving beyond simple dyadic interactions to more complex social auditory scenes.
Author of this paper? Send a correction.