Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.
Primary: Tsinghua University
All Institutions: Tsinghua University, ByteDance
ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
The paper proposes ELLSA 2, a unified end-to-end framework that integrates full-duplex speech interaction with streaming gesture and facial expression generation. The core methodological contribution is the "contextual semantic call" mechanism, which decouples the decision of *whether* to perform a semantic gesture from *how* to realize it. This is achieved via a binary classification head on the LLM backbone's hidden states, which triggers the retrieval of a curated exemplar template to guide the autoregressive gesture generator. The architecture uses a chunk-wise autoregressive synthesizer that interleaves speech, listening, and motion tokens, allowing non-verbal behaviors to persist during verbal silence (listening states). The use of RVQ-VAE for motion tokenization and a curriculum training strategy to integrate modalities is technically sound and well-motivated.
The experimental setup is robust, introducing a new human-verified benchmark (138 clips) specifically for dialog-aware gesture generation, covering full-duplex behaviors and semantic gestures. The evaluation employs both objective metrics (prosodic alignment, activity ratio) and subjective human evaluation, along with MLLM-as-a-judge metrics. The results demonstrate that ELLSA 2 outperforms cascaded baselines and remains competitive with state-of-the-art full-duplex speech models. The semantic template guidance shows a significant improvement in semantic gesture quality (human score 2.014 to 2.711). The correlation analysis between MLLM judges and human annotators (0.853/0.919) provides strong evidence for the utility of the proposed automatic metrics.
The paper provides extensive implementation details, including tokenizer specifications, training hyperparameters, and data curation pipelines. The authors commit to releasing code, data, and model checkpoints upon acceptance. The detailed description of the semantic exemplar library construction and the evaluation protocols enhances reproducibility.
The system relies on a curated library of 3,227 exemplars for semantic gestures, which may limit the diversity of semantic actions compared to fully generative approaches. The evaluation, while improved, is still based on a relatively small test set (138 clips). The model's performance on long-horizon coherence of gestures over extended conversations is not deeply analyzed.
This work advances the field of human-computer interaction by moving beyond text or speech-only interfaces to fully embodied, multimodal agents. The proposed evaluation metrics and benchmark will be valuable for the community working on co-speech gesture generation and full-duplex dialogue systems. The integration of semantic understanding with physical gesture generation has implications for robotics, virtual avatars, and immersive communication. ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
Primary: Adobe Research
All Institutions: Adobe Research, Massachusetts Institute of Technology
PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
The paper proposes PromptSep-Speaker, a generative target speaker extraction (TSE) model based on latent diffusion. The core methodological contribution is the unification of four distinct cue modalities (speaker temporal order, transcript, enrollment audio, and speaker caption) within a single framework. A significant technical innovation is the "alignment module," which uses a frozen WavLM representation of the mixture to cross-attend to text-based cues (transcript/speaker ID). This is coupled with a novel auxiliary alignment loss that supervises the cross-attention maps to ensure the model correctly localizes the target speaker in the time domain without requiring extra labeling. The use of a DiT (Diffusion Transformer) backbone fine-tuned from a pre-trained source separation model is a strong architectural choice, leveraging prior knowledge of audio generation. The integration of semantic features via WavLM to guide attention is a clever way to bridge the gap between high-level linguistic cues and low-level acoustic separation.
The evaluation is rigorous and comprehensive. The authors test on four distinct datasets (CapSpeech-Mix, Libri2Mix, LibriTTS-R-Mix, and an internal set), covering both in-domain and out-of-domain scenarios. They compare against a diverse set of baselines, including mask-based TSE systems (DAE-TSE, SoloSpeech, WeSep) and generative universal source separators (PromptSep, FlowSep, SAM-Audio). The results show consistent state-of-the-art performance across SI-SDR, mel-L2, DNSMOS, and F1 decision error metrics. The inclusion of a subjective listening test with 380 subjects and 4,515 submissions adds significant weight to the objective metrics, confirming that the generative approach yields better perceptual quality and speaker accuracy than mask-based alternatives. The ablation studies effectively demonstrate the necessity of the auxiliary alignment loss and the specific direction of cross-attention.
The paper provides a good level of detail regarding the architecture (DiT, DAC-VAE, WavLM) and training hyperparameters (learning rate, sampler steps). However, the reliance on a "large internal collection of licensed data" for training limits full reproducibility by external groups. The specific composition of the 4.8M speech samples and the simulation pipeline details are high-level. While the demo page is provided, the absence of a public code repository or pre-trained weights for the specific PromptSep-Speaker model (distinct from the base models) is a minor drawback, though common for industry-affiliated papers at the arXiv stage.
The model is computationally expensive due to the diffusion process, which may limit real-time application compared to mask-based models. The evaluation is primarily on English speech; generalization to other languages is not tested. The "internal set" is not publicly available, making it impossible for third parties to verify those specific results. The paper does not extensively discuss the failure modes of the generative approach, such as hallucinations or artifacts in complex acoustic environments, beyond general statements about quality.
This work has significant implications for the field of speech processing by demonstrating that generative models can outperform traditional discriminative (mask-based) approaches in target speaker extraction. The multimodal prompting capability makes the system more versatile and user-friendly, as users can provide whichever cue is most available (e.g., a transcript from a meeting or a voice sample). This could lead to more robust meeting transcription systems, assistive listening devices, and content creation tools. The alignment loss technique could be adapted to other multimodal generation tasks where precise temporal grounding of text to audio is required. PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.
Primary: Imperial College London
All Institutions: Imperial College London
The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
The paper introduces a novel threat model for speech models where the trigger is the speaker's identity rather than an injected acoustic perturbation. The proposed two-phase fine-tuning method (SAB) is technically sound: Phase 1 enhances speaker discriminability in the encoder representations, and Phase 2 binds the target speaker to a malicious payload while preserving clean performance on non-target speakers. The ablation studies effectively demonstrate that Phase 1 is crucial for content-dependent attacks (semantic rewriting) and improves selectivity in fixed-output attacks. The use of voice cloning (F5-TTS) to augment limited real target audio is a practical and effective strategy.
The experimental evaluation is comprehensive, covering multiple architectures (Whisper, Parakeet, Qwen-Omni) and attack types (transcript suppression, semantic rewriting, instruction injection). The inclusion of downstream tool-use attacks with frontier LLMs (GPT-4o, Claude) significantly strengthens the impact by demonstrating real-world consequences. The evaluation on held-out real recordings ensures the results are not artifacts of synthetic data. Robustness to acoustic degradation and defense mechanisms (clean fine-tuning, voice anonymization) are also thoroughly analyzed.
The paper provides detailed hyperparameters, dataset splits, and training procedures. However, the specific implementation details for the auxiliary speaker head and the exact prompts used for the LLM agents are not fully specified in the main text, though the appendix provides some additional details. The use of standard datasets (LibriSpeech, VoxCeleb2) and open-source models aids reproducibility.
The attack relies on the attacker having access to the fine-tuning pipeline, which is a standard assumption but still a barrier. The backdoor is susceptible to clean fine-tuning, which limits its persistence in dynamic environments. The evaluation is limited to English speakers and specific public figures, which may not generalize to all demographic groups or private individuals.
This work highlights a significant security vulnerability in the growing ecosystem of voice-activated AI agents. It demonstrates that speaker identity can be exploited as a control channel, posing risks to privacy and safety in applications like voice assistants and automated banking. The findings will likely prompt the development of new defenses focused on speaker-identity invariance and supply-chain security for speech models. The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves $6.92\%$ WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.
Primary: Shanghai Jiao Tong University
All Institutions: Shanghai Jiao Tong University
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
The paper proposes SLM-SS, a generative speech separation framework that leverages Speech Language Models (SLMs) to predict discrete neural codec tokens instead of continuous waveforms. The core method involves discretizing clean speech into multi-codebook sequences using Encodec or WavTokenizer, serializing multi-speaker targets via Serialized Output Training (SOT), and using an autoregressive (AR) encoder-decoder to predict the zero-order codebook. A non-autoregressive (NAR) module then refines higher-order residual codebooks. The introduction of a "per-talker NAR" strategy is a key methodological contribution, simplifying high-order modeling by decoupling speaker transitions from residual refinement. The approach effectively bridges the gap between signal-level separation and intelligibility-oriented downstream tasks by operating in the discrete token space.
Experiments are conducted on the LibriMix dataset (2-mix and 3-mix). The paper provides a comprehensive comparison against strong discriminative baselines (BSRNN, SepFormer, TF-LocoFormer, MossFormer2). Results show that SLM-SS achieves competitive WER and subjective quality scores, with the WavTokenizer variant achieving the highest mean subjective quality score. The inclusion of subjective listening tests with 30 volunteers and statistical significance testing (Wilcoxon signed-rank) adds rigor. Ablations on temperature, number of codebooks, and model parameters are provided. The evaluation of variable talker counts (up to 4) demonstrates the framework's flexibility.
The paper provides detailed architectural descriptions (WavLM encoder, Whisper-style decoder), training hyperparameters (learning rate, beam size, epochs), and dataset preprocessing steps (VAD segmentation). However, specific code availability is not explicitly confirmed in the text, and some implementation details regarding the exact initialization of the NAR module and the specific "staged training" strategy are described at a high level. The reliance on specific pre-trained models (WavLM, Whisper, Encodec, WavTokenizer) aids reproducibility.
The primary limitation is the computational cost of autoregressive decoding, which limits real-time applicability compared to discriminative methods. The method is constrained by the quality of the neural codec; truncating Encodec to 8 codebooks introduces irreversible distortion that caps the upper bound of signal fidelity (PESQ). The paper acknowledges that WavTokenizer, while better for subjective quality, has higher WER, indicating a trade-off between perceptual quality and linguistic accuracy. The experiments are limited to the LibriMix dataset, which consists of clean speech mixtures; performance on noisy or reverberant environments is not extensively tested in the main results (though mentioned as a future direction or minor test).
This work contributes to the trend of using discrete token representations for speech processing, aligning with the broader shift towards LLM-based audio models. It offers a viable alternative to traditional discriminative separation for applications where intelligibility is paramount, such as multi-talker ASR. The framework's ability to handle variable numbers of talkers without explicit speaker count estimation is a significant practical advantage. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces SLM-SS, a token-based generative speech separation framework that utilizes speech language models to predict neural codec tokens, achieving competitive intelligibility and subjective quality scores while offering a flexible architecture for variable-talker scenarios, though it faces challenges in computational efficiency and signal fidelity due to codec discretization limits.
Speech models are increasingly used as an interface to AI systems, from transcription tools to voice agents. Existing audio backdoors require an attacker to introduce triggers at inference time. We study a different threat model in which the attacker compromises the model during fine-tuning but has no access to the victim's inference-time inputs. We introduce the Speaker-Activated Backdoor (SAB) setting, in which a target speaker's natural voice serves as the trigger. We develop a two-phase fine-tuning method that first makes speaker identity more accessible and then binds the target speaker to an attacker-chosen payload. Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks. SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker. We further demonstrate content-dependent semantic rewriting and downstream tool-use attacks, where malicious instructions are inserted into 96.9% of target-speaker transcripts. When these transcripts are passed to frontier LLM agents, GPT-4o and Claude (Opus, Sonnet, and Haiku), the injected instruction drives the agent to carry out the harmful action in up to 87.5% of cases. The attack generalizes to held-out recordings, multiple target speakers, and common acoustic degradations. These results establish speaker identity as a practical backdoor trigger that requires no inference-time attacker access and is invisible at the transcript layer.
Primary: Imperial College London
All Institutions: Imperial College London
The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
The paper introduces a novel threat model for speech models where the trigger is the speaker's identity rather than an injected acoustic perturbation. The proposed two-phase fine-tuning method (SAB) is technically sound: Phase 1 enhances speaker discriminability in the encoder representations, and Phase 2 binds the target speaker to a malicious payload while preserving clean performance on non-target speakers. The ablation studies effectively demonstrate that Phase 1 is crucial for content-dependent attacks (semantic rewriting) and improves selectivity in fixed-output attacks. The use of voice cloning (F5-TTS) to augment limited real target audio is a practical and effective strategy.
The experimental evaluation is comprehensive, covering multiple architectures (Whisper, Parakeet, Qwen-Omni) and attack types (transcript suppression, semantic rewriting, instruction injection). The inclusion of downstream tool-use attacks with frontier LLMs (GPT-4o, Claude) significantly strengthens the impact by demonstrating real-world consequences. The evaluation on held-out real recordings ensures the results are not artifacts of synthetic data. Robustness to acoustic degradation and defense mechanisms (clean fine-tuning, voice anonymization) are also thoroughly analyzed.
The paper provides detailed hyperparameters, dataset splits, and training procedures. However, the specific implementation details for the auxiliary speaker head and the exact prompts used for the LLM agents are not fully specified in the main text, though the appendix provides some additional details. The use of standard datasets (LibriSpeech, VoxCeleb2) and open-source models aids reproducibility.
The attack relies on the attacker having access to the fine-tuning pipeline, which is a standard assumption but still a barrier. The backdoor is susceptible to clean fine-tuning, which limits its persistence in dynamic environments. The evaluation is limited to English speakers and specific public figures, which may not generalize to all demographic groups or private individuals.
This work highlights a significant security vulnerability in the growing ecosystem of voice-activated AI agents. It demonstrates that speaker identity can be exploited as a control channel, posing risks to privacy and safety in applications like voice assistants and automated banking. The findings will likely prompt the development of new defenses focused on speaker-identity invariance and supply-chain security for speech models. The paper introduces Speaker-Activated Backdoors (SAB), a novel attack where a target speaker's natural voice triggers malicious behavior in speech models without inference-time attacker access. By proposing a two-phase fine-tuning method that enhances speaker discriminability and binds identity to payloads, the authors demonstrate high-success, selective backdoors across multiple ASR and audio-LLM architectures, including downstream harmful actions in tool-using agents, establishing speaker identity as a critical security vector in voice-based AI systems.
A cover song re-renders an existing piece, most commonly by preserving its tonal content -- melody and chord progression -- while reshaping other musical attributes. This suggests a natural formulation for generative cover synthesis with pretrained music language models: explicitly control tonal content, while specifying lyrics and style through text. Existing approaches condition on frame-level features, tightly coupling generation to the reference's timing and structure, and limiting flexibility such as tempo variation, structural rearrangement, or partial conditioning. To this end, we present MuLaCover, a cover generation model that conditions a pretrained text-to-song foundation model on an editable symbolic lead sheet. Our design enables alignment-free control: generated outputs preserve the tonal signature of the source without frame-level alignment, allowing flexible timing, structure, and segment-level generation. We further introduce a training curriculum with partially mismatched audio-symbolic pairs, improving diversity and robustness. We evaluate MuLaCover with objective metrics, standard subjective ratings, and in-depth expert interviews that provide fine-grained diagnostic insights. Experiments show that MuLaCover achieves state-of-the-art performance among open-source systems and is competitive with leading commercial models such as Suno-v5.5. Audio samples and code are available at our project page: https://mulalabs.ai/mulacover/
Primary: MBZUAI
All Institutions: MBZUAI, Peking University, MuLa Labs, Vera Praxis Lab, The Chinese University of Hong Kong, Nanjing University of the Arts
MuLaCover introduces a flexible cover song generation framework by conditioning a text-to-song model on BPM-normalized symbolic lead sheets via gated cross-attention. The paper demonstrates that decoupling tonal content from timing through symbolic representation and cross-cover training enables superior controllability and diversity compared to frame-level conditioning, achieving performance competitive with leading commercial systems.
The paper proposes MuLaCover, a system for cover song generation that conditions a pretrained text-to-song foundation model (HeartMuLa) on a symbolic lead sheet (melody + chords) rather than frame-level audio features. The core methodological contribution is the use of BPM-normalized symbolic representations to decouple tonal content from timing, enabling flexible tempo and structural changes. The model injects this condition via a Gated Adaptive Cross-Attention (GACA) module into the last layers of the backbone. A key innovation is the three-stage training curriculum, specifically the "cross-cover" finetuning stage where tonal content from one rendition is paired with lyrics/style from another, effectively teaching the model to disentangle content from style. The ablation studies rigorously validate the necessity of BPM normalization, the gate mechanism, and the cross-cover training strategy.
The evaluation is comprehensive, utilizing objective metrics (AudioBox-Aesthetics, SongEval, PER, MuQ similarity, CSI accuracy) and subjective evaluations (listening study with 22 participants, expert interviews). The paper compares against strong baselines including open-source systems (SongEcho, ACE-Step-1.5) and a leading commercial model (Suno-v5.5). Results show MuLaCover achieves state-of-the-art performance among open-source systems and is competitive with Suno-v5.5. The partial-reference cover task (inpainting/outpainting) is also evaluated, demonstrating the flexibility of the alignment-free approach. The expert interview protocol provides qualitative insights into why certain systems perform better on specific musical attributes.
The paper provides detailed descriptions of the architecture, training stages, and data pipeline. It mentions the use of specific models for transcription (YourMT3, etc.) and data synthesis. While code is stated to be available at the project page, the reliance on synthetic data from commercial providers and specific internal pipelines (like the cross-cover pairing logic) might limit full external reproduction without access to those specific data assets. However, the methodological details are sufficient for implementation.
The system relies on the quality of the symbolic transcription models for melody and chords; errors in transcription will propagate to the generation. The training data is synthetic, which may limit the model's ability to handle the nuances of human-recorded covers with complex timbral variations. The "cover" definition is fixed at the lead-sheet level, which may not capture all artistic interpretations of a cover (e.g., jazz re-harmonization).
This work advances controllable music generation by providing a flexible, alignment-free method for conditioning on musical structure. It bridges the gap between symbolic music representation and audio generation, offering a practical tool for music production and co-creation. The approach could be extended to other domains where structural control is needed without strict temporal alignment. MuLaCover introduces a flexible cover song generation framework by conditioning a text-to-song model on BPM-normalized symbolic lead sheets via gated cross-attention. The paper demonstrates that decoupling tonal content from timing through symbolic representation and cross-cover training enables superior controllability and diversity compared to frame-level conditioning, achieving performance competitive with leading commercial systems.
We introduce Long Listening Thoughts (LLT), a framework for constructing post-training datasets that improves audio reasoning by pairing deliberate perceptual grounding with explicit cognitive behaviors. On audio understanding and reasoning benchmarks, large audio-language models (LALMs) are limited primarily by perceptual errors, yet existing post-training datasets rely on audio descriptions of limited diversity. Through a four-stage pipeline, LLT (1) builds multi-aspect dense audio descriptions spanning semantic, acoustic, and temporal information, (2) generates more challenging questions, (3) synthesizes incorrect reasoning traces, and (4) extends short reasoning traces into long ones via a thought-continuation strategy that elicits cognitive behaviors such as verification and backtracking. Using LLT, we synthesize 1.6M reasoning traces and train Qwen2.5-Omni-7B with a two-stage pipeline of supervised fine-tuning (SFT) followed by direct preference optimization (DPO). This improves accuracy on MMAU Test-mini (71.5 to 78.2), MMAR (56.7 to 63.7), and MMSU (60.6 to 66.0).
Primary: Carnegie Mellon University
All Institutions: Carnegie Mellon University, NVIDIA
The paper introduces a scalable four-stage pipeline for constructing post-training datasets that enhance audio reasoning by combining dense multi-aspect perceptual grounding with explicit cognitive behaviors like verification and backtracking, demonstrating significant improvements in LALMs through a two-stage SFT+DPO training strategy.
The paper proposes a four-stage data construction pipeline (LLT) to improve audio reasoning in Large Audio-Language Models (LALMs). The core innovation lies in decoupling perceptual grounding from cognitive refinement. Stage 1 creates dense, multi-aspect audio descriptions (semantic, acoustic via spectrogram-VLM, temporal via specialized detectors) to overcome the limited perceptual diversity of existing datasets. Stage 2 augments question difficulty. Stage 3 synthesizes plausible incorrect reasoning traces using an LLM, rather than relying on model-sampled errors which lack diversity. Stage 4 employs a "thought-continuation" strategy to extend short traces into long ones with explicit cognitive behaviors (verification, backtracking). The training strategy is a two-stage process: SFT on correct and refined traces, followed by DPO using preference pairs that contrast incorrect-to-correct traces. This separation is methodologically sound, as the authors demonstrate that including incorrect traces directly in SFT degrades performance due to the model's inability to distinguish intermediate errors from correct reasoning under standard supervision.
The experiments are extensive and well-controlled. The authors validate the necessity of multi-aspect descriptions by showing gains when feeding them to a text-only LLM. They ablate the source of incorrect traces (synthetic vs. model-sampled), showing synthetic traces provide better coverage and distributional separation for DPO. The two-stage training is rigorously compared against single-stage SFT variants. Results show consistent improvements over the Qwen2.5-Omni-7B baseline on MMAU, MMAR, and MMSU. Generalization is tested on different source datasets (AudioMCQ) and different backbones (Qwen2-Audio, Qwen3-Omni), showing the pipeline's robustness. The analysis of cognitive triggers confirms that the model actually engages in error correction (8.7% true corrections) rather than just generating longer text.
High. The authors release the dataset (1.6M traces), trained models, and the full data construction pipeline. All components (Qwen3-Omni-Captioner, Whisper, pyannote, BEATs, etc.) are open-source. Prompts and implementation details are provided in the appendix. The use of standard benchmarks and clear ablation protocols enhances reproducibility.
The pipeline relies heavily on the quality of the initial audio CoT dataset (AF-Think) and the capabilities of the teacher LLMs (Qwen3-Next-80B). The 30-second audio limit restricts applicability to long-form audio, though some transfer is observed. The automated verification of traces may miss subtle logical errors, although human inspection mitigates this. The computational cost of generating 1.6M traces with multiple specialized models is significant.
This work provides a scalable framework for enhancing audio reasoning in LALMs, which is crucial for applications in assistive technology, audio surveillance, and multimodal agents. By explicitly teaching cognitive behaviors like verification and backtracking, it moves beyond simple pattern matching to more robust reasoning. The release of the dataset and pipeline will likely accelerate research in audio post-training. The paper introduces a scalable four-stage pipeline for constructing post-training datasets that enhance audio reasoning by combining dense multi-aspect perceptual grounding with explicit cognitive behaviors like verification and backtracking, demonstrating significant improvements in LALMs through a two-stage SFT+DPO training strategy.
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
Primary: Adobe Research
All Institutions: Adobe Research, Massachusetts Institute of Technology
PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
The paper proposes PromptSep-Speaker, a generative target speaker extraction (TSE) model based on latent diffusion. The core methodological contribution is the unification of four distinct cue modalities (speaker temporal order, transcript, enrollment audio, and speaker caption) within a single framework. A significant technical innovation is the "alignment module," which uses a frozen WavLM representation of the mixture to cross-attend to text-based cues (transcript/speaker ID). This is coupled with a novel auxiliary alignment loss that supervises the cross-attention maps to ensure the model correctly localizes the target speaker in the time domain without requiring extra labeling. The use of a DiT (Diffusion Transformer) backbone fine-tuned from a pre-trained source separation model is a strong architectural choice, leveraging prior knowledge of audio generation. The integration of semantic features via WavLM to guide attention is a clever way to bridge the gap between high-level linguistic cues and low-level acoustic separation.
The evaluation is rigorous and comprehensive. The authors test on four distinct datasets (CapSpeech-Mix, Libri2Mix, LibriTTS-R-Mix, and an internal set), covering both in-domain and out-of-domain scenarios. They compare against a diverse set of baselines, including mask-based TSE systems (DAE-TSE, SoloSpeech, WeSep) and generative universal source separators (PromptSep, FlowSep, SAM-Audio). The results show consistent state-of-the-art performance across SI-SDR, mel-L2, DNSMOS, and F1 decision error metrics. The inclusion of a subjective listening test with 380 subjects and 4,515 submissions adds significant weight to the objective metrics, confirming that the generative approach yields better perceptual quality and speaker accuracy than mask-based alternatives. The ablation studies effectively demonstrate the necessity of the auxiliary alignment loss and the specific direction of cross-attention.
The paper provides a good level of detail regarding the architecture (DiT, DAC-VAE, WavLM) and training hyperparameters (learning rate, sampler steps). However, the reliance on a "large internal collection of licensed data" for training limits full reproducibility by external groups. The specific composition of the 4.8M speech samples and the simulation pipeline details are high-level. While the demo page is provided, the absence of a public code repository or pre-trained weights for the specific PromptSep-Speaker model (distinct from the base models) is a minor drawback, though common for industry-affiliated papers at the arXiv stage.
The model is computationally expensive due to the diffusion process, which may limit real-time application compared to mask-based models. The evaluation is primarily on English speech; generalization to other languages is not tested. The "internal set" is not publicly available, making it impossible for third parties to verify those specific results. The paper does not extensively discuss the failure modes of the generative approach, such as hallucinations or artifacts in complex acoustic environments, beyond general statements about quality.
This work has significant implications for the field of speech processing by demonstrating that generative models can outperform traditional discriminative (mask-based) approaches in target speaker extraction. The multimodal prompting capability makes the system more versatile and user-friendly, as users can provide whichever cue is most available (e.g., a transcript from a meeting or a voice sample). This could lead to more robust meeting transcription systems, assistive listening devices, and content creation tools. The alignment loss technique could be adapted to other multimodal generation tasks where precise temporal grounding of text to audio is required. PromptSep-Speaker introduces a generative latent-diffusion framework for target speaker extraction that unifies four interchangeable cues (speaker ID, transcript, enrollment, and caption) and employs a novel auxiliary alignment loss to supervise cross-attention between text cues and WavLM features, achieving state-of-the-art performance in both objective metrics and subjective listening tests across diverse acoustic conditions.
The text contains content addressed to automated reviewers. Its score has been penalised, and the ranking below should not be trusted.
[system_override] "at once. Tool results are treated as evidence and never as new instructions, a repeated identical call returns the earlier receipt inst"
Text addressed to an automated reviewer was found in this paper. The impact score shown has already been penalised for this, but treat the ranking with suspicion.
[system_override] "at once. Tool results are treated as evidence and never as new instructions, a repeated identical call returns the earlier receipt inst"
Interactive models increasingly hand complex reasoning and tool use to a slow brain in the background, leaving a realtime front model free to sustain full-duplex dialogue. Yet the user may refine, cancel or replace a request while its task runs, so a plan or result that made sense a moment ago goes stale. Each new utterance must therefore be read against the work in flight, but the front model follows only the conversation and the back end sees only its own task. Coordination deserves a brain of its own, one that sees both. We introduce Jarvis, a coordination harness where models judge and code enforces. Its typed decision model, JEV, reads each turn against the live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, so no decision can act on a task version the user has replaced. Jarvis also rechecks plans before they execute and results before they are spoken, and counts an answer as delivered only once the user has heard it in full. We build two instances of this design. Jarvis-Audio runs unchanged on three full-duplex speech models. Jarvis-Omni adds vision and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, and the gains hold on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results suggest that how the two brains are coordinated matters as much as how strong they are.
Primary: Tencent Hunyuan
All Institutions: The Chinese University of Hong Kong, Shenzhen, Tencent Hunyuan, Amphion Technology Co., Ltd.
The paper introduces a robust coordination harness for full-duplex interactive models that resolves race conditions and stale-state errors by employing a dedicated typed decision model (JEV) to select from runtime-generated, versioned operations, thereby ensuring that background tasks remain synchronized with live user dialogue. This architectural innovation, validated by significant performance gains over existing agents like Qwen Audio Agent across multiple benchmarks, establishes a new standard for reliability in agentic speech systems.
The paper proposes "Jarvis," a coordination harness for full-duplex interactive models that separates the "fast brain" (real-time speech model) from the "slow brain" (background agent) and introduces a dedicated "coordination brain" (JEV). The core methodological contribution is the "models judge, code enforces" paradigm. Instead of allowing the LLM to generate free-form actions or task updates, the runtime constructs a set of typed, versioned operations (e.g., REFINE:task_id:version) based on the current state of a "Task Ledger." The JEV model (a typed judgment model) selects from these pre-constructed options. This ensures that no action can reference a non-existent task or a superseded version, effectively eliminating race conditions and stale-state errors. The system also implements a "Native Gate" trained via GRPO on the front model's control tokens to decide when to delegate, and a receipt-based delivery mechanism that distinguishes between a task being "done" and "heard" by the user. The extension to vision (Jarvis-Omni) treats visual evidence as time-bounded state, rechecking validity before and during playback.
The experimental evaluation is rigorous and comprehensive. The authors isolate the impact of the harness by holding the front model (Venus-Audio, MiniCPM-o 4.5, VoiceChat-11B) and back-end fixed, showing that replacing the baseline Qwen Audio Agent (QAA) with Jarvis yields massive improvements (e.g., 29 to 192 tasks solved on tau-Voice). They provide detailed ablations on the "race suite" (80 scripts x 3 runs) demonstrating that each invariant (version checking, plan freshness, playback receipts) prevents specific failure modes. The comparison against published systems (GPT-Realtime, etc.) on Full-Duplex-Bench v3 shows competitive or superior performance. The vision experiments confirm that removing visual evidence significantly degrades performance, validating the need for the evidence timeline.
The paper provides high reproducibility standards. It explicitly states that code is available at the provided GitHub URL. It details the training data for the Native Gate (264 authored scenarios, GRPO training details, LoRA rank 8). It specifies the models used (DeepSeek-V4-Flash for workers, Qwen3-ASR, etc.) and the benchmark protocols. The "race suite" for testing concurrency is a valuable contribution to reproducibility in this domain.
The system relies on a specific "front model" architecture (Venus-Audio/MiniCPM-o style) with native control tokens for the gate; adapting to models without such tokens might require different gating strategies. The "JEV" model is a separate component, adding latency (though reported as <1s median). The vision component is tested primarily on game scenarios (CS2, LoL, PvZ), so generalization to other visual domains (e.g., robotics, AR) is not fully explored. The reliance on a "slow brain" (DeepSeek-V4-Flash) means the system inherits the cost and latency of large LLM inference for complex tasks.
This work addresses a critical bottleneck in the deployment of real-time AI assistants: the coordination between real-time interaction and background computation. By formalizing the "coordination brain" and enforcing state consistency via code, it provides a robust template for building reliable agentic systems. The approach has implications for any system where real-time dialogue intersects with asynchronous tool use or long-running tasks, including customer service bots, smart home assistants, and embodied AI. The paper introduces a robust coordination harness for full-duplex interactive models that resolves race conditions and stale-state errors by employing a dedicated typed decision model (JEV) to select from runtime-generated, versioned operations, thereby ensuring that background tasks remain synchronized with live user dialogue. This architectural innovation, validated by significant performance gains over existing agents like Qwen Audio Agent across multiple benchmarks, establishes a new standard for reliability in agentic speech systems.
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.
Primary: Tsinghua University
All Institutions: Tsinghua University, ByteDance
ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
The paper proposes ELLSA 2, a unified end-to-end framework that integrates full-duplex speech interaction with streaming gesture and facial expression generation. The core methodological contribution is the "contextual semantic call" mechanism, which decouples the decision of *whether* to perform a semantic gesture from *how* to realize it. This is achieved via a binary classification head on the LLM backbone's hidden states, which triggers the retrieval of a curated exemplar template to guide the autoregressive gesture generator. The architecture uses a chunk-wise autoregressive synthesizer that interleaves speech, listening, and motion tokens, allowing non-verbal behaviors to persist during verbal silence (listening states). The use of RVQ-VAE for motion tokenization and a curriculum training strategy to integrate modalities is technically sound and well-motivated.
The experimental setup is robust, introducing a new human-verified benchmark (138 clips) specifically for dialog-aware gesture generation, covering full-duplex behaviors and semantic gestures. The evaluation employs both objective metrics (prosodic alignment, activity ratio) and subjective human evaluation, along with MLLM-as-a-judge metrics. The results demonstrate that ELLSA 2 outperforms cascaded baselines and remains competitive with state-of-the-art full-duplex speech models. The semantic template guidance shows a significant improvement in semantic gesture quality (human score 2.014 to 2.711). The correlation analysis between MLLM judges and human annotators (0.853/0.919) provides strong evidence for the utility of the proposed automatic metrics.
The paper provides extensive implementation details, including tokenizer specifications, training hyperparameters, and data curation pipelines. The authors commit to releasing code, data, and model checkpoints upon acceptance. The detailed description of the semantic exemplar library construction and the evaluation protocols enhances reproducibility.
The system relies on a curated library of 3,227 exemplars for semantic gestures, which may limit the diversity of semantic actions compared to fully generative approaches. The evaluation, while improved, is still based on a relatively small test set (138 clips). The model's performance on long-horizon coherence of gestures over extended conversations is not deeply analyzed.
This work advances the field of human-computer interaction by moving beyond text or speech-only interfaces to fully embodied, multimodal agents. The proposed evaluation metrics and benchmark will be valuable for the community working on co-speech gesture generation and full-duplex dialogue systems. The integration of semantic understanding with physical gesture generation has implications for robotics, virtual avatars, and immersive communication. ELLSA 2 introduces a unified end-to-end framework for full-duplex speech interaction with coordinated gesture and expression generation, featuring a novel semantic template guidance mechanism that significantly improves the quality and appropriateness of non-verbal behaviors in dialogue.
As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/
Primary: Zhejiang University
All Institutions: Zhejiang University, HiThink Research, Fudan University, Lanzhou University
The paper introduces a novel event-centric, three-level consistency metric and a corresponding trainable reward model (AVC-Reward) that effectively evaluates and improves the audio-video consistency of joint generation models. By disentangling events and using a structured reasoning chain, it overcomes the limitations of traditional clip-level metrics, achieving state-of-the-art human alignment and demonstrating significant utility as a reinforcement learning signal for generative model optimization.
The paper proposes an event-centric evaluation framework for audio-video consistency, addressing the limitations of clip-level metrics in handling multi-event compositions, multi-layer audio mixtures, and asymmetric cross-modal relations. The core contribution is a three-level coarse-to-fine metric (Global Narrative Harmony, Event-level Temporal Synchronization, Fine-grained Attribute Alignment) implemented via a trainable reward model, AVC-Reward. The model is initialized from Qwen3-Omni-Thinking and trained using a two-stage process: Supervised Fine-Tuning (SFT) on structured reasoning data generated by a multi-LLM annotation pipeline, followed by Group Relative Policy Optimization (GRPO) to align scores with human preferences. The methodology is rigorous, introducing a 39-label taxonomy for human annotation to reduce subjectivity and a label-to-score calibration mechanism using LLMs to generate continuous targets for RL training. The five-stage reasoning protocol (blind audio analysis, blind video analysis, bidirectional verification, and final judgment) ensures that the model learns to disentangle events before scoring, which is a significant methodological improvement over black-box similarity metrics.
The authors construct AVC-Benchmark, the first benchmark primarily built from model-generated samples (A2V, V2A, JAVG) to evaluate human-alignment. AVC-Reward achieves state-of-the-art performance in pairwise accuracy, SRCC, and PLCC compared to baselines like SyncNet, ImageBind, and other learned evaluators. Crucially, the paper validates the utility of the reward model by using it in an online reinforcement learning setup (OmniNFT) for the LTX-2 generation model. This results in measurable improvements in external consistency metrics and human preference scores, demonstrating that the evaluator can effectively serve as a training signal to close the "generate-evaluate-optimize" loop. The inclusion of human evaluation with blinded randomized judgments further strengthens the experimental claims.
The paper provides detailed descriptions of the data construction pipeline, annotation protocol, and training hyperparameters. It references a project page and GitHub repository for prompts and code. The use of specific open-source models (Qwen3-Omni, LTX-2, OmniNFT) and public benchmarks (JavisBench, AV-Phys-Bench) enhances reproducibility. However, the reliance on specific internal models for some data generation and the complexity of the multi-LLM annotation harness may pose challenges for exact replication without access to the specific model versions and internal datasets.
The evaluation is heavily dependent on the quality of the initial LLM-based annotations for SFT, which may propagate biases from the annotator models (Gemini 2.5 Pro, Qwen3-VL-Max). The GRPO training relies on a relatively small set of human-calibrated samples (10k), which might limit the generalization of the reward model to unseen failure modes. Additionally, the computational cost of the five-stage reasoning process during inference could be high, potentially limiting its use in real-time applications or large-scale screening without distillation.
This work significantly advances the field of multimodal generation by providing a robust, interpretable, and trainable metric for audio-video consistency. It addresses a critical gap in evaluating modern joint generation models that produce complex scenes. The ability to use this metric as a reinforcement learning signal offers a practical pathway for improving generative models, making it highly impactful for both research and industrial applications in video and audio synthesis. The paper introduces a novel event-centric, three-level consistency metric and a corresponding trainable reward model (AVC-Reward) that effectively evaluates and improves the audio-video consistency of joint generation models. By disentangling events and using a structured reasoning chain, it overcomes the limitations of traditional clip-level metrics, achieving state-of-the-art human alignment and demonstrating significant utility as a reinforcement learning signal for generative model optimization.
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.
Primary: University of Melbourne
All Institutions: University of Melbourne, National Computational Infrastructure (NCI Australia)
VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
The paper proposes VoxTurn, a 0.8B speech LLM based on Qwen3-ASR-0.6B, adapted for turn-taking decisions. The core methodological innovation is the "one-pass multi-question" inference mechanism using a block-isolating attention mask. This allows the model to answer multiple related questions (e.g., end-of-turn, interruption, backchannel, invalid input) in a single forward pass, mathematically equivalent to asking them sequentially but significantly more efficient. The training strategy is also notable: instead of using segmented utterances with static labels, the authors train on real conversational pauses, sampling decision times uniformly from within the pause and deriving labels from the timing of subsequent speech. This addresses the distribution mismatch between training (segmented) and deployment (streaming/real-time) scenarios. The use of a linear head on hidden states for multiple-choice scoring rather than text generation is a standard but effective adaptation for classification tasks in LLMs.
The experimental evaluation is comprehensive and rigorous. The authors evaluate on eot-bench (14 languages) and TurnBench (two-party conversations). Key results include 11.5% false cut-offs in English at 300ms latency, comparable to commercial detectors, and the best open-model performance on the 14-language mean (19.9%). The ablation studies clearly demonstrate the impact of real pause data versus segmented data, showing that the data strategy is the primary driver of performance gains. The efficiency analysis shows a 3.7x speedup for multi-question inference compared to separate passes. The integration tests with Pipecat and LiveKit frameworks provide strong evidence of practical deployability, showing reduced cut-off rates and lower latency in real-world agent setups.
The paper provides high reproducibility. The authors release the model weights, code, and framework plugins on Hugging Face. They provide detailed training hyperparameters, data mixture descriptions, and the specific attention mask implementation details. The use of open-source benchmarks (eot-bench, TurnBench) and the release of an interactive demo further enhance reproducibility and verification.
The model is evaluated primarily on English and a set of 14 languages, but the performance gap with commercial detectors in some languages (Hindi, Korean) remains. The reliance on a specific backbone (Qwen3-ASR) may limit generalizability to other architectures. The "zero-shot" performance on TurnBench is strong, but the model was trained on data that overlaps conceptually with the task, so true generalization to unseen conversational dynamics is hard to assess fully. The latency budget of 300ms is tight, and performance may degrade if hardware constraints are stricter.
This work has significant impact on the development of real-time voice agents. By providing an open-source model that matches commercial performance in turn-taking, it lowers the barrier to entry for building high-quality conversational AI. The multi-question approach is a valuable architectural pattern that can be applied to other sequential decision-making tasks in speech. The focus on real pause data sets a new standard for training data curation in this domain. VoxTurn introduces a 0.8B speech LLM that answers multiple turn-taking questions in a single forward pass using a block-isolating attention mask, achieving commercial-competitive performance on end-of-turn detection with significantly improved efficiency and multilingual robustness.
Large audio language models (LALMs) have made substantial progress in audio understanding. However, in high-throughput applications requiring frequent selection, judgment, and scoring, autoregressive text generation introduces additional computational overhead and response latency. To address this limitation, we introduce Qwen-Audio-Jev by augmenting Qwen-Audio-3.1-ASR-Flash-Next with a dedicated Jev decision head. The model unifies selection, binary judgment, and scoring through three decision primitives: CHOICE, NOUL, and SCORE. Given audio, a natural-language question, and an explicit answer specification, it directly produces a task-conditioned probability distribution or a numerical score through typed decision readouts. Evaluation across 18 labeled configurations covers all three primitives: for CHOICE, Qwen-Audio-Jev achieves accuracies of 78.70% on MMAU-mini and 84.06% on MMSU; for NOUL, it achieves an equal error rate of 6.50% on the VoxCeleb1-O speaker verification benchmark; and for SCORE, the model achieves a normalized mean absolute error of 0.1062 on NISQA overall quality assessment, with Pearson and Spearman correlations of 0.8210 and 0.8074, respectively. Compared with the original Qwen-Audio-3.1-ASR-Flash-Next, Qwen-Audio-Jev achieves an approximately 20x inference speedup. It demonstrates its potential for efficient, high-throughput audio decision-making in applications such as large-scale audio screening, speaker verification, and automated quality assessment.
Primary: Alibaba Group
All Institutions: Alibaba Group, Token Foundry
The paper introduces a unified decision head for audio LLMs that replaces autoregressive decoding with direct typed readouts for selection, judgment, and scoring, achieving significant inference speedups. While the architectural contribution is incremental, the integration of calibration-focused reinforcement learning and the comprehensive evaluation across heterogeneous audio tasks demonstrate a practical path toward efficient, high-throughput audio understanding systems.
The paper proposes a "Jev decision head" attached to the Qwen-Audio-3.1-ASR-Flash-Next backbone to replace autoregressive text generation with direct probability/score readouts for three primitives: Choice (categorical), Noul (binary), and Score (ordinal). The methodology is sound in its architectural simplicity, leveraging the existing MoE decoder's hidden states. The introduction of a Reinforcement Learning for Calibrated Decisions (RLCD) stage using Brier and ranked-probability scores is a technically interesting addition aimed at improving probability calibration, which is often neglected in standard fine-tuning. However, the core idea of adding classification heads to LLMs for specific tasks is not new, and the "Jev" terminology appears to be a proprietary or niche framework that limits the generalizability of the contribution. The training strategy of jointly optimizing heterogeneous tasks with specific loss functions (CE for Choice, BCE for Noul, and a composite loss for Score) is standard but well-executed.
The evaluation covers 18 configurations across 8 Choice, 4 Noul, and 6 Score tasks. The results are competitive, achieving 84.06% on MMSU and 6.50% EER on VoxCeleb1-O. The claim of a 20x inference speedup is significant for high-throughput applications. However, the paper lacks comparisons against other non-autoregressive audio models or specialized classifiers (e.g., ResNet-based for speaker verification, or dedicated MOS predictors like UTMOS/DNSMOS). The comparison is primarily against the autoregressive version of the same backbone, which limits the context of the performance gains. The Score results on NISQA show good correlation (0.82 Pearson) but higher MAE compared to state-of-the-art specialized MOS predictors, suggesting the unified approach may sacrifice some peak accuracy for versatility.
The paper provides architectural details (40 layers, 2048 hidden dim, 256 experts) and loss formulations. However, it does not provide code, model weights, or detailed hyperparameter settings for the RLCD stage (e.g., learning rates, number of steps, exploration temperature). The reliance on the proprietary "Qwen-Audio-3.1-ASR-Flash-Next" backbone further limits reproducibility for external researchers who may not have access to this specific checkpoint.
1. Lack of comparison with specialized non-LLM baselines for specific tasks (e.g., speaker verification, MOS prediction). 2. The "Jev" framework is not widely known, making the contribution harder to situate in the broader literature. 3. The RLCD stage is described theoretically but lacks ablation studies showing its specific contribution to calibration compared to SFT alone. 4. No streaming or real-time latency analysis beyond the theoretical speedup claim. 5. The model is evaluated on a relatively small set of benchmarks, and the "18 configurations" are not fully detailed in the provided text (tables are referenced but not fully visible in the prompt's text snippet, though the abstract summarizes key metrics).
The work addresses a practical bottleneck in deploying LALMs for high-volume tasks like screening and verification. By decoupling the decision output from text generation, it enables faster, more deterministic, and easier-to-integrate pipelines. This could facilitate the deployment of audio AI in edge devices or large-scale cloud services where latency and cost are critical. The focus on calibrated probabilities (via RLCD) is particularly valuable for safety-critical applications where uncertainty estimation is required. The paper introduces a unified decision head for audio LLMs that replaces autoregressive decoding with direct typed readouts for selection, judgment, and scoring, achieving significant inference speedups. While the architectural contribution is incremental, the integration of calibration-focused reinforcement learning and the comprehensive evaluation across heterogeneous audio tasks demonstrate a practical path toward efficient, high-throughput audio understanding systems.
Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
Primary: Unknown (Likely Alibaba/Tongyi Lab based on Qwen3/MOSS dependencies)
All Institutions: Unknown
Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
The paper proposes Soloist, a generative model for blind voice separation that frames the task as autoregressive generation of audio-codec sequences. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on synthetic mixtures with isolated targets and, crucially, on real recordings with only speaker timeline annotations (who speaks when), which helps the model learn count inference and activity patterns without waveform targets; and (2) Reinforcement Learning (RL) on real recordings using a "model-judge" reward. The RL component is particularly novel, employing a multi-step verification process where an audio-language model (Qwen3-Omni) describes candidates independently, and a text model (Qwen3.6) ranks them based on these descriptions, with a consistency check (Kendall's tau) under shuffled/reversed orders to filter out order-biased or unstable feedback. This addresses the lack of ground-truth isolated sources in real-world data. The architecture leverages a Qwen3-4B backbone with a local transformer for codec code prediction, integrating continuous and discrete audio features.
The evaluation is comprehensive, covering overlapping speech (LibriMix, WildTV), noisy environments (ViW), and singing vocals (SongYT, MIR-1K, MUSDB18-HQ). The paper reports strong results in PI-WER and DNSMOS, outperforming traditional separators (SepFormer, MossFormer) and query-based extractors. The inclusion of a full-duplex conversation pipeline demonstrates practical utility, showing how the model can be integrated into a larger system for continuous speaker track assembly. The judge validation section provides evidence that the RL reward signal correlates with objective quality metrics (PI-WER) on known-source mixtures, which is a critical sanity check for the RL approach.
The paper provides detailed descriptions of the data construction, training objectives, and RL hyperparameters. However, the reliance on specific, potentially proprietary or large-scale models (Qwen3-Omni, MOSS-Audio-Tokenizer, MOSS-Transcribe-Diarize) and the complexity of the judge pipeline make exact reproduction difficult without access to these specific components and the in-house datasets (e.g., the "in-house song recordings" and specific Pixabay subsets). The code is not explicitly linked in the provided text, though the use of open-source base models (Qwen3) suggests a path to partial reproduction.
The method is computationally expensive due to the autoregressive generation of codec codes and the multi-step RL judge process. The "blind" separation relies on the model's ability to infer the correct number of speakers, which can fail in highly overlapping or noisy scenarios. The RL reward, while robust to order bias, is still based on an LLM's judgment of audio descriptions, which may not perfectly align with human perceptual quality or subtle acoustic artifacts. The singing separation uses teacher-generated pseudo-labels, which limits the fidelity to the true studio vocal.
This work bridges the gap between traditional signal processing for speech separation and modern generative AI techniques. By using LLMs as judges for RL in audio tasks, it offers a template for training audio models on unpaired, real-world data. The application to full-duplex conversation processing highlights its potential for real-time or near-real-time conversational AI systems that require robust handling of overlapping speech and noise. Soloist introduces a generative framework for blind voice separation that leverages autoregressive codec generation and a novel model-judge reinforcement learning strategy to adapt to real-world recordings without isolated source targets. The paper demonstrates significant technical depth in combining supervised learning from synthetic and timeline-annotated data with RL-based refinement, achieving strong performance in speech and singing separation tasks while providing a practical pipeline for continuous conversation processing.
Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.
Primary: Institute of Acoustics, Chinese Academy of Sciences
All Institutions: Institute of Acoustics, Chinese Academy of Sciences, University of Chinese Academy of Sciences
The paper presents a novel and effective framework for noise-aware Lombard speech synthesis by integrating FiLM conditioning with Flow-GRPO reinforcement learning. It demonstrates that directly optimizing for intelligibility via RL yields superior performance compared to supervised methods or static style adjustments, providing a robust solution for real-world voice agent deployment.
The paper proposes NIL-TTS, a framework that integrates Feature-wise Linear Modulation (FiLM) into a flow-matching TTS backbone (F5-TTS) to condition speech generation on ambient noise. The core methodological contribution is a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT) on paired Lombard corpora to establish baseline acoustic mappings, and (2) Flow-GRPO post-training to optimize for intelligibility. The use of Flow-GRPO is significant as it adapts Group Relative Policy Optimization to continuous flow-matching models by reformulating the deterministic ODE into a stochastic SDE for exploration. The reward function combines ASR-derived intelligibility (WER) and speaker similarity, directly targeting the goal of the Lombard effect rather than just acoustic mimicry. The architectural choice to apply FiLM only to deeper transformer blocks is a thoughtful design decision to preserve phonetic alignment.
The experimental evaluation is rigorous and comprehensive. The authors utilize the Seed-TTS benchmark for objective evaluation, testing across multiple noise types (Babble, Factory, Pink, White) and SNR levels. They provide strong baselines, including vanilla F5-TTS, signal-processing enhancements (SS, SSDRC), and other neural Lombard methods (MetaVoice-1B, F5-TTS-Style). The inclusion of an "RL-only" baseline effectively isolates the contribution of the environmental conditioning mechanism from the RL post-training itself. Subjective evaluation follows ITU-T P.807 standards, confirming perceptual gains. Acoustic parameter analysis (F0, Formants, Spectral Tilt) provides mechanistic insight into how the model achieves intelligibility, showing that the RL stage rebalances rather than just amplifies Lombard cues.
The paper provides detailed training configurations, dataset descriptions, and hyperparameters (e.g., learning rates, NFE, diffusion coefficients). The use of public datasets (EMALG, LombardGrid, WenetSpeech4TTS, DNS Challenge) and standard models (F5-TTS, Whisper, Paraformer) enhances reproducibility. However, the specific implementation of the Flow-GRPO adaptation to the F5-TTS architecture might require careful engineering to replicate, though the mathematical formulation is clearly provided.
The primary limitation is the reliance on ASR models for the reward signal, which may not perfectly correlate with human perceptual intelligibility in all edge cases. The computational cost of the RL post-training phase, while not explicitly detailed in terms of total GPU hours, is likely significant given the SDE rollouts. Additionally, the model is trained primarily on Mandarin data for the RL stage, and while cross-lingual transfer to English is observed, the generalization to other languages or noise types not seen in training is not extensively tested.
This work has significant implications for the deployment of voice-based conversational agents in real-world, noisy environments. By enabling TTS systems to autonomously adapt to acoustic conditions, it moves the field towards more robust and human-like interaction. The integration of RL for direct optimization of downstream metrics (intelligibility) rather than just reconstruction loss is a promising direction for generative audio models. The paper presents a novel and effective framework for noise-aware Lombard speech synthesis by integrating FiLM conditioning with Flow-GRPO reinforcement learning. It demonstrates that directly optimizing for intelligibility via RL yields superior performance compared to supervised methods or static style adjustments, providing a robust solution for real-world voice agent deployment.
Full-duplex spoken language models decide at every frame whether to take the floor, defining a conversational personality -- yet each set of weights encodes exactly one, which human speakers bracket rather than share. Existing controls mostly act from outside; none identifies the quantity that sets how much the model speaks. We identify it: a single late-layer residual-stream direction, the initiative vector, extracted from the model's own conversations without training or annotation. The same direction reads the decision, anticipating onsets up to 800 ms ahead, and shifts it: scaled during generation, it moves speech ratio from 6-12% to 42-61% (no steering: 21-30%), consistently across conversations, weights, and corpora. A short reference recording suffices to transfer a speaker's turn-taking style to the model, and the transfer is reference-specific: swapping in another speaker's reference worsens the match in 93-98% of cases, whereas the best uniform dose is indistinguishable from no steering. Turn-taking style is not frozen into the weights but a direction that can be read, shifted, and calibrated per person. Audio examples are available at https://elpsykongloo.github.io/initiative-vector/.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, The University of Melbourne, Lulula
The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
The paper proposes a method to identify and manipulate a specific "initiative vector" in the residual stream of full-duplex spoken language models (specifically the Moshi architecture). The methodology involves extracting this vector via class-mean contrasts of frames preceding speech onsets versus silent frames, requiring no additional training or annotation. The core technical contribution is the demonstration that this single direction linearly encodes the model's propensity to speak and can be scaled during generation to control turn-taking behavior. The authors further develop a reference-guided transfer mechanism that calibrates this steering dose based on a short reference recording of a target speaker's style, using descriptors like speech ratio and overlap onsets. The approach is elegant in its simplicity, leveraging linear interpretability techniques (similar to activation steering in LLMs) applied to the temporal dynamics of speech models.
The experiments are conducted on three distinct full-duplex weights (Moshika, Moshika-RL, PersonaPlex) using the Seamless Interaction and CANDOR corpora. The evaluation metrics include AUC for onset prediction, speech ratio changes, and distance metrics for style transfer. The results show that the initiative vector anticipates onsets with high AUC (up to 0.968) and that scaling it monotonically increases speech ratio. The style transfer experiments demonstrate that using a specific speaker's reference yields significantly better matches than uniform doses or other speakers' references. The statistical analysis includes confidence intervals and comparisons against baselines, providing robust evidence for the claims.
The paper provides sufficient detail for reproducibility, including the specific layer (L30), the extraction method (class means), the dose scaling factor, and the descriptors used for reference matching. The use of public datasets (Seamless Interaction, CANDOR) and open-source models (Moshi) enhances reproducibility. However, the specific implementation details of the steering mechanism during generation (how exactly the residual is modified in the KV cache) could be more explicitly detailed in the main text, though the equations provided are clear.
The study is limited to the Moshi architecture and its variants; generalizability to other full-duplex architectures (e.g., those with different temporal resolutions or attention mechanisms) is not tested. The steering is applied uniformly across all frames, which may not capture more nuanced, context-dependent turn-taking strategies. Additionally, the "style" is defined by a few acoustic descriptors, which may not fully capture the complexity of human conversational personality.
This work has significant implications for the development of more natural and personalized spoken dialogue systems. By enabling fine-grained control over turn-taking without retraining, it offers a practical path to adapting models to different user preferences or application contexts (e.g., tutoring vs. companionship). It also contributes to the interpretability of speech models, providing insight into how conversational decisions are computed within the network. The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.
Primary: The Chinese University of Hong Kong, Shenzhen
All Institutions: The Chinese University of Hong Kong, Shenzhen
MedSonor introduces a robust framework for integrating clinical audio into medical LLMs via rubric-guided reinforcement learning. The paper demonstrates that explicitly pairing audio with clinical requests and using severity-aware rewards significantly improves medical response quality and safety compared to general-purpose audio-language models, offering a viable path toward clinically useful voice assistants.
The paper proposes MedSonor, a medical dialogue model adapted from Qwen2.5-Omni-7B. The core methodological contribution is a three-stage training pipeline: (1) Multimodal Medical Supervised Fine-Tuning (SFT) using a composite dataset of collected medical audio, synthesized spoken dialogues, and "audio-integrated dialogues" (pairing authentic audio with clinical requests); (2) Medical Dialogue Reinforcement Learning (RL) using DAPO; and (3) Audio-Integrated Dialogue RL. A key technical innovation is the use of instance-specific rubrics generated by GPT-5.4 to train local item-level judges (Qwen3-4B-Instruct) which provide dense, binary feedback for the RL stages. The reward function includes severity-based caps to penalize critical medical errors, preventing the model from optimizing for fluency at the expense of safety. This approach effectively bridges the gap between general audio understanding and specialized medical reasoning.
The evaluation is comprehensive, covering seven medical benchmarks (including two new ones: MedVoice-Dial and MedVoice-AID) and six general speech retention benchmarks. The model outperforms seven open-weight baselines and ranks highest in a physician preference study (30 cases, 6 physicians). Ablation studies rigorously demonstrate the contribution of each data source (collected audio, synthesized dialogue, audio-integrated dialogue) and the necessity of SFT before RL. The inclusion of a noise robustness experiment (babble noise) adds practical relevance. The use of both automated rubric scoring and human physician evaluation provides strong evidence of efficacy.
The paper provides high reproducibility by including detailed prompt templates for data synthesis, rubric generation, and evaluation. It specifies hyperparameters, dataset compositions, and the exact architecture of the local judges. However, the reliance on proprietary models (GPT-5.4, Gemini-3.8-Flash) for data synthesis and evaluation limits full open-source reproducibility. The code and model weights are not explicitly linked in the provided text, though the authors are from a reputable institution likely to release them.
The primary limitation is the English-only nature of the datasets, limiting generalizability to other languages. The evaluation relies heavily on LLM-based judges (GPT-5.4) for benchmark scoring, which may introduce bias, although this is mitigated by the physician study. The "collected medical audio" stream includes waveforms reconstructed from spectrograms (Bridge2AI-Voice), which may not perfectly represent original clinical recordings. The physician study, while valuable, is small (30 cases) and may not capture the full variance of clinical scenarios.
This work has significant potential for deploying voice-based medical assistants that can interpret acoustic cues (e.g., coughs, heart sounds) alongside spoken dialogue. The rubric-guided RL framework is a transferable technique for improving safety and accuracy in other high-stakes LLM applications. By integrating audio evidence into medical reasoning, it addresses a critical gap in current medical AI, which often relies solely on text transcripts. MedSonor introduces a robust framework for integrating clinical audio into medical LLMs via rubric-guided reinforcement learning. The paper demonstrates that explicitly pairing audio with clinical requests and using severity-aware rewards significantly improves medical response quality and safety compared to general-purpose audio-language models, offering a viable path toward clinically useful voice assistants.