MedSonor: Learning to Use Clinical Audio in Medical LLMs
Abstract
Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MedSonor introduces a robust framework for integrating clinical audio into medical LLMs via rubric-guided reinforcement learning. The paper demonstrates that explicitly pairing audio with clinical requests and using severity-aware rewards significantly improves medical response quality and safety compared to general-purpose audio-language models, offering a viable path toward clinically useful voice assistants.
Methodology
The paper proposes MedSonor, a medical dialogue model adapted from Qwen2.5-Omni-7B. The core methodological contribution is a three-stage training pipeline: (1) Multimodal Medical Supervised Fine-Tuning (SFT) using a composite dataset of collected medical audio, synthesized spoken dialogues, and "audio-integrated dialogues" (pairing authentic audio with clinical requests); (2) Medical Dialogue Reinforcement Learning (RL) using DAPO; and (3) Audio-Integrated Dialogue RL. A key technical innovation is the use of instance-specific rubrics generated by GPT-5.4 to train local item-level judges (Qwen3-4B-Instruct) which provide dense, binary feedback for the RL stages. The reward function includes severity-based caps to penalize critical medical errors, preventing the model from optimizing for fluency at the expense of safety. This approach effectively bridges the gap between general audio understanding and specialized medical reasoning.
Results & evaluation
The evaluation is comprehensive, covering seven medical benchmarks (including two new ones: MedVoice-Dial and MedVoice-AID) and six general speech retention benchmarks. The model outperforms seven open-weight baselines and ranks highest in a physician preference study (30 cases, 6 physicians). Ablation studies rigorously demonstrate the contribution of each data source (collected audio, synthesized dialogue, audio-integrated dialogue) and the necessity of SFT before RL. The inclusion of a noise robustness experiment (babble noise) adds practical relevance. The use of both automated rubric scoring and human physician evaluation provides strong evidence of efficacy.
Reproducibility
The paper provides high reproducibility by including detailed prompt templates for data synthesis, rubric generation, and evaluation. It specifies hyperparameters, dataset compositions, and the exact architecture of the local judges. However, the reliance on proprietary models (GPT-5.4, Gemini-3.8-Flash) for data synthesis and evaluation limits full open-source reproducibility. The code and model weights are not explicitly linked in the provided text, though the authors are from a reputable institution likely to release them.
Limitations
The primary limitation is the English-only nature of the datasets, limiting generalizability to other languages. The evaluation relies heavily on LLM-based judges (GPT-5.4) for benchmark scoring, which may introduce bias, although this is mitigated by the physician study. The "collected medical audio" stream includes waveforms reconstructed from spectrograms (Bridge2AI-Voice), which may not perfectly represent original clinical recordings. The physician study, while valuable, is small (30 cases) and may not capture the full variance of clinical scenarios.
Broader impact
This work has significant potential for deploying voice-based medical assistants that can interpret acoustic cues (e.g., coughs, heart sounds) alongside spoken dialogue. The rubric-guided RL framework is a transferable technique for improving safety and accuracy in other high-stakes LLM applications. By integrating audio evidence into medical reasoning, it addresses a critical gap in current medical AI, which often relies solely on text transcripts. MedSonor introduces a robust framework for integrating clinical audio into medical LLMs via rubric-guided reinforcement learning. The paper demonstrates that explicitly pairing audio with clinical requests and using severity-aware rewards significantly improves medical response quality and safety compared to general-purpose audio-language models, offering a viable path toward clinically useful voice assistants.
Author of this paper? Send a correction.