Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction
Abstract
Extracting patient medical conditions from code-switched clinical spoken dialogues is challenging due to rapid turn-taking and highly overlapped speech. We present a robust system evaluated on the DISPLACE-M dataset of real-world Hinglish medical conversations. We propose an End-to-End Neural Diarization with Vector Clustering approach (EEND-VC) to accurately resolve dense and speaker overlaps in Doctor-Patient Conversations (DoPaCo). For transcription, we adapt a Qwen3 ASR model via domain-specific fine-tuning, Devanagari script normalization, and dialogue-level LLM error correction, achieving an 18.59% tcpWER. We benchmark open and proprietary LLMs on medical condition extraction, comparing our text-based cascade system against a multimodal End-to-End (E2E) audio framework. While proprietary E2E models set the performance ceiling, our open cascaded architecture is highly competitive, as it achieved first place out of 25 participants in the DISPLACE-M challenge. All implementations are publicly released.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper contributes a novel and effective pipeline for extracting medical conditions from code-switched Hinglish dialogues, addressing a critical gap in the field of speech processing. The methodology is innovative, and the results demonstrate substantial technical impact, making it a valuable addition to the literature on multilingual and code-switched speech recognition.
Methodology
The paper presents a well-structured methodology that integrates an End-to-End Neural Diarization with Vector Clustering (EEND-VC) approach tailored for the complexities of code-switched Hinglish medical dialogues. The use of a domain-adapted ASR model (Qwen3) and the innovative error correction mechanism utilizing dialogue-level LLMs demonstrates a thoughtful approach to addressing the challenges posed by overlapping speech and spontaneous dialogue. The modular architecture allows for flexibility and adaptability as new models emerge, which is a significant advantage in the rapidly evolving field of speech processing.
Results & evaluation
The experimental setup is robust, utilizing the DISPLACE-M dataset, which is specifically designed for the task at hand. The authors benchmark their system against both open-source and proprietary models, providing a comprehensive analysis of performance metrics. Achieving first place in the DISPLACE-M challenge adds credibility to their results. However, the paper could benefit from more detailed ablation studies to further clarify the contributions of each component in their pipeline.
Reproducibility
The authors mention that all implementations are publicly released, which is a positive aspect for reproducibility. However, the paper lacks specific URLs or links to the code repositories, which would enhance accessibility for other researchers looking to replicate or build upon this work.
Limitations
The paper does not extensively discuss the limitations of the proposed methods, particularly concerning the potential biases in the training data or the challenges of generalizing the approach to other languages or dialects. Additionally, while the results are promising, the reliance on proprietary models for benchmarking may limit the applicability of findings to open-source solutions.
Broader impact
The research has significant implications for healthcare applications, particularly in regions where code-switching is prevalent. By improving the extraction of medical conditions from dialogues, the work could enhance patient care and facilitate better communication between healthcare providers and patients. The public release of their system also encourages further research and development in this area. This paper contributes a novel and effective pipeline for extracting medical conditions from code-switched Hinglish dialogues, addressing a critical gap in the field of speech processing. The methodology is innovative, and the results demonstrate substantial technical impact, making it a valuable addition to the literature on multilingual and code-switched speech recognition.
Author of this paper? Send a correction.