When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition
Abstract
Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the systematic evaluation of audio-language models' ability to leverage clinical context for dysarthric speech recognition, revealing critical insights into their limitations and potential for improvement. The comprehensive analysis of methodology, experiments, and implications underscores its significance in advancing research in accessible speech recognition technologies.
Methodology
The paper introduces a systematic evaluation protocol for assessing the effectiveness of clinical context in improving dysarthric speech recognition using audio-language models. The methodology is robust, employing a variety of prompt conditions and a comprehensive analysis of nine different models. The use of LoRA for context-dependent fine-tuning is a notable innovation, allowing the model to adapt to clinical prompts effectively. The authors provide detailed descriptions of their experimental setup, including the hierarchical structure of prompts and the clinical profiles used, which enhances the reproducibility of their findings.
Results & evaluation
The experiments are well-designed, utilizing the Speech Accessibility Project (SAP) dataset, which is the largest dysarthric speech corpus available. The evaluation includes zero-shot baselines and a thorough analysis of the models under different prompting conditions. The results are statistically significant, revealing that most models do not leverage clinical context effectively, which is a critical finding for the field. The paper also includes subgroup analyses that highlight specific gains for certain speaker populations, adding depth to the evaluation.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including the training and evaluation protocols, which should allow for reproducibility. However, the lack of publicly available code or models limits the immediate reproducibility of the results. The authors mention hardware specifications and training configurations, which is helpful for others looking to replicate their work.
Limitations
The authors acknowledge several limitations, including the reliance on a single dataset, the coarse nature of clinical annotations, and the underrepresentation of severe cases in their analyses. They also note that the clinical profiles used may not capture the full variability present in dysarthric speech, which could affect the generalizability of their findings.
Broader impact
The findings have significant implications for the development of more inclusive automatic speech recognition systems, particularly for individuals with dysarthria. By identifying the shortcomings of current models in utilizing clinical context, the paper paves the way for future research aimed at improving accessibility in voice-based technologies. The ethical considerations discussed regarding the use of clinical labels also highlight the need for sensitivity in deploying such technologies. The main contribution of this paper is the systematic evaluation of audio-language models' ability to leverage clinical context for dysarthric speech recognition, revealing critical insights into their limitations and potential for improvement. The comprehensive analysis of methodology, experiments, and implications underscores its significance in advancing research in accessible speech recognition technologies.
Author of this paper? Send a correction.