CARE: A Multimodal Corpus for Studying Speech and Non-Verbal Communication Across Multiple Medical Conditions
Abstract
Automatic analysis of multimodal speech has shown strong potential for computationally detecting and monitoring a wide range of neurological, psychiatric, and respiratory conditions. However, progress in this field is limited by existing publicly accessible datasets, which are often small in scale, focused on a single condition or disease, and primarily speech focused. Moreover, if key confounding variables such as education, medication use, comorbidities, or mood state are insufficiently documented, the reliability and interpretability of computational analyses are further compromised. To address these limitations, we introduce CARE v1.0, a curated multimodal English dataset of approximately 144 hours of short video interviews collected from 612 individuals across 12 medical conditions plus a control cohort. For each video, a comprehensive set of clinically relevant multimodal descriptors is provided, alongside structured metadata covering factors such as medication, life impacts, and expressed emotions. The corpus's breadth and heterogeneity support a wide range of applications, including automatic disease and symptom detection, multimodal modelling of speech and non-verbal behaviour under emotionally charged contexts, and studies of disease trajectories and coping processes.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Methodology
The paper introduces CARE v1.0, a large-scale multimodal corpus derived from the HEXI platform. The methodology focuses on data curation and feature engineering rather than novel algorithmic development. The authors employ a rigorous pipeline for extracting multimodal descriptors: audio features (eGeMAPS, DisVoice, XLS-R, TRILLsson), visual features (OpenFace for facial landmarks/AUs, gaze, head pose, MediaPipe for body keypoints), and linguistic features (WhisperX alignment, sentence embeddings). A key methodological contribution is the use of LLMs (gpt-oss-120b) to extract structured metadata (demographics, emotions, comorbidities) from unstructured narrative texts, validated against a manually annotated subset. This approach addresses a significant gap in existing datasets by providing rich, clinically relevant metadata alongside raw multimodal signals.
Results & evaluation
The experimental section is primarily descriptive, focusing on the statistical properties of the dataset (distribution of conditions, demographics, emotion frequencies) and the validation of the LLM-based metadata extraction pipeline. The authors compare five LLMs for metadata extraction, demonstrating that gpt-oss-120b performs best in terms of accuracy and adherence to constraints. They provide a baseline evaluation workflow for a binary classification task (Control vs. Parkinson's) using an SVM on aggregated features. While this demonstrates the utility of the dataset, the lack of extensive benchmarking against state-of-the-art multimodal models limits the assessment of the dataset's ability to drive performance improvements in specific tasks. The validation of the metadata extraction is solid, but the downstream ML performance is only illustrative.
Reproducibility
The paper excels in reproducibility. The dataset is hosted on Hugging Face, and the code for feature extraction and the example evaluation workflow is publicly available. The detailed documentation of the feature extraction pipeline, including specific tools (OpenFace, DiariZen, WhisperX) and parameter settings, allows other researchers to replicate the feature computation. The use of deterministic settings for LLM inference (temperature=0) further ensures reproducibility of the metadata annotations.
Limitations
The dataset is skewed towards White UK/Irish participants, limiting its generalizability to other ethnic groups. The control group is heterogeneous (caregivers, bereaved, professionals), which may introduce confounding variables if not carefully handled in downstream analyses. The reliance on LLMs for metadata extraction, while validated, introduces potential biases and hallucinations (though rare, as noted). The dataset is cross-sectional, limiting longitudinal studies of disease trajectories, although the authors mention this as a potential future application. The "in-the-wild" nature of some video clips (varying quality, lighting) may pose challenges for robust computer vision models.
Broader impact
CARE v1.0 has significant potential impact on the field of digital health and affective computing. By providing a large, multimodal, and clinically annotated dataset, it enables research into automated detection and monitoring of a wide range of neurological, psychiatric, and respiratory conditions. The inclusion of diverse medical conditions and rich metadata supports the development of more robust and interpretable computational models. The dataset can facilitate studies on the interplay between speech, non-verbal behavior, and emotional states in health contexts, contributing to a better understanding of disease manifestations and coping processes. This paper presents CARE v1.0, a comprehensive multimodal corpus of 144 hours of video interviews from 612 individuals across 12 medical conditions, enriched with automatically extracted multimodal descriptors and structured metadata, addressing critical gaps in scale, diversity, and clinical context in existing digital health datasets.
Author of this paper? Send a correction.