A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning
Abstract
Emergency Medical Technicians (EMTs) operate in high-pressure environments, making rapid, life-critical decisions under heavy cognitive and operational loads. We present EMSGlass, a smart-glasses system powered by EMSNet, the first multimodal multitask model for Emergency Medical Services (EMS), and EMSServe, a low-latency multimodal serving framework tailored to EMS scenarios. EMSNet integrates text, vital signs, and scene images to construct a unified real-time understanding of EMS incidents. Trained on real-world multimodal EMS datasets, EMSNet simultaneously supports up to five critical EMS tasks with superior accuracy compared to state-of-the-art unimodal baselines. Built on top of PyTorch, EMSServe introduces a modality-aware model splitter and a feature caching mechanism, achieving adaptive and efficient inference across heterogeneous hardware while addressing the challenge of asynchronous modality arrival in the field. By optimizing multimodal inference execution in EMS scenarios, EMSServe achieves 1.9x -- 11.7x speedup over direct PyTorch multimodal inference. A user study evaluation with six professional EMTs demonstrates that EMSGlass enhances real-time situational awareness, decision-making speed, and operational efficiency through intuitive on-glass interaction. In addition, qualitative insights from the user study provide actionable directions for extending EMSGlass toward next-generation AI-enabled EMS systems, bridging multimodal intelligence with real-world emergency response workflows.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of EMSGlass, a multimodal multitask learning system that enhances the capabilities of Emergency Medical Technicians through real-time situational awareness and decision-making support. This work represents a notable step forward in applying machine learning to critical real-world applications, showcasing both innovative methodology and practical implications for improving emergency medical services.
Methodology
The paper introduces EMSGlass, a smart-glasses system that employs EMSNet, a multimodal multitask learning model designed specifically for Emergency Medical Services (EMS). The methodology is innovative as it combines various data modalities—text, vital signs, and scene images—into a cohesive model that addresses multiple tasks simultaneously. The use of a modality-aware model splitter and feature caching mechanism in EMSServe is a significant contribution that enhances the efficiency of multimodal inference in real-time scenarios. The integration of these components is well-articulated, showcasing a thoughtful approach to the challenges faced in high-pressure EMS environments.
Results & evaluation
The authors conducted experiments using real-world multimodal EMS datasets, demonstrating that EMSNet outperforms state-of-the-art unimodal baselines across five critical EMS tasks. The reported speedup of 1.9x to 11.7x over direct PyTorch multimodal inference is compelling and indicates a strong technical contribution. Additionally, the user study involving six professional EMTs provides qualitative insights into the system's effectiveness in enhancing situational awareness and decision-making speed, although the sample size is relatively small.
Reproducibility
The paper lacks detailed implementation information, such as code availability or specific dataset descriptions, which could hinder reproducibility. While the authors mention using PyTorch, further details on the training process, hyperparameters, and evaluation metrics would be beneficial for others attempting to replicate the study.
Limitations
One limitation is the small user study sample size, which may not fully capture the diverse experiences and needs of all EMTs. Additionally, the paper does not address potential challenges in deploying such a system in varied real-world environments or the adaptability of the model to different EMS protocols.
Broader impact
The potential applications of EMSGlass extend beyond emergency medical services, as the framework could be adapted for other high-stakes environments requiring rapid decision-making under pressure. The integration of multimodal intelligence into wearable technology represents a significant advancement in the field, with implications for improving operational efficiency and outcomes in emergency response scenarios. The main contribution of this paper is the development of EMSGlass, a multimodal multitask learning system that enhances the capabilities of Emergency Medical Technicians through real-time situational awareness and decision-making support. This work represents a notable step forward in applying machine learning to critical real-world applications, showcasing both innovative methodology and practical implications for improving emergency medical services.
Author of this paper? Send a correction.