DIVINE: Coordinating Multimodal Disentangled Representations for Oro-Facial Neurological Disorder Assessment
Abstract
In this study, we present a multimodal framework for predicting neuro-facial disorders by capturing both vocal and facial cues. We hypothesize that explicitly disentangling shared and modality-specific representations within multimodal foundation model embeddings can enhance clinical interpretability and generalization. To validate this hypothesis, we propose DIVINE a fully disentangled multimodal framework that operates on representations extracted from state-of-the-art (SOTA) audio and video foundation models, incorporating hierarchical variational bottlenecks, sparse gated fusion, and learnable symptom tokens. DIVINE operates in a multitask learning setup to jointly predict diagnostic categories (Healthy Control,ALS, Stroke) and severity levels (Mild, Moderate, Severe). The model is trained using synchronized audio and video inputs and evaluated on the Toronto NeuroFace dataset under full (audio-video) as well as single-modality (audio-only and video-only) test conditions. Our proposed approach, DIVINE achieves SOTA result, with the DeepSeek-VL2 and TRILLsson combination reaching 98.26% accuracy and 97.51% F1-score. Under modality-constrained scenarios, the framework performs well, showing strong generalization when tested with video-only or audio-only inputs. It consistently yields superior performance compared to unimodal models and baseline fusion techniques. To the best of our knowledge, DIVINE is the first framework that combines cross-modal disentanglement, adaptive fusion, and multitask learning to comprehensively assess neurological disorders using synchronized speech and facial video.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DIVINE, a fully disentangled multimodal framework that enhances the assessment of neuro-facial disorders by effectively integrating audio and visual cues through advanced representation learning techniques. This work is significant as it addresses critical challenges in clinical assessments, providing a foundation for future research and practical applications in the field.
Methodology
The proposed DIVINE framework introduces a novel approach to disentangling shared and modality-specific representations in a multimodal setting, leveraging advanced techniques such as hierarchical variational bottlenecks and sparse gated fusion. The integration of symptom tokens for clinical interpretability is a significant methodological contribution that enhances the model's applicability in real-world clinical scenarios. The multitask learning setup is well-justified, allowing for simultaneous classification and severity estimation, which is crucial for neuro-facial disorder assessment.
Results & evaluation
The experiments are robust, utilizing the Toronto NeuroFace dataset with a well-defined evaluation strategy that includes cross-validation and various test conditions (full, audio-only, video-only). The reported results demonstrate significant improvements over baseline methods and unimodal models, establishing a new state-of-the-art performance. However, the paper could benefit from more detailed comparisons with existing methods in terms of computational efficiency and model complexity.
Reproducibility
The paper provides a GitHub repository with code, model checkpoints, and evaluation scripts, which is a positive aspect for reproducibility. However, the absence of detailed hyperparameter settings and specific training configurations in the main text may hinder full reproducibility for some readers.
Limitations
The study's limitations include a reliance on a single dataset, which may affect the generalizability of the findings. The authors also acknowledge the need for cross-dataset validation, which is essential for establishing the robustness of the model in diverse clinical settings. Additionally, the ethical implications of using the model in clinical practice are briefly mentioned but could be explored in greater depth.
Broader impact
The DIVINE framework has the potential to significantly impact clinical workflows by providing a more objective and interpretable assessment tool for neuro-facial disorders. Its use in decision-support systems could enhance the accuracy and efficiency of clinical evaluations, but careful consideration of ethical deployment and clinician oversight is necessary to mitigate risks associated with automated decision-making. The main contribution of this paper is the introduction of DIVINE, a fully disentangled multimodal framework that enhances the assessment of neuro-facial disorders by effectively integrating audio and visual cues through advanced representation learning techniques. This work is significant as it addresses critical challenges in clinical assessments, providing a foundation for future research and practical applications in the field.
Author of this paper? Send a correction.