Multimodal Decision Support System for Improved Diagnosis and Healthcare Decision Making

Abstract

Access to quality health care continues to be a major challenge for remote or overlooked regions that do not have the necessary medical infrastructures and human resources. This research proposes integrating structured data-laboratory results, vitals and demographics- with unstructured medical data-clinical notes, free-text diagnosis-finalized into Multimodal Decision Support Systems (MDSS) that closes the healthcare gap by enhancing the diagnostic accuracy and treatment recommendations. Our innovative approach entails employing random forest classifiers for structured data and BERT-based embeddings for unstructured data and fuses their predictive outputs through late fusion technique. Among various evaluated fusion methods, including simple average, weighted average and stacked fusion, the stacked fusion approach resulted in achieving maximum diagnostic accuracy, i.e., 87 against individual models, thus taking diagnostic accuracy improvement into huge consideration as well as significant reductions in misdiagnosis, in addition to last but not least, personalized healthcare recommendation, especially to rural populations. The evaluation of this system used the MIMICIV dataset and showed improved performance in risk prediction and analysis of patient outcomes. The first generation of our smart health care assistant will feature video consultations and multi-lingual support, as well as real-time processing capabilities to allow access to high-quality health care for these populations. Future improvements in optimizing data imputation, enhancing interpretability and ensuring HIPAA and GDPR compliance will make for secure and ethical data usage. This work will build ground work for AI-personal healthcare solutions with a long-term goal of bridging the gap between rural and urban patient populations in access to care.