VHealth-MFusion: a fusion-based multimodal deep learning framework for pneumonia tele-diagnosis in virtual hospital environments.
Authors
Affiliations (9)
Affiliations (9)
- Department of Computer Science, Virginia Tech, Alexandria, VA, United States.
- Information Technology Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University, Riyadh, Saudi Arabia.
- Information Systems Department, College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University, Riyadh, Saudi Arabia.
- Information Systems Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia.
- Computer Engineering Department, College of Engineering, Al Yamamah University, Riyadh, Saudi Arabia.
- Trauma and Acute Care Surgery Unit, College of Medicine, King Saud University, Riyadh, Saudi Arabia.
- Department of Pulmonary Medicine, Rashid Hospital, Dubai, United Arab Emirates.
- Hind Bint Maktoum College of Nursing and Midwifery, Mohammed Bin Rashid University of Medicine and Health Sciences (MBRU), Dubai, United Arab Emirates.
- Department of Radiology and Medical Imaging, King Khalid University Hospital, King Saud University, Riyadh, Saudi Arabia.
Abstract
Virtual hospital systems support remote triage and diagnosis across distributed healthcare environments by integrating radiographic imaging with structured clinical information for clinical decision making. However, many existing diagnostic frameworks do not fully address real world challenges commonly encountered in scalable virtual healthcare settings. These challenges include severe class imbalance, incomplete clinical attributes, heterogeneous multimodal inputs, and variability in data quality across institutions. This study introduces <i>VHealth-MFusion</i>, a fusion-based hierarchical multimodal deep learning framework that integrates chest X-ray (CXR) imaging and structured clinical data within a unified convolutional neural network (CNN) and multilayer perceptron (MLP) architecture. Using pneumonia Tele-Diagnosis as a use case, the proposed framework combines radiographic feature extraction through a CNN branch with structured clinical representation learning through an MLP branch, followed by late multimodal fusion for multiclass respiratory disease classification. The framework was evaluated using a hierarchical multimodal dataset containing CXR images across multiple diagnostic categories, including Normal, Bacterial, Adenovirus, Influenza, MERS-CoV, and SARS-CoV-2, together with 632 structured clinical variables comprising demographic information, vital signs, symptoms, and laboratory findings. Under a controlled dataset-consistent comparison against an established hierarchical multimodal CNN baseline, <i>VHealth-MFusion</i> achieved 97.2% classification accuracy, outperforming the baseline accuracy of 95.9%. The findings demonstrate that multimodal integration of radiographic imaging and structured clinical information can improve diagnostic robustness and support more reliable Tele-Diagnosis workflows within virtual hospital environments. Overall, the proposed framework contributes a practically oriented multimodal diagnostic architecture designed to facilitate scalable remote clinical services and informed decision making within digitally enabled care systems where imaging and structured clinical information are jointly available.