A Systematic Review of Deep Learning and Machine Learning Applications in Longitudinal Multimodal Clinical Data

Jinqian Pan, Tienyu Chang, Mengxian Lyu, Weimin Meng, Qingyu Wang, Yiling Ma, Ziyi Chen, Xiaohan Li, Chengkun Sun, Renjie Liang, Jennifer Fishe, Jie Xu

Abstract Artificial intelligence (AI) is increasingly applied in healthcare to support diagnosis and treatment by leveraging longitudinal multimodal data, despite persistent challenges such as missing data and heterogeneity. This systematic review, conducted in accordance with PRISMA guidelines, analyzed deep learning (DL) and machine learning (ML) applications to longitudinal multimodal clinical data published between 2020 and 2024. An initial search identified 1,124 records; a total of 54 items were included (53 studies meeting inclusion criteria and 1 dataset descriptor). Relevant research studies increased substantially over the review period, from 3 studies in 2020 to 21 in 2024. Prediction (n = 23) and classification (n = 19) were the most commonly addressed tasks. Structured electronic health records (EHRs) and medical imaging were the most frequently used data sources, reported in 48 and 33 studies, respectively, and were often used in combination. Regarding missing data, 34 studies reported general handling strategies, and 10 explicitly addressed missing modalities. While methodological variability and performance limitations remain, these findings highlight the growing use of DL and ML to analyze multimodal data, which may inform clinical decision support.