This study is a retrospective cohort analysis of high-risk patients with stroke treated at Ramathibodi Hospital in Bangkok, Thailand, from January 2010 to December 2020. The study included patients aged 18 years or older with at least 1 diagnosis of HT, AF, DM, or DLP. Participants were excluded if they had a prior stroke at the initial hospital visit or had only 1 visit during the study period.

The patient cohort was identified from Ramathibodi Hospital's electronic database using the International Classification of Diseases, 10th Revision (ICD-10) codes for risk factors and clinical features, such as HT (I10-I16), DM (E08-E13), AF (I48), and DLP (E78). The primary end points of interest were the development of ischemic stroke (I63) and hemorrhagic stroke (I61), as indicated by their respective ICD-10 codes. The features and criteria used in this study can be found in Multimedia Appendix 1.

Each patient was followed up until stroke occurrence, loss to follow-up, or censoring at study end (December 31, 2020). The latter 2 events were censored on their final visit or study end date, respectively.

Baseline study predictive features included age, sex, AF, HT, DM, DLP, SBP, plasma glucose (PG), serum creatinine, BMI, low-density lipoprotein and high-density lipoprotein (HDL), triglyceride level, and medications (antihypertensives, antiplatelets, oral hypoglycemics and insulin, statin and nonstatin lipid-lowering drugs, and anticoagulants). These baseline features were identified and retrieved when patients were first identified in our electronic medical records. The missing data in this study, assumed to be missing at random, were filled in using multiple imputation by chained equations via scikit-learn’s IterativeImputer [23,24]. The percentage of missing data and features used in multiple imputation by chained equations for each imputed variable are detailed in Table S1-S3 in Multimedia Appendix 1. Continuous data were categorized on the basis of previous literature to improve interpretation and as a requirement of the BN model [25]. Details of discretization are provided in Table S4 in Multimedia Appendix 1. We randomly separated the data by hospital numbers into development and test sets with a ratio of 80:20; each patient appeared in only 1 data set to maintain independence between the data sets. Characteristics of patients between the 2 data sets are comparable, see Table S5 in Multimedia Appendix 1.

We compared model performance and explainability between LR, CPH, BN, TAN, XGBoost, and EBM. We normalized continuous variables and used recursive feature elimination to select features in the LR model, whereas feature selection in XGBoost and EBM included self-selecting features during node splitting [26]. We manually selected features in the BN and TAN based on stroke pathophysiology and considered the appropriate network structure.

Scikit-learn served as the ML library for LR and XGBoost, with hyperparameter tuning using grid and random searches with successive halving (HalvingGridSearchCV and HalvingRandomSearchCV) and assigned imbalance ratio as weights to counter imbalanced class effects. We extracted LR coefficients and XGBoost’s features’ importance together with Shapley Additive Explanations (SHAP) to represent their explainability [27]. We constructed EBM using the open-source package InterpretML (Microsoft) [21]. Variable and interaction effects were plotted to determine their impact on the outcome.

We built a BN using GeNIe Modeler (BayesFusion, LLC) software based on the known causal pathways of disease [28] and trained it using discretized data. TAN structures were also determined using the training data and GeNIe (BayesFusion, LLC) software. The architectural details of the BN and TAN are shown in Multimedia Appendix 2. Models were evaluated with C-statistics and F₁-scores. C-statistics, or area under receiver operating characteristics curve (AUC-ROC), provide a measure of a models’ ability to accurately distinguish between positive and negative classes (0.5 being no predictive ability beyond chance and 1 being perfect prediction), while the F₁-score represents a measurement of the balance between precision and recall in binary classification, which computes by harmonic mean between precision and recall.

The data were anonymized to ensure confidentiality and privacy protection. This study was approved by Human Research Ethics Committee, Faculty of Medicine Ramathibodi Hospital, Mahidol University (COA. MURA2021/255). The committee waived the need to obtain consent for the collection, analysis, and publication of the retrospectively obtained and anonymized data for this noninterventional study.

We investigated a retrospective cohort of patients at high risk of developing stroke to develop prediction models for stroke occurrence. The models identified AF, HT, or SBP or antihypertensive medication, anticoagulant medication, HDL, age, and statin use as important features in predicting stroke using both conventional LR and ML models. Our findings provide robust, transparent, and explainable ML models for stroke risk prediction using routinely collected clinical data accessible in general health care settings.

Explainability and transparency of risk prediction models are important for facilitating the prescribing of individualized treatments (precision medicine) in real-world clinical settings [29]. Improved patient understanding also leads to empowerment and improved medication or treatment adherence [30]. Improved understanding and use of ML models in both prehoc and post hoc analyses are growing. According to Arrieta et al [18], LR, CPH, BN, TAN, and EBM models are considered transparent and understandable in themselves, while XGBoost requires post hoc analysis to improve explainability, including local interpretable model-agnostic explanations [31], SHAP, partial dependence plots, feature importance [32], or DeepLIFT [17,18,33]. Explainability can be classified into 3 types: application-grounded, human-grounded, and functionality-grounded [34]. Application- and human-grounded categories involve human interpretability, that is, models that are easily comprehensible to a layperson, without the need for specialized technical knowledge or expertise; functionality grounded refers to the methods or algorithms used and their quantitative evaluation.

Some studies have explored the benefits of white-box ML prediction models, such as BN and EBM. For example, Park et al [35] used BN with a TAN algorithm to predict 3-month functional outcomes after stroke with an AUC-ROC of 0.889. Kanwar et al [36] used a BN-derived risk prediction model that improved the prediction of 1-year survival in patients with pulmonary arterial HT compared to the Kaplan-Meier method in REVEAL (version 2.0), with an AUC-ROC of 0.8 versus 0.76 [37,38]. Lou et al [39] showed that EBM approaches could achieve accuracy close to that provided by random forest models while providing good interpretability. White-box EBM approaches are also known as “glass-box” models that allow for interaction terms between variables within the model. All of these models performed well, and their white-box nature enables transparency, making them useful for clinicians to explain and translate medical knowledge for a more confident application in clinical settings. A previous study compared multiple ML models to predict stroke and address the class imbalance problem using a multilayer perceptron classifier to achieve the lowest false-negative rate (18.60%) and SHAP to investigate the impact of risk factors on stroke prediction [40]. However, this approach was considered a post hoc analysis and not representative of a white-box model.

To date, investigation of the predictive capabilities of multiple explainable models in the context of stroke risk assessment using real-world data has been limited. Our study addresses this knowledge gap through a novel approach that compares the performance metrics for several explainable models, resulting in significantly improved predictive accuracy, further informing the existing literature. ML models generally outperform traditional statistical methods, supported by AUC-ROCs that represent sufficient improvement to be clinically actionable, that is, AUC-ROCs over 0.80-0.85. This does not mean that all ML methods are superior to traditional statistical methods in all applications, and users should keep an open mind. Another benefit of transparent and explainable models is the rational and selective approach to the choice of predictors. Data mining methods used without regard to causative pathways can include variables that cause collider bias and reduce model performance or may lead to embedded bias within the observational data.

We seek to further contextualize our study findings in relation to the existing literature while also acknowledging the unique characteristics of our study population. The significant features we identified as contributing to stroke risks, such as AF and HT, have been extensively reported previously, providing validation of our findings [41]. However, it is important to note that our study offers additional insight into the strength and interaction of these risk factors to improve our understanding of stroke risk. They also show the potential to improve model performance over traditional approaches even when starting with an identical data set.

The integration of these models into clinical workflows could provide real-time, personalized risk assessments, guiding clinicians toward more targeted and effective interventions. For instance, a patient identified as high risk could be prioritized for aggressive preventive measures, such as rigorous lifestyle modification, counseling, or intensified medication regimens. Conversely, if a patient is predicted to have a lower stroke risk, they may avoid unnecessary treatments and potential side effects, or they might require less intensive follow-up within the hospital setting. This individualized approach would enhance the personalization of stroke prevention strategies, potentially improving patient outcomes.

In addition, the interpretability of the models used helps health care professionals to better understand the key drivers of the predicted stroke risk. This transparency could facilitate more informed and confident decision-making, bridging the gap between complex ML algorithms and their practical application in a clinical setting. Ultimately, these advances could lead to more efficient and effective personalized health care, underpinned by evidence-based, data-driven decisions.

There were several limitations to our study. First, other important epidemiological factors, such as smoking status, education, and alcohol consumption, were not included within our risk prediction models as the information was not recorded in the electronic medical records. Many variables rely on the accuracy of ICD-10 coding, which may be subject to miscoding or misdiagnosis that would reduce model performance. Our study cohort encompassed a somewhat narrow demographic range of an “at-risk” population. We recognize that stroke risk factors may vary across different populations, highlighting the need for externally validating our stroke prediction model in the future before wider application.

Our study demonstrates predictive accuracy and explainability for stroke risk prediction models in high-risk patients. The key findings highlight the impact of AF, HT, and blood pressure control as significant risk factors for stroke emphasizing the potential benefits of screening and early detection, especially within patients for whom these risk factors are prominent. Furthermore, our findings confirm the robustness and interpretability of ML models such as XGBoost, EBM, and BN in handling complex, real-world health data and the potential to improve model performance even when starting with the same data set as traditional approaches.

Looking ahead, we anticipate significant opportunities for further research using these approaches. The continued evolution of ML techniques provides an avenue for refining prediction models, possibly by incorporating additional or alternative feature sets. Moreover, future studies could explore the effects of different interventions on stroke risk, such as lifestyle modifications or novel therapeutic agents. In doing so, our understanding of stroke prevention and management may be enhanced, potentially improving patient outcomes. By pushing the boundaries of explainable ML in health care, these findings hold the potential to revolutionize clinical practice, empowering physicians and patients with clear, actionable insights for better health outcomes.