Opportunities and challenges in developing risk prediction models with electronic health records data: A systematic review

Objective: Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data. Methods: We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation. Results: We identified 107 articles from 15 different countries. Studies were generally very large (median sample size1/426 100) and utilized a diverse array of predictors. Most used validation techniques (n1/494 of 107) and reported model coefficients for reproducibility (n1/483). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n1/437) and employed relatively few predictor variables (median1/427 variables). Less than half of the studies were multicenter (n1/450) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow- up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71). Conclusions: EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.