Since the passage of the Health Information Technology for Economic and Clinical Health (HITECH) Act in 2009 1 2 , there has been an increase in the rate of electronic health record (EHR) adoption. As of 2012, the rate of EHR adoption with at least basic functionality was 44.4% in non-federal acute care hospitals 3 and 39.6% in office-based physician practices 4 .

The transition to EHRs has created new opportunities for research 5 6 7 . The secondary use of EHR data provides a more efficient and less expensive alternative to clinical trials, the current gold standard of medical research 8 9 . This is especially important in the current fiscal climate, where federal funding of medical research is becoming increasingly limited.

There are, however, potential caveats to the secondary use of EHR data 10 . EHRs suffer from data quality problems 11 12 13 , which may affect the internal validity of retrospective studies. One of these data quality problems is insufficient data. Sufficiency can be conceptualized as a type of completeness, which is one of several categories of data quality that are relevant to EHR data reuse 14 . When EHR data are complete according to the requirements of a given task, those data can be considered to be sufficient for that task. Required data may be missing for different reasons: a data point was observed but not documented 12 or it was never observed in the first place, either because the observation was not clinically necessary or because it could not be performed. Regardless of the reason, missing data is very common in today’s EHR databases, leading to datasets that may not be sufficient for work relying on the secondary use of EHR data. Although it has been pointed out that the missing data may cause records to be “visually complete but intellectually insufficient,” 15 the causal effect of health status on data sufficiency is not the focus of this study. Instead, we focus on the correlation between the sufficiency of electronic health record data for clinical research and the underlying patient health status.

In a clinical trial a study sample is chosen based on predefined eligibility criteria. The data necessary to answer the research question is then prospectively collected for every participant. This approach ensures that all required data are present and trustworthy, but may come at the expense of limited external generalizability due to the non-representativeness of the sample 16 . In contrast, studies relying on the use of EHR data are thought to have greater external validity, having drawn their participants from actual patients receiving regular care in actual health care settings. In such studies, however, participants must be chosen based not only on the eligibility criteria but also upon the availability of sufficient data for extraction 17 18 19 . Example sufficiency requirements include “a sub-population who have sufficient health record data at institution {X} frequenting the {X} hospital system for routine care” and “total number of individuals that have male gender and serum creatinine 1.5 mg/dL or female gender and serum creatinine 1.3 mg/dL. The patients need to have at least 2 values over the threshold.” In a study by Green et al., of 122,270 patients satisfying eligibility criteria, only 59.7% had sufficient data 19 . Patients without the data necessary to determine eligibility or perform the analyses of interest cannot, by definition, be included in the study sample. The addition of this frequently overlooked sufficiency requirement has the potential to lead to bias in the selection of patients for inclusion in EHR based studies, which may limit their external validity.

The proportion of patients in a given population with sufficient data varies from study to study, as it depends on the research question and the necessary kinds of data required for answering that question 14 20 . We have previously demonstrated the contextual nature of EHR sufficiency, as well as the high variability of sufficient patient records in a large-scale analysis of the NewYork-Presbyterian Hospital Clinical Data Warehouse 14 . This variability is not always random; it is more likely that the pattern of data quantity is related to one or more of the variables of interest 21 . Our preliminary work indicates that the patient records containing sufficient data, i.e., those best-suited for secondary use in research, tend to belong to the sickest patients 22 .

As Lee et al. point out, many studies assume that the addition of a requirement for a visit in a given time frame (visit-based sampling) produces a sample that is representative of the population from which it is derived. However, as their work demonstrates this assumption is wrong, and the imposition of just this one sufficiency requirement biases the population towards sicker and older patients 23 . Sufficient visit data is one common way patients are selected for inclusion in EHR based research. Another common sufficiency requirement is based on laboratory and/or medication data. Some studies require just the presence of a specific laboratory value or medication order while others also impose a minimum threshold for the number of each.

This paper reports an in-depth exploration of the relationship between patient illness severity and quantity of available data, as well as the potential clinical confounders of this relationship. We demonstrate that, because of the data sufficiency requirements for sampling, the cohorts being identified for research may not be representative of the broader patient population, thus compromising the external validity of research conducted using EHR data.

We hypothesized that the health records of sicker patients would be more likely to have sufficient data for research, and that this relationship would hold true when controlling for possible covariates. We also hypothesized that other patient- and procedure-related factors, such as age, sex, admission status, and the emergent nature of the procedure, would independently affect EHR data quantity.

The mean age of patients in our sample was 45.0 (SD = 23.9) and ranged from one year to 102. Sixty-one percent of our cohort was female. Most cases were non-emergent (88.8%) with more outpatients (41.6%) than same-day admissions (32.7%) or inpatients (25.6%). The most frequently occurring diagnostic categories in our dataset were Complications of pregnancy, childbirth, and puerperium (19.3%), Diseases of the digestive system (12.3%), and Neoplasms (11.1%). The most common procedure categories were Anesthesia procedures, which includes procedures for analgesia during labor and delivery (17.5%), Operations on the digestive system (16.3%), and Operations on the musculoskeletal system (13.2%). Table  2 presents descriptive statistics of counts of days with laboratory results and medication orders within subcategories.

ASA Class = American Society of Anesthesiologists Physical Status Classification; ICD-9 = International Classification of Diseases, Ninth Revision; CPT = Current Procedural Terminology; Dz. = Diseases; * Denotes ICD and CPT categories that contain other similar categories: Congenital anomalies contains Certain conditions originating in the perinatal period; Symptoms, signs and ill-defined conditions contains Infections and parasitic diseases, Diseases of the blood and blood forming organs, Mental disorders, Diseases of the skin and subcutaneous tissue, & Supplementary; Operations on the nose, mouth and pharynx contains Operations on the ear; Operations on the female genital organs contains Obstetrical procedures; Operations on the integumentary system contains Operations on the hemic and lymphatic system; Anesthesia procedures contains Other.

Table  3 shows the effect of each variable as a whole and of each level of the primary outcome variable (ASA Class) on the estimated number of days with laboratory results and medication orders based on the parameter estimates from the negative binomial model. Effects, standard errors and 95% confidence intervals for individual variable levels are expressed as ratios comparing that level to the reference level for the variable, such that an effect of 2.0 for ASA 3 indicates that ASA 3 is estimated to have 2 times the number of days as ASA 1. These ratios were obtained by exponentiating the model regression coefficients. ASA class, subject sex, age, admission status, ICD-9 category, and CPT category were significantly associated with the counts of days with laboratory results and medication orders, while emergency status was associated only with laboratory results.

ASA Class = American Society of Anesthesiologists Physical Status Classification; ICD-9 = International Classification of Diseases, Ninth Revision; CPT = Current Procedural Terminology; SE = Standard Error.

*The effects are ratios, obtained by exponentiating the model regression coefficients, of the expected number of days (with either laboratory results or medication orders) for each ASA Class compared to ASA 1, that is, an effect of 2.0 for ASA 3 indicates that ASA 3 is estimated to have 2 times the number of days as ASA 1.

^†Statistically significant at the 0.05 significance level.

Our primary variable of interest, ASA class, had a significant association with the counts of days with laboratory results and medication orders. Controlling for all other variables, the estimated count of days with laboratory results for ASA 2 was 1.47 times, for ASA 3 was 3.38 times, and for ASA 4 was 5.05 times the count of days with laboratory results for ASA 1. The pairwise differences for counts of days with laboratory results between all four ASA classes were statistically significant. Similarly, the estimated count of days with medication orders for ASA 2 was 1.74 times, for ASA 3 was 4.78 times, and for ASA 4 was 6.85 times the count of days with medication orders for ASA 1. All pairwise comparisons between the four ASA classes for counts of days with medication orders were statistically significant.

The results of the negative binomial regression model demonstrate the relationship between patient health status and EHR data sufficiency. The less healthy the patient, as measured by ASA status, the more data that patient is likely to have, as represented by counts of days with laboratory results and medication orders, and the more likely they are to satisfy sufficiency requirements. This relationship holds true even when controlling for a number of likely confounders, including sex, age, emergent status, patient type, diagnosis, and procedure, which suggests that even within specific, well-defined cohorts, sicker patients are likely to have more data than healthier patients.

These findings highlight an important but usually overlooked problem inherent to studies using EHR data: the selection of records with sufficient data, as measured by human imposed sufficiency requirements, for research may bias the sample towards patients who are sicker than the population from which the sample is drawn. The findings from this study are consistent with previous work exploring the complex relationships among data quality, bias, and health status. In one example, Wennberg et al. used insurance claims data to demonstrate bias in comorbidity measurement by showing that Charlson Comorbity Index scores are associated with the frequency of physician visits 25 , suggesting that data quality is compromised by differences in healthcare utilization. Similarly, Collins et al. identified a relationship between patient mortality and increased rates of nursing documentation, suggesting that more acutely ill patients are likely to have more thoroughly documented records 36 37 . In a study of a pneumonia severity index using EHR data, Hripcsak et al. found that the addition of cohort selection criteria that required the presence of sufficient data to make a reliable diagnosis substantially limited the sample size and significantly altered the mortality rates 38 . They note that the addition of simple sample restraints, while beneficial in their case, has the potential to significantly narrow the sample, leading to the possibility of bias.

We observed a direct correlation between severity of illness and data sufficiency in spite of the presence of sub-populations in our study sample in which this correlation should not exist: living organ donors and pregnant women with uncomplicated pregnancies presenting for management of labor and delivery. These patients tend to be healthy, but have more data in their records, resulting from laboratory testing performed as part of routine prenatal care or organ donor evaluation. Our 10,000-patient sample contained 1,802(18.0%) such patients, of whom 1,746(96.9%) were classified as ASA 1 or 2 (relatively healthy). The average number of days with laboratory results for patients in this group (6.5) is nearly double that of all other patients in the study (3.4). Despite the presence of such a large number of healthy patients with a high degree of EHR sufficiency, our original hypothesis — that sicker patients have better EHR data sufficiency for research — was confirmed. (See Additional file 1: Table S1 for results of the negative binomial model with pregnant patients and living organ donors excluded).

In addition to confirming our primary hypothesis, we discovered that many other variables are independently associated with data sufficiency. These include admission status at time of assessment, age, emergency classification of the procedure, procedure type (CPT category) and primary diagnosis type (ICD-9 category). Potential biases in these other characteristics of the study population should be considered when selecting populations based on sufficiency requirements this population is studied.

In this analysis, we established the correlation between the degree of patient sickness and the sufficiency of data of their health records, for inclusion in research. This finding is important for researchers reusing EHR data. EHR-based studies sample patients based on sufficiency requirements with the aim of selecting only those records containing sufficient data to overcome the data missingness problem. This strategy turns out to introduce a hidden bias towards sick patients because sicker patient have records with a higher degree of data sufficiency and are more likely to be included in EHR-based studies; therefore, this selection process biases the study populations towards those comprised of sicker patients. The more stringent the sufficiency requirements, the sicker the resultant sample population. This is a problem unique to studies that rely on the secondary use of data initially collected for purposes other than research. Those involved in the secondary use of EHR data for research, as well as consumers of this research, should be aware of this sampling bias problem and exercise caution when applying results to real world populations.

The authors declared that they have no competing interests.

AR and NGW carried out data extraction and analysis and wrote the manuscript together. SW performed the statistical analyses. CW identified the research question and directed the experiment. All authors read and approved the final manuscript.