Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes
Early Warning Systems (EWS) and Early Warning Indictors (EWI) have recently emerged as an attractive domain for states and school districts interested in predicting student outcomes using data that schools already collect with the intention to better time and tailor interventions. However, current diagnostic measures used across the domain do not consider the dual issues of sensitivity and specificity of predictors, key components for considering accuracy. We apply signal detection theory using Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) analysis adapted from the engineering and medical domains, and using the pROC package in R. Using nationally generalizable data from the Education Longitudinal Study of 2002 (ELS:2002) we provide examples of applying ROC accuracy analysis to a variety of predictors of student outcomes, such as dropping out of high school, college enrollment, and postsecondary STEM degrees and careers.
Keywords: ROC, AUC, Early Warning System, Early Warning Indicator, signal detection theory, dropout, college enrollment, Postsecondary STEM Degree, hard STEM career, soft STEM career
- Bowers Zhou 2019 ROC AUC for EWIS JESPAR.pdf application/pdf 1 MB Download File
Also Published In
- Journal of Education for Students Placed At Risk
More About This Work
- Academic Units
- Education Leadership
- Published Here
- March 5, 2019
This document is a pre-print of this manuscript, published in the Journal of Education for Students Placed At Risk (JESPAR). Citation:
Bowers, A.J., Zhou, X. (2019) Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes. Journal of Education for Students Placed At Risk, 24(1) 20-46. https://doi.org/10.1080/10824669.2018.1523734
This research was supported by a grant from the National Science Foundation (NSF IIS-1546653). Any opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of funding agencies.