Academic Commons

Articles

Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes

Bowers, Alex J.; Zhou, Xiaoliang

Early Warning Systems (EWS) and Early Warning Indictors (EWI) have recently emerged as an attractive domain for states and school districts interested in predicting student outcomes using data that schools already collect with the intention to better time and tailor interventions. However, current diagnostic measures used across the domain do not consider the dual issues of sensitivity and specificity of predictors, key components for considering accuracy. We apply signal detection theory using Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) analysis adapted from the engineering and medical domains, and using the pROC package in R. Using nationally generalizable data from the Education Longitudinal Study of 2002 (ELS:2002) we provide examples of applying ROC accuracy analysis to a variety of predictors of student outcomes, such as dropping out of high school, college enrollment, and postsecondary STEM degrees and careers.

Keywords: ROC, AUC, Early Warning System, Early Warning Indicator, signal detection theory, dropout, college enrollment, Postsecondary STEM Degree, hard STEM career, soft STEM career

Files

  • thumnail for Bowers Zhou 2019 ROC AUC for EWIS JESPAR.pdf Bowers Zhou 2019 ROC AUC for EWIS JESPAR.pdf application/pdf 1 MB Download File

Also Published In

Title
Journal of Education for Students Placed At Risk
DOI
https://doi.org/10.1080/10824669.2018.1523734

More About This Work

Academic Units
Education Leadership
Published Here
March 5, 2019

Notes

This document is a pre-print of this manuscript, published in the Journal of Education for Students Placed At Risk (JESPAR). Citation:
Bowers, A.J., Zhou, X. (2019) Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes. Journal of Education for Students Placed At Risk, 24(1) 20-46. https://doi.org/10.1080/10824669.2018.1523734

This research was supported by a grant from the National Science Foundation (NSF IIS-1546653). Any opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of funding agencies.

Academic Commons provides global access to research and scholarship produced at Columbia University, Barnard College, Teachers College, Union Theological Seminary and Jewish Theological Seminary. Academic Commons is managed by the Columbia University Libraries.