Academic Commons

Theses Doctoral

Developing an approach to determine generalizability: A review of efficacy and effectiveness trials funded by the Institute of Education Sciences

Fellers, Lauren Ashley

Since its establishment the Institute of Education Sciences has been creating opportunities and driving standards to generate research in education that is high quality rigorous, and relevant. This dissertation is an analysis of current practices in Goal III and Goal IV studies, in order to (1) better understand of the types of schools that agree to take part in these studies, and (2) an assess how representative these schools are in comparison to important policy relevant populations. This dissertation focuses on a subset of studies that were funded from 2005-2014 by the Department of Education, IES, under the NCER grants-funding arm. Studies included were those whose interventions were aimed at elementary students across core curriculum and ELL program areas. Study schools were compared to two main populations, the U.S population of elementary schools and Title I elementary schools, as well as these populations on a state level. The B-index, proposed by Tipton (2014) was the main value of comparison used to assess the compositional similarity, or generalizability, of study schools to these identified inference populations. The findings show that across all studies included in this analysis, participating schools were representative of the U.S. population of schools, B-index = 0.9. Comparisons were also made between this collection of schools and the respective populations at the state level. Results showed that these schools were not representative of any individual states (no B-index values were greater than 0.90). Across all included studies, schools that agreed to participate were more often located in urban areas, had higher rates of FRL students, had more minority students enrolled, and had more total students, in both district and school, than those schools in the population of U.S. schools. It is clear that the movement of education research is to be relevant to a larger audience. Through this study it is clear that, across studies, we are achieving some representation in IES funded studies. However, the finer comparisons, study samples to individual state and individual studies to these populations, show limited similarity between study schools and populations of interest to policy makers using these study findings to make decisions about their schools.


  • thumnail for Fellers_columbia_0054D_13746.pdf Fellers_columbia_0054D_13746.pdf binary/octet-stream 1.47 MB Download File

More About This Work

Academic Units
Measurement and Evaluation
Thesis Advisors
Tipton, Laura Elizabeth
Ph.D., Columbia University
Published Here
January 31, 2017