Theses Doctoral

Process Data Applications in Educational Assessment

Qi, Jitong

The widespread adoption of computer-based testing has opened up new possibilities for collecting process data, providing valuable insights into the problem-solving processes that examinees engage in when answering test items. In contrast to final response data, process data offers a more diverse and comprehensive view of test takers, including construct-irrelevant characteristics. However, leveraging the potential of process data poses several challenges, including dealing with serial categorical responses, navigating nonstandard formats, and handling the inherent variability. Despite these challenges, the incorporation of process data in educational assessments holds immense promise as it enriches our understanding of students' cognitive processes and provides additional insights into their interactive behaviors. This thesis focuses on the application of process data in educational assessments across three key aspects.

Chapter 2 explores the accurate assessment of a student's ability by incorporating process data into the assessment. Through a combination of theoretical analysis, simulations, and empirical study, we demonstrate that appropriately integrating process data significantly enhances assessment precision.

Building upon this foundation, Chapter 3 takes a step further by addressing not only the target attribute of interest but also the nuisance attributes present in the process data to mitigate the issue of differential item functioning. We present a novel framework that leverages process data as proxies for nuisance attributes in item response functions, effectively reducing or potentially eliminating differential item functioning. We validate the proposed framework using both simulated data and real data from the PIAAC PSTRE items.

Furthermore, this thesis extends beyond the analysis of existing tests and explores enhanced strategies for item administration. Specifically, in Chapter 4, we investigate the potential of incorporating process data in computerized adaptive testing. Our adaptive item selection algorithm leverages information about individual differences in both measured proficiency and other meaningful traits that can influence item informativeness. A new framework for process-based adaptive testing, encompassing real-time proficiency scoring and item selection is presented and evaluated through a comprehensive simulation study to demonstrate the efficacy.

Files

  • thumnail for Qi_columbia_0054D_18078.pdf Qi_columbia_0054D_18078.pdf application/pdf 1.89 MB Download File

More About This Work

Academic Units
Statistics
Thesis Advisors
Liu, Jingchen
Ying, Zhiliang
Degree
Ph.D., Columbia University
Published Here
August 23, 2023