2019 Theses Doctoral

Cognitive Diagnostic Models-based Automatic Item Generation: Item Feature Exploration and Calibration Model Selection

Bai, Yu

One of the most significant challenges for test developers is the creation and production of effective test items. Automatic Item Generation (AIG) presents a highly-efficient approach to developing items at a relatively low cost. Research is conducted on the AIG system to explore item characteristics (or features) that impact item parameters, and to develop the appropriate calibration models for the items generated. Current research has focused on developing the AIG system within a framework of Item Response Theory. However, there may be additional benefits to developing an AIG system based on Cognitive Diagnostic Models (CDM), since both AIG and CDM development start with developing cognitive models. It remains to be seen, however, to what extent the cognitive model of CDMs (Q-matrix) may be helpful to the AIG system.
This research aims to assess the feasibility of adopting Q-matrix for content-related features to predict the item parameters through an empirical study (study 1). In addition, four calibration models were proposed and evaluated in a simulation study with conditions representing possible types of variations due to the CDM-based AIG process (study 2). Overall, the results of study 1 showcased the impact of the Q-matrix on item parameters that Q-matrix either alone or together with universal features explained a good amount of the variations in item parameters, especially in parameter g. The results of study 2 were promising, suggesting that the calibration models provided unbiased estimations of the person and item parameters.
This research explores potential issues that may be encountered by the CDM-based AIG system and provides evidence of the advantages of building an AIG within the CDM framework.