2024 Reports

Challenges and Solutions for Measuring Level 3 of the Kirkpatrick Model

Chandler, Thomas; Hendra, Richard; Huang, Shuyang; DeVincenzo, Joshua L.; Yang, Yaxuan

As disasters in the United States increase in frequency and intensity, preparedness is now more critical than ever. To that end, training disaster management professionals on hazard mitigation, preparation, response, and recovery is a cornerstone of new planning efforts. Annually, the U.S. government, at all levels, invests significant resources in training programs. Taking advantage of "blue sky days" to learn best practices, work through real-life scenarios, and practice using tools and templates is essential. But how do we know if a given emergency management training has actually been effective? Are participants really incorporating what they have learned into their everyday work practices or even as part of an actual disaster response? This conceptual white paper addresses the challenges and opportunities of incorporating the Kirkpatrick Model’s Level 3 evaluation into determining whether a training program has been successful in the long term. The authors address data collection challenges, threats to validity of causal inferences, as well as ways in which learning management systems can improve Level 3 evaluation capabilities.

Keywords: Emergency management training; Evaluation; Kirkpatrick Model; Level 3; Knowledge transfer; Learning management system; Social desirability bias; Recall bias; Survey response rate; Online learning; Distance learning; Threats to validity of causal inferences