2013 Reports

Metamorphic Runtime Checking of Applications without Test Oracles

Murphy, Christian; Kaiser, Gail E.; Bell, Jonathan Schaffer; Su, Fang-Hsiang

Challenges arise in testing applications that do not have test oracles, i.e., for which it is impossible or impractical to know what the correct output should be for general input. Metamorphic testing, introduced by Chen et al., has been shown to be a simple yet effective technique in testing these types of applications: test inputs are transformed in such a way that it is possible to predict the expected change to the output, and if the output resulting from this transformation is not as expected, then a fault must exist. Here, we improve upon previous work by presenting a new
technique called Metamorphic Runtime Checking, which automatically conducts metamorphic testing of both the entire application and individual functions during a program's execution. This new approach improves the scope, scale, and sensitivity of metamorphic testing by allowing for the identification of more properties and execution of more tests, and increasing the likelihood of detecting faults not be found by application-level properties alone. We also discuss a technique for automatically discovering functions' metamorphic properties, and present the results of new studies that demonstrate that Metamorphic Runtime Checking advances the state of the art in testing applications without oracles.