2014 Reports

Metamorphic Runtime Checking of Applications Without Test Oracles

Bell, Jonathan Schaffer; Murphy, Christian; Kaiser, Gail E.

For some applications, it is impossible or impractical to know what the correct output should be for an arbitrary input, making testing difficult. Many machine-­learning applications for “big data”, bioinformatics and cyberphysical systems fall in this scope: they do not have a test oracle. Metamorphic Testing, a simple testing technique that does not require a test oracle, has been shown to be effective for testing such applications. We present Metamorphic Runtime Checking, a novel approach that conducts metamorphic testing of both the entire application and individual functions during a program’s execution. We have applied Metamorphic Runtime Checking to 9 machine-­‐learning applications, finding it to be on average 170% more effective than traditional metamorphic testing at only the full application level.