Home

Empirical Evaluation of Approaches to Testing Applications without Test Oracles

Christian Murphy; Gail E. Kaiser

Title:
Empirical Evaluation of Approaches to Testing Applications without Test Oracles
Author(s):
Murphy, Christian
Kaiser, Gail E.
Date:
Type:
Technical reports
Department:
Computer Science
Permanent URL:
Series:
Columbia University Computer Science Technical Reports
Part Number:
CUCS-039-10
Publisher:
Department of Computer Science, Columbia University
Publisher Location:
New York
Abstract:
Software testing of applications in fields like scientific computing, simulation, machine learning, etc. is particularly challenging because many applications in these domains have no reliable "test oracle" to indicate whether the program's output is correct when given arbitrary input. A common approach to testing such applications has been to use a "pseudo-oracle", in which multiple independently-developed implementations of an algorithm process an input and the results are compared. Other approaches include the use of program invariants, formal specification languages, trace and log file analysis, and metamorphic testing. In this paper, we present the results of two empirical studies in which we compare the effectiveness of some of these approaches, including metamorphic testing, pseudo-oracles, and runtime assertion checking. We also analyze the results in terms of the software development process, and discuss suggestions for practitioners and researchers who need to test software without a test oracle.
Subject(s):
Computer science
Item views:
165
Metadata:
View

In Partnership with the Center for Digital Research and Scholarship at Columbia University Libraries/Information Services.