Home

Testing and Validating Machine Learning Classifiers by Metamorphic Testing

Xiaoyuan Xie; Joshua W. K. Ho; Christian Murphy; Gail E. Kaiser; Baowen Xu; Tsong Yueh Chen

Title:
Testing and Validating Machine Learning Classifiers by Metamorphic Testing
Author(s):
Xie, Xiaoyuan
Ho, Joshua W. K.
Murphy, Christian
Kaiser, Gail E.
Xu, Baowen
Chen, Tsong Yueh
Date:
Type:
Technical reports
Department:
Computer Science
Permanent URL:
Series:
Columbia University Computer Science Technical Reports
Part Number:
CUCS-002-10
Abstract:
Machine Learning algorithms have provided important core functionality to support solutions in many scientific computing applications - such as computational biology, computational linguistics, and others. However, it is difficult to test such applications because often there is no "test oracle" to indicate what the correct output should be for arbitrary input. To help address the quality of scientific computing software, in this paper we present a technique for testing the implementations of machine learning classification algorithms on which such scientific computing software depends. Our technique is based on an approach called "metamorphic testing", which has been shown to be effective in such cases. Also presented is a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has very high effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficient to test for the correctness of a supervised classification program. Metamorphic testing is strongly recommended as a complementary approach. Finally we discuss how our findings can be used in other areas of computational science and engineering.
Subject(s):
Computer science
Item views:
575
Metadata:
text | xml

In Partnership with the Center for Digital Research and Scholarship at Columbia University Libraries/Information Services | Terms of Use