Testing and Validating Machine Learning Classifiers by Metamorphic Testing

Xiaoyuan Xie; Joshua W. K. Ho; Christian Murphy; Gail E. Kaiser; Baowen Xu; Tsong Yueh Chen

Testing and Validating Machine Learning Classifiers by Metamorphic Testing
Xie, Xiaoyuan
Ho, Joshua W. K.
Murphy, Christian
Kaiser, Gail E.
Xu, Baowen
Chen, Tsong Yueh
Technical reports
Computer Science
Persistent URL:
Columbia University Computer Science Technical Reports
Part Number:
Department of Computer Science, Columbia University
Publisher Location:
New York
Machine Learning algorithms have provided important core functionality to support solutions in many scientific computing applications - such as computational biology, computational linguistics, and others. However, it is difficult to test such applications because often there is no "test oracle" to indicate what the correct output should be for arbitrary input. To help address the quality of scientific computing software, in this paper we present a technique for testing the implementations of machine learning classification algorithms on which such scientific computing software depends. Our technique is based on an approach called "metamorphic testing", which has been shown to be effective in such cases. Also presented is a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has very high effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficient to test for the correctness of a supervised classification program. Metamorphic testing is strongly recommended as a complementary approach. Finally we discuss how our findings can be used in other areas of computational science and engineering.
Computer science
Item views
text | xml
Suggested Citation:
Xiaoyuan Xie, Joshua W. K. Ho, Christian Murphy, Gail E. Kaiser, Baowen Xu, Tsong Yueh Chen, , Testing and Validating Machine Learning Classifiers by Metamorphic Testing, Columbia University Academic Commons, .

Columbia University Libraries | Policies | FAQ