To Kill a Mutant: An Empirical Study of Mutation Testing Kills
Mutation testing has been used and studied for over four decades as a method to assess the strength of a test suite. This technique adds an artificial bug (i.e., a mutation) to a program to produce a mutant, and the test suite is run to determine if any of its test cases are sufficient to detect this mutation (i.e., kill the mutant). In this situation, a test case that fails is the one that kills the mutant. However, little is known about the nature of these kills. In this paper, we present an empirical study that investigates the nature of these kills. We seek to answer questions, such as: How are test cases failing so that they contribute to mutant kills? How many test cases fail for each killed mutant, given that only a single failure is required to kill that mutant? How do program crashes contribute to kills, and what are the origins and nature of these crashes? We found several revealing results across all subjects, including the substantial contribution of "crashes" to test failures leading to mutant kills, the existence of diverse causes for test failures even for a single mutation, and the specific types of exceptions that commonly instigate crashes. We posit that this study and its results should likely be taken into account for practitioners in their use of mutation testing and interpretation of its mutation score, and for researchers who study and leverage mutation testing in their future work.
Thu 20 JulDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | ISSTA 9: Testing 2Technical Papers at Amazon Auditorium (Gates G20) Chair(s): Cristian Cadar Imperial College London | ||
10:30 15mTalk | A Comprehensive Study on Quality Assurance Tools for Java Technical Papers Han Liu East China Normal University, Sen Chen Tianjin University, Ruitao Feng UNSW, Chengwei Liu Nanyang Technological University, Kaixuan Li East China Normal University, Zhengzi Xu Nanyang Technological University, Liming Nie Nanyang Technological University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University DOI | ||
10:45 15mTalk | Transforming Test Suites into Croissants Technical Papers Yang Chen University of Illinois at Urbana-Champaign, Alperen Yildiz Sabanci University, Darko Marinov University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign DOI | ||
11:00 15mTalk | SlipCover: Near Zero-Overhead Code Coverage for Python Technical Papers Juan Altmayer Pizzorno University of Massachusetts Amherst, Emery D. Berger University of Massachusetts Amherst DOI | ||
11:15 15mTalk | To Kill a Mutant: An Empirical Study of Mutation Testing Kills Technical Papers Hang Du University of California at Irvine, Vijay Krishna Palepu Microsoft, James Jones University of California at Irvine DOI | ||
11:30 15mTalk | Systematically Producing Test Orders to Detect Order-Dependent Flaky Tests Technical Papers Chengpeng Li University of Texas at Austin, M. Mahdi Khosravi Middle East Technical University, Wing Lam George Mason University, August Shi University of Texas at Austin DOI | ||
11:45 15mTalk | Extracting Inline Tests from Unit Tests Technical Papers Yu Liu University of Texas at Austin, Pengyu Nie University of Texas at Austin, Anna Guo University of Texas at Austin, Milos Gligoric University of Texas at Austin, Owolabi Legunsen Cornell University DOI |