Thu 20 Jul 2023 11:15 - 11:30 at Amazon Auditorium (Gates G20) - ISSTA 9: Testing 2 Chair(s): Cristian Cadar

Mutation testing has been used and studied for over four decades as a method to assess the strength of a test suite. This technique adds an artificial bug (i.e., a mutation) to a program to produce a mutant, and the test suite is run to determine if any of its test cases are sufficient to detect this mutation (i.e., kill the mutant). In this situation, a test case that fails is the one that kills the mutant. However, little is known about the nature of these kills. In this paper, we present an empirical study that investigates the nature of these kills. We seek to answer questions, such as: How are test cases failing so that they contribute to mutant kills? How many test cases fail for each killed mutant, given that only a single failure is required to kill that mutant? How do program crashes contribute to kills, and what are the origins and nature of these crashes? We found several revealing results across all subjects, including the substantial contribution of "crashes" to test failures leading to mutant kills, the existence of diverse causes for test failures even for a single mutation, and the specific types of exceptions that commonly instigate crashes. We posit that this study and its results should likely be taken into account for practitioners in their use of mutation testing and interpretation of its mutation score, and for researchers who study and leverage mutation testing in their future work.

Thu 20 Jul

Displayed time zone: Pacific Time (US & Canada) change

10:30 - 12:00
ISSTA 9: Testing 2Technical Papers at Amazon Auditorium (Gates G20)
Chair(s): Cristian Cadar Imperial College London
10:30
15m
Talk
A Comprehensive Study on Quality Assurance Tools for Java
Technical Papers
Han Liu East China Normal University, Sen Chen Tianjin University, Ruitao Feng UNSW, Chengwei Liu Nanyang Technological University, Kaixuan Li East China Normal University, Zhengzi Xu Nanyang Technological University, Liming Nie Nanyang Technological University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University
DOI
10:45
15m
Talk
Transforming Test Suites into Croissants
Technical Papers
Yang Chen University of Illinois at Urbana-Champaign, Alperen Yildiz Sabanci University, Darko Marinov University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign
DOI
11:00
15m
Talk
SlipCover: Near Zero-Overhead Code Coverage for Python
Technical Papers
Juan Altmayer Pizzorno University of Massachusetts Amherst, Emery D. Berger University of Massachusetts Amherst
DOI
11:15
15m
Talk
To Kill a Mutant: An Empirical Study of Mutation Testing Kills
Technical Papers
Hang Du University of California at Irvine, Vijay Krishna Palepu Microsoft, James Jones University of California at Irvine
DOI
11:30
15m
Talk
Systematically Producing Test Orders to Detect Order-Dependent Flaky Tests
Technical Papers
Chengpeng Li University of Texas at Austin, M. Mahdi Khosravi Middle East Technical University, Wing Lam George Mason University, August Shi University of Texas at Austin
DOI
11:45
15m
Talk
Extracting Inline Tests from Unit Tests
Technical Papers
Yu Liu University of Texas at Austin, Pengyu Nie University of Texas at Austin, Anna Guo University of Texas at Austin, Milos Gligoric University of Texas at Austin, Owolabi Legunsen Cornell University
DOI