Transforming Test Suites into Croissants
Software developers often rely on regression testing to ensure that recent changes made to the source code do not introduce bugs. Flaky tests, which non-deterministically pass or fail regardless of any change to the code, can negatively impact the effectiveness of the regression testing. While state-of-the-art is advancing the techniques for test-flakiness detection and mitigation, the community is missing a systematic approach for generating high-quality benchmarks of flaky tests to compare the effectiveness of such techniques. Inspired by the power of mutation testing in evaluating the fault-detection ability of tests, this paper proposes Croissant, a framework for injecting flakiness into the test suites to assess the effectiveness of test-flakiness detection tools in finding these tests. Croissant implements 18 flakiness-inducing mutation operators. We designed these operators to allow controlling the non-determinism involved in flakiness, i.e., making many mutants deterministically pass or fail to observe flaky behavior. Our extensive empirical evaluation of Croissant on the test suites of 15 real-world projects confirms the ability of designed mutation operators to generate high-quality mutants, and their effectiveness in challenging test-flakiness detection tools in revealing flaky tests.
Thu 20 JulDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | ISSTA 9: Testing 2Technical Papers at Amazon Auditorium (Gates G20) Chair(s): Cristian Cadar Imperial College London | ||
10:30 15mTalk | A Comprehensive Study on Quality Assurance Tools for Java Technical Papers Han Liu East China Normal University, Sen Chen Tianjin University, Ruitao Feng UNSW, Chengwei Liu Nanyang Technological University, Kaixuan Li East China Normal University, Zhengzi Xu Nanyang Technological University, Liming Nie Nanyang Technological University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University DOI | ||
10:45 15mTalk | Transforming Test Suites into Croissants Technical Papers Yang Chen University of Illinois at Urbana-Champaign, Alperen Yildiz Sabanci University, Darko Marinov University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign DOI | ||
11:00 15mTalk | SlipCover: Near Zero-Overhead Code Coverage for Python Technical Papers Juan Altmayer Pizzorno University of Massachusetts Amherst, Emery D. Berger University of Massachusetts Amherst DOI | ||
11:15 15mTalk | To Kill a Mutant: An Empirical Study of Mutation Testing Kills Technical Papers Hang Du University of California at Irvine, Vijay Krishna Palepu Microsoft, James Jones University of California at Irvine DOI | ||
11:30 15mTalk | Systematically Producing Test Orders to Detect Order-Dependent Flaky Tests Technical Papers Chengpeng Li University of Texas at Austin, M. Mahdi Khosravi Middle East Technical University, Wing Lam George Mason University, August Shi University of Texas at Austin DOI | ||
11:45 15mTalk | Extracting Inline Tests from Unit Tests Technical Papers Yu Liu University of Texas at Austin, Pengyu Nie University of Texas at Austin, Anna Guo University of Texas at Austin, Milos Gligoric University of Texas at Austin, Owolabi Legunsen Cornell University DOI |