Thu 20 Jul 2023 10:45 - 11:00 at Amazon Auditorium (Gates G20) - ISSTA 9: Testing 2 Chair(s): Cristian Cadar

Software developers often rely on regression testing to ensure that recent changes made to the source code do not introduce bugs. Flaky tests, which non-deterministically pass or fail regardless of any change to the code, can negatively impact the effectiveness of the regression testing. While state-of-the-art is advancing the techniques for test-flakiness detection and mitigation, the community is missing a systematic approach for generating high-quality benchmarks of flaky tests to compare the effectiveness of such techniques. Inspired by the power of mutation testing in evaluating the fault-detection ability of tests, this paper proposes Croissant, a framework for injecting flakiness into the test suites to assess the effectiveness of test-flakiness detection tools in finding these tests. Croissant implements 18 flakiness-inducing mutation operators. We designed these operators to allow controlling the non-determinism involved in flakiness, i.e., making many mutants deterministically pass or fail to observe flaky behavior. Our extensive empirical evaluation of Croissant on the test suites of 15 real-world projects confirms the ability of designed mutation operators to generate high-quality mutants, and their effectiveness in challenging test-flakiness detection tools in revealing flaky tests.

Thu 20 Jul

Displayed time zone: Pacific Time (US & Canada) change

10:30 - 12:00
ISSTA 9: Testing 2Technical Papers at Amazon Auditorium (Gates G20)
Chair(s): Cristian Cadar Imperial College London
10:30
15m
Talk
A Comprehensive Study on Quality Assurance Tools for Java
Technical Papers
Han Liu East China Normal University, Sen Chen Tianjin University, Ruitao Feng UNSW, Chengwei Liu Nanyang Technological University, Kaixuan Li East China Normal University, Zhengzi Xu Nanyang Technological University, Liming Nie Nanyang Technological University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University
DOI
10:45
15m
Talk
Transforming Test Suites into Croissants
Technical Papers
Yang Chen University of Illinois at Urbana-Champaign, Alperen Yildiz Sabanci University, Darko Marinov University of Illinois at Urbana-Champaign, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign
DOI
11:00
15m
Talk
SlipCover: Near Zero-Overhead Code Coverage for Python
Technical Papers
Juan Altmayer Pizzorno University of Massachusetts Amherst, Emery D. Berger University of Massachusetts Amherst
DOI
11:15
15m
Talk
To Kill a Mutant: An Empirical Study of Mutation Testing Kills
Technical Papers
Hang Du University of California at Irvine, Vijay Krishna Palepu Microsoft, James Jones University of California at Irvine
DOI
11:30
15m
Talk
Systematically Producing Test Orders to Detect Order-Dependent Flaky Tests
Technical Papers
Chengpeng Li University of Texas at Austin, M. Mahdi Khosravi Middle East Technical University, Wing Lam George Mason University, August Shi University of Texas at Austin
DOI
11:45
15m
Talk
Extracting Inline Tests from Unit Tests
Technical Papers
Yu Liu University of Texas at Austin, Pengyu Nie University of Texas at Austin, Anna Guo University of Texas at Austin, Milos Gligoric University of Texas at Austin, Owolabi Legunsen Cornell University
DOI