DeepAtash: Focused Test Generation for Deep Learning Systems
When deployed in the operation environment, Deep Learning (DL) systems often experience the so-called development to operation (dev2op) data shift, which causes a lower prediction accuracy on field data as compared to the one measured on the test set during development. To address the dev2op shift, developers must obtain new data with the newly observed features, as these are under-represented in the train/test set, and must use them to fine tune the DL model, so as to reach the desired accuracy level.
In this paper, we address the issue of acquiring new data with the specific features observed in operation, which caused a dev2op shift, by proposing DeepAtash, a novel search-based focused testing approach for DL systems.
DeepAtash targets a cell in the feature space, defined as a combination of feature ranges, to generate misbehaviour-inducing inputs with predefined features.
Experimental results show that DeepAtash was able to generate up to 29X more targeted, failure-inducing inputs than the baseline approach. The inputs generated by DeepAtash were useful to significantly improve the quality of the original DL systems through fine tuning not only on data with the targeted features, but quite surprisingly also on inputs drawn from the original distribution.
Wed 19 JulDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | ISSTA 5: Improving Deep Learning SystemsTechnical Papers at Smith Classroom (Gates G10) Chair(s): Michael Pradel University of Stuttgart | ||
10:30 15mTalk | Understanding and Tackling Label Errors in Deep Learning-Based Vulnerability Detection (Experience Paper) Technical Papers XuNie Huazhong University of Science and Technology; Beijing University of Posts and Telecommunications, Ningke Li Huazhong University of Science and Technology, Kailong Wang Huazhong University of Science and Technology, Shangguang Wang Beijing University of Posts and Telecommunications, Xiapu Luo Hong Kong Polytechnic University, Haoyu Wang Huazhong University of Science and Technology DOI | ||
10:45 15mTalk | Improving Binary Code Similarity Transformer Models by Semantics-Driven Instruction Deemphasis Technical Papers Xiangzhe Xu Purdue University, Shiwei Feng Purdue University, Yapeng Ye Purdue University, Guangyu Shen Purdue University, Zian Su Purdue University, Siyuan Cheng Purdue University, Guanhong Tao Purdue University, Qingkai Shi Purdue University, Zhuo Zhang Purdue University, Xiangyu Zhang Purdue University DOI | ||
11:00 15mTalk | CILIATE: Towards Fairer Class-Based Incremental Learning by Dataset and Training Refinement Technical Papers Xuanqi Gao Xi’an Jiaotong University, Juan Zhai University of Massachusetts Amherst, Shiqing Ma UMass Amherst, Chao Shen Xi’an Jiaotong University, Yufei Chen Xi’an Jiaotong University; City University of Hong Kong, Shiwei Wang Xi’an Jiaotong University DOI Pre-print | ||
11:15 15mTalk | DeepAtash: Focused Test Generation for Deep Learning Systems Technical Papers DOI | ||
11:30 15mTalk | Systematic Testing of the Data-Poisoning Robustness of KNN Technical Papers Yannan Li University of Southern California, Jingbo Wang University of Southern California, Chao Wang University of Southern California DOI | ||
11:45 15mTalk | Semantic-Based Neural Network Repair Technical Papers DOI | ||