ACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Deep learning (DL) applications are prevalent nowadays as they can help with multiple tasks. DL libraries are essential for building DL applications. Furthermore, DL operators are the important building blocks of the DL libraries, that compute the multi-dimensional data (tensors). Therefore, bugs in DL operators can have great impacts. Testing is a practical approach for detecting bugs in DL operators. In order to test DL operators effectively, it is essential that the test cases pass the input validity check and are able to reach the core function logic of the operators. Hence, extracting the input validation constraints is required for generating high-quality test cases. Existing techniques rely on either human effort or documentation of DL library APIs to extract the constraints. They cannot extract complex constraints and the extracted constraints may differ from the actual code implementation.
To address the challenge, we propose ACETest, a technique to automatically extract input validation constraints from the code to build valid yet diverse test cases which can effectively unveil bugs in the core function logic of DL operators. For this purpose, ACETest can automatically identify the input validation code in DL operators, extract the related constraints and generate test cases according to the constraints. The experimental results on popular DL libraries, TensorFlow and PyTorch, demonstrate that ACETest can extract constraints with higher quality than state-of-the-art (SOTA) techniques. Moreover, ACETest is capable of extracting 96.4% more constraints and detecting 1.95 to 55 times more bugs than SOTA techniques. In total, we have used ACETest to detect 108 previously unknown bugs on TensorFlow and PyTorch, with 87 of them confirmed by the developers. Lastly, five of the bugs were assigned with CVE IDs due to their security impacts.
Wed 19 JulDisplayed time zone: Pacific Time (US & Canada) change
15:30 - 17:00 | ISSTA Online 4: Testing and Analysis of DL SystemsTechnical Papers at Smith Classroom (Gates G10) Chair(s): Elena Sherman Boise State University | ||
15:30 10mTalk | A Tale of Two Approximations: Tightening Over-Approximation for DNN Robustness Verification via Under-Approximation Technical Papers Zhiyi Xue East China Normal University, Si Liu ETH Zurich, Zhaodi Zhang East China Normal University, Yiting Wu East China Normal University, Min Zhang East China Normal University DOI | ||
15:40 10mTalk | In Defense of Simple Techniques for Neural Network Test Case Selection Technical Papers Shenglin Bao Fudan University, Chaofeng Sha Fudan University, Bihuan Chen Fudan University, Xin Peng Fudan University, Wenyun Zhao Fudan University DOI | ||
15:50 10mTalk | ROME: Testing Image Captioning Systems via Recursive Object Melting Technical Papers BoXi Yu Chinese University of Hong Kong, Zhiqing Zhong Chinese University of Hong Kong, Jiaqi Li Chinese University of Hong Kong, Yixing Yang Chinese University of Hong Kong, Shilin He Microsoft Research, Pinjia He Chinese University of Hong Kong DOI | ||
16:00 10mTalk | ACETest: Automated Constraint Extraction for Testing Deep Learning Operators Technical Papers Jingyi Shi Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yang Xiao Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yuekang Li University of New South Wales, Yeting Li Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, DongSong Yu Zhongguancun Laboratory, Chendong Yu Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Hui Su Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yufeng Chen Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Wei Huo Institute of Information Engineering at Chinese Academy of Sciences DOI | ||
16:10 10mTalk | Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing Technical Papers Yisong Xiao Beihang University, Aishan Liu Beihang University; Institute of Dataspace, Li Tianlin Nanyang Technological University, Xianglong Liu Beihang University; Institute of Dataspace; Zhongguancun Laboratory DOI | ||
16:20 10mTalk | CoopHance: Cooperative Enhancement for Robustness of Deep Learning Systems Technical Papers Quan Zhang Tsinghua University, Yongqiang Tian University of Waterloo, Yifeng Ding University of Illinois at Urbana-Champaign, Shanshan Li National University of Defense Technology, Chengnian Sun University of Waterloo, Yu Jiang Tsinghua University, Jiaguang Sun Tsinghua University DOI | ||
16:30 10mTalk | Back Deduction Based Testing for Word Sense Disambiguation Ability of Machine Translation Systems Technical Papers Jun Wang Nanjing University, Yanhui Li Nanjing University, Xiang Huang Nanjing University, Lin Chen Nanjing University, Xiaofang Zhang Soochow University, Yuming Zhou Nanjing University DOI | ||
16:40 10mTalk | CydiOS: A Model-Based Testing Framework for iOS Apps Technical Papers Shuohan Wu Hong Kong Polytechnic University, Jianfeng Li Xi’an Jiaotong University, Hao Zhou Hong Kong Polytechnic University, Yongsheng Fang Beijing University of Posts and Telecommunications, Kaifa ZHAO Hong Kong Polytechnic University, Haoyu Wang Huazhong University of Science and Technology, Chenxiong Qian University of Hong Kong, Xiapu Luo Hong Kong Polytechnic University DOI | ||