Testing the Compiler for a New-Born Programming Language: An Industrial Case Study (Experience Paper)
Due to the critical role of compilers, many compiler testing techniques have been proposed, two most notable categories among which are grammar-based and metamorphic-based techniques. All of them have been extensively studied for testing mature compilers. However, it is typical to develop a new compiler for a new-born programming language in practice. In this scenario, the existing techniques are hardly applicable due to some major reasons: (1) no reference compilers to support differential testing, (2) lack of program analysis tools to support most of metamorphic-based compiler testing, (3) substantial implementation effort incurred by different programming language features. Hence, it is unknown how the existing techniques perform in this new scenario.
In this work, we conduct the first exploration (i.e., an industrial case study) to investigate the performance of the existing techniques in this new scenario with substantial adaptations. We adapted grammar-based compiler testing to this scenario by synthesizing new test programs based on code snippets and using compilation crash as test oracle due to the lack of reference compilers for differential testing. We also adapted metamorphic-based compiler testing to this scenario by constructing equivalent test programs under any inputs to relieve the dependence on program analysis tools. We call the adapted techniques SynFuzz and MetaFuzz, respectively.
We evaluated both SynFuzz and MetaFuzz on two versions of a new compiler for a new-born programming language in a global IT company. By comparing with the testing practice adopted by the testing team and the general fuzzer (AFL), SynFuzz can detect more bugs during the same testing time, and both SynFuzz and MetaFuzz can complement the other two techniques. In particular, SynFuzz and MetaFuzz have detected 11 previously unknown bugs, all of which have been fixed by the developers. From the industrial case study, we summarized a series of lessons and suggestions for practical use and future research.
Tue 18 JulDisplayed time zone: Pacific Time (US & Canada) change
15:30 - 17:00 | ISSTA Online 3: Empirical StudiesTechnical Papers at Bezos Seminar Room (Gates G04) Chair(s): Jordan Samhi University of Luxembourg | ||
15:30 10mTalk | Understanding Breaking Changes in the Wild Technical Papers Dhanushka Jayasuriya University of Auckland, Valerio Terragni University of Auckland, Jens Dietrich Victoria University of Wellington, Samuel Ou University of Auckland, Kelly Blincoe University of Auckland DOI | ||
15:40 10mTalk | LiResolver: License Incompatibility Resolution for Open Source Software Technical Papers Sihan Xu Nankai University, Ya Gao Nankai University, Lingling Fan Nankai University, Linyu Li Nankai University, Xiangrui Cai Nankai University, Zheli Liu Nankai University DOI | ||
15:50 10mTalk | An Empirical Study on Concurrency Bugs in Interrupt-Driven Embedded Software Technical Papers Chao Li Beijing Institute of Control Engineering; Beijing Sunwise Information Technology, Rui Chen Beijing Institute of Control Engineering; Beijing Sunwise Information Technology, Boxiang Wang Beijing Institute of Control Engineering; Beijing Sunwise Information Technology, Zhixuan Wang Xidian University, Tingting Yu Beijing Institute of Control Engineering; Beijing Sunwise Information Technology, Yunsong Jiang Beijing Institute of Control Engineering; Beijing Sunwise Information Technology, Mengfei Yang China Academy of Space Technology DOI | ||
16:00 10mTalk | An Empirical Study of Functional Bugs in Android AppsACM SIGSOFT Distinguished Paper Technical Papers Yiheng Xiong East China Normal University, Mengqian Xu East China Normal University, Ting Su East China Normal University, Jingling Sun East China Normal University, Jue Wang Nanjing University, He Wen East China Normal University, Geguang Pu East China Normal University, Jifeng He East China Normal University, Zhendong Su ETH Zurich DOI | ||
16:10 10mTalk | Testing the Compiler for a New-Born Programming Language: An Industrial Case Study (Experience Paper) Technical Papers Yingquan Zhao Tianjin University, Junjie Chen Tianjin University, Ruifeng Fu Tianjin University, Haojie Ye Huawei, Zan Wang Tianjin University DOI | ||
16:20 10mTalk | An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors Technical Papers Kunlun Ren Huazhong University of Science and Technology, Qiang Weizhong Huazhong University of Science and Technology, Yueming Wu Nanyang Technological University, yi zhou Huazhong University of Science and Technology, Deqing Zou Huazhong University of Science and Technology, Hai Jin Huazhong University of Science and Technology DOI | ||
16:30 10mTalk | Security Checking of Trigger-Action-Programming Smart Home Integrations Technical Papers Lei Bu Nanjing University, Qiuping Zhang Nanjing University, Suwan Li Nanjing University, Jinglin Dai Nanjing University, Guangdong Bai University of Queensland, Kai Chen Institute of Information Engineering at Chinese Academy of Sciences, Xuandong Li Nanjing University DOI | ||
16:40 10mTalk | Third-Party Library Dependency for Large-Scale SCA in the C/C++ Ecosystem: How Far Are We? Technical Papers Ling Jiang Southern University of Science and Technology, Hengchen Yuan Southern University of Science and Technology, Qiyi Tang Tencent Security Keen Lab, Sen Nie Tencent Security Keen Lab, Shi Wu Tencent Security Keen Lab, Yuqun Zhang Southern University of Science and Technology DOI | ||
16:50 10mTalk | Who Judges the Judge: An Empirical Study on Online Judge Tests Technical Papers Kaibo Liu Peking University, Yudong Han Peking University, Jie M. Zhang King’s College London, Zhenpeng Chen University College London, Federica Sarro University College London, Mark Harman University College London, Gang Huang Peking University; National Key Laboratory of Data Space Technology and System, Yun Ma Peking University DOI | ||