AGORA: Automated Generation of Test Oracles for REST APIsACM SIGSOFT Distinguished Artifact
Test case generation tools for REST APIs have grown in number and complexity in recent years. However, their advanced capabilities for automated input generation contrast with the simplicity of their test oracles, which limit the types of failures they can detect to crashes, regressions, and violations of the API specification or design best practices. In this paper, we present AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants—properties of the output that should always hold. In practice, AGORA aims to learn the expected behavior of an API by analyzing previous API requests and their corresponding responses. For this, we extended the Daikon tool for dynamic detection of likely invariants, including the definition of new types of invariants and the implementation of an instrumenter called Beet. Beet converts any OpenAPI specification and a collection of API requests and responses to a format processable by Daikon. As a result, AGORA currently supports the detection of up to 105 different types of invariants in REST APIs. AGORA achieved a total precision of 81.2% when tested on a dataset of 11 operations from 7 industrial APIs. More importantly, the test oracles generated by AGORA detected 6 out of every 10 errors systematically seeded in the outputs of the APIs under test. Additionally, AGORA revealed 11 bugs in APIs with millions of users: Amadeus, GitHub, Marvel, OMDb and YouTube. Our reports have guided developers in improving their APIs, including bug fixes and documentation updates in GitHub. Since it operates in black-box mode, AGORA can be seamlessly integrated into existing API testing tools.
Thu 20 JulDisplayed time zone: Pacific Time (US & Canada) change
13:30 - 15:00 | ISSTA 12: Web and Smart ContractsTechnical Papers at Smith Classroom (Gates G10) Chair(s): Martin Kellogg New Jersey Institute of Technology | ||
13:30 15mTalk | Enhancing REST API Testing with NLP Techniques Technical Papers Myeongsoo Kim Georgia Institute of Technology, Davide Corradini University of Verona, Saurabh Sinha IBM Research, Alessandro Orso Georgia Institute of Technology, Michele Pasqua University of Verona, Rachel Tzoref-Brill IBM Research, Mariano Ceccato University of Verona DOI | ||
13:45 15mTalk | AGORA: Automated Generation of Test Oracles for REST APIsACM SIGSOFT Distinguished Artifact Technical Papers Juan C. Alonso University of Seville, Sergio Segura University of Seville, Antonio Ruiz-Cortés University of Seville DOI | ||
14:00 15mTalk | ωTest: WebView-Oriented Testing for Android Applications Technical Papers Jiajun Hu Hong Kong University of Science and Technology, Lili Wei McGill University, Yepang Liu Southern University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology DOI | ||
14:15 15mTalk | NodeRT: Detecting Races in Node.js Applications Practically Technical Papers Jingyao Zhou Nanjing University, Lei Xu Nanjing University, Gongzheng Lu Suzhou City University, Weifeng Zhang Nanjing University of Posts and Telecommunications, Xiangyu Zhang Purdue University DOI | ||
14:30 15mTalk | iSyn: Semi-automated Smart Contract Synthesis from Legal Financial Agreements Technical Papers Pengcheng Fang Case Western Reserve University, Zhenhua Zou Tsinghua University, Xusheng Xiao Arizona State University, Zhuotao Liu Tsinghua University DOI | ||
14:45 15mTalk | Automated Generation of Security-Centric Descriptions for Smart Contract BytecodeACM SIGSOFT Distinguished Paper Technical Papers Yu Pan University of Utah, Zhichao Xu University of Utah, Levi Taiji Li University of Utah, Yunhe Yang University of Utah, Mu Zhang University of Utah DOI |