Test case generation tools for REST APIs have grown in number and complexity in recent years. However, their advanced capabilities for automated input generation contrast with the simplicity of their test oracles, which limit the types of failures they can detect to crashes, regressions, and violations of the API specification or design best practices. In this paper, we present AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants—properties of the output that should always hold. In practice, AGORA aims to learn the expected behavior of an API by analyzing previous API requests and their corresponding responses. For this, we extended the Daikon tool for dynamic detection of likely invariants, including the definition of new types of invariants and the implementation of an instrumenter called Beet. Beet converts any OpenAPI specification and a collection of API requests and responses to a format processable by Daikon. As a result, AGORA currently supports the detection of up to 105 different types of invariants in REST APIs. AGORA achieved a total precision of 81.2% when tested on a dataset of 11 operations from 7 industrial APIs. More importantly, the test oracles generated by AGORA detected 6 out of every 10 errors systematically seeded in the outputs of the APIs under test. Additionally, AGORA revealed 11 bugs in APIs with millions of users: Amadeus, GitHub, Marvel, OMDb and YouTube. Our reports have guided developers in improving their APIs, including bug fixes and documentation updates in GitHub. Since it operates in black-box mode, AGORA can be seamlessly integrated into existing API testing tools.

Thu 20 Jul

Displayed time zone: Pacific Time (US & Canada) change

13:30 - 15:00
ISSTA 12: Web and Smart ContractsTechnical Papers at Smith Classroom (Gates G10)
Chair(s): Martin Kellogg New Jersey Institute of Technology
13:30
15m
Talk
Enhancing REST API Testing with NLP Techniques
Technical Papers
Myeongsoo Kim Georgia Institute of Technology, Davide Corradini University of Verona, Saurabh Sinha IBM Research, Alessandro Orso Georgia Institute of Technology, Michele Pasqua University of Verona, Rachel Tzoref-Brill IBM Research, Mariano Ceccato University of Verona
DOI
13:45
15m
Talk
AGORA: Automated Generation of Test Oracles for REST APIsACM SIGSOFT Distinguished Artifact
Technical Papers
Juan C. Alonso University of Seville, Sergio Segura University of Seville, Antonio Ruiz-Cortés University of Seville
DOI
14:00
15m
Talk
ωTest: WebView-Oriented Testing for Android Applications
Technical Papers
Jiajun Hu Hong Kong University of Science and Technology, Lili Wei McGill University, Yepang Liu Southern University of Science and Technology, Shing-Chi Cheung Hong Kong University of Science and Technology
DOI
14:15
15m
Talk
NodeRT: Detecting Races in Node.js Applications Practically
Technical Papers
Jingyao Zhou Nanjing University, Lei Xu Nanjing University, Gongzheng Lu Suzhou City University, Weifeng Zhang Nanjing University of Posts and Telecommunications, Xiangyu Zhang Purdue University
DOI
14:30
15m
Talk
iSyn: Semi-automated Smart Contract Synthesis from Legal Financial Agreements
Technical Papers
Pengcheng Fang Case Western Reserve University, Zhenhua Zou Tsinghua University, Xusheng Xiao Arizona State University, Zhuotao Liu Tsinghua University
DOI
14:45
15m
Talk
Automated Generation of Security-Centric Descriptions for Smart Contract BytecodeACM SIGSOFT Distinguished Paper
Technical Papers
Yu Pan University of Utah, Zhichao Xu University of Utah, Levi Taiji Li University of Utah, Yunhe Yang University of Utah, Mu Zhang University of Utah
DOI