PhysCov: Physical Test Coverage for Autonomous Vehicles
Adequately exercising the behaviors of autonomous vehicles is fundamental to their validation. However, quantifying an autonomous vehicle's testing adequacy is challenging as the system's behavior is influenced both by its \textit{state} as well as its \textit{physical environment}. To address this challenge, our work builds on two insights. First, data sensed by an autonomous vehicle provides a unique spatial signature of the physical environment inputs. Second, given the vehicle's current state, inputs residing outside the autonomous vehicle's physically reachable regions are less relevant to its behavior. Building on those insights, we introduce an abstraction that enables the computation of a physical environment-state coverage metric, \textit{PhysCov}. The abstraction combines the sensor readings with a physical reachability analysis based on the vehicle's state and dynamics to determine the region of the environment that may affect the autonomous vehicle. It then characterizes that region through a parameterizable geometric approximation that can trade quality for cost. Tests with the same characterizations are deemed to have had similar internal states and exposed to similar environments and thus likely to exercise the same set of behaviors, while tests with distinct characterizations will increase \textit{PhysCov}. A study on two simulated and one real system's dataset examines \textit{PhysCovs}'s ability to quantify an autonomous vehicle's test suite, showcases its characterization cost and precision, investigates its correlation with failures found and potential for test selection, and assesses its ability to distinguish among real-world scenarios.
Wed 19 JulDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | ISSTA 6: Testing 1Technical Papers at Habib Classroom (Gates G01) Chair(s): Karine Even-Mendoza King’s College London | ||
10:30 15mTalk | Synthesizing Speech Test Cases with Text-to-Speech? An Empirical Study on the False Alarms in Automated Speech Recognition Testing Technical Papers Julia Kaiwen Lau Monash University Malaysia, Kelvin Kai Wen Kong Monash University Malaysia, Julian Hao Yong Monash University Malaysia, Per Hoong Tan Monash University Malaysia, Zhou Yang Singapore Management University, Zi Qian Yong Monash University Malaysia, Joshua Chern Wey Low Monash University Malaysia, Chun Yong Chong Monash University Malaysia, Mei Kuan Lim Monash University Malaysia, David Lo Singapore Management University DOI | ||
10:45 15mTalk | PhysCov: Physical Test Coverage for Autonomous Vehicles Technical Papers Carl Hildebrandt University of Virginia, Meriel von Stein University of Virginia, Sebastian Elbaum University of Virginia Link to publication DOI Pre-print | ||
11:00 15mTalk | BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems Technical Papers Mingfei Cheng Singapore Management University, Yuan Zhou Nanyang Technological University, Xiaofei Xie Singapore Management University DOI | ||
11:15 15mTalk | Building Critical Testing Scenarios for Autonomous Driving from Real Accidents Technical Papers Xudong Zhang Institute of Software at Chinese Academy of Sciences, Yan Cai Institute of Software at Chinese Academy of Sciences DOI | ||
11:30 15mTalk | Virtual Reality (VR) Automated Testing in the Wild: A Case Study on Unity-Based VR Applications Technical Papers Dhia Elhaq Rzig University of Michigan - Dearborn, Nafees Iqbal University of Michigan at Dearborn, Isabella Attisano Villanova University, Xue Qin Villanova University, Foyzul Hassan University of Michigan at Dearborn DOI | ||
11:45 15mTalk | Concept-Based Automated Grading of CS-1 Programming Assignments Technical Papers Zhiyu Fan National University of Singapore, Shin Hwei Tan Concordia University, Canada, Abhik Roychoudhury National University of Singapore DOI |