Machine learning (ML) systems have achieved remarkable performance across a wide area of applications. However, they frequently exhibit unfair behaviors in sensitive application domains (e.g., employment and loan), raising severe fairness concerns. To evaluate and test fairness, engineers often generate individual discriminatory instances to expose unfair behaviors before model deployment.
However, existing baselines ignore the naturalness of generation and produce instances that deviate from the real data distribution, which may fail to reveal the actual model fairness since these unnatural discriminatory instances are unlikely to appear in practice.
To address the problem, this paper proposes a framework named Latent Imitator (LIMI) to generate more natural individual discriminatory instances with the help of a generative adversarial network (GAN), where we imitate the decision boundary of the target model in the semantic latent space of GAN and further samples latent instances on it.
Specifically, we first derive a surrogate linear boundary to coarsely approximate the decision boundary of the target model, which reflects the nature of the original data distribution. Subsequently, to obtain more natural instances, we manipulate random latent vectors to the surrogate boundary with a one-step movement, and further conduct vector calculation to probe two potential discriminatory candidates that may be more closely located in the real decision boundary. Extensive experiments on various datasets demonstrate that our LIMI outperforms other baselines largely in effectiveness ($\times$9.42 instances), efficiency ($\times$8.71 speeds), and naturalness (+19.65%) on average. In addition, we empirically demonstrate that retraining on test samples generated by our approach can lead to improvements in both individual fairness (45.67% on $IF_r$ and 32.81% on $IF_o$) and group fairness (9.86% on $SPD$ and 28.38% on $AOD$). Our codes can be found on our website.

Wed 19 Jul

Displayed time zone: Pacific Time (US & Canada) change

15:30 - 17:00
ISSTA Online 4: Testing and Analysis of DL SystemsTechnical Papers at Smith Classroom (Gates G10)
Chair(s): Elena Sherman Boise State University
15:30
10m
Talk
A Tale of Two Approximations: Tightening Over-Approximation for DNN Robustness Verification via Under-Approximation
Technical Papers
Zhiyi Xue East China Normal University, Si Liu ETH Zurich, Zhaodi Zhang East China Normal University, Yiting Wu East China Normal University, Min Zhang East China Normal University
DOI
15:40
10m
Talk
In Defense of Simple Techniques for Neural Network Test Case Selection
Technical Papers
Shenglin Bao Fudan University, Chaofeng Sha Fudan University, Bihuan Chen Fudan University, Xin Peng Fudan University, Wenyun Zhao Fudan University
DOI
15:50
10m
Talk
ROME: Testing Image Captioning Systems via Recursive Object Melting
Technical Papers
BoXi Yu Chinese University of Hong Kong, Zhiqing Zhong Chinese University of Hong Kong, Jiaqi Li Chinese University of Hong Kong, Yixing Yang Chinese University of Hong Kong, Shilin He Microsoft Research, Pinjia He Chinese University of Hong Kong
DOI
16:00
10m
Talk
ACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Technical Papers
Jingyi Shi Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yang Xiao Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yuekang Li University of New South Wales, Yeting Li Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, DongSong Yu Zhongguancun Laboratory, Chendong Yu Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Hui Su Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Yufeng Chen Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Wei Huo Institute of Information Engineering at Chinese Academy of Sciences
DOI
16:10
10m
Talk
Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing
Technical Papers
Yisong Xiao Beihang University, Aishan Liu Beihang University; Institute of Dataspace, Li Tianlin Nanyang Technological University, Xianglong Liu Beihang University; Institute of Dataspace; Zhongguancun Laboratory
DOI
16:20
10m
Talk
CoopHance: Cooperative Enhancement for Robustness of Deep Learning Systems
Technical Papers
Quan Zhang Tsinghua University, Yongqiang Tian University of Waterloo, Yifeng Ding University of Illinois at Urbana-Champaign, Shanshan Li National University of Defense Technology, Chengnian Sun University of Waterloo, Yu Jiang Tsinghua University, Jiaguang Sun Tsinghua University
DOI
16:30
10m
Talk
Back Deduction Based Testing for Word Sense Disambiguation Ability of Machine Translation Systems
Technical Papers
Jun Wang Nanjing University, Yanhui Li Nanjing University, Xiang Huang Nanjing University, Lin Chen Nanjing University, Xiaofang Zhang Soochow University, Yuming Zhou Nanjing University
DOI
16:40
10m
Talk
CydiOS: A Model-Based Testing Framework for iOS Apps
Technical Papers
Shuohan Wu Hong Kong Polytechnic University, Jianfeng Li Xi’an Jiaotong University, Hao Zhou Hong Kong Polytechnic University, Yongsheng Fang Beijing University of Posts and Telecommunications, Kaifa ZHAO Hong Kong Polytechnic University, Haoyu Wang Huazhong University of Science and Technology, Chenxiong Qian University of Hong Kong, Xiapu Luo Hong Kong Polytechnic University
DOI