Reading notes: Adversarial Examples Are Not Bugs, They Are Features

-
Source: Ilyas, Andrew, et al. "Adversarial examples are not bugs, they are features." arXiv preprint arXiv:1905.02175 (2019).  https://arxiv.org/abs/1905.02175

Introduction

Over the past few years, the adversarial attacks - which aim to force the machine learning systems to make misclassification by adding slightly perturbation - have received signification attention in the community. There are a lot of works show that intentional perturbation which is imperceptible to human can easily fool a deep learning classifier. In response to the threat, there has been much work on defensive techniques that help models against adversarial examples. But none of them really answer the fundamental question: Why do these adversarial attacks arise?

By far, some previous research works view the adversarial examples as aberrations come from the high dimensional nature of the input space that will eventually disappear when we have enough training dataset or better training algorithms. Different from this point of view, a group of MIT researchers proposes a new perspective on the adversarial examples in this paper:

"Adversarial vulnerability is a direct result of our models’ sensitivity to well-generalizing features in the data."

The researchers suggest that the classifiers are often trained to maximize accuracy solely without much prior context about the human-related concept on classes. Inevitable, classifiers would use any available signals to deliver the result even some signals are incomprehensible to humans or likely be ignored by humans. For example, it is more natural to use "eyes" or "ears" than to use "fur" to distinguish dog and cat as shown in the following figure. They posit that the model relies on those "non-robust" features, leads to adversarial perturbations.

Cat or Dog?

 Experiments

To support this theory, they show that it is possible to disentangle robust and non-robust features in some standard image dataset. A conceptual description of the experiments is shown in Figure 1. Given a training dataset, they can construct a "robustified" dataset, in which samples primarily contain robust features. By training on this dataset, the model should have good robust accuracy (Figure 1(a)). On the other hand, they construct a dataset with images that attacked to predict a wrong label. They use the perturbed images paired with a misclassified label to build a new dataset. They show that this dataset suffices to train a classifier with good performance on the standard test set (Figure 2(b)). This indicates that non-robust features are indeed valuable features.


Results

Figure 2 shows the experiment results. By restricting the dataset to only robust feature, the standard training results in good accuracy in both standard and adversarial setting. While training on "non-robust" dataset, the model is vulnerable to adversarial. Overall, the findings corroborate the hypothesis that adversarial examples arise from non-robust features of data itself.


The standard training on the "mislabeled" dataset actually generalize to the original test set, as shown in Table 1, which indicates that non-robust feature is useful for classification in the standard-setting.

The experiment establishes adversarial vulnerability as a purely human-centric phenomenon, since, from the supervised learning point of view, non-robust features are as important as robust features. For the future, if we want a model to be human-specified robustness, we need to incorporate priors into architecture or training process. 

More in the Paper

In the paper, the authors also provide a precise framework for discussing the robust and non-robust feature, more experiments result and a theoretical model for studying these features.











Comments

Post a Comment

Popular posts from this blog

Reading notes: Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation

-
Image
This blog is the reading note for the paper " Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation " by Uktku Ozbulak, et al., MICCAI 2019.  Broadly speaking, the authors try to generate more realistic adversarial attacks on medical image segmentation tasks. Specifically, they design a very novel generation method to produce an adversarial example that misleads the trained model to predict a targeted  realistic prediction mask with limited perturbation. Introduction Recent studies have developed various deep learning-based models in medical imaging systems. They have shown great performance over some clinical prediction tasks with minimum labor expenses. However, deep learning-based models  come with a major security problem: they can be totally fooled by so-called adversarial examples. Adversarial examples are inputs of the model with imperceptibly small perturbation that lead to misclassification. Consequently, the users of t...
Read more

Reading notes: Federated Learning with Only Positive Labels

-
Image
This blog is the reading note for the paper "Federated Learning with Only Positive Labels" by Yu, Felix X., et al. ICML 2020. Broadly speaking, the authors consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class. Specifically, they propose a generic framework namely Federated Averaging with Spreadour (FedAwS), where the server imposes a geometric regularizer after each round to encourage classes to spread-out in the embedding space. They show that FedAwS can almost match the performance of conventional learning both theoretically and empirically. Introduction In this work, the authors consider learning a classification model in the federated learning setup, where each user has only access to a single class. Examples of such settings include decentralized training of face recognition models or speaker identification models, where the classifier of the users has sensitive ...
Read more

Reading notes: Degenerative Adversarial NeuroImage Nets: Generating Images that Mimic Disease Progression

-
Image
This blog is the reading note for the paper "Degenerative Adversarial NeuroImage Nets: Generating Images that Mimic Disease Progression" by Ravi, Daniele, et al. MICCAI 2019. Broadly speaking, the authors try to simulating images representative of neurodegenerative diseases. Specifically, they designed a novel network named Degenerative Adversarial NeuroImage Net (DaniNet) to produce accurate and convincing synthetic images that emulate disease progression. Introduction  Disease progression modelling help to map out longitudinal change during chronic diseases. However, modelling temporal neurodegeneration of full resolution MRI is still a challenging problem. Several traditional simulators proposed in the literature are extremely resource-demanding and is not scalable to high resolution image. [1] proposed a deep learning framework based on GAN to manipulate MRI, but their assumption is over-simplified: 1. disease progression is modelled linearly and 2. morpholog...
Read more