Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision

Introduction

Most of today’s machine learning methods rely on the assumption that the training and testing data are drawn from a same distribution . One implication is that models are susceptible to poor real-world performance when the test data differs from what is observed during training. This limited capability to generalise partly arises because supervised training essentially amounts to identifying correlations between given examples and their labels. However, correlations can be spurious, in the sense that they may reflect dataset-specific biases or sampling artifacts, rather than intrinsic properties of the task of interest . When spurious correlations do not hold in the test data, the model’s predictive performance suffers and its output becomes unreliable and unpredictable. For example, an image recognition system may rely on common co-occurrences of objects, such as people together with a dining table, rather visual evidence for each recognized object. This system could then hallucinate people when a table is observed (Fig. 4).

A model capable of generalization and extrapolation beyond its training distribution should ideally capture the causal mechanisms at play behind the data. Acquiring additional training examples from the same distribution cannot help in this process . Rather, we need either to inject strong prior assumptions in the model, such as inductive biases encoded in the architecture of a neural network, or a different type of training information. Ad hoc methods such as data augmentation and domain randomization fall in the former category, and they only defer the limits of the system by hand-designed rules.

In this paper, we show that many existing datasets contain an overlooked signal that is informative about their causal data-generating process. This information is present in the form of groupings of training examples, and it is often discarded by the shuffling of points occurring during stochastic training. We show that this information can be used to learn a model that is more faithful to the causal model behind the data. This training signal is fundamentally different and complementary to the labels of individual points. We use pairs of minimally-dissimilar, differently-labeled training examples, which we interpret as counterfactuals of one another. In some datasets, such pairs are provided explicitly . In others, they can be identified from existing annotations .

The intuition for our approach is that relations between pairs of counterfactual examples indicate what changes in the input space map to changes in the space of labels. In a classification setting, this serves to constrain the geometry of a model’s decision boundary between classes. Loosely speaking, we complement the traditional “curve fitting” to individual training points of standard supervised learning, with “aligning the curve” with pairs of counterfactual training points.

We describe a novel training objective (gradient supervision) and its implementation on various architectures of neural networks. The vector difference in input space between pairs of counterfactual examples serves to supervise the orientation of the gradient of the network. We demonstrate the benefits of the method on four tasks in computer vision and natural language processing (NLP) that are notoriously prone to poor generalization due to dataset biases: visual question answering (VQA), multi-label image classification, sentiment analysis, and natural language inference. We use annotations from existing datasets that are usually disregarded, and we demonstrate significant improvements in generalization to out-of-distribution test sets for all tasks.

In summary, the contributions of this paper are as follows.

We propose to use relations between training examples as additional information in the supervised training of neural networks (Section 3.1). We show that they provide a fundamentally different and complementary training signal to the fitting of individual examples, and explain how they improve generalization (Section 3.3).

We describe a novel training objective (gradient supervision) to use this information and its implementation on multiple architectures of neural networks (Section 4).

We demonstrate that the required annotations are present in a number of existing datasets in computer vision and NLP, although they are usually discarded. We show that our technique brings improvements in out-of-distribution generalization on VQA, multi-label image classification, sentiment analysis, and natural language inference.

Related work

This work proposes a new training objective that improves the generalization capabilities of models trained with supervision. This touches a number of core concepts in machine learning.

The predictive performance of machine learning models rests on the fundamental assumption of statistical similarity of the distributions of training and test data. There is a growing interest for evaluating and addressing the limits of this assumption. Evaluation on out-of-distribution data is increasingly common in computer vision and NLP . These evaluations have shown that some of the best models can be right for the wrong reasons . This happens when they rely on dataset-specific biases and artifacts rather than intrinsic properties of the task of interest. When these biases do not hold in the test data, the predictive performance of the models can drop dramatically .

When poor generalization is viewed as a deficiency of the training data, it is often referred to as dataset biases. They correspond to correlations between inputs and labels in a dataset that can be exploited by a model to exhibit strong performance on a test set containing these same biases, without actually solving the task of interest. Several popular datasets used in vision-and-language and NLP have been shown to exhibit strong biases, leading to an inflated sense of progress on these tasks.

Recent works have discussed generalization from a causal perspective . This sheds light on the possible avenues for addressing the issue. In order to generalize perfectly, a model should ideally capture the real-world causal mechanisms at play behind the data. The limits of identifiability of causal models from observational data have been well studied . In particular, additional data from a single biased training distribution can not solve the problem. The alternative options are to use strong assumptions (e.g. inductive biases, engineered architectures, hand-designed data augmentations), or additional data, collected in controlled conditions and/or of a different type than labeled examples. This work uses the latter option, using pairings of training examples that represent counterfactuals of one another. Recent works that follow this line include the principle of invariant risk minimization (IRM ). IRM uses multiple training environments, i.e. non-IID training distributions, to discover generalizable invariances in the data. Teney et al. showed that existing datasets could be automatically partitioned to create these environments, and demonstrated improvements in generalization for the task of visual question answering (VQA).

Generalization is also related to the wide area of domain adaptation . Our objective in this paper is not to adapt to a particular new domain, but rather to learn a model that generalizes more broadly by using annotations indicative of the causal mechanisms of the task of interest. In domain adaptation, the idea of finding a data representation that is invariant across domains is limiting, because the true causal factors that our model should rely on may differ in their distribution across training domains. We refer the reader to for a formal discussion of these issues.

The growing popularity of high-level tasks in vision-and-language has brought the issue of dataset biases to the forefront. In VQA, language biases cause models to be overly reliant on the presence of particular words in a question. Improving the data collection process can help but it only addresses precisely identified biases and confounders. Controlled evaluations for VQA now include out-of-distribution test sets . Several models and training methods have been proposed with significant improvements. They all use strong prior knowledge about the task and/or additional annotations (question types) to improve generalization. Some methods also supervise the model’s attention with ground truth human attention maps . All of these methods are specific to VQA or to captioning whereas we describe a much more general approach.

Evaluating generalization overlaps with the growing interest in adversarial examples for evaluation . The term has been used to refer both to examples purposefully generated to fool existing models , but also to hard natural examples that current models struggle with . Our method is most related to the use of these examples for adversarial training. Existing methods focus mostly on the generation of these examples then mix them with the original data in a form of data augmentation . We argue that this shuffling of examples destroys valuable information. In many datasets, we demonstrate that relations between training points contain valuable information. The above methods also aim at improving robustness to targeted adversarial attacks, which often use inputs outside the manifold of natural data. Most of them rely on prior knowledge and unsupervised regularizers whereas we seek to exploit additional supervision to improve generalization on natural data.

Proposed approach

We start with an intuitive motivation for our approach, then describe its technical realization. In Section 3.3, we analyze more formally how it can improve generalization. In Section 4, we demonstrate its application to a range of tasks.

Our motivating intuition is that many datasets contain information that is indicative of the shape of an ideal $f$ (in the sense of being faithful to the data-generating process, see Section 3.3) between training points. In particular, we are interested in pairs of training examples that are counterfactuals of one another. Given a labeled example $({\boldsymbol{x}}_{1},y_{1})$, we define its counterfactuals as examples such as $({\boldsymbol{x}}_{2},y_{2})$ that represents an alternative premise ${\boldsymbol{x}}_{2}$ (“counter to the facts”) that lead to different outcome $y_{2}$. These points represent “minimal changes” ($||{\boldsymbol{x}}_{1}-{\boldsymbol{x}}_{2}||\ll$, in a semantic sense) such that their label $y_{1}\neq y_{2}$. All possible counterfactuals to a given example ${\boldsymbol{x}}_{1}$ constitute a distribution. We assume the availability of samples from it, forming pairs such as $\{({\boldsymbol{x}}_{1},y_{1}),({\boldsymbol{x}}_{2},y_{2})\}$. The counterfactual relation is undirected.

Some existing datasets explicitly contain pairs of counterfactual examples . For example, contains sentences (movie reviews) with positive and negative labels. Annotators were instructed to edit a set of sentences to flip the label, thus creating counterfactual pairs (see examples in Fig. 1). Existing works simply use these as additional training point. Our contribution is to use the relation between these pairs, which is usually discarded. In other datasets, counterfactual examples can be created by masking parts of the input, thus creative negative examples. In Section 4, we apply this approach to the COCO and VQA v2 datasets.

2 Gradient supervision

To exploit relations between counterfactual examples, we introduce an auxiliary loss that supervises the gradient of the network $f_{\boldsymbol{\theta}}$. We denote the gradient of the network with respect to its input at a point ${\boldsymbol{x}}_{i}$ with ${{\boldsymbol{g}}}_{i}={\nabla_{\boldsymbol{x}}f}({\boldsymbol{x}}_{i})$. Our new gradient supervision (GS) loss encourages ${{\boldsymbol{g}}}_{i}$ to align with a “ground truth” gradient vector $\hat{\boldsymbol{g}}_{i}$:

This definition is a cosine distance between ${{\boldsymbol{g}}}_{i}$ and $\hat{\boldsymbol{g}}_{i}$. Assuming $\{({\boldsymbol{x}}_{i},{y}_{i}),({\boldsymbol{x}}_{j},{y}_{j})\}$ is a pair of counterfactual examples, a “ground truth” gradient at ${\boldsymbol{x}}_{i}$ is obtained as $\hat{\boldsymbol{g}}_{i}={\boldsymbol{x}}_{j}$-${\boldsymbol{x}}_{i}$. This represents the translation in the input space that should change the network output from ${y}_{i}$ to ${y}_{j}$. Minimizing Eq. 1 encourages the network’s gradient to align with this vector at the training points. Assuming $f$ is continuously differentiable, it also constrains the shape of $f$ between training points. This makes $f$ more faithful to the generating process behind the training data (see Section 3.3). Also note that the GS loss uses a local linearization of the network. Although deep networks are highly non-linear globally, first-order approximations have found multiple uses, for example in providing explanations and generating adversarial examples . In our application, this approximation is reasonable since pairs of counterfactual examples lie close to one another and to the classification boundary, by definition.

In cases where the network output ${\boldsymbol{y}}$ is a vector, a ground truth gradient is only available for classes for which we have positive examples. Denoting such a class gt, we apply the GS loss only on the gradient of this class, using ${{\boldsymbol{g}}}_{i}=\nabla_{\boldsymbol{x}}f_{i}({\boldsymbol{x}}_{i})$. If a softmax is used, the output for one class depends on that of the others, so the derivative of the network is taken on its logits to make it dependent on one class only.

3 How gradient supervision improves generalization

By training a machine learning model $f_{\theta}$, we seek to approximate an ideal $\mathcal{F}$ that represents the real-world process attributing the correct label $y=\mathcal{F}({\boldsymbol{x}})$ to any possible input ${\boldsymbol{x}}$. Let us considering the Taylor expansion of $f$ at a training point ${\boldsymbol{x}}_{j}$:

Our definition of a pair of counterfactual examples (${\boldsymbol{x}}_{i}$, ${\boldsymbol{x}}_{j}$) (Section 3.1) implies that $({\boldsymbol{x}}_{i}$-${\boldsymbol{x}}_{j})^{m}$ approaches 0 ($m>1$). For such a pair of nearby points, the terms beyond the first order virtually vanish. It follows that the distance between $f({\boldsymbol{x}}_{j})$ and $f({\boldsymbol{x}}_{i})$ is maximized when the dot product ${\nabla_{\boldsymbol{x}}f}({\boldsymbol{x}}_{i})\,.\,({\boldsymbol{x}}_{i}$-${\boldsymbol{x}}_{j})$ is maximum. This is precisely the desired behavior of $f$ in the vicinity of ${\boldsymbol{x}}_{i}$ and ${\boldsymbol{x}}_{j}$, since their ground truth labels $y_{i}$ and $y_{j}$ are different by our definition of counterfactuals. This leads to the definition of the GS loss in Eq. 1. Geometrically, it encourages the gradient of $f$ to align with the vector pointing from a point to its counterfactual, as illustrated in Fig. 3.

The conventional empirical risk minimization with non-convex functions leads to large numbers of local minimas. They correspond to multiple plausible decision boundaries with varied capability for generalization. Our approach essentially modifies the optimization landscape for the parameters $\theta$ of $f$ such that the minimizer found after training is more likely to reflect the ideal $\mathcal{F}$.

Applications

The proposed method is applicable to datasets with counterfactual examples in the training data. They are sometimes provided explicitly . Most interestingly, we show that they can be also be generated from existing annotations .

We selected four classical tasks in vision and language that are notoriously subject to poor generalization due to dataset biases. Our experiments aim (1) to measure the impact of gradient supervision on performance for well-known tasks, and (2) to demonstrate that the necessary annotations are available in a variety of existing datasets. We therefore prioritized the breadth of experiments and the use of simple models (details in supp. mat.) rather than chasing the state of the art on any particular task. The method should readily apply to more complex models for any of these tasks.

The task of visual question answering (VQA) involves an image and a related question, to which the model must determine the correct answer among a set of approximately 2,000 candidate answers. Models trained on existing datasets (e.g. VQA v2 ) are notoriously poor at generalization because of dataset biases. These models rely on spurious correlations between the correct answer and certain words in the question. We use the training/test splits of VQA-CP that were manually organized such that the correlation between the questions’ prefixes (first few words) and answers differ at training/test time. Most methods evaluated on VQA-CP use the explicit knowledge of this fact or even of the ground truth set of prefixes, which defeats the purpose of evaluating generalization. As discussed in the introduction, strong background assumptions are one of the two options to improve generalization beyond a set of labels. Our method, however, follows the other option of using a different type of data, and does not rest on the knowledge of the construction of VQA-CP.

We build counterfactual examples for VQA-CP using annotations of human attention from . Given a question/image/answer triple $({\boldsymbol{q}},{\boldsymbol{I}},\boldsymbol{a})$, we build its counterfactual counterpart $({\boldsymbol{q}},{\boldsymbol{I}}^{\prime},\boldsymbol{a}^{\prime})$ by editing the image and answer. The image ${\boldsymbol{I}}$ is a set of features pre-extracted with a bottom-up attention model (typically a matrix of dimensions $N\times 2048$). We build ${\boldsymbol{I}}^{\prime}$ ($N^{\prime}\times 2048$, $N^{\prime}\leq N$) by masking the features whose bounding boxes overlap with the human attention map past a certain threshold (details in supp. mat.). The vector $\boldsymbol{a}$ is a binary vector of correct answers over all candidates. We simply set all entries in $\boldsymbol{a}^{\prime}$ to zero.

Experimental setting.

For training, we use the training split of VQA-CP, minus 8,000 questions held out as an “in-domain” validation set (as in ). We generate counterfactual versions of the training examples that have a human attention map (approx. 7% of them). For evaluation, we use (1) our “in-domain” validation set (held out from the training set), (2) the official VQA-CP test set (which has a different correlation between prefixes and answers), and (3) a new focused test set.

The focused test set contains the questions from VQA-CP test from which we only keep image features of regions looked at by humans to answer the questions. We essentially perform the opposite of the building of counterfactual examples, and mask regions where the human attention is below a low threshold. Answering questions from the focused test set should intuitively be easier, since the background and distracting image regions have been removed. However, a model that relies on context (question or irrelevant image regions) rather than strictly on the relevant visual evidence will do poorly on the focused test set. This serves to measure robustness beyond the question biases that VQA-CP was specifically designed for.

Results

We present results of our method applied on top of two existing models. The first (weak baseline) is the popular BUTD model . The second (strong baseline) is the “unshuffling” method of , which was specifically tuned to address the language biases evaluated with VQA-CP. We compare the baseline model with the same model trained with the additional counterfactual data, and then with the additional GS loss. The performance improves on most question types with each of these additions. The “focused” provides an out-of-distribution evaluation complementary to the VQA-CP test set (which only accounts for language biases). It shows the improvements expected from our method to a larger extent that the VQA-CP test set. This suggests that evaluating generalization in VQA is still not completely addressed with the current benchmarks. Importantly, the improvements over both the weak and strong baselines indicate that the proposed method is not redundant with existing methods that specifically address the language biases measured by VQA-CP, like the strong baseline. Additional details are provided in the supplementary material.

2 Multi-label image classification

We apply our method to the COCO dataset . Its images feature objects from 80 classes. They appear in common situations such that the patterns of co-occurrence are highly predictable: a bicycle often appears together with a person, and a traffic light often appears with cars, for example. These images serve as the basis of a number of benchmarks for image detection , captioning , visual question answering , etc. They all inherit the biases inherent to the COCO images which is an increasing cause of concern. A method to improve generalization in this context has a wide potential impact.

We consider a simple multi-label classification task that captures the core issue of dataset biases that affect higher-level tasks (captioning for example ). Each image is associated with a binary vector of size 80 that represents the presence of at least one object of the corresponding class in the image. The task is to predict this binary vector. Performance is measured with the mean average precision (mAP) over all classes. The model is a feed-forward neural network that performs an 80-class binary classification with sigmoid outputs, over pre-extracted ResNet-based visual features. We pre-extract these features with the bottom-up attention model of Anderson et al. . They are spatially pooled into a 2048-dimensional vector. The model is trained with a standard binary cross-entropy loss (details in the supplementary material).

Generating counterfactual examples.

Counterfactual examples can be generated using existing annotation in COCO. Agarwal et al. used the inpainter GAN to edit images by masking selected objects. This only requires the original labels and bounding boxes. The edited images represent a “minimal change” that makes the corresponding label negative, which agrees with our definition of counterfactuals. The vector of ground truth labels for edited images are edited accordingly. For training, we use all images produced by from the COCO train2014 split (original and edited versions). For evaluation, we use their images from the val2014 split (original and edited version, evaluated separately). We also create an additional evaluation split named “Hard edited images”. It contains a subset of edited images with patterns of classes that never appear in the training set.

Results.

We first compare the baseline model trained with the original images only, and then with the original and edited images (Table 2). The performance improves (71.8$\rightarrow$72.1%), which is particularly clear when evaluated on edited images (58.1$\rightarrow$64.0%). This is because the patterns of co-occurrence in the training data cannot blindly relied on with the edited images. The images in this set depict situations that are unusual in the training set, such as a surfboard without a person on top, or a man on a tennis court who is not holding a racquet. A model that relies on common co-occurrences in the training set rather than strictly on visual evidence can do well on the original images, but not on edited ones. An improvement from additional data is not surprising. It is still worth emphasizing that the edited images were generated “for free” using existing annotations in COCO.

Training the model with the proposed gradient supervision (GS) further improves the precision (72.1$\rightarrow$72.9%). This is again more significant on the edited images (64.0$\rightarrow$65.2%). The improvement is highest on the set of “hard edited images” (56.0$\rightarrow$57.7%). As an ablation, we train the GS model with random pairwise relations instead of relations between counterfactual pairs. The performance is clearly worse, showing that the value of GS is in leveraging an additional training signal, rather than setting arbitrary constraints on the gradient like existing unsupervised regularizers . In Fig. 4, we provide qualitative examples from the evaluation sets where the predictions of our model improve over the baseline.

3 NLP Tasks: sentiment analysis and natural language inference

The task of sentiment analysis is to assign a positive or negative label to a text snippet, such as a movie or restaurant review. For training, we use the extension of the IMDb dataset of movie reviews by Kaushik et al. . They collected counterfactual examples by instructing crowdworkers to edit sentences from the original dataset to flip their label. They showed that a standard model trained on the original data performs poorly when evaluated on edited data, indicating that it relies heavily on dataset biases (e.g. the movie genre being predictive of the label). They then used edited data during training (simply mixing it with the original data) and showed much better performance in all evaluation settings, even when controlling for the amount of additional training examples. Our contribution is to use GS to leverage the relations between the pairs of original/edited examples.

The task of natural language inference (NLI) is to classify a pair of sentences, named the premise and the hypothesis, into $\{$entailment, contradiction, neutral$\}$ according to their logical relationship. We use the extension of the SNLI dataset by Kaushik et al. . They instructed crowdworkers to edit original examples to change their labels. Each original example is supplemented with versions produced by editing the premise or the hypothesis, to either of the other two classes. The original and edited data together are therefore four times as large as the original data alone.

We first compare a model trained with the original data, and with the original and edited data as simple augmentation (Table 3). The improvement is significant when tested on edited data (55.3$\rightarrow$88.7%). We then train the model with our GS loss. The added improvement is visible on both the original data (82.0$\rightarrow$83.8%) and on the edited data (88.7$\rightarrow$91.2%). The evaluation on edited examples is the more challenging setting, because spurious correlations from the original training data cannot be relied on. The ablation that uses GS with random relations completely fails, confirming the value of the supervision with relations between pairs of related examples.

We additionally evaluate the model on out-of-sample data with three additional test sets: Amazon Reviews , Semeval 2017 (Twitter data) , and Yelp reviews . The model trained on IMDb is applied without any fine-tuning to these, which constitutes a significant challenge in terms of generalization. We observe a clear gain over the data augmentation baseline on all three.

Results on NLI.

We perform the same set of experiments on NLI. The fairest point of comparison is again the model trained with the original and edited data. Using the GS loss on top of it brings again a clear improvement (Table 3), both when evaluated on standard test data and on edited examples. As an additional measure of generalization, we also evaluate the same models on the dev. set of MultiNLI without any fine-tuning. There is a significant domain shift between the datasets. Using the edited examples for data augmentation actually hurts the performance here, most likely because they constitute very “unnatural” sentences, such that easy-to-pick-up language cues cannot be relied on. Using GS (with always uses the edited data as augmentations as well) brings back the performance higher, and above the baseline trained only on the original data.

Limitations.

Our NLP experiments were conducted with simple models and relatively little data. The current state of the art in sentiment analysis and NLI is achieved by transformer-based models trained on vastly more data. Kaushik et al. showed that counterfactual examples are much more valuable than the same amount of standard data, including for fine-tuning a BERT model for NLI. The application of our technique to the extremely-large data regime, including with large-scale language models, is an exciting direction for future work.

Conclusions

We proposed a new training objective that improves the generalization capabilities of neural networks by supervising their gradient, and using an unused training signal found in many datasets. While most machine learning models rely on identifying correlations between inputs and output, we showed that relations between counterfactual examples provide a fundamentally different, complementary type of information. We showed theoretically and empirically that our technique can shape the decision boundary of the model to be more faithful to the causal mechanisms that generated the data. Practically speaking, the model is then more likely to be “right for the right reasons”. We showed that this effect brings significant improvements on a number of tasks when evaluated with out-of-distribution test data. We demonstrated that the required annotations can be extracted from existing datasets for a number of tasks.

There is a number of additional tasks and datasets on which our method can readily apply . Scaling up the technique to state-of-the-art models in vision and NLP is another exciting direction for future work.

Supplementary material

Appendix 0.A Application to VQA

We generate the counterfactual examples by masking image features on-the-fly, during training, according the the human attention maps of . We use image features from , which correspond to bounding boxes in the image. We mask the features whose boxes overlap with a fraction of the the human attention map above a fixed threshold. We use the precomputed overlap score from , which is a scalar in $$, and set the threshold at 0.2 (setting it at 0 would mask the occasional boxes that encompass nearly the whole image, which is not desirable). This value was set manually by verifying for the intended effect on a few training examples (that is, masking most of the relevant visual evidence). See Fig. 6 for examples of original questions and their counterfactual versions.

Experimental setting.

Our experiments use a validation set (8,000 questions chosen at random) held out from the original VQA-CP training set. Note that most existing methods evaluated in VQA-CP use the extremely unsanitary practice of using the VQA-CP test split for model selection. This is extremely concerning since the whole purpose of VQA-CP is to evaluate generalization to an out-of-distribution test set. The variance in evaluating the ‘number’ and ‘yes/no’ questions is moreover extremely high, because the number of reasonable answers on each of these types is very limited. For example, a model that answers yes or no at random, or produces constantly either answer, can fare extremely well (upwards of 62% accuracy) on these questions. This can very well result from a buggy implementation or a “lucky” random seed, identified by model selection on the test set (!). This is the reason why we include an evaluation on the ‘other’ type of questions in isolation. All of these issues have been pointed out by a few authors .

Our focused test set is a subset of the official VQA-CP test set. It is created in a similar manner as the counterfactual examples. We mask features that overlap with human attention maps below (instead of above) a threshold of 0.8. This value was set manually by verifying for the intended effect on a few examples (masking the background but not the regions necessary to answer the question). The focused test set is much smaller than the official test set since it only comprises questions for which a human attention map is available.

Models.

Our baseline model follows the general description of Teney et al. . We use the features of size 36$\times$2048 provided by Anderson et al. . Our ‘strong baseline’ uses the additional procedure described in on top of this baseline, using the code provided by the authors.

Existing methods.

The method presented in could have constituted an ideal point of comparison with ours, as it was evaluated on VQA-CP and used human attention maps. However, after extensive discussions with the authors, we still have not been able to replicate any of the performance claimed in the paper. We found a number of errors in the paper, as well as inconsistencies in the reported results, and an extreme sensitivity to a single hyperparameter (their reported results were obtained with a single run on a single random seed). We chose not to mention this work in our main paper until these issues have been resolved.

Why not use the same technique for the VQA and COCO experiments ? Inpainting in pixel space vs masking image features.

The two approaches are applicable in both cases. The only reason was to showcase the use of multiple techniques to generate counterfactual examples. The human attention map are specific to VQA and not applicable to the COCO experiments.

Appendix 0.B Application to image classification with COCO

We use the edited images released by together with the corresponding original images from COCO. The edited images were created with the inpainter GAN to mask ground truth bounding boxes of specific objects. The images come from the COCO splits train2014 and val2014. We keep this separation for our experiments as follows. Images from train2014 (323,116 counting original and edited ones) are used for training, except a random subset (1,000 images) that we hold out for validation (model selection, early stopping). Images from val2014 (3,361 original and 3,361 edited) are used exclusively for testing.

We identified a subset (named Hard edited) of the edited images from val2014 whose ground truth vector (which indicated the classes appearing in the image) is never seen during training (614 images).

The set of edited images provided by is a non-standard subset of COCO, so no directly-comparable results have been published for the multi-label classification task that we consider.

Model.

We pre-extract image features from all images with the ResNet-based, bottom-up attention model . These features are averaged across spatial locations, giving a single vector of dimensions 2048 to represent each image. Our model is a 3-layer ReLU MLP of size 64, followed by a linear/sigmoid output layer of size 80 (corresponding to the 80 COCO classes). This baseline model was first tuned for best performance on the validation set (tuning the number of a layers and their size, the batch size, and learning rate), before adding the proposed GS loss. The model is optimized with AdaDelta, mini-batches of size 512, and a binary cross-entropy loss.

Performance is measured with a standard mean average precision (mAP) (as defined in the Pascal VOC challenge) over all 80 classes.

The Fig. 4 in the paper shows the input image with the scores of the top-$k$ predicted labels by the baseline and by our method. The $k$ corresponds to the number of ground truth labels of each image.

Random baseline.

In our ablations, this model is identical to the standard baseline, but it is trained with a randomly shuffled training set. We shuffle the inputs $\{{\boldsymbol{x}}_{i}\}_{i}$ and the ground truth labels $\{y_{i}\}_{i}$ of all training examples. The model is thus not getting any relevant training signal from any example. It can only leverage static dataset biases (i.e. a class imbalance).

Appendix 0.C Application to NLP tasks

We use the subset of the IMDb dataset for which Kaushik et al. obtained counterfactual examples. We use their ‘paired’ version of the data, which only contains original examples that do have an edited version. For training, we use the ‘train’ split of original and edited data (3414 examples). For validation (model selection, early stopping), we use the ‘dev’ set of paired examples. For testing, we use the ‘test’ split, reporting accuracy over the original and edited examples separately. For testing on other datasets, we use a random subset (2000 examples) of the test sets of Amazon Reviews , Semeval 2017 (Twitter data) , and Yelp reviews similarly to .

Sentiment analysis model.

We first optimized a simple baseline model on the validation set (tuning the number of a layers, embedding sizes, batch size, and learning rate). We then added the proposed gradient supervision, tuned its hyperparameters on the validation set (regularizer weight) then reported the performance on the test sets at the epoch of best performance on the validation set. The sentences are tokenized and trimmed to a maximum of 32 tokens. The model encodes a sentence as a bag of words, using word embeddings of size 50, averaged to the exact length of each sentence (i.e. not including the padding of the shorter sentences). The vocabulary is limited to the 20,000 most frequent words in the dataset. The averaged vector is passed to a simple linear classifier with a sigmoid output. All weights, including word embeddings, are initialized from random values, and optimized with AdaDelta, in mini-batches of size 32, with a binary cross-entropy loss. The best weight for the GS regularizer was found to be $\lambda$=20. To reduce the noise in the evaluation due to the small size of the training set, we use an ensemble of 6 identical models trained in parallel. The reported results uses the output of the ensemble, that is the average of the logits of the 6 models.

NLI data.

The experiments on NLI follow a similar procedure to those on sentiment analysis. We use the subset of the SNLI dataset for which Kaushik et al. collected counterfactual examples. We use their biggest version of the data, named ‘all combined’, that contains counterfactual examples with edited premises and edited hypotheses. For testing, we evaluate accuracy separately on original and edited examples (edited premises and edited hypotheses combined). For testing transfer, we use the ‘dev’ set of MultiNLI . Whereas the SNLI dataset contains sentence pairs derived from image captions, MultiNLI is more diverse. It contains sentences from transcribed speech, popular fiction, and government reports. Compared to SNLI, it contains more linguistic diversity and complexity.

NLI model.

The premise and hypothesis sentences are tokenized and trimmed to a maximum of 32 tokens. They are encoded separately as bags of words, using frozen Glove embeddings (dimension 300), then a learned linear/ReLU projection to dimension 50, and an average to the length of each sentence (without using the padding). They are then passed through a batch normalization layer, then concatenated, giving a vector of size 100. The vector is passed through 3 linear/ReLU layers, then a final linear/sigmoid output layer. The model is trained with AdaDelta, with mini-batches of size 512, and a binary cross-entropy loss. The best weight for the GS regularizer was found to be $\lambda$=0.01. Similarly to our experiments on sentiment analysis, we evaluate an ensemble of 6 copies of the model described above.

References