Adversarial Filters of Dataset Biases

Introduction

Large-scale neural networks have achieved superhuman performance across many popular AI benchmarks, for tasks as diverse as image recognition (ImageNet; Russakovsky et al., 2015), natural language inference (SNLI; Bowman et al., 2015), and question answering (SQuAD; Rajpurkar et al., 2016). However, the performance of such neural models degrades considerably when tested on out-of-distribution or adversarial samples, otherwise known as data “in the wild” (Eykholt et al., 2018; Jia & Liang, 2017). This phenomenon indicates that high performance of the strongest AI models is often confined to specific datasets, implicitly making a closed-world assumption. In contrast, true learning of a task necessitates generalization, or an open-world assumption. A major impediment to generalization is the presence of spurious biases – unintended correlations between input and output – in existing datasets (Torralba & Efros, 2011). Such biases or artifactsWe will henceforth use biases and artifacts interchangeably. are often introduced during data collection (Fouhey et al., 2018) or during human annotation (Rudinger et al., 2017; Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Geva et al., 2019). Not only do dataset biases inevitably bias the models trained on them, but they have also been shown to significantly inflate model performance, leading to an overestimation of the true capabilities of current AI systems (Sakaguchi et al., 2020; Hendrycks et al., 2019).

Many recent studies have investigated task or dataset specific biases, including language bias in Visual Question Answering (Goyal et al., 2017), texture bias in ImageNet (Geirhos et al., 2018), and hypothesis-only reliance in Natural Language Inference (Gururangan et al., 2018). These studies have yielded domain-specific algorithms to address the found biases. However, the vast majority of these studies follow a top-down framework where the bias reduction algorithms are essentially guided by researchers’ intuitions and domain insights on particular types of spurious biases. While promising, such approaches are fundamentally limited by what the algorithm designers can manually recognize and enumerate as unwanted biases.

Our work investigates AFLite, an alternative bottom-up approach to algorithmic bias reduction. AFLiteStands for Lightweight Adversarial Filtering. was recently proposed by Sakaguchi et al. (2020)—albeit very succinctly—to systematically discover and filter any dataset artifact in crowdsourced commonsense problems. AFLite employs a model-based approach with the goal of removing spurious artifacts in data beyond what humans can intuitively recognize, but those which are exploited by powerful models. Figure 1 illustrates how AFLite reduces dataset biases in the ImageNet dataset for object classification.

This paper presents the first theoretical understanding and comprehensive empirical investigations into AFLite. More concretely, we make the following four novel contributions.

First, we situate AFLite in a theoretical framework for optimal bias reduction, and demonstrate that AFLite provides a practical approximation of AFOpt, the ideal but computationally intractable bias reduction method under this framework (§2).

Second, we present an extensive suite of experiments that were lacking in the work of Sakaguchi et al. (2020), to validate whether AFLite truly removes spurious biases in data as originally assumed. Our baselines and thorough analyses use both synthetic (thus easier to control) datasets (§3) as well as real datasets. The latter span benchmarks across NLP (§4) and vision (§5) tasks: the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) datasets for natural language inference, QNLI (Wang et al., 2018a) for question answering, and the ImageNet dataset (Russakovsky et al., 2015) for object recognition.

Third, we demonstrate that models trained on AFLite-filtered data generalize substantially better to out-of-domain samples, compared to models that are trained on the original biased datasets (§4, §5). These findings indicate that spurious biases in datasets make benchmarks artificially easier, as models learn to overly rely on these biases instead of learning more transferable features, thereby hurting out-of-domain generalization.

Finally, we show that AFLite-filtering makes widely used AI benchmarks considerably more challenging. We consistently observe a significant drop in the in-domain performance even for state-of-the-art models on all benchmarks, even though human performance still remains high; this suggests that currently reported performance on benchmarks might be inflated. For instance, the best model on SNLI-AFLite achieves only 63% accuracy, a 30% drop compared to its accuracy on the original SNLI. These findings are especially surprising since AFLite maintains an identical train-test distribution, while retaining a sizable training set.

In summary, AFLite-filtered datasets can serve as upgraded benchmarks, posing new research challenges for robust generalization.

AFLite

Large datasets run the risk of prioritizing performance on the data-rich head of the distribution, where examples are plentiful, and discounting the tail. AFLite seeks to minimize the ability of a model to exploit biases in the head of the distribution, while preserving the inherent complexity of the tail. In this section, we provide a formal framework for studying such bias reduction techniques, revealing that AFLite can be viewed as a practical approximation of a desirable but computationally intractable optimum bias reduction objective.

Let $\Phi$ be any feature representation defined over a dataset $\mathcal{D}=(X,Y)$. AFLite seeks a subset $S\subset\mathcal{D},|S|\geq n$ that is maximally resilient to the features uncovered by $\Phi$. In other words, for any identically-distributed train-test split of $S$, learning how to best exploit the features $\Phi$ on the training instances should not help models generalize to the held-out test set.

Let $\mathcal{M}$ denote a family of classification models (e.g., logistic regression, support vector machine classifier, or a particular neural architecture) that can be trained on subsets $S$ of $\mathcal{D}=(X,Y)$ using features $\Phi(X)$. We define the representation bias of $\Phi$ in $S$ w.r.t $\mathcal{M}$, denoted $\mathcal{R}(\Phi,S,\mathcal{M})$, as the best possible out-of-sample classification accuracy achievable by models in $\mathcal{M}$ when predicting labels $Y$ using features $\Phi(X)$. Given a target minimum reduced dataset size $n$, the goal is to find a subset $S\subset\mathcal{D}$ of size at least $n$ that minimizes this representation bias in $S$ w.r.t. $\mathcal{M}$:

Eq. (1) corresponds to optimum bias reduction, referred to as AFOpt. We formulate $\mathcal{R}(\Phi,S,\mathcal{M})$ as the expected classification accuracy resulting from the following process. Let $q:2^{S}\to$ be a probability distribution over subsets $T=(X^{T},Y^{T})$ of $S$. The process is to randomly choose $T$ with probability $q(T)$, train a classifier $M_{T}\in\mathcal{M}$ on $S\setminus T$, and evaluate its classification accuracy f_{M_{T}}\big{(}\Phi(X^{T}),Y^{T}\big{)} on $T$. The resulting accuracy on $T$ itself is a random variable, since the training set $S\setminus T$ is randomly sampled. We define the expected value of this classification accuracy to be the representation bias:

The expectation in Eq. (2), however, involves a summation over exponentially many choices of $T$ even to compute the representation bias for a single $S$. This makes optimizing Eq. (1), which involves a search over $S$, highly intractable. To circumvent this challenge, we refactor $\mathcal{R}(\Phi,S,\mathcal{M})$ as a sum over instances $i\in S$ of the aggregate contribution of $i$ to the representation bias across all $T$. Importantly, this summation has only $|S|$ terms, allowing more efficient computation. We call this the predictability score $p(i)$ for $i$: on average, how reliably can label $y_{i}$ be predicted using features $\Phi(x_{i})$ when a model from $\mathcal{M}$ is trained on a randomly chosen training set $S\setminus T$ not containing $i$. Instances with high predictability scores are undesirable as their feature representation can be exploited to confidently correctly predict such instances.

With some abuse of notation, for each $i\in S$, we denote $q(i)\triangleq\sum_{T\ni i}q(T)$ the marginal probability of choosing a subset $T$ that contains $i$. The ratio $\frac{q(T)}{q(i)}$ is then the probability of $T$ conditioned on it containing $i$. Let f_{M_{T}}\big{(}\Phi(x_{i}),y_{i}\big{)} be the classification accuracy of $M_{T}$ on $i$. Then the expectation in Eq. (2) can be written in terms of $p(i)$ as follows:

where $p(i)$ is the predictability score of $i$ defined as:

Implementation Algorithm 1 provides an implementation of AFLite. The algorithm takes as input a dataset $D=(X,Y)$, a representation $\Phi(X)$ we are interested in minimizing the bias in, a model family $\mathcal{M}$ (e.g., linear classifiers), a target dataset size $n$, size $m$ of the support of the expectation in Eq. (4), training set size $t$ for the classifiers, size $k$ of each slice, and an early-stopping filtering threshold $\tau$. Importantly, for efficiency, $\Phi(X)$ is provided to AFLite in the form of pre-computed embeddings for all of $X$. In practice, to obtain $\Phi(X)$, we train a first “warm-up” model on a small fraction of the data based on the learning curve in low-data regime, and do not reuse this data for the rest of our experiments. Moreover, this fraction corresponds to the training size $t$ for AFLite and it remains unchanged across iterations. We follow the iterative filtering approach, starting with $S=D$ and iteratively removing some instances with the highest predictability scores using the greedy slicing strategy. Slice size $k$ and number of partitions $m$ are determined by the available computation budget.

At each filtering phase, we train models (linear classifiers) on $m$ different random partitions of the data, and collect their predictions on their corresponding test set. For each instance $i$, we compute its predictability score as the ratio of the number of times its label $y_{i}$ is predicted correctly, over the total number of predictions for it. We rank the instances according to their predictability score and use the greedy slicing strategy of removing the top-$k$ instances whose score is not less than the early-stopping threshold $\tau$. We repeat this process until fewer than $k$ instances pass the $\tau$ threshold in a filtering phase or fewer than $n$ instances remain. Appendix §A.5 provides details of hyperparameters used across different experimental settings, to be discussed in the following sections.

Synthetic Data Experiments

We present experiments under a synthetic setting, to evaluate whether AFLite successfully removes examples with spurious correlations from a dataset. We synthesize a dataset comprising two-dimensional data, arranged in concentric circles, at four different levels of separation, as shown in Figure 2. The label (color) indicates the circular region the data point is situated in. As is evident, a linear function is inadequate for separating the two classes; it requires a more complex non-linear model such as a support vector machine (SVM) with a radial basis function (RBF) kernel.

To simulate spurious correlations in the data, we add class-specific artificially constructed features (biases) sampled from two different Gaussian distributions. These features are only added to $75\%$ of the data in each class, while for the rest of the data, we insert random (noise) features. The bias features make the task solvable through a linear function. Furthermore, for the first dataset, with the largest separation, we flipped the labels of some biased samples, making the data slightly adversarial even to the RBF. Both models can clearly leverage the biases, and demonstrate improved performance over a baseline without biases.We use standard implementations from scikit-learn: https://scikit-learn.org/stable/.

Once we apply AFLite, as expected, the number of biased samples is reduced considerably, making the task hard once again for the linear model, but still solvable for the non-linear one. The filtered dataset is shown in the bottom half of Fig. 2, and the captions indicate the performance of a linear and an SVM model (detailed results for each are provided in Appendix §A.3, for better visibility). Under each separation level, our results show that AFLite indeed removes examples with spurious correlations from a dataset. Moreover, AFLite removes most of the flipped examples in the first dataset.

NLP Experiments

As our first real-world data evaluation for AFLite, we consider out-of-domain and in-domain generalization for a variety of language datasets. The primary task we consider is natural language inference (NLI) on the Stanford NLI dataset (Bowman et al., 2015, SNLI). Each instance in the NLI task consists of a premise-hypothesis sentence pair, the task involves predicting whether the hypothesis either entails, contradicts or is neutral to the premise.

Experimental Setup We use feature representations from RoBERTa-large, $\phi_{\textit{RoBERTa}}$ (Liu et al., 2019b), a large-scale pretrained masked language model. This is extracted from the final layer before the output layer, trained on a random $10\%$ sample (warm-up) of the original training set. The resultant filtered NLI dataset, $D(\phi_{\textit{RoBERTa}})$, is compared to the original dataset $D$ as well as a randomly subsampled dataset $D_{182k}$, with the same sample size as $D(\phi_{\textit{RoBERTa}})$, amounting to only a third of the full data $D$. The same RoBERTa-large architecture is used to train the three NLI models.

As motivated in Section §1, large-scale architectures often learn to solve datasets rather than the underlying task by overfitting on unintended correlations between input and output in the data. However, this reliance might be hurtful for generalization to out-of-distribution examples, since they may not contain the same biases. We evaluate AFLite for this criterion on the NLI task.

Gururangan et al. (2018), among others, showed the existence of certain annotation artifacts (lexical associations etc.) in SNLI which make the task considerably easier for most current methods. This spurred the development of several out-of-distribution test sets which carefully control for the presence of said artifacts. We evaluate on four such out-of-distribution datasets: HANS (McCoy et al., 2019b), NLI Diagnostics (Wang et al., 2018a), Stress tests (Naik et al., 2018) and Adversarial NLI (Nie et al., 2019) (c.f. Appendix §A.4 for details). Given that these benchmarks are collected independently of the original SNLI task, the biases from SNLI are less likely to carry over.However, these benchmarks might contain their own biases (Liu et al., 2019a).

Table 1 shows results on three out of four diagnostic datasets (HANS, NLI-Diagnostics and Stress), where we perform a zero-shot evaluation of the models. Models trained on SNLI-AFLite consistently exceed or match the performance of the full model on the benchmarks above, up to standard deviation. To control for the size, we compare to a baseline trained on a random subsample of the same size ($D_{182k}$). AFLite models report higher generalization performance suggesting that the filtered samples are more informative than a random subset. In particular, AFLite substantially outperforms challenging examples on the HANS benchmark, which targets models purely relying on lexical and syntactic cues. Table 2 shows results on the Adversarial NLI benchmark, which allows for evaluation of transfer capabilities, by finetuning models on each of the three training datasets (Rd1, Rd2 and Rd3). A RoBERTa-large model trained on SNLI-AFLite surpasses the performance in all three settings.

2 In-distribution Benchmark Re-estimation

AFLite additionally provides a more accurate estimation of the benchmark performance on several tasks. Here we simply lower the AFLite early-stopping threshold, $\tau$ in order to filter most biased examples from the data, resulting in a stricter benchmark with 92k train samples.

In addition to RoBERTa-large, we consider here pre-computed embeddings from BERT-large (Devlin et al., 2019), and GloVe (Pennington et al., 2014), resulting in three different feature representations for SNLI: $\phi_{\textit{BERT}}$, $\phi_{\textit{RoBERTa}}$ from RoBERTa-large (Liu et al., 2019b), and $\Phi_{\textit{ESIM+GLoVe}}$ which uses the ESIM model (Chen et al., 2016) with GloVe embeddings. Table 3 shows the results for SNLI. In all cases, applying AFLite substantially reduces overall model accuracy, with typical drops of 15-35% depending on the models used for learning the feature representations and those used for evaluation of the filtered dataset. In general, performance is lowest when using the strongest model (RoBERTa) for learning feature representations. Results also highlight the ability of weaker adversaries to produce datasets that are still challenging for much stronger models with a drop of 13.7% for RoBERTa using $\Phi_{\textit{ESIM+GLoVe}}$ as feature representation.

To control for the reduction in dataset size by filtering, we randomly subsample $D$, creating $D_{92k}$ whose size is approximately equal to that of $D(\phi_{\textit{RoBERTa}})$. All models achieve nearly the same performance as their performance on the full dataset – even when trained on just one-fifth the original data. This result further highlights that current benchmark datasets contain significant redundancy within its instances.

We also include two other baselines, which target known dataset artifacts in NLI. The first baseline uses Point-wise Mutual Information (PMI) between words in a given instance and the target label as its only feature. Hence it captures the extent to which datasets exhibit word-association biases, one particular class of spurious correlations. While this baseline is relatively weaker than other models, its performance still reduces by nearly 13% on the $D(\phi_{\textit{RoBERTa}})$ dataset. The second baseline trains only the hypothesis of an NLI instance (-HypOnly). Such partial input baselines (Gururangan et al., 2018) capture reliance on lexical cues only in the hypothesis, instead of learning a semantic relationship between the hypothesis and premise. This reduces performance by almost 24% before and after filtering with RoBERTa. AFLite, which is agnostic to any particular known bias in the data, results in a drop of about 30% on the same dataset, indicating that it might be capturing a larger class of spurious biases than either of the above baselines.

Finally, to demonstrate the value of the iterative, ensemble-based AFLite algorithm, we compare with a baseline where using a single model, we filter out the most predictable examples in a single iteration — a non-iterative, single-model version of AFLite. A RoBERTa-large model trained on this subset (of the same size as $D(\phi_{\textit{RoBERTa}})$) achieves a dev accuracy of $72.1\%$. Compared to the performance of RoBERTa on $D(\phi_{\textit{RoBERTa}})$ ($62.6\%$, see Table 3), it makes this baseline a sensible yet less effective approach. In particular, this illustrates the need for an iterative procedure involving models trained on multiple partitions of the remaining data in each iteration.

MultiNLI and QNLI

We evaluate the performance of another large-scale NLI dataset multi-genre NLI (Williams et al., 2018, MultiNLI), and the QNLI dataset (Wang et al., 2018a) which is a sentence-pair classification version of the SQuAD (Rajpurkar et al., 2016) question answering task.QNLI is stylized as an NLI classification task, where the task is to determine whether or not a sentence contains the answer to a question. Results before and after AFLite are reported in Table 4. Since RoBERTa resulted in the largest drops in performance across the board in SNLI, we only experiment with RoBERTa as adversary for MultiNLI and QNLI. While RoBERTa achieves over $90\%$ on both original datasets, its performance drops to $66.2\%$ for MultiNLI and to $77.7\%$ for QNLI on the filtered datasets. Similarly, partial input baseline performance also decreases substantially on both dataset compared to their performance on the original dataset. Overall, our experiments indicate that AFLite consistently results in reduced accuracy on the filtered datasets across multiple language benchmark datasets, even after controlling for the size of the training set.

Table 3 shows that human performance on SNLI-AFLite is lower than that on the full SNLI.Measured based on five annotator labels provided in the original SNLI validation data. This indicates that the filtered dataset is somewhat harder even for humans, though to a much lesser degree than any model. Indeed, removal of examples with spurious correlations could inadvertently lead to removal of genuinely easy examples; this might be a limitation of a model-based bias reduction approach such as AFLite (see Appendix §A.8 for a qualitative analysis). Future directions for bias reduction techniques might involve additionally accounting for unaltered human performance before and after dataset reduction.

Vision Experiments

We evaluate AFLite on image classification through ImageNet (ILSVRC2012) classification. On ImageNet, we use the state-of-the-art EfficientNet-B7 model (Tan & Le, 2019) as our core feature extractor $\Phi_{\text{EN-B7}}$. The EfficientNet model is learned from scratch on a fixed 20% sample of the ImageNet training set, using RandAugment data augmentation (Cubuk et al., 2019). We then use the 2560-dimensional features extracted by EfficientNet-B7 as the underlying representation for AFLite to use to filter the remaining dataset, and stop when data size is 40% of ImageNet.

In Table 5, we report performance of image classification models on ImageNet-A, a dataset with out-of-distribution images (Hendrycks et al., 2019). As shown, all EfficientNet models struggle on this task, even when trained on the entire ImageNet.Notably, there is a large difference in the degree of out-of-distribution generalization performance for NLP and vision tasks. NLP tasks benefit from the availability of pretrained representations from large language models, such as RoBERTa. In vision, however, while (pre)training on ImageNet alone is often sufficient to learn competitive features, such strong pretrained representations are not available. Moreover, ImageNet has many classes and a skewed distribution of data (Vodrahalli et al., 2018). Hence, it is considerably harder to find a smaller subset of data which generalizes well to adversarial challenge sets, such as ImageNet-A. However, we find that training on AFLite-filtered data leads to models with greater generalization, in comparison to training on a randomly sampled ImageNet of the same size, leading to up to 2% improvement in performance.

In-distribution Image Classification

In Table 6, we present ImageNet accuracy across the EfficientNet and ResNet (He et al., 2016) model families before and after filtering with AFLite. For evaluation, the Imagenet-AFLite filtered validation set is much harder than the standard validation set (also see Figure 1). While the top performer after filtering is still EfficientNet-B7, its top-1 accuracy drops from 84.4% to 63.5%. A model trained on a randomly filtered subsample of the same size though suffers much less, most likely due to reduction in training data.

Overall, these results suggest that image classification – even within a subset of the closed world of ImageNet – is far from solved. These results echo other findings that suggest that common biases that naturally occur in web-scale image data, such as towards canonical poses (Alcorn et al., 2019) or towards texture rather than shape (Geirhos et al., 2018), are problems for ImageNet-trained classifiers.

Related Work

AFLite is related to Zellers et al. (2018)’s adversarial filtering (AF) algorithm, yet distinct in two key ways: it is (i) much more broadly applicable (by not requiring over generation of data instances), and (ii) considerably more lightweight (by not requiring re-training a model at each iteration of AF). Variants of this AF approach have recently been used to create other datasets such as HellaSwag (Zellers et al., 2019) and Abductive NLI (Bhagavatula et al., 2019) by iteratively perturbing dataset instances until a target model cannot fit the resulting dataset. While effective, these approaches run into three main pitfalls. First, dataset curators need to explicitly devise a strategy of collecting or generating perturbations of a given instance. Second, the approach runs the risk of distributional bias where a discriminator can learn to distinguish between machine generated instances and human-generated ones. Finally it requires re-training a model at each iteration, which is computationally expensive especially when using a large model such as BERT as the adversary. In contrast, AFLite focuses on addressing dataset biases from existing datasets instead of adversarially perturbing instances. AFLite was earlier proposed by Sakaguchi et al. (2020) to create the Winogrande dataset. This paper presents more thorough experiments, theoretical justification and results from generalizing the proposed approach to multiple popular NLP and Vision datasets.

Data Selection for Debiased Representations

Li & Vasconcelos (2019) recently proposed REPAIR, a method to remove representation bias by dataset resampling. The motivation in REPAIR is to learn a probability distribution over the dataset that favors instances that are hard for a given representation. In contrast to AFLite, the implementation of REPAIR relies on in-training classification loss as opposed to out-of-sample generalization accuracy. RESOUND (Li et al., 2018) quantifies the representation biases of datasets, and uses them to assemble a new K-class dataset with smaller biases by sampling an existing C-class dataset ($C>K$). Dataset distillation (Wang et al., 2018b) optimizes for a different objective function compared to AFLite: it aims to synthesize a small number of instances to approximate the model trained on the original data. Dasgupta et al. (2018) introduce an NLI dataset that cannot be solved using only word-level knowledge and requires some compositionality. The authors show that debiasing training corpora and augmenting them with minimal contrasting examples makes models more suited to learn the compositional structure of language. Finally, Sagawa et al. (2020) analyze the tension between over-parameterization and using all the data available. It advocates for subsampling the majority groups as opposed to upweighting minority groups in order to achieve low worst-group error. This is in line with the filtering approach that AFLite adapts, as well as the out-of-distribution and robustness results we observe.

Learning Objectives for Debiasing

Another line of related work focuses on removing bias in data representations via the design of learning objectives for debiasing. Arjovsky et al. (2019) propose Invariant Risk Minimization as an objective that promotes learning representations of the data that are stable across environments. Instead of learning optimal classifiers, AFLite aims to remove instances that exhibit artifacts in a dataset. Belinkov et al. (2019) propose an adversarial removal technique that encourages models to learn representations free of hypothesis-only biases. He et al. (2019) propose DRiFt, a debiasing algorithm that first learns a biased model using only known biased features and then trains a debiased model that fits the residuals of the biased model. Similarly, Clark et al. (2019) propose learning a naive classifier using only bias features, to be used in an ensemble along with other classifiers containing more general features. Each of the previous approaches target only known NLI biases, based on prior knowledge; we show AFLite is capable of removing even those examples which exhibit previously unknown spurious biases. Finally, Elazar & Goldberg (2018) show that adversarial training effectively mitigate demographic information leakage, but fail to remove it completely when dealing with text data.

Conclusion

We present a deep-dive into AFLite – an iterative greedy algorithm that adversarially filters out spurious biases from data for accurate benchmark estimation. We provide a theoretical framework supporting AFLite, and show its effectiveness in bias reduction on synthetic and real datasets, providing extensive analyses. We apply AFLite to four datasets, including widely used benchmarks such as SNLI and ImageNet. On out-of-distribution and adversarial test sets designed for such benchmarks, we show that models trained on the AFLite-filtered subsets achieve better performance, indicating higher generalization abilities. Moreover, we show that the strongest performance on the resulting filtered datasets drops significantly (by 30 points for SNLI and 20 points for ImageNet). We hope that dataset creators will employ AFLite to identify unknown dataset artifacts before releasing new challenge datasets for more reliable estimates of task progress on future AI benchmarks. All datasets and code for this work will be made public.

Acknowledgments

We would like to thank Noah A. Smith, Nicholas Lourie, Ana Marasovic̀ and Daniel Khashabi for insightful discussions about this work as well as the anonymous reviewers for their valuable feedback. This research was supported in part by NSF (IIS-1524371), the National Science Foundation Graduate Research Fellowship under Grant No. DGE 1256082, DARPA CwC through ARO (W911NF15-1- 0543), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI. Computations on beaker.org were supported in part by credits from Google Cloud.

References

Appendix A Appendix

All three strategies could be further improved by considering not only the predictability score of the top-$k$ instances but also (via retraining without these instances) how their removal would influence the predictability scores of other instances in the next step. We found our computationally lighter approaches to work well even without the additional overhead of such look-ahead. AFLite implements the greedy slicing approach, and can thus be viewed as a scalable and practical approximation of (intractable) AFOpt for optimum bias reduction. We leave the empirical investigation into other proposed strategies for future work.

A.2 Slice Sampling Details

A.3 Results on Synthetic Data Experiments

As discussed in Section §3, Figure 2 shows the effect of AFLite on four synthetic datasets containing data arranged in concentric circles at four degrees of class separation. For greater visibility, we have provided the accuracies of the SVM with RBF kernel and logistic regression in Table 7.

In summary, a stronger model such as the SVM is more robust to the presence of artifacts than a simple linear classifier. Thus, the implications for real datasets is to move towards models designed for reasoning about a specific task, hence avoiding a dependence on spurious artifacts.

A.4 NLI Out-of-distribution Benchmarks

We describe the four out-of-distribution evaluation benchmarks for NLI from Section §4.1 below:

HANS (McCoy et al., 2019b) contains evaluation examples designed to avoid common structural heuristics (such as word overlap) which could be used by models to correctly predict NLI inputs, without true inferential reasoning.

NLI Diagnostics (Wang et al., 2018a) is a set of hand-crafted examples designed to demonstrate model performance on several fine-grained semantic categories, such as logical reasoning and commonsense knowledge.

Stress tests for NLI (Naik et al., 2018) are a collection of tests targeting the weaknesses of strong NLI models, to check if these are robust to semantics (competence), irrelevance (distraction) and typos (noise).

Adversarial NLI (Nie et al., 2019) consists of premises collected from Wikipedia and other news corpora, and human generated hypotheses, arranged at different tiers of the challenge they present to a model, using a human and model in-the-loop procedure.

Recent work (McCoy et al., 2019a) has observed large variance on out-of-distribution test sets with random seeds. Hence, we report the mean and variance across 5 random seeds in all settings in Table 1. Since Adversarial NLI involves finetuning the model, and not just reporting on a different test set, we skip this step in Table 2.

A.5 Hyperparameters for AFLite

Table 8 shows hyperparameters used to run AFLite to obtain filtered subsets for in-distribution benchmark estimation on different datasets. Target dataset size, $n$ and the early stop filtering threshold $\tau$ are interdependent, as the predictability score threshold determines what examples to keep, which in turn influences the desired size of the dataset, $n$. For ImageNet, we set $n=640K$ and do not control for $\tau$. We use much larger values for $t$ and $k$ for ImageNet than in all NLP experiments, where the use of powerful language representations (such as RoBERTa) allows us to get reasonable performance even with smaller training sets; ImageNet does not offer any such benefits arising from pretrained representations.

For all out-of-distribution NLP experiments, we explicitly control for the size of $n$, as discussed in the corresponding sections in the paper. In these cases, we typically end up using slightly larger $n$, allowing for the final models to get more exposure to task data which is, to a degree, helpful for out-of-distribution generalization. In ImageNet, we use the same hyperparameters in both sets of experiments. In particular, we explicitly set $n=182K$ for SNLI, and $n=640K$ for ImageNet AFLite-filtering for the out-of-distribution generalization experiments.

A.6 Hyperparameters for NLP experiments

For all NLP experiments, our implementation is based on the GLUE (Wang et al., 2018a) experiments in the Transformers repository (Wolf et al., 2019) from Huggingface.https://github.com/huggingface/transformers We used the Adam optimizer (Kingma & Ba, 2014) for every training set up, with a learning rate of 1e-5, and an epsilon value of 1e-8. We trained for 3 epochs for all *NLI tasks, maintaining a batch size of 92. All above hyperparameters were selected using a grid search; we kept other hyperparameters unaltered from the original HuggingFace repository. Each experiment was performed on a single Quadro RTX 8000 GPU.

A.7 Hyperparameters for ImageNet

We trained our ImageNet models using v3-512 TPU pods. For EfficientNet (Tan & Le, 2019), we used RandAugment data augmentation (Cubuk et al., 2019) with 2 layers, and a magnitude of 28, for all model sizes. We trained our models using a batch size of 4096, a learning rate of 0.128, and kept other hyperparameters the same as in (Tan & Le, 2019). We trained for 350 epochs for all dataset sizes - so when training on 20% or 40% of ImageNet (or a smaller dataset), we scaled the number of optimization steps accordingly. For ResNet (He et al., 2016), we used a learning rate of 0.1, a batch size of 8192, and trained for 90 epochs.

A.8 Qualitative Analysis of SNLI

Table 9 shows some examples removed and retained by AFLite on the NLI dataset.