An Investigation of Why Overparameterization Exacerbates Spurious Correlations

Introduction

The typical goal in machine learning is to minimize the average error on a test set that is independent and identically distributed (i.i.d.) to the training set. A large body of prior work has shown that overparameterization—increasing model size beyond the point of zero training error—improves average test error in a variety of settings, both empirically (with neural networks, e.g., Nakkiran et al. 2019) and theoretically (with linear and random projection models, e.g., Belkin et al. 2019; Mei & Montanari 2019).

However, recent work has also demonstrated that models with low average error can still fail on particular groups of data points (Blodgett et al. 2016; Hashimoto et al. 2018; Buolamwini & Gebru 2018). This problem of high worst-group error arises especially in the presence of spurious correlations, such as strong associations between label and background in image classification (McCoy et al. 2019; Sagawa et al. 2020). To mitigate this problem, common approaches reduce the worst-group training loss, e.g., through distributionally robust optimization (DRO) or simply upweighting the minority groups. Sagawa et al. 2020 showed these approaches improve worst-group error on strongly regularized neural networks but fail to help standard neural networks that can achieve zero training error, suggesting that increasing model capacity by reducing regularization—and perhaps by increasing overparameterization as well—can exacerbate spurious correlations.

In this paper, we investigate why overparameterization exacerbates spurious correlations under the above approach of upweighting minority groups. We first confirm on two image datasets (Figure 2) that directly increasing overparameterization (i.e., increasing model size) indeed hurts worst-group error, leading to models that are highly inaccurate on the minority groups where the spurious correlation does not hold (Section 3). In contrast, their underparameterized counterparts obtain much better worst-group error, but do worse on average. We also confirm that models trained via empirical risk minimization (i.e., without upweighting the minority) have poor worst-group test error regardless of whether they are under- or overparameterized. Through simulations on a synthetic setting, we further identify two properties of the training data that modulate the effect of overparameterization: (i) the relative sizes of the majority versus minority groups, and (ii) how informative the spurious features are relative to the core features (Section 4).

Why does overparameterization exacerbate spurious correlations? Underparameterized models do not rely on spurious features because that would incur high training error on the (upweighted) minority groups where the spurious correlation does not hold. In contrast, overparameterized models can always obtain zero training error by memorizing training examples, and instead rely on their inductive bias to pick a solution—which features to use and which examples to memorize—out of all solutions with zero training error. Our results suggest an intuitive story of why overparameterization can hurt: because overparameterized models can have an inductive bias towards “memorizing” fewer examples (Figure 1). If (i) the majority groups are sufficiently large and (ii) the spurious features are more informative than the core features for these groups, then overparameterized models could choose to use the spurious features because it entails less memorization, and therefore suffer high worst-group test error. We test this intuition through simulations and formalize it in a theoretical analysis (Section 5).

Our analysis also leads to the counterintuitive result that on overparameterized models, subsampling the majority groups is much more effective at improving worst-group error than upweighting the minority groups. Indeed, an overparameterized model trained on a subset of $<$5% of the data performs similarly (on average and on the worst group) to an underparameterized model trained on all the data (Section 6). This suggests a possible tension between using overparameterized models and using all the data; average error benefits from both, but improving worst-group error seems to rely on using only one but not both.

Setup

Spurious correlation setup. We adopt the setting studied in Sagawa et al. 2020, where each example comprises the input features $x$, a label (core attribute) $y\in\mathcal{Y}$, and a spurious attribute $a\in\mathcal{A}$. Each example belongs to a group $g\in\mathcal{G}=\mathcal{Y}\times\mathcal{A}$, where $g=(y,a)$. Importantly, the spurious attribute $a$ is correlated with the label $y$ in the training set. We focus on the binary setting in which $\mathcal{Y}=\{1,-1\}$ and $\mathcal{A}=\{1,-1\}$.

Applications. We study two image classification tasks (Figure 2). In the first task, the label is spuriously correlated with demographics: specifically, we use the CelebA dataset (Liu et al. 2015) to classify hair color between the labels $\mathcal{Y}=\{\text{blonde, non-blonde}\}$, which are correlated with the gender $\mathcal{A}=\{\text{female, male}\}$. In the second task, the label is spuriously correlated with image background. We use the Waterbirds dataset (based on datasets from Wah et al. 2011; Zhou et al. 2017 and modified by Sagawa et al. 2020) to classify between the labels $\mathcal{Y}=\{\text{waterbird, landbird}\}$, which are spuriously correlated with the image background $\mathcal{A}=\{\text{water background, land background}\}$. See Appendix A.5 for more dataset details.

Objectives and metrics. We evaluate a model $w$ by its worst-group error,

However, in line with Sagawa et al. 2020, we find that models trained via ERM have poor worst-group test error regardless of whether they are under- or overparameterized (Appendix A.1). To achieve low worst-group test error, prior work proposed modified objectives that focus on the worst-group loss, such as group distributionally robust optimization (group DRO) which directly optimizes for the worst-group training loss (Hu et al. 2018; Sagawa et al. 2020) or reweighting (Shimodaira 2000; Byrd & Lipton 2019). Sagawa et al. 2020 showed that both approaches can help worst-group loss, though group DRO is typically more effective. For simplicity, we focus on the well-studied reweighting approach, which optimizes

where $\hat{p}_{g}$ is the fraction of training examples in group $g$. The intuition behind reweighting is that it makes each group contribute the same weight to the training objective: that is, minority groups are upweighted, while majority groups are downweighted. Note that this approach requires the groups $g$ to be specified at training time, though not at test time.

Overparameterization hurts worst-group error

Sagawa et al. 2020 observed that decreasing $L_{2}$ regularization hurts worst-group error. Though increasing overparameterization and reducing regularization can have different effects (Zhang et al. 2017; Mei & Montanari 2019), this suggests that overparameterization might similarly exacerbate spurious correlations. Here, we show that directly increasing overparameterization (model size) indeed hurts worst-group error even though it improves average error.

Results. Overparameterization improves average test error across both datasets, in line with prior work (Belkin et al. 2019; Nakkiran et al. 2019) (Figure 3). However, in stark contrast, overparameterization hurts worst-group error: the best worst-group test error is achieved by an underparameterized model with non-zero training error. On CelebA, the smallest model (width 1) has 12.4% worst-group training error but comparatively low worst-group test error of 25.6%. As width increases, training error goes to zero but worst-group test error gets worse, reaching $>$60% for overparameterized models with zero training error. Similarly, on Waterbirds, an underparameterized model with $90$ random features and worst-group training error of 17.7% obtains the best worst-group test error of 26.6%, while overparameterized models with zero training error yield worst-group test error of 42.4% at best.

In Appendix A.2, we also confirm that stronger regularization improves worst-group error but hurts average error in overparameterized models, while it has little effect on both worst-group and average error in underparameterized models. However, we focus on understanding the effect of overparameterization in the remainder of the paper.

Discussion. Why does overparameterization hurt worst-group test error? We make two observations. First, in the overparameterized regime, the smallest groups incur the highest test error (blonde males in CelebA and waterbirds on land background in Waterbirds), despite having zero training error. In other words, overparameterized models perfectly fit the minority points at training time, but seem to do so by using patterns that do not generalize. We informally refer to this behavior as “memorizing” the minority points.

Second, underparameterized models do obtain low worst-group error by learning patterns that generalize to both majority and minority groups. Therefore, overparameterized models should also be able to learn these patterns while attaining zero training error (e.g., by memorizing the training points that the underparameterized model cannot fit). Despite this, overparameterized models seem to learn patterns that generalize well on the majority but do not work on the minority (such as the spurious attributes $a$ in Figure 2).

What makes overparameterized models memorize the minority instead of learning patterns that generalize well on both majority and minority groups? We study this question in the next two sections: in Section 4, we use simulations to understand properties of the data distribution that give rise to this trend, and in Section 5 we analyze a simplified linear setting and show how the inductive bias of models towards memorizing fewer points can lead to overparameterized models choosing to use spurious correlations.

Simulation studies

The discussion in Section 3 suggests two properties of the training distribution that modulate the effect of overparameterization on worst-group error. Intuitively, overparameterized models should be more incentivized to use the spurious features and memorize the minority groups if (i) the proportion of the majority group, $p_{\mathsf{maj}}$, is higher, and (ii) the ratio of how informative the spurious features are relative to the core features, $r_{\mathsf{s:c}}$, is higher. In this section, we use simulations to confirm these intuitions and probe how $p_{\mathsf{maj}}$ and $r_{\mathsf{s:c}}$ affect worst-group error in overparameterized models.

Data distribution. We construct a synthetic dataset that replicates the empirical trends in Section 3. As in Section 2, the label $y\in\{1,-1\}$ is spuriously correlated with a spurious attribute $a\in\{1,-1\}$. We divide our training data into four groups accordingly: two majority groups with $a=y$, each of size $n_{\mathsf{maj}}/2$, and two minority groups with $a=-y$, each of size $n_{\mathsf{min}}/2$. We define $n=n_{\mathsf{maj}}+n_{\mathsf{min}}$ as the total number of training points, and $p_{\mathsf{maj}}=n_{\mathsf{maj}}/n$ as the fraction of majority examples. The higher $p_{\mathsf{maj}}$ is, the more strongly $a$ is correlated with $y$ in the training data.

The core and spurious features are both noisy and encode their respective attributes at different signal-to-noise ratios. We define the spurious-core information ratio (SCR) as $r_{\mathsf{s:c}}=\sigma_{\mathsf{core}}^{2}/\sigma_{\mathsf{spu}}^{2}$. The higher the SCR, the more signal there is about the spurious attribute in the spurious features, relative to the signal about the label in the core features.

Compared to the image datasets we studied in Section 3, this synthetic dataset offers two key simplifications. First, the only differences between groups stem from their differences in $(y,a)$, which isolates the effect of flipping the spurious attribute $a$. In contrast, in real datasets, groups can differ in other ways, e.g., more label noise in one group. Second, the relative difficulty of estimating $y$ versus $a$ is completely governed by changing $\sigma_{\mathsf{core}}^{2}$ and $\sigma_{\mathsf{spu}}^{2}$. In contrast, real datasets have additional complications, e.g., estimating $y$ might involve a more complex function of the input $x$ than estimating $a$, and there might be an inductive bias towards learning a simpler model over a more complex one.

In all of the experiments below, we fix the total number of training points $n$ to $3000$, and set $d=100$ (so each input $x$ has $2d=200$ dimensions). Unless otherwise specified, we set the majority fraction $p_{\mathsf{maj}}=0.9$ and the noise levels $\sigma_{\mathsf{spu}}^{2}=1$ and $\sigma_{\mathsf{core}}^{2}=100$ to encourage the model to use the spurious features over the core features.

2 Observations on synthetic dataset

The synthetic dataset replicates the trends we observe on real datasets. Figure 4 shows how average and worst-group error change with the number of parameters/random projections $m$. This matches the trends we obtained on CelebA and Waterbirds in Section 3. The best worst-group test error of 28.5% is achieved by an underparameterized model, whereas highly overparameterized models achieve high worst-group test error that plateaus at around 55%. In contrast, the average test error is better for overparameterized models than for underparameterized models.

Overparameterized models use spurious features. Figure 5-Right shows that overparameterized models have high test error on minority groups ($a=-y$) despite zero training error, but perform very well on the majority groups ($a=y$). Since the only difference between the minority and majority groups in the synthetic dataset is the relative signs of the core and spurious attributes, this suggests overparameterized models are using spurious features and simply memorizing the minority groups to get zero training error, consistent with our discussion in Section 3. In contrast, the underparameterized model has low training and test errors across all groups, suggesting that it relies mainly on core features.

These results imply that the degradation in the worst-group test error is due to the spurious features. We confirm that overparameterization no longer hurts when we “remove” the spurious features by replacing them with noise centered around zero (i.e., we replace the mean of $x_{\mathsf{spu}}$ by 0). In this case, the best worst-group test error is now obtained by an overparameterized model, as shown in Figure 5-Left.

3 Distributional properties

What properties of the training data make overparameterization hurt worst-group error? We study (i) $p_{\mathsf{maj}}$, which controls the relative size of majority to minority groups, and (ii) $r_{\mathsf{s:c}}$, the relative informativeness of spurious to core features. In the synthetic dataset, overparameterization hurts worst-group test error only when both are sufficiently high. In contrast, overparameterization helps average test error regardless; see Appendix A.3.

Effect of the majority fraction $p_{\mathsf{maj}}$. We observe that increasing $p_{\mathsf{maj}}=n_{\mathsf{maj}}/n$, which controls the relative size of the majority versus minority groups, makes overparameterization hurt worst-group error more (Figure 6). When the groups are perfectly balanced with $p_{\mathsf{maj}}=0.5$, overparameterization no longer hurts the worst-group test error, with overparameterized models achieving better worst-group test error than all underparameterized models. This suggests that group imbalance can be a key factor inducing the detrimental effect of overparameterization.

Effect of the spurious-core information ratio $r_{\mathsf{s:c}}$. Next, we characterize the effect of $r_{\mathsf{s:c}}=\sigma_{\mathsf{core}}^{2}/\sigma_{\mathsf{spu}}^{2}$, which measures the relative informativeness of the spurious versus core features. A high $r_{\mathsf{s:c}}$ means that the spurious features are more informative. We vary $r_{\mathsf{s:c}}$ by changing $\sigma_{\mathsf{spu}}^{2}$ while keeping $\sigma_{\mathsf{core}}^{2}=100$ fixed, since this does not change the best possible worst-group test error (with a model that uses only the core features $x_{\mathsf{core}}$). Figure 6 shows that the higher $r_{\mathsf{s:c}}$ is, the more overparameterization hurts. As $r_{\mathsf{s:c}}$ increases, the spurious features become more informative, and overparameterized models rely more on them than the core features; underparameterized models outperform overparameterized models only for sufficiently large $r_{\mathsf{s:c}}\geq 1$. Note that increasing $r_{\mathsf{s:c}}$ does not significantly affect the worst-group test error in the underparameterized regime, since the core features $x_{\mathsf{core}}$ are unaffected. In contrast, increasing the majority fraction $p_{\mathsf{maj}}$ hurts the worst-group test error in both underparameterized and overparameterized models.

4 An intuitive story

We return to the question of what makes overparameterized models memorize the minority instead of learning patterns that generalize on both majority and minority groups. The simulation results above show that of all overparameterized models that achieve zero training error, the inductive bias of the model class and training algorithm favors models that use spurious features which generalize only for the majority groups, instead of learning to use core features that also generalize well on the minority groups.

What is the nature of this inductive bias? Consider a model that predicts the label $y$ by returning its estimate of the spurious attribute $a$ from $x_{\mathsf{spu}}$, taking advantage of the fact that $y$ and $a$ are correlated in the training data. To get achieve zero training error, it will need to memorize the points in the minority group, e.g., by exploiting variations due to noise in the features $x$. On the other hand, consider a model that predicts $y$ by returning a direct estimate of $y$ based on the core features $x_{\mathsf{core}}$. Because $x_{\mathsf{core}}$ provides a noisier estimate of $y$ than $x_{\mathsf{spu}}$ does for $a$, this model will need to memorize all points for which $x_{\mathsf{core}}$ gives an inaccurate prediction of $y$ due to noise. Since the estimators of the core and spurious attributes are equally easy to learn, the main difference between these two models is the number of examples to be memorized.

We therefore hypothesize that the inductive bias favors memorizing as few points as possible. This is consistent with the results above: the model uses $x_{\mathsf{spu}}$ and memorizes the minority points only when the fraction of minority points is small (high majority fraction $p_{\mathsf{maj}}$). Similarly, the model uses $x_{\mathsf{spu}}$ over $x_{\mathsf{core}}$ to fit the majority points only when the spurious features are less noisy (high $r_{\mathsf{s:c}}$) and therefore require less memorization to obtain zero training error than the core features. In the next section, we make this intuition formal by analyzing a related but simpler linear setting.

Theoretical analysis

In this section, we show how the inductive bias against memorization leads to overparameterization exacerbating spurious correlations. Our analysis explicates the effect of the inductive bias and the importance of the data parameters $p_{\mathsf{maj}}$ and $r_{\mathsf{s:c}}$ discussed in Section 4.

The synthetic setting discussed in Section 4 is difficult to analyze because of the non-linear random projections, so we introduce a linear explicit-memorization setting that allows us to precisely define the concept of memorization. For clarity, we refer to the previous synthetic setting in Section 4 as the implicit-memorization setting. In Appendix A.4, we show empirically that models in these two settings behave similarly in the overparameterized regime, though they differ in the underparameterized regime.

In the previous implicit-memorization setting, we varied model size and memorization capacity by varying the number of random projections of the input. In the new explicit-memorization setting, we instead use linear models that act directly on the input and introduce explicit “noise features” that can be used to memorize. We vary the memorization capacity by varying the number of explicit noise features.

where $\sigma_{\mathsf{noise}}^{2}$ is a constant. The scaling by $1/N$ ensures that for large $N$, the norm of the noise vectors $\|x_{\mathsf{noise}}\|_{2}^{2}\approx\sigma_{\mathsf{noise}}^{2}$ is approximately constant with high probability. Intuitively, when $N$ is large, overparameterized models can use $x_{\mathsf{noise}}$ to fit a training point $x$ without affecting its predictions on other points, thereby memorizing $x$. We formalize this notion of memorization later in Section 5.2.

As before, the training data is composed of four groups, each corresponding to a combination of the label $y\in\{-1,1\}$ and the spurious attribute $a\in\{-1,1\}$: two majority groups with $a=y$, each of size $n_{\mathsf{maj}}/2$, and two minority groups with $a=-y$, each of size $n_{\mathsf{min}}/2$. Combined, there are $n$ training examples $\{(x^{(i)},y^{(i)})\}_{i=1}^{n}$.

In the underparameterized regime, the training data is not linearly separable and we simply have $\hat{w}^{\mathsf{rw}}=\hat{w}^{\mathsf{rw}}_{0}/\|\hat{w}^{\mathsf{rw}}_{0}\|_{2}$. In the overparameterized regime where $N\gg n$, the training data is linearly separable, and Rosset et al. 2004 showed that $\hat{w}^{\mathsf{rw}}={\hat{w}^{\mathsf{mm}}}$, where ${\hat{w}^{\mathsf{mm}}}$ is the max-margin classifier

The equivalence $\hat{w}^{\mathsf{rw}}={\hat{w}^{\mathsf{mm}}}$ holds regardless of the reweighting by $1/\hat{p}_{g}$: if we define the ERM estimator ${\hat{w}^{\mathsf{erm}}}$ analogously to (5) without the reweighting, it is also equal to ${\hat{w}^{\mathsf{mm}}}$. We will therefore analyze ${\hat{w}^{\mathsf{mm}}}$ in the overparameterized regime since it subsumes both $\hat{w}^{\mathsf{rw}}$ and ${\hat{w}^{\mathsf{erm}}}$.

We also note that if we use gradient descent to directly optimize the unregularized logistic regression objective (either reweighted or not), the resulting solution after scaling to unit norm also converges to ${\hat{w}^{\mathsf{mm}}}$ as the number of gradient steps goes to infinity (Soudry et al. 2018).

2 Analysis of worst-group error

We now state our main analytical result: in the explicit-memorization setting, the worst-group test error of a sufficiently overparameterized model is greater than $1/2$ (worse than random) under certain settings of $\sigma_{\mathsf{spu}}^{2},\sigma_{\mathsf{core}}^{2},n_{\mathsf{maj}},n_{\mathsf{min}}$. In contrast, underparameterized models attain reasonable worst-group error even under such a setting.

For any $p_{\mathsf{maj}}\geq\bigl(1-\frac{1}{2001}\bigr)$, $\sigma_{\mathsf{core}}^{2}\geq 1$, $\sigma_{\mathsf{spu}}^{2}\leq\frac{1}{16\log 100n_{\mathsf{maj}}}$, $\sigma_{\mathsf{noise}}^{2}\leq\frac{n_{\mathsf{maj}}}{600^{2}}$ and $n_{\mathsf{min}}\geq 100$, there exists $N_{0}$ such that for all $N>N_{0}$ (overparameterized regime), with high probability over draws of the data,

where ${\hat{w}^{\mathsf{mm}}}$ is the max-margin classifier.

However, for $N=0$ (underparameterized regime), with $p_{\mathsf{maj}}=\bigl(1-\frac{1}{2001}\bigr)$, $\sigma_{\mathsf{core}}^{2}=1$, and $\sigma_{\mathsf{spu}}^{2}=0$, and in the asymptotic regime with $n_{\mathsf{maj}},n_{\mathsf{min}}\rightarrow\infty$, we have

where $\hat{w}^{\mathsf{rw}}$ minimizes the reweighted logistic loss.

The result in the overparameterized regime applies to the max-margin classifier ${\hat{w}^{\mathsf{mm}}}$, which as discussed above subsumes both $\hat{w}^{\mathsf{rw}}$ and ${\hat{w}^{\mathsf{erm}}}$ when the data is linearly separable. The proof of Theorem 1 appears in Appendix B.

The conditions on $\sigma_{\mathsf{spu}}^{2}$ and $\sigma_{\mathsf{core}}^{2}$ in Theorem 1 above imply high spurious-core information ratio $r_{\mathsf{s:c}}$. Theorem 1 therefore provides a setting where high $p_{\mathsf{maj}}$ and high $r_{\mathsf{s:c}}$ provably make overparameterized models obtain high worst-group error, matching the trends we observed upon varying $p_{\mathsf{maj}}$ and $r_{\mathsf{s:c}}$ in the implicit-memorization setting (Figure 6). Furthermore, underparameterized models obtain reasonable worst-group error despite these conditions, mirroring the observations in earlier sections.

3 Overparameterization and memorization

We now sketch the key ideas in the proof of Theorem 1 (full proof in Appendix B), focusing first on the overparameterized regime. We start by establishing an inductive bias towards learning the minimum-norm model that fits the training data. We then define memorization and show how the minimum-norm inductive bias translates into a bias against memorization. Finally, we illustrate how the bias against memorization leads to learning the spurious feature and suffering high worst-group error.

Minimum-norm inductive bias. Define a separator as any model that correctly classifies all of the training points $(x,y)$ with margin $yw\cdot x\geq 1$. Then from standard duality arguments, ${\hat{w}^{\mathsf{mm}}}$ can be rewritten as ${\hat{w}^{\mathsf{minnorm}}}/\|{\hat{w}^{\mathsf{minnorm}}}\|$, the scaled version of the minimum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$

Since scaling does not affect the 0-1 test error, it suffices to analyze ${\hat{w}^{\mathsf{minnorm}}}$. Equation (9) shows that out of the set of all separators (which all perfectly fit the training data), the inductive bias favors the separator with the minimum norm. We now discuss how this minimum-norm inductive bias favors less memorization.

Memorization. For convenience, we denote the three components of a model $w$ as

In the overparameterized regime when $N\gg n$, a model can “memorize” a training point $x^{(i)}$ via $w_{\mathsf{noise}}$, in particular by putting a large weight $\alpha^{(i)}$ in the direction of $x^{(i)}$ (Equation (11)):

Because the noise vectors of the training points (high-dimensional Gaussians) are nearly orthogonal for large $N$, the component $\alpha^{(i)}{x_{\mathsf{noise}}^{(i)}}$ affects the prediction on $x^{(i)}$, but not on any other training or test points.

This ability to memorize plays a crucial role in making overparameterized models obtain high worst-group error. Intuitively, the minimum-norm inductive bias favors less memorization in overparameterized models. Roughly speaking, models that memorize more have larger weights $|\alpha^{(i)}|$ on the noise vectors ${x_{\mathsf{noise}}^{(i)}}$. Since these noise vectors are nearly orthogonal and have similar norm, this translates into a larger norm $\|w_{\mathsf{noise}}\|_{2}^{2}$.

Comparing using $x_{\mathsf{core}}$ versus using $x_{\mathsf{spu}}$. To illustrate how the inductive bias against memorization leads to high worst-group error, we consider two extreme sets of separators: (i) ones that use the spurious feature but not the core feature, denoted by $\mathcal{W}^{\mathsf{use-spu}}$ (ii) ones that use the core feature but not the spurious feature, denoted by $\mathcal{W}^{\mathsf{use-core}}$.

In scenario (i), using the spurious feature $x_{\mathsf{spu}}$ alone allows models to fit the majority groups very well. Thus, models that use $x_{\mathsf{spu}}$ only need to memorize the minority points. In Proposition 1, we construct a separator $w^{\mathsf{use-spu}}\in\mathcal{W}^{\mathsf{use-spu}}$ and show that its norm only scales with the number of minority points $n_{\mathsf{min}}$.

Conversely, in scenario (ii), using the core feature $x_{\mathsf{core}}$ alone allows models to fit all groups equally well. However, when $r_{\mathsf{s:c}}$ is high, $x_{\mathsf{core}}$ is noisier than $x_{\mathsf{spu}}$, so models that use $x_{\mathsf{core}}$ still need to memorize a constant fraction of all the training points. In Proposition 2, we show that norms of all separators $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$ are lower bounded by a quantity linear in the total number of training points $n$.

When the majority fraction $p_{\mathsf{maj}}$ is sufficiently large such that $n_{\mathsf{min}}\ll n$, the separator $w^{\mathsf{use-spu}}$ that uses $x_{\mathsf{spu}}$ will have a lower norm than any separator $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$ that uses $x_{\mathsf{core}}$. Since the inductive bias favors the minimum-norm separator, it prefers a separator $w^{\mathsf{use-spu}}$ that memorizes the minority points and suffers high worst-group error over any $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$.

When $\sigma_{\mathsf{core}}^{2},\sigma_{\mathsf{spu}}^{2}$ satisfy the conditions in Theorem 1, there exists $N_{0}$ such that for all $N>N_{0}$, with high probability, there exists a separator $w^{\mathsf{use-spu}}\in\mathcal{W}^{\mathsf{use-spu}}$ such that

for some constants $\gamma_{1},\gamma_{2}>0$.

To simplify exposition in this sketch, suppose that the noise vectors ${x_{\mathsf{noise}}^{(i)}}$ are orthogonal and have constant norm $\|{x_{\mathsf{noise}}^{(i)}}\|_{2}^{2}=\sigma_{\mathsf{noise}}^{2}$. We construct a separator $w^{\mathsf{use-spu}}\in\mathcal{W}^{\mathsf{use-spu}}$ that does not use the core feature $x_{\mathsf{core}}$ as follows. Set $w^{\mathsf{use-spu}}_{\mathsf{spu}}=\gamma_{1}$ for some large enough constant $\gamma_{1}>0$. This is sufficient to satisfy the margin condition on the majority points: since $\sigma_{\mathsf{spu}}^{2}$ is very small, w.h.p. all majority training points satisfy $y^{(i)}({x_{\mathsf{spu}}^{(i)}}\gamma_{1})\geq 1$.

However, for the minority training points, the spurious attribute $a$ does not match the label $y$, and in order to satisfy the margin condition with a positive $w^{\mathsf{use-spu}}_{\mathsf{spu}}$, these $n_{\mathsf{min}}$ minority points have to be memorized. Since $\sigma_{\mathsf{spu}}^{2}$ is very small, the decrease in the margin due to $w^{\mathsf{use-spu}}_{\mathsf{spu}}=\gamma_{1}$ is at most $-\rho\gamma_{1}$ w.h.p. for some constant $\rho$ that depends on $\sigma_{\mathsf{spu}}^{2}$. To satisfy the margin condition, it thus suffices to set $\alpha^{(i)}_{\mathsf{use-spu}}=y^{(i)}(1+\rho\gamma_{1})/\sigma_{\mathsf{noise}}^{2}$, and the bound on the norm follows. The full proof appears in Section B.2.6. ∎

When $\sigma_{\mathsf{core}}^{2},\sigma_{\mathsf{spu}}^{2}$ satisfy the conditions in Theorem 1 and $n_{\mathsf{min}}\geq 100$, there exists $N_{0}$ such that for all $N>N_{0}$, with high probability, all separators $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$ satisfy

Subsampling

Our results above highlight the role of the majority fraction $p_{\mathsf{maj}}$ in determining if overparameterization hurts worst-group test error. When $p_{\mathsf{maj}}$ is large, the inductive bias favors using spurious features because it entails memorizing only a relatively small number of minority points, while the alternative of using core features requires memorizing a large number of majority points. This suggests that reducing the memorization cost of using core features by directly removing some majority points could induce overparameterized models to obtain low worst-group error.

Here, we show that this approach of subsampling the majority group achieves good worst-group test error on the datasets studied above. Subsampling creates a new group-balanced dataset by randomly removing training points in all other groups to match the number of points from the smallest group (Japkowicz & Stephen 2002; Haixiang et al. 2017; Buda et al. 2018). We then train a model to minimize the average loss on this subsampled dataset. For a precise description, see Appendix A.6.

Figure 7 shows that overparameterized models trained via subsampling (Equation 15) obtain low worst-group error on the CelebA, Waterbirds, and synthetic (implicit-memorization) datasets. Across all three datasets, training via subsampling makes increasing overparameterization help both average and worst-group test error. Moreover, overparameterized models trained on subsampled data are comparable to or better than the best models trained on the full dataset (i.e., underparameterized models trained with reweighting).

Subsampling seems wasteful since it throws away a large fraction of the training data: we only use 3.4% of the full training data for CelebA, 4.6% for Waterbirds, and 10% for the synthetic dataset. However, the results above show that subsampling in overparameterized models matches or outperforms reweighting with underparameterized models. For example, on CelebA, an overparameterized model trained via subsampling obtains 11.1% average test and 15.1% worst-group test error, whereas an underparameterized model trained with reweighting obtains 11.3% average and 25.6% worst-group test error.

Subsampling vs. reweighting. Both subsampling and reweighting artificially balance the groups in the training data, and previous work on imbalanced datasets has concluded that reweighting is typically at least as effective as subsampling (Buda et al. 2018). However, we find a clear difference between subsampling and reweighting in the overparameterized regime: increasing overparameterization with reweighting increases worst-group error, while doing so with subsampling decreases worst-group error. The intuition developed in Sections 4 and 5 shed some light on this difference. Consider an overparameterized model: as in Section 5.1, reweighting does not change the learned model which is the max-margin classifier. However, subsampling reduces $p_{\mathsf{maj}}$. Recall that the inductive bias favors spurious features when the alternative of using core features requires memorizing a large number of training points. By reducing $p_{\mathsf{maj}}$, we reduce this memorization cost associated with core features, thereby inducing the model to use core features and achieve low worst-group test error.

Related work

The effect of overparameterization. The effect of overparameterization on average test error has been widely studied. In what is commonly referred to as “double descent”, increasing model size beyond zero training error decreases test error, despite conventional wisdom that overfitting should increase test error. This behavior has been observed empirically (Belkin et al. 2019; Opper 1995; Advani & Saxe 2017; Nakkiran et al. 2019) and shown analytically in high-dimensional regression (Hastie et al. 2019; Bartlett et al. 2019; Mei & Montanari 2019). These works focus on average test error and are consistent with our findings there. However, our focus is on worst-group test error, particularly when the groups are defined based on spurious attributes, and in this paper we establish that worst-group test error can behave quite differently from average test error.

Increasing overparameterization can actually improve model robustness to some types of distributional shifts (Hendrycks et al. 2019; Hendrycks & Dietterich 2019; Yang et al. 2020). In this light, our results show that the effect of overparameterization on model robustness can depend heavily on the dataset (e.g., properties like $p_{\mathsf{maj}}$ and $r_{\mathsf{s:c}}$), type of distributional shift, and training procedure.

Worst-group error. Prior work on improving worst-group error focused on the underparameterized regime, with methods based on weighting/sampling (Shimodaira 2000; Japkowicz & Stephen 2002; Buda et al. 2018; Cui et al. 2019), distributionally robust optimization (DRO) (Ben-Tal et al. 2013; Namkoong & Duchi 2017; Oren et al. 2019), and fair algorithms (Dwork et al. 2012; Hardt et al. 2016; Kleinberg et al. 2017). Our focus is on the overparameterized, zero-training-error regime; here, previous methods based on reweighting and DRO are ineffective (Wen et al. 2014; Byrd & Lipton 2019; Sagawa et al. 2020). As mentioned in Section 1, Sagawa et al. 2020 demonstrated that stronger $L_{2}$-regularization can improve worst-group error on neural networks (when coupled with reweighting or group DRO). Similarly Cao et al. 2019 show that data-dependent regularization can improve error on rare labels. While their work focuses on developing methods to improve worst-group error, our focus is on understanding the mechanisms by which overparameterization hurts worst-group error.

Discussion

Our work shows that overparameterization hurts worst-group error on real datasets that contain spurious correlations. We studied the implicit- and explicit-memorization settings to provide a potential story for why this might occur: there can be an inductive bias towards solutions that do not need to memorize as many training points, and this can favor models that exploit the spurious correlations.

However, our synthetic settings make several simplifying assumptions, e.g., they suppose that the model prefers the spurious feature because it is less noisy than the core feature. This assumption need not always apply, and different assumptions might also lead to overparameterization exacerbating spurious correlations. For example, there might exist a true classifier based on the core features which has high accuracy but which is relatively more complex (e.g., high parameter norm) and therefore not favored by the training procedure. Studying the effect of overparameterization in settings such as those is important future work.

We also observed that subsampling allows overparameterized models to achieve low average and worst-group test error, despite eliminating a large fraction of training examples. In contrast, when using the full training data, only underparameterized models attain low worst-group test error under our current training methods. These observations call for future work to develop methods that can exploit both the statistical information in the full training data as well as the expressivity of overparameterized models, so as to attain good worst-group and average test error.

We are grateful to Yair Carmon, John Duchi, Tatsunori Hashimoto, Ananya Kumar, Yiping Lu, Tengyu Ma, and Jacob Steinhardt for helpful discussions and suggestions. SS was supported by a Stanford Graduate Fellowship, AR was supported by a Google PhD Fellowship and Open Philanthropy Project AI Fellowship, and PWK was supported by the Facebook Fellowship Program.

Reproducibiltity

Code is available at https://github.com/ssagawa/overparam_spur_corr. All code, data, and experiments are available on the Codalab platform at https://worksheets.codalab.org/worksheets/0x1db77e603a8d48c8abebd67fce39cf8b.

References

Appendix A Supplemental experiments

In the main text, we focused on reweighted models, trained with the reweighted objective on the full data (Sections 3-5), as well as subsampled models, trained on subsampled data with the ERM objective (Section 6). Here, we study the effect of overparameterization on ERM models, trained with the ERM objective on the full data. Consistent with prior work, we observe that ERM models obtain poor worst-group error (near or worse than random), regardless of whether the model is underparameterized or overparameterized (Sagawa et al. 2020). We also confirm that overparameterization helps average test error (see, e.g., Nakkiran et al. 2019; Belkin et al. 2019; Mei & Montanari 2019).

We first consider the CelebA and Waterbirds dataset, following the experimental set-up of Section 3 but now training with the standard ERM objective (Equation (2)) instead of the reweighted objective (Equation (3)).

On these datasets, overparameterization helps the average test error (Figure 8). As model size increases past the point of zero training error, the average test error decreases. The best average test error is obtained by highly overparameterized models with zero training error—4.6% for CelebA at width 96, and 4.2% for Waterbirds at 6,000 random features.

In contrast, the worst-group error is consistently high across model sizes: it is consistently worse than random ($>$50%) for CelebA and nearly random (44%) for Waterbirds (Figure 8). These worst-group errors are much worse than those obtained by reweighted, underparameterized models ($25.6\%$ for CelebA and $26.6\%$ for Waterbirds; see Section 3). Thus, while overparameterization helps ERM models achieve better test error, these models all fail to yield good worst-group error regardless of the degree of overparameterization.

We also evaluate the effect of overparameterization on ERM models on the synthetic dataset introduced in Section 4. As above, ERM models fail to achieve reasonable worst-group test error across model sizes, but improve in average test error as model size increases (Figure 8). The best average test error is obtained by a highly overparameterized model with zero training error—9.0% error at 9,000 random features—while the worst-group test error is nearly random or worse ($>48$%) across model sizes.

A.2 Stronger L2L_{2} regularization improves worst-group error in overparameterized reweighted models

In the main text, we studied models with default/weak or no $L_{2}$ regularization. In this section, we study the role of $L_{2}$ regularization in modulating the effect of overparameterization on worst-group error by changing the hyperparameter $\lambda$ that controls $L_{2}$ regularization strength. Overall, we find that increasing $L_{2}$ regularization (to the point where models do not have zero training error) improves worst-group error but hurts average error in overparameterized reweighted models. In contrast, $L_{2}$ regularization has little effect on both worst-group and average error in the underparameterized regime.

In the main text, we trained ResNet10 models with default, weak regularization ($\lambda=0.0001$) on the CelebA dataset, and unregularized logistic regression on the Waterbirds and synthetic datasets. Here, we consider strongly-regularized models with $\lambda=0.1$ for both types of models; unlike before, these models no longer achieve zero training error even when overparameterized. Figure 9 shows the results of varying model size on strongly-regularized ERM, reweighted, and subsampled models on the three datasets.

On all three datasets, with strong regularization, ERM models continue to yield poor worst-group test error across model sizes, with similar or worse worst-group test error compared to with weak/ no regularization. Conversely, strongly-regularized subsampled models continue to achieve low worst-group test error across model sizes.

Where strong regularization has a large effect is on reweighted models. With reweighting, we find that strong regularization improves worst-group error in overparameterized models: across all three datasets, the worst-group test error in the overparameterized regime is much lower for the strongly-regularized models than their weakly regularized or unregularized counterparts (Figure 3). These results are consistent with similar observations made in Sagawa et al. 2020. However, even though strongly-regularized overparameterized models outperform weakly-regularized overparameterized models, overparameterization can still hurt the worst-group error in strongly-regularized reweighted models. On the CelebA and synthetic datasets, with $\lambda=0.1$, the best worst-group error is still obtained by an underparameterized model for the CelebA and synthetic datasets, though overparameterization seems to help worst-group error on the Waterbirds dataset at least in the range of model sizes studied.

Given a fixed overparameterized model size, how does its performance change with the $L_{2}$ regularization strength $\lambda$? We study this with the logistic regression model on the Waterbirds and synthetic datasets, using a model size of $m=10,000$ random features and varying the $L_{2}$ regularization strength from $\lambda=10^{-9}$ to $\lambda=10^{2}$. We did not run this experiment on the CelebA dataset for computational reasons, as doing so would have required tuning a different learning rate for each choice of regularization strength.

Results are in Figure 10. As before, ERM models obtain poor worst-group error regardless of the regularization strength, and subsampled models are relatively insensitive to regularization, achieving reasonable worst-group error at most settings of $\lambda$.

For reweighted models, however, having the right level of regularization is critical for obtaining good worst-group test error. On both datasets, the best worst-group test error is obtained by strongly-regularized models that do not achieve zero training error. In contrast, increasing regularization strength hurts average error, with the best average test error attained by models with nearly zero regularization.

In the above experiments, we kept either model size or regularization strength fixed, and varied the other. Here, we vary both: we consider $L_{2}$ regularization strengths $\lambda\in\{10^{-9},10^{-6},0.001,0.1,10\}$ and investigate the effect of increasing model size for each $\lambda$. We plot the results for Waterbirds and the synthetic dataset in Figure 11 and Figure 12 respectively.

For reweighted models, the results match what we observed above. Strengthening $L_{2}$ regularization reduces the detrimental effect of overparameterization on worst-group error. For any fixed model size in the overparameterized regime, the worst-group test error improves as $\lambda$ increases up to a certain value. Worst-group test error seems to plateau at different values as model size increases, depending on the regularization strength, though we note that it is possible that further increasing model size beyond the range we studied might lead models with different regularization strengths to eventually converge. Further empirical studies as well as theoretical characterization of the interaction between regularization and overparameterization are needed to confirm this phenomenon.

Given sufficiently large $\lambda$ (e.g., $\lambda=10$ for both Waterbirds and synthetic datasets), overparameterized models seem to outperform underparameterized models, at least for the range of model sizes studied. However, we caution that this trend does not seem to hold on the CelebA dataset (Figure 9).

Finally, in contrast with its effects on overparameterized models, regularization seems to only have a modest effect on worst-group test error in the underparameterized regime.

Figure 13 shows how the average test error changes as a function of model size under different settings of the majority fraction $p_{\mathsf{maj}}$ and the spurious-core ratio $r_{\mathsf{s:c}}$ on the synthetic dataset introduced in Section 4. As expected, overparameterization helps the average test error regardless of SCR and the majority fraction.

A.4 Comparison between implicit and explicit implicit memorization

However, in the highly underparameterized regime, the RP models do poorly because of model misspecification (owing to a small number of random projections), whereas the linear models can still learn to use $x_{\mathsf{core}}$ and therefore do well.

A.5 Experimental details

For the CelebA dataset, we use the official train-val-test split from Liu et al. 2015, with the Blond_Hair attribute as the target $y$ and the Male as the spurious association $a$.

For the Waterbirds dataset, we follow the setup in Sagawa et al. 2020; for convenience, we reproduce some details of how it was constructed here. This dataset was obtained by combining bird images from the CUB dataset (Wah et al. 2011) with backgrounds from the Places dataset (Zhou et al. 2017). The CUB dataset comes with annotations of bird species. For the Waterbirds dataset, each bird was labeled was a waterbird if it was a seabird or waterfowl in the CUB dataset; otherwise, it was labeled as a landbird. Bird images were cropped using the provided segmentation masks and placed on either a land (bamboo forest or broadleaf forest) or water (ocean or natural lake) background obtained from the Places dataset.

For Waterbirds, we follow the same train-val-test split as in Sagawa et al. 2020. Note that in these validation and test sets, landbirds and waterbirds are uniformly distributed on land and water backgrounds so that accuracy on the rare groups can be more accurately estimated. When calculating average test accuracy, we therefore first compute the average test accuracy over each group and then report a weighted average, with weights corresponding to the relative proportion of each group in the skewed training dataset.

We post-process Waterbirds by extracting feature representations taken from the last layer of a ResNet18 model pre-trained on ImageNet. We use the Pytorch torchvision implementation of the ResNet18 model for this. All models on the Waterbirds dataset in our paper are logistic regression models trained on top of this (fixed) feature representation.

We used a modified ResNet10 with variable widths, following the approach in Nakkiran et al. 2019 and extending the torchvision implementation. We trained all ResNet10 models with stochastic gradient descent with momentum of 0.9 and a batch size of 128, with the $L_{2}$ regularization parameter $\lambda$ was passed in to the optimizer as the weight decay parameter. In the experiments in the main text, we used the default setting of $\lambda=10^{-4}$. We used a fixed learning rate instead of a learning rate schedule and selected the largest learning rate for which optimization was stable, following Sagawa et al. 2020. This resulted in learning rates of 0.01 and 0.0001 for $\lambda=10^{-4}$ and $\lambda=0.1$, respectively, across all training procedures. As in the original ResNet paper (He et al. 2016), we used batch normalization (Ioffe & Szegedy 2015) and no dropout (Srivastava et al. 2014), and for simplicity, we trained all models without data augmentation.

We trained for 50 epochs for ERM and reweighted models and 500 epochs for subsampled models (due to smaller number of examples per epoch). We found that worst-group error can be unstable across epochs due to the small sample size and relatively large learning rate, so in our results we report the error averaged over the last 10 epochs.

We used the logistic regression implementation from scikit-learn, training with the L-BFGS solver until convergence with tolerance 0.0001, and setting the regularization parameter as $C=1/(n\lambda)$. For unregularized models, we set $\lambda=10^{-9}$ for numerical stability.

A.6 Subsampling

Formally, given a set of groups $\mathcal{G}$ and a dataset D comprising a set of $n$ training points with their group identities $\{(x^{(i)},y^{(i)},g^{(i)})\}$, the subsampling procedure involves two steps. First, we group training points based on group identities:

For each group $g$, we select a subset $\text{D}^{\textsf{ss}}_{g}\subseteq\text{D}_{g}$ uniformly at random from $\text{D}_{g}$ such that each subset has the same number of points as the smallest group in the training set. We form a new dataset $\text{D}^{\textsf{ss}}$ by combining these subsets:

Note that $\text{D}^{\textsf{ss}}$ is group-balanced, with $p_{\mathsf{maj}}=0.5$. We then train a model by minimizing the average loss on $\text{D}^{\textsf{ss}}$,

Since $\text{D}^{\textsf{ss}}$ is group-balanced, the reweighted training loss (Equation 3) has the same weight on all training points and minimizing the reweighted objective on $\text{D}^{\textsf{ss}}$ is equivalent to minimizing the average loss objective above.

Appendix B Proof of Theorem 1

Here, we detail the proof of Theorem 1 presented in Section 5. We structure the proof by splitting Theorem 1 into two smaller theorems: one for the overparameterized regime (Appendix B.2), and another for the underparameterized regime (Appendix B.3).

Further, by the representer theorem, we decompose $\hat{w}_{\mathsf{noise}}$ as

Note that $\alpha^{(i)}(w)$ is equivalent to the $\alpha^{(i)}$ referred to in the main text. Recall that we define memorization of each training point $x^{(i)}$ by the weight $\alpha^{(i)}$ as follows.

The component $\alpha^{(i)}(\hat{w}){x_{\mathsf{noise}}^{(i)}}$ serves to “memorize” $x^{(i)}$ when $N$ is sufficiently large, as it affects the prediction on $x^{(i)}$ but not on any other training or test points (because noise vectors are nearly orthogonal when $N$ is large). In the proof, we set the constant $\gamma^{2}$ appropriately (based on other parameter settings in Theorem 1) to get the required result.

Finally, let $G_{\mathsf{maj}},G_{\mathsf{min}}$ denote the indices of training points in the majority and minority group respectively.

B.2 Overparameterized regime

In our explicit-memorization set-up, sufficiently overparameterized models provably have high worst-group error under certain settings of $\sigma_{\mathsf{spu}}^{2},\sigma_{\mathsf{core}}^{2},n_{\mathsf{maj}},n_{\mathsf{min}}$ as stated in Theorem 1 (restated below as Theorem 2).

For any $p_{\mathsf{maj}}\geq\bigl(1-\frac{1}{2001}\bigr)$, $\sigma_{\mathsf{core}}^{2}\geq 1$, $\sigma_{\mathsf{spu}}^{2}\leq\frac{1}{16\log 100n_{\mathsf{maj}}}$, $\sigma_{\mathsf{noise}}^{2}\leq\frac{n_{\mathsf{maj}}}{600^{2}}$ and $n_{\mathsf{min}}\geq 100$, there exists $N_{0}$ such that for all $N>N_{0}$ (overparametrized regime), with high probability over draws of the data,

where ${\hat{w}^{\mathsf{mm}}}$ is the max-margin classifier.

In Section 5, we sketched key ideas in the proof by considering special families of separators: because the minimum-norm inductive bias favors less memorization, models can prefer to learn the spurious feature and memorize the minority examples (entailing high worst-group error), instead of learning the core feature and memorizing some fraction of all training points (possibly attaining reasonable worst-group error). We now provide the full proof of Theorem 2, generalizing the above key concepts by considering all separators.

Recall from Section 5 that we consider the maximum-margin classifier ${\hat{w}^{\mathsf{minnorm}}}$:

In other words, ${\hat{w}^{\mathsf{minnorm}}}$ is the minimum-norm separator, where separator is a classifier with zero training error and required margins, satisfying $y^{(i)}(w\cdot x^{(i)})\geq 1$ for all $i$. We analyze the worst-group error of the minimum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$ as outlined below:

We first upper bound the fraction of majority examples memorized by the minimum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$. We show that there exists a separator that can use spurious features and needs to memorize only the minority points (Lemma 1) for the parameter settings in Theorem 2 where $\sigma_{\mathsf{spu}}$ is sufficiently small. Since the norm of a separator is roughly scales with the number of points memorized ($|\alpha^{(i)}(\hat{w})|\geq\gamma^{2}/\sigma_{\mathsf{noise}}^{2}$), we have an upper bound on the number of training points memorized by ${\hat{w}^{\mathsf{minnorm}}}$. Since the number of majority points is much larger than the number of minority points, this says that only a small fraction of majority points could be memorized by ${\hat{w}^{\mathsf{minnorm}}}$.

Next, we observe that since the core feature is noisy as per the parameter setting in Theorem 2, if we do not use the spurious feature, a constant fraction of majority points have to be memorized if spurious features are not used. Conversely, if less than this fraction of majority points can be memorized, the separator must use spurious features. Since using spurious features leads to higher worst-group test error, this reveals a trade-off between the worst-group test error of a separator and the fraction of majority points that it memorizes at training time. Succinctly, smaller fraction memorized implies the use of spurious features which in turn implies higher worst-group test error. Smaller worst-group test error requires eliminating the use of spurious features which would lead to a large fraction of majority points requiring memorization in order for a classifier to be a separator. We formalize the above trade-off between the worst-group test error and fraction of majority examples to be memorized in Proposition 3.

Combining the two steps together, since ${\hat{w}^{\mathsf{minnorm}}}$ memorizes only a small fraction of majority points by virtue of being the minimum norm separator, ${\hat{w}^{\mathsf{minnorm}}}$ suffers high worst-group test error.

We now formally prove Theorem 2, invoking propositions that we prove in subsequent sections.

In the first part of the proof, we show that the minimum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$ “memorizes” a small fraction of the majority examples. Formally, we study the quantity $\delta_{\text{maj-train}}\left(\hat{w},\gamma^{2}\right)$ defined as follows.

Consider a separator $\hat{w}$ on training data $\{(x^{(i)},y^{(i)})\}_{i=1}^{n}$. Let $\delta_{\text{maj-train}}\left(\hat{w},\gamma^{2}\right)$ be the fraction of training examples that $\hat{w}$ $\gamma$-memorizes in the majority groups:

We provide an upper bound on $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$ (Lemma 4) by first bounding $\|{\hat{w}^{\mathsf{minnorm}}}\|$ and then bounding $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$ in terms of $\|{\hat{w}^{\mathsf{minnorm}}}\|$.

There exists a separator $w^{\mathsf{use-spu}}$ that satisfies $y^{(i)}(w^{\mathsf{use-spu}}\cdot x^{(i)})\geq 1,~\forall i\in G_{\mathsf{maj}},G_{\mathsf{min}}$. The norm of this separator gives a bound on $\|{\hat{w}^{\mathsf{minnorm}}}\|$ as follows. For the parameter settings under Theorem 2, with high probability, we have

for constants $u=1.3125,s=\frac{2.61}{\sigma_{\mathsf{noise}}^{2}}$.

In order to get an upper bound on $\|{\hat{w}^{\mathsf{minnorm}}}\|$, we compute the norm of a particular separator. Concretely, we consider a separator $w^{\mathsf{use-spu}}$ of the following form:

Now it remains to select appropriate values of constants $u$ and $s$ such that $y^{(i)}(w^{\mathsf{use-spu}}\cdot x^{(i)})\geq 1$ is satisfied for all training examples.

For majority points, this involves setting $u$ large enough such that the less noisy spurious feature can be used to obtain the required margin. Without loss of generality, assume $y^{(i)}=1$. Formally, for $i\in G_{\mathsf{maj}}$,

The first inequality follows from the fact that $\sigma_{\mathsf{spu}}$ is small enough under the parameter settings of Theorem 2 to allow a uniform bound on ${x_{\mathsf{spu}}^{(i)}}$ (Lemma 5). The second inequality follows from setting the number of random features $N$ to be large enough so that the noise features are near orthogonal (Lemma 8). Conversely, we have

Notice that the condition in Equation 23 requires that $u$ be greater than $0$. Since the minority points have spurious attribute $a=-y$, we need to set $s$ to be large enough so that $w^{\mathsf{use-spu}}$ as defined above separates the minority points. Just as before, we set $y=1$ WLOG. For $i\in G_{\mathsf{min}}$, we have

The steps are similar to the condition for majority points, with the key difference that the contribution from the noise term involves $s\|{x_{\mathsf{noise}}^{(i)}}\|_{2}^{2}$ (Lemma 9).

A set of parameters that satisfies both conditions above Equation 24 and Equation 23 is the following:

We use the fact that $c_{1}<1/2000$ (From Lemma 9).

This follows from bounds on $\|{x_{\mathsf{noise}}^{(i)}}\|_{2}^{2}$ (Lemma 9) and sum of less than $n^{2}$ terms involving $s^{2}{x_{\mathsf{noise}}^{(i)}}\cdot{x_{\mathsf{noise}}^{(j)}}$ (using Lemma 8). ∎

For a separator $\hat{w}$ with bounded $\alpha^{(i)}(\hat{w})^{2}\leq\frac{10n}{\sigma_{\mathsf{noise}}^{2}}$ for all $i=1,\dots,n$, its norm can be bounded with high probability as

under the parameter settings of Theorem 2.

The result follows bounded norms (Lemma 9), bounded dot products (Lemma 8), and the definition of $\delta_{\text{maj-train}}\left(\hat{w},\gamma^{2}\right)$ (Definition 3).

We now apply Lemma 1 and Lemma 2 in order to bound $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$, showing that the fraction of majority points that are memorized is small for appropriate choice of $\gamma$.

To invoke Lemma 2, we first show that the coefficient $\alpha^{(i)}({\hat{w}^{\mathsf{minnorm}}})$ is bounded above with high probabiltity.

Under the parameter settings of Theorem 2, with high probability, $\alpha^{(i)}({\hat{w}^{\mathsf{minnorm}}})$ is bounded above for $i=1,\dots,n$ as

Let $\max\limits_{i}\alpha^{(i)}({\hat{w}^{\mathsf{minnorm}}})=\frac{M}{\sigma_{\mathsf{noise}}^{2}}$.

From the upper bound on $\|{\hat{w}^{\mathsf{minnorm}}}\|_{2}^{2}$ (Lemma 1), we have

Since $c_{1}<1/2000$, and $n\geq 2000$, setting $u=1.3125,s\sigma_{\mathsf{noise}}^{2}=2.61$, we get $M^{2}\leq 10n$. ∎

Now, we are ready to show that $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$ is small.

Under the parameter settings of Theorem 2, the following is true with high probability.

Applying Lemma 2 to ${\hat{w}^{\mathsf{minnorm}}}$ by invoking the bounds on $\alpha^{(i)}({\hat{w}^{\mathsf{minnorm}}})$ (Lemma 3),

with high probability. Putting this together with Lemma 1, we have

where in the last step we substitute the constants $\gamma^{2}=9/10,u=1.3125,s\sigma_{\mathsf{noise}}^{2}=2.61,n_{\mathsf{maj}}/n_{\mathsf{min}}\leq 1/2000$ and $\sigma_{\mathsf{noise}}^{2}\leq n_{\mathsf{maj}}/360000$. ∎

B.2.2 Concentration Inequalities

With probability $>1-1/100$, if $\sigma_{\mathsf{spu}}\leq\frac{1}{4\sqrt{\log 100n}}$,

This follows from standard subgaussian concentration and union bound over $n=n_{\mathsf{maj}}+n_{\mathsf{min}}$ points.

For $N=\Omega(\text{poly}(n))$, with probability greater than $1-1/2000$,

This follows from Corollary 1 and union bound over $n^{2}$ pairs of training points.

For $N=\Omega(\text{poly}(n))$, with probability greater than $1-1/2000$,

This follows from Lemma 6 and union bound over $n$ training points. In particular, we can set $c_{1}<1/2000$ for large enough $N$.

In the previous section, we proved that $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$, the fraction of majority training samples that can have coefficient on the noise vectors greater than $\gamma^{2}/\sigma_{\mathsf{noise}}^{2}$ in the max margin separator ${\hat{w}^{\mathsf{minnorm}}}$ is bounded for suitable value of $\gamma$. We showed this using the fact that the norm of ${\hat{w}^{\mathsf{minnorm}}}$ is the smallest among all separators and the observation that the squared norm of a separator roughlty scales proportional the number of training points that have large coefficient along the noise vectors.

What does small $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$ imply? We now show that the bound on $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$ has an important consequence on the worst-group error $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$; low $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma\right)$ would imply high worst-group error $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$. We show that there is a trade-off between the worst-group test error of a separator and the fraction of majority points that it “memorizes” at training time. If a model that has low worst-group test error must use the core feature and not the spurious feature, and to obtain zero training error such a model would memorize a potentially large fraction of majority and minority points. In contrast, if the model instead uses only the spurious feature, then the worst-group test error would be high, but it would memorize only a small fraction of majority examples at training time; because we assume that the spurious feature is much less noisy than the core feature ($\sigma_{\mathsf{core}}\gg\sigma_{\mathsf{spu}}$), much fewer majority examples would need to be memorized. To summarize, a large $\hat{w}_{\mathsf{spu}}$ would require smaller fraction of majority points to be memorized $\delta_{\text{maj-train}}\left(\hat{w},\gamma^{2}\right)$ but increase the worst-group test error $\text{Err}_{\mathsf{wg}}({\hat{w}})$. We formalize the above trade-off between the worst-group error and fraction of majority examples to be memorized in Proposition 3.

For the minimum norm separator ${\hat{w}^{\mathsf{minnorm}}}$, under the parameter settings of Theorem 2, with high probability,

for some constants $c_{3},c_{4}<1/1000$ and $\Phi$ the Gaussian CDF.

For any separator $\hat{w}$ that spans the training points and satisfies

under the parameter settings of Theorem 2, with high probability,

for some constants $c_{1}<1/2000;c_{5},c_{6}<1/1000$ and $\Phi$ the Gaussian CDF.

As mentioned before, we see that the spurious component weight ${\hat{w}^{\mathsf{minnorm}}}_{\mathsf{spu}}$ has opposite effects on the two quantities; $\text{Err}_{\mathsf{wg}}({\hat{w}})$ increases with increase $\hat{w}_{\mathsf{spu}}$, but $\delta_{\text{maj-train}}\left(\hat{w},\gamma\right)$ decreases with increase in $\hat{w}_{\mathsf{spu}}$. This dependence can be exploited to relate the two quantities to each other as follows.

In other words, if the $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma\right)$ is low, then $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$ would need to be high.

B.2.4 Worst-group error is high

Recall from part 1 that $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma\right)<1/200$ for appropriate choice of $\gamma$, and from part 2 the trade-off between $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma\right)$ and $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$ (Equation (50)). As a final step, we need to bound the quantities on the RHS of Equation (50). All the constants are small, and $\gamma^{2}=9/10,\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},9/10\right)\leq 1/200$ (Lemma 4) which allows us to write

We have hence proved that the minimum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$ incurs high worst-group error with high probability under the specified conditions.

B.2.5 Proof of Proposition 3

We bound the expected worst-group error $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$, which is the expected worst-group loss over the data distribution. Below, we lower bound the worst-group error $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$ by bounding the error on a particular group: minority positive points which have label $y=1$ and spurious attribute $a=-1$. The test error is the probability that a test example $x$ from this group gets misclassified, i.e. ${\hat{w}^{\mathsf{minnorm}}}\cdot x<0$.

In the last step, we rewrite for convenience $x_{\mathsf{core}}=y+\sigma_{\mathsf{core}}z_{1}$ and $x_{\mathsf{spu}}=a+\sigma_{\mathsf{spu}}z_{2}$, where $z_{1},z_{2}\sim\mathcal{N}(0,1)$.

We use the properties of high-dimensional Gaussian random vectors to bound the quantity ${\hat{w}^{\mathsf{minnorm}}}_{\mathsf{noise}}\cdot x_{\mathsf{noise}}$. Recall that ${\hat{w}^{\mathsf{minnorm}}}_{\mathsf{noise}}$ can be written as

From the expression above, we see that $\text{Err}_{\mathsf{wg}}({{\hat{w}^{\mathsf{minnorm}}}})$ increases as the spurious component ${\hat{w}^{\mathsf{minnorm}}}_{\mathsf{spu}}$ increases. This is because in the minority group, the spurious feature is negatively correlated with the label.

We now compute a lower bound on $\delta_{\text{maj-train}}\left({\hat{w}^{\mathsf{minnorm}}},\gamma^{2}\right)$, which is the number of majority points (where $a=y$) that are “memorized.” Intuitively, we want to show that the fraction depends on $\hat{w}_{\mathsf{spu}}-\hat{w}_{\mathsf{core}}$. The more the core feature is used relative to the spurious feature, the larger fraction of points need to be memorized because the core feature is more noisy.

First, consider a separator $\hat{w}$ with some core and spurious components $\hat{w}_{\mathsf{core}}$ and $\hat{w}_{\mathsf{spu}}$. Recall that $\hat{w}_{\mathsf{noise}}=\sum\limits_{i}\alpha^{(i)}(\hat{w}){x_{\mathsf{noise}}^{(i)}}$ and $y^{(i)}(\hat{w}\cdot x^{(i)})\geq 1$ by the definition of separators. For a given $\hat{w}_{\mathsf{core}}$ and $\hat{w}_{\mathsf{spu}}$, we want to bound the fraction of majority points ($a=y$) which can have $\alpha^{(i)}(\hat{w})<\frac{\gamma^{2}}{\sigma_{\mathsf{noise}}^{2}}$. We focus only on separators with bounded memorization, i.e. those that satisfy ${\alpha^{(i)}(\hat{w})}^{2}\leq\frac{10n}{\sigma_{\mathsf{noise}}^{4}}$. Note that from Lemma 3, w.h.p., the mininum-norm separator ${\hat{w}^{\mathsf{minnorm}}}$ satifies this condition.

We bound the above by bounding a related quantity: the fraction of points that are memorized in the training distribution in expectation. We then use concentration to relate it to the fraction of the training set.

Formally, we have fixed quantities $\hat{w}_{\mathsf{core}}$ and $\hat{w}_{\mathsf{spu}}$. The training set is generated as per the usual data generating distribution. As before, we are interested in separators on the training set. For any majority training point, the coefficient $\alpha^{(i)}(\hat{w})$ in a separator is a random variable. Since training point $i$ is separated, we have

From Lemma 8, Lemma 6, and the condition on $\alpha^{(i)}(\hat{w})$, this implies with high probability that

B.2.6 Proof of Proposition 1

The proposition follows directly from Lemma 1.

The constant $\gamma_{1}=u=1.3125$ and $\gamma_{2}=s\sigma_{\mathsf{noise}}^{2}(2+c_{1})=2.61(2+c_{1})$ for $c_{1}<1/2000$. ∎

B.2.7 Proof of Proposition 2

To bound the norm for all $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$, we provide a lower bound on the norm of the minimum-norm separator in the set $\mathcal{W}^{\mathsf{use-core}}$:

We bound the $\|{\bar{w}}^{\mathsf{use-core}}\|$ in two steps:

We first provide a lower bound for $\|{\bar{w}}^{\mathsf{use-core}}\|$ in terms of the fraction of training points memorized $\delta_{\text{train}}\left({\bar{w}}^{\mathsf{use-core}},\gamma^{2}\right)$ (defined formally below) in Corollary 2.

We then provide a lower bound for $\delta_{\text{train}}\left({\bar{w}}^{\mathsf{use-core}},\gamma^{2}\right)$ in Corollary 3.

We first formally define $\delta_{\text{train}}\left(\hat{w},\gamma^{2}\right)$.

For a separator $\hat{w}$ on training data $\{(x^{(i)},y^{(i)})\}_{i=1}^{n}$, let $\delta_{\text{train}}\left(\hat{w},\gamma^{2}\right)$ be the fraction of training examples that $\hat{w}$ $\gamma$-memorizes:

For a separator $\hat{w}$ with bounded $\alpha^{(i)}(\hat{w})^{2}\leq\frac{10n}{\sigma_{\mathsf{noise}}^{2}}$ for all $i=1,\dots,n$, its norm can be bounded with high probability as

Similarly to the proof of Lemma 2, the result follows bounded norms (Lemma 9), bounded dot products (Lemma 8), and the definition of $\delta_{\text{train}}\left(\hat{w},\gamma^{2}\right)$ (Definition 4).

The result follows from applying Lemma 10 to ${\bar{w}}^{\mathsf{use-core}}$, invoking the bounds on any individual component $\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})$ obtained below in Lemma 11. ∎

Below, we bound $\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})$, where $\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})$ is the component of training point $i$ to the classifier ${\bar{w}}^{\mathsf{use-core}}$ via the representer theorem.

With high probability, $i=1,\dots,n$, $\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})$ can be bounded as follows.

As a first step, we upper bound the norm of ${\bar{w}}^{\mathsf{use-core}}$ by the norm of another separator $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$, using the fact that ${\bar{w}}^{\mathsf{use-core}}$ is the minimum-norm separator in $\mathcal{W}^{\mathsf{use-core}}$. In particular, we construct a separator $w^{\mathsf{use-core}}\in\mathcal{W}^{\mathsf{use-core}}$ that “memorizes” all training points, of the following form:

This is analogous to the construction of $w^{\mathsf{use-spu}}\in\mathcal{W}^{\mathsf{use-spu}}$ (Lemma 1), and similar calculations can be used to obtain a suitable value $\alpha$ to ensure that $w^{\mathsf{use-core}}$ is a separator with high probability. We provide it below for completeness. We show that the following condition is sufficient to satisfy the margin constraints $y^{(i)}w^{\mathsf{use-core}}\cdot x^{(i)}\geq 1$ for all $i=1,\dots,n$ with high probability:

for $c_{1}<1/2000$. We obtain the above condition by applying Lemma 8 and Lemma 9 to the margin condition.

Thus, we can construct $w^{\mathsf{use-core}}$ by setting some constant $\alpha\sigma_{\mathsf{noise}}^{2}\leq 2$.

Now that we have constructed $w^{\mathsf{use-core}}$, we can bound the norm of the minimum norm separator ${\bar{w}}^{\mathsf{use-core}}$ by the norm of $w^{\mathsf{use-core}}$. The following is true with high probability,

Finally, we bound $\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})$ for all $i$ by bounding $\max\limits_{i}\alpha^{(i)}({\bar{w}}^{\mathsf{use-core}})=\frac{M}{\sigma_{\mathsf{noise}}^{2}}$. As we showed in the proof of Lemma 3, following is true with high probability:

Combined with the upper bound on $\|{\bar{w}}^{\mathsf{use-core}}\|_{2}^{2}$ (Equation (80)), we have

Since $c_{1}<1/2000$, and $n\geq 2000$, setting $\alpha\sigma_{\mathsf{noise}}^{2}=2$ yields $M^{2}\leq 10n$ with high probability. ∎

Under the parameter settings of Theorem 2, with high probability,

for some constants $c_{1}<1/2000;c_{5},c_{6}<1/1000$ where $\Phi$ is the Gaussian CDF.

The result follows from applying Proposition 3 (which computes a bound on the majority fraction of points that is $\gamma-$memorized) to ${\bar{w}}^{\mathsf{use-core}}$, invoking Lemma 11, and plugging in ${\bar{w}}^{\mathsf{use-core}}_{\mathsf{spu}}=0$. Note that when ${\bar{w}}^{\mathsf{use-core}}_{\mathsf{spu}}=0$, $\delta_{\text{train}}\left({\bar{w}}^{\mathsf{use-core}},\gamma^{2}\right)=\delta_{\text{maj-train}}\left({\bar{w}}^{\mathsf{use-core}},\gamma^{2}\right)$. ∎