No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems

Introduction

In many real-world classification tasks, each labeled class consists of multiple semantically distinct subclasses that are unlabeled. Because models are typically trained to maximize global metrics such as average performance, they often underperform on important subclasses . This phenomenon—recently termed hidden stratification—can lead to skewed assessments of model quality and result in unexpectedly poor performance when models are deployed . For instance, a medical imaging model trained to classify between benign and abnormal lesions may achieve high overall performance, yet consistently mislabel a rare but critical abnormal subclass as “benign” . As another example, a well-known model for classifying chest radiographs was shown to perform substantially worse at recognizing pneumothorax (collapsed lung) on the subclass of pneumothorax images without a chest drain–which is worrisome since chest drains are the common form of treatment for the condition, so the drain-free subclass is in fact the clinically important one .

Modern robust optimization techniques can improve performance on poorly-performing groups when the group identities are known . However, in practice, a key obstacle is that subclasses are often unlabeled, or even unidentified. This makes even detecting such performance gaps—let alone mitigating them—a challenging problem. Nevertheless, recent empirical evidence encouragingly suggests that feature representations of deep neural networks often carry information about unlabeled subclasses (e.g., Figure 1). Motivated by this observation, we propose a method for addressing hidden stratification, by both measuring and improving worst-case subclass performance in the setting where subclass labels are unavailable. Our work towards this is organized into four main sections.

First, in Section 3 we propose a simple generative model of the data labeling process. Using this model, we show that when label annotations are insufficiently fine-grained—as is often the case in real-world datasets—hidden stratification can naturally arise. For instance, an image classification task might be to classify birds vs. frogs; if labels are only provided for these broad classes, they may fail to capture visually meaningful finer-grained, intra-class variation (e.g., “bird in flight” versus “bird in nest”). We show that in the setting of our generative model, standard training via empirical risk minimization (ERM) can result in arbitrarily poor performance on underrepresented subclasses.

Second, in Section 4 we use insights from this generative model to motivate George, a two-step procedure for alleviating hidden stratification by first estimating the subclass labels and then exploiting these estimates to train a robust classifier. To estimate subclass labels, we train a standard model on the task, and split each class (or “superclass,” for clarity) into estimated subclasses via unsupervised clustering in the model’s feature space. We then exploit these estimated subclasses by training a new model to optimize worst-case performance over all estimated subclasses using group distributionally robust optimization (GDRO) . In this way, our framework allows ML practitioners to automatically detect poorly-performing subclasses and improve performance on them, without needing to resort to expensive manual relabeling of the data.

Third, in Section 5 we use our generative framework to prove that—under sufficiently strong conditions on the data distribution and the quality of the recovered clusters—George can reduce the subclass performance gap, attaining the same asymptotic sample complexity rates as if the true subclass labels were known.

Fourth, in Section 6 we empirically validate the ability of George to both measure and mitigate hidden stratification on four image classification tasks, comprising both robustness benchmarks and real-world datasets. We demonstrate that the first step of George—training an ERM model and clustering the superclass features—often recovers clusters that align closely with true subclasses. We evaluate the ability of these clusters to measure the worst-case subclass (i.e., “robust”) performance: on average, the gap between worst-case cluster performance and worst-case subclass performance is less than half the gap between overall and worst-case subclass performance, indicating that George enables more accurate measurement of robust performance. Next, we show that the second stage of George—retraining a robust model using cluster assignments as proxy subclass labels—reduces average worst-case subclass error rates by 22% on these datasets. For comparison, the state-of-the-art “oracle” GDRO method that does require subclass labels reduces average worst-case subclass error rates by 51%. As an extension, we show that leveraging recent pretrained image embeddings for clustering can substantially further improve the robust performance of George, in some cases to nearly match the performance of GDRO trained using the true subclass labels.Code for George can be found at https://github.com/HazyResearch/hidden-stratification.

Background

Our work builds upon prior work from three main areas: robust optimization, representation learning, and unsupervised clustering. We provide a more extensive discussion of related work in Appendix A.

Distributionally Robust Optimization. Robustness and fairness is an active research area in machine learning . Distributionally robust optimization (DRO) attempts to guarantee good performance in the presence of distribution shift, e.g., from adversarial perturbations or evaluation on arbitrary subpopulations . Because these notions of robustness can be pessimistic , others investigate group DRO (GDRO), which optimizes worst-case performance over a known set of groups (partitions) of the data . A major obstacle to applying GDRO methods in practice is that group labels are often unavailable; in our work, we aim to address this issue in the classification setting.

Representation Learning & Clustering. Our approach relies on estimating unknown subclass labels by clustering a feature representation of the data. Techniques for learning semantically useful image features include autoencoder-based methods , the use of unsupervised auxiliary tasks , and pretraining on massive datasets . Such features may be used for unsupervised identification of classes, either using clustering techniques or an end-to-end approach . It has also been observed that when a model is trained on coarse-grained class labels, the data within each class can often be separated into distinct clusters in model feature space (e.g., ). While we primarily focus on the latter approach, we also evaluate the utility of pretrained embeddings as a source of features for clustering.

2 Problem Setup

We are given $n$ datapoints $x_{1},\dots,x_{n}\in\mathcal{X}$ and associated superclass labels $y_{1},\dots,y_{n}\in\{1,\dots,B\}$.We assume $B>1$ since otherwise the “classification problem” is trivial. In addition, each datapoint $x_{i}$ is associated with a latent (unobserved) subclass label $z_{i}\in\{1,\dots,C\}$. We assume that $\{1,\dots,C\}$ is partitioned into disjoint nonempty sets $S_{1},\dots,S_{B}$ such that if $z_{i}\in S_{b}$, then $y_{i}=b$; in other words, the subclass label $z_{i}$ determines the superclass label $y_{i}$. Let $S_{b}$ denote the set of all subclasses comprising superclass $b$, and $S(c)$ denote the superclass corresponding to subclass $c$.

Our goal is to classify examples from $\mathcal{X}$ into their correct superclass. Given a function class $\mathcal{F}$, it is typical to seek a classifier $f\in\mathcal{F}$ that maximizes overall population accuracy:

By contrast, we seek to maximize the robust accuracy, defined as the worst-case expected accuracy over all subclasses:

To approximate (2), if we knew $z_{1},...,z_{n}$ we could minimize the worst-case per-subclass training risk by solving:

Modeling Hidden Stratification

In Section 3.1, we introduce a generative model of the data labeling process. In Section 3.2, we use this model to explain how hidden stratification can occur, and show that in the setting of this model ERM can attain arbitrarily poor robust risk compared to GDRO.

In real datasets, individual datapoints are typically described by multiple different attributes, yet often only a subset of these are captured by the class labels. For example, a dataset might consist of images labeled “cat” or “dog.” These coarse class labels may not capture other salient attributes (color, size, breed, etc.); these attributes can be interpreted as latent variables representing different subclasses.

A key assumption is that the subclasses are “meaningful” in some sense, rather than just arbitrary groups of datapoints. Thus, rather than attempting to enforce good performance on all possible subsets of the data, we assume some meaningful structure on the subclass data distributions. We model this via the Gaussian assumption on $\mathcal{P}(V_{i}|\vec{Z})$, which is similar to that often made for the latent space of GANs . Consequently, the data distribution is a mixture of Gaussians in the “canonical feature space” $\vec{V}$, which facilitates further theoretical analysis (Section 5). Our generative model also bears similarity to that of , who use a hierarchical data-generation model to analyze the behavior of DRO methods in the presence of distribution shift.

2 What Causes Hidden Stratification, and When Can It Be Fixed?

We now use our generative model to help understand why hidden stratification can occur, and present a simple example in which ERM is provably suboptimal in terms of the robust risk.

We distinguish between two main causes of hidden stratification: inherent hardness and dataset imbalance. First, certain subclasses are “inherently harder” to classify because they are more similar to other superclasses. We define the inherent hardness of a task as the minimum attainable robust error; inherent hardness thus lower bounds the worst-case subclass error of any model. See Appendix D for more discussion.

Second, imbalance in subclass sizes can cause ERM to underserve rare subclasses, since it optimizes for average-case performance. We provide a simple concrete example (3.1) below. Unlike inherent hardness, robust performance gaps arising from dataset imbalances can be resolved if subclass labels are known, by using these labels to minimize the objective in Equation (4) via GDRO.

Figure 3 depicts an example distribution generated by the model in Section 3.1. In this example, the binary attribute vector $\vec{Z}$ has dimension 2, i.e., $\vec{Z}=(Z_{1},Z_{2})$, while only $Z_{2}$ determines the superclass label $Y$, i.e., $Y=Z_{2}$. The latent attribute $Z_{1}$ induces two subclasses in each superclass, each distributed as a different Gaussian in feature space, with mixture proportions $\alpha$ and $1-\alpha$ respectively. For linear models with regularized logistic loss, there exists a family of distributions of this form such that as the proportion $\alpha$ of the rare subclasses goes to , the worst-case subclass accuracy of ERM is only $O(\alpha)$, while that of GDRO is $1-O(\alpha)$. (See Appendix D.1 for the specific parameters of the per-subclass distributions in this example and a proof of the claim.)

Example 3.1 illustrates that when the dataset is imbalanced—i.e., the distribution of the underlying attributes $\vec{Z}$ is highly nonuniform—knowledge of subclass labels can improve robust performance. We thus ask: how well can we estimate subclass labels if they are not provided? In the extreme, if two subclasses of a superclass have the same distribution in feature space, we cannot distinguish them. However, the model must then perform the same on each subclass, since its prediction is a fixed function of the features! Conversely, if one subclass has higher average error, it must lie “further across” the decision boundary, meaning that the two subclasses must be separable to some degree; the larger the accuracy gap, the more separable the subclasses are. We formalize this in Appendix D.3.

George: A Framework for Mitigating Hidden Stratification

Inspired by the insights of Section 3, we propose George, an algorithm to mitigate hidden stratification. A schematic overview of George is provided in Figure 4.

Under the generative model of Section 3.1, each subclass is described by a different Gaussian in latent feature space. This suggests that a natural approach to identify the subclasses is to transform the data into feature space, and then cluster the data into estimated subclasses. However, this feature space is unknown. To obtain a surrogate for this feature space, we leverage the empirical observation that feature representations of deep neural networks trained on a superclass task can carry information about unlabeled subclasses . Next, to improve performance on these estimated subclasses, we minimize the maximum per-cluster average loss, by using the clusters as groups in the GDRO objective . We provide more details below, and pseudocode in Appendix B (Algorithm 1).

In practice, we apply UMAP dimensionality reduction before clustering, as we find it improves results (Appendix B). Additionally, based on the insight of Section 3.2 that subclasses with high loss differences are more separable, we also use the loss component (i.e., the component of the activation vector orthogonal to the decision boundary) as an alternative representation.

We first tried using standard clustering methods (such as $k$-means and Gaussian mixture model clustering) in our work. By visual inspection, we found that these methods often failed to capture smaller clusters, even if they were well-separated. However, missing small clusters like this is problematic for George, since these small clusters frequently correspond to rare, low-performing subclasses. Additionally, these methods require specification of $k$. We apply over-clustering (clustering using a larger $k$) to remedy this problem in an efficient manner. Naive overclustering also has drawbacks as it still requires manual specification of $k$, and if $k$ is set too large, several clusters can be spurious and result in overly pessimistic and unstable measurements of robust performance (as we explore in Appendix C.5). Thus, we develop a fully automated criterion based on the commonly used Silhouette (SIL) criterion to search for the number of clusters $k$, over-cluster to find smaller clusters that were missed, and filter out the spurious overclusters. Empirically, our clustering approach significantly improves performance over “vanilla” clustering; we hope that it may be of independent interest as well. We describe our procedures in more detail in Appendix B.

$k$ and other clustering and dimensionality reduction hyperparameters are selected automatically based on an unsupervised SIL criterion as described further in Appendix B.

2 Step 2: Exploiting Approximate Subclass Labels

Analysis of George

We now analyze a simple mixture model data distribution, based on the generative model presented in Section 3.1. We show that in this setting, unlike ERM, George converges to the optimal robust risk at the same sample complexity rate as GDRO when it is able to recover the true latent features $\vec{Z}$ (and when “soft” group predictions are used). Specifically, Example 3.1 shows that the robust risk of ERM can be arbitrarily worse than that of GDRO, for data generated according to the generative model in Section 3.1. By contrast, if the subclass labels estimated by George are sufficiently accurate, then the objective minimized in Step 2 of George well approximates the true GDRO objective (4). In Theorem 1, we use this to show that, when each subclass is described by a different Gaussian in feature space, George (with soft group predictions) achieves the same optimal asymptotic sample complexity rates as GDRO trained with true subclass labels. We sketch the argument below; full proofs are deferred to Appendix D.

First, suppose we could compute the true data distribution $\mathcal{P}(x,y,z)$. Our goal is to minimize the maximum per-subclass training loss by solving Eq. (4). Even with infinite data, we cannot estimate the individual $z_{i}$’s to arbitrary accuracy, so we cannot directly compute the objective in (4). However, we can estimate the per-subclass losses as follows: for each training example $(x_{i},y_{i})$, we use $\mathcal{P}$ to compute the probability that it comes from subclass $c$, and use that to weight the loss corresponding to that example. In Lemma 1, we show that when the training data is randomly sampled from $\mathcal{P}$, this yields an unbiased estimate of the average per-subclass empirical risk.

Theorem 1 implies that if each subclass is Gaussian in feature space, and we have access to this feature space (i.e., we can invert the mapping $g$ from features $\vec{Z}$ to data $X$), then we can cluster the features to estimate $\hat{\mathcal{P}}$, and the robust generalization performance of the model that minimizes the resulting perturbed training loss $\hat{R}_{\text{robust}}$ scales the same as does that of the minimizer of the true robust training loss $R_{\text{robust}}$, in terms of the amount of data required. This underscores the importance of recovering a “good” feature space; empirically, we show in Appendix C that the choice of model architecture can indeed dramatically impact the model feature space and thus the ability to recover subclasses.

Experiments

We empirically validate that George can mitigate hidden stratification across four datasets. In Section 6.2, we show that when subclass labels are unavailable, George improves robust performance over standard training methods. In Section 6.3, we analyze the clusters returned by George to understand the reasons for this improvement; we confirm that George identifies clusters that correspond to poorly-performing subclasses, which enables accurate measurement of robust performance. In Section 6.4, we ablate the contributions that George’s robust training objective and George’s improved measurement of validation robust performance each make to the performance gains of George. Finally, in Section 6.5, we evaluate the use of recent pretrained image embeddings as a source of features for George, and find that this further improves performance of George on some applications. Additional details on datasets, model architectures, and experimental procedures are provided in Appendix B.

Waterbirds. Waterbirds, a robustness benchmark introduced to evaluate GDRO in , contains images of land-bird and water-bird species on either land or water backgrounds. The task is to classify images as “land-bird” or “water-bird”; however, 95% of land (water)-birds are on land (water) backgrounds, causing ERM to frequently misclassify both land-birds on water and water-birds on land.

Undersampled MNIST (U-MNIST). We design U-MNIST as a modified version of MNIST , where the task is to classify digits as ‘$<$5’ and ‘$\geq$5’ (digits 0-9 are the subclasses). In addition, we remove 95% of ‘8’s; due to its rarity, it is challenging for ERM to perform well on the ‘8’ subclass.

CelebA. CelebA is a common face classification dataset also used as a robustness benchmark in . The task is to classify faces as “blond” or “not blond.” Because only 6% of blond faces are male, ERM performs poorly on this rare subclass.

ISIC. The ISIC skin cancer dataset is a public real-world dataset for classifying skin lesions as “malignant” or “benign.” 48% of benign images contain a colored patch. Of the non-patch examples, 49% required histopathology (a biopsy) to diagnose. We report AUROC for ISIC, as is standard .

2 End-to-End Results

We first show that George substantially improves the worst-case subclass accuracy, while modestly affecting overall accuracy. (Recall that we refer to worst-case subclass accuracy as “robust accuracy” [Eq. (2)].) We train models on each dataset in Section 6.1 using (a) ERM, (b) George, and (c) GDRO with true subclass labels (“Subclass-GDRO”), and report both robust and overall performance metrics in Table 1. Compared to ERM, training with George improves robust accuracy by up to 22 points, and substantially reduces the gap between the robust error of the ERM model and that of the subclass-GDRO model—despite the fact that George does not require subclass labels. In Appendix C, we show that George also outperforms other subclass-agnostic baselines, such as GDRO trained using the superclasses as groups.

On Waterbirds, U-MNIST, and CelebA, George significantly improves worst-case subclass accuracy over ERM. On ISIC, all methods perform similarly in terms of both AUROC on the non-patch subclass and AUROC on the clinically meaningful histopathology subclass. On CelebA, although George improves upon ERM, it substantially underperforms subclass-GDRO. However, this gap can be closed when improved features are used: if we cluster pretrained BiT embeddings rather than ERM features and use the resulting cluster assignments for the second stage of George, the robust accuracy improves to nearly match that of subclass-GDRO. We describe this experiment in Section 6.5.

In terms of overall performance, ERM generally performs best (as it is designed to optimize for average-case performance), followed by George and then subclass-GDRO. However, this difference is generally much smaller in magnitude than the increase in robust performance.

3 Clustering Results

Step 1 of George is to train an ERM model and cluster the data of each superclass in its feature space. We analyze these clusters to better understand George’s behavior. First, in Section 6.3.1 we show that George finds clusters that align well with poorly-performing human-labeled subclasses. This helps explain why the second step of George, running GDRO using the cluster assignments as groups, improves performance on these subclasses (as demonstrated in Section 6.2). Next, in Section 6.3.2 we show that George can discover meaningful subclasses that were not labeled by human annotators. Finally, in Section 6.3.3 we show that the worst-case performance measured on the clusters returned by George is a good approximation of the true robust performance.

We evaluate the ability of George to identify clusters that correspond to the true subclasses. We focus on identification of poorly-performing subclasses, as these determine robust performance. In Table 2, we compute the precision and recall of the cluster returned by George that most closely aligns with each given subclass. Precision is the fraction of cluster examples with that subclass label; recall is the fraction of subclass examples assigned to the cluster. For each poorly-performing subclass, George identifies a cluster with high recall and better-than-random precision. (Interestingly, while the precision and recall are substantially better than random in all cases, they are often still quite far from the optimal value of 1.0, but Step 2 of George nevertheless achieves substantial worst-group performance gains.)

We note that the lower recall on ISIC is because the no-patch subclass is often split into multiple clusters; in fact, this subclass is actually composed of two semantically distinct groups as discussed below. If these sub-clusters are combined into one, the precision and recall of the resulting cluster at identifying no-patch examples are $>0.99$ and $>0.97$ respectively.

3.2 Unlabeled Subclass Discovery

In addition to yielding clusters aligned with human-annotated subclasses, our procedure can identify semantically meaningful subclasses that were not specified in the human-provided schema. On U-MNIST, 60% of trials of George partition the “7” subclass into two subclusters, each containing stylistically different images (Figure 6c, Appendix C). On ISIC, 70% of George trials reveal distinct benign clusters within the no-patch subclass (see Figure 6g-i). In these trials, at least 77% of images in one of these no-patch clusters required histopathology (biopsy & pathologist referral), while such images made up $<$7% of each other cluster. In other words, the no-patch subclass split into “histopathology” and “non-histopathology” clusters, where the former datapoints were harder for clinicians to classify. We comment on the real-world importance of the ISIC result in particular. Naively, the overall AUROC on ISIC obtained using the ERM model is 0.957, which suggests a high-performing model; however, our clustering reveals that a large fraction of the benign images contain a “spurious” brightly colored patch, which makes them very easy to classify. The model performs substantially worse on examples without such a patch, and worse still on “more difficult” examples for which a clinician also utilized a histopathology examination to make a diagnosis. Thus, if deployed in practice with a target sensitivity value in mind, the appropriate way to set an operating point for this model is in fact cluster-dependent; if a single operating point were set using the aggregate ROC curve, the true sensitivity on the histopathology subclass would be substantially lower than intended. This means that even simply measuring hidden stratification via Step 1 of George can provide crucial information that would help avoid spurious false negatives at test time—the worst type of error a medical screening application can make.

3.3 Estimating Robust Accuracy

We show that the clusters returned by George enable improved measurement of worst-case subclass performance. Specifically, we measure the worst-case performance across any cluster returned by George (which we call the “cluster-robust” performance) and compare this to the true robust performance and the overall performance. We present results for both ERM and George in Table 3. In most cases, the cluster-robust performance is much closer to the true robust performance than the overall performance is. On ISIC, cluster-robust performance even yields a better estimate of robust performance on the histopathology subclass than does performance on the patch/no-patch subclass. By comparing cluster-robust performance to overall performance, we can detect hidden stratification (and estimate its magnitude) without requiring subclass labels.

In addition, improvements in robust performance from George compared to ERM are accompanied by increases in cluster-robust performance; by comparing the cluster-robust performance of ERM and George, we can estimate how much George improves hidden stratification.

4 Effects of Validation Metric

George’s improvement of robust performance has two potential explanations: (1) minimizing the cluster-robust training loss is a better surrogate objective for the true robust performance than minimizing the overall training loss, and (2) selecting the best model checkpoint based on validation cluster-robust performance is better than selecting based on overall validation performance. To decouple these two effects, in Table 8 we display the test robust performance for ERM and George when using the true robust validation performance as a checkpointing metric. This change generally improves performance for both methods, but George still significantly outperforms ERM on all datasets except ISIC. This shows that, for the goal of maximizing robust performance, the GDRO objective with cluster labels indeed performs better than the ERM objective.

In Table 8, we also display the effects of changing the frequencies of subclasses in the validation set. As described in Appendix B.2, by default we re-weight the validation and test sets of U-MNIST and Waterbirds so that the effective frequencies of each subclass are the same as they are in the training set. If we turn off this reweighting, the cluster-robust validation performance is a more accurate measure of the true robust performance, since the frequency of the underperforming subclass increases in the validation set. Using this unreweighted metric to select the best model checkpoint increases the true robust performance of George to 83.3%—an improvement of over 22 points compared to ERM checkpointed against average accuracy on the same unreweighted validation set. We stress that the true validation subclass labels are still assumed to be unknown in this experiment. Having training and validation sets with different distributions is realistic in many situations.

We remark that in typical supervised learning settings, model selection—encompassing both hyperparameter tuning and selection of a checkpoint from the training trajectory—is done with the help of a validation set on which the desired metric of interest can be computed. By contrast, the setting we study in George is more challenging not only due to the absence of training group labels, but also because the absence of validation group labels means that this selection metric (worst-group accuracy / AUROC) cannot even be computed exactly on the validation set. The results of this section are encouraging in that they suggest that George’s cluster-robust performance is an acceptable proxy metric for model selection.Our hyperparameter tuning procedure is described in Section B.3; we also do not use validation set group labels to tune any of George’s hyperparameters.

5 Extension: Leveraging Pretrained Embeddings

As an alternative to training an ERM model, we assess whether recent pretrained image embeddings (BiT ) can provide better features for Step 1 of George. Specifically, we modify Step 1 of George to compute BiT embeddings for the datapoints, cluster the embeddings, and use these cluster assignments as estimated subclass labels in Step 2 of George. This modification (George-BiT) dramatically improves robust accuracy on CelebA to 87.3% ($\pm 1.3\%$), nearly matching subclass-GDRO.Overall accuracy drops somewhat to 91.5%.

The CelebA BiT clusters align much better with the true subclasses (cf. Table 2), which helps explain this improvement. Similarly, cluster-robust accuracy measured using the BiT clusters is much closer to the true robust accuracy: for the George-BiT model, average cluster-robust performance on the BiT clusters is $83.3\pm 1.3\%$, and for ERM it is $33.9\pm 2.5\%$ [compared to the true robust accuracy of 40.3%].

Despite its excellent performance on CelebA, the default George implementation outperforms George-BiT on the other datasets, suggesting that BiT is not a panacea: on these datasets, the task-specific information contained in the representation of the trained ERM model seems to be important for identifying meaningful clusters. See Appendix B.3.6 for additional evaluations and discussion. Extending Step 1 of George to enable automatically selecting between different representations (e.g., BiT vs. ERM) is a compelling future topic.

Conclusion

We propose George, a two-step approach for measuring and mitigating hidden stratification without requiring access to subclass labels. George’s first step, clustering the features of an ERM model, identifies clusters that provide useful approximations of worst-case subclass performance. George’s second step, using these cluster assignments as groups in GDRO, yields significant improvements in worst-case subclass performance. We analyze George in the context of a simple generative model, and show that under suitable assumptions George achieves the same asymptotic sample complexity rates as if we had access to true subclass labels. We empirically validate George on four datasets, and find evidence that it can reduce hidden stratification on real-world machine learning tasks. Interesting directions for future work include further exploring different ways to learn representations for the first stage of George, developing better unsupervised metrics to choose between representations and clustering methods, and characterizing when ERM learns representations that enable separation of subclasses.

Acknowledgments

We thank Arjun Desai, Pang Wei Koh, Shiori Sagawa, Zhaobin Kuang, Karan Goel, Avner May, Esther Rolf, and Yixuan Li for helpful discussions and feedback.

We gratefully acknowledge the support of DARPA under Nos. FA86501827865 (SDH) and FA86501827882 (ASED); NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, Total, the HAI-AWS Cloud Credits for Research program, the Schlumberger Innovation Fellowship program, and members of the Stanford DAWN project: Facebook, Google, VMWare, and Ant Financial. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ONR, or the U.S. Government.

References

Appendix

Appendix A Extended Related Work

Our work builds on several active threads in the machine learning literature.

Our motivating problem is that of hidden stratification, wherein models trained on superclass labels exhibit highly variable performance on unlabeled subclasses . This behavior has been observed in a variety of studies spanning both traditional computer vision and medical machine learning . Of note is the work of , who propose that the existence of “distribution shift” at the subclass level may substantially affect measures of test set performance for image classification models on CIFAR-10. They use a simple mixture model between an “easy” and a “hard” subclass to demonstrate how changes that would not be detectable at the superclass level could affect aggregate performance metrics. extend these ideas by developing notions of visual hardness, and suggest that better loss function design would be useful for improving the performance of machine learning models on harder examples. studies how to automatically find large, interpretable underperforming data slices in structured datasets.

Our approach is also inspired by the literature on causality and machine learning, and in particular by the common assumption that the data provided for both training and evaluation are independent and identically distributed (IID) . This is often untrue in real-world settings; in particular, classes in real-world datasets are often composed of multiple subclasses, and the proportions of these subclasses may change between training and evaluation settings—even if the overall class compositions are the same. Many of the guarantees from statistical learning theory break down in the presence of such non-IID data , suggesting that models trained using traditional Empirical Risk Minimization (ERM) are likely to be vulnerable to hidden stratification. This motivates the use of the maximum (worst-case) per-subclass risk, rather than the overall average risk, as the objective to be optimized.

The first stage of the technique we propose for addressing hidden stratification (George) relies heavily on our ability to identify latent subclasses via unsupervised clustering of neural representations learned via ERM. This has been an area of substantial recent activity in machine learning, and has provided several important conclusions upon which we build in our work. The work of and , for instance, demonstrate the utility of a simple autoencoded representation for performing unsupervised clustering in the feature space of a trained model. While the purpose of these works is often to show that deep clustering can be competitive with semi-supervised learning techniques, the mechanics of clustering in model feature space explored by these works are important for our present study. Indeed, we directly leverage the conclusion of that Uniform Manifold Approximation and Projection (UMAP) works well as a dimensionality reduction technique for deep clustering in the current study.

Further, the fact that work such as directly uses neural representation clustering to estimate the presence of novel classes in a given dataset provides an empirical basis for our approach, which uses a model trained with ERM to approximately identify unlabeled subclasses within each superclass. Similarly, demonstrate excellent semi-supervised image classification performance by maximizing mutual information between the class assignments of each pair of images. Their work demonstrates not only the utility of a clustering-style objective in image classification, but also suggests that overclustering – using more clusters than naturally exist in the data – can be beneficial for clustering deep feature representations in a manner that is helpful for semi-supervised classification.

A related, but different, approach is that of , who explicitly attempt to identify subcategories of classes via a graph and SVM-based “subcategory mining” framework in order to improve overall task performance. The subcategory mining algorithm is quite complicated and uses manually extracted features (rather than automatically learned features, e.g., from CNNs); in addition, this work is geared towards improving overall performance, rather than ensuring good performance on all subcategories. Nevertheless, it is an important piece of prior literature.

The second stage of Georgedepends on our ability to optimize the worst-case classification loss over existing subgroups. This formulation draws a clear connection between our work and the literature on fairness in machine learning , which is at least partially concerned with ensuring that trained models do not disadvantage a particular group in practice. While there exist a wide variety of definitions for algorithmic fairness , the common idea that models should be optimized such that they respect various notions of fairness is similar to the motivation behind our work.

Distributionally robust optimization studies the problem of optimizing for worst-case performance with respect to some “uncertainty set” of distributions. A multitude of recent papers have explored optimizing distributionally robust objectives in slightly different contexts. Most relevant to our work is the study of , who propose the group DRO algorithm for training classifiers with best worst-case subgroup performance (in other words, the “uncertainty sets” in this case are the per-group distributions). Crucially, this algorithm demonstrates improved worst-case subclass performance in cases where triplets ($x,y,g$) are known for every data point, with $x$ is the input data, $y$ is the true label, and $g$ is a true subgroup label. While present preliminary evidence that group DRO can work well in the presence of noisy $g$, the efficacy of the algorithm in this setting remains functionally unexplored. We leverage the group DRO algorithm as an optimizer for minimizing the worst-case loss with respect to our approximately identified subclasses.

We also discuss other works on DRO. , for instance, considers the general problem of optimizing the worst-case loss over any possible subdistribution of the data above a specified size; while conceptually important, the goal of optimizing over arbitrary subdistributions of a minimum size is rather pessimistic compared to assuming more structure on these subdistributions (such as in GDRO). built upon these ideas to design efficient methods for large-scale DRO, but the uncertainty sets considered are also less structured compared to group DRO and therefore generally give poorer results when the goal is in fact to optimize worst-group performance on a specific set of groups. Concurrently with our work, studies how to train models with respect to group-level fairness constraints, given only noisy versions of the groups; they show how to solve this problem efficiently via a DRO formulation and a reweighting approach based on soft group assignments. While their formulation encompasses several fairness objectives, such as “equal opportunity” (equal true positive rates across groups) and “equal outcome” (equal positive prediction rates across groups), it does not directly optimize for worst-group performance. Additionally, they assume that the noisy group labels are provided and their marginal distribution is the same as that of the true groups, whereas we do not assume any foreknowledge of the (noisy or true) groups.

Similar to the goal of optimizing worst-group performance is optimizing for group Pareto fairness, i.e., seeking solutions that are Pareto-efficient in terms of the performances on each group. and both study this (more general) problem in the case where the group labels are known. In addition, follow-ups to these papers that were concurrent to our work explored this problem in the setting where the group labels are unknown ; however, both of these works focus on simpler structured datasets rather than more challenging settings such as image classification.

Other relevant techniques include invariant risk minimization, which attempts to train classifiers that are optimal across data drawn from a mixture of distributions (i.e., a non-IID setting) ; methods from slice-based learning that learn feature representations optimized for ensuring high performance on specific subsets, or “slices” of the data ; mixture-of-experts models, which explicitly handle learning models for multiple different subsets of data ; and techniques for building robust classifiers via domain adaptation . While our work is closely related to these directions, a major difference is that we handle the setting where the different subclasses (i.e., groups, environments, slices, etc.) are unidentified.

A final research thread that is closely related to the work presented here focuses on deep representation learning in the absence of ground truth labels. Our methods are similar in spirit to those from weak supervision , which focuses on training models using noisy labels that are often provided programmatically. Our work can be seen as analyzing a new form of fine-grained weak supervision for DRO-style objectives, which is drawn from unsupervised clustering of an ERM representation. Another related line of work is representation learning for few-shot learning ; however, our work fundamentally differs in the sense that we assume no access to ground truth subclass labels.

Other methods aim to automatically learn classes via an iterative approach. An early work of this type is , which uses iterative clustering and ERM training to learn highly effective feature representations for image classification. More recently, used a self-supervised task to learn semantically meaningful features, and then generate labels using an iteratively refining approach. Our work differs from these in that we do assume access to ground truth superclass labels—which provide much more information than in the fully-unlabeled setting—and use clustering within each superclass to generate approximate labels. In addition, our primary end goal is not accurate identification of the subclasses, but ensuring good worst-case performance among all subclasses.

Finally, other works aim to promote a notion of “diversity” among feature representations by adding different regularizers. In , such a regularizer was introduced in the context of latent space models, to better capture infrequently observed patterns and improve model expressiveness for a given size. More recently, introduced a regularizer that aims to promote diversity of the predicted logits. They showed that this method could also lead to estimation of subclasses within a superclass, without requiring subclass labels. However, this work focused on improving overall performance, and specifically improvement of knowledge distillation; by contrast, our goal is to improve robust performance. Nevertheless, integrating these recent ideas into our work is an interesting avenue for future work, to potentially further improve the feature learning stage.

Appendix B Experimental Details

We provide pseudocode for George in Algorithm 1, to complement the detailed description of our methodology in Section 4.We note that the final step of George—training a model to minimize the maximum per-cluster risk—can also be done when “soft” (probabilistic) cluster labels are given instead of hard assignments; see Appendix D.5. Note that our model class $\mathcal{F}$ (as per the notation in Section 2.2) is a class of neural networks, composed of a “featurizer” module $f_{\theta}$ and a “linear classification head” $L$ that takes the feature representation to a prediction.

Note that our framework is not constrained by the specific choice of clustering algorithm, dimensionality reduction algorithm, or robust optimization algorithm. While we use GDRO throughout this work, other training techniques that encourage good robust performance could be swapped in for GDRO during “Step 2” of George.

B.2 Dataset Details

Below, we describe the datasets used for evaluation in more detail. [We provide PyTorch dataloaders to support each one in our code.]

Each dataset contains labeled subclasses; although the George procedure does not use the subclass labels at any point, we use them to assess how well George (a) can estimate the subclass labels and (b) can estimate and improve worst-case subclass performance.

We remark that while we evaluate on binary classification tasks in this work, George can readily be applied in principle to tasks with any amount of superclasses and subclasses.

Undersampled MNIST (U-MNIST) is a binary dataset that divides data from the standard MNIST dataset (which has 60,000 training points) into two superclasses: numbers less than five, and numbers greater than or equal to five. Crucially, the “8” subclass is subsampled at 5% of its usual frequency in MNIST. The rarity of the “8” subclass makes this task much more challenging than the default MNIST task, in terms of robust performance. We use data drawn from the original MNIST validation set as our test set; we create a separate validation set of 12,000 points by sampling from the MNIST training set, and use the remainder for training.

On the validation (and test) sets, we do not actually undersample the 8’s, as this would leave only 50-60 “8” examples; instead, we downweight these examples when computing validation/test accuracies and losses, to mimic the rarity that would be induced by actually undersampling but still allow for more stable accuracy measurements.

B.2.2 Waterbirds

The Waterbirds dataset (4,795 training points) used in this work was introduced in . Similar to our approach for U-MNIST, Sagawa et al. create more balanced validation and test sets to allow for stable measurements, and downweight the examples from rare subclasses during evaluation (as we do for U-MNIST as well); we follow the same procedure.

B.2.3 ISIC

The dataset from the International Skin Imaging Collaboration (ISIC) website is, at time of writing, comprised of 23,906 images and their corresponding metadata . We extract the ISIC dataset directly from the site’s image archive, which is accessible through a public API.https://isic-archive.com/api/v1/ We only use images whose metadata explicitly describe them as “benign” or “malignant.” We use these descriptors in order to formulate the problem as a binary classification task that classifies images as either normal or abnormal. Other possible descriptors that exist in the image metadata (which we filter out) include “indeterminate,” “indeterminate/benign,” “indeterminate/malignant,” or no description. We created pre-set training, validation, and test splits from these images by randomly assigning 80% of examples to the training set, 10% to the validation set, and 10% to the test set.

We derive true subclass information from the image metadata. In particular, we observed that an image belongs in the benign patch subclass if and only if it is an image from the SONIC data repository . As detailed in Section 6.3, we are retroactively able to identify the histopathology subclass through analysis of the diagnosis confirmation type of each image. Images in the histopathology subclass were explicitly mentioned as such—other possible diagnosis confirmation types include “single image expert consensus,” “serial imaging showing no change,” “confocal microscopy with consensus dermoscopy,” or no confirmation type.

B.2.4 CelebA

The CelebA dataset is a standard face classification dataset containing over 200,000 examples ($\approx 163,000$ train) of celebrity faces, each annotated with 40 different attributes. The images contain a wide variety of poses, backgrounds, and other variations. The task is to classify faces as “blond” or “not blond,” as in . We use the standard (pre-set) train/validation/test splits for this task.

B.3 Methods

For each dataset, we perform ten separate trials of each method with different random seeds. The exception is CelebA, on which we perform five trials instead due to the larger dataset size. In all result tables, ‘$X\pm Y$’ intervals represents a 95% confidence interval, where $X$ is the mean of the per-trial results and $Y$ is the interval half-width, calculated as standard deviation times 1.96 divided by the square root of the number of trials. Similarly, in all plots, error bars denote 95% confidence intervals computed the same way.

B.3.2 Baselines

In addition to ERM, we run two additional baseline methods: superclass-GDRO and random-GDRO. Superclass-GDRO minimizes the maximum loss over each superclass, i.e., runs GDRO using the superclasses as groups. Since we assume knowledge of the training superclass labels, this does not require additional information at training time. Random-GDRO runs GDRO using randomly chosen groups within each superclass, where the groups are chosen to have the same sizes as the true subclasses. Since we do not assume the subclass sizes are known, this is not a method that would be useful in practice; rather, it helps highlight the difference between running GDRO with labels that do not align well with the true subclasses, and running GDRO with labels that do. Results on each dataset are presented in Figure 5.

B.3.3 ERM Training Details

The first stage of George is to train a model for each application using ERM. The activations of the resulting model are clustered and used in the second stage of our procedure. Inspired by the results of , we explored using either a standard ERM model or an ERM model with high regularization for this stage, selecting between the two based on the quality of the resulting clustering as measured by the Silhouette score (an unsupervised metric). Below, we detail the ERM hyperparameter settings for each dataset.

Our U-MNIST model is a simple 4-layer CNN, based on a publicly available LeNet5 implementation;https://github.com/activatedgeek/LeNet-5 based on this implementation, we fix the learning rate at 2e-3 and use the Adam optimizer. Each model is trained for 100 epochs. Because the original implementation does not specify a weight decay, we search over weight decay values of $[10^{-3},10^{-4},10^{-5}]$, and choose the setting with highest average validation accuracy over three trials with different random seeds. Our final hyperparameters are recorded in Table 5.

Our Waterbirds model uses the torchvision implementation of a 50-layer Residual Network (ResNet-50), initialized with pretrained weights from ImageNet (as done in ). For the standard ERM model we use hyperparameters reported by : weight decay of 1e-4, learning rate of 1e-3, SGD with momentum 0.9, and 300 epochs. For the high-regularization ERM model (which ends up being the one used in Stage 1 of George), the weight decay and learning rate are 1.0 and 1e-5 respectively (as done in for the high-regularization model).

Our CelebA model also uses a torchvision pretrained ResNet-50, as done in . We use the hyperparameters reported by : weight decay of 1e-4, learning rate of 1e-4, SGD with momentum 0.9, and 50 epochs. For the high-regularization ERM model (which ends up being the one used in Stage 1 of George), the weight decay and learning rate are 0.1 and 1e-5 respectively (as done in for the high-regularization model). However, we train on 4 GPUs instead of 1. (This change does not substantially affect the results; our ERM and subclass-GDRO results are similar to those reported in .)

Our ISIC model also uses a torchvision pretrained ResNet-50. Models were trained for 20 epochs using SGD with momentum 0.9 (as done in ). Because these hyperparameters were unavailable in the literature for this architecture and task, we grid searched over weight decay values in [0.01, 0.001, 0.0001] and learning rates in [0.0005, 0.001, 0.005], selecting the values that maximize the overall AUROC on the validation set, averaged over three trials per hyperparameter setting.

B.3.4 Clustering Details

We apply a consistent clustering procedure to each dataset, which is designed to encourage discovery of clusters of varied sizes, while still being computationally efficient. We emphasize that while the clustering procedure outlined below yields adequate end-to-end results on our datasets, optimizing this part of the George procedure represents a clear avenue for future work. In particular, we use the Silhouette score as a metric to select between feature representations and number of clusters; while this is a serviceable heuristic, it has several flaws (and in the case of BiT embeddings, misleadingly suggests that they are not a suitable representation due to their low Silhouette score).

Dimensionality Reduction: As recommended by , we use UMAP for dimensionality reduction before clustering; clustering is faster when the data is low-dimensional, and we find that UMAP also typically improves the results. As an alternative to UMAP, we also use the component of the representation that is orthogonal to the decision boundary, which we refer to as the “loss component,” as a single-dimensional representation; this can improve clustering on datasets, especially when performance on certain subclasses is particularly poor (as discussed further in Appendix D.3).We experimented with concatenating the UMAP and loss representations, but found this to reduce performance. When the loss component is used to identify clusters, we find that applying higher regularization to the initial ERM model further improves clustering quality, as this regularization “pushes examples further apart” along the loss direction, and adopt this convention in our experiments.The loss component is used for Waterbirds and CelebA (non-BiT version). For both datasets, the weight decay and learning rate used for the high-regularization ERM model are the same as the ones used for the GDRO models on that dataset.

In each experiment, we select the representation and the number of clusters $k$ based on the parameter setting that achieves the highest average per-cluster Silhouette score. (For all experiments, we set the number of UMAP neighbors to 10 and the minimum distance to 0; further information about these hyperparameters can be found in .)

The fact that simply using the “loss component” can yield reasonable results is arguably surprising, as this essentially amounts to just picking the examples that the original network got wrong (or closer to wrong than others). Nevertheless, especially on tasks with severe data imbalances and “spurious features” (e.g., Waterbirds and CelebA), the rare subclasses do tend to be misclassified at far higher rates, so simply picking the misclassified examples can be a crude but effective heuristic.

Global Clustering: For each superclass, we search over $k\in 2,\dots,10$ to find the clustering that yields the highest average Silhouette score, using the dimensionality reduction procedure identified above. We similarly perform a search over clustering techniques ($k$-means, GMM, etc.), and find that GMM models achieve high average Silhouette scores most often in our applications. Given that GMM clustering also aligns with our theoretical analysis, we use this approach for all datasets. We refer to this global clustering as $f_{C,G}$.

Overclustering: For each superclass, we take the clustering $f_{C,G}$ achieving the highest average Silhouette score, and then split each cluster $c_{i}$ into $F$ sub-clusters $c_{i1},\dots,c_{iF}$, where $F$ denotes the “overclustering factor” (fixed to 5 for all experiments). For each sub-cluster $c_{ij}$ whose Silhouette score exceeds the Silhouette score of the corresponding points in the original clustering, and which contains at least $s_{min}$ points (for a small threshold value $s_{min}$), the global clustering $f_{C,G}$ is updated to include $c_{ij}$ as a new cluster (and its points are removed from the base cluster $c_{i}$). The overclustering factor $F$ was coarsely tuned via visual inspection of clustering outputs (without referencing the true subclass labels); the threshold value $s_{min}$ is used to prevent extremely small clusters, as these can lead to instability when training with GDRO and/or highly variable estimates of validation cluster-robust accuracy. (Note: We do not apply overclustering to 1-dimensional representations, as it tends to create strange within-interval splits.)

B.3.5 Dimensionality Reduction and Clustering: Further Details

Dimensionality reduction for this dataset used 2 UMAP components and no loss component, as UMAP achieved higher SIL scores. Our clustering procedure consistently identifies a cluster with a high proportion of the low-frequency “8” subclass. As detailed in the main body, we also often observe a small additional cluster with a high concentration of “7”s written with crosses through the main vertical bar (see Figure 6); performance on this subset is low (below 90%), which explains why cluster-robust performance actually underestimates the true subclass performance on U-MNIST.

Dimensionality reduction for this dataset used only 1 component (the loss component); this significantly outperformed UMAP both in terms of SIL score and final robust performance. We observe that while our procedure does not yield clusters with absolutely high frequencies of the minority classes (as shown in Table 2), George still identifies clusters with high enough precision (i.e., high enough proportions of the poorly-performing subclasses) such that the second stage of George can substantially improve performance on these subclasses.

Dimensionality reduction for ISIC used 2 UMAP components and no loss component. Patch and non-patch examples lie in different clusters over 99% of the time. Within the non-patch subclass, on most trials, histopathology examples mostly lie in a different cluster from non-histopathology examples. Similarly, the patch examples often further separate into clusters based on the color of the patch (Figure 6).

Despite the fact that clustering reveals the non-patch and histopathology subclasses with fairly high fidelity (as also shown in Table 2), we do not observe significant improvements in performance on either subset. We hypothesize that this is due to these subsets being “inherently harder.” For example, we find that even Subclass-GDRO, which uses the true patch vs. non-patch subclass labels, fails to significantly improve performance on the non-patch subclass compared to ERM, and in fact fails to significantly reduce the training loss on it compared to ERM despite being explicitly trained to do so. This suggests that the issue causing underperformance on these subsets may be due to other factors than the training optimization algorithm (such as model capacity).

Dimensionality reduction for CelebA (without BiT) used only 1 component (the loss component). We observe that clustering does not do a good job of identifying the subclasses of either superclass; thus, it is not surprising that the default version George (i.e., without BiT) performs poorly. In fact, George performs poorly even compared to the non-ERM baselines. By contrast, George-BiT does significantly better; the clustering on the (nearly balanced) non-blond superclass attains approximately 95% accuracy at distinguishing between men and women, and the clustering on the blond superclass also significantly improves over the default version of George.

B.3.6 BiT Details

As an alternative to representations from a trained ERM model, we explore the use of BiT embeddings , as discussed in Section 6.5. We use the ResNet-50 version of BiT embeddings; specifically, BiT embeddings are the activations of the penultimate layer of a network pretrained on massive quantities of image data (see for more details). The remainder of George proceeds the same as usual: the embeddings are clustered and then the cluster assignments are used in the GDRO objective.

For BiT, we experimented with both clustering the BiT embeddings directly (under the hypothesis that the BiT embedding space itself is a good representation), and clustering after dimensionality reduction with UMAP. We found clustering raw embeddings generally performed somewhat better; thus, we show results for clustering the raw embeddings. Due to the high dimensionality of these embeddings (2048-d), we use $k$-means clustering when clustering the BiT embeddings, although the rest of our procedure remains the same.

We find that BiT embeddings significantly improve the end-to-end robust performance results on CelebA; however, they perform worse than the standard version of George on all other datasets, indicating that the task-specific information is important for these other tasks to learn a “good” representation that can be clustered to find superclasses. Indeed, we find that on these other tasks, the BiT clustering is worse than clustering the activations of the ERM model, in terms of precision and recall at identifying poorly-performing subclasses. [For example, when BiT embeddings are used on MNIST, the “8”s are never identified as their own cluster.]

Surprisingly, the clustered BiT embeddings uniformly have a much lower Silhouette score than the clustered ERM embeddings, even for CelebA. Thus, our current unsupervised representation and clustering selection technique would not have identified the BiT embeddings as better for CelebA. Improving the representation and clustering selection metric to do a better job at automatically choosing among different representations is an interesting avenue for future work. We note that if a small validation set with subclass labels is available, such a set could be used to select between different clusterings by measuring the degree of overlap of the clusters with the true subclasses, as well as used to measure which representation and clustering technique eventually leads to the best validation robust accuracy; however, in general we do not assume any prior knowledge about the subclasses in this work.

B.3.7 GDRO Training Details

In the final step of George, we train a new model (with the same architecture) using the group DRO approach of with weak subclass labels provided by our cluster assignments, and compare to GDRO models trained using (a) superclass labels only (b) random subclass labels and (c) human-annotated subclass labels. Below, we describe the hyperparameter search procedure for each such model and dataset. Unless otherwise stated, all other hyperparameters (batch size, momentum, # epochs, etc.) are the same as those for ERM.

In the case of U-MNIST, we ran a hyperparameter search over weight decay in [1e-3, 1e-4, 1e-5], and $C$ (the group size adjustment parameter from ) in . We find performance to be fairly insensitive to the hyperparameters, so choose weight decay of 1e-5 and $C=0$ for simplicity and consistency with ERM.

For Waterbirds, we use hyperparameters provided by , so no additional hyperparameter tuning is required. These hyperparameters are presented in Table 5.

For CelebA, we again use hyperparameters provided by , so no additional tuning is required. These are presented in Table 5.

Each type of ISIC model is hyperparameter searched over the same space as the original ERM model, in addition to searching over group size adjustment parameter $C$ in . We found performance to be fairly insensitive to both. Hyperparameters with highest validation performance were used in the final runs, and are reported in Table 5.

B.4 Hyperparameters

In Table 5, we present the selected hyperparameters for the final runs of each dataset and method.

Appendix C Additional Experimental Results

In this section, we provide additional ablation experiments.

In Figures 6 and 7, we visualize the representations returned by George, as well as the clusters it finds and representative examples from each cluster.

C.2 Comparing Cluster-Robust Performance and True Robust Performance

In addition to the results of Table 3 which show that the cluster-robust performance is a good approximation for the true robust performance, we find that the cluster-robust performance typically tracks closely with the true robust performance throughout training (with the exception of CelebA without BiT clusters). For example, Figure 8 plots the validation cluster-robust accuracy and validation true robust accuracy from a randomly selected training run on Waterbirds. Both metrics are quite close to each other throughout training (while the overall accuracy is significantly higher).

C.3 Runtime

In Table 6, we present runtimes for the standard version of George broken down by stage.

As the default implementation of George involves first training an ERM model, dimensionality-reducing and then clustering its activations, and then training a “robust” model, the total runtime is roughly 2-3$\times$ long as that of simply training an ERM model. For George-BiT, no ERM model is trained (and we do not apply dimensionality reduction), so the runtime is just the runtime of the clustering stage plus the runtime of training the GDRO model (“Step 2”). On our datasets, the total runtime of George-BiT is less than 1.5x times the runtime of George. For instance, on CelebA with BiT embeddings (the largest and most expensive dataset), the entire clustering stage (including the time taken to compute the BiT embeddings of the datapoints) takes 46 minutes, while the time taken to train the ERM model is roughly 2.5 hours. (Clustering the BiT embeddings is more expensive because they are 2048-dimensional.)

Note that the runtime of typical clustering algorithms scales superlinearly in the number of datapoints; while the clustering runtime is usually less than the training time for the datasets we evaluate on, a remedy for larger datasets could be to only use a random subset of the data for clustering (which typically does not significantly worsen the cluster quality). In addition, we did not attempt to optimize the dimensionality reduction and clustering routines themselves. As we search over $k$ from 2 to 10 for each superclass, and then overcluster, this is 20 different clusterings in total, along with computing the Silhouette score for each one (which is also expensive as it involves computing pairwise distances). If we instead fixed $k$ (for instance), the total clustering runtime would be less than 7 minutes even for CelebA with BiT embeddings.

The runtime of George can be substantially reduced by training the second (robust) model for fewer epochs. On Waterbirds and CelebA, we can recover over 70% of the worst-case performance improvement of George even when we limit the total runtime to 1.3$\times$ that of ERM, simply by training for fewer epochs in the second stage. On U-MNIST, if we additionally adjust the LR decay schedule so that decay occurs before the end of the shortened training, and fix $k$ to 5 to avoid the expensive search over $k$ as described above, we can achieve this as well. (On ISIC, the ERM model itself already attains nearly the same robust AUROC on the histopathology subclass, and higher on the non-patch subclass, than the George model.)

C.4 Label Noise

We ran experiments in which a fixed percentage of the data of each subclass was randomly given an incorrect superclass label. With a minor modification (discarding small clusters), George empirically works well in the presence of label noise when the total number of corrupted labels in each superclass is less than the size of the smallest subclass. Up to this noise threshold, George attains +3 points robust accuracy on MNIST and +4 points robust AUROC on ISIC compared to ERM. However, ensuring subgroup-level robustness if there is a larger group of “wrong” examples is difficult because differentiating “real” subclasses from noise becomes challenging. Thus, we do not consider applying label noise to Waterbirds as the smallest subclass (water-birds on land) is only 1% of the data; similarly, the smallest subclass on CelebA (blond males) is only 3% of the data.

In fact, our clustering approach can even be used to help identify incorrectly labeled training examples. First, if a small “gold” set of correctly labeled examples is available, the clustering found on the training data could be evaluated on this gold set; clusters consisting of mostly incorrectly labeled training examples should have very few members in the gold set. If such a “gold” set is not available, the clusters still allow for much more rapid inspection of the data for incorrect labels, since a few representative examples from each cluster can be inspected instead of a brute-force search through all the training images for incorrectly labeled images. Finally, if one has prior knowledge of the frequency of the rarest subclass in the training data, one can simply discard training examples belonging to poorly-performing clusters smaller than this threshold, treating them as incorrectly labeled.

C.5 Fixing k𝑘k

If the number of clusters $k$ is held fixed (rather than automatically chosen based on Silhouette score), robust performance tends to initially improve with $k$, before decreasing as large values of $k$ cause fragmented clusters that are less meaningful. For example, robust accuracies on U-MNIST using 2, 5, 10, 25, and 100 clusters per superclass are 95.0%, 96.3%, 95.9%, 94.4%, 90.8% respectively. We also observe similar trends on the other datasets.

C.6 Effect of Model Choice on Subclass Recovery

As suggested in Section 4, choosing an appropriate model class $\mathcal{F}$ for the featurizer $f_{\theta}$ is important. In particular, $\mathcal{F}$ should ideally contain the inverse of the true generative function $g$, in order to recover the latent features $\vec{V}$ from the data $X$. We demonstrate the importance of model architecture on the ability to separate subclasses in the model feature space by comparing the feature representations of two simple networks on a superclass of the U-MNIST dataset (described in Section 6.1). Figure 9 shows that the choice of model family can strongly affect the learned feature representation of the initial model and its ability to provide useful information about the subclass. On this dataset, the feature space of a simple fully connected network (Figure 9a) yields substantially less separation between the known subclasses than does that of a simple convolutional network (Figure 9b), which displays clusters that clearly correspond to semantically meaningful subclasses.

C.7 Additional Classification Metrics

In Table 7, we compare George and ERM in terms of both per-subclass averaged accuracy (SCAA) and average precision on the test set. As expected, George slightly decreases average precision, as it trades off some average-case performance for better worst-case performance, and George typically increases per-subclass averaged accuracy (except on U-MNIST, where there is a very slight decrease), due to the fact that it significantly improves performance on poorly-performing subclasses while only slightly decreasing performance on other subclasses.

In Table 8, to complement Table 8 we report the robust test accuracies and average test accuracies for models trained with ERM or George, but where the model checkpoint is selected using the true validation robust accuracy.

C.8 Empirical Validation of Lemma 1

Appendix D Derivations and Proofs

We restate Example 3.1 from Section 3.2 below:

The binary attribute vector $\vec{Z}$ has dimension 2, i.e., $\vec{Z}=(Z_{1},Z_{2})$, while only $Z_{2}$ determines the superclass label $Y$, i.e., $Y=Z_{2}$. The latent attribute $Z_{1}$ induces two subclasses in each superclass, each distributed as a different Gaussian in feature space, with mixture proportions $\alpha$ and $1-\alpha$ respectively. For linear models with regularized logistic loss, as the proportion $\alpha$ of the rare subclasses goes to , the worst-case subclass accuracy of ERM is only $O(\alpha)$, while that of GDRO is $1-O(\alpha)$.

Specifically, we consider the following distribution setup: \vec{Z}\,{\in}\,\{\scalebox{0.75}[1.0]{-}1,+1\}^{2}, with {p(\vec{Z}=(\scalebox{0.75}[1.0]{-}1,\scalebox{0.75}[1.0]{-}1))}={p(\vec{Z}=(+1,+1))}=\tfrac{1-\alpha}{2}, p(\vec{Z}=(\scalebox{0.75}[1.0]{-}1,+1))=p(\vec{Z}=(+1,\scalebox{0.75}[1.0]{-}1))=\alpha/2, and $p(V_{1}|Z_{1})=\mathcal{N}(4Z_{1},\alpha^{2})$, $p(V_{2}|Z_{1},Z_{2})=\mathcal{N}(Z_{1}+3Z_{2},\alpha^{2})$, and the label $Y=h(Z_{1},Z_{2})$ simply equals $Z_{2}$. We assume the observed data $X=(V_{1},V_{2})$, i.e., the observed data is the same as the “underlying features” $\vec{V}$.

Thus, the superclass $Y=-1$ is made up of a “big” subclass with distribution $\mathcal{N}((-4,-4),\alpha^{2}\textbf{I})$ and relative mixture weight $1-\alpha$ [corresponding to $\vec{Z}=(-1,-1)$], and a “small” subclass with distribution $\mathcal{N}((+4,-2),\alpha^{2}\textbf{I})$ and relative mixture weight $\alpha$ [corresponding to $\vec{Z}=(+1,-1)$], where I denotes the $2\times 2$ identity matrix. The superclass $Y=+1$ is made up of a “big” subclass with distribution $\mathcal{N}((+4,+4),\alpha^{2}\textbf{I})$ and relative mixture weight $1-\alpha$ [corresponding to $\vec{Z}=(+1,+1)$], and a “small” subclass with distribution $\mathcal{N}((-4,+2),\alpha^{2}\textbf{I})$ and relative mixture weight $\alpha$ [corresponding to $\vec{Z}=(-1,+1)$].

For notational simplicity in the following analysis, we will henceforth rename the label $Y=-1$ as $Y=0$. The prediction of the logistic regression model on a given sample $(x_{1},x_{2})$ is $\sigma(w_{1}x_{1}+w_{2}x_{2})=\sigma(w^{T}x)$, where $\sigma(x):=\log(\tfrac{1}{1+e^{-x}})$ denotes the sigmoid function and $w_{1},w_{2}$ are the weights of the model. The decision boundary is the line $w^{T}x=0$; examples with $w^{T}x<0$ are classified as $Y=0$, else they are classified as $Y=1$. [For simplicity of exposition, we assume there is no bias term, and assume that we regularize the norm of the classifier so that $\left\lVert w\right\rVert_{2}\leq R$ for some constant $R$, as changing the parameter norm does not change the decision boundary. Note that neither assumption is necessary, but they serve to simplify the analysis.]

By 1-Lipschitz continuity of the logistic loss and Jensen’s inequality,

Similarly, the loss on the $Z_{1}=-1$ subclass is bounded in the range $\log(1+e^{-w^{T}(-4,2)})\pm\alpha\sqrt{2/\pi}\cdot\left\lVert w\right\rVert_{2}$. So, the total loss is bounded in $(1-\alpha)\log(1+e^{-w^{T}(4,4)})+\alpha\log(1+e^{-w^{T}(-4,2)})\pm\alpha\sqrt{2/\pi}\cdot\left\lVert w\right\rVert_{2}$. When $\alpha$ is sufficiently small, the first term is $\Theta(1)$, while the latter two are $O(\alpha)$ (under the assumption that $\left\lVert w\right\rVert_{2}$ is bounded). For a fixed value of $\left\lVert w\right\rVert_{2}$, the first term is minimized when $w/\left\lVert w\right\rVert_{2}=(\tfrac{1}{\sqrt{2}},\tfrac{1}{\sqrt{2}})$, so that $w^{T}(4,4)$ is as large as possible. A $\Theta(\alpha)$-scale perturbation to the direction $w/\left\lVert w\right\rVert_{2}$ results in an increase of $\Theta(\alpha)$ to the quantity $(1-\alpha)\log(1+e^{-w^{T}(4,4)})$. Thus, whenever $\alpha$ is sufficiently small, $w/\left\lVert w\right\rVert_{2}$ must be $(\tfrac{1}{\sqrt{2}},\tfrac{1}{\sqrt{2}})+O(\alpha)$ in order to minimize the loss subject to the $\left\lVert w\right\rVert_{2}\leq R$ constraint. In other words, the regularized ERM solution converges to $(w_{1},w_{2})=(\tfrac{1}{\sqrt{2}},\tfrac{1}{\sqrt{2}})$ as $\alpha\downarrow 0$.

For the $Z_{1}=-1$ subclass, $w^{T}x$ is a normal random variable with mean $-4w_{1}+2w_{2}$ and variance $\alpha\left\lVert w\right\rVert_{2}^{2}$. When $\alpha$ is sufficiently small and $w/\left\lVert w\right\rVert_{2}=(\tfrac{1}{\sqrt{2}},\tfrac{1}{\sqrt{2}})+O(\alpha)$, the quantity $-4w_{1}+2w_{2}$ is negative with magnitude $O(1)$—and thus, since examples with $w^{T}x<0$ are classified as $Y=0$, this means that for sufficiently small $\alpha$ the fraction of the subclass $Z_{1}=-1$ classified correctly as $Y=1$ is only $O(\alpha)$.

By contrast, the GDRO solution minimizes the maximum per-subclass loss. Since each subclass has the same covariance $\alpha^{2}\textbf{I}$, the GDRO decision boundary is the line that separates the superclass means and has maximum distance to any subclass mean. After normalization to have $\left\lVert w\right\rVert_{2}=1$, this is the line $(-\tfrac{1}{\sqrt{5}},\tfrac{4}{\sqrt{5}})$; the true solution will be some multiple of this (depending on $\alpha$ and $R$), giving rise to the same boundary. As $\alpha\downarrow 0$, the accuracy of this decision boundary is $1-O(\alpha)$, since the variance of each subclass is $O(\alpha^{2}\textbf{I})$. ∎

D.2 Proofs from Section 5

where $\pi_{\min}$ denotes the minimum true subclass proportion (i.e., $\pi_{\min}=\min_{c}p(z=c)$).

D.2.2 Proof of Theorem 1

First, we show that within each superclass, if the per-subclass distributions are estimated well, then the per-subclass estimated risks will be close to the true per-subclass risks.

First, we bound $|\hat{w}(x,c)-w(x,c)|$ using the triangle inequality:

By definition, $\hat{p}(x|y=b)={\sum\limits_{c\in S_{b}}}\hat{p}(z=c|y=b)\hat{p}(x|z=c)$, so $\hat{p}(x|z=c)\leq\dfrac{\hat{p}(x|y=S(c))}{\hat{p}(z=c|y=S(c))}$. Thus,

as by definition $\hat{p}(z=c|y=S(c))\geq\hat{p}(z=c)\geq\hat{\pi}_{\min}$. So,

Total variation in estimated per-subclass distributions: Gaussian case

In order to apply Lemma 2, we first need to relate the total variation error $\epsilon$ between the mixtures to the total variation error between the individual mixture components; we show that when $\epsilon$ is small enough, then the total variation error between corresponding mixture components is ${O}(\epsilon)$ as well. We state this formally in Lemma 3 (proved later in this section).

Let $\mathcal{P}$ and $\hat{\mathcal{P}}$ be two $k$-component Gaussian mixtures, and suppose the $k$ components of $\mathcal{P}$, denoted by $p_{1},\dots,p_{k}$, are distinct Gaussian distributions and all have nonzero mixture weights $m_{1},\dots,m_{k}$. Similarly denote the $k$ components of $\hat{\mathcal{P}}$ by $\hat{p}_{1},\dots,\hat{p}_{k}$, with mixture weights $\hat{m}_{1},\dots,\hat{m}_{k}$. There exists a constant $c(\mathcal{P})$ depending only on the parameters of $\mathcal{P}$ such that for all sufficiently small $\epsilon>0$, whenever $TV(\mathcal{P},\hat{\mathcal{P}})\leq\epsilon$ there exists some permutation $P:[k]\rightarrow[k]$ such that $\max\limits_{c\in[k]}\,TV(p_{c},\hat{p}_{P(c)})\leq c(\mathcal{P})\cdot\epsilon$.

In addition, we use the following standard result from learning theory to relate the minimizer of the estimated robust training loss $\hat{R}_{\text{robust}}$ to the minimizer of the true robust training loss $R_{\text{robust}}$.

Theorem 1 Proof

Using the preceding lemmas, we will now prove Theorem 1.

Note that a technical requirement of the above argument is that the samples we use to estimate $\hat{\mathcal{P}}$ should be independent from those we use to compute the robust loss; for this to hold, we may randomly sample half of the examples to learn the distribution $\hat{\mathcal{P}}$ (and its mixture components), and then use the other half to minimize the robust loss. This does not change the asymptotic dependence on the number of samples $n$. [In practice, however, we use all examples in both phases, to get the most out of the data.] ∎

Lemma 3 Proof

Before we prove Lemma 3, we first provide a simple lemma bounding the total variation distance of two Gaussians in terms of the Euclidean distance between their parameters, directly based on the results from .

The one-dimensional case is shown in Theorem 1.3 of . The higher-dimensional case follows from Theorems 1.1 and 1.2 of . Note that the constant $c$ does not depend on $\epsilon$, although it may depend on $d$. ∎

For convenience, we restate Lemma 3 below. See 3

Note that, since we assumed the mixture components are unique and $m_{i}\neq 0$ for all $i$, the only global minima of $q$ [where $q$ evaluates to , which means that $\mathcal{P}$ and $\hat{\mathcal{P}}$ are the same distribution] are where $(m_{\pi(i)}^{\prime},\mu_{\pi(i)}^{\prime},\mathbf{\Sigma}_{\pi(i)}^{\prime})=(m_{i},\mu_{i},\mathbf{\Sigma}_{i})$ for all $1\leq i\leq k$, for some permutation $\pi$—in other words, when the two distributions have the exact same mixture components and mixture weights up to permutation. Note that $q$ is continuous on its domain. Further, it is not hard to see that the $\epsilon$-sublevel sets of $q$ are compact for sufficiently small $\epsilon$, and therefore $\lim\limits_{\epsilon\rightarrow 0}\{({m}_{1}^{\prime},\dots,{\mu}_{1}^{\prime},\dots,{\mathbf{\Sigma}}_{1}^{\prime},\dots):q({m}_{1}^{\prime},\dots,{\mu}_{1}^{\prime},\dots,{\mathbf{\Sigma}}_{1}^{\prime},\dots)\leq\epsilon\}$ is exactly the set of global minima of $q$. Thus, for a fixed distribution $\mathcal{P}$, as $\epsilon\rightarrow 0$, the set of points such that $q({m}_{1}^{\prime},\dots,{\mu}_{1}^{\prime},\dots,{\mathbf{\Sigma}}_{1}^{\prime},\dots)\leq\epsilon$ is contained in the union of sets of $\infty$-norm radius $O(\delta(\epsilon))$ around each of the global minima of $q$, where $\delta(\epsilon)\rightarrow 0$ as $\epsilon\rightarrow 0$. In other words, when $\epsilon$ is sufficiently small then the set of all Gaussian mixtures $\mathcal{P}^{\prime}$ with $TV(\mathcal{P},\mathcal{P}^{\prime})\leq\epsilon$ is the set of all mixtures $\mathcal{P}^{\prime}$ whose parameters $\{m_{i}^{\prime},\mu_{i}^{\prime},\mathbf{\Sigma}_{i}^{\prime}\}$ are $O(\delta(\epsilon))$-close to those of the true distribution $\mathcal{P}$, up to permutation. In particular, if $TV(\mathcal{P},\mathcal{P}^{\prime})\leq\epsilon$, then for each individual Gaussian component $\mathcal{N}_{\mu_{i},\mathbf{\Sigma}_{i}}$ in $\mathcal{P}$, there exists a component $\mathcal{N}_{\mu_{j}^{\prime},\mathbf{\Sigma}_{j}^{\prime}}$ in $\mathcal{P}^{\prime}$ whose parameters are $O(\delta(\epsilon))$-close to it, i.e., $\max\left\{|m_{i}-m_{j}|,\left\lVert\mu_{i}-\mu_{j}^{\prime}\right\rVert_{\infty},\left\lVert\mathbf{\Sigma}_{i}^{\prime}-\mathbf{\Sigma}_{j}^{\prime}\right\rVert_{\infty}\right\}\leq O(\delta(\epsilon))$.

We now argue that $\lim\limits_{\epsilon\rightarrow 0}\tfrac{\delta(\epsilon)}{\epsilon}$ must be a constant (i.e., that $\delta(\epsilon)$ is $\Theta(\epsilon)$ as $\epsilon\rightarrow 0$) in order for the total variation between the two mixtures to be $\leq\epsilon$. We do so by Taylor expanding a set of quantities whose magnitudes lower bound the total variation between $\mathcal{P}$ and $\mathcal{P}^{\prime}$, and showing that these quantities are locally linear in the parameter differences between $\mathcal{P}^{\prime}$ and $\mathcal{P}$ when these differences are sufficiently small.

Assume $TV(\mathcal{P},\mathcal{P}^{\prime})\leq\epsilon$, so as argued before $\left\lVert\vec{v}^{\prime}-\pi(\vec{v})\right\rVert_{2}$ is $O(\delta(\epsilon))$ for some permutation $\pi$ and some function $\delta$ with $\lim\limits_{x\rightarrow 0}\delta(x)=0$. [More precisely, $\lim\limits_{\epsilon\rightarrow 0}\,\,\max\limits_{\vec{v}^{\prime}:TV(\mathcal{P},\mathcal{P}^{\prime})\leq\epsilon}\,\,\min\limits_{\pi:\pi(k)}\left\lVert\vec{v}^{\prime}-\pi(\vec{v})\right\rVert_{2}=0$.] Without loss of generality, we will henceforth simply write $\vec{v}$ in place of $\pi(\vec{v})$. For notational simplicity, let’s write $\delta:=\left\lVert\vec{v}^{\prime}-\vec{v}\right\rVert_{2}$. As $h$ is smooth around $\vec{v}$, we can Taylor expand $h$ about the point $\vec{v}=\vec{v}^{\prime}$ [i.e., $\{m_{1}^{\prime}=m_{1},...,\mu_{1}^{\prime}=\mu_{1},...,\sigma_{1}^{2\prime}=\sigma_{1}^{2},...,\sigma_{k}^{2\prime}=\sigma_{k}^{2}\}$] to get

Define $f_{i}(x)=\tfrac{1}{2}\text{erfc}\left(\tfrac{\mu_{i}-x}{\sqrt{2}\sigma_{i}}\right)$ for $1\leq i\leq k$, $-\tfrac{m_{i}e^{-(x-\mu_{i})^{2}/(2\sigma_{i}^{2})}}{\sqrt{2\pi}\sigma_{i}}$ for $k+1\leq i\leq 2k$, and $\tfrac{m_{i}e^{-(x-\mu_{i})^{2}/(2\sigma_{i}^{2})}(\mu_{i}-x)}{2\sqrt{2\pi}\sigma_{i}^{3}}$ for $2k+1\leq i\leq 3k$. So $\nabla h(\vec{v};c_{j})^{T}=(f_{1}(c_{j}),\dots,f_{3k}(c_{j}))^{T}$.

So, $\left\lVert\delta\right\rVert_{\infty}-O(\left\lVert\delta\right\rVert_{\infty}^{2})\leq 2\epsilon\left\lVert\mathbf{A}^{-1}\right\rVert_{\infty}$, and thus $\left\lVert\delta\right\rVert_{\infty}$ [which is, by definition, the maximum error in any parameter $m_{1},\dots,m_{k},\mu_{1},\dots,\mu_{k},\sigma_{1}^{2},\dots,\sigma_{k}^{2}$ up to permutation] is $O(\epsilon)$. But then, the total variation between each pair of mixture components $\mathcal{N}_{\mu_{i},\sigma_{i}^{2}}$ and $\mathcal{N}_{\mu_{i}^{\prime},\sigma_{i}^{2\prime}}$ is also $O(\epsilon)$, by Lemma 5 and norm equivalence.

Thus, when $d=1$, if the total variation between the two Gaussian mixtures $\mathcal{P}$ and $\hat{\mathcal{P}}$ is $O(\epsilon)$, the total variation between each mixture component must also be $O(\epsilon)$ [where the big-O notation suppresses all parameters that depend on the true distribution $\mathcal{P}$], as desired. (Recall that total variation is always in $$.)

Now suppose $d>1$. Similarly to before, we have

Once again, this set of $k+kd+kd(d+1)/2$ partial derivatives, considered as functions of $\vec{c}_{j}$, comprise a linearly independent set of functions, since the $(\mu_{i},\mathbf{\Sigma}_{i})$ pairs are unique. The remainder of the proof proceeds analogously to the $d=1$ case. ∎

D.3 Subclass Performance Gaps Enable Distinguishing Between Subclasses

In this section, we give simple intuition for why a performance gap between two subclasses of a superclass implies that it is possible to discriminate between the two subclasses in feature space to a certain extent.

Suppose the setting is binary classification, and one of the superclasses has two subclasses with equal proportions in the dataset. Suppose we have access to a model whose training accuracy on one subclass is $x$, while its training accuracy on the other subclass is $y$, where $1\geq x>y\geq 0$.

Of the correctly classified examples, $\tfrac{x}{x+y}>\tfrac{1}{2}$ fraction of them are from the first subclass; similarly, of the incorrectly classified examples, $\tfrac{1-y}{2-x-y}>\tfrac{1}{2}$ fraction of them are from the second subclass.

This means that if we form “proxy subclasses” by simply splitting the superclass into the correctly classified training examples and incorrectly classified training examples, the resulting groups can in fact be a good approximation of the true subclasses! This is illustrated in Figure 11. For instance, suppose $x=0.9$ and $y=0.6$. Then $\tfrac{x}{x+y}=0.6$ and $\tfrac{1-y}{2-x-y}=0.8$ - so, 60% of the examples in the first group are from subclass 1, and 80% of those in the second group are from subclass 2, which is much better than randomly guessing the true subclasses (in which the concentration of each subclass in each guessed group will approach 50% as $n\rightarrow\infty$). In the extreme case, if one subclass has accuracy $1$ and the other has accuracy , then the superclass decision boundary separates them perfectly (no matter their proportions).

Combined with other information, this helps explain why looking at the way each example is classified (such as the loss of the example or related error metrics) can be helpful to discriminate between the subclasses.

D.4 Inherent Hardness

We define the “inherent hardness” of a (task, function class) pair as the minimum attainable robust error, i.e.,

where the function class is denoted by $\mathcal{F}$. (This can be thought of as the “Bayes robust risk.”) We allow the function $f$ to be stochastic: i.e., for a given input $x$, it may output a fixed probability distribution over the possible labels, in which case we define $\mathbf{1}(f(x)=y)$ as the probability assigned by $f$ to the label $y$, given input $x$. By definition, the inherent hardness lower bounds the robust error attained by any classifier in $\mathcal{F}$, regardless of how it is trained or how much data is available. The only way to improve robust performance is therefore to either make the model class $\mathcal{F}$ more expressive (i.e., include more functions in $\mathcal{F}$) or to collect new data such that the covariates $x$ include more information that can be used to distinguish between different classes. (Of course, both of these changes would be expected to improve overall performance as well, if sufficient data is available.) Thus, addressing hidden stratification effects caused by “inherent hardness” is beyond the scope of this work. A simple example of an “inherently hard” task (i.e., a task with nonzero “inherent hardness”) is shown in Figure 12; no classifier can get perfect accuracy on every subclass, because the two superclasses overlap and thus it is impossible to distinguish between them in the region of overlap. Nevertheless, it is possible to attain perfect accuracy on some subclasses in this example, meaning that there will still be performance gaps between the subclasses.

D.5 GDRO with Soft Group Assignments

In preliminary experiments, we found hard clustering to work better than the “soft clustering” approach described in this section; as it also has the advantage of simplicity, all final experiments were performed with hard clustering.