M2m: Imbalanced Classification via Major-to-minor Translation

Introduction

The recent success of deep neural networks (DNNs) across various computer vision problems has emerged due to the access to large-scale, annotated datasets collected from our visual world . Despite having several well-organized datasets in research, e.g., CIFAR and ILSVRC , real-world datasets usually suffer from its expensive data acquisition process and the labeling cost. This commonly leads a dataset to have a “long-tailed” label distribution . Such class-imbalanced datasets make the standard training of DNN harder to generalize , particularly if one requires a class-balanced performance metric for a practical reason.

A natural approach in an attempt to bypass this class-imbalance problem is to re-balance the training objective artificially with respect to the class-wise sample sizes. Two of such methods are representative: (a) re-weighting the given loss function by a factor inversely proportional to the sample frequency in a class-wise manner , and (b) re-sampling the given dataset so that the expected sampling distribution during training can be balanced, either by “over-sampling” the minority classes or “under-sampling” the majority classes .

However, naïvely re-balancing the objective usually results in harsh over-fitting to minority classes, since they cannot handle, the lack of minority information in essence. Several attempts have been made to alleviate this issue: Cui et al. proposed the concept of “effective number” of samples as alternative weights in the re-weighting method. Cao et al. found that both re-weighting and re-sampling can be much more effective when applied at the later stage of training, in case of neural networks. In the context of re-sampling, SMOTE is a widely-used variant of the over-sampling method that mitigates the over-fitting via data augmentation, and several variants of SMOTE have been suggested accordingly . A major drawback of these SMOTE-based methods is that they usually perform poorly when there exist only a few samples in the minority classes, i.e., under regime of “extreme” imbalance, because they synthesize a new minority sample only using the existing samples of the same class.

Another line of research attempts to prevent the over-fitting with a new regularization scheme that minority classes are more penalized, where the margin-based approaches generally suit well as a form of data-dependent regularizer . There have also been works that view the class-imbalance problem in the framework of active learning or meta-learning .

In this paper, we revisit the over-sampling framework and propose a new way of generating minority samples, coined Major-to-minor Translation (M2m). In contrast to other over-sampling methods, e.g., SMOTE that applies data augmentation to minority samples to mitigate the over-fitting issue, we attempt to generate minority samples in a completely different way. The proposed M2m does not use the existing minority samples for the over-sampling. Instead, it use the majority (non-minority) samples and translate them to the target minority class using another classifier independently trained under the given imbalanced dataset. Our key finding is that, this method turns out to be very effective on learning more generalizable features in imbalanced learning: it does not overly use the minority samples, and leverages the richer information of the majority samples simultaneously.

Our minority over-sampling method consists of three components to improve the sampling quality. First, we propose an optimization objective for generating synthetic samples: a majority sample can be translated into a synthetic minority sample via optimizing it, while not affecting the performance of the majority class (even the sample is labeled to the minority class). Second, we design a sample rejection criterion based on the observation that generation from more majority class is more preferable. Third, based on the proposed rejection criterion, we suggest an optimal distribution for sampling a majority seed to be translated in our generation process.

We evaluate our method on various imbalanced classification problems, covering synthetically imbalanced datasets from CIFAR-10/100 and ImageNet , and real-world imbalanced datasets including CelebA , SUN397 , Twitter and Reuters datasets. Despite its simplicity, our method significantly improves the balanced test accuracy compared to previous re-sampling or re-weighting methods across all the tested datasets. Our results even surpass those from LDAM , a current state-of-art margin-based method. Moreover, we found our method is particularly effective under “extreme” imbalance: in the case of Reuters of the most severe imbalance, we could improve the balanced accuracy by (relatively) ${17.1\%}$ and ${9.2\%}$ upon standard training and LDAM, respectively.

M2m: Major-to-minor translation

Our method is primarily based on over-sampling technique , a traditional and principled way to balance the class-imbalanced training objective via sampling minority classes more frequently. In other words, we assume a “virtually balanced” training dataset $\mathcal{D}_{\tt bal}$ made from $\mathcal{D}$ such that the class $k$ has $N_{1}-N_{k}$ more samples, and the classifier $f$ is trained on $\mathcal{D}_{\tt bal}$ instead of $\mathcal{D}$.

A key challenge in over-sampling is to prevent over-fitting on minority classes, as the objective modified is essentially much biased to a few samples of minority classes. In contrast to most prior works that focus on performing data augmentation directly on minority samples to mitigate this issue , we attempt to augment minority samples in a completely different way: our method does not use the minority samples for the augmentation, but the majority samples.

Consider a scenario of training a neural network $f$ on a class-imbalanced dataset $\mathcal{D}$. The proposed Major-to-minor Translation (M2m) attempts to construct a new balanced dataset $\mathcal{D}_{\tt bal}$ for training $f$, by adding synthetic minority samples that are translated from other samples of (relatively) majority classes. There could be multiple ways to perform this “Major-to-minor” translation. In particular, a recent progress on cross-domain generation via generative adversarial networks has made this more attractive, provided that much computational cost for additional training is acceptable. In this paper, on the other hand, we explore a much simpler and efficient approach: we translate a majority sample by optimizing it to maximize the target minority confidence of another baseline classifier $g$. Here, we assume the classifier $g$ is a pre-trained neural network on $\mathcal{D}$ so that performs well (at least) on the training imbalanced dataset, e.g., via standard ERM training. This implies that, $g$ may be over-fitted to minority classes and does not necessarily generalize well under the balanced test dataset. We found this mild assumption on $g$ is fairly enough to capture the information in the small minority classes and could generate surprisingly useful synthetic minority samples by utilizing the diversity of majority samples. On the other hand, $f$ is the target network that we aim to train to perform well on the balanced testing criterion.

During the training of $f$, M2m utilizes the given classifier $g$ to generate new minority samples, and the generated samples are added to $\mathcal{D}$ to construct $\mathcal{D}_{\tt bal}$ on the fly. To obtain a single synthetic minority $x^{*}$ of class $k$, our method solves an optimization problem starting from another training sample $x_{0}$ of a (relatively) major class $k_{0}<k$:

where $\mathcal{L}$ denotes the cross entropy loss and $\lambda>0$ is a hyperparameter. In other words, our method “translates” a majority seed $x_{0}$ into $x^{*}$, so that $g$ confidently classifies it as minority class $k$. The generated sample $x^{*}$ is then labeled to class $k$ and fed into $f$ for training to perform better on $\mathcal{D}_{\tt bal}$ and match the prediction of $f$ to that of $g$. We do not force $f$ in (2) to classify $x^{*}$ to class $k$ as well, but we restrict that $f$ to have lower confidence on the original class $k_{0}$ by imposing a regularization term $\lambda\cdot f_{k_{0}}(x)$. Here, the regularization term $\lambda\cdot f_{k_{0}}(x)$ on the logit reduces the risk when $x^{*}$ is labeled to $k$, whereas it may contain significant features of $x_{0}$ in the viewpoint of $f$. Intuitively, one can regard the overall process as teaching $f$ to learn novel minority features which $g$ considers it significant, i.e., via extension of the decision boundary from the knowledge $g$. Figure 2 illustrates the basic idea of our method.

2 Underlying intuition on M2m

One may understand our method better by considering the case when $g$ is an “oracle” (possibly the Bayes optimal) classifier, e.g., (roughly) humans. Here, solving (2) essentially requires a transition of the original input $x_{0}$ of class $k_{0}$ with $100\%$ confidence to another class $k$ with respect to $g$: this would let $g$ “erase and add” the features related to the class $k_{0}$ and $k$, respectively. Hence, in this case, our process corresponds to collecting more in-distribution minority data, which may be argued as the best way one could do to resolve the class-imbalance problem.

An intriguing point here is, however, that neural network models are very far from this ideal behavior, even when they achieve super-human performance. Instead, when $f$ and $g$ are neural networks, (2) often finds $x^{*}$ that is very close to $x_{0}$, i.e., similar to the phenomenon of adversarial examples . Nevertheless, we found our method still effectively improves the generalization of minority classes even in such cases. This observation is, in some sense, aligned to a recent claim that adversarial perturbation is not a “bug” in neural networks, but a “generalizable” feature .

In this paper, we hypothesize this counter-intuitive effectiveness of our method comes from mainly in two aspects: (a) the sample diversity in the majority dataset is utilized to prevent over-fitting on the minority classes, and (b) another classifier $g$ is enough to capture the information in the small minority dataset. In this respect, adversarial examples from a majority to a minority can be regarded as one of natural ways to leverage the diverse features in majority examples useful to improve the generalization of the minority classes. It is also notable that our over-sampling method does not completely replace the existing dataset. Instead, our method only augment the minority classes, and our finding is that this augmentation turns out to be very effective than naïvely duplicating minority examples as done by the standard over-sampling. We further discuss a more detailed analysis to verify these claims, by performing an extensive ablation study in Section 3.4.

3 Detailed components of M2m

An important factor that affects the quality of the synthetic minority samples in our method is the quality of $g$, especially for $g_{k_{0}}$: a better $g_{k_{0}}$ would more effectively “erase” important features of $x_{0}$ during the translation, thereby making the resulting minority samples more reliable. In practice, however, $g$ is not that perfect: the synthetic samples still contain some discriminative features of the original class $k_{0}$, in which it may even harm the performance of $f$. This risk of “unreliable” generation becomes more harsh when $N_{k_{0}}$ is small, as we assume that $g$ is also trained on the given imbalanced dataset $\mathcal{D}$.

To alleviate this risk, we consider a simple criterion for rejecting each of the synthetic samples randomly with probability depending on $k_{0}$ and $k$:

where $(\cdot)^{+}:=\max(\cdot,0)$, and $\beta\in[0,1)$ is a hyperparameter which controls the reliability of $g$: the smaller $\beta$, the more reliable $g$. For example, if $\beta=0.999$, the synthetic samples are accepted with probability more than $99\%$ if $N_{k_{0}}-N_{k}>4602$. When $\beta=0.9999$, on the other hand, it requires $N_{k_{0}}-N_{k}>46049$ to achieve the same goal. This exponential modeling of the rejection probability is motivated by the effective number of samples , a heuristic recently proposed to model the observation that the impact of adding a single data point exponentially decreases at larger datasets. When a synthetic sample is rejected, we simply replace it by an existing minority sample from the original dataset $\mathcal{D}$ to obtain the balanced dataset $\mathcal{D}_{\tt bal}$.

Optimal seed sampling.

Another design choice of our method is how to choose a (majority) seed sample $x_{0}$ with class $k_{0}$ for each generation in (2). Based on the rejection criterion proposed in (3), we design a sampling distribution $Q(k_{0}|k)$ for selecting the class $k_{0}$ of initial point $x_{0}$ given target class $k$, by considering two aspects: (a) $Q$ maximizes the acceptance probability $P_{\tt accept}(k_{0}|k)$ under our rejection criterion, and (b) $Q$ chooses diverse classes as much as possible, i.e., the entropy $H(Q)$ is maximized. Namely, we are interested in the following optimization:

It is elementary to check that $Q=P_{\tt accept}$ is the solution of the above optimization. Hence, due to the rejection probability (3), we choose:

Once $k_{0}$ is selected, a sample $x_{0}$ is sampled uniformly at random among samples having the class $k_{0}$. The overall procedure of M2m is summarized in Algorithm 1.

Practical implementation via re-sampling.

In practice of training a neural network $f$, e.g., stochastic gradient descent (SGD) with a mini-batch sampling, M2m is implemented using a batch-wise re-sampling. More precisely, in order to simulate the generation of $N_{1}-N_{k}$ samples for any $k=1,2,\cdots,K$, we perform the generation with probability $\frac{N_{1}-N_{y_{i}}}{N_{1}}=1-N_{y_{i}}/N_{1}$, for all $i$ in a given class-balanced mini-batch $\mathcal{B}=\{(x_{i},y_{i})\}_{i=1}^{m}$.Obtaining such a class-balanced mini-batch can be done via standard re-sampling. For a single generation at index $i$, we first sample $k_{0}\sim Q(k_{0}|y_{i})$ following (5) until $k_{0}\in\{y_{i}\}_{i=1}^{m}$, and select a seed $x_{0}$ of class $k_{0}$ randomly inside $\mathcal{B}$. Then, we solve the optimization (2) from $x_{0}$ toward class $y_{i}$ via gradient descent for a fixed number of iterations $T$ with a step size $\eta$. We accept the result sample $x^{*}$ only if $\mathcal{L}(g;x^{*},y_{i})$ is less than $\gamma>0$ for stability. Finally, if accepted, we replace $(x_{i},y_{i})$ in $\mathcal{B}$ by $(x^{*},y_{i})$.

Experiments

We evaluate our method on various class-imbalanced classification tasks: synthetically-imbalanced variants of CIFAR-10/100 , ImageNet-LTResults on ImageNet-LT can be found in the supplementary material. , CelebA , SUN397 , Twitter , and Reuters datasets.Code is available at https://github.com/alinlab/M2m Figure 3 illustrates the class-wise sample distributions for the datasets considered in our experiments. The more details on the tested datasets are given in the supplementary material. To evaluate the classification performance of the models on the balanced test distribution, we mainly report two popular metrics: the balanced accuracy (bACC) and the geometric mean scores (GM) , which are defined by the arithmetic and geometric mean over class-wise sensitivity (i.e., recall), respectively. We remark that bACC is essentially equivalent to the standard accuracy metric for balanced datasets. All the values and error bars in this section are mean and standard deviation across three random trials, respectively. Overall, our results clearly demonstrate that minority synthesis via translating from majority consistently improves the efficiency of over-sampling, in terms of the significant improvement of the generalization in minority classes compared to other re-sampling baselines, across all the tested datasets. We also perform an ablation study to verify the effectiveness of our main ideas.

We consider a wide range of baseline methods, as listed in what follows: (a) empirical risk minimization (ERM): training on the cross-entropy loss without any re-balancing; (b) re-sampling (RS) : balancing the objective from different sampling probability for each sample; (c) SMOTE : a variant of re-sampling with data augmentation; (d) re-weighting (RW) : balancing the objective from different weights on the sample-wise loss; (e) class-balanced re-weighting (CB-RW) : a variant of re-weighting that uses the inverse of effective number for each class, defined as $(1-\beta^{N_{k}})/(1-\beta)$. Here, we use $\beta=0.9999$; (f) deferred re-sampling (DRS) and (g) deferred re-weighting (DRW) : re-sampling and re-weighting is deferred until the later stage of the training, repsectively; (h) focal loss (Focal) : the objective is up-weighted for relatively hard examples to focus more on the minority; (i) label-distribution-aware margin (LDAM) : the classifier is trained to impose larger margin to minority classes. Roughly, the considered baselines can be classified into three categories: (i) “re-sampling” based methods - (b, c, f), (ii) “re-weighting” based methods - (d, e, g), and (iii) different loss functions - (a, h, i).

Training details.

We train every model via stochastic gradient descent (SGD) with momentum of weight $0.9$. The initial learning rate is set to $0.1$, and “step decay” is performed during training where the exact scheduling across datasets is specified in the supplementary material. Although it did not affect much to our method, we also adopt the “linear warm-up” learning rate strategy in the first 5 epochs, as the performance of some baseline methods, e.g., re-weighting, highly depends on the use of this strategy. For CIFAR-10/100 and CelebA, we train ResNet-32 for 200 epochs with mini-batch size 128, and set a weight decay of $2\times 10^{-4}$. In case of SUN397, the pre-activation ResNet-18 model is used instead.We remark this model is larger than ResNet-32 used for CIFAR and CelebA datasets, as it has roughly 4$\times$ more channels. We ensure that all the input images are normalized over the training dataset, and have the size of 32$\times$32 either by cropping or re-sizing, to be compatible with the given architectures. For Twitter and Reuters datasets, we train 2-layer fully-connected networks for 15 epochs with mini-batch size 64, and with a weight decay of $5\times 10^{-5}$.

Details on M2m.

When our method is applied, we use another classifier $g$ of the same architecture to $f$ that is pre-trained on the given (imbalanced) dataset via standard ERM training. Also, in a similar manner to that of , we use the deferred scheduling to our method, i.e., we start to apply our method after the standard ERM training for a fixed number of epochs. The actual scheduling across datasets is specified in the supplementary material. We choose hyperparameters in our method from a fixed set of candidates, namely $\beta\in\{0.9,0.99,0.999\}$, $\lambda\in\{0.01,0.1,0.5\}$ and $\gamma\in\{0.9,0.99\}$ based on the validation set. Unless otherwise stated, we fix $T=10$ and $\eta=0.1$ when performing a single generation step.

2 Long-tailed CIFAR datasets

We consider a “synthetically long-tailed” variant of CIFAR datasets (CIFAR-LT-10/100) in order to evaluate our method on various levels of imbalance, where the original datasets are class-balanced. To simulate the long-tailed distribution frequently appeared in imbalanced datasets, we control the imbalance ratio $\rho>1$ and artificially reduce the training sample sizes of each class except the first class, so that: (a) $N_{1}/N_{K}$ equals to $\rho$, and (b) $N_{k}$ in between $N_{1}$ and $N_{K}$ follows an exponential decay across $k$. We keep the test dataset unchanged during this process, i.e., it is still perfectly balanced, thereby measuring accuracy on this dataset is equivalent to measuring the balanced accuracy. We consider two imbalance ratios $\rho\in\{100,10\}$ each for CIFAR-LT-10 and 100. See Figure 3(a) and 3(b) for a detailed illustration of the sample distribution.

Table 1 summarizes the main results. In overall, the results show that our method consistently improves the bACC by a large margin, across all the tested baselines. These results even surpass the “LDAM+DRW” baseline , which is known to be the state-of-the-art to the best of our knowledge. Moreover, we point out, in most cases, our method could further improve bACC when applied upon the LDAM training scheme (see “LDAM+M2m”): this indicates that the performance gain from our method is fairly orthogonal to that of LDAM, i.e., the margin-based approach, which suggests a new promising direction of improving the generalization when a neural network model suffers from a problem of small data.

3 Real-world imbalanced datasets

We further verify the effectiveness of M2m on four well-known, naturally imbalanced datasets: CelebA , SUN397 , Twitter and Reuters datasets. More detailed information for each of these datasets is demonstrated in Figure 3 and the supplementary material.

CelebA is originally a multi-labeled dataset, and we port this to a 5-way classification task by filtering only the samples with five non-overlapping labels about hair colors. We also subsampled the full dataset by $1/20$ while maintaining the imbalance ratio $\rho\approx 10.7$, in attempt to make the task more difficult. We denote the resulting dataset by CelebA-5.

Although Twitter and Reuters datasets are from natural language processing, we also evaluate our method on them to test the effectiveness under much extreme imbalance. Here, we remark that the imbalance ratio $N_{1}/N_{k}$ of these two datasets are about 150 and 710, respectively, which are much higher than the other image datasets tested. In case of Reuters, we exclude the classes having less than 5 samples in the test set for more reliable evaluation, resulting a dataset of 36 classes.

Table 2 shows the results. Again, M2m performs best amongst other baseline methods, demonstrating the effectiveness of our method under natural imbalance, as well as wider applicability of our algorithm beyond image classification. Remarkably, the significant results on Reuters dataset compared to the others suggest that our method can be even more effective under a regime of “extremely” imbalanced datasets, as Reuters has a much larger imbalance ratio than the others.

4 Ablation study

We conduct an extensive ablation study to present a detailed analysis of the proposed method. All the experiments in this section are performed with ResNet-32 models, trained on CIFAR-LT-10 with the imbalance ratio $\rho=100$. We additionally report the balanced test accuracy over majority and minority classes, namely “Major” and “Minor” respectively, to further identify the relative impacts on those two classes separately. We divide the whole classes into “majority” and “minority” classes, so that the majority classes consist of top-$k$ frequent classes with respect to the training set where $k$ is the minimum number that $\sum_{k}N_{k}$ exceeds $50\%$ of the total. We denote the minority classes as the remaining classes. We provide more discussion in the supplementary material.

In Section 2.1, we hypothesize that the effectiveness of our method mainly comes from utilizing a much diversity in the majority samples to prevent the over-fitting to the minority classes. To verify this, we consider an ablation that the candidates of “seed samples” are limited: more concretely, we control the size of seed sample pools per each class to a fixed subset of the training set, made before training $f$. In Table 3, the accuracy of minority classes is progressively increased as seed sample pools become diverse. This clear trend indicates that M2m makes use of the diversity of majority classes for preventing the over-fitting to the minority classes.

The effect of λ𝜆\lambda.

In the optimization objective (2) for the generation step in M2m, we impose a regularization term $\lambda\cdot f_{k_{0}}(x)$ to improve the quality of synthetic samples: they might confuse $f$ if themselves still contain important features of the original class in a viewpoint of $f$. To verify the effect of this term, we consider an ablation that $\lambda$ is set to , and compare the performance to the original method. As reported in Table 4, we found a certain level of degradation in the balanced test accuracy at this ablation, which shows the effectiveness of the proposed regularization.

Over-sampling from the scratch.

As specified in Section 3.1, we use the “deferred” scheduling to our method by default, i.e., we start to apply our method after the standard ERM training for a fixed number of epochs. We have also considered a simple ablation where this strategy is not used, namely “M2m-RS”. The results in Table 4 show that M2m-RS still outperforms any other baselines (reported in Table 1) except the ones that the deferred scheduling is used, i.e., DRS and DRW, and this further verifies the effectiveness of our method.

Labeling as a targeted class.

Our primary assumption on the pre-trained classifier $g$ does not require that $g$ itself to generalize well on the minority classes (see Section 2.1). This implies that solving (2) with $g$ may not end up with a synthetic sample that contains generalizable features of the target minority class. To examine how much the generated samples would be correlated to the target classes, we consider another ablation upon M2m-RS:Here, we attempt to opt out any potential effect from using DRS, for more clearer evaluation. instead of labeling the generated sample as the target class, the ablated method “M2m-RS-Rand” labels it to a “random” class chosen from all the possible classes (except for the target and original classes). The results shown in Table 4 indicate that M2m-RS-Rand generalizes much worse than its counterpart M2m-RS on the minority classes, which indeed confirms that the correctly-labeled synthetic samples could improve the generalization of the minority classes.

Comparison of t-SNE embeddings.

To further validate the effectiveness of our method, we visualize and compare the penultimate features learned from various training methods (including ours) using t-SNE . Each embedding is computed from a randomly-chosen subset of training samples in the CIFAR-LT-10 ($\rho=100$), so that it consists of 50 samples per each class. Figure 4 illustrates the results, and shows that the embedding from our training method (M2m) is of much separable features compared to other methods: one could successfully distinguish each cluster under the M2m embedding (even though they are from minority classes), while others have some obscure region.

Comparison of cumulative false positive.

The use of adversarial examples.

As mentioned in Section 2.2, the generation under M2m often ends up with a synthetic minority sample that is very close to the original (before translation) as like the adversarial example. This indeed happens when $f$ and $g$ are neural networks as assumed here, i.e., ResNet-32, as illustrated in Figure 6. To understand more on how such adversarial perturbations affect our method, we consider a simple ablation, which we call “M2m-Clean”: recall that our method synthesizes a minority sample $x^{*}$ from a seed majority sample $x_{0}$. This ablation uses the “clean” $x_{0}$ instead of $x^{*}$ for over-sampling. Under the identical training setup, we notice a significant reduction in the balanced accuracy of M2m-Clean compared to the original M2m (see Table 4). This observation reveals that the adversarial perturbations ablated are extremely crucial to make our method to work, regardless of a small noise.

Conclusion

We propose a new over-sampling method for imbalanced classification, called Major-to-minor Translation (M2m). We found the diversity in majority samples could much help the class-imbalanced training, even with a simple translation method using a pre-trained classifier. This suggests a promising way to overcome the long-standing class-imbalance problem, and exploring more powerful methods to perform this Major-to-minor translation, e.g., CycleGAN , would be an interesting future research. The problems we explored in this paper also lead us to an essential question that whether an adversarial perturbation could be a good feature. Our findings suggest that it could be, at least for the purpose of imbalanced learning, where the minority classes suffer over-fitting due to insufficient data. We believe our method could open up a new direction of research both in imbalanced learning and adversarial examples.

Acknowledgements

This work was supported by Samsung Electronics and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).

References

Appendix A Details on the datasets

CIFAR-10/100 datasets consist of 60,000 RGB images of size $32\times 32$, 50,000 for training and 10,000 for testing. Each image in the two datasets is corresponded to one of 10 and 100 classes, respectively. In our experiments, we construct “synthetically long-tailed” variants of CIFAR-10/100, namely CIFAR-LT-10/100, respectively . We hold-out 10% of the test set to construct a validation set, and use the remaining for testing. We use ResNet-32 with a mini-batch size 128, and set a weight decay of $2\times 10^{-4}$. We train the network for 200 epochs with an initial learning rate of 0.1. We follow the learning rate schedule used by for fair comparison: the initial learning rate is set to $0.1$, and we decay it by a factor of $100$ at $160$-th and $180$-th epoch. When the deferred scheduling is used, e.g., DRS, DRW and our method, it is applied after 160 epochs of standard training.

CelebA-5.

CelebFaces Attributes (CelebA) dataset is a multi-labeled face attributes dataset. It is originally composed of 202,599 number of RGB face images with 40 binary attributes annotations per image. We port this CelebA to a 5-way classification task by filtering only the samples with five non-overlapping labels about hair colors: namely, “blonde”, “black”, “bald”, “brown”, and “gray”. This is in a similar manner as done in . We denote the resulting dataset by CelebA-5. We pick out 50 and 100 samples per each class for validation and testing. We use ResNet-32 with a mini-batch size 128, and set a weight decay of $2\times 10^{-4}$. We train the network for 90 epochs with an initial learning rate of 0.1. We decay the learning rate by 0.1 at epoch 30 and 60. When the deferred scheduling is used, it is applied after 60 epochs of standard training.

SUN397.

Scene UNderstanding (SUN) is a dataset for a scene categorization. It originally consists of 108,754 RGB images which are labeled with 397 classes. For the inputs, center patches are first extracted and they are resized to 32$\times$32. We hold-out 10 and 40 samples per each class for validation and testing, respectively, as the dataset itself does not provide any separated split for testing. We use pre-activation ResNet-18 which roughly has $4\times$ more channels with a mini-batch size 128, and set a weight decay of $2\times 10^{-4}$. We train the network for 90 epochs with an initial learning rate of 0.1. We decay the learning rate by 0.1 at epoch 30 and 60. When the deferred scheduling is used, it is applied after 60 epochs of standard training.

Twitter.

Twitter is a dataset for a part-of-speech (POS) tagging task in social media text with 25 classes. Each sample is a pair of a token and a tag, e.g., “(books, common noun)” and “(#acl, hashtag)”, where each token is embedded into a 50-dimensional vector via a pre-defined word-embedding [hendrycks2016baseline]. We discarded two classes with zero test samples and obtained 14,614 training samples with 23 classes. We use 2-layer fully-connected network with a hidden layer size of 256 and a ReLU nonlinearity. We set a mini-batch size 64 and a weight decay of $5\times 10^{-5}$. We train the network for 15 epochs with an initial learning rate 0.1 and decay the learning rate by 0.1 at epoch 10. When the deferred scheduling is used, it is applied after 10 epochs of standard training.

Reuters.

Reuters is a dataset for a text categorization task which predicts the subject of a given text. As an input, 1000-dimensional bag-of-words vectors are given, which are processed from a news story document. It is originally composed of 52 classes, but we discarded the classes that have less than 5 test samples for a reliable evaluation, obtaining a subset of the full dataset of 36 classes with 6436 training samples. We hold-out 10% of training samples to construct a validation set. We use 2-layer fully-connected network with a hidden layer size of 256 and a ReLU nonlinearity. We set a mini-batch size 64 and a weight decay of $5\times 10^{-5}$. We train the network for 15 epochs with an initial learning rate 0.1 and decay the learning rate by 0.1 at epoch 10. When the deferred scheduling is used, it is applied after 10 epochs of standard training.

Appendix B More results from ablation study

As mentioned, our method introduces another classifier $g$ to generate synthetic minority $x^{*}$ independently from the training classifier $f$. This is because using $f$ itself instead of $g$ in the optimization objective (2) would let the synthetic samples already confident in the target minority class to $f$, and this makes the overall training process redundant. To further validate the importance of using $g$, we consider an ablation called “M2m-Self”: instead of using $g$, “M2m-Self” uses $f$ for generating minority samples. As reported in Table 6, one could immediately see that M2m-Self only shows marginal improvement from DRS, which is much inferior than the original M2m.

Using multiple classifiers for generation.

Since our method is not restricted to use the only one pre-trained classifier $g$ in the optimization (2), the multiple classifiers $g_{i}$ for $i=1,\dots,m$ can be used to improve the quality of generation. To verify the additional gain from multiple classifiers, we consider an ablation called “M2m-Ensemble”: use the ensemble of the classifiers ($m=2$) for generation instead of the single classifier. Here, we use the same architecture ResNet-32 for $g_{1}$ and $g_{2}$ and use a higher $\gamma$ due to the smoothed prediction from the ensemble. The results in Table 6 show that M2m-Ensemble slightly perform better than M2m. It indicates that our method can benefit from the stronger classifier.

Rejection criteria.

We also propose a sample rejection criteria to alleviate the risk of unreliable generation, possibly due to a weak generalization of $g$. To verify the effect of this rejection criteria, we consider an ablation, namely “M2m-No-Reject”, which does not use this rejection policy in training. In other words, all the generated samples are used to train $f$. The results in Table 6 show that M2m-No-Reject performs significantly worse than M2m. This indeed confirms the gain from using the proposed rejection criteria.

The effect of γ𝛾\gamma.

As specified in Algorithm 1 in the main paper, we set a threshold $\gamma$ to filter out the synthetic samples which the generation objective is not sufficiently minimized, mainly due to the limited budget. To evaluate the practical effectiveness of using $\gamma$, here we consider an ablation that this thresholding is not used, equivalently when $\gamma=\infty$. As reported in Table 6, we indeed observe a performance degradation by not using $\gamma$. This reveals that the confidence level in $g$ affects the final quality of the generation.

Appendix C Results on ImageNet-LT

We additionally evaluate our method on ImageNet-LT dataset, a subset of ImageNet dataset with a synthetic imbalance following the Pareto distribution of the power $\alpha=6$. It is composed of 115,846 training samples with 1,000 categories, 1,280 images in the maximal class and 5 images in the minimal class. A more detailed distribution is presented in Figure 7. We use the randomly-resized cropping and the horizontal flipping as a data augmentation, and all the images are resized to 128$\times$128. We hold-out 20 samples per class randomly from the original ImageNet training set to form a validation set, and the original (roughly balanced) ImageNet validation set is used for testing. We use ResNet-50 with a mini-batch size 256 and set a weight decay of $10^{-4}$. We train the network for 200 epochs with an initial learning rate of 0.1 and it is decayed by 0.1 at epoch 160 and 180. When the deferred scheduling is used, e.g., DRS, DRW and our method, it is applied after 160 epochs of standard training. We evaluate our method with followings which show the best performance among the baselines in the experiments in the main paper: (a) ERM-DRS and (b) LDAM-DRW . We report the balanced accuracy (bACC) and the geometric mean scores (GM). As reported in Table 6, our method, M2m, significantly outperforms the baselines. In the case of ERM loss, compare to DRS, M2m shows 3.43 % and 4.75 % relative gains in bACC and GM, respectively. Furthermore, with a margin-based loss function LDAM, the improvement is much enlarged.