The Devil is in the Tails: Fine-grained Classification in the Wild

Introduction

During the past five years we have witnessed dramatic improvement in the performance of visual recognition algorithms [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.]. Human performance has been approached or achieved in many instances. Three concurrent developments have enabled such progress: (a) the invention of ‘deep network’ algorithms where visual computation is learned from the data rather than hand-crafted by experts [Fukushima and Miyake(1982), LeCun et al.(1989)LeCun, Boser, Denker, Henderson, Howard, Hubbard, and Jackel, Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton], (b) the design and construction of large and well annotated datasets [Fei-Fei et al.(2004)Fei-Fei, Fergus, and Perona, Everingham and et al.(2005), Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei, Lin et al.(2014a)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] supplying researchers with a sufficient amount of data to train learning-based algorithms, and (c) the availability of inexpensive and ever more powerful computers, such as GPUs [Lindholm et al.(2008)Lindholm, Nickolls, Oberman, and Montrym], for algorithm training.

Large annotated datasets yield two additional benefits, besides supplying deep nets with sufficient training fodder. The first is offering common performance benchmarks that allow researchers to compare results and quickly evolve the best algorithmic architectures. The second, more subtle but no less important, is providing researchers with a compass – a definition of the visual tasks that one ought to try and solve. Each new dataset pushes us a bit further towards solving real world challenges. We wish to focus here on the latter aspect.

One goal of visual recognition is to enable machines to recognize objects in the world. What does the world look like? In order to better understand the nature of visual categorization in the wild we examined three real-world datasets: bird species, as photographed worldwide by birders who are members of eBird [Sullivan et al.(2009)Sullivan, Wood, Iliff, Bonney, Fink, and Kelling], tree species, as observed along the streets of Pasadena [Wegner et al.(2016)Wegner, Branson, Hall, Schindler, and Perona], and plants and animal species, as photographed by the iNaturalist (www.inaturalist.org) community. One salient aspect of these datasets is that some species are very frequent, while most species are represented by only few specimens (Fig 1). In a nutshell: the world is long-tailed, as previously noted in the context of subcategories and object views [Salakhutdinov et al.(2011)Salakhutdinov, Torralba, and Tenenbaum, Zhu et al.(2014)Zhu, Anguelov, and Ramanan]. This is in stark contrast with current datasets for visual classification, where specimen distribution per category is almost uniformly distributed (see [Lin et al.(2014a)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Fig. 5(a)).

With this observation in mind, we ask whether current state-of-the-art classification algorithms, whose development is motivated and benchmarked by uniformly distributed datasets, deal well with the world’s long tails. Humans appear to be able to generalize from few examples, can our algorithms do the same? Our experiments show that the answer is no. While, when data is abundant, machine vision classification performance can currently rival humans, we find that this is emphatically not the case when data is scarce for most classes, even if a few are abundant.

This work is organized in the following: In Section 2 we review the related work. We then describe the datasets and training process in Section 3 followed by an analysis of the experiments in Section 4. We summarize and conclude in Section 5.

Related Work

Fine-Grained Visual Classification – The vision community has released many fine-grained datasets covering several domains such as birds [Welinder et al.(2010)Welinder, Branson, Mita, Wah, Schroff, Belongie, and Perona, Wah et al.(2011)Wah, Branson, Welinder, Perona, and Belongie, Berg et al.(2014)Berg, Liu, Lee, Alexander, Jacobs, and Belhumeur, Van Horn et al.(2015)Van Horn, Branson, Farrell, Haber, Barry, Ipeirotis, Perona, and Belongie], dogs [Khosla et al.(2011)Khosla, Jayadevaprakash, Yao, and Fei-Fei, Liu et al.(2012)Liu, Kanazawa, Jacobs, and Belhumeur], airplanes [Maji et al.(2013)Maji, Rahtu, Kannala, Blaschko, and Vedaldi, Vedaldi et al.(2014)Vedaldi, Mahendran, Tsogkas, Maji, Girshick, Kannala, Rahtu, Kokkinos, Blaschko, Weiss, et al.], flowers [Nilsback and Zisserman(2006)], leaves [Kumar et al.(2012)Kumar, Belhumeur, Biswas, Jacobs, Kress, Lopez, and Soares], trees [Wegner et al.(2016)Wegner, Branson, Hall, Schindler, and Perona] and cars[Krause et al.(2013)Krause, Stark, Deng, and Fei-Fei, Lin et al.(2014b)Lin, Morariu, Hsu, and Davis]. These datasets were constructed to be uniform, or to contain "enough" data for the task. The recent Pasadena Trees dataset [Wegner et al.(2016)Wegner, Branson, Hall, Schindler, and Perona] is the exception. Most fine-grained research papers present a novel model for classification [Xu et al.(2015)Xu, Huang, Zhang, and Tao, Lin et al.(2015)Lin, RoyChowdhury, and Maji, Farrell et al.(2011)Farrell, Oza, Zhang, Morariu, Darrell, and Davis, Krause et al.(2015)Krause, Jin, Yang, and Fei-Fei, Xie et al.(2015)Xie, Yang, Wang, and Lin, Branson et al.(2014)Branson, Van Horn, Perona, and Belongie, Gavves et al.(2015)Gavves, Fernando, Snoek, Smeulders, and Tuytelaars, Simon and Rodner(2015), Göring et al.(2014)Göring, Rodner, Freytag, and Denzler, Shih et al.(2015)Shih, Mallya, Singh, and Hoiem, Zhang et al.(2014)Zhang, Donahue, Girshick, and Darrell, Berg and Belhumeur(2013), Chai et al.(2013)Chai, Lempitsky, and Zisserman, Xiao et al.(2015)Xiao, Xu, Yang, Zhang, Peng, and Zhang, Zhang et al.(2016)Zhang, Wei, Wu, Cai, Lu, Nguyen, and Do, Pu et al.(2014)Pu, Jiang, Wang, and Xue]. While these methods often achieve state-of-the-art performance at the time of being published, it is often the case that the next generation of convolutional networks can attain the same level of performance without any custom modifications. In this work we use the Inception-v3 model [Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna], pretrained on ImageNet for our experiments. Some of the recent fine-grained papers have investigated augmenting the original datasets with additional data from the web [Krause et al.(2016)Krause, Sapp, Howard, Zhou, Toshev, Duerig, Philbin, and Fei-Fei, Xu et al.(2015)Xu, Huang, Zhang, and Tao, Xie et al.(2015)Xie, Yang, Wang, and Lin, Van Horn et al.(2015)Van Horn, Branson, Farrell, Haber, Barry, Ipeirotis, Perona, and Belongie]. Krause et al\bmvaOneDot[Krause et al.(2016)Krause, Sapp, Howard, Zhou, Toshev, Duerig, Philbin, and Fei-Fei] investigated the collection and use of a large, noisy dataset for the task of fine-grained classification and found that off the shelf CNN models can readily take advantage of these datasets to increase accuracy and reach state-of-the-art performance. Krause et al\bmvaOneDotmention, but do not investigate, the role of the long tail distribution of training images. In this work we specifically investigate the effect of this long tail on the model performance.

Imbalanced Datasets – Techniques to handle imbalanced datasets are typically split into two regimes: algorithmic solutions and data solutions. In the first regime, cost-sensitive learning [Elkan(2001)] is employed to force the model to adjust its decision boundaries by incurring a non-uniform cost per misclassification; see [He and Garcia(2009)] for a review of the techniques. The second regime concerns data augmentation, achieved either through over-sampling the minority classes, under sampling the majority classes or synthetically generating new examples for the minority classes. When using mini batch gradient descent (as we do in the experiments), oversampling the minority classes is similar to weighting these classes more than the majority classes, as in cost-sensitive learning. We conduct experiments on over-sampling the minority classes. We also employ affine [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] and photometric [Howard(2013)] transformations to synthetically boost the number of training examples.

Transfer Learning – Transfer learning [Pan and Yang(2010)] attempts to adapt the representations learned in one domain to another. In the era of deep networks, the simplest form of transfer learning is using features extracted from pretrained ImageNet [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] or Places [Zhou et al.(2014)Zhou, Lapedriza, Xiao, Torralba, and Oliva] networks, see [Sharif Razavian et al.(2014)Sharif Razavian, Azizpour, Sullivan, and Carlsson, Donahue et al.(2014)Donahue, Jia, Vinyals, Hoffman, Zhang, Tzeng, and Darrell]. The next step is actually fine-tuning [Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik] these pretrained networks for the target task [Yosinski et al.(2014)Yosinski, Clune, Bengio, and Lipson, Agrawal et al.(2014)Agrawal, Girshick, and Malik, Oquab et al.(2014)Oquab, Bottou, Laptev, and Sivic, Huh et al.(2016)Huh, Agrawal, and Efros]. This has become the standard method for obtaining baseline numbers on a new target dataset and often leads to impressive results [Azizpour et al.(2015)Azizpour, Sharif Razavian, Sullivan, Maki, and Carlsson], especially when the target dataset has sufficient training examples. More sophisticated transfer learning methods [Long et al.(2015)Long, Cao, Wang, and Jordan, Tzeng et al.(2015)Tzeng, Hoffman, Darrell, and Saenko] are aimed at solving the domain adaptation problem. In this work, we are specifically interested in a single domain, which happens to contain a long tail distribution of training data for each class. We investigate whether there is a transfer of knowledge from the well represented classes to the sparsely represented classes.

Low Shot Learning – We experiment with a minimum of 10 training images per class, which falls into the realm of low shot learning, a field concerned with learning novel concepts from few examples. In [Wang and Hebert(2016a)] Wang and Herbet learn a regression function from classifiers trained on small datasets to classifiers trained on large datasets, using a fixed feature representation. Our setup is different in that we want to allow our feature representation to adapt to the target dataset and we want a model that can classify both the well represented classes and the sparsely represented classes. The recent work of Hariharan and Girshick in [Hariharan and Girshick()] explored this setup specifically, albeit in the broad domain of ImageNet. The authors propose a low shot learning benchmark and implement a loss function and feature synthesis scheme to boost performance on under represented classes. However, their results showed marginal improvement when using a high capacity model (at 10 images per class the ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] model performed nearly as well as their proposed method). Our work aims to study the relationship between the well represented classes and the sparse classes, within a single domain. Metric learning tackles the low shot learning problem by learning a representation space where distance corresponds to similarity. While these techniques appear promising and provide benefits beyond classification, they do not hold up well against simple baseline networks for the specific task of classification [Rippel et al.(2015)Rippel, Paluri, Dollar, and Bourdev].

Experiment Setup

We consider three different types of datasets: uniform, long tail and approximate long tail. We used images from eBird (ebird.org) to construct each of these datasets. These images are real world observations of birds captured by citizen scientists and curated by regional experts. Each dataset consists of a training, validation, and test split. When placing images into each split we ensure that a photographer’s images do not end up in multiple splits for a single species. The test set is constructed to contain as many different photographers as possible (e.g\bmvaOneDot30 images from 30 different photographers). The validation set is similarly constructed and the train set is constructed from the remaining photographers.

Uniform Datasets – The uniform datasets allow us to study the performance of the classification model under optimal image distribution conditions. These datasets have the same number of images per class: either 10, 100, 1K, or 10K. The total number of classes can be either 10, 100, or 1K. We we did not analyze a uniform dataset with 1K classes containing 1K or 10K images each due to a lack of data from eBird. Each smaller dataset is completely contained within the larger dataset (e.g\bmvaOneDotthe 10 class datasets are contained within the 100 class datasets, etc.). The test and validation sets are uniform, with 30 and 10 images for each class respectively, and remain fixed for a class across all uniform datasets.

Approx. Long Tail Datasets – To conveniently explore the effect of moving from a uniform dataset to a long tail dataset we constructed approximate long tail datasets, see Figure 1. These datasets consist of 1K classes split into two groups: the head classes and the tail classes. All classes within a group have the same number of images. We study two different sized splits: a 10 head, 990 tail split and a 100 head, 900 tail split. The 10 head split can have 10, 100, 1K, or 10K images in each head class. The 100 head split can have 10, 100, or 1K images in each head class. The tail classes from both splits can have 10 or 100 images. We use the validation and test sets from the 1K class uniform dataset for all of the approximate long tail datasets. This allows us to compare the performance characteristics of the different datasets in a reliable way, and we can use the 1K class uniform datasets as reference points.

Long Tail Datasets – The full eBird dataset, with nearly 3 million images, is not amenable to easy experimentation. Rather than training on the full dataset we would prefer to model the image distribution and use it to construct smaller, tunable datasets, see Figure 1. We did this by fitting a two piece broken power law to the eBird image distribution. Each class $i\in[1,N]$, is put into the head group if $i<=h$ otherwise it is put into the tail group, where $h$ is the number of head classes. Each head class $i$ contains $y\cdot{}i^{a_{1}}$ images, where $y$ is the number of images in the most populous class and $a_{1}$ is the power law exponent for the head classes. Each tail class $i$ has $y\cdot{}h^{(a_{1}-a_{2})}\cdot{}i^{a_{2}}$ where $a_{2}$ is the power law exponent for the tail classes. We used linear regression to determine that $a_{1}=-0.3472$ and $a_{2}=-1.7135$. We fixed the minimum number of images for a class to be 10. This leaves us with 2 parameters that we can vary: $y$, which shifts the distribution up and down, and $h$ which shifts the distribution left and right. We analyze four long tail datasets by selecting $y$ from {1K, 10K} and $h$ from {10, 100}. Each resulting dataset consists of a different number of classes and therefore has a different test and validation split. We keep to the pattern of reserving 30 test images and 10 validation images for each class.

2 Model Training & Testing Details

Model – We use the Inception-v3 network[Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna], pretrained from ILSVC 2012 as the starting point for all experiments. The Inception-v3 model exhibits good trade-off between size of the model (27.1M parameters) and classification accuracy on the ILSVC (78% top 1 accuracy) as compared to architectures like AlexNet and VGG. We could have used the ResNet-50 model but opted for Inception-v3 as it is currently being used by the eBird development team.

Training – We have a fixed training regime for each dataset. We fine-tune the pretrained Inception-v3 network (using TensorFlow[Abadi et al.(2015)Abadi, Agarwal, Barham, Brevdo, Chen, Citro, Corrado, Davis, Dean, Devin, Ghemawat, Goodfellow, Harp, Irving, Isard, Jia, Jozefowicz, Kaiser, Kudlur, Levenberg, Mané, Monga, Moore, Murray, Olah, Schuster, Shlens, Steiner, Sutskever, Talwar, Tucker, Vanhoucke, Vasudevan, Viégas, Vinyals, Warden, Wattenberg, Wicke, Yu, and Zheng]) by training all layers using a batch size of 32. Unless noted otherwise, batches are constructed by randomly sampling from the pool of all training images. The initial learning rate is 0.0045 and is decayed exponentially by a factor or 0.94 every 4 epochs. Training augmentation consists of taking random crops from the image whose area can range from 10% to 100% of the image, and whose aspect ratio can range from 0.7 to 1.33. The crop is randomly flipped and has random random brightness and saturation distortions applied.

Testing – We use the validation loss to stop the training by waiting for it to steadily increase, signaling that the model is overfitting. We then consider all models up to this stopping point and use the model with the highest validation accuracy for testing. At test time, we take a center crop of the image, covering 87.5% of the image area. We track top 1 image accuracy as the metric, as is typically used in fine-grained classification benchmarks. Note that image accuracy is the same as class average accuracy for datasets with uniform validation and test sets, as is the case for all of our experiments.

Experiments

We first study the performance characteristics of the uniform datasets. We consider two regimes: 1) we extract feature vectors from the pretrained network and train a linear SVM. 2) We fine-tune the pretrained network, see section 3.2 for the training protocol. We use the activations of the layer before the final fully connected layer as our features for the SVM, and used the validation set to tune the penalty parameter. Figure 2 plots the error achieved under these two different regimes. We can see that fine-tuning the neural network is beneficial in all cases except the extreme case of 10 classes with 10 images each (in which case the model overfit quickly, even with extensive hyperparameter sweeps). The neural network scales incredibly well with increasing number of classes, incurring a small increase in error for 10x increase in the number of classes. This should be expected given that the network was designed for 1000-way ImageNet classification. At 10k images per class the network is achieving 96% accuracy on 10 bird species, showing that the network can achieve high performance given enough data. For the network, a 10x increase in data corresponds to at least a 2x error reduction. Keep in mind that the opposite is true as well: as we remove 10x images the error rate increases by at least 2x. These uniform dataset results will be used as reference points for the long tail experiments.

2 Uniform vs Natural Sampling

The long tail datasets present an interesting question when it comes to creating the training batches: Should we construct batches of images such that they are sampled uniformly from all classes, or such that they are sampled from the natural distribution of images? Uniformly sampling from the classes will result in a given tail image appearing more frequently in the training batches than a given head image, i.e\bmvaOneDotwe are oversampling the tail classes. To answer this question, we trained a model for each of our approximate long tail datasets using both sampling methods and compared the results. Figure 3 plots the error achieved with the different sampling techniques on three different splits of the classes (all classes, the head classes, and the tail classes). We see that both sampling methods often converge to the same error, but the model trained with natural sampling is typically as good or better than the model trained with uniform sampling. Figure 4 visualizes the performance of the classes under the two different sampling techniques for two different long tail datasets. These figures highlight that the head classes clearly benefit from natural sampling and the center of mass of the tail classes is skewed slightly towards the natural sampling. The results for the long tail dataset experiments in the following sections use natural sampling.

3 Transferring Knowledge from the Head to the Tail

Section 4.1 showed that the Inception-v3 architecture does extremely well on uniform datasets: achieving 96% accuracy on the 10 class, 10K images per class dataset, 87.3% accuracy on the 100 class, 1K images per class dataset, and 71.5% accuracy on the 1K class, 100 images per class dataset. The question we seek to answer is: How is performance affected when we move to a long tail dataset? Figure 5 summarizes our findings for the approximate long tail datasets. Starting with a dataset of 1000 classes and 10 images in each class, the top 1 accuracy across all classes is 33.2% (this is the bottom, left most blue point in the figure). If we designate 10 of the classes as head classes, and 990 classes as tail classes, what happens when we increase the number of available training data in the head classes (traversing the bottom blue line in Figure 5)? We see that the head class accuracy approaches the peak 10 class performance of 96% accuracy (reaching 94.7%), while the tail classes have remained near their initial performance of 33.2%.

We see a similar phenomena even if we are more optimistic regarding the number of available training images in the tail classes, using 100 rather than 10 (the top blue line in Figure 5). The starting accuracy across all 1000 classes, each with 100 training images, is 71.5%. As additional images are added to the head classes, the accuracy on the head classes again approaches the peak 10 class performance (reaching 94%) while the tail classes are stuck at 71%.

We can be optimistic in another way by moving more classes into the head, therefore making the tail smaller. We now specify 100 classes to be in the head, leaving 900 classes for the tail (the green points in 5). We see a similar phenomena even in this case, although we do see a slight improvement for the tail classes when the 100 head classes have 1k images each. These experiments reveal that there is very little to no transfer learning occurring within the network. Only the classes that receive additional training data actually see an improvement in performance, even though all classes come from the same domain. To put it plainly, an additional 10K bird images covering 10 bird species does nothing to help the network learn a better representation for the remaining bird species.

To confirm the results on the approximate long tail datasets we experimented on four long tail distributions modeled after the actual eBird dataset, see Section 3.1 for details on the datasets. For these experiments we trained three separate models, one trained with all classes and the other two trained with the head classes or tail classes respectively. Figures 5 and 6 show the results. We see the same recurring story: the tail performance is not affected by the head classes. Training a model solely on the tail classes is as good, or even better, than training jointly with the head classes, even though the head classes are from the same domain and are doubling the size of the training dataset. The network is not transferring knowledge from the head classes to the tail classes.

See the supplementary material for exact performance values of the various experiments.

Discussion and Conclusions

The statistics of images in the real world is long-tailed: a few categories are highly represented and most categories are observed only rarely. This in stark contrast with the statistics of popular benchmark datasets, such as ImageNet [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei], COCO [Lin et al.(2014a)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] and CUB200 [Wah et al.(2011)Wah, Branson, Welinder, Perona, and Belongie], where the training images are evenly distributed amongst classes.

We experimentally explored the performance of a state-of-the-art classification model on approximate and realistic long-tailed datasets. We make four observations which, we hope, will inform future research in visual classification.

First, performance is excellent, even in challenging tasks, when the number of training images exceeds many thousands. For example, the species classification error rate is about 4% in the eBird dataset when each species is trained with 104 images (see Fig.2). This is in line with the performance observed on ImageNet and COCO, where current algorithms can rival humans.

Second, if the number of training images is sufficient, classification performance suffers only minimally from an increase in the number of classes (see Fig.2). This is indeed good news, as we estimate that there are tens of millions of object categories that one might eventually attempt to classify simultaneously.

Third, the number of training images is critical: classification error more than doubles every time we cut the number of training images by a factor of 10 (see Fig.2). This is particularly important in a long-tailed regime since the tails contain most of the categories and therefore dominate average classification performance. For example: the largest long tail dataset from our experiments contains 550,692 images and yields an average classification error of 48.6% (see Fig. 5). If the same 550,692 images were distributed uniformly amongst the 2215 classes the average error rate would be about 27% (see Fig. 2). Another way to put it: Collecting the eBird dataset took a few thousand motivated birders about 1 year. Increasing its size to the point that its top 2000 species contained at least 104 images would take 100 years (see Fig. 1). This is a long time to wait for excellent accuracy.

Fourth, on the datasets tested, transfer learning between classes is negligible with current classification models. Simultaneously training on well-represented classes does little or nothing for the performance on those classes that are least represented. The average classification accuracy of the models will be dominated by the poor tail performance, and adding data to the head classes will not improve the situation.

Our findings highlight the importance of continued research in transfer and low shot learning [Fei-Fei et al.(2004)Fei-Fei, Fergus, and Perona, Hariharan and Girshick(), Wang and Hebert(2016a), Wang and Hebert(2016b)] and provide baselines for future work to compare against. When we train on uniformly distributed datasets we sweep the world’s long tails under the rug and we do not make progress in addressing this challenge. As a community we need to face up to the long-tailed challenge and start developing algorithms for image collections that mirror real-world statistics.

Supplementary

2 Increasing Performance on the Head Classes

The experiments in Section 4.3 showed that we should not expect the tail classes to benefit from additional head class training data. While we would ultimately like to have a model that performs well on the head and tail classes, for the time being we may have to be content with optimizing for the classes that have sufficient training data, i.e\bmvaOneDotthe head classes. In this section we explore whether we can use the tail to boost performance on the head classes. For each experiment the model is trained on all classes specified in the training regime (which may be the head classes only, or could be the head and the tail classes), but at test time only head test images are used and only the head class predictions are considered (e.g\bmvaOneDota model trained for 1000 way classification will be restricted to make predictions for the 10 head classes only).

We first analyze the performance of the head classes in a uniform dataset situation, where we train jointly with tail classes that have the same number of training images as the head classes. This can be considered the best case scenario for transfer learning as the source and target datasets are from the same distribution and there are many more source classes than target classes. Figure 7(a) shows the results. In both the 10 head class situation and 100 head class situation we see drops in error when jointly training the head and tail (dashed lines) as compared to the head only model (solid lines).

The next experiments explore the benefit to the head in the approximate long tail datasets. Figures 7(b) and 7(c) show the results. We found that there is a benefit to training with the long tail, between 6.3% and 32.5% error reduction, see Table 4. The benefit of the tail typically decreases as the ratio of head images to tail images increases. When this ratio exceeds 10 it is worse to use the tail during training.

In these experiments we have been monitoring the performance of all classes during training with a uniform validation set (10 images per class). This validation set is our probe into the model and we use it to select which iteration of the model to use for testing. We now know that using as much tail data as possible is beneficial. This raises the following question: Can we monitor solely the head classes with the validation set and still recover an accurate model? If the answer is yes then we will be able to place all tail images in the training set rather than holding some out for the validation set. The results shown in Figure 8 show that this is possible and that it actually produces a more accurate model for the head classes.

References