SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning

Introduction

The human visual system has an ability to recognize new visual classes (for instance, greebles ) based on a few examples that is, currently, unmatched by computer vision. The development of computer-vision systems that can perform such few-shot learning is important, e.g., for developing systems that can recognize the millions of natural or man-made classes that appear in the world .

Few-shot learning is generally studied in a learning setting in which the visual-recognition system is first trained to recognize a collection of base classes from a large number of training examples. Subsequently, the system receives a small number of training examples (so-called “shots”) for a few novel visual classes that it needs to recognize thereafter. In order to be robust to overfitting, a successful few-shot learning model must efficiently re-use what it learned from training on the base classes for the novel classes.

Many current few-shot learners extract image features using a convolutional network, and use a combination of meta-learning and nearest-neighbor classification to perform the recognition . Prior studies suggest that using meta-learning outperforms “vanilla” nearest neighbor classification .

This study challenges the status quo by demonstrating that nearest-neighbor classifiers can achieve state-of-the-art performance on popular few-shot learning benchmarks without meta-learning. Specifically, we find that applying simple feature transformations on the features before nearest-neighbor classification leads to very competitive few-shot learning results. For example, we find that a nearest-neighbor classifier that uses DenseNet features to which mean subtraction and L2-normalization are applied outperforms a long list of recent, arguably more complex few-shot learning approaches on the popular miniImageNet and tieredImageNet benchmarks (see Table 1 and 2). These observations generalize to other convolutional network architectures . We refer to our few-shot learner as SimpleShot. We hope to re-establish nearest-neighbor classification as an obvious but competitive baseline for few-shot learning.

Nearest Neighbors for Few-Shot Learning

Denoting an image by $\mathbf{I}$, we assume we are given a training set, $\mathcal{D}_{\text{base}}=\{(\mathbf{I}_{1},y_{1}),\dots,(\mathbf{I}_{N},y_{N})\}$, that contains $N$ labeled images from $A$ base classes; that is, $y_{n}\in\{1,\dots,A\}$. Furthermore, we assume we are given a support set $\mathcal{D}_{\text{support}}$ of labeled images from $C$ novel classes, where each novel class has $K$ examples. The goal of few-shot learning is to construct a model that accurately recognizes the $C$ novel classes. This learning setting is referred to as the $K$-shot $C$-way setting.

In multi-shot settings, we use a nearest-centroid approach. Specifically, we compute the averaged feature vector (centroid) for each class in $\mathcal{D}_{\text{support}}$ and treat each of the centroids as a one-shot example for the corresponding class. We then apply Equation 1 on the centroids.

1 Feature Transformations

In this study, we use the Euclidean distance, $d(\hat{{\bm{x}}},\hat{{\bm{x}}}^{\prime})=\|\hat{{\bm{x}}}-\hat{{\bm{x}}}^{\prime}\|_{2}$, as the distance measure for nearest-neighbors classification. We only consider two feature transformations that are well-established and may be considered trivial but, empirically, we find that they can have a positive effect on the accuracy of the SimpleShot few-shot learner.

We compute the mean feature vector on the base classes, $\bar{{\bm{x}}}=\frac{1}{|\mathcal{D}_{\text{base}}|}\sum_{{\bm{x}}\in\mathcal{D}_{\text{base}}}{\bm{x}}$, and subtract it from a feature vector $\hat{{\bm{x}}}$ to normalize it: $\hat{{\bm{x}}}\leftarrow\hat{{\bm{x}}}-\bar{{\bm{x}}}$. Centering (or mean subtraction) in itself does not alter Euclidean distances between feature vectors, but can become effective in combination with L2-normalization.

L2-normalization (L2N).

Experiments

Following prior work, we measure the efficacy of feature transformations in nearest-neighbor classifiers for few-shot learning in a series of image-recognition experiments.Code at https://github.com/mileyan/simple_shot.

The miniImageNet dataset is a subset of ImageNet that is commonly used to study few-shot learning. The dataset contains 100 classes and has a total of 600 examples per class. Following and subsequent work, we split the dataset to have 64 base classes, 16 validation classes, and 20 novel classes. Following and subsequent studies, we resize the images to $84\times 84$ pixels via rescaling and center cropping.

We also perform experiments on the tieredImageNet dataset , which is also constructed from ImageNet but contains 608 classes. The dataset is split into 351, 97, and 160 classes for base, validation, and novel classes, respectively. The class split is performed using WordNet to ensure that all the base classes are semantically unrelated to the novel classes. Again, we resize images to $84\times 84$ pixels.

Following , we also perform experiments on the CIFAR-100 dataset, which contains 100 image classes. Each of the classes in the dataset has 600 images of size $32\times 32$ pixels. We follow and split the classes into 60 base, 20 validation, and 20 novel classes.

Evaluation protocol.

Following , we measure the accuracy of SimpleShot and the other few-shot learners by drawing 10,000 $K$-shot $C$-way tasks from the novel classes: each task has $C$ novel classes and $K$ labeled (support) images and 15 test (query) images per class. Following prior work, we focus on one-shot and five-shot, five-way tasks.

We average observed accuracies over all test images and over all the tasks, and report the resulting average accuracy and $95\%$ confidence interval.

Model and implementation details.

We evaluate our methods using five different convolutional-network architectures as the basis for the feature-generating function $f_{\theta}(\mathbf{I})$. We study five different network architectures:

Four-layer convolutional networks (Conv-4): We follow to implement this baseline model.

Wide residual networks (WRN-28-10) : We follow and use the architecture with 28 convolutional layers and a widening factor of 10.

Dense convolutional networks (DenseNet-121) : We use the standard 121-layer architecture but remove the first two down-sampling layers (i.e., we set their stride to $1$) and change the first convolutional layer to use a kernel of size $3\times 3$ (rather than $7\times 7$) pixels.

Residual networks (ResNet-10/18) : We use the standard 18-layer architecture but we remove the first two down-sampling layers and we change the first convolutional layer to use a kernel of size $3\times 3$ (rather than $7\times 7$) pixels. Our ResNet-10 contains 4 residual blocks; the ResNet-18 contains 8 blocks.

MobileNet : We use the standard architecture for ImageNet but, again, we remove the first two down-sampling layers from the network.

We train all networks for 90 epochs from scratch using stochastic gradient descent to minimize the cross-entropy loss of $A$-way classification ($A$ is the number of base classes). We perform the data augmentation proposed in . We set the initial learning rate to $0.1$ and use a batch size of $256$ images. On miniImageNet, We shrink the learning rate by $10$ at 45 and 66 epoch respectively. On tieredImageNet, we divide the learning rate by $10$ after every 30 epochs. We perform early stopping according to the one-shot five-way accuracy (measured using SimpleShot (L2N)) on the validation classes.

Feature transformations.

We evaluate the effectiveness of three feature transformations in our experiments:

CL2N: Centered and then L2-normalized features.

These transforms are followed by nearest-neighbor classification using the Euclidean distance measure.

Comparison.

We compare our baselines to a range of state-of-the-art few-shot learners . We do not compare to approaches that were developed for semi-supervised and transductive learning settings, as such approaches use the statistics of query examples or statistics across the few-shot tasks. We note that the network architectures used in prior studies may have slight variations; we have tried our best to eliminate the effect of such variations on our observations as much as possible.For example, we report results for ResNet-10 models because it is the shallowest ResNet architecture used in prior work on few-shot learning.

2 Results

Table 1, 2, and 3 present our results on miniImageNet, tieredImageNet, and CIFAR-100, respectively. In line with prior work, we observe that nearest-neighbor classifiers using “vanilla” Euclidean distance (UN) do not perform very well. However, simply applying L2-normalization (L2N) consistently leads to accuracy gains of at least $3\%$ on these datasets. Subtracting the mean before L2-normalization (CL2N) leads to another improvement of $1\!-\!3\%$.

Our SimpleShot nearest-neighbor / nearest-centroid classifiers achieve accuracies that are comparable with or better than the state-of-the-art. For example, on the miniImageNet dataset, our simple methods obtain the highest one-shot and five-shot accuracies for three of five network architectures.

We perform a simple experiment measuring the effectiveness of feature transformations at various stages of convolutional-network training. We train a DenseNet on miniImageNet for 90 epochs, and measure the one-shot five-way accuracy on 10,000 tasks sampled from the validation classes after each epoch. The results of this experiment are shown in Figure 1: they show that nearest-neighbor classifiers using C2LN feature transformation consistently outperform their UN and L2N counterparts. This suggests that our observations on the role of feature transformations do not depend on how long the network is trained.

We also investigate the effect of feature transformations on more complex few-shot learning algorithms. Specifically, we trained a Conv-4 architecture with the ProtoNet loss, which uses unnormalized Euclidean distances. After training, we apply feature transformations before computing pairwise Euclidean distances between features in a nearest-neighbor approach. Table 4 presents the results of this experiment, which shows that CL2N normalization can also improve the performance of ProtoNet.

Conclusion

We analyzed the effect of simple feature transformations in nearest-neighbor classifiers for few-shot learning. We observed that such transformations — in particular, a combination of centering and L2-normalization — can improve the quality of the representation to a degree that the resulting classifiers outperforms several state-of-the-art approaches to few-shot learning. We hope that the SimpleShot classifiers studied in this paper will be used as a competitive baseline in future studies on few-shot learning.

The authors thank Han-Jia Ye for helpful discussions. Y.W. and K.Q.W. are supported by grants from the NSF (III-1618134, III-1526012, IIS-1149882, IIS-1724282, and TRIPODS-1740822), the Bill and Melinda Gates Foundation, and the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875); and are also supported by Zillow, SAP America Inc., and Facebook.

References

Appendix A Meta-iNat Results

We also investigate the role of feature transformations in SimpleShot on the long-tailed iNaturalist dataset . Following the meta-iNat benchmark , we split the dataset to have 908 base classes and 227 novel classes. We follow the evaluation setup of and perform 227-way multi-shot evaluation. (In the meta-iNat benchmark, the number of shots varies per class.) We train all networks for 90 epochs using stochastic gradient descent. We set the initial learning rate to be $0.1$ and batch size to be $256$. We scale the learning rate by $0.1$ after every $30$ epochs.

The results of our meta-iNat experiments with SimpleShot are presented in Table 5. The table reports the averaging the accuracy on each class over all test classes (per class) and the average accuracy over all test images (mean). To the best of our knowledge, our highest accuracy of 62.13% (per class) and 65.09% (mean) is the current state-of-the-art on the meta-iNat benchmark. Figure 2 shows the absolute accuracy improvement (in %) of each of the classifiers compared to the baseline nearest-neighbor classifier without feature normalization (UN). In line with prior experiments, L2-normalization (L2N) leads to accuracy improvements in few-shot learning. Different from the other experiments, centering after L2-normalization (CL2N) does not improve the accuracy of SimpleShot further.