Deep Metric Transfer for Label Propagation with Limited Annotated Data

Introduction

We address the problem of object recognition from a very small amount of labeled data. This problem is of particular importance when limited labels can be collected due to either time or financial constraints. Though this is a difficult challenge, we are encouraged by evidence from cognitive science suggesting that infants can quickly learn new concepts from very few examples .

Many recognition problems in computer vision are concerned with learning on few labeled data. Semi-supervised learning, transfer learning, and few-shot recognition all aim to achieve fast generalization from few examples, by leveraging unlabeled data or labeled data from other domains.

The fundamental difficulty of this problem is that naive supervised training with very few examples results in severe over-fitting. Because of this, prior work in semi-supervised learning rely on strong regularizations such as augmentations , temporal consistency , and adversarial examples to improve performance. Some related works in few-shot learning do not even refine an online classifier. Instead, they simply apply the similarity metric learned from training categories to new categories without adaptation. Meta-learning seeks to optimize an online parametric classifier with few samples, but under the assumption that just a few steps of optimization will lead to effective generalization with less overfitting. These approaches indirectly address the inherent problem of limited training data.

In this paper, we propose a new framework of label propagation via metric transfer to tackle the problem of limited training data. We propagate labels to an unlabeled dataset, so that training a supervised model with great learning capacity no longer faces over-fitting. This approach is related to work on “pseudo-labeling” , where the model is bootstrapped from limited data and trained on the new data/label pairs it infers. However, that is unlikely to work well when the labeled data is scarce, since the initial model is likely to be poor. Instead of bootstrapping, our work transfers the metric learned from another related domain, and thus provides a much better generalization ability.

Our approach works with three data domains: a source domain to learn a similarity metric, few labeled examples to define the target problem, and an unlabeled dataset in which to propagate labels. As in Figure 1, we first learn a similarity metric on the source domain, which can be either labeled or unlabeled. Supervised learning or unsupervised (self-supervised) learning is used to learn the metric accordingly. Then, given few observations of the target problem, we propagate the labels from these observations to the unlabeled dataset using the metric learned in the source domain. This creates an abundance of labeled data for learning a classifier. Finally, we train a standard supervised model using the propagated labels.

The main contribution of this work is the metric transfer approach for label propagation. By studying different combinations of metric pretraining methods (e.g. unsupervised, supervised) and label propagation algorithms (e.g. nearest neighbors, spectral clustering), we find that our metric transfer approach on unlabeled data is general enough to work effectively for many settings. For semi-supervised learning on CIFAR10 and ImageNet, we obtain an absolute $20\%$ improvement over the state-of-the-art when labeled data is limited ($5-10$ labels per category). We also achieve a $10\%$ improvement on transferring representations from ImageNet to CIFAR10 for transfer learning, and $6\%$ improved performance for few-shot recognition on the mini-ImageNet benchmark.

Due to this generic framework, our work also brings individual insights into the respective tasks we studied: 1) for semi-supervised learning, algorithms may better develop from unsupervised learning, as opposed to using unlabeled data for regularization. 2) for transfer learning, we propose an alternative method for transferring knowledge other than the dominant finetuning approach. 3) for few-shot recognition, in certain scenarios, unlabeled data in the target domain is more beneficial than labeled data in the source domain.

Related Work

Large-scale Recognition. To solve a computer vision problem, it has become a common practice to build a large-scale dataset and train deep neural networks on it. This philosophy has achieved unprecedented success on many important computer vision problems . However, constructing a large-scale dataset is often time-consuming and expensive, and this has motivated work on unsupervised learning and problems defined on few labeled samples.

Semi-supervised Learning. Semi-supervised learning is a problem that lies in between supervised learning and unsupervised learning. It aims to make more accurate predictions by leveraging a large amount of unlabeled data than by relying on the labeled data alone. In the era of deep learning, one line of work leverages unlabeled data through deep generative models . However, training of generative models is often unstable, making it tricky to work with recognition tasks. Recent efforts on semi-supervised learning focus on regularization by self-ensembling through consistency loss, such as temporal ensembling , adversarial ensembling , teacher-student distillation , and cross-view ensembling . The pseudo-labeling approach initializes a model on a smalled labeled dataset and bootstraps on the new data it predicts. This tends to fail when the labeled set is small.

Our work is most closely related to the transductive approaches . Prior work in computer vision shows that label propagation can work well with handcrafted GIST descriptors. We bring it to the context of deep learning, and demonstrate that metric transfer may further improve the accuracy of label propagation.

Few-shot Recognition. Given some training data in training categories, few-shot recognition requires the classifier to generalize to new categories from observing very few examples, often 1-shot or 5-shot. A body of work approaches this problem by offline metric learning , where a generic similarity metric is learned on the training data and directly transferred to the new categories using simple nearest neighbor classifiers without further adaptation. Recent works on meta-learning take a learning-to-learn approach using online algorithms. In order not to overfit to the few examples, they develop meta-learners to find a common embedding space, which can be further finetuned with fast convergence to the target problem. Recent works using meta-learning consider the combined problem of semi-supervised learning and few-shot recognition, by allowing access to unlabeled data in few-shot recognition. This drives few-shot recognition into more realistic scenarios. We follow this setting as we study few-shot recognition.

Transfer Learning. Since the inception of the ImageNet challenge , transfer learning has emerged almost everywhere in visual recognition, such as in object detection and semantic segmentation , by simply transferring the network weights learned on ImageNet classification and finetuning on the target task. When the pretraining task and the target task are closely related, this tends to generalize much better than training from scratch on the target task alone. Domain adaptation seeks to address a much more difficult scenario where there is a large gap between the inputs of the source and target domains , for example, between real images and synthetic images. What we study in this paper is metric transfer. Different from prior work that employ metric transfer just to reduce the distribution divergence of different domains, we use metric transfer to propagate labels. Through this, we show that metric propagation is an effective method for learning with small data.

Approach

To deal with the shortage of labeled data, our approach is to enlarge it by propagating labels from annotated images to unlabeled data using the similarity metric between data pairs. The creation of much more labeled data enables us to train deep neural networks to their full learning capacity.

Our framework works on three data domains: the source domain $\mathcal{S}$, the target domain $\mathcal{T}$, and additional unlabeled data $\mathcal{U}$. The source domain $\mathcal{S}$ can be labeled or unlabeled with abundant data, and it is used to learn a generic similarity metric between data pairs. The target domain $\mathcal{T}$ only has few labeled data, but it defines the problem we want to optimize. The unlabeled data $\mathcal{U}$ is the resource in which to propagate labels, and may potentially contain similar classes to the task defined in $\mathcal{T}$. It may or may not have overlapping classes with $\mathcal{S}$.

The approach we propose in this paper is very general, suggesting that a spectrum of metric pretraining and label propagation algorithms can all work well in this framework. Below we introduce our method in details, and overview several metric learning and label propagation methods we used for our experiments.

The source domain $\mathcal{S}$ is used for pretraining a similarity metric between data pairs. Ideally, we desire the metric to capture the inherent structure in the target domain $\mathcal{T}$, so that transferring labels from $\mathcal{T}$ is reliable and useful. For this to happen, we usually hold some prior knowledge about the source $\mathcal{S}$ and the target $\mathcal{T}$. For example, the source domain is sampled from the same distribution as the target domain, but is completely unannotated, or the source domain is annotated with a different task but is closely related to the target. Formally, a similarity metric $s_{ij}$ between data $x_{i}$ and $x_{j}$ can be defined as

where $f$ is the similarity function to be learned. In this work, we use deep neural networks as a parametric model of this similarity function. The metric can be trained with either supervised or unsupervised methods, depending on whether labels are given in the source domain $\mathcal{S}$. We briefly review the training algorithms as follows.

Unsupervised Metric Pretraining Recently, there has been growing interest in unsupervised learning and self-supervised learning. Different algorithms are based on different data properties (e.g. color , context , motion ) and thus may vary in performance on the target task we may want to transfer. However, it is not our intent to give a comprehensive comparison over various methods and choose the best one. Instead, we show that general unsupervised transfer is beneficial for label propagation and leads to improved performance.

In this work, we utilize two unsupervised learning methods: instance discrimination and colorization . For instance discrimination, we treat each instance as a class, and maximize the probability of each example belonging to the class of itself,

For colorization, the idea is to learn a mapping from grayscale images to colorful ones. Following the original paper , instead of predicting raw pixel colors, we quantize the color space into soft bins $q$, and use the cross-entropy loss on the soft bins,

where $h,w$ are spatial indices. We follow previous work for applying ResNet to colorization, where we use a base network to map inputs to features, and a head network of three convolutional layers to convert features to colors. Since colorization does not automatically output a metric, we use the Euclidean distance on the features from the base network to measure similarity.

Supervised Metric Pretraining In some scenarios, we have access to a labeled dataset, such as PASCAL VOC and ImageNet, having commonalities with the target task. Traditional metric learning with supervision minimizes the intra-class distance and maximizes the inter-class distance of the labeled samples. For this purpose, many types of loss functions such as contrastive loss, triplet loss , and neighborhood analysis have been proposed. In this work, we use neighborhood analysis to learn our metric. Concretely, we maximize the likelihood of each example being supported by other examples belonging to the same category,

2 Label Propagation

Given a target $\mathcal{T}$ represented by a small number of labeled examples, and a unlabeled set $\mathcal{U}$, we propagate labels from $\mathcal{T}$ to $\mathcal{U}$ using the similarity function $f(\cdot)$ learned from $\mathcal{S}$. Suppose $\mathcal{T}=\{(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n_{t}},y_{n_{t}})\}$, and $\mathcal{U}=\{x_{n_{t}+1},x_{n_{t}+2},...,x_{n_{t}+n_{u}}\}$, where $n_{t}$ and $n_{u}$ are the number of images in $\mathcal{T}$, $\mathcal{U}$ respectively. Label $y_{i}$ is represented as a vector with the ground-truth class element set to $1$ and the others set to $-1$. We consider two propagation algorithms.

Naive Nearest Neighbors A straightforward propagation approach is to vote for the class of an unlabeled sample based on its similarity to each of the exemplars in the target set $\mathcal{T}$. For an unlabeled example $x_{u}\in\mathcal{U}$, we calculate its logits $z_{u,c}$ for every class $c$,

where $I(\cdot)$ is the indicator function, $W_{i,u}=\text{exp}\left(f(x_{i},x_{u})\right))$ denotes the similarity between example $i$ and $u$, and $n_{t,c}$ is the number of labeled images available for class $c$.

The nearest neighbor propagation method is essentially a one-step random walk where the similarity metric acts as the transition matrix and the indicator function acts as the initial distribution. The effectiveness of such one-step propagation depends heavily on the quality of the similarity metric.

In general, it is hard to learn such a metric well, especially when limited supervision is available, because of the visual diversity of images. Figure 2 (left) shows a typical similarity matrix computed from unsupervised features. Data points in the similarity matrix are sparsely connected, thus limiting the one-step label propagation approach.

Constrained Spectral Clustering Constrained spectral clustering may potentially relieve such a problem. Instead of propagating labels by one step as in the naive nearest neighbor approach, constrained spectral clustering propagates labels through multiple steps by taking advantage of structure within the unlabeled dataset. It computes a spectral embedding from the original similarity metric, which is then used as the new metric for label propagation. The spectral embedding is formulated as

where $\lambda_{j}$ and $e_{j}$ are the eigenvalues and eigenvectors of the normalized Laplacian in ascending order. The Laplacian matrix $L_{\text{sym}}$ is derived from the original similarity metric as $L_{\text{sym}}=I-D^{-1/2}WD^{-1/2}$, with degree matrix $D=\text{diag}(d)$ and $d_{i}=\sum_{j}W_{ij}$. Parameter ${\eta}$ is the total number of eigen components used.

Due to its globalized nature, spectral clustering is able to pass messages between distant areas, which is in contrast to the local behavior of the naive nearest neighbors approach. The embedded metric is usually densely connected and better aligned with object classes, as illustrated in Figure 2 (right). Using the same voting approach as in Eqn (5), labeled propagation can be more accurate than using the original raw similarity metric.

Constrained spectral clustering is also efficient. By following the common practice of using $k$-nearest neighbors to build the similarity graph , propagating labels to $50k$ images takes about 10 seconds on a regular GPU.

3 Confidence Weighted Supervised Training

Given the logits $z_{i}$, the pseudo label $\hat{y}_{i}$ is estimated as

With the estimated pseudo labels on the unlabeled data, we have considerably more data for training a classifier. However, the pseudo labels may not be accurate, and directly using these labels may lead to degraded performance. For example, not all the data in the unlabeled set are related to the target problem. Here, we devise a simple weighting mechanism to compensate for inaccurate labels.

Given the logits $z_{i}$ produced by the label propagation algorithm, we first normalize it into a probabilistic distribution,

where $c$ indexes the dimension of categories, and the temperature $\tau$ controls the sharpness of the distribution. We then define the confidence measure $\alpha_{i}$ of the pseudo label as the difference between the maximum response and the second largest response,

A high value of $\alpha_{i}$ indicates a confident estimate of the pseudo label, and a low value of $\alpha_{i}$ indicates an ambiguous estimate. In Figure 3, we measure the accumulated accuracy of pseudo labels on validation data sorted by this confidence. It can be seen that our confidence measure gives a good indication of the quality of pseudo labels.

where $\hat{y}_{i}$ is the pseudo label for example $i$, and $p(\cdot)$ is the softmax probability output of the classification network.

In practice, since some pseudo labels have relatively low confidence, e.g. $\alpha<0.01$, and thus contribute negligibly to the overall learning criterion, we may safely discard those examples to speed up learning.

Experiments

Through experiments, we show that, with unlabeled data, metric propagation is able to effectively label lots of data when little labeled data is given. We verify our approach on semi-supervised learning, where an unsupervised metric is transferred, and on transfer learning, where supervised metrics generalize across different data distributions, and on few-shot recognition, where the metric can generalize across open-set object categories. While studying few-shot recognition, we leverage an extra unlabeled data for label propagation, which is also known as semi-supervised few-shot recognition .

Our approach has two major hyper-parameters: the number of the eigenvectors $\eta$ for spectral clustering and the temperature $\sigma$ controlling the confidence distribution. Different parameter settings may slightly change the performance. We use $\eta=200$ and $\sigma=40$ across the experiments. A detailed analysis is provided in the supplementary materials.

We follow a recent evaluation paper , which gives a comprehensive benchmark for state-of-the-art semi-supervised learning approaches. A majority of our ablation studies are conducted on CIFAR10 , while we also test our method on ImageNet. On CIFAR10, we use the same Wide-ResNet architecture with 28 layers and a width factor of 2. We report performance varying the number of labeled examples from $50$ to $8,000$ of $50,000$ examples.

For training our model, we pretrain the metric using the unlabeled split, and propagate labels to the same unlabeled set. This means $\mathcal{S}=\mathcal{U}$ in our framework. We use SGD for optimization with an initial learning rate of 0.01 and a cosine decay schedule. We fix the total number of optimization iterations to $200K$ as opposed to fixing optimization epochs, because it gives more consistent comparisons when the number of labeled data varies.

Our label propagation algorithm needs a pretrained similarity metric to guide it. The pretrained metric can be learned by supervised methods using limited labeled data, or by unsupervised methods using large-scale unlabeled data. Here, we consider three metric pretraining methods:

supervised bootstrapping on limited labeled data.

self-supervised learning by image colorization .

unsupervised learning by instance discrimination .

We train the models using the optimal parameters for each pretraining method. Then we use cosine similarity in the feature space for propagating labels to the unlabeled data.

In Table 1, we evaluate the quality of pseudo labels as the mean average precision (mAP) sorted by the confidence as in Figure 3. Table 2 lists the final semi-supervised recognition accuracy. We can see that both unsupervised methods generalize much better than the supervised bootstrapping method most of the time, until the labeled set is relatively large with 4000 labels. This confirms our claim that unsupervised transfer is the key for label propagation. For the unsupervised methods, non-parametric metric learning performs better than colorization, probably because it explicitly learns a similarity metric. We also include the result of the naive baseline which trains from scratch using limited labeled data without label propagation.

Study of different label propagation schemes.

Given the pretrained metrics, there are various ways to transfer the metrics. We consider three possible solutions:

no propagation, only transfer network weights.

The first baseline is a common practice, which basically transfers the network weights and then finetunes on the labeled data. The second is much weaker than the third because it only considers one-hop distances, without taking into account the similarities between unlabeled pairs.

The results are summarized in Table 1 and Table 2. Compared to the state-of-the-art performance in Table 4, even a simple finetuning approach outperforms the state-of-the-arts when the labeled data is small. For example, by finetuning from instance discrimination, we achieve $62.46\%$ with $250$ labeled data, significantly outperforming the state-of-the-art result of $47.07\%$. This suggests that unsupervised pretraining generally improves semi-supervised learning.

When unlabeled data is used for label propagation, metric transfer can be much stronger than just weight transfer, improving the performance to $71.26\%$ with $250$ labeled data. It is also evident that the spectral clustering method performs better than weighted nearest neighbors because of its globalization behavior.

Scalability to large network architectures.

In contrast to prior methods which face over-fitting issues, our approach can easily scale to larger network architectures. Here, we keep all the learning parameters unchanged, and experiment with a wider version of Wide-ResNet-28 with a width factor of 10. We consider a state-of-the-art method mean-teacher for comparison. In Table 3, mean-teacher only shows a limited improvement of about $2-3\%$. Our method enjoys consistently significant gains from a larger network on all the testing scenarios. It achieves an unprecedented $73.13\%$ accuracy using only $50$ labels with Wide-ResNet-28-10.

Comparison to the state-of-the-art on CIFAR10.

We compare our approach to state of the art methods in Figure 4. Ours is particularly stronger when the labeled set is small, but this advantage diminishes as the labeled set grows. However, as most prior approaches focus on self-ensembling, ours is orthogonal to them. We examine the complementarity of our method by combining it with each of the prior approaches. To do so, we generate our most confident $10K$ pseudo labels (about $20\%$ of the full data), and use it as ground-truth for the other algorithms. For fair comparisons, we run public codehttps://github.com/brain-research/realistic-ssl-evaluation with our generated pseudo labels. In Table 4, combining our approach leads to improved performance for all of the methods.

Comparison to the state-of-the-art on ImageNet.

We notice that few literature report semi-supervised classification performance on ImageNet consistently. In this paper, we consider finetuning from an unsupervised model trained with instance discrimination as our baseline. We vary the number of labeled examples from $1\%$ to $4\%$ of the entire ImageNet. We use ResNet-50 to pretrain the unsupervised model, and split the dataset into 10 chunks for spectral clustering to speed up the computation. In Table 5, finetuning from unsupervised model significantly improves upon training from scratch, and our label propagation approach outperforms the finetuning approach. Notable, ours is $18\%$ better when $1\%$ labeled data is available.

2 Transfer Learning

We also examine whether the proposed metric transfer can work across different data distributions. We pretrain the metric on the source $\mathcal{S}$ ImageNet, and transfer it to the unlabeled $\mathcal{U}$ CIFAR10. For this, we study supervised and unsupervised pretraining for transfer learning.

Transferring from labeled ImageNet. We resize ImageNet images to a resolution of $32\times 32$ and pretrain the metric on them by supervised learning. We keep the network architecture WideResNet-28-2 for meaningful comparison with the semi-supervised settings in Sec 4.1. This obtains an accuracy of $42\%$ on the ImageNet validation set. Then we transfer the metric to CIFAR10. This transfer is conducted by network finetuning and by metric propagation. In Table 6, we can see that simple network finetuning can reach the best results obtained in the semi-supervised settings of the previous subsection. By using label propagation with spectral clustering, we can observe a large improvement, yielding $86.07\%$ accuracy with just $250$ labeled images. This illustrates the generality of our metric transfer approach, where supervised transfer can also take advantage of unlabeled data to improve generalization.

Transferring from unlabeled ImageNet. Instead of supervised training which encodes prior knowledge about object categories, we treat ImageNet images as unlabeled and repeat the previous experiment. Different from the earlier unsupervised experiments, this setting involves substantially more unlabeled data, which could potentially lead to a better unsupervised metric. However, our results suggest otherwise. When propagating to CIFAR10, the unsupervised metric learned from ImageNet is inferior to the metric learned from CIFAR10. This is possibly due to the data distribution gap between CIFAR10 and ImageNet. Nevertheless, our unsupervised transfer from ImageNet still surpasses the state-of-the-art in the semi-supervised setting when labeled samples are limited.

3 Few-Shot Recognition

Few-shot recognition targets a more challenging scenario, the generalization across object categories (a.k.a. open-set recognition). Originally, the problem is defined with numerous labeled examples in a source dataset, and few examples in the target categories. Recent works also explore the scenario where extra unlabeled data is available for this problem. This fits into our framework for studying label propagation via metric transfer.

We follow the protocols in for conducting the experiments, because it introduces distractor categories in the unlabeled set. The experiments are evaluated on the mini-ImageNet dataset, consisting of a total of $100$ categories, with $64$ for training, $16$ for validation and $20$ for testing. Images in each category are split into $40\%$ as labeled, and $60\%$ as unlabeled. Training uses only the labeled split in the training categories. During evaluation, a testing episode is constructed by sampling few-shot labeled observations from the labeled split in the testing categories, and all of the unlabeled images in all the testing categories. A testing episode requires the model to find useful information in the unlabeled set to aid recognition from the few-shot observations. Unlike , which includes five distractor categories in the unlabeled set, we consider all $20$ categories in the testing set, which better reflects practical scenarios. We test $300$ episodes and report the results.

We follow prior work by using a shallow architecture with four convolutional layers and a final fully connected layer. Each convolutional layer has 64 channels, interleaved with ReLU, subsampling and a batch normalization layer. Images are resized to $84\times 84$ to train the model. We use the spectral embedding approach for label propagation. During online training, we use an initial learning rate of $0.01$ with a total of 30 epochs and decrease the learning rate to be $5$ times smaller after $20$ epochs.

Transfer from supervised models. We use a recent supervised metric learning approach SNCA as the baseline. After label propagation and finetuning on the new data, our supervised propagation obtains a significant boost of $6\%$ over SNCA. Prior work improves upon its baselines, but fails to make further improvement because of limited training data. In Figure 5, we visualize the top retrievals from the unlabeled set in the one-shot scenario. These retrievals not only belong to the same class as the ground truth, but their diversity facilitates a strong classifier.

Transfer from unsupervised models. We also investigate pretraining the metric without labels, using instance discrimination for learning the metric. Surprisingly, in Table 7, our unsupervised propagation obtains better performance than the offline metric learning approach with annotations , by $0.5\%$ in 1-shot recognition and $2\%$ for 5-shot. This suggests that leveraging unlabeled data in the target problem may possibly be more beneficial than using labeled samples in the source domain.

Discussions

The effectiveness of label propagation depends heavily on the learned metric, so advances in metric learning should lead to improved results. Since the prevalent pretraining methods in deep learning use softmax classification, we hope to draw more attention to pretraining networks with metric learning.

Currently, we study metric pretraining and label propagation separately. It may be beneficial to formulate them jointly in an end-to-end framework.

Our algorithm takes advantage of the unlabeled dataset $\mathcal{U}$ to create more training data. The overall performance is affected by the relevance of image content in the unlabeled set $\mathcal{U}$ to that of the target $\mathcal{T}$, as this impacts the ability to effectively propagate labels.

Appendix A1 Ablations of Model Parameters

Our model depends on two parameters: the number of eigen components $\eta$ used for spectral clustering, and the temperature $\sigma$ used for controlling the confidence. We used $\eta=200$ and $\sigma=40$ in our main submission. In Figure 6, we show the effects of the two parameters respectively.

The number of eigenvectors $\eta$ works well in the range between $30$ and $200$. We can see a trade-off of the value $\eta$ for performance under various number of labeled samples. Smaller $\eta$ benefits very few labeled samples, while larger $\eta$ benefits comparably more labeled samples. For the temperature parameter $\sigma$, it is generally robust for a wide range of values between $10$ to $100$.

Appendix A2 Additional Visualizations

We provide more retrieval visualizations in the CIFAR10 and mini-ImageNet dataset in Figure 7 and Figure 8. For CIFAR10, we show the top retrievals for each class in the unlabeled set given $250$ labeled examples. For mini-ImageNet, we show the top retrievals in the 5-class 1-shot scenario.

References