CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

Introduction

One of the key factors that drive recent advances in large-scale image recognition is massive collections of labeled images like ImageNet and COCO . However, it is normally expensive and time-consuming to collect large-scale manually labeled datasets. In practice, for fast development of new image recognition tasks, a widely used surrogate is to automatically collect noisy labeled data from Internet . Yet many studies have shown that label noise can affect accuracy of the induced classifiers significantly , making it desirable to develop algorithms for learning in presence of label noise.

Learning with label noise can be categorized by type of supervision: methods that rely on human supervision and methods that do not. For instance, some of the large-scale training data were constructed using classifiers trained on manually verified seed images to remove label noise (e.g. LSUN and Places ). Some studies for learning convolutional neural networks (CNNs) with noise also rely on manual labeling to estimate label confusion . The methods using human supervision exhibit a disadvantage in scalability as they require labeling effort for every class. For classification tasks with millions of classes , it is infeasible to have even one manual annotation per class. In contrast, methods without human supervision (e.g. model predictions-based filtering and unsupervised outliers removal ) are scalable but often less effective and more heuristic. Going with any of the existing approaches, either all the classes or none need to be manually verified. It is difficult to have both scalability and effectiveness.

In this work, we strive to reconcile this gap. We observe that one of the key ideas for learning from noisy data is finding “class prototypes” to effectively represent classes. Methods learn from manually verified seed images like and methods assume majority correctness like belong to this category. Inspired by this observation, we develop an attention mechanism that learns how to select representative seed images in a reference image set collected for each class with supervised information, and transfer the learned knowledge to other classes without explicit human supervision through transfer learning. This effectively addresses the scalability problem of the methods that rely on human supervision.

Thus, we introduce “label cleaning network” (CleanNet), a novel neural architecture designed for this setting. First, we develop a reference set encoder with the attention mechanism to encode a set of reference images of a class to an embedding vector that represents that class. Second, in parallel to reference set embedding, we also build a query embedding vector for each individual image and impose a matching constraint in training to require a query embedding to be similar to its class embedding if the query is relevant to its class. In other words, the model can tell whether an image is mislabeled by comparing its query embedding with its class embedding. Since class embeddings generated from different reference sets represents different classes where we wish the model to adapt to, CleanNet can generalize to classes without explicit human supervision. Fig. 1 illustrates the end-to-end differentiable model.

As the first step of this work, we demonstrate that CleanNet is an effective tool for label noise detection. Simple thresholding based on the similarity between the reference set and the query image lead to good results compared with existing methods. Label noise detection not only is useful for training image classifiers with noisy data, but also has important values in applications like image search result filtering and linking images to knowledge graph entities.

CleanNet predicts the relevance of an image to its noisy class label. Therefore, we propose to use CleanNet to assign weights to image samples according to the image-to-label relevance to guide training of the image classifier. On the other hand, as a better classifier provides more discriminative convolutional image features for learning CleanNet, we refresh the CleanNet using the newly trained classifier. We introduce a unified learning scheme to train the CleanNet and image classifier jointly.

To summarize, our contributions include a novel neural architecture CleanNet that is designed to make label noise detection and learning from noisy data with human supervision scalable through transfer learning. We also propose a unified scheme for training CleanNet and the image classifier with noisy data. We carried out comprehensive experimentation to evaluate our method for label noise detection and image classification on three large datasets with real-world label noise: Clothing1M , WebVision , and Food-101N. Food-101N contains 310K images we collected from Internet with the Food-101 taxonomy , and we added “verification label” that verifies whether a noisy class label is correct for an imageFood-101N will be available at kuanghuei.github.io/CleanNetProject.. Experimental results show that CleanNet can reduce label noise detection error rate on held-out classes where no human supervision available by 41.5% compared to current weakly supervised methods. It also achieves 47% of the performance gain of verifying all images with only 3.2% images verified on an image classification task.

Related Work

Label noise reduction. Our method belongs to the category of approaches that address label noise by demoting or removing mislabeled instances in training data. One of the popular approaches is unsupervised outlier removal (e.g. One-Class SVM , UOCL , and DRAE ). Using this approach for label noise detection relies on an assumption that outliers are mislabeled. However, outliers are often not well defined, and therefore removing them presents a challenge . Another approach that also needs no human supervision is weakly supervised label noise reduction . For example, Thongkam et al. proposed a classification filtering method that learns an SVM from noisy data and removes instances misclassified by the SVM. Weakly supervised methods are often heuristic, and we are not aware of any large dataset actually built with these methods. On the other hand, label noise reduction using human supervision has been widely studied for dataset constructions. For instance, Yu et al. proposed manually labeling seed images and then training multilayer perceptrons (MLPs) to remove mislabeled images. Similarly, the Places dataset was constructed using an AlexNet trained on manually verified seed images. However, methods using human supervision exhibit a disadvantage in scalability as they require human supervision for every class to be cleansed.

Direct neural network learning with label noise. Some methods were developed for directly learning neural network with label noise . Azadi et al. developed a regularization method to actively select image features for training, but it depends on features pre-trained for other tasks and hence is less effective. Zhuang et al. proposed attention in random sample groups but did not compare with standard CNN classifiers, and thus is less practical. Methods proposed by Xiao et al. and Patrini et al. rely on manual labeling to estimate label confusion for real-world label noise. However, such labeling is required for all classes and much more expensive than simply verifying whether the noisy class labels are correct. Veit et al. proposed an architecture that learns from human verification to clean noisy labels, but their approach does not generalize to classes that are not manually verified as opposed to our method. Chen et al. , which relies on specific data sources, and Li et al. , which uses knowledge graph, could be difficult to generalize and thus are beyond the scope of this paper.

Transfer learning with neural network. There is a large body on literature of learning neural joint embeddings for transfer learning . Tsai et al. trained visual-semantic embeddings with supervised and unsupervised objectives using labeled and unlabeled data to improve robustness of embeddings for transfer learning. Recently Liu et al. and Tzeng et al. exploited adversarial objectives for domain adaptation. Inspired by , we also incorporate unsupervised objectives in this work.

Scalable Learning with Label Noise

We focus on learning an image classifier from a set of images with label noise using transfer learning. Specifically, assume we have a dataset of $n$ images, i.e., $X=\{(x_{1},y_{1}),...,(x_{n},y_{n})\}$, where $x_{i}$ is the $i$-th image and $y_{i}\in\{1,...,L\}$ is its class label, where $L$ is the total number of classes. Note that the class labels are noisy, means some of the images’ labels are incorrect.

In this section, we present the CleanNet, a joint neural embedding network, which only requires a fraction of the classes being manually verified to provide the knowledge of label noise that can be transferred to other classes. We then integrate CleanNet and conventional convolutional neural network (CNN) into one system for image classifier training with label noise. Specifically, we introduce the designs and properties of CleanNet in Section 3.1. In Section 3.3 we integrate CleanNet and the CNN into one framework for image classifier learning from noisy data.

The overall architecture of CleanNet is shown in Fig. 1. It consists of two parts: a reference set encoder and a query encoder. The reference set encoder $f_{s}(\cdot)$ learns to focus on representative features in a noisy reference image set, which is collected for a specific class, and outputs a class-level embedding vector. Since using all the images in the reference set is computationally expensive, we first create a representative subset, and extract one visual feature vector from each image in that subset to form a representative feature vector set, i.e., let $V_{c}^{s}$ denotes the representative reference feature vector set for class $c$ (reference feature set).

We explored two pragmatic approaches to select $V_{c}^{s}$. The first one is random sampling a subset from all images in class $c$ and extract features using a pre-trained CNN $f_{v}(\cdot)$ as shown in Fig. 1. The second approach is running K-means on the extracted features of all images in class $c$ to find K cluster centroids and use them as $V_{c}^{s}$. The K-means step is ignored in the figures. Since the K-means approach shows slightly better result on a held-out set, we choose it for experiments hereafter. We select 50 feature vectors to form $V_{c}^{s}$.

In parallel to reference set encoder, we also develop a query encoder $f_{q}(\cdot)$. Let $q$ denote a query image labeled as class $c$. The query encoder $f_{q}(\cdot)$ maps the query image feature $v^{q}=f_{v}(q)$ to a query embedding $\phi^{q}=f_{q}(v^{q})$. We impose a matching constraint such that the query embedding $\phi^{q}$ is similar to its class embedding $\phi_{c}^{s}=f_{s}(V_{c}^{s})$ if the query $q$ is relevant to its class label $c$. In other words, we decide whether a query is mislabeled by comparing its query embedding vector with its class embedding vector. Since the class labels are noisy, we can further mark up a query image and its class label by a manual “verification label”. The verification label for each image is defined as

Note that, to reduce human labeling effort, most of the verification labels are -1, means no human verification available.

The model learns the matching constraint from the supervision given by the verification labels, such that a query embedding is similar to its class embedding if the query image $q$ truly belongs to its class label, and transfer to different classes where no human verification available. In the following, we present how we build the reference set encoder, query encoder, and objectives for learning the matching constraint.

Reference set encoder. The architecture of the reference set encoder is depicted in Fig. 2. It maps a reference feature set $V_{c}^{s}$ for class $c$ to a class embedding vector $\phi_{c}^{s}$. First, a two-layer MLP projects each image feature to a hidden representation $h_{i}$. Next, we learn an attention mechanism to encode representative features to a fixed-length hidden representation as class prototype:

As shown in Eq. (4), the importance of each $h_{i}$ is measured by the similarity between $u_{i}$ and a context vector $u$. Similar to , the context vector $u$ is learned during training. Driven by the matching constraint, this attention mechanism learns how to pay attention on the most representative features for classes. This model learns from supervised information, i.e., the manual verification label, and adapts to other classes without explicit supervision. An example of this attention mechanism is shown in Fig. 3. Finally, a one-layer MLP maps the hidden representation to the class embedding $\phi_{c}^{s}$.

Query encoder. As illustrated in Fig. 1, we adopt a 5-layer autoencoder as the query encoder and incorporate autoencoder reconstruction error into learning objectives. Taking this strategy, as proposed in , forces the query embedding to preserve semantic information of all the classes including those classes without verification labels, because images without verification label can now be used in training with this unsupervised objective. It has been proven effective for improving domain adaptation performance.

Given a query image feature vector $v^{q}$, the autoencoder maps $v^{q}$ to a hidden representation $\phi^{q}$ and seek to reconstruct $v^{q}$ from $\phi^{q}$. The reconstruction error is defined as

where $r(v^{q})$ is the reconstructed representation.

Learning objectives based on matching constraint. With the supervision from human verification labels, the similarity between class embedding $\phi_{c}^{s}$ and query embedding $\phi^{q}$ is maximized if a query is relevant to its class label ($l=1$); otherwise the similarity is minimized ($l=0$). We adopt the cosine similarity loss with margin to impose this constraint:

where $\cos(\cdot)$ is the normalized cosine similarity, $\omega$ is negative sample weight for balancing positive and negative samples, and $\rho$ is the margin set to 0.1 in this work. The case $l=-1$ is ignored in the loss function since this supervised objective only utilizes query images with verification label.

On the other hand, images without verification label can also be utilized to learn the matching constraint. Similar to , we introduce an unsupervised self-reinforcing strategy that applies pseudo-verification to images without verification label. To be specific, a query is treated as relevant if $\cos(\phi^{q},\phi_{c}^{s})$ is larger than the margin $\rho$:

where $\rho$ is the same margin as in Eq. (6). From Eq. (7) and Eq. (8), we can see that for queries that are initially treated as relevant, the model learns to further push up the similarity between queries and reference sets; for queries that are initially treated as irrelevant, they are ignored.

Total loss. To summarize the training objectives, our model is learned by minimizing a total loss combining both supervised and unsupervised objectives:

where $\beta$ and $\gamma$ are selected through hyper-parameter search, and $t$ indicates whether a query image has verification label. $\beta$ and $\gamma$ are set to 0.1 in this work. During training, we randomly sample images without verification label as queries for a fraction of a mini-batch (usually $1/2$).

Note that the parameters of the attentional reference set encoder and the query encoder are tied across all classes so the information learned from classes that have human verification labels can be transferred to other classes that have no human verification label.

2 CleanNet for Label Noise Detection.

From a relevance perspective, CleanNet can be used to rank all the images with label noise for a class by cosine similarity $\cos(\phi^{q},\phi_{c}^{s})$. We can simply perform thresholding for label noise detection:

where $\delta$ is a threshold selected through cross-validation. We observe that the threshold is not very sensitive to different classes in most cases, and therefore we usually select an uniform threshold for all classes so that verification labels are not required for all classes for cross-validation.

3 CleanNet for Learning Classifiers

CleanNet predicts the relevance of an image to its noisy class label by comparing the query embedding of the image to its class embedding that represents the class. That is, the distance between two embeddings can be used to decide how much attention we should pay to a data sample in training the image classifier. Specifically, we assign attention weights on data samples based on the cosine similarity:

where $V_{c}^{s}$ is the reference image feature set that represents the prototype of class $y=c$. Eq. (12) defines a soft weighting on an image $x$ with noisy class label $y=c$. Similarly, we also define a hard weighting as

where $\delta$ is a threshold as in Eq. (11). In essence, hard weighting is equivalent to explicit label noise removal. With $w_{soft}$ or $w_{hard}$, we define the weighted classification learning objective as

where $H(x,y=c)$ is negative log likelihood:

Integrating CleanNet and the image classifier. Learning the image classifier relies on CleanNet to assign proper attention weights to data samples. On the other hand, better classifier provides more discriminative features which are critical for CleanNet learning. Therefore, we integrate CleanNet and the CNN-based image classifier into one framework for end-to-end learning of image classifiers with label noise. The overall architecture of this framework is illustrated in Fig. 4. The structure of a CNN-based image classifier is split into fully-connected layer(s) and convolutional layers $f_{cl}$ that can be used for feature extraction.

Alternating training. We adopt an alternative training scheme to learn the proposed classification system. At step 1, we first train a classifier from noisy data with all sample weights set to 1. At step 2, parameters of convolutional layers $f_{cl}$ are copied to feature extractor $f_{v}$ and a CleanNet is trained to convergence. At step 3, the classifier are fine-tuned using the sample weights proposed by CleanNet. A similar alternating process can continue till the classifier stops improving. For more iterations of learning classifier, we fix the convolutional layers and only fine-tune the fully-connected layers.

Experiments

Table 1 lists the statistics of the datasets.

Food-101N: We collect 310k images from Google, Bing, Yelp, and TripAdvisor using the Food-101 taxonomy, and avoid foodspotting.com where the original Food-101 was collected. The estimated noisy class label accuracy is 80%. We manually add 55k verification label for training and 5k for testing label noise detection. Image classification is evaluated on Food-101 test set.

Clothing1M : Clothing1M is a public large-scale dataset designed for learning from noisy data with human supervision. It consists of 1M images with noisy class labels from 14 fashion classes. The estimated accuracy of class labels is 61.54%. There are also three sets of images, with the size of 50k, 14k, 10k, respectively, which have correct class labels provided by human labelers – we call them clean sets. There are some images overlap between the three clean sets and the noisy set. For those overlapped images, we can then verify whether the noisy class label (as in the noisy set) is correct given the human labels on these images, and hence obtain verification labels for these images. Through this process, we obtain 25k and 7k verification labels for training and validation, respectively. The state of the art result of image classification on Clothing1M is reported in .

WebVision: WebVision contains 2.4M noisy labeled images crawled from Flickr and Google using the ILSVRC taxonomy . We conveniently verify noisy class labels using the Inception-ResNet-V2 model pre-trained on ILSVRC. Noisy class label of an image is verified as relevant if it falls in top-5 predictions. Otherwise, the noisy class label is marked as mislabeled. We randomly obtain 250 “pseudo-verification labels” for each class for training. For evaluating image classification, we use 50k WebVision validation set and 50k ILSVRC 2012 validation set.

2 Label Noise Detection

We first evaluate CleanNet for the task of label noise detection. The label noise detection problem can be viewed as a binary classification problem for each class, and hence the results and comparisons are reported in average error rate over all the classes. We compare with the following categories of existing baseline methods:

Supervised: Supervised methods learn a binary classification from verification labels for each class. We consider neural networks (2-layer MLP, used in for data construction), kNN, SVM, label prop , and label spread . We also explored MLPs of more layers but 2-layer shows the best results.

Unsupervised: We consider DRAE , the state of the art unsupervised outlier removal. Empirically, DRAE shows better results than one-class SVM .

Weakly supervised: Like unsupervised method, weakly supervised methods do not require verification labels. We compare with a widely used classification filtering method: we train a CNN model on noisy data and predict top-K classes for each training image. An image is classified as relevant to its class label if the class is in top-K predictions. Otherwise, it is classified as mislabeled. K is selected on the validation set.

We provide two additional baselines: naive baseline that treats all class labels as correct, and average baseline that simply averages reference features as a class embedding vector and use query feature as a query embedding vector.

CleanNet and all the baselines depend on a CNN to extract image features. We fine-tune the ImageNet pre-trained ResNet-50 models on noisy data, same as step 1 in the alternating training scheme, and extract the pool5 layer as image features. Implementations of kNN, SVM, label prop, and label spread are from scikit-learn . We re-implemented DRAE and MLP in our experimentation.

In the following, we will evaluate CleanNet for label noise detection under two scenarios: Full supervision: verification labels in all classes are available for learning CleanNet; Transfer learning: only a fraction of classes contains verification labels for learning CleanNet.

Full supervision. In Table 2, we report the label noise detection results in terms of average error rate over all the classes. CleanNet gives error rate of 9.61% on Food-101N and 15.91% on Clothing1M. Comparing to MLP at 10.42% on Food-101N and 16.09% on Clothing1M, we validate that CleanNet performs similar to the best supervised baseline. Comparing to classification filtering at 16.60% on Food-101N and 23.55% on Clothing1M, the results demonstrate effectiveness of adding verification labels for human supervision for label noise detection. CleanNet* denotes the results of CleanNet using image features extracted from the classifiers retrained with data cleansed by CleanNet, and shows improvements (6.99% on Food-101N and 15.77% on Clothing1M). However, improvements become negligible with more iterations.

Transfer learning. We choose Food-101N to demonstrate label noise detection with CleanNet under the setting of transfer learning, where verification labels in $n$ classes are held out for CleanNet (Lists of the held-out classes are available in the Food-101N dataset.). Here we also consider MLP that uses all verification labels and classification filtering that needs no verification labels. We ONLY evaluate the results on $n$ held-out classes to demonstrate the results on classes without explicit human supervision. The results are shown in Fig. 5. First, we observe that CleanNet can reduce label noise detection error rate on held-out classes where no human supervision available by 41.5% relatively ($n=10$) compared to classification filtering. CleanNet consistently outperforms classification filtering, the weakly-supervised baseline. We also observe that the result of CleanNet with 50/101 classes held out (11.02%) is still comparable to the result of MLP which is based on supervised learning (10.12%).

3 Learning Classifiers with Label Noise

In this subsection, we present experiments for learning image classification models with label noise using the proposed CleanNet-based learning framework. Experimentation in this section is based on ResNet-50.

Experiments on Food-101N. Table 3 lists the results on Food-101N using verification labels in all classes. We observe that the performance of smooth soft weighting ($w_{soft}$) (83.95%) without need for thresholding outperforms hard weighting ($w_{hard}$) (83.47%). Fig. 6 presents the results of image classification using the proposed CleanNet-based method when verification labels in $n$ classes are held out. For these $n$ held-out classes, the information needed for cleaning up the noisy class labels are transferred from other classes through CleanNet. It is observed that there are still 2.1% and 1.75% accuracy gain when 50/101 and 70/101 classes are held out. This validates that labeling effort on a small fraction of classes can still lead to significant gains.

Fig. 7 shows examples of predictions by CleanNet. The cosine similarity score between the image and the reference set of its class is shown for each example. Because of transfer learning, CleanNet can assign reasonable scores to images from classes where no training images belonging to it are manually verified.

Experiments on Clothing1M. For Clothing1M, we consider the state of the art result reported in , which also used ResNet-50. used the part of data in Clothing1M that has both noisy and correct class labels to estimate confusion among classes and modeled this information in loss function. Since we only compare the noisy class label to the correct class label for an image to verify whether the noisy class label is correct, we lose the label confusion information, and thus these numbers are not directly comparable. However, labeling the correct classes like Clothing1M (only 14 classes) is not scalable in number of classes because having labeling workers select from a large number of classes is time-consuming and unlikely to be accurate.

Table 4 lists the results of image classification using verification labels in all classes. Using CleanNet significantly improves the accuracy from 68.94% (#1) to 74.69% (#6) on 1M noisy training data. We also follow to fine-tune the best model trained on 1M noisy set on the 50k clean training set. Our proposed method achieves 79.90%, which is comparable to the state of the art 80.38% reported in which benefits from the extra label confusion information.

Experiments on WebVision. As opposed to Food-101N and Clothing1M which are fine-grained tasks, WebVision experiments sheds light on general image classification at very large scale. As mentioned in Sec. 4.1, the pseudo-verification labels are model-based so that we can obtain for all images. This property allows us to explore how to select classes for adding verification labels and compare to the upper bound scenario where all noisy class labels are verified without any cost. We define how to add verification labels as “verification conditions”, listed in Table 5. Table 6 shows the experimental results using CleanNet and soft weighting ($w_{soft}$). We observe that verifying every image (every-image) improves the top-1 accuracy from 67.76% to 70.31% on the WebVision validation set. With only 3.20% and 1.2% images verified, semantic-308 and random-118 give 47% and 29% of the performance gain of every-image on the WebVision validation set respectively. Note that we only include 250 verification labels for each class for all experiments using CleanNet. The results again confirm that labeling on a fraction of classes is effective because of transfer learning by CleanNet.

Conclusion

In this work, we highlighted the difficulties of having both scalability and effectiveness of human supervision for label noise detection and classification learning from noisy data. We introduced CleanNet as a transfer learning approach to reconcile the issue by transferring supervised information of transferring the correctness of labels to classes without explicit human supervision. We empirically evaluate our proposed methods on both general and fine-grained image classification datasets. The results show that CleanNet outperforms methods using no human supervision by a large margin when small fraction of classes is manually verified. It also matches existing methods that require extensive human supervision when sufficient classes are manually verified. We believe this work creates a novel paradigm that efficiently utilizes human supervision to better address label noise in large-scale image classification tasks.

Acknowledgement

The authors thank Xi Chen, Yu-Hsiang Bosco Chiu, Yandong Guo and Po-Sen Huang for their thoughtful feedbacks and discussions. Thanks also to Li Huang and Arun Sacheti for helping develop the Food-101N dataset.

References