Symmetric Cross Entropy for Robust Learning with Noisy Labels

Introduction

Modern deep neural networks (DNNs) are often highly complex models that have hundreds of layers and millions of trainable parameters, requiring large-scale datasets with clean label annotations such as ImageNet for proper training. However, labeling large-scale datasets is a costly and error-prone process, and even high-quality datasets are likely to contain noisy (incorrect) labels. Therefore, training accurate DNNs in the presence of noisy labels has become a task of great practical importance in deep learning.

Recently, several works have studied the dynamics of DNN learning with noisy labels. Zhang et.al argued that DNNs exhibit memorization effects whereby they first memorize the training data for clean labels and then subsequently memorize data for the noisy labels. Similar findings are also reported in that DNNs first learn clean and easy patterns and eventually memorize the wrongly assigned labels. Further evidence is provided in that DNNs first learn simple representations via subspace dimensionality compression and then overfit to noisy labels via subspace dimensionality expansion. Different findings are reported in , where DNNs with a specific activation function (i.e., tanh) undergo an initial label fitting phase then a subsequent representation compression phase where the overfitting starts. Despite these important findings, a complete understanding of DNN learning behavior, particularly their learning process for noisy labels, remains an open question.

In this paper, we provide further insights into the learning procedure of DNNs by investigating the learning dynamics across classes. While Cross Entropy (CE) loss is the most commonly used loss for training DNNs, we have found that DNN learning with CE can be class-biased: some classes (“easy” classes) are easy to learn and converge faster than other classes (“hard” classes). As shown in Figure 1(a), even when labels are clean, the class-wise test accuracy spans a wide range during the entire training process. As further shown in Figure 1(b), this phenomenon is amplified when training labels are noisy: whilst easy classes (e.g., class 6) already overfit to noisy labels, hard classes (e.g., class 3) still suffer from significant under learning (class accuracy significantly lower than clean label setting). Specifically, class 3 (bottom curve) only has an accuracy of ${\small\sim}60\%$ at the end, considerably less than the ${\small>}90\%$ accuracy of class 6 (top curve). Label Smoothing Regularization (LSR) is a widely known technique to ease overfitting issues, as shown in Figure 1(c), which still exhibits significant under learning on hard classes. Comparing the overall test accuracy (solid red curve) in Figure 1, a low test accuracy (under learning) on hard classes is a barrier to high overall accuracy. This is a different finding from previous belief that poor performance is simply caused by overfitting to noisy labels. We also visualize the learned representations for the noisy label case in Figure 2(b): some clusters are learned comparably well to those learned with clean labels (Figure 2(a)), while some other clusters do not have clear separated boundaries.

Intuitively, CE requires an extra term to improve its learning on hard classes, and more importantly, this term needs to be tolerant to label noise. Inspired by the symmetric KL-divergence, we propose such a noise tolerant term, namely Reverse Cross Entropy (RCE), which combined with CE forms the basis of the approach Symmetric cross entropy Learning (SL). SL not only promotes sufficient learning (class accuracy close to clean label setting) of hard classes, but also improves the robustness of DNNs to noisy labels. As a preview of this, we can inspect the improved learning curves of class-wise test accuracy and representations in Figure 1(d) and 2(c). Under the same 40% noise setting, the variation of class-wise test accuracy has been narrowed by SL to 20% with 95% the highest and 75% the lowest (Figure 1(d)), and the learned representations are of better quality with more separated clusters (Figure 2(c)), both of which are very close to the clean settings.

Compared to existing approaches that often involve architectural or non-trivial algorithmic modifications, SL is extremely simple to use. It requires minimal intervention to the training process and thus can be straightforwardly incorporated into existing models to further enhance their performance. In summary, our main contributions are:

We provide insights into the class-biased learning procedure of DNNs with CE loss and find that the under learning problem of hard classes is a key bottleneck for learning with noisy labels.

We propose a Symmetric Learning (SL) approach, to simultaneously address the hard class under learning problem and the noisy label overfitting problem of CE. We provide both theoretical analysis and empirical understanding of SL.

We empirically demonstrate that SL can achieve better robustness than state-of-the-art methods, and can be also easily incorporated into existing methods to significantly improve their performance.

Related Work

Different approaches have been proposed to train accurate DNNs with noisy labels, and they can be roughly divided into three categories: 1) label correction methods, 2) loss correction methods, and 3) refined training strategies.

The idea of label correction is to improve the quality of the raw labels. One common approach is to correct noisy labels to their true labels via a clean label inference step using complex noise models characterized by directed graphical models , conditional random fields , neural networks or knowledge graphs . These methods require the support from extra clean data or an expensive detection process to estimate the noise model.

Loss correction methods modify the loss function for robustness to noisy labels. One approach is to model the noise transition matrix that defines the probability of one class changed to another class . Backward and Forward are two such correction methods that use the noise transition matrix to modify the loss function. However, the ground-truth noise transition matrix is not always available in practice, and it is also difficult to obtain accurate estimation . Work in augments the correction architecture by adding a linear layer on top of the neural network. Bootstrap uses a combination of raw labels and their predicted labels. There is also research that defines noise robust loss functions, such as Mean Absolute Error (MAE) , but a challenge is that training a network with MAE is slow due to gradient saturation. Generalized Cross Entropy (GCE) loss applies a Box-Cox transformation to probabilities (power law function of probability with exponent $q$) and can behave like a weighted MAE. Label Smoothing Regularization (LSR) is another technique using soft labels in place of one-hot labels to alleviate overfitting to noisy labels.

Refined training strategies design new learning paradigms for noisy labels. MentorNet supervises the training of a StudentNet by a learned sample weighting scheme in favor of probably correct labels. Decoupling training strategy trains two networks simultaneously, and parameters are updated when their predictions disagree. Co-teaching maintains two networks simultaneously during training, with one network learning from the other network’s most confident samples. These studies all require training of an auxiliary network for sample weighting or learning supervision. D2L uses subspace dimensionality adapted labels for learning, paired with a training process monitor. The iterative learning framework iteratively detects and isolates noisy samples during the learning process. The joint optimization framework updates DNN parameters and labels alternately. These methods either rely on complex interventions into the learning process, which may be challenging to adapt and tune, or are sensitive to hyperparameters like the number of training epochs and learning rate.

Weakness of Cross Entropy

We begin by analyzing the Cross Entropy (CE) and its limitations for learning with noisy labels.

2 Weakness of CE under Noisy Labels

We now highlight some weaknesses of CE for DNN learning with noisy labels, based on empirical evidence on CIFAR-10 dataset (10 classes of natural images). To generate noisy labels, we randomly flip a correct label to one of the other 9 incorrect labels uniformly (e.g., symmetric label noise), and refer to the portion of incorrect labels as the noise rate. The network used here is an 8-layer convolutional neural network (CNN). Detailed experimental settings can be found in Section 5.1.

We first explore in more detail the class-biased phenomenon shown in Figure 1(a) and 1(b), focusing on three distinct learning stages: early (the $10$-th epoch), middle (the $50$-th epoch) and later (the $100$-th epoch) stages, with respect to total 120 epochs of training. As illustrated in Figure 3, CE learning starts in a highly class-biased manner (the blue curves) for both clean labels and 40% noisy labels. This is because patterns inside of samples are intrinsically different. For clean labels, the network eventually manages to learn all classes uniformly well, reflected by the relatively flat accuracy curve across classes (the green curve in Figure 3(a)). However, for noisy labels, the class-wise test accuracy varies significantly across different classes, even at the later stage (the green curve in Figure 3(b)). In particular, the network struggles to learn hard classes (e.g., class $2/3$) with up to a 20% gap to the clean setting, whereas some easy classes (e.g., class $1/6$) are better learned and already start overfitting to noisy labels (accuracy drops from epoch 50 to 100). It appears that the under learning of hard classes is a major cause for the overall performance degradation, due to the fact that the accuracy drop caused by overfitting is relatively small.

We further investigate the reason behind the under learning of CE on hard classes from the perspective of representations. Due to their high similarities in representations to some other classes (see the red cluster for class 3 in Figure 2(a)), the predictions for hard class examples are likely to assign a relatively large probability to those similar classes. Under the noisy label scenario, class 3 has become even more scattered into other classes (red cluster in Figure 2(b)). As a consequence, no visible cluster was learned by CE, even though there are still 60% correct labels in this scenario. Further delving into these 60% clean portion of class 3 samples, we show, in Figure 4(a), their prediction confidence output of the neural network. Although the confidence at class 3 is the highest, it is only around 0.5, while for the other classes, the confidence is around 0.05 or 0.1 which is actually a relatively high value and an indication of insufficient learning of class 3 even on the clean labeled part. Another evidence of under learning can be obtained from Figure 4(b), where hard classes (e.g., class $2/3$) have fewer true positive samples throughout intermediate stages of learning.

Clearly, CE by itself is not sufficient for learning of hard classes, especially under the noisy label scenario. We note that this finding sheds new insights into DNN learning behavior under label noise, and differs from previous belief that DNNs overfit to all classes in general . In the next section, we propose a symmetric learning approach that can address both the hard class under learning and noisy label overfitting problems of CE.

Symmetric Cross Entropy Learning

In this section, we propose Symmetric cross entropy Learning (SL), an approach that strikes a balance between sufficient learning and robustness to noisy labels. We also provide theoretical analysis about the formulation and behavior of SL.

Given two distributions $q$ and $p$, the relation between the cross entropy (denoted as $H(q,p)$) and the KL-divergence (denoted as $KL(q\|p)$) is:

where $H(q)$ is the entropy of $q$. In the context of classification, $q=q(k|\mathbf{x})$ is the ground truth class distribution conditioned on sample $\mathbf{x}$, whilst $p=p(k|\mathbf{x})$ is the predicted distribution over labels by the classifier $f$. From the perspective of KL-divergence, classification is to learn a prediction distribution $p(k|\mathbf{x})$ that is close to the ground truth distribution $q(k|\mathbf{x})$, which is to minimize the KL-divergence $KL(q\|p)$ between the two distributionsIn practice, the $H(q(k|\mathbf{x}))$ term is a constant for a given class distribution and therefore omitted from Eq. (2) giving the CE loss in Eq. (1)..

In information theory, given a true distribution $q$ and its approximation $p$, $KL(q\|p)$ measures the penalty on encoding samples from $q$ using code optimized for $p$ (penalty in the number of extra bits required). In the context of noisy labels, we know that $q(k|\mathbf{x})$ does not represent the true class distribution, instead $p(k|\mathbf{x})$ can reflect the true distribution to a certain extent. Thus, in addition to taking $q(k|\mathbf{x})$ as the ground truth, we also need to consider the other direction of KL-divergence, that is $KL(p||q)$, to punish coding samples that come from $p(k|\mathbf{x})$ when using a code for $q(k|\mathbf{x})$. The symmetric KL-divergence is:

Transferring this symmetric idea from KL-divergence to cross entropy gives us the Symmetric Cross Entropy (SCE):

where $RCE=H(p,q)$ is the reverse version of $H(q,p)$, namely, Reverse Cross Entropy. The RCE loss for a sample $\mathbf{x}$ is:

The sample-wise SCE loss can then be defined as:

While the RCE term is noise tolerant as will be proved in Section 4.2, the CE term is not robust to label noise . However, CE is useful for achieving good convergence , which will be verified empirically in Section 5. Towards more effective and robust learning, we propose a flexible symmetric learning framework with the use of two decoupled hyperparameters (e.g., $\alpha$ and $\beta$), with $\alpha$ on the overfitting issue of CE while $\beta$ for flexible exploration on the robustness of RCE. Formally, the SL loss is:

2 Theoretical Analysis

Similarly, we can prove the case for asymmetric noise, please refer Appendix A for details. ∎

Gradient analysis: We next derive the gradients of a simplified SL with $\alpha,\beta=1$ to give a rough idea of how its learning process differs from that of CEComplete derivations can be found in the Appendix B.. For brevity, we denote $p_{k}$, $q_{k}$ as abbreviations for $p(k|\mathbf{x})$ and $q(k|\mathbf{x})$. Consider the case of a single true label, the gradient of the sample-wise RCE loss with respect to the logits $z_{j}$ can be derived as:

where $\frac{\partial p_{k}}{\partial z_{j}}$ can be further derived based on whether $k=j$:

According to Eq. (9) and the ground-truth distribution for the case of one single label (e.g., $q_{y}=1$, and $q_{k}=0$ for $k\neq y$), the gradients of SL can be derived as:

3 Discussion

An easy addition to improve CE would be to upscale its gradients with a larger coefficient (e.g., ‘2CE’, ‘5CE’). However, this will cause more overfitting (see the ‘5CE’ curve in the following Section 5 Figure 9(a)). There are also other options to consider, such as MAE. Although motivated from completely different perspectives, that is, CE and RCE are measures of (information theoretic) uncertainty, while MAE is a measure of distance, we can surprisingly show that MAE is a special case of RCE at $A=-2$, when there is a single true label for $\mathbf{x}$ (e.g. $q(y|\mathbf{x})=1$ and $q(k\neq y|\mathbf{x})=0$). For MAE, we have,

That is, when $A=-2$, RCE is reduced to exactly MAE. Meanwhile, different from the GCE loss (i.e., a weighted MAE) , SL is a combination of two symmetrical learning terms.

Experiments

We first provide some empirical understanding of our proposed SL approach, then evaluate its robustness against noisy labels on MNIST, CIFAR-10, CIFAR-100, and a large-scale real-world noisy dataset Clothing1M.

Noise setting: We test two types of label noise: symmetric (uniform) noise and asymmetric (class-dependent) noise. Symmetric noisy labels are generated by flipping the labels of a given proportion of training samples to one of the other class labels uniformly. Whilst for asymmetric noisy labels, flipping labels only occurs within a specific set of classes , for example, for MNIST, flipping $2\to 7$, $3\to 8$, $5\leftrightarrow 6$ and $7\to 1$; for CIFAR-10, flipping TRUCK $\to$ AUTOMOBILE, BIRD $\to$ AIRPLANE, DEER $\to$ HORSE, CAT $\leftrightarrow$ DOG; for CIFAR-100, the 100 classes are grouped into 20 super-classes with each has 5 sub-classes, then flipping between two randomly selected sub-classes within each super-class.

We conduct experiments on CIFAR-10 dataset with symmetric noise towards a deeper understanding of SL.

Experimental setup: We use an 8-layer CNN with 6 convolutional layers followed by 2 fully connected layers. All networks are trained using SGD with momentum 0.9, weight decay $10^{-4}$ and an initial learning rate of 0.01 which is divided by 10 after 40 and 80 epochs (120 epochs in total). The parameter $\alpha$, $\beta$ and $A$ in SL are set to 0.1, 1 and -6 respectively.

Class-wise learning: The class-wise test accuracy of SL on 40% noisy labels has already been presented in Figure 1(d). Here we provide further results for 60% noisy labels in Figure 5. Under both settings, each class is more sufficiently learned by SL than CE, accompanied by accuracy increases. Particularly for the hard classes (e.g., classes $2/3/4/5$), SL significantly improves their learning performance. This is because SL facilitates an adaptive pace to encourage learning from hard classes. During learning, samples from easy classes can be quickly learned to have a high probability $p_{k}>0.5$, while samples from hard classes still have a low probability $p_{k}<0.5$. SL will balance this discrepancy by increasing the learning speed for samples with $p_{k}<0.5$ while decreasing the learning speed for those with $p_{k}>0.5$.

Prediction confidence and distribution: In comparison to the low confidence of CE on the clean samples in Figure 4(a), we train the same network using SL under the same setting. As shown in Figure 6(a), on the clean portion of class 3 samples, SL successfully pulls up the confidence to 0.95, while at the same time, pushes down the residual confidence at other classes to almost 0. As further shown in Figure 6(b), the prediction distributions demonstrate that each class contains more than 4000 true positive samples, including the hard classes (e.g., class $2/3/4/5$). Some classes (e.g., class $1/6/7/8/9$) even obtain close to 5000 true positive samples (the ideal case). Compared to the earlier results in Figure 4(b), SL achieves considerable improvement on each class.

Representations: We further investigate the representations learned by SL compared to that learned by CE. We extract the high-dimensional representation at the second last dense layer, then project to a 2D embedding using t-SNE . The projected representations are illustrated in Figures 2 and 7 for 40% and 60% noisy labels respectively. Under both settings, the representations learned by SL are of significantly better quality than that of CE with more separated and clearly bounded clusters.

Parameter analysis: We tune the parameters of SL: $\alpha$, $\beta$ and $A$. As $\beta$ can be reflected by $A$, here we only show results of $\alpha$ and $A$. We tested A in $$with step 2 and$\alpha\in[10^{-2},1]$on CIFAR-10 under 60$\alpha$(e.g., 1.0/0.5) tends to cause more overfitting, while small$\alpha$(e.g., 0.1/0.01) can help ease the overfitting of CE. Nevertheless, the convergence can become slow when$\alpha$is too small (e.g., 0.01), a behaviour like the single RCE. For this reason, a relatively large$\alpha$can help convergence on difficult datasets such as CIFAR-100. As for parameter$A$, if the overfitting of CE is well controlled by$\alpha=0.1$, SL is not sensitive to$A$(Figure 8(b)). However, if CE overfitting is not properly addressed, SL becomes mildly sensitive to$A$ (Figure 8(c)).

Ablation study: For a comprehensive understanding of each term in SL, we further conduct a series of ablation experiments on CIFAR-10 under 60% noisy labels. Figure 9(a) presents the following experiments: 1) removing the RCE term; 2) removing the CE term; 3) upscaling the CE term; and 4) upscaling the RCE term. We can observe that simply upscaling CE does not help learning, or even leads to more overfitting. The RCE term itself does not exhibit overfitting even when upscaled, but it converges slowly. But when CE and RCE are combined into the SL framework, the performance is drastically improved.

2 Robustness to Noisy Labels

Baselines: We compare SL with 5 recently proposed noisy label learning methods as well as the standard CE loss: (1) Forward : Training with label correction by multiplying the network prediction with the ground truth noise matrix; (2) Bootstrap : Training with new labels generated by a convex combination of the raw labels and the predicted labels; (3) GCE : Training with a noise robust loss encompassing both MAE and CE; (4) D2L : Training with subspace dimensionality adapted labels; (5) Label Smoothing Regularization (LSR) : Training with CE on soft labels, rather than the one-hot labels; and (6) CE: Training with standard cross entropy loss.

Experimental setup: Experiments are conducted on MNIST , CIFAR-10 and CIFAR-100 . We use a 4-layer CNN for MNIST, the same network as Section 5.1 for CIFAR-10 and a ResNet-44 for CIFAR-100. Parameters for the baselines are configured according to their original papers. For our SL, we set $A=-4$ for all datasets, and $\alpha=0.01,\beta=1.0$ for MNIST, $\alpha=0.1,\beta=1.0$ for CIFAR-10, $\alpha=6.0,\beta=0.1$ for CIFAR-100 (a dataset known for hard convergence)For 40% asymmetric noise, $\beta$ is set to 5.0 for CIFAR-10 and $\alpha$ is set to 2.0 for CIFAR-100. Other parameters are unchanged.. All networks are trained using SGD with momentum 0.9, weight decay $5\times 10^{-3}$ and an initial learning rate of 0.1. The learning rate is divided by 10 after 10 and 30 epochs for MNIST (50 epochs in total), after 40 and 80 epochs for CIFAR-10 (120 epochs in total), and after 80 and 120 epochs for CIFAR-100 (150 epochs in total). Simple data augmentation techniques (width/height shift and horizontal flip) are applied on CIFAR-10 and CIFAR-100. For symmetric noise, we test varying noise rates $\eta\in[0\%,80\%]$, while for asymmetric noise, we test noise rates $\eta\in[0\%,40\%]$.

Robustness performance: The classification accuracies are reported in Table 1. As can be seen, SL improves on the baselines via a large margin for almost all noise rates and all datasets. Note that Forward sometimes also delivers a relatively good performance, as we directly provide it with the ground truth noise matrix. We also find that SL can be more effective than GCE, particularly for high noise rates. The complete learning procedures of SL and baselines on CIFAR-10 are illustrated in Figure 9(b). SL shows a clear advantage over other methods, especially in the later stages of learning with noisy labels. This is likely because that, in the later stages of DNN learning, other methods all suffer to some extent from under learning on hard classes, while SL ensures sufficient learning on them.

Enhancing existing methods with SL: We introduce some general principles to incorporate SL into existing methods to further enhance their performance. For methods that use robust loss functions or label corrections, the RCE term of SL can be directly added to the loss function, while for methods that still use the standard CE loss without label corrections, SL can be used with small $\alpha$ and large $\beta$ to replace the existing loss function. This is to avoid overfitting while promote sufficient learning. As a proof-of-concept, we conduct experiments to enhance Forward and LSR with SL. For “Forward+SL”, we add the RCE term to the Forward loss with $\beta=1.0/0.1$ for symmetric/asymmetric noise respectively, while for “LSR+SL”, we use the SL loss with the same setting in Table 1. Results on CIFAR-10 are presented in Table 2. Both the enhanced methods demonstrate a clear performance improvement over their original versions (Forward or LSR) both on symmetric and asymmetric noise. However, in some scenarios, the enhanced methods are still not as good as SL. This often occurs when there is a large performance gap between the original methods and SL. We believe that with more adaptive incorporation and careful parameter tuning, SL can be combined with existing approaches to achieve even better performance.

3 Experiments on Real-world Noisy Dataset

In the above experiments, we have seen that SL achieves excellent performance on datasets with manually corrupted noisy labels. Next, we assess its applicability for a real-world large-scale noisy dataset: Clothing1M .

The Clothing1M dataset contains 1 million images of clothing obtained from online shopping websites with 14 classes: T-shirt, Shirt, Knitwear, Chiffon, Sweater, Hoodie, Windbreaker, Jacket, Down Coat, Suit, Shawl, Dress, Vest, and Underwear. The labels are generated by the surrounding text of images and are thus extremely noisy. The overall accuracy of the labels is ${\small\sim 61.54\%}$, with some pairs of classes frequently confused with each other (e.g., Knitwear and Sweater), which may contain both symmetric and asymmetric label noise. The dataset also provides $50k$, $14k$, $10k$ manually refined clean data for training, validation and testing respectively, but we did not use the $50k$ clean data. The classification accuracy on the $10k$ clean testing data is used as the evaluation metric.

Experimental setup: We use ResNet-50 with ImageNet pretrained weights similar to . For preprocessing, images are resized to $256\times 256$, with mean value subtracted and cropped at the center of $224\times 224$. We train the models with batch size 64 and initial learning rate $10^{-3}$, which is reduced by $1/10$ after 5 epochs (10 epochs in total). SGD with a momentum 0.9 and weight decay $10^{-3}$ are adopted as the optimizer. Other settings are the same as Section 5.2.

Results: As shown in Table 3, SL obtains the highest performance compared to the baselines. We also find that Forward achieves a relatively good result, though it requires the use of the part of data that both have noisy and clean labels to obtain the noise transition matrix, which is not often available in real-world settings. SL only requires the noisy data and does not require extra auxiliary information.

Conclusions

In this paper, we identified a deficiency of cross entropy (CE) used in DNN learning for noisy labels, in relation to under learning of hard classes. To address this issue, we proposed the Symmetric cross entropy Learning (SL), boosting CE symmetrically with the noise robust Reverse Cross Entropy (RCE), to simultaneously addresses its under learning and overfitting problems. We provided both theoretical and empirical understanding on SL, and demonstrated its effectiveness against various types and rates of label noise on both benchmark and real-world datasets. Overall, due to its simplicity and ease of implementation, we believe SL is a promising loss function for training robust DNNs against noisy labels, and an attractive framework to be used along with other techniques for datasets containing noisy labels.

References

Appendix A Proof for Theorem 1

For asymmetric or class-dependent noise, $1-\eta_{y}$ is the probability of a label being correct (i.e., $k=y$), and the noise condition $\eta_{yk}<1-\eta_{y}$ generally states that a sample $\mathbf{x}$ still has the highest probability of being in the correct class $y$, though it has probability of $\eta_{yk}$ being in an arbitrary noisy (incorrect) class $k\neq y$. Considering the noise transition matrix between classes $[\eta_{ij}],\forall i,j\in\{1,2,\cdots,K\}$, this condition only requires that the matrix is diagonal dominated by $\eta_{ii}$ (i.e., the correct class probability $1-\eta_{y}$). Following the symmetric case, here we have,

As $f^{\ast}_{\eta}$ is the minimizer of $R^{\eta}(f)$, $R^{\eta}(f_{\eta}^{\ast})-R^{\eta}(f^{\ast})\leq 0$. So, from Eq.(12), we have,

Appendix B Gradient Derivation of SL

The complete derivartion of the simplified SL ($\alpha,\beta=1$) with respect to the logits is as follows:

Combining Eq. (16) and (17) into Eq. (14), we can obtain:

If $q_{j}=q_{y}=1$, then the gradient of SL is: