Training Binary Neural Networks through Learning with Noisy Supervision

Introduction

Deep convolutional neural networks (CNNs) have achieved much success in many real-world applications such as image recognition (He et al., 2016; Han et al., 2018c), object detection (Ren et al., 2015), and semantic segmentation (Chen et al., 2016). These CNN models usually consume high computational resource, and thus they cannot be easily deployed on embedded devices. A series of model compression and acceleration methods (Han et al., 2016; Chen et al., 2020) have been proposed to reduce the number of parameters and FLOPs of CNNs, including network pruning (Han et al., 2016; Li et al., 2017; Shu et al., 2019), tensor decomposition (Denton et al., 2014), knowledge distillation (Hinton et al., 2015; Chen et al., 2019), efficient model design (Howard et al., 2017; Han et al., 2020), and model quantization (Gupta et al., 2015; Hubara et al., 2016). These methods have significantly promoted the development of deep learning towards real-world mobile applications.

Binary neural networks (BNNs) (Hubara et al., 2016; Rastegari et al., 2016; Lin et al., 2017; Liu et al., 2018; Shen et al., 2020) push neural network quantization to the extreme. 1-bit weights and activations in BNNs can dramatically save computational cost for better real-time inference. BinaryNet (Hubara et al., 2016) first proposed binary networks with both 1-bit weights and activations. XNOR-Net (Rastegari et al., 2016) further improved BinaryNet by introducing channel-wise scale factor of weights and activation. Dorefa-Net (Zhou et al., 2016) utilized layer-wise scale factor to achieve XNOR-Net like performance. ABC-Net (Lin et al., 2017) enhanced the performance by using more weight bases and activation bases, but computation complexity is increased at the same time. These approaches have constantly boosted the performance of BNNs. For example, binary AlexNet in XNOR-Net (Rastegari et al., 2016) achieves a 44.2% top-1 accuracy on the ImageNet classification task, while reducing the convolution parameters for nearly 32$\times$ over the full-precision model (56.6% top-1 accuracy).

In constructing the binary neural network, most of existing approaches employ hard thresholding (e.g. sign function) to quantize the weights and process each element independently. “Straight through estimator” (STE) (Bengio et al., 2013) is applied to calculate the gradient of sign function. However, the performance of the existing BNNs is far worse than that of the full-precision counterparts, e.g. up to 11.6% accuracy drop of binary AlexNet in XNOR-Net. Simply binarizing each individual weight independently does not fully explore the relationship between neurons and may not bring in the optimal solution. Moreover, estimated gradients by STE often lead to inaccurate weights that contain the noise, i.e. some of the binary weights have been incorrectly flipped into the opposite values.

In this paper, we learn to binarize neurons with noisy supervision, as shown in Fig. 1. In contrast to classical hand crafted rules to binarize the weights, we suggest a mapping from full-precision neurons to the binary ones. This mapping function can be approximated with a neural network that treats full-precision weights in a filter as a whole for the input. To help the learning of the mapping function, we take the pretrained binary weights as noisy supervisions that are close to the ideal binary neurons. An unbiased estimator is introduced for learning with noisy supervisions and avoiding noise disturbance. In an end-to-end fine-tuning, the proposed method can be a nice alternative of the sign function in BNNs by mining the relationship between neurons and taking advantage of noisy supervisions. Theoretical analysis suggests that the introduced unbiased estimator can converge to the optimal solution of binary weights under the clean distribution. Experiments on benchmark datasets including CIFAR-10 and ImageNet demonstrate that BNNs established using the proposed Learning with Noisy Supervision (LNS) method achieve state-of-the-art performance.

Related Work

In this section, we give a brief review of the related work in the field of binary neural networks and learning with noisy labels.

Binary neural networks with extremely low memory and computation cost appeal great interest from the community. Binaryconnect (Courbariaux et al., 2015) was proposed to train deep neural networks with binary weights. BinaryNet (Hubara et al., 2016) further quantize both the weights and the activations to 1-bit values, starting the research on pure binary neural networks. XNOR-Net (Rastegari et al., 2016) introduce channel-wise scale factor to improve performance. Dorefa-Net (Zhou et al., 2016) simplifies to layer-wise scale factor and achieve similar performance with XNOR-Net. ABC-Net (Lin et al., 2017) proposes to enhance the performance by using more weight bases and activation bases, but it admittedly needs more memory and computation cost than BinaryNet. There are also several works designing blocks or architectures for binary networks (Liu et al., 2018; Shen et al., 2019). Bireal-Net (Liu et al., 2018) introduces layer-wise identity short-cut, and AutoBNN (Shen et al., 2019) widen or squeeze the channels in an automatic manner. All these binary models utilize STE (Bengio et al., 2013) for gradient back-propagation which would introduce inaccurate gradients for model optimization.

Some works propose new gradient calculation approach instead of STE. Bireal-Net (Liu et al., 2018) and DSQ (Gong et al., 2019) use specially designed activation function for back-propagation. PCNN (Gu et al., 2019) proposes a new discrete back-propagation via projection algorithm to build BNNs. However, most of existing methods quantize each weight independently and ignore their internal relationship.

2 Learning with Noisy Labels

To learn more accurate predictions and correct biased information from noisy labels, a number of methods are proposed for learning with noisy labels, which can be divided into three categories:

Label correction aims to correct the wrong labels in the raw labels. The existing methods usually utilize a clean label inference module to correct the noisy labels to the true ones. The inference module can be modeled by neural networks (Lee et al., 2018), graphical models (Xiao et al., 2015), or conditional random fields (Vahdat, 2017). However, the extra clean data or expensive noise detection process is required in these methods, which is unpractical in real-world applications.

Refined training strategies introduce new learning framework for robustness to noisy labels (Jiang et al., 2018; Han et al., 2018b; Yu et al., 2019; Wang et al., 2018; Tanaka et al., 2018). These methods such as MentorNet (Jiang et al., 2018) and Co-teaching (Han et al., 2018b; Yu et al., 2019), change the standard learning process with complex interventions which usually need much effort to adapt and tune.

Loss correction methods improve the standard loss function to suit for noisy labels. One common approach is modeling the noise transition matrix which defines the probability of one class flipped to another one (Natarajan et al., 2013). Backward and Forward (Patrini et al., 2017) introduce two alternative procedures for loss correction, provided knowing the noise transition matrix. A linear layer is added on top of the neural networks for noisy prediction correction in (Goldberger & Ben-Reuven, 2017). Masking (Han et al., 2018a) derive a structure-aware probabilistic model to incorporate the structure prior. Noise robust loss function is another technique dealing with noisy labels, such as generalized cross entropy (Zhang & Sabuncu, 2018), label smoothing regularization (Pereyra et al., 2017), and symmetric cross entropy (Wang et al., 2019).

Binary weights can also be recognized as prediction of a binary classifier, and the biased weights derived from the sign function are exactly the noisy label. Therefore, we present to develop a mapping that can correct the noisy binary weights and obtain BNNs with better performance.

Approach

In this section we detail the formulation of our method, including binary weight mapping model and an unbiased estimator with noisy supervision.

where $\circledast$ represents convolution operation with binary operations. During training, the back-propagation process of the quantization follows the straight through estimator (Bengio et al., 2013):

The simple sign function to binarize the weights cannot take the relationship between elements into consideration and may not be the optimal. In fact, the ideal process to transform the full-precision weights to binary could be complicated and unknown. Instead of trying to fit the hand-crafted binarization rules, we propose to binarize neurons through a learned mapping function as shown in Fig. 2. The full-precision weights are taken as a whole, and thus their internal relation can be fully explored and exploited by the mapping model to accomplish the binarization. Formally, the binarization process can be written as

where $f_{\theta}$ is the mapping model with training parameters $\theta$. Compared to sign function, the binarization function in Eq. 4 is learnable and more flexible, which can approximate the binarization for the need to quantize the weights. Sign function only operates on each individual element independently, while the mapping function approximated by a neural network (Eq. 4) can quantize each element by considering its connections with other elements.

2 Learning with Noisy Supervision

If the ground-truth binary weights $Q$ are provided, we can force the mapping model to learn the target under the supervision:

where $\|\cdot\|_{F}$ is the Frobenius norm of a tensor. Denoting $\hat{q}$ as each element in the predictions $\hat{Q}$, and $q$ as the corresponding ground-truth label in $Q$, Eq. 5 can be represented as the following for simplicity:

The loss in Eq. 5 is simple and easy to optimize. However, the ground-truth $Q$ is hard to obtain in practice.

where $\rho_{+1}$ is the probability that the negative weight is flipped into $+1$, $\rho_{-1}$ is the probability that the positive weight is flipped into $-1$, and $\rho_{+1}+\rho_{-1}<1$. The noise rates $\rho_{+1}$ and $\rho_{-1}$ are two hyper-parameters. In binary neural networks, the number of positive weights and negative weights are similar, so the noise rate $\rho_{+1}$ and $\rho_{-1}$ should be similar as well, i.e. $\rho=\rho_{+1}=\rho_{-1}$.

(Natarajan et al., 2013) With probability at least $1-\delta$,

Then the gradient to $\theta$ is given by

where $\frac{\partial\hat{q}}{\partial\theta}$ can be calculated with standard chain rule as $f_{\theta}(\cdot)$ is a neural network. The latent weights $W$ are the input to the mapping model $f_{\theta}(\cdot)$ to output the binary weight predictions $\hat{q}$, so the gradients are

where $\alpha$ is the trade-off hyper-parameter. The proposed LNS method can be embedded into the training process of binary neural networks and trained in the end-to-end manner. The forward and back-propagation process of our method are listed in Algorithm 1. After training, we obtain the optimized latent weights $W$ and mapping parameters $\theta$. We transform the latent full-precision weights to binary weights $\hat{Q}$ using the mapping neural network, and only keep those binary weights for the inference.

Experiments

In this section, we evaluate the proposed method on two image classification datasets: CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (ILSVRC12) (Deng et al., 2009), and compare our method with other BNNs.

CIFAR-10 dataset (Krizhevsky & Hinton, 2009) consists of 60,000 32$\times$32 color images belonging to 10 categories, with 6,000 images per category. There are 50,000 training images and 10,000 test images. For hyper-parameter tuning, 10,000 training images are randomly sampled for validation and the rest images are for training. Data augmentation strategy includes random crop and random flipping as in (He et al., 2016) during training. For testing, we evaluate the single view of the original image for fair comparison.

ImageNet

ImageNet ILSVRC 2012 (Deng et al., 2009) is a large-scale image classification dataset which contains over 1.2 million high-resolution natural images for training and 50k validation images in 1,000 classes. The commonly used data augmentation strategy including random crop and flipping in PyTorch examples (Paszke et al., 2019) is adopted for training. We report the single-crop evaluation result using $224\times 224$ center crop from images.

Implementation Details

All the models are implemented using PyTorch (Paszke et al., 2019) and conducted on NVIDIA Tesla V100 GPUs. For CIFAR-10, ResNet-20 is used as baseline model. The binary baseline models are trained for 400 epochs with a batch size of 128 and an initial learning rate $0.1$. We use the SGD optimizer with the momentum of 0.9 and set the weight decay to 0. Our method is fine-tuned based on the pretrained baseline for 120 epochs using SGD optimizer. The learning rate starts from 0.01 and decayed by 0.1 every 30 epochs. For ImageNet, AlexNet and ResNet-18 are adopted for evaluation. We train the binary baseline models for 120 epochs with a batch size of 256. SGD optimizer is applied with the momentum of 0.9 and the weight decay of 0. The learning rate is set as 0.1 initially and is multiplied by 0.1 at the 70$th$, 90$th$ and 110$th$ epoch, respectively. Our method is fine-tuned from the pretrained baseline for 45 epochs with the initial learning rate 0.01 which is decayed by 0.1 every 15 epochs.

In each layer, there is a neural network for binary weight mapping. We simply use a three-layer CNN with weight shape of $2c\times c\times 3\times 3$, $2c\times 2c\times 3\times 3$ and $c\times 2c\times 3\times 3$, respectively. We set padding as 1 and stride as 1 in every layer of the mapping model to keep the size of output unchanged. Batch normalization and ReLU activation are inserted after the intermediate convolutional layers. The mapping model is updated for several epochs for warm start meanwhile the other weights are fixed before fine-tuning.

2 Experiments on CIFAR-10

We first conduct detailed studies on CIFAR-10 dataset for the proposed method. The widely used ResNet-20 architecture is adopted as the basic architecture, and Dorefa-Net is used as the baseline quantization method. Following the common setting in (Zhou et al., 2016), all the layers except for the first convolutional layer and the last fully-connected layer for classification are quantized into 1-bit. We train the baseline binary model for 400 epochs and obtain an accuracy of 85.06%. Based on this pretrained model, we further fine-tune with or without our method.

To verify the effectiveness of our method, we fine-tune the baseline model without mapping model and our method for 120 epochs with all the same experimental settings as stated in implementation details. In our method, we setting the hyper-parameters as $\alpha=1.0$ and $\rho=0.005$. We run them 5 times and show the best, mean and standard values in Table 1. After fine-tuning using our method, we can see that our method without noisy supervision achieves a mean accuracy of 85.36%, adding noisy supervision further improve the accuracy to 85.56%, while simply fine-tuning achieves 85.26%. Both simple fine-tuning and our method can improve the baseline model, but the performance of our method is much better than simple fine-tuning. The results indicate the effectiveness of the proposed binary neuron mapping and the corresponding noisy supervision. The highest accuracy of our method can achieve 85.78%, which is the state-of-the-art as shown in the latter analysis.

We also plot the loss curve and accuracy curve to observe the effect of our method during training. The cross entropy loss curves of simple fine-tuning and our method are shown in Fig. 3(a), and the test accuracy curves of them are shown in Fig. 3(b). The initial loss value and accuracy are 0.37 and 85.06%, respectively, from the pretrained baseline model. At first, we find that Fine-tuning has a much larger loss than our method with the same initial learning rate (lr=0.01), so we decrease the learning rate. Although the loss in Fine-tuning (lr=0.001/0.0001) is decreased, the accuracy on test set has no improvement. From Fig 3(a), we can see that the simple fine-tuning changes the loss at the first as it disturbs the pretrained binary weights largely, while our method does not change the loss much as the mapping model only changes a small portion of the weights. When we decrease the learning rate at the 30$th$, 60$th$ and 90$th$ epoch, the loss values in Fine-tuning and our method will have a relatively large drop. At the last several epochs, simple fine-tuning method with different learning rate achieve a smaller train loss, but a lower test accuracy than our method (Fig. 3(b)). This means that our method can alleviate over-fitting by imposing a noisy supervision on each layer as the noisy weights are often those that over-fit the training data.

From the loss curve and accuracy curve, we know that the training process of our method is more stable than that of simple fine-tuning method. We show the flip rate of the binary weights after each epoch in Fig. 3(c), where flip rate means the ratio of binary weights that are flipped into the opposite values. The flip rate decrease gradually during training in all these curves. The flip rate in Fine-tuning is always higher than our method, which means our method only change a small portion of binary weights which are likely to be noise to explore better performance.

Analysis of Hyper-parameters.

There are two hyper-parameters in our method, i.e. $\alpha$ for balancing the cross entropy classification loss and the noisy neuron correction loss, and $\rho$ for controlling the noise rate in the binary weight transformation. We run all the models 5 times and report the mean and std values of the accuracy on CIFAR-10 validation set.

We first fix $\rho=0.01$ and tune $\alpha$ in range of $\{0,0.01,$ $0.02,0.05,0.1,0.2,0.5,1,2,5\}$ to see the influence of $\alpha$. The results are shown in Fig. 4. The mean accuracy is 85.35% when $\alpha=0$, which is higher than the simple fine-tuning. This verifies the effectiveness of the mapping model without self-supervision. When we increase the value of $\alpha$, the accuracy is improved over that at $\alpha=0$. The highest mean accuracy occurs around $\alpha=1$, i.e. 85.53%. We can see that our method works at a large range of $\alpha$ and can choose $\alpha$ around 1 for the best performance on CIFAR-10.

For the noise rate $\rho$, we fix $\alpha$ as 1 and test $\rho$ in $\{0,$ $0.001,0.002,0.005,0.01,0.02,0.03,0.04,0.05,0.1,0.2\}$. From the results in Fig. 5, we can see that the mean accuracy at different noise rate is different. When $\rho$ is small (around 0.001), the mean accuracy is about 85.4%, which is higher than baseline and simple fine-tuning method. Our method achieve the best mean accuracy at $\rho=0.005$, which means the ground-truth noise rate is about 0.5%. When $\rho$ is too large, such as $\rho=0.2$, the mean accuracy drops significantly, even below the simple fine-tuning method. This is due to that there are not so much noise in the pretrained binary weights, setting $\rho$ too large disturbs the good weights and is harmful to the performance.

Comparison with SOTA.

We compare our method with some other state-of-the-art binary neural networks such as Dorefa-Net (Zhou et al., 2016), XNOR-Net (Rastegari et al., 2016), and DSQ (Gong et al., 2019). The results of comparison are listed in Table 2. Note that only the best accuracy is reported for other methods. From the results, our method outperforms the competitors by a large margin and achieves the state-of-the-art result (85.78% accuracy).

3 Experiments on ImageNet

In order to validate our method on large-scale dataset, we conduct more experiments on ImageNet classification dataset. Two common network architectures, i.e. AlexNet (Krizhevsky et al., 2012) and ResNet-18 (He et al., 2016), are used for experiments.

We test the effect of our method by deploying the proposed method on ResNet-18. The baseline here is the binary ResNet-18 which is quantized using Dorefa-Net (Zhou et al., 2016). We fine-tune the pretrained binary ResNet-18 with or without our method for 60 epochs. These two models use the same hyper-parameter settings. The simple fine-tuning without using our method can improve the Top-1 accuracy to 52.8%. Our method achieves 53.1%, much higher than the baseline and the simple fine-tuning.

Comparison with SOTA.

While the ablation study has evaluated the effectiveness of the proposed method, we also compare our method with the state-of-the-art methods to show the superiority of our method. The compared binary neural network methods include BinaryNet (Hubara et al., 2016), Dorefa-Net (Zhou et al., 2016), XNOR-Net (Rastegari et al., 2016), Bireal-Net (Liu et al., 2018), PCNN (Gu et al., 2019), Bop (Helwegen et al., 2019), GBCN (Liu et al., 2019), etc. Two representative 2-bit neural networks, i.e. HWGQ (Cai et al., 2017) and TBN (Wan et al., 2018), are also included. Following the common settings (Hubara et al., 2016; Liu et al., 2018), we do not quantize the first convolutional layer and the last fully connected layer for classification.

In ResNet-18 experiments, except for BinaryNet and ABC-Net, all the other methods including our method do not quantize the down-sample layers for fair comparison. The statistics of the compared methods are listed in Table 3. The FLOPs are calculated as real-valued floating-point multiplication plus 1/64 of the amount of 1-bit multiplication as the binary operations including AND and POPCOUNT can be performed in a parallel of 64 by the mainstream CPUs (Liu et al., 2018). We use the ResNet-18 architecture in Bireal-Net as baseline and insert PReLU activation (He et al., 2015) after every binary convolutional layer. This strong baseline has a Top-1 accuracy of 59.0%. Our method is fine-tuned based on the pretrained baseline and finally achieve 59.4% Top-1 and 81.7% Top-5 accuracies, which are higher than the compared models and achieve the state-of-the-art results for binary ResNet-18. It is encouraging to see that our method can beat some methods with 2-bit activations, such as HWGQ (Cai et al., 2017) and TBN (Wan et al., 2018). This gives us the confidence to achieve higher performance with lower bit-width in neural networks.

We also compare our method with several state-of-the-art models for AlexNet architecture which does not has residual connections. from the results in Table 5, tt can be seen that our method outperforms the compared models such as BinaryNet (Hubara et al., 2016), and Dorefa-Net (Zhou et al., 2016), which validates the superiority of our method for different architectures. Moreover, our method with only layer-wise scale factor can achieve higher Top-1 accuracy than XNOR-Net (Rastegari et al., 2016) which uses channel-wise scale factor and more parameters.

Conclusion

Acknowledgement

The authors thank the anonymous reviewers for their helpful comments in revising the paper. This work was supported in part by National Key R&D Program of China (2017YFB1002701), NSFC (61632003,61672502), Macau S&T Development Fund (0018/2019/AKP), UM Research Fund (MYRG2019-00006-FST), and in part by the Australian Research Council under Project DE-180101438.

References