Deep neural networks are robust to weight binarization and other non-linear distortions

Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve K. Esser, Dharmendra Modha

Introduction

Deep neural networks (DNNs) trained using backpropagation have been shown to perform exceptionally well on a wide range of classification tasks . Typically, these networks use a high precision representation for weights (e.g., 32-bit floating point) for both training and for inference. Considering just inference tasks, a long standing goal has been to reduce the precision of weights without sacrificing performance, with the aim of lowering the network’s computational and memory footprint. This has practical applications that include network compression, running networks faster and more efficiently on conventional hardware , and running networks on specialized hardware designed specifically for reduced precision representations .

Here, we present a number of new findings:

We show that many networks that perform well when their weights are binarized, also perform well for other kinds of distortions. These distortions include additive and multiplicative noise, as well as applying non-linear distortions to the weights.

We report that using weight projections other than quantization during training also lead to robust networks. Furthermore, we show examples where standard backprop with weight clipping can learn a base level of robustness, although performance is slightly reduced.

Based on these observations, we propose a new stochastic projection rule inspired by BinaryConnect , except our rule projects weights to random intervals (as opposed to quantized values). This rule results in new state of the art performance for CIFAR-10 with $8.25\%$ in the binary case and $7.64\%$ in the non-binary case.

The organization of our paper is as follows: After reviewing related work in Section 2, we describe our training algorithm in Section 3. Next in Section 4, we use this algorithm to train six networks on the CIFAR-10 dataset, exploring different combinations of weight projections and weight clipping parameters. To help frame our results, we first delve into a curious finding in Section 4.1.1. Specifically, we show that a network trained using quantized weights also has low test error for non-quantized weights, suggesting that it is robust to weight distortions. Following this lead in Section 4.1.2, we uncover that this network (and others) are robust to distortions beyond weight quantization. In Section 4.1.3 we try to tease apart the aspects of training that lead to these robust networks, and report for the first time that non-quantized projections (and even no projections at all) can also lead to robustness. A new stochastic projection rule is explored in Section 4.1.4. Then in Sections 4.2 and 4.2.1 we check that our findings hold up for ImageNet. Section 5 puts forth two theoretical ideas on how backprop is able to find these robust solutions. Finally, we conclude and discuss future directions in Section 6.

Related work

Our current work is related to a flurry of recent research on DNNs with low precision weights. The major advance has been the discovery of a backprop-based learning method that quantizes weights during propagations. This training method was introduced in with impressive results on CIFAR-10, and developed in the context of neuromorphic hardware in . proposed a similar rule for contrastive divergence. Here we build on this work by exploring a more general class of weight projections that are not restricted to quantization. Another recent approach approximates weights using binary values during training, formulated as a constrained optimization problem, with promising results on ImageNet . We expect their findings are consistent with our results, however, this is left for future work. Other approaches have developed a probabilistic interpretation of discrete weights , however, they have not yet been extended to convnets or datasets larger than MNIST.

DNNs with projected weights

Testing, just like training, is also performed for a particular projection, however it is important to note that testing and training projections are independently specified. We often refer to projections applied during testing as distortions.

Results

In this section, we explore the performance and robustness of networks trained using different weight projections and clip settings, on both CIFAR-10 (Section 4.1) and ImageNet (Section 4.2).

CIFAR-10 is an image classification dataset of small ( $32\times 32$ ) color images with $10$ categories . For our experiments, we do not use data augmentation. To train, we use the ADAM learning rule with a learning rate of $0.003$ , and a batch size of $50$ ; a square hinge loss; and batch normalization . These results were obtained in TensorFlow , except for a control network that used Caffe .

1.2 Robustness to weight distortions beyond -1,+1

We investigate the premise that networks that perform well with binary weights, also perform well for many types weight distortions. Here, we focus on networks trained using quantization-based weight projections (based on the networks from BinaryConnect), where allowed weight states during training are discrete. Specifically, we consider Tr-Sign-C and Tr-Stoch-C under three distortions: Te-AddNorm, Te-MultUnif, and Te-Power. NiN-ctrl is also shown for comparison.

Bringing these results together, it appears that backprop is finding paths to solutions that are generally robust to weight distortions, and not just the specific quantizations used during training.

1.3 Learning robust networks with and without weight projections

Next, we tried removing weight projections altogether. We trained networks Tr-None-C (no projection with clipping) and Tr-None-NC (no projection without clipping). Putting these networks through the same battery of tests, we observe that Tr-None-C is more robust than Tr-None-NC, although they both exhibit the same basic trends. Notably Tr-None-C still achieves a test error of $11.3\%$ even when its weights are quantized to binary, even though it was never trained with binary weights. Later on in Section 5, we hypothesize how weight clipping can be viewed as a type of regularization, which may help explain these results.

1.4 Stochastic multiplicative projection

In terms of performance, Tr-StochM-C achieves $7.64\%$ error for Te-None and $8.25\%$ for Te-Sign (Table 2), which are state of the art for this data set without data augmentation, to the best of our knowledge. The network also exhibits a high degree of robustness (Figure 2A-C). It is interesting to note that using $\gamma=0.5$ during training, the expected value of the projected weight is no longer the same as $w_{ki}$ , which was originally thought to be important for these stochastic projections to work properly .

2 Towards a robust AlexNet

To see whether our results extend beyond CIFAR-10, we moved to ImageNet (ILSVRC2012), which is a dataset with $\sim 1.2$ M training images and 1K classes . We use an AlexNet with $5$ convolution and $2$ fully connected layers, modified with batch norm layers. These experiments were run in MatConvnet using SGD with no momentum and a batch size of 512.

Before benchmarking, we tested whether weight projections are needed to obtain robust networks. Accordingly, two AlexNets were trained without projections for $20$ K iterations (8.5 epochs): one without weight clipping (Tr-None-NC), and one with weight clipping (Tr-None-C). In both cases the top-5 error was reported for Te-None, Te-Round, and Te-Sign. Focusing on network Tr-None-NC (Figure 4A), we find that while Te-None reaches $28.1\%$ This network reaches $22.5\%$ after $100$ K iterations., the test error for Te-Round and Te-Sign are significantly worse. So our previous CIFAR-10 result for Tr-None-NC did not hold up for ImageNet.

Network Tr-None-C tells a different story (Figure 4B): the network has similar performance for Te-None, Te-Round, and Te-Sign, at $35\%$ , $34.7\%$ and $40.5\%$ respectively. This confirms our previous observation that weight clipping during training can influence network robustness. Although in this case, the peak error after $20$ K iterations is about $7\%$ higher for Te-None compared to network Tr-None-NC; to help to mitigate this effect, we use a clipping scheduling to increase clip values during training in later experiments. Also we observe that, Te-Round performs better than Te-Sign.

Hypotheses on learning robust networks

We put forth two ideas on how backprop can find paths to robust solutions.

First, we explore the idea that imposing constraints on weights can act as a regularizer, similar to DropConnect . This idea was first suggested in for the case of weight binarization. Here, we examine how imposing weight clipping (without weight projections) can also act as regularizer in the context of proximal methods; see for an excellent review on proximal methods. Consider minimizing $\mathcal{L}(w)+\mathcal{G}(w)$ where $w$ is a vector containing all the weights, $\mathcal{L}(\cdot)$ is the loss, and $\mathcal{G}(\cdot)$ is a regularizer. For convex $\mathcal{L}(\cdot)$ , the proximal gradient method is

However, if the weights are projected with $\phi_{\theta}$ before computing forward and backward steps, then backprop only has access to $\mathcal{L}(\phi_{\theta}(w))$ . By sampling from its distribution over several minibatches, gradient descent is traversing the alternative error surface

Hence the solution obtained by minimizing $E_{\text{data}}E_{\phi_{\theta}}[\mathcal{L}(\phi_{\theta}(w))]$ necessarily provides the most robustness against distortions under $\phi_{\theta}$ to its weights. Furthermore, the surface $E_{\text{data}}E_{\phi_{\theta}}[\mathcal{L}(\phi_{\theta}(w))]$ is a smoothed-over version of the surface $E_{\text{data}}\left[\mathcal{L}(w)\right]$ , where the smoothness is controlled by the distribution of the noise source underlying $\phi_{\theta}$ ’s stochasticity. During training, typically, backprop estimates the gradient of $E_{\text{data}}\left[\mathcal{L}(w)\right]$ which takes presumed stochasticity of the data into account to produce a smoothed estimate. In our case, backprop estimates the gradient of $E_{\text{data}}E_{\phi_{\theta}}[\mathcal{L}(\phi_{\theta}(w))]$ by additionally sampling the weight space in the neighborhood of $w$ . This provides additional gradient smoothing even when $\phi_{\theta}$ is deterministic to some extent. Thus, both the error surface and its gradient are smoother which may explain the empirical results observed in this paper. During testing, because the objective function is $\mathcal{L}(w)$ , using either $w$ directly or with a deterministic projection should yield the best results. For future work, we envision cooling the noise source underlying $\phi_{\theta}$ ’s stochasticity as training progress so that eventually

Taking a step back, it appears that in all these cases (weight clipping, deterministic projections, and stochastic projections) the weight gradients are distorted relative to the gradients from standard backprop. Viewing these distortions as a type of noise may bridge our findings with recent work that suggests adding explicit gradient noise results in better overall performance .

Conclusion and future work

We expand on previous work demonstrating that networks trained with backprop can become robust to specific weight distortions, such as binary weights. Here we show that imposing certain weight distortions during training leads to a regime where the network becomes robust not only to that distortion, but to an entire family of distortions as well.

Based on this observation, we proposed a novel rule that stochastically projects each weight to a random interval based on its current value. We hypothesize that this rule, similar to the stochastic projection rule in BinaryConnect, is not optimizing the weight values directly, but instead optimizing the neighborhood of the weight vector. In practice, our rule leads to state of the art performance for CIFAR-10 for both binary and non-binary weighted networks.

Our finding that a network can achieve $89\%$ on CIFAR-10 with $0.68$ bits per weight may also be of practical interest. One potential application is that weights can be implemented with noisy devices, which could have implications for neuromorphic computing.

More recently, research on binary weights has been extended to also include binary neuron activations . Training these networks is similar to the binary weight case, namely binary activations are imposed during training. We hypothesize that the neuron outputs in these models are also robust to distortions. If confirmed, this would suggest that imposing other activation constraints could improve performance. We cannot help but speculate that the built-in robustness to noise in synapses and neurons is an inherent characteristic of the brain and may prove invaluable in opening new directions in deep learning and neuromorphic computing.