Spectral Norm Regularization for Improving the Generalizability of Deep Learning

Introduction

Deep learning has been successfully applied to several machine learning tasks including visual object classification , speech recognition , and natural language processing . A well-known method of training deep neural networks is stochastic gradient descent (SGD). SGD can reach a local minimum with high probability over the selection of the starting point , and all local minima attain similar loss values . However, the performance on test data, that is, generalizability, can be significantly different among these local minima. Despite the success of deep learning applications, we lack the understanding of generalizability, even though there has been progress .

While understanding the generalizability of deep learning is an interesting topic, it is important from a practical point of view. For example, suppose that we are training a deep neural network using SGD and want to parallelize the process using multiple GPUs or nodes to accelerate the training. A well-known method for achieving this is synchronous SGD , which requires a large minibatch to effectively exploit parallel computation on multiple GPUs or nodes. However, it is reported that models trained through synchronous SGD with large minibatches exhibit poor generalization , and a method is required to resolve this problem.

In this study, we consider the generalizability of deep learning from the perspective of sensitivity to input perturbation. Intuitively, if a trained model is insensitive or sensitive to the perturbation of an input, then the model is confident or not confident about the output, respectively. As the performance on test data is important, models that are insensitive to the perturbation of test data are required. Note that adversarial training is designed to achieve insensitivity to the perturbation of training data, and it is not always effective for achieving insensitivity to the perturbation of test data.

To obtain insensitivity to the perturbation of test data, we propose a simple and effective regularization method, referred to as spectral norm regularization. As the name suggests, spectral norm regularization prevents the weight matrices used in neural networks from having large spectral norms. Through this, even though test data are not known in advance, a trained model is ensured to exhibit slight sensitivity to the perturbation of test data.

Using several real-world datasets, we experimentally confirm that models trained using spectral norm regularization exhibit better generalizability than models trained using other baseline methods. It is claimed in that the maximum eigenvalue of the Hessian predicts the generalizability of a trained model. However, we show that the insensitivity to the perturbation of test data is a more important factor for predicting generalizability, which further motivates the use of spectral norm regularization. Finally, we show that spectral norm regularization effectively reduces the spectral norms of weight matrices.

The rest of this paper is organized as follows: We review related works in Section 2. In Section 3, we explain spectral norm regularization and compare it with other regularizing techniques . Experimental results are provided in Section 4, and conclusions are stated in Section 5.

Related Works

A conventional method of understanding the generalizability of a trained model is the notion of the flatness/sharpness of a local minimum . A local minimum is (informally) referred to as flat if its loss value does not increase significantly when it is perturbed; otherwise, it is referred to as sharp. In general, the high sensitivity of a training function at a sharp local minimizer negatively affects the generalizability of the trained model. In , this is explained in more detail through the minimum description length theory, which states that statistical models that require fewer bits to describe generalize better .

It is known that SGD with a large minibatch size leads to a model that does not generalize well . In , this problem is studied based on the flatness/sharpness of the obtained local minima. They formulated a flat local minimum as a local minimum at which all eigenvalues of the Hessian are small (note that all eigenvalues are non-negative at a local minimum). Using a proxy measure, they experimentally showed that SGD with a smaller minibatch size tends to converge to a flatter minimum.

The notion of flat/sharp local minima considers the sensitivity of a loss function against the perturbation of model parameters. However, it is natural to consider the sensitivity of a loss function against the perturbation of input data, as we discuss in this paper. In , the perturbation to training data that increases the loss function the most is considered, and the resulting perturbed training data are referred to as adversarial examples. It is reported in that training using adversarial examples improves test accuracy.

Recently, showed that the classical notions of Rademacher complexity and the VC dimension are not adequate for understanding the generalizability of deep neural networks.

Note that the spectral norm of a matrix is equal to its largest singular value. Singular values have attracted attention in the context of training recurrent neural networks (RNN). In , it is shown that by restricting the weight matrices in RNN to be unitary or orthogonal, that is, matrices with all singular values equal to one, the problem of diminishing and exploding gradients can be prevented and better performance can be obtained.

Spectral Norm Regularization

In this section, we explain spectral norm regularization and how it reduces the sensitivity to test data perturbation.

which corresponds to the largest singular value of $A$. Hence, the function $f_{\Theta}$ is insensitive to the perturbation of ${\bm{x}}$ if the spectral norm of $W_{\Theta,{\bm{x}}}$ is small.

2 Details of spectral norm regularization

We describe how to handle convolutions because they are used widely in recent applications of deep neural networks. Consider a convolutional layer with $a$ input channels, $b$ output channels, and a $k_{w}\times k_{h}$-sized kernel. This implies that the convolution has $abk_{w}k_{h}$ parameters. Note that a value in an output channel is determined using $ak_{w}k_{h}$ values in the input channels. Hence, we align the parameters as a matrix of size $b\times ak_{w}k_{h}$ and apply the abovementioned power iteration method to the matrix to calculate its spectral norm and gradient.

3 Comparison with other regularization methods

We now compare the spectral norm regularization with other regularization techniques.

Weight decay, or the Frobenius norm regularization, is a well-known regularization technique for deep learning. It considers the following problem:

Adversarial training.

Adversarial training considers the following problem:

Jacobian regularization.

The Jacobian regularization promotes the smoothness of a model against input perturbation. However, this regularization is impractical because calculating the derivative of a Jacobian with respect to parameters is computationally expensive. To resolve the issue, Gu et al. proposed an alternative method that regularizes layer-wise Jacobians:

Experiments

In this section, we experimentally demonstrate the effectiveness of spectral norm regularization on classification tasks over other regularization techniques, and confirm that the insensitivity to test data perturbation is an important factor for generalization.

All the training methods discussed here are based on stochastic gradient descent (SGD). We consider two regimes for the choice of the mini-batch size. In the small-batch regime, we set the mini-batch size to $B=64$, and in the large-batch regime, we set it to $B=4096$. In our experiments, we compared the following four problems:

Vanilla problem (vanilla): As a vanilla problem, we considered empirical risk minimization without any regularization, that is, $\mathop{\text{minimize}}_{\Theta}\frac{1}{K}\sum_{i=1}^{K}L(f_{\Theta}({\bm{x}}_{i}),{\bm{y}}_{i})$, where $L$ is the cross entropy.

Weight decay (decay): We considered the problem (2), where $L$ is the cross entropy. We selected the regularization factor $\lambda=10^{-4}$.

Adversarial training (adversarial): We considered the problem (3), where $L$ is the cross entropy. We selected $\alpha=0.5$ and $\epsilon=1$, as suggested in .

Spectral norm regularization (spectral): We considered the problem (1), where $L$ is the cross entropy. We selected the regularization factor $\lambda=0.01$.

We trained models using Nesterov’s accelerated gradient descent with momentum 0.9. We decreased the learning rate by a factor of $1/10$ when the half and the three quarters of the training process have passed. We optimized the hyper-parameters through a grid search and selected those that showed a reasonably good performance for every choice of neural network and mini-batch size.

In our experiments, we used the following four settings on the model and dataset.

The VGG network (VGGNet for short) on the CIFAR-10 dataset .

The network in network (NIN) model on the CIFAR-100 dataset .

The densely connected convolutional network (DenseNet) having a depth of 40 on the CIFAR-100 dataset.

DenseNet having a depth of 22 on the STL-10 dataset . The depth was decreased because of the memory consumption issue.

We preprocessed all the datasets using global contrast normalization. We further applied data augmentation with cropping and horizontal flip on STL-10 because the number of training data samples is only 5,000, which is small considering the large mini-batch size of 4096. The learning rate of SGD was initialized to $0.1$ in the small-batch regime and $1.0$ in the large-batch regime for the NIN and DenseNet models on the CIFAR-100 dataset, and was initialized to $0.01$ in the small-batch regime and $0.1$ in the large-batch regime for the VGGNet and DenseNet models on the CIFAR-10 dataset.

First, we look at the test accuracy obtained by each method, which is summarized in the left columns of Table 1. In the small-batch regime, decay and spectral show better test accuracies than the other two methods. In the large-batch regime, spectral clearly achieves the best test accuracy for every model. Although the test accuracy decreases as the mini-batch size increases, as reported in , the decrease in the test accuracy of spectral is less significant than those of the other three methods.

Next, we look at the generalization gap. We define the generalization gap at a threshold $\alpha$ as the minimum difference between the training and test accuracies when the test accuracy exceeds $\alpha$. The generalization gap of each method is summarized in the right columns of Table 1. For each setting, we selected the threshold $\alpha$ so that every method achieves a test accuracy that (slightly) exceeds this threshold. For each of the settings, except for the DenseNet model on the STL-10 dataset, spectral clearly shows the smallest generalization gap followed by decay, which validates the effectiveness of spectral.

Figure 1 shows the training curve of the DenseNet model on the CIFAR-100 dataset. The results for other settings are given in Appendix A. As we can observe, in both the small-batch and large-batch regimes, spectral shows the smallest generalization gaps. The generalization gap of decay increases as the mini-batch size increases, whereas that of spectral does not increase significantly. Investigating the reason behind this phenomena is an interesting future work.

The choice of the mini-batch size and the training method is not important to obtain a model with a good training accuracy; all of them exceeds 95%. However, obtaining a model with a good test accuracy, or a small generalization gap, is important. In the subsequent sections, we investigate which property of a trained model determines its generalization gap.

To summarize, spectral consistently achieves the small generalization gap and shows the best test accuracy, especially in the large-batch regime.

2 Sensitivity to the perturbation of the input

3 Maximum eigenvalue of the Hessian with respect to the model parameters

In , it is claimed that the maximum eigenvalue of the Hessian of the loss function defined with the training data predicts the generalization gap well. To confirm this claim, we computed the maximum eigenvalue of the DenseNet model trained on the CIFAR-100 dataset, shown in Figure 4. As it is computationally expensive to compute the Hessian, we approximated its maximum eigenvalue by using the power iteration method because we can calculate the Hessian-vector product without explicitly calculating the Hessian . We also computed the maximum eigenvalue of the Hessian of the loss function defined with the test data.

We can observe that, for vanilla, larger eigenvalues (in both the training and test data) are obtained if the mini-batch size is increased, which confirms the claim of . However, the models trained with regularizations tend to have larger eigenvalues, although they have better generalizability. In particular, the models trained by spectral have the largest eigenvalues, although they have the best generalizability as we have seen in Section 4.1.

This phenomena can be understood as follows: If we train a model without regularization, then a small perturbation does not significantly affect the Frobenius and spectral norms of weight matrices because they are already large. However, if we train a model with regularization, then because of these small norms a small perturbation may significantly affect those norms, which may cause a significant change in the output.

To summarize, this experiment indicates that the maximum eigenvalue of the Hessian of the loss function is not a suggestive measure for predicting generalizability.

4 Singular values of weight matrices

Finally, we look at the singular values of weight matrices in the models trained by each method. Figures 4 shows the singular values of a weight matrix taken from the DenseNet model on the CIFAR-100 dataset. The matrix is selected arbitrarily because all matrices showed similar spectra.

We can observe that the spectrum of vanilla is highly skewed, and adversarial and decay shrink the spectrum while maintaining the skewed shape. In contrast, the spectrum of spectral is flat. This behavior is as expected because the spectral norm regularization tries to reduce the largest singular value. Because the maximum singular value obtained by spectral is low, we obtain less sensitivity to the perturbation of the input.

Conclusions

In this work, we hypothesized that a high sensitivity to the perturbation of the input data degrades the performance of the data. In order to reduce the sensitivity to the perturbation of the test data, we proposed the spectral norm regularization method, and confirmed that it exhibits a better generalizability than other baseline methods through experiments. Experimental comparison with other methods indicated that the insensitivity to the perturbation of the test data plays a crucial role in determining the generalizability.

There are several interesting future directions to pursue. It is known that weight decay can be seen as a regularization in MAP estimation derived from a Gaussian prior to the model parameters. Is it possible to understand spectral norm regularization as a derivation of a prior? We also need a theoretical understanding of the effect of spectral norm regularization on generalization. It is known that, in some ideal cases, weight decay improves generalizability by preventing neural networks from fitting noises . Can we extend this argument to spectral norm regularization?

Acknowledgement

The authors thank Takuya Akiba and Seiya Tokui for helpful discussions.

References

Appendix A Accuracy

The training curves for the VGGNet, NIN, and DenseNet models on the STL-10 dataset are shown in Figure 5. We can observe that, in every setting, spectral shows the smallest generalization gap or the best test accuracy, which demonstrates that spectral can effectively reduce the generalization gap without suppressing the model complexity significantly.

Appendix B Sensitivity to the perturbation of the input