Improving training of deep neural networks via Singular Value Bounding

I Introduction

Deep learning methods keep setting the new state-of-the-art for many computer vision problems, with image classification and object detection as the prominent examples. These practical successes are largely achieved by newly proposed deep architectures that have huge model capacities, including the general ones such as Inception and ResNet , and also specially designed ones such as Faster R-CNN and FCN . Training of these ultra-deep/ultra-wide networks are enabled by modern techniques such as Batch Normalization (BN) and residual learning .

In spite of these practical successes, however, optimization of deep networks remains an active topic in deep learning research. Until recently, deep networks are considered to be difficult to train. Researchers argue for different reasons causing such difficulties, such as the problem of vanishing/exploding gradients , internal shift of feature statistics , and also the proliferation of saddle points . To address these issues, different schemes of parameter initialization , shortcut connections , normalization of internal activations , and second-order optimization methods are respectively proposed.

In this work, we focus on another important issue to address the difficulty of training deep neural networks. In particular, given the high-dimensional solution space of deep networks, it is unclear on the properties of the (arguably) optimal solutions that can give good performance at inference. Without knowing this, training by a specified objective function easily goes to unexpected results, partially due to the proliferation of local optima/critical points . For example, it is empirically observed in that adding extra layers to a standard convolutional network (ConvNet) does not necessarily give better image classification results. This unclear issue is further compounded by other (aforementioned) optimization difficulties.

Existing deep learning research has some favors on the solutions of network parameters, and also on network architectures that can give desirable solutions. In particular, Arpit et al. study the properties of network parameters that can ensure accurate recovery of the true signals of hidden representations, and prove that for sparse true signals, e.g., those out of Rectified Linear Units (ReLU) activations , strong recovery can be achieved if the weight matrix is highly incoherent. Saxe et al. advocate orthogonal initialization of weight matrices, and theoretically analyze its effects on learning efficiency using deep linear networks. Practical results on image classification using orthogonal initialization are also presented in . In terms of the favored properties on network architectures, the development of Inception models relies on the Hebbian principles , and ResNet argues for residual learning by shortcut connections.

In this paper, we are inspired by the analysis of orthogonal initialization in , and aim to constrain the solutions of weight matrices in the orthogonal feasible set during the whole process of network training. To this end, we propose a simple yet effective method called Singular Value Bounding (SVB). In SVB, all singular values of each weight matrix are simply bounded in a narrow band around the value of $1$ (Section III). When using stochastic gradient descent (SGD) or its variants for network training, this amounts to turning SVB on by every a specified number of iterations. We present theoretical analysis, using deep linear networks, to show how such learned networks are better on forward-propagation to achieve training objectives, and backward-propagation of training errors (Section IV).

Batch normalization is a very effective method to improve and accelerate network training. We prove that in the framework of our theoretical analysis, trainable parameters in BN may cause ill-conditioned layer transform. We thus propose Bounded Batch Normalization (BBN), a technique that improves BN by removing this risk without sacrificing all its other benefits. BBN achieves this by simply bounding the values of BN parameters during training.

We present benchmark image classification experiments using both ConvNets and modern network architectures (Section VI). Our results show that SVB indeed improves over SGD based methods for training various architectures of deep networks, and in many cases with a large margin. Our proposed BBN further improves over BN. In particular, we achieve the state-of-the-art results of $3.06\%$ error rate on CIFAR10 and $16.90\%$ on CIFAR100 , using off-the-shelf network architectures (Wide ResNets ). Our preliminary results on the large-scale ImageNet dataset are consistent with those on moderate-scale ones.

II Related works

In this section, we briefly review the closely related deep learning methods that also pay attention to the properties of network solutions.

Saxe et al. theoretically study the gradient descent learning dynamics of deep linear networks, and give similar empirical insights for deep nonlinear networks. They further suggest that using orthogonal initialization of weight matrices can achieve learning efficiency similar to that of unsupervised pre-training. Mishkin and Matas present promising results on image classification, using the orthogonal initialization idea in . Our theoretical analysis in Section IV follows , but are different in the following aspects. We focus on studying the conditions when network training converges, while focuses on the conditions right after network initialization. Our analysis centers around our proposed SVB method, and we discuss how SVB can resolve the issues that appear as the network training proceeds. We also extend our theoretical analysis to BN , and propose a new BBN method that improves over BN for training modern deep networks.

Arpit et al. also study the properties of network parameters that can have good performance, but from a signal recovery point of view. In particular, they study the reverse data-generating properties of auto-encoders where input samples are generated from the true signals of hidden representations. They prove that for sparse true signals, e.g., those out of ReLU activations, strong recovery can be achieved if the weight matrix is highly incoherent. Different from , our main concern is on the properties of network parameters that can give good image classification performance by feed-forward computations.

In , a soft constraint technique is proposed to deal with the vanishing gradient in training recurrent neural networks (RNNs). The soft constraint regularizes the learning of weight matrices so that those better to achieve norm preservation of error signals across layers are favored. In contrast, our proposed SVB method directly controls the singular values of weight matrices, and norm preservation of error signals is only part of our benefits.

A recent work from Wisdom and Powers et al. shows full-capacity unitary recurrence matrices can be used in RNNs, and can be optimized over the differentiable manifold of unitary matrices. This improves over , where unitary recurrent matrices are restricted to be a product of parameterized unitary matrices. In contrast, we focus on convolutional networks in this work, where weight matrices are not square. We only enforce column or row vectors of weight matrices of ConvNets to be near orthogonal, while giving them more flexibility to better learn to the training tasks. This relaxation from strict orthogonality enables us to use very simple algorithms compatible with standard SGD based training. Practicably, we observe our SVB algorithm is just as efficient as SGD based training, while achieving the property of near orthogonality.

Other very recent relevant research includes that develops geometric framework to analyze network optimization on sub-manifolds of certain normalized kernels, including orthonormal weight matrices, and proposes a SGD based algorithm to optimize on the sub-manifolds with guaranteed convergence. In , Wang et al. propose Extended Data Jacobian Matrix (EDJM) as a network analyzing tool, and study how the spectrum of EDJM affects performance of different networks of varying depths, architectures, and training methods. Based on these observations, they propose a spectral soft regularizer that encourages major singular values of EDJM to be closer to the largest one (practically implemented on weight matrix of each layer). This is related, but different from our proposed hard constraint based SVB method.

III The proposed Singular Value Bounding algorithm

Training of deep neural networks is usually based on SGD or its variants . Given the training loss function ${\cal{L}}\left(\{\mathbf{x}_{i},\mathbf{y}_{i}\}_{i=1}^{K};\Theta\right)$, SGD updates $\Theta$ based on a simple rule of $\Theta_{t+1}\leftarrow\Theta_{t}-\eta\frac{\partial{\cal{L}}}{\partial{\Theta_{t}}}$, where $\eta$ is the learning rate. The gradient $\frac{\partial{\cal{L}}}{\partial{\Theta_{t}}}$ is usually computed from a mini-batch of training samples. Network training proceeds by sampling for each iteration $t$ a mini-batch from $\{\mathbf{x}_{i},\mathbf{y}_{i}\}_{i=1}^{K}$, until a specified number $T$ of iterations or the training loss plateaus.

Existing deep learning research suggests that in order to get good performance, initializations of $\Theta$ matter. In particular, scaled random Gaussian matrices are proposed in as the initializations of weight matrices $\{\mathbf{W}^{l}\}_{l=1}^{L}$, and random orthogonal ones are advocated in . Given different initializations, these methods train deep networks using SGD or its variants. Theoretical analysis in and empirical results in demonstrate some advantages of orthogonal initializations over Gaussian ones. In this work, we are interested in pushing a step further to know what solutions of $\Theta$ matter when network training converges, rather than just at the initialization. For the orthogonal case, our empirical results (cf. Figure 1) show that as the training proceeds, singular value spectra of weight matrices diverge from their initial conditions. We are thus motivated to investigate along this line, from the empirical observations of Figure 1 and also the theoretical analysis in .

More specifically, we propose a simple yet very effective network training method, which preserves the orthogonality of weight matrices during the procedure of network training. This amounts to solving the following constrained optimization problem

where $\cal{O}$ stands for the set of matrices whose row or column vectors are orthogonal (or near orthogonal). Compared with standard SGDs, the feasible set of problem (III) for $\{\mathbf{W}^{l}\}_{l=1}^{L}$ is much reduced. We approximately solve this problem based on SGD (or its variants): we simply bound, after every $T_{svb}$ iterations of SGD training, all the singular values of each $\mathbf{W}^{l}$, for $l=1,\dots,L$, in a narrow band $[1/(1+\epsilon),(1+\epsilon)]$ around the value of $1$, where $\epsilon$ is a specified small constant. Algorithm 1 presents details of our proposed Singular Value Bounding (SVB) method. In Section IV, we present theoretical analysis on deep linear networks to justify the advantages of our proposed SVB on forward propagation to achieve training objectives, and backward propagation of training errors. In Section V, we prove that BN could cause an ill-conditioned layer transform. To improve BN and make BN be compatible with SVB, we propose Bounded Batch Normalization (BBN) (cf. Algorithm 2), which removes such a risk by directly controlling the learning of BN parameters. Experiments in Section VI on image classification show that SVB improves over SGD based methods, and in many cases with a large margin. And BBN further improves the performance on deep networks with BN layers.

Empirical computation cost Applying SVB to network training amounts to solving singular value decompositions (SVD) for weight matrices of all the network layers. We note that this cost can be amortized by doing SVB every $T_{svb}$ number of iterations. We usually apply SVB once every epoch of SGD training (for CIFAR10 with a batch size of $128$, this amounts to doing SVB once every $391$ iterations). The wall-clock time caused by SVB is practically negligible. In fact, we often observe even faster training when using SVB, possibly due to the better conditioning of weight matrices resulting from SVB.

IV Propagations of all directions of variations with Singular Value Bounding

In this section, we present theoretical analysis on deep linear networks to discuss the importance of forward-propagating all the directions of training objectives and backward-propagating those of training errors, in order to better train deep neural networks. Our analyses resemble, but are different from, those in (cf. Section II for details of the difference). These analyses justify our proposed SVB algorithm, and are supported by the experimental results reported in Section VI.

We start our analysis of optimal network solutions with a simple two-layer linear network that computes $\mathbf{W}^{2}\mathbf{W}^{1}\mathbf{x}$, where we have used a linear activation $f(\mathbf{z})=\mathbf{z}$ and ignored the bias for simplicity. Using squared Euclidean distance as the training criterion gives the following loss function ${\cal{L}}=\frac{1}{2K}\sum_{i=1}^{K}\|\mathbf{y}_{i}-\mathbf{W}^{2}\mathbf{W}^{1}\mathbf{x}_{i}\|_{2}^{2}$. To minimize $\cal{L}$ with respect to (w.r.t.) $\mathbf{W}^{1}$ and $\mathbf{W}^{2}$, we note that the optimal solutions are characterized by the gradients

As suggested in , when we initialize $\mathbf{W}^{1}$ and $\mathbf{W}^{2}$ as

Since $\mathbf{R}$ is fixed, the above conditions ensure that $\mathbf{W}^{1}$ and $\mathbf{W}^{2}$ are optimized along their respective independent directions of variations. Denote $s_{m}$ and $t_{m}$, $m=1,\dots,\min(N_{y},N_{1},N_{x})$, are the $m^{th}$ diagonal entries of $\mathbf{S}^{1}$ and $\mathbf{S}^{2}$ respectively. By change of optimization variables, (IV-A) can be further simplified as the following equations for each $m^{th}$ direction of variations

In fact, the gradients (5) w.r.t. $s_{m}$ and $t_{m}$ arise from the following energy function

showing that the product of optimal pairs $s_{m}$ and $t_{m}$ approaches $\sigma_{m}$.

We subsequently extend the analysis from (IV-A) to (6) for a deep linear network of $L$ layers. With the same loss function $\cal{L}$ of squared Euclidean distance, the optimal weight matrix $\mathbf{W}^{l}$ of the $l^{th}$ layer is characterized by the gradient

where $\prod_{i=l}^{l^{\prime}}\mathbf{W}^{i}=\mathbf{W}^{l^{\prime}}\mathbf{W}^{l^{\prime}-1}\cdots\mathbf{W}^{l}$ with the special case that $\prod_{i=l}^{l^{\prime}}\mathbf{W}^{i}=\mathbf{I}$ when $l>l^{\prime}$, and we have assumed in (7) that $\mathbf{C}^{xx}=\mathbf{I}$. Similar to (3), when we initialize weight matrices of the deep network as $\mathbf{W}^{l}=\mathbf{R}^{l+1}\mathbf{S}^{l}\mathbf{R}^{l\top}$ for any $l\in\{1,\dots,L\}$, where each $\mathbf{R}^{l}$ is an orthogonal matrix with the special cases that $\mathbf{R}^{1}=\mathbf{V}^{x}$ and $\mathbf{R}^{L+1}=\mathbf{U}^{y}$, and each $\mathbf{S}^{l}$ is an diagonal matrix with nonnegative entries, and keep $\{\mathbf{R}^{l}\}_{l=1}^{L+1}$ fixed during optimization Alternatively, one might relax this constraint and update $\{\mathbf{W}^{l}\}_{l=1}^{L}$ using standard methods such as SGD, and change the left and right singular vectors of each updated $\mathbf{W}^{l}$ to satisfy $\mathbf{W}^{l}=\mathbf{R}^{l+1}\mathbf{S}^{l}\mathbf{R}^{l\top}$ (with varying sets of $\{\mathbf{R}^{l}\}_{l=1}^{L+1}$). However, this would cause mixing of different directions in the connecting output/input spaces across layers., the gradient (7) at optimal solutions can be derived as

By change of optimization variables, (8) can be further simplified as the following independent gradient for the $m^{th}$ direction of variations with $m\leq M=\min(N_{y},\dots,N_{l},\dots,N_{x})$

which turns out to be the gradient of the energy function

The positive scalar $\sigma_{m}$ in (10) represents the strength of the $m^{th}$ direction of input-output correlations. It is usually fixed given provided training data. To characterize the conditions under which the minimum energy of (10) can be achieved, denote $s_{m}^{l_{\max}}=\max(s_{m}^{1},\dots,s_{m}^{L})$ and $s_{m}^{l_{\min}}=\min(s_{m}^{1},\dots,s_{m}^{L})$. One can easily prove that when $L\rightarrow\infty$, it is necessary that $s_{m}^{l_{\max}}>1$ and $s_{m}^{l_{\min}}<1$. Conversely, the sufficient conditions for not achieving the minimum energy of (10) are either $s_{m}^{l_{\max}}<1$ or $s_{m}^{l_{\min}}>1$, when $L\rightarrow\infty$.

For any fixed and finite $\sigma_{m}$, our proposed SVB algorithm is potentially able to achieve the minimum energy of (10) (although it does not meet the assumptions used to derive (10)), by choosing an appropriate value of $\epsilon$ so that values of $\{s_{m}^{l}\}_{l=1}^{L}$ are properly learned to range in a narrow band $\left[1/(1+\epsilon),1+\epsilon\right]$. This applies to any of the $M$ directions of input-output correlations. Existing network training methods have no such constraints, and $\{s_{m}^{l}\}$ of all layers/directions are free to be scaled up or down, resulting in very uneven magnitude distribution of $\{\{s_{m}^{l}\}_{l=1}^{L}\}_{m=1}^{M}$. Consequently, training easily falls in local minima that minimize (10) for certain directions, but not for all of the $M$ ones. And only parts of the input-output correlations are taken into account during learning.

Our derivation from (7) to (8) requires that the output singular vectors of the weight matrix of layer $l$ be the input singular vectors of that of layer $l+1$. However, it does not hold true in the SGD based Algorithm 1, where weight matrices are updated without such constraints. Consider a two-layer basic component $\mathbf{W}^{l+1}\mathbf{W}^{l}$ in (7), which propagates signal activations (and hence information of input variations) from layer $l$ to layer $l+1$. After SGD updating, Algorithm 1 computes SVDs of the updated $\mathbf{W}^{l+1}$ and $\mathbf{W}^{l}$, resulting in $\mathbf{W}^{l+1}\mathbf{W}^{l}=\mathbf{U}^{l+1}\mathbf{S}^{l+1}\mathbf{V}^{l+1\top}\mathbf{U}^{l}\mathbf{S}^{l}\mathbf{V}^{l\top}$. While one may initialize $\mathbf{W}^{l+1}$ and $\mathbf{W}^{l}$ such that $\mathbf{V}^{l+1}=\mathbf{U}^{l}$, after SGD updating, they are generally not equal. Denote $\mathbf{M}=\mathbf{S}^{l+1}\mathbf{V}^{l+1\top}\mathbf{U}^{l}\mathbf{S}^{l}$, we have

where $\mathbf{M}_{m,m^{\prime}}$ is the $(m,m^{\prime})$ entry of $\mathbf{M}$, $\mathbf{v}_{m}^{l+1}$ is the $m^{th}$ column of $\mathbf{V}^{l+1}$, $\mathbf{u}_{m^{\prime}}^{l}$ is the $m^{\prime th}$ column of $\mathbf{U}^{l}$, and $s_{m}^{l+1}$ and $s_{m^{\prime}}^{l}$ are respectively the $m^{th}$ and $m^{\prime th}$ singular values of $\mathbf{S}^{l+1}$ and $\mathbf{S}^{l}$. By projecting $\mathbf{u}_{m^{\prime}}^{l}$ onto $\mathbf{v}_{m}^{l+1}$, $\mathbf{v}_{m}^{l+1\top}\mathbf{u}_{m^{\prime}}^{l}$ represents the mixing of the $m^{\prime th}$ direction of variations in the output space of layer $l$ with the $m^{th}$ one in the input space of layer $l+1$. By bounding $s_{m}^{l+1}$ and $s_{m^{\prime}}^{l}$, our proposed SVB algorithm controls both the independent (when $m=m^{\prime}$ and the assumptions from (7) to (8) hold), and the mixing strengths of propagation across layers. Without such constraints, some directions of variations could be over-amplified while others are strongly attenuated, when signals are propagated from lower layers to higher layers.

IV-B The backward propagation

For a deep linear network that performs cascaded computations of $\mathbf{x}^{l}=\mathbf{W}^{l}\mathbf{x}^{l-1}$ for $l=1,\dots,L$, the gradient of loss function $\cal{L}$ w.r.t. the output activation $\mathbf{x}^{l}$ of layer $l$ is written as

where $\frac{\partial{\cal{L}}}{\partial{\mathbf{x}^{L}}}$ contains the error vector for back-propagation. For any $i\in\{l+1,\dots,L\}$, assume $\mathbf{W}^{i}$ satisfies the condition $\mathbf{W}^{i}=\mathbf{R}^{i+1}\mathbf{S}^{i}\mathbf{R}^{i\top}$ that we have used to derive from (7) to (10), with analysis similar to the forward propagation case, we have

where $\mathbf{r}_{m}^{L+1}$ (or $\mathbf{r}_{m}^{l+1}$) denotes the $m^{th}$ column of $\mathbf{R}^{L+1}$ (or $\mathbf{R}^{l+1}$), and $M=\min(N_{L},\dots,N_{l})$. As the network goes deep (i.e., $L$ becomes large), $\prod_{i=l+1}^{L}s_{m}^{i}$ would either explode or vanish if $\{s_{m}^{i}\}_{i=l+1}^{L}$ do not satisfy a necessary condition similar to the one for achieving the minimum of (10). Consequently, the $m^{th}$ component error vector $\left(\prod_{i=l+1}^{L}s_{m}^{i}\right)\mathbf{r}_{m}^{l+1}\mathbf{r}_{m}^{L+1\top}\frac{\partial{\cal{L}}}{\partial{\mathbf{x}^{L}}}$ in (13) would either explode or vanish. When $\epsilon\rightarrow 0$ in Algorithm 1, our proposed method guarantees that all the $M$ components of the error vector would propagate to lower layers without attenuation or explosion. In the ideal case of $N_{L}=\dots=N_{l}$, our method also guarantees that $\|\frac{\partial{\cal{L}}}{\partial{\mathbf{x}^{l}}}\|_{2}=\|\frac{\partial{\cal{L}}}{\partial{\mathbf{x}^{L}}}\|_{2}$, i.e., to preserve the norm of error vector. Without such constraints on singular values of $\{\mathbf{W}^{i}\}_{i=l+1}^{L}$, it is still possible that the norm of error vector is preserved by amplifying some singular values while shrinking others, as the way advocated in . However, its norm preservation is achieved in a rather anisotropic way.

V Compatibility with Batch Normalization

In this section, we investigate how our proposed network training algorithm could be compatible with Batch Normalization . BN addresses a network training issue called internal covariate shift, which slows down the training since distributions of each layer’s inputs keep changing during the training process. BN alleviates this issue by inserting into network trainable normalization layers, which normalize each layer’s neuron activations as zero mean and unit variance in a mini-batch and neuron-wise manner.

Inserting $\mathbf{z}=\mathbf{W}\mathbf{x}$ into (14) we get

which is simply a standard network layer with change of variables. The following lemma suggests that we may bound the entries $\{\gamma_{i}/\varsigma_{i}\}_{i=1}^{N}$ of the product of the diagonal matrices $\Gamma$ and $\Sigma$, to make our proposed SVB algorithm be compatible with BN.

Proof of the lemma is given in Appendix A. Lemma 1 suggests that for a deep network with BN layers, the trainable parameters $\{\gamma_{i}\}_{i=1}^{N}$, together with sample statistics $\{\varsigma_{i}\}_{i=1}^{N}$, could change the conditioning of layer transform, and consequently the behaviors of signal propagation across network layers. In particular, when absolute values $\{|\gamma_{i}/\varsigma_{i}|\}_{i=1}^{N}$ of the diagonal entries of $\Gamma\Sigma$ for all the network layers simultaneously drift up or down away from the value of $1$, signal propagation would be susceptible to explosion or attenuation when the network goes deep. One direct way to remove this risk is to control the values of $\{\gamma_{i}/\varsigma_{i}\}_{i=1}^{N}$, e.g., to let them be around $1$. However, this would also remove an important benefit of BN. More specifically, the introduction of trainable scaling parameters $\{\gamma_{i}\}_{i=1}^{N}$ in BN is to make sure that after neuron-wise normalization by $\{\varsigma_{i}\}_{i=1}^{N}$ (and $\{\mu_{i}\}_{i=1}^{N}$), the change to layer outputs is compensated by $\{\gamma_{i}\}_{i=1}^{N}$, so that the BN transform is overall an identity transform . One might expect that the value of each $\varsigma_{i}$ in $\Sigma$ is similar to that of the corresponding $\gamma_{i}$ in $\Gamma$. However, this is not the case in practice. In fact, $\{\gamma_{i}\}_{i=1}^{N}$ bring additional and significant benefits to training of deep neural networks: the decoupled $\{\gamma_{i}\}_{i=1}^{N}$ enable scales of the magnitude of features at different network layers become freely adjustable for better training objectives. This advantage is also leveraged in recent works such as to improve network training. Inspired by this scheme of BN, we introduce a decoupled scalar $\alpha$ from $\Gamma\Sigma$, and propose to control the re-scaled version $\{\frac{1}{\alpha}\gamma_{i}/\varsigma_{i}\}_{i=1}^{N}$, instead of $\{\gamma_{i}/\varsigma_{i}\}_{i=1}^{N}$, to make our proposed SVB be compatible with BN. We set $\alpha=\frac{1}{N}\sum_{i=1}^{N}\gamma_{i}/\varsigma_{i}$ during network training. Note that re-scaling the magnitude scales of features at different layers is equivalent to simultaneously scaling up $\{s_{m}^{l}\}_{m=1}^{M}$ of all the $M$ directions in (10) for certain layers, while simultaneously scaling down for other layers, and this does not cause sacrifice of propagation of certain directions of input-output correlations. Algorithm 2 presents our improved BN transform called Bounded Batch Normalization (BBN). We note that in Algorithm 2, we do not take the absolute values. This is because values of $\{\gamma_{i}\}_{i=1}^{N}$ are usually initialized as $1$, and they are empirically observed to keep positive during the process of network training. Experiments in Section VI show that image classification results are improved when using BBN instead of BN, demonstrating a consistency between our theoretical analysis and practical results.

VI Experiments

In this section, we present image classification results to show the efficacy of our proposed SVB and BBN algorithms for training deep neural networks. We use benchmark datasets including CIFAR10, CIFAR100 , and ImageNet . CIFAR10 is intensively used for our controlled studies. We investigate how SVB and BBN perform on standard ConvNets, and also the modern architectures of (pre-activation versions of) ResNets , Wide ResNets , and Inception-ResNets .

In this section, we use ConvNets to study the behaviors of our proposed SVB algorithm on deep network training. We choose modern convolutional architectures from . The networks start with a conv layer of $16$ $3\times 3$ filters, and then sequentially stack three types of $2X$ conv layers of $3\times 3$ filters, each of which has the feature map sizes of $32$, $16$, and $8$, and filter numbers $16$, $32$, and $64$, respectively. Spatial sub-sampling of feature maps is achieved by conv layers of stride $2$. The networks end with a global average pooling and fully-connected layers. Thus for each network we have $6X+2$ weight layers in total. We use networks of $X=3$ and $X=6$ for our studies, which give $20$- and $38$-layer networks respectively.

The used CIFAR10 dataset consists of $10$ object categories of $60,000$ color images of size $32\times 32$ ($50,000$ training and $10,000$ testing ones). We use raw images without pre-processing. The data augmentation follows the standard manner in : during training, we zero-pad $4$ pixels along each image side, and sample a $32\times 32$ region crop from the padded image or its horizontal flip; during testing, we simply use the original non-padded image.

Figure 2 shows that for each depth case, our results using SVB are consistently better than those from standard SGD with momentum. For $20$-layer network, SGD with momentum gives an error rate of $9.21$, and our best result using SVB improves the error rate to $8.03$. For $38$-layer network, SGD with momentum gives an error rate of $12.08$, and our best result using SVB improves the error rate to $9.90$. These results verify that bounding the singular values of weight matrices indeed improves the conditioning of layer transform.

Replacing BN with BBN further improves the result to $7.85$ for the $20$-layer network, and to $9.66$ for the $38$-layer network. The improvement is however less significant as compared with that from SGD with momentum to SVB. This shows that in the case of plain ConvNets, SVB has largely improved the conditioning of layer transform, and the ill-conditioning caused by BN is not severe. Indeed, in the more complex ResNet type architectures, BBN improves over BN effectively at a higher accuracy level, as presented shortly.

Comparative results in Figure 2 also suggest that with the increase of network layers, training becomes more difficult: results of deeper network ($X=6$) are worse than those of shallower one ($X=3$). This is consistent with the observations in . Although our SVB and BBN improve the performance, they do not solve this training difficulty of plain ConvNets.

VI-B Ablation studies using ResNet

We conduct experiments to investigate whether our proposed SVB and BBN methods are effective for “residual learning” . We use an architecture similar to those presented in Section 4.2 in , but change it to the pre-activation version . The network construction is based on the ConvNets presented in Section VI-A, and we use an “identify shortcut” to connect every two conv layers of $3\times 3$ filters, and use a “projection shortcut” when sub-sampling of feature maps is needed. We use a network of $X=11$ for experiments in this section, which gives $68$ weight layers.

VI-C Comparisons with the state-of-the-art

We apply our proposed methods to Wide ResNet , and a pre-activation version of Inception-ResNet , and compare with the state-of-the-art results on CIFAR10 and CIFAR100. The CIFAR100 dataset has the same number of $32\times 32$ color images as CIFAR10 does, but it has $100$ object categories and each category contains one tenth images of those of CIFAR10. We use raw data without pre-processing, and do data augmentation using the same manner as for CIFAR10.

VI-D Preliminary results on ImageNet

We present preliminary results on ImageNet , which has $1.28$ million images of $1000$ classes for training, and $50$ thousand images for validation. The data augmentation scheme follows . We investigate how SVB and BBN may help large-scaling learning, for which we use the pre-activation version of Inception-ResNet . We use the same parameter settings as those for the CIFAR10 experiments in Section VI-C, except the learning rate that starts from $0.045$. Table IV shows that SVB and BBN indeed improve the large-scale learning, with a similar performance gain for top-1 and top-5 errors. The improvement is however lower than what we expected. We are interested for further studies in future research. We note that our architecture is almost identical to , but we did not manage to get the results in , possibly due to the different choices of gradient descent methods ( uses RMSProp while ours are based on SGD with momentum).

VII Conclusions

In this work, we present a simple yet effective method called Singular Value Bounding, to improve training of deep neural networks. SVB iteratively projects SGD based updates of network weights into a near orthogonal feasible set, by constraining all singular values of each weight matrix in a narrow band around the value of $1$. We further propose Bounded Batch Normalization, a method to remove the risk of ill-conditioned layer transform caused by batch normalization. We present theoretical analysis to justify our proposed methods. Experiments on benchmark image classification tasks show the efficacy.

References

Appendix A

Let $\mathbf{x}^{*}=\arg\max_{\mathbf{x}\neq 0}\frac{\|\widetilde{\mathbf{W}}\mathbf{x}\|_{2}}{\|\mathbf{x}\|_{2}}$, we have

We next consider the special case of $M\leq N$ and $\textrm{rank}(\mathbf{W})=M$. Without loss of generality, we assume diagonal entries $\{g_{i}\}_{i=1}^{M}$ of $\mathbf{G}$ are all positive and ordered. By definition we have $\widetilde{\mathbf{W}}=\mathbf{I}\mathbf{G}\mathbf{W}$, where $\mathbf{I}$ is an identity matrix of size $M\times M$. Let $\mathbf{V}=\left[\mathbf{W}^{\top},\mathbf{W}^{\bot\top}\right]$, where $\mathbf{W}^{\bot}$ denotes the orthogonal complement of $\mathbf{W}$, we thus have the SVD of $\widetilde{\mathbf{W}}$ by construction as $\widetilde{\mathbf{W}}=\mathbf{I}\left[\mathbf{G},\mathbf{0}\right]\mathbf{V}^{\top}$. When some values of $\{g_{i}\}_{i=1}^{M}$ are not positive, the SVD can be constructed by changing the signs of the corresponding columns of either $\mathbf{I}$ or $\mathbf{V}$. Since matrix singular values are uniquely determined (while singular vectors are not), singular values of $\widetilde{\mathbf{W}}$ are thus exactly $\{|g_{i}|\}_{i=1}^{M}$. ∎