Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks

Introduction

Deep learning achieves great successes in almost all real-world applications ranging from image processing (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012) to Go games (Silver et al., 2016). Understanding and explaining the success of deep learning has thus become a central problem for theorists. One of the mysteries is that the neural networks used in practice are often heavily over-parameterized such that they can even fit random labels to the input data (Zhang et al., 2017), while they can still achieve very small generalization error (i.e., test error) when trained with real labels.

There are multiple recent attempts towards answering the above question and demystifying the success of deep learning. Soudry and Carmon (2016); Safran and Shamir (2016); Arora et al. (2018a); Haeffele and Vidal (2015); Nguyen and Hein (2017) showed that over-parameterization can lead to better optimization landscape. Li and Liang (2018); Du et al. (2019b) proved that with proper random initialization, gradient descent (GD) and/or stochastic gradient descent (SGD) provably find the global minimum for training over-parameterized one-hidden-layer ReLU networks. Arora et al. (2019a) analyzed the convergence of GD to global optimum for training a deep linear neural network under a set of assumptions on the network width and initialization. Du et al. (2019a); Allen-Zhu et al. (2019b); Zou et al. (2019) studied the convergence of gradient-based method for training over-parameterized deep nonlinear neural networks. Specifically, Du et al. (2019a) proved that gradient descent can converge to the global minima for over-parameterized deep neural networks with smooth activation functions. Allen-Zhu et al. (2019b); Zou et al. (2019) independently proved the global convergence results of GD/SGD for deep neural networks with ReLU activation functions in the over-parameterization regime. However, in such an over-parametrized regime, the training loss function of deep neural networks may have potentially infinitely many global minima, but not all of them can generalize well. Hence, convergence to the global minimum of the training loss is not sufficient to explain the good generalization performance of GD/SGD.

There are only a few studies on the generalization theory for learning neural networks in the over-parameterization regime. Brutzkus et al. (2018) showed that SGD learns over-parameterized networks that provably generalize on linearly separable data. Song et al. (2018) showed that when training two-layer networks in a suitable scaling limit, the SGD dynamic is captured by a certain non-linear partial differential equation with nearly ideal generalization error. Li and Liang (2018) relaxed the linear separable data assumption and proved that SGD learns an over-parameterized network with a small generalization error when the data comes from mixtures of well-separated distributions. Allen-Zhu et al. (2019a) proved that under over-parameterization, SGD or its variants can learn some notable hypothesis classes, including two and three-layer neural networks with fewer parameters. Arora et al. (2019b) provided a generalization bound of GD for two-layer ReLU networks based on a fine-grained analysis on how much the network parameters can move during GD. Nevertheless, all these results are limited to two or three layer neural networks, and cannot explain the good generalization performance of gradient-based methods for deep neural networks. For deep neural networks, existing generalization error bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018a; Golowich et al., 2018; Dziugaite and Roy, 2017; Arora et al., 2018b; Li et al., 2018a; Neyshabur et al., 2018b; Wei et al., 2019) are mostly based on uniform convergence and independent of the training algorithms. Daniely (2017) established a generalization bound for over-parameterized neural networks trained with one-pass SGD. However, they considered a setting where the training of hidden layers are neglectable and only the output layer training is effective.

In this paper, we aim to answer the following question:

Why gradient descent can learn an over-parameterized deep neural network that generalizes well?

Specifically, we consider learning deep fully connected ReLU networks with cross-entropy loss using over-parameterization and gradient descent.

The following theorem gives an informal version of our main results.

Under certain data distribution assumptions, for any $\epsilon>0$, if the number of nodes per each hidden layer is set to $\widetilde{\Omega}(\epsilon^{-14})$ and the sample size $n=\widetilde{\Omega}(\epsilon^{-4})$, then with high probability, gradient descent with properly chosen step size and random initialization method learns a deep ReLU network and achieves a population classification error at most $\epsilon$.

Here in Theorem 1.1 we use $\widetilde{O}(\cdot)$ and $\widetilde{\Omega}(\cdot)$ to hide some logarithmic terms in standard Big-O and Big-Omega notations. The result of Theorem 1.1 holds for ReLU networks with arbitrary constant number of layers, as long as the data distribution satisfies certain separation condition, which will be discussed in Section 4.2.

Our contributions. Our main contributions are as follows:

We provide a generalization error bound specifically suitable for wide neural networks of arbitrary depth. The bound enjoys better dependency in terms of the network width compared with existing generalization error bounds for deep neural networks (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018a; Golowich et al., 2018; Arora et al., 2018b; Li et al., 2018a; Wei et al., 2019). Moreover, we also provide an optimization result on the convergence of gradient descent for over-parameterized neural networks. Combining these two results together gives an algorithm dependent bound of expected error that is independent of the network width.

We investigate two types of data distribution assumptions, and show that under each of them, gradient descent can train an over-parameterized neural network to achieve $\epsilon$ expected error provided $\widetilde{O}(\epsilon^{-4})$ training examples. The data distribution assumptions we consider in this paper are standard and have been studied in recent literature. This demonstrates that our analysis can give meaningful generalization bounds even for very wide neural networks, and can provide insights on the practical success of over-parameterized neural networks.

2 Notation

We use the following standard asymptotic notations. For two sequences $\{a_{n}\}$ and $\{b_{n}\}$, we write $a_{n}=O(b_{n})$ if $a_{n}\leq C_{1}b_{n}$ for some absolute constant $C_{1}>0$, and $a_{n}=\Omega(b_{n})$ if $a_{n}\geq C_{2}b_{n}$ for some absolute constant $C_{2}>0$. In addition, we use $\widetilde{O}(\cdot)$ and $\widetilde{\Omega}(\cdot)$ to hide some logarithmic terms in Big-O and Big-Omega notations.

Additional Related Work

There is a huge body of literature towards building the foundations of deep learning, and we are not able to include every work in this paper. In this section, we briefly review and comment additional work that is most related to ours and was not discussed in Section 1.

Representation power of deep neural networks. A line of research has shown that deeper neural networks have higher expressive power (Telgarsky, 2015, 2016; Lu et al., 2017; Liang and Srikant, 2017; Yarotsky, 2017, 2018; Hanin, 2017; Hanin and Sellke, 2017) than shallow neural networks. This to certain extent explains the advantage of deep neural networks with over-parameterization. Lin and Jegelka (2018) proved that ResNet (He et al., 2016) with one hidden node per layer is a universal approximator to any Lebesgue integrable function.

Optimization landscape of neural networks. Many studies (Haeffele and Vidal, 2015; Kawaguchi, 2016; Freeman and Bruna, 2017; Hardt and Ma, 2017; Safran and Shamir, 2018; Xie et al., 2017; Nguyen and Hein, 2017; Soltanolkotabi et al., 2018; Zhou and Liang, 2017; Yun et al., 2018; Du and Lee, 2018; Venturi et al., 2018; Gao et al., 2019) investigated the optimization landscape of neural networks with different activation functions. However, these results only apply to one-hidden layer neural networks, or deep linear networks, or rely on some stringent assumptions on the data and/or activation functions. In fact, they do not hold for non-linear shallow neural networks (Yun et al., 2019) or three-layer linear neural networks (Kawaguchi, 2016). Furthmore, Yun et al. (2019) showed that small nonlinearities in activation functions create bad local minima in neural networks.

Implicit bias/regularization of GD and its variants. A bunch of papers (Gunasekar et al., 2017; Soudry et al., 2018; Ji and Telgarsky, 2019; Gunasekar et al., 2018a, b; Nacson et al., 2019; Li et al., 2018b) have studied implicit regularization/bias of GD, stochastic gradient descent (SGD) or mirror descent for matrix factorization, logistic regression, and deep linear networks. However, generalizing these results to deep non-linear neural networks turns out to be challenging and is still an open problem.

Connections between deep learning and kernel methods. Daniely (2017) uncovered the connection between deep neural networks with kernel methods and showed that SGD can learn a function that is comparable with the best function in the conjugate kernel space of the network. Jacot et al. (2018) showed that the evolution of a DNN during training can be described by a so-called neural tangent kernel, which makes it possible to study the training of DNNs in the functional space. Belkin et al. (2018); Liang and Rakhlin (2019) showed that good generalization performance of overfitted/interpolated classifiers is not only an intriguing feature for deep learning, but also for kernel methods.

Recovery guarantees for shallow neural networks. A series of work (Tian, 2017; Brutzkus and Globerson, 2017; Li and Yuan, 2017; Soltanolkotabi, 2017; Du et al., 2018a, b; Zhong et al., 2017; Zhang et al., 2019; Cao and Gu, 2019b) have attempted to study shallow one-hidden-layer neural networks with ground truth parameters, and proved recovery guarantees for gradient-based methods such as gradient descent (GD) and stochastic gradient descent (SGD). However, the assumption of the existence of ground truth parameters is not realistic and the analysis of the recovery guarantee can hardly be extended to deep neural networks. Moreover, many of these studies need strong assumptions on the input distribution such as Gaussian, sub-Gaussian or symmetric distributions.

Distributional view of over-parameterized networks. Mei et al. (2018); Chizat and Bach (2018); Sirignano and Spiliopoulos (2019); Rotskoff and Vanden-Eijnden (2018); Wei et al. (2019) took a distributional view of over-parametrized networks, used mean field analysis to show that the empirical distribution of the two-layer neural network parameters can be described as a Wasserstein gradient flow, and proved that Wasserstein gradient flow converges to global optimima under certain structural assumptions. However, their results are limited to two-layer infinitely wide neural networks. Very recently, Yang (2019) studied the scaling limit of wide multi-layer neural networks.

Problem Setup and Training Algorithm

Given $n$ training examples $(\bm{x}_{1},y_{1}),\ldots,(\bm{x}_{n},y_{n})$ drawn independently from $\mathcal{D}$, the training of the neural network can be formulated as an empirical risk minimization (ERM) problem as follows:

Here we introduce the details of the algorithm we use to solve the empirical risk minimization problem (3.1). The entire training algorithm is summarized in Algorithm 1.

Main Theory

In this section we present our main result. We first introduce several assumptions.

We have $M/m=O(1)$, where $M=\max\{m_{1},\ldots,m_{L}\}$, $m=\min\{m_{1},\ldots,m_{L}\}$.

Assumption 4.2 essentially says that the width of each layer in the deep neural network is in the same order, and the neural work architecture is balanced. Throughout this paper, we always assume Assumptions 4.1 and 4.2 hold. We therefore omit them in our theorem statements.

For the ease of exposition we introduce the following definitions.

For the collection of random parameters $\mathbf{W}^{(0)}=\{\mathbf{W}_{l}^{(0)}\}_{l=1}^{L}$ generated in Algorithm 1, we call

the $\tau$-neighborhood of $\mathbf{W}^{(0)}$.

The definition of $\mathcal{W}_{\tau}$ is motivated by the observation that in a small neighborhood of initialization, deep ReLU networks satisfy good scaling and landscape properties. It also provides a small subset of the entire hypothesis space and enables a sharper capacity bound based on Rademacher complexity for the generalization gap between empirical and generalization errors.

For a collection of parameter matrices $\mathbf{W}=\{\mathbf{W}_{l}\}_{l=1}^{L}$, we define its empirical surrogate error $\mathcal{E}_{S}(\mathbf{W})$ and population surrogate error $\mathcal{E}_{\mathcal{D}}(\mathbf{W})$ as follows:

In this section, we provide (i) a generalization bound for neural networks with parameters in a neighborhood of random initialization, (ii) a convergence guarantee of gradient descent for training over-parameteried neural networks. Combining these two results gives a bound on the expected error of neural networks trained by gradient descent.

For any $\delta>0$, there exist absolute constants $\overline{C},\overline{C}^{\prime},\underline{C}$ such that, if

then with probability at least $1-\delta$,

for all $\mathbf{W}\in\mathcal{W}_{\tau}$.

For neural networks initialized with He initialization (He et al., 2015), the generalization bound given by Theorem 4.5 has a better dependency in network width $m$ compared with existing uniform convergence based generalization error bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018a; Golowich et al., 2018; Arora et al., 2018b; Li et al., 2018a; Wei et al., 2019). For instance, $\mathbf{W}\in\mathcal{W}_{\tau}$ implies $\|\mathbf{W}_{l}^{\top}-\mathbf{W}_{l}^{(0)\top}\|_{2,1}\leq\sqrt{m}\tau$ and $\|\mathbf{W}_{l}\|_{2}=\widetilde{O}(1)$. Plugging these bounds into the generalization bound given by Bartlett et al. (2017)

or the bound given by Neyshabur et al. (2018a)

results in a generalization bound of the order $\widetilde{O}(m\tau/\sqrt{n})$. In comparison, when $\tau$ is small enough, our bound on the generalization gap is in the order of $\widetilde{O}(\tau\cdot\sqrt{m/n})$, which has a better dependency in $m$. Note that for over-parameterized neural networks, gradient descent indeed converges to a global minima that is very close to initialization, as we will show in Theorem 4.7. Therefore, while the previously mentioned uniform convergence based generalization bounds hold for more general settings and are more suitable when the weight matrices are not close enough to random initialization, our bound in Theorem 4.5 provides a sharper result that is specifically designed for the over-parameterized setting.

Theorem 4.5 in particular suggests that if gradient descent finds a parameter configuration with small surrogate error in $\mathcal{W}_{Rm^{-1/2}}$ for some $R$ independent of $m$, then the obtained neural network has a generalization bound decreasing in $m$. The following lemma shows that under a gradient lower bound assumption, gradient descent indeed converges to a global minima in $\mathcal{W}_{Rm^{-1/2}}$ with $R$ independent of $m$.

Suppose that the training loss function $L_{S}(\mathbf{W})$ satisfies the following inequality

for all $\mathbf{W}\in\mathcal{W}_{\tau}$, where $B$ is independent of $m$, and $\tau=\widetilde{O}(B^{-1}\epsilon^{-1}m^{-1/2})$. For any $\epsilon,\delta>0$, there exist absolute constants $\overline{C},\underline{C}$ and $m^{*}=\widetilde{O}(L^{12}B^{-4}\epsilon^{-2})\cdot\log(1/\delta)$ such that, if $m\geq m^{*}$, then with probability at least $1-\delta$, Algorithm 1 with step size $\eta=O(L^{-3}B^{2}m^{-1})$ generates $K=\widetilde{O}(L^{3}B^{-4}\epsilon^{-2})$ iterates $\mathbf{W}^{(1)},\ldots,\mathbf{W}^{(K)}$ that satisfy:

$\mathbf{W}^{(k)}\in\mathcal{W}_{\tau}$, $k\in[K]$.

There exists $k\in\{0,\ldots,K-1\}$ such that $\mathcal{E}_{S}(\mathbf{W}^{(k)})\leq\epsilon$.

Combining Theorems 4.5 and 4.7 directly gives the following corollary:

Suppose that the training loss function $L_{S}(\mathbf{W})$ satisfies inequality (4.1) for all $\mathbf{W}\in\mathcal{W}_{\tau}$, where $B$ is independent of $m$, and $\tau=\widetilde{O}(B^{-1}\epsilon^{-1}m^{-1/2})$. For any $\epsilon,\delta>0$, there exist absolute constants $\overline{C},\underline{C}$ and $m^{*}=\widetilde{O}(L^{12}B^{-4}\epsilon^{-2})\cdot\log(1/\delta)$ such that, if $m\geq m^{*}$, then with probability at least $1-\delta$, Algorithm 1 with step size $\eta=O(L^{-3}B^{2}m^{-1})$ finds a point $\mathbf{W}^{(k)}$ that satisfies

within $K=\widetilde{O}(L^{3}B^{-4}\epsilon^{-2})$ iterations.

2 Generalization Error Bounds under Specific Data Distribution Assumptions

In this section we introduce two specific data distributions that have been studied in the literature, and show that if one of them holds, then (4.1) holds with a $B$ independent of $m$ and $n$. Assumption 4.10 below is related to a random feature model studied in Rahimi and Recht (2009).

Denote by $p(\overline{\mathbf{u}})$ the density of standard Gaussian random vectors. Define

We assume that there exist an $f(\cdot)\in\mathcal{F}$ and a constant $\gamma>0$ such that $y_{i}\cdot f(\mathbf{x}_{i})\geq\gamma$ for all $i\in[n]$.

$\mathcal{F}$ defined in Assumption 4.10 corresponds to the random feature function class studied in Rahimi and Recht (2009) when the feature function is chosen to be ReLU. Assumption 4.10 essentially states that there exists a function $f$ in the function class $\mathcal{F}$ that can separate the data distribution $\mathcal{D}$ with a constant margin $\gamma$. According to the definition of $\mathcal{F}$, each value of $\overline{\mathbf{u}}$ can be considered as a node in an infinite-width one-hidden-layer ReLU network, and the corresponding product $c(\overline{\mathbf{u}})p(\overline{\mathbf{u}})$ can be considered as the second-layer weight. Therefore $\mathcal{F}$ contains infinite-width one-hidden-layer ReLU networks whose second-layer weights decay faster than $p(\overline{\mathbf{u}})$. Also note that Assumption 4.10 is strictly milder than linearly separable assumption.

The following corollary gives an expected error bound of neural networks trained by gradient descent under Assumption 4.10.

Under Assumption 4.10, for any $\epsilon,\delta>0$, there exist

such that, if $m\geq m^{*}(\epsilon,L,\gamma,\delta)$ and $n\geq n^{*}(\epsilon,L,\gamma,\delta)$, then with probability at least $1-\delta$, Algorithm 1 with step size $\eta=O(4^{-L}L^{-3}\gamma^{2}m^{-1})$ finds a point $\mathbf{W}^{(k)}$ that satisfies

We now introduce another data distribution assumption which has been made in Daniely (2017).

The conjugate kernel of fully connected neural networks is defined recursively as

We assume that there exists a function $f$ in the reproducing kernel Hilbert space (RKHS) $\mathcal{H}$ induced by the conjugate kernel function $\kappa^{(L-1)}(\cdot,\cdot)$ with $\|f\|_{\mathcal{H}}\leq 1$ such that $y_{i}\cdot f(\mathbf{x}_{i})\geq\gamma>0$.

Under Assumption 4.12, we have the following result.

Under Assumption 4.12, for any $\epsilon,\delta>0$, there exist

such that, if $m\geq m^{*}(\epsilon,L,\gamma,\delta)$ and $n\geq n^{*}(\epsilon,L,\gamma,\delta)$, then with probability at least $1-\delta$, Algorithm 1 with step size $\eta=O(L^{-3}\gamma^{2}m^{-1})$ finds a point $\mathbf{W}^{(k)}$ that satisfies

Corollary 4.13 shows that under Assumption 4.12, a neural network trained by gradient descent can achieve $\epsilon$-expected error given $\widetilde{O}(\epsilon^{-4})$ training examples. We remark that although Daniely (2017) studied the same assumption, our result is not a re-derivation of the results given by Daniely (2017), because while they considered one-pass SGD and square loss, while we consider GD and cross-entropy loss. More importantly, Assumption 4.12 is just one specific setting our result can cover, and therefore Corollary 4.13 demonstrates the power of our general theory.

A follow-up work Cao and Gu (2019a) studied the generalization performance of over-parameterized neural networks trained with one-pass SGD, and relate the generalization bound to the neural tangent kernel function studied in recent work (Jacot et al., 2018). We remark that their generalization bound is based on an online-to-batch conversion argument, which cannot be applied to the standard gradient descent algorithm we study in this paper. Therefore our result and their result are not directly comparible.

Proof of the Main Theory

In this section we provide the proofs of the main results given in Section 4. The omitted proof can be found in the supplementary material.

Here we provide the detailed proof of Theorem 4.5. We first present the lemma below, which gives an upper bound on the gradients of $L_{S}(\mathbf{W})$, and relates the gradients with the empirical surrogate error $\mathcal{E}_{S}(\mathbf{W})$.

for some large enough absolute constant $\overline{C}$ and small enough absolute constant $\underline{C}$, then with probability at least $1-\delta$, for all $\mathbf{W}\in\mathcal{W}_{\tau}$ and $l\in[L]$,

Lemma 5.2 below reveals the fact that near initialization, the neural network function is almost linear in terms of its weight parameters. As a consequence, the empirical loss function $L_{S}(\mathbf{W})$ is almost smooth in a small neighborhood around $\mathbf{W}^{(0)}$.

for some large enough absolute constant $\overline{C}$ and small enough absolute constant $\underline{C}$, then there exists an absolute constant $C$ such that with probability at least $1-\delta$, for all $\widetilde{\mathbf{W}},\widehat{\mathbf{W}}\in\mathcal{W}_{\tau}$,

Let $\mathcal{F}_{\tau}=\{f_{\mathbf{W}}(\mathbf{x}):\mathbf{W}\in\mathcal{W}_{\tau}\}$. We consider the empirical Rademacher complexity (Bartlett and Mendelson, 2002; Mohri et al., 2018; Shalev-Shwartz and Ben-David, 2014) of $\mathcal{F}_{\tau}$ defined as follows

where $C_{1}$ is an absolute constant. We now bound the term $\widehat{\mathfrak{R}}_{n}[\mathcal{F}_{\tau}]$. By definition, we have

Now by Lemma 5.1, we have \big{\|}\nabla_{\mathbf{W}_{l}}f_{\mathbf{W}^{(0)}}(\bm{x}_{i})\big{\|}_{F}\leq C_{3}\sqrt{m} for all $l\in[L]$, where $C_{3}$ is an absolute constant. Therefore $I_{2}\leq C_{3}L\tau\cdot\sqrt{m/n}$. Plugging in the bounds of $I_{1}$ and $I_{2}$ into (5.1) and applying Markov’s inequality

2 Proof of Theorem 4.7

The following lemma is given by Zou et al. (2018), which gives a bound on the neural network output at initialization.

For any $\delta>0$, with probability at least $1-\delta$, $|f_{\mathbf{W}^{(0)}}(\bm{x}_{i})|\leq C\sqrt{\log(n/\delta)}$ for all $i\in[n]$, where $C$ is an absolute constant.

Set $\tau=\widetilde{O}(B^{-1}\epsilon^{-1}m^{-1/2})$. Then there exist $\eta=O(L^{-3}B^{2}m^{-1})$, $K=\widetilde{O}(L^{3}B^{-4}\epsilon^{-2})$ and $m^{*}=\widetilde{O}(L^{12}B^{-4}\epsilon^{-2})\cdot\log(1/\delta)$ such that when $m\geq m^{*}$, all assumptions of Lemmas 5.1, 5.2 hold, and

for some small enough absolute constant $\nu$. We now prove by induction that $\mathbf{W}^{(k)}\in\mathcal{W}(\mathbf{W}^{(0)},\tau/2)$, $k\in\{0\}\cup[K]$. By definition clearly we have $\mathbf{W}^{(0)}\in\mathcal{W}(\mathbf{W}^{(0)},\tau/2)$. Suppose that $\mathbf{W}^{(k)}\in\mathcal{W}(\mathbf{W}^{(0)},\tau/2)$ for all $k=0,\ldots,t$. Then for all $l\in[L]$ we have

where the last inequality follows by Lemma 5.1 and the definition of $\tau$ and $\eta$ (note that a comparison between (4.1) and Lemma 5.1 implies that $B=O(1)$). Therefore $\mathbf{W}^{(t+1)}\in\mathcal{W}_{\tau}$. Plugging in the gradient upper bound given by Lemma 5.1 and assumption (4.1) into the result of Lemma 5.2, we obtain

for all $k=0,\ldots,t$, where $C_{1},C_{2}$ are absolute constants. Plugging in the bounds (5.3) and (5.4), we have

for all $k=0,\ldots,t$. Combining (5.6) with Lemma 5.1 gives

3 Proof of Corollary 4.11

In this section we give the proof of Corollary 4.11. The following lemma verifies that under Assumption 4.10, (4.1) indeed holds with $B$ independent in both $m$ and $n$.

for some large enough absolute constant $\overline{C}$ and small enough absolute constant $\underline{C}$, then with probability at least $1-\delta$, there exists an absolute constant $C$ such that

for all $\mathbf{W}\in\mathcal{W}_{\tau}$.

Corollary 4.11 directly follows by plugging in $B=O(2^{-L}\gamma)$ given by Lemma 5.4 and the assumptions $m\geq\widetilde{O}(L^{24}2^{8L}\gamma^{-8})\cdot\epsilon^{-14}$, $n\geq\widetilde{O}(L^{4}4^{L}\gamma^{-2})\cdot\epsilon^{-4}$ into Corollary 4.9. ∎

4 Proof of Corollary 4.13

In this section we give the proof of Corollary 4.13. Similar to the proof of Corollary 4.11, we mainly need to derive a gradient lower bound of the form (4.1). The result is given in the following lemma, which gives a similar result in part of the proof of Claim 1 in Daniely (2017).

for some large enough absolute constant $\overline{C}$ and small enough absolute constant $\underline{C}$, then with probability at least $1-\delta$, there exists an absolute constant $C$ such that

for all $\mathbf{W}\in\mathcal{W}_{\tau}$.

Corollary 4.13 directly follows by plugging in $B=O(\gamma)$ given by Lemma 5.5 and the assumptions $m\geq\widetilde{O}(L^{24}\gamma^{-8})\cdot\epsilon^{-14}$, $n\geq\widetilde{O}(L^{4}\gamma^{-2})\cdot\epsilon^{-4}$ into Corollary 4.9. ∎

Conclusions and Future Work

In this paper, we provided a generalization guarantee of gradient descent for training deep ReLU networks under over-parameterization, which hold under mild data distribution assumptions. Although we only focus on gradient descent and cross-entropy loss for binary classification, our results can be extended to stochastic gradient descent, other loss functions and multi-class classification. In addition, we will derive generalization bounds for deep learning based on the “small-ball” assumption proposed in Mendelson (2014). Another interesting direction is to investigate the generalization of gradient descent using stability-based analysis (Hardt et al., 2016).

Acknowledgements

This research was sponsored in part by the National Science Foundation CAREER Award IIS-1906169, IIS-1903202, and Salesforce Deep Learning Research Award. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

Appendix A Matrix Product Representation for Deep ReLU Networks

where we use the following matrix product notation:

Since this paper studies the generalization performance of neural network learning, we frequently need to study the training examples $(\bm{x}_{1},y_{1}),\ldots,(\bm{x}_{n},y_{n})$ as well as a test example $(\mathbf{x},y)\sim\mathcal{D}$. To distinguish the $i$-th example in the training sample and the $l$-th layer output of the test input $\mathbf{x}$, we use the following notations:

For $i=1,\ldots,n$, $l=1,\ldots,L$, we use $\bm{x}_{i}$ to denote the $i$-th training input, and $\bm{x}_{l,i}$ the output of the $l$-th layer with input $\bm{x}_{i}$.

For $l=1,\ldots,L$, we denote by $\mathbf{x}_{l}$ the output of the $l$-th layer with test input $\mathbf{x}$.

Appendix B Proof of Main Results in Section 5

In this section we provide proofs of theorems and lemmas given in Section 5.

Before we prove Lemma 5.1, we need the following technical lemma, which is a simplified version of Theorem 5.3 given in Zou et al. (2019). It characterizes several basic scaling properties of deep ReLU networks around random initialization. Here we use the following extension of the notations introduced in Section A: we denote by $\widetilde{\bm{x}}_{l,i}$ and $\widehat{\bm{x}}_{l,i}$ the hidden outputs of the ReLU network with input $\bm{x}_{i}$ and weights $\widetilde{\mathbf{W}}$, $\widehat{\mathbf{W}}$ respectively. Similar notations are also used for the binary diagonal matrices $\widetilde{\bm{\Sigma}}_{l}(\bm{x}_{i})$ and $\widehat{\bm{\Sigma}}_{l}(\bm{x}_{i})$.

There exist absolute constants $\overline{C},\underline{C}$ such that, for any $\delta>0$, if

then with probability at least $1-\delta$, the following results hold uniformly for all $\widehat{\mathbf{W}},\widetilde{\mathbf{W}}\in\mathcal{W}_{\tau}$:

$\|\widetilde{\mathbf{W}}_{l}\|_{2},\|\widetilde{\bm{x}}_{l,i}\|_{2}\leq\overline{C}$ for all $l\in[L]$ and $i\in[n]$.

\big{\|}\prod_{r=l}^{L}\widetilde{\bm{\Sigma}}_{r}(\bm{x}_{i})\widetilde{\mathbf{W}}_{r}^{\top}\big{\|}_{2}\leq\overline{C}L for all $l\in[L]$ and $i\in[n]$.

$\|\widehat{\bm{x}}_{l,i}-\widetilde{\bm{x}}_{l,i}\|_{2}\leq\overline{C}\sqrt{L}\cdot\sum_{r=1}^{l}\|\widehat{\mathbf{W}}_{r}-\widetilde{\mathbf{W}}_{r}\|_{2}$ for all $l\in[L]$ and $i\in[n]$.

$\|\widehat{\bm{\Sigma}}_{l}(\bm{x}_{i})-\widetilde{\bm{\Sigma}}_{l}(\bm{x}_{i})\|_{0}\leq\overline{C}L^{4/3}\tau^{2/3}m$ for all $l\in[L]$ and $i\in[n]$.

\big{\|}\mathbf{v}^{\top}\big{[}\prod_{r=l}^{L}\widetilde{\bm{\Sigma}}_{r}(\bm{x}_{i})\widetilde{\mathbf{W}}_{r}^{\top}\big{]}\big{\|}_{2}\leq\overline{C}\sqrt{m} for all $l\in[L]$ and $i\in[n]$.

We are now ready to give the proof of Lemma 5.1.

Denote $\widetilde{y}_{i}=f_{\mathbf{W}}(\bm{x}_{i})$. Then by definition, we have

for all $i\in[n]$ and $l\in[L]$, where $C_{1}$ is an absolute constant. By the triangle inequality, we have

for all $l\in[L]$, where $C_{2}$ is an absolute constant. This completes the proof. ∎

B.2 Proof of Lemma 5.2

For $i\in[n]$, denote by $\widehat{y}_{i}$, $\widetilde{y}_{i}$ the outputs of the network with input $\bm{x}_{i}$ and parameter matrices $\widehat{\mathbf{W}}$, $\widetilde{\mathbf{W}}$ respectively. Then we have

and therefore $f_{\widehat{\mathbf{W}}}(\bm{x}_{i})-f_{\widetilde{\mathbf{W}}}(\bm{x}_{i})=I_{1}+I_{2}+I_{3}$, where

For $I_{1}$, note that by Lemma B.1, for any $l=1,\ldots,L$ we have

where the first inequality follows by checking the non-zero diagonal entries of $\widehat{\bm{\Sigma}}_{l}(\bm{x}_{i})-\widetilde{\bm{\Sigma}}_{l}(\bm{x}_{i})$, and $C_{1},C_{2}$ are absolute constants. Therefore by Lemma B.1 we have

where $C_{3}$ is an absolute constant. For $I_{2}$, by Lemma B.1 we have

where $C_{4}$ is an absolute constant. For $I_{3}$, we have

Denote $\Delta_{i}=\widehat{y}_{i}-\widetilde{y}_{i}$. Then

Plugging in the bound of \big{|}f_{\widehat{\mathbf{W}}}(\bm{x}_{i})-F_{\widetilde{\mathbf{W}},\widehat{\mathbf{W}}}(\bm{x}_{i})\big{|} gives

where $C_{5}$ is an absolute constant. Moreover, by Lemma B.1, clearly we have

B.3 Proof of Lemma 5.4

Here we give the proof of Lemma 5.4. Again, we extend the notations introduced in Section A by denoting $\bm{x}_{l,i}^{(k)}$ and $\bm{\Sigma}_{l}^{(k)}(\bm{x}_{i})$ the network hidden layer outputs and binary diagonal matrices with input $\bm{x}_{i}$ and weights $\mathbf{W}^{(k)}$ respectively.

We first introduce the following two lemmas, which are based on Assumption 4.10. Lemma B.2 below shows that under our data distribution assumptions, the hidden layer outputs of the deep ReLU network is linearly separable with high probability. Lemma B.3 takes advantage of this linearly separable property and further gives a lower bound result with respect to the initialized weights, which plays an essential role in the proof of our gradient lower bound.

For any $\delta>0$, if $m\geq C\cdot 4^{L}\cdot L^{2}\gamma^{-2}\log(nL/\delta)$ for some large enough absolute constant $C$, then with probability at least $1-\delta$, there exist $\bm{\alpha}_{1}\in S^{m_{1}-1},\ldots,\bm{\alpha}_{L}\in S^{m_{L}-1}$ such that $y_{i}\cdot\langle\bm{\alpha}_{l},\bm{x}_{l,i}^{(0)}\rangle\geq 2^{-(l+1)}\gamma$ for all $i\in[n]$ and $l\in[L]$.

For any $\delta>0$, under the same assumptions as Lemma B.2, with probability at least $1-\delta$, the inequality

by (iii) and (iv) in Lemma B.1, with probability at least $1-\delta/2$ we have

where $C_{1}$ is an absolute constant. Therefore by the assumption that $\tau\leq C8^{-L}\cdot L^{-2}\gamma^{3}$ for some small enough absolute constant $C$, we have

B.4 Proof of Lemma 5.5

where $C$ is an absolute constant. This finishes the proof. ∎

Appendix C Proof of Lemmas in Appendix B

By Assumption 4.10, there exists $c(\overline{\mathbf{u}})$ with $\|c(\cdot)\|_{\infty}\leq 1$ such that

Since $\|c(\cdot)\|_{\infty}\leq 1$, we have $\|\widetilde{\bm{\alpha}}_{1}\|_{2}^{2}=m_{1}^{-1}\cdot\sum_{j=1}^{m_{1}}c^{2}(\sqrt{m_{1}/2}\mathbf{w}_{j})\leq 1$. For any $i\in[n]$, we have

for some absolute constant $C_{1}$. Therefore by Hoeffding inequality and union bound, with probability at least $1-\delta/4$ we have

for all $i\in[n]$, where $C_{2}$ is an absolute constant. Set $\bm{\alpha}_{1}=\widetilde{\bm{\alpha}}_{1}/\|\widetilde{\bm{\alpha}}_{1}\|_{2}$. Then by $\|\widetilde{\bm{\alpha}}_{1}\|_{2}\leq 1$, we have

for all $l=2,\ldots,L$, where $C_{3}$ is an absolute constant. Therefore since $\|\widetilde{\bm{\alpha}}_{1}\|_{2}=1$, we have $\|\widetilde{\bm{\alpha}}_{l}\|_{2}\leq 2^{l-1}$ for all $l=2,\ldots,L$. Moreover, for any $i\in[n]$ and $l=2,\ldots,L$, by definition, we have $\mathbf{w}_{l,j}^{(0)\top}\stackrel{{\scriptstyle d}}{{=}}-\mathbf{w}_{l,j}^{(0)\top}$, $j\in[m_{l}]$, and therefore

where $C_{4},C_{5}$ are absolute constants, by Bernstein’s inequality and a union bound, with probability at least $1-\delta/2$ we have

for all $i\in[n]$ and $l=2,\ldots,L$, where $C_{6}$ is an absolute constant. Therefore we have

for all $i\in[n]$ and $l=2,\ldots,L$. Setting $\bm{\alpha}_{l}=\widetilde{\bm{\alpha}}_{l}/\|\widetilde{\bm{\alpha}}_{l}\|_{2}$, we obtain

C.2 Proof of Lemma B.3

for all $i\in[n]$, where $C_{1}$ is an absolute constant. Hence we have

where the first inequality follows by Jensen’s inequality, the second and third inequality follows by Lemma B.2, and the last inequality is by (C.1). This completes the proof. ∎

References