Mean Field Analysis of Neural Networks: A Law of Large Numbers

Introduction

Neural networks have achieved immense practical success over the past decade. Neural networks are nonlinear statistical models whose parameters are estimated from data using stochastic gradient descent. They have been employed as critical components of many important technologies in a variety of industries. This practical success has sparked significant interest in their mathematical analysis. Currently, there is limited mathematical understanding of neural networks. This paper analyzes the asymptotic behavior of neural networks, rigorously proving that the empirical distribution of their parameters converges to the solution of a nonlinear partial differential equation (PDE).

Neural network models have revolutionized fields such as image, text, and speech recognition. They are actively used in a variety of applications. In image recognition, neural networks are able to accurately identify and recognize objects in images using only the raw pixels. Neural networks are used for image recognition in applications such as self-driving cars, image searches on search engines such as Google, and facial recognition for security systems (see , , , and ). In speech recognition, neural networks are used to develop computer systems that automatically understand human speech (see , , , and ). Applications include voice control of certain systems in vehicles, transcription (automatically converting human speech to written text), interactive voice response for customer service, and spoken commands for smartphones. In text recognition, neural networks are used to automatically translate text from one language (e.g., English) to another language (e.g., Italian); see and . They have also been used for automatically generating summaries of long documents; see and .

In addition, there is growing interest in applying neural networks to engineering, robotics, medicine, and finance. Neural networks are being used in reduced-form models of the Navier-Stokes equation in turbulent conditions (see and ). , , and describe applications in robotics. Neural networks have been used to identify cancer and to model protein folding . In finance, neural networks have been used to model loan default and prepayment risk and to model high frequency financial data . Neural networks have also been used to solve high-dimensional PDEs in financial applications .

Due to the impact that neural networks have had on practical applications, there is a significant interest in better understanding their mathematical properties. However, the existing literature is relatively limited, with only a few recent papers such as , , and . There also exist classical results regarding the approximation power of neural networks , , and .

Our result characterizes neural networks with a single hidden layer in the asymptotic regime of large network sizes and large numbers of stochastic gradient descent iterations. We rigorously prove that the empirical distribution of the neural network parameters will weakly converge to a distribution. This distribution satisfies a nonlinear partial differential equation. The proof relies upon weak convergence analysis for interacting particle systems. The result can be considered a “law of large numbers” for neural networks when both the network size and the number of stochastic gradient descent steps grow to infinity.

Recently, rigorously established a weak convergence result for a class of machine learning algorithms. Weak convergence analysis has been widely used in other fields (for example, see , , , , , , and for a non-exhaustive list). In fact, mean field analysis has been actively used for many years to study biological neural networks and physical systems of interacting particles; see for example , , , , and the references therein.

Upon completion of this work, we became aware of the very recent work of where a related PDE limit result for neural networks is derived; see also the recent work . Our convergence analysis, setup, and assumptions are different. In , it is assumed that the gradient of the neural network is a priori globally Lipschitz and bounded and under this assumption a similar PDE result as well as certain rates of convergence are established. In our work, we do not assume that the gradient of the neural network is a priori globally Lipschitz or bounded. Often, neural network models (and their gradients) are not globally Lipschitz and not bounded. In this paper, we assume that the data come from a distribution that has its first moments bounded and that the initialization is done according to distributions with certain moment bounds. Based on this assumption, we rigorously prove relative compactness of the pre-limit measure valued process, identification of its limit, and uniqueness of the limit point in the appropriate space. Our method of proof leverages on weak convergence analysis in an appropriate Skorokhod space for measure-valued processes (similar to the approaches in and ). In particular, the relative compactness and uniqueness proof addresses the challenge of neural networks not being a priori globally Lipschitz nor globally bounded using the structure of the stochastic gradient descent algorithm.

where the data $(Y,X)$ is assumed to have a joint distribution $\pi(dx,dy)$. We shall write $\mathcal{X},\mathcal{Y}$ for the state spaces of $X$ and $Y$, respectively. The parameters $\theta=(c^{1},\ldots,c^{N},w^{1},\ldots,w^{N})$ are estimated using stochastic gradient descent:

where $\alpha$ is the learning rate and $(x_{k},y_{k})\sim\pi(dx,dy)$. Stochastic gradient descent minimizes (1.2) using a sequence of noisy (but unbiased) gradient descent steps $\nabla_{\theta}[(y_{k}-g_{\theta_{k}}^{N}(x_{k}))^{2}]$. Note that typically $\nabla_{\theta}[(y-g_{\theta}^{N}(x))^{2}]$ is not a priori globally Lipschitz nor globally bounded as a function of $\theta$. Stochastic gradient descent typically converges more rapidly than gradient descent for large datasets. For this reason, stochastic gradient descent is widely used in machine learning.

The neural network’s output can be re-written in terms of the empirical measure:

$\left\langle f,h\right\rangle$ denotes the inner product of $f$ and $h$. For example, $\left\langle c\sigma(w\cdot x),\nu^{N}_{k}\right\rangle=\int c\sigma(w\cdot x)\nu^{N}_{k}(dc,dw)$.

Our main results are stated below. Theorem 1.2 (and the associated Remark 1.3) is a law of large numbers describing the distribution of the trained parameters when $N$ is large. Theorem 1.6 describes the behavior of individual parameters when $N$ is large. Theorem 1.6 is a “propagation of chaos” result. Section 1.1 presents several insights provided by these asymptotic results.

where $\nabla f=(\partial_{c}f,\nabla_{w}f)$.

Since weak convergence to a constant implies convergence in probability, Theorem 1.2 leads to the stronger result of convergence in probability

for every $\delta>0$ and where $d_{E}$ is the metric for $D_{E}([0,T])$.

Assume Assumption 1.1. Suppose that $\bar{\mu}_{0}$ admits a density $p_{0}(c,w)$ and there exists a unique solution to the nonlinear partial differential equation

such that $p(t,c,w)$ vanishes as $|c|,\parallel w\parallel\rightarrow\infty$. Then, we have that the solution to the measure evolution equation (1.7) is such that

Notice that by setting $\theta=(c,w)$, the partial differential equation for $p(t,\theta)$ in Corollary 1.4 can be written as

where $\textrm{div}_{\theta}$ is the divergence operator with respect to the variable $\theta$ and $v(\theta,p(t,\cdot))$ is defined as

In addition, Theorem 1.2 and Corollary 1.4 imply that the objective function $L^{N}(\theta)$ from (1.2) satisfies

In Theorem 1.6 we prove that the neural network has the “propagation of chaos” property.

The law of large numbers (1.7) suggests several interesting characteristics of trained neural networks (at least in the setting studied in this paper).

As $N\rightarrow\infty$, the neural network converges (in probability) to a deterministic model. This is despite the fact that the neural network is randomly initialized and it is trained on a random sequence of data samples via stochastic gradient descent.

The learning rate $\alpha$ was assumed to be constant and to not decay with time. However, notice that the hidden layer has been normalized by $1/N$ and it is this normalization by $1/N$ in the hidden layer that replaces the role of the learning rate decay, enabling convergence.

The propagation of chaos result (1.10) indicates that, as $N\rightarrow\infty$, the dynamics of the weights $(c^{i}_{k},w^{i}_{k})$ will become independent of the dynamics of the weights $(c^{j}_{k},w^{j}_{k})$ for any $i\neq j$. Note that the dynamics $(c^{i}_{k},w^{i}_{k})$ are still random due to the random initialization. However, the dynamics of the $i$-th set of weights will be uncorrelated with the dynamics of the $j$-th set of weights in the limit as $N\rightarrow\infty$.

In order to illustrate some aspects of the theoretical results of this paper, we performed the following numerical study.

Figure 1 displays the convergence of the distribution of the parameters in a trained neural network as the number of hidden units $N\rightarrow\infty$. The neural network has a single hidden layer followed by a softmax function. Figure 1 reports the distribution of the parameters connecting the hidden layer to the softmax function. The distributions are presented as histograms. The neural network is trained on the MNIST dataset, which is a standard image dataset in machine learning . The dataset includes $60,000$ images of handwritten numbers $\{0,1,2,\ldots,9\}$. The neural network is trained to identify the handwritten numbers using only the image pixels as an input (i.e., it learns to recognize images as a human would). In the MNIST dataset, each image has $784$ pixels. A pixel takes values in $\{0,1,\ldots,255\}$.The pixel values are normalized to $$ for the purposes of training the neural network. Neural networks can achieve 98-99% out-of-sample accuracy on the MNIST dataset.

Figure 1 shows that the distribution of parameters converges to a fixed distribution as $N\rightarrow\infty$. This can be seen by the fact that the distributions for $N=10,000$, $N=100,000$, and $N=250,000$ are nearly identical. A priori it is unclear if the distribution of neural network parameters should converge as $N\rightarrow\infty$. Our theory and numerical results confirm that this is indeed the case. Indeed, as $N$ gets large, we see that the empirical distribution of the parameters connecting the hidden layer to the softmax function converges to a specific deterministic distribution.

2 Overview of the Proof

Relative Compactness

For each $\eta>0$, there is a compact subset $\mathcal{K}$ of E such that

Given now that $\lim_{L\to\infty}\sum_{j=1}^{\infty}\frac{C}{(L+j)^{3/2}}=0$, the proof of the lemma is concluded. ∎

We start by noticing that a Taylor expansion gives for $0\leq s\leq t\leq T$

for points $\bar{c}^{i},\bar{w}^{i}$ in the segments connecting $c^{i}_{\left\lfloor Ns\right\rfloor}$ with $c^{i}_{\left\lfloor Nt\right\rfloor}$ and $w^{i}_{\left\lfloor Ns\right\rfloor}$ with $w^{i}_{\left\lfloor Nt\right\rfloor}$, respectively.

Let’s now establish a bound on $|c^{i}_{\left\lfloor Nt\right\rfloor}-c^{i}_{\left\lfloor Ns\right\rfloor}|$ for $s<t\leq T$. Let $0<p<1$.

where Assumption 1.1 was used. Let’s now establish a bound on $\parallel w^{i}_{\left\lfloor Nt\right\rfloor}-w^{i}_{\left\lfloor Ns\right\rfloor}\parallel$ for $s<t\leq T$. Making use of the uniform bounds established in Lemma 2.1, we obtain similarly to the previous bound

Now, we return to equation (2.1). By Lemma 2.1, the quantities $(\bar{c}^{i}_{\left\lfloor Nt\right\rfloor},\bar{w}^{i}_{\left\lfloor Nt\right\rfloor})$ are bounded in expectation for $0<s<t\leq T$. Therefore, for $0<s<t\leq T$,

where $C<\infty$ is some unimportant constant. Then, the statement of the Lemma follows. ∎

Given Lemmas 2.2 and 2.3, Theorem 8.6 of Chapter 3 of , gives the statement of the lemma. (See also Remark 8.7 B of Chapter 3 of regarding replacing $\sup_{N}$ with $\lim_{N}$ in the regularity condition B of Theorem 8.6.) ∎

We conclude this section with the proof of the a-priori bound of Lemma 2.1.

We start by establishing some useful a-priori bounds on $c_{k}^{i}$ and $w_{k}^{i}$. The unimportant finite constant $C<\infty$ may change from line to line. We first observe that

where to derive the last line we used the definition of $g_{\theta_{k}}^{N}(x)$ via (1.1) and the uniform boundedness assumption on $\sigma$. Then, we subsequently obtain that

Let us now define $m_{k}^{N}=\frac{1}{N}\sum_{i=1}^{N}|c_{k}^{i}|$ and $b_{k}^{N}=\frac{1}{N}\sum_{i=1}^{N}|c_{0}^{i}|+\sum_{j=1}^{k}\frac{\alpha C|y_{j-1}|}{N}$. Then we have

which by the discrete Gronwall lemma gives the bound

for a possibly different constant that may depend on $T$, where the relation $k/N\leq T$ was used in the last step. Going back now to the bound for $c_{k}^{i}$ we obtain

Raising this to power $1\leq p\leq 4$, we have for a constant $C_{p}$ that may depend on $p$

Let us bound now each of the terms on the right hand side of the last display. We have for some constant $C_{p}<\infty$ that may change from line to line

for a constant $C<\infty$ that may change from line to line. Taking now expectation, using Assumption 1.1, the a-priori bound (2.2) and the fact that $k/N\leq T$ we obtain

Identification of the Limit

for points $\bar{c}^{i}_{k},\bar{w}^{i}_{k}$ in the segments connecting $c^{i}_{k+1}$ with $c^{i}_{k}$ and $w^{i}_{k+1}$ with $w^{i}_{k}$, respectively. Notice now that the uniform bounds of Lemma 2.1 and the relation (1.3) imply that as $N$ gets large

The term $O_{p}\left(N^{-2}\right)$ Recall that when we write $Z=O_{p}(b)$ we mean that $Z/b$ is stochastically bounded. is a result of $f\in C^{2}_{b}$, the bounds from Lemma 2.1 as well as the moment bounds on $(x_{k},y_{k})$ from Assumption 1.1. We next define the drift and martingale components:

Combining the different terms together, we then obtain

Next, we define the scaled versions of $D^{1,N},D^{2,N},M^{1,N}$ and $M^{2,N}$:

The scaled empirical measure satisfies, as $N$ grows,

In fact as we show below $M^{1,N}(t)$ and $M^{2,N}(t)$ converge to in $L^{2}$ as $N\rightarrow\infty$.

Let $\mathcal{F}_{k}^{N}$ be the $\sigma-$algebra generated by $(c^{i}_{0},w^{i}_{0})_{i=1}^{N}$ and $(x_{j},y_{j})_{j=0}^{k-1}$. If $j>k$, then

Let $\pi^{N}$ be the probability measure of a convergent subsequence of $\left(\mu^{N}\right)_{0\leq t\leq T}$. Each $\pi^{N}$ takes values in the set of probability measures \mathcal{M}\big{(}D_{E}([0,T])\big{)}. Relative compactness, proven in Section 2, implies that there is a subsequence $\pi^{N_{k}}$ which weakly converges. We must prove that any limit point $\pi$ of a convergent subsequence $\pi^{N_{k}}$ will satisfy the evolution equation (1.7).

Let $\pi^{N_{k}}$ be a convergent subsequence with a limit point $\pi$. Then $\pi$ is a Dirac measure concentrated on $\bar{\mu}\in D_{E}([0,T])$ and $\bar{\mu}$ satisfies the measure evolution equation (1.7).

Then, by the proof of Lemma 3.1, we obtain for large $N$

Since $F(\cdot)$ is continuous and $F(\mu^{N})$ is uniformly bounded (due to the uniform boundedness results of Section 2),

It remains to prove that the evolution equation (1.7) has a unique solution. This is the content of Section 4.

Uniqueness

We prove uniqueness of a solution to the evolution equation (1.7). We will set up a Picard type of iteration and prove that it has a unique fixed point through a contraction mapping. We start by noticing that we can write

We remark here that a solution to (4.1), $\bar{\mu}_{\cdot}$, is associated to the nonlinear random process $Z_{t}$ (see for example ) satisfying the random ordinary differential equation (ODE)

This ODE is random due to the random initial data.

It is clear that if $(\mu_{t})_{t\in[0,T]}$ is a fixed point of $H$, then $\text{Law}(Z_{t})=H_{t}(\mu_{\cdot})$ is a solution to (4.1). Conversely, if $(Z_{t})_{t\in[0,T]}$ is a solution to (4.2) then its law will be a fixed point of $H$, implying that $\text{Law}(Z_{t})=H_{t}(\mu)$. In addition, if $\mu$ is a weak measure valued solution to (4.1), then it must be a fixed point of $H$ and thus satisfy (4.2), proving our result.

Lemma 4.1 shows that there is regularity in time and it also provides us with some useful a-priori uniform bounds.

Let $1\leq p\leq 4$ and $T<\infty$ be given. Then, there are constants $C_{1},C_{2},C_{3}<\infty$, depending on $p$, such that

and for every $0\leq s\leq t\leq T$ we have that

Let’s examine $c_{t}$ first and establish a bound on its growth. The constant $C$ may change from line to line and it may also depend upon the final time $T$ and on $p$.

Therefore, by Gronwall’s inequality, there exists a constant $C_{2}<\infty$ such that

for $0\leq s\leq T$. Therefore, returning to (4.4) and recalling Assumption 1.1 we get that uniformly in $t\in[0,T]$, there exist constants $C_{1},C_{2}<\infty$ such that

and the claimed bound follows by taking supremum over all $t\in[0,T]$, expectation and using the previously derived uniform bound for $c_{t}$. Let us now prove the second statement of the lemma. Similarly to the calculations above and using the uniform moment bounds on $c_{t}$ and $w_{t}$ together with Assumption 1.1, we have

For $m,m^{\prime}\in M_{T}$ and $p\geq 1$ define the metric

where $P(m,m^{\prime})$ is the set of probability measures on $C_{T}\times C_{T}$ such that the marginal distributions are $m$ and $m^{\prime}$, respectively.

Now we show existence and uniqueness of a fixed point $\text{Law}(c_{t},w_{t})$ for the mapping $H$, as defined via (4.5). If a solution to (4.2) exists, then it must be a fixed point of $H$ (defined via equation (4.5)). This is an immediate consequence of Lemma 4.1. Therefore, if $H$ has a unique solution, there can be at most one solution to (4.2). If (4.2) has at most one solution, (4.1) has at most one solution. Therefore, if $H$ has a unique fixed point, this proves uniqueness for (4.1).

Due to Lemma 4.1 we need only to consider the space of measures $M_{T}$ that have bounded moments up to order $p=4$. By known results, see for example , the space of measures with $p=4$ finite moments endowed with the $D_{T,4}$ metric is complete and separable. Due to closedness, the space of measures with bounded moments of order four is a complete and separable metric space when endowed with the Wasserstein metric $D_{T,4}$. Therefore, in the arguments below we work with the space of measures that have bounded the first four moments and we consider the metric $D_{T,4}$. We will show that there is a unique fixed point by proving a contraction.

Lemma 4.2 shows that for a large enough bound, $H(m)$ maps from a subspace of bounded moments to the same subspace of bounded moments.

Assume that the measure $m$ is such that $\left\langle\sup_{0\leq t\leq T}(|c_{t}|^{4}+\left\lVert w_{t}\right\rVert^{4}),m\right\rangle<K$ (where $K<\infty$ will be chosen below). Using the same steps as in Lemma 4.1, we can show that for some $T_{1}<T$ (to be chosen later),

We can now prove a contraction and then apply the Banach fixed-point theorem to prove that there is a unique fixed point.

For two elements $m^{1},m^{2}\in M_{T}$, let us set $\text{Law}(Y^{i}_{\cdot})=\text{Law}((c^{i}_{\cdot},w^{i}_{\cdot}))=H(m^{i}_{\cdot})$ for $t\in[0,T]$ with $i=1,2$. So, let $(c_{t}^{1},w_{t}^{1})$ satisfying (4.5) with $m=m^{1}$ and $(c_{t}^{2},w_{t}^{2})$ satisfying (4.5) with $m=m^{2}$. The processes $(c_{t}^{1},w_{t}^{1})$ and $(c_{t}^{2},w_{t}^{2})$ have the same initial conditions. That is,

We now prove a contraction for the mapping $H$ for some $0<T_{0}<T$. By definition, $(c_{t}^{1},w_{t}^{1})$ and $(c_{t}^{2},w_{t}^{2})$ have marginal distributions $H(m^{1})$ and $H(m^{2})$, respectively, on the time interval $[0,T_{0}]$. Once this is proven, we can extend this to the entire interval $[0,T]$ since $T_{0}$ is not affected by the input measures $m^{1},m^{2}$ or by which subinterval of $[0,T]$ we are considering. We have the following lemma.

Let $m^{1},m^{2}\in M_{T}$ and $T<\infty$. Then, there exist constants $C_{1},C_{2}<\infty$ that may depend on $T$ such that

for any $0<t<T$. In addition, if $\bar{\mu}(0,dc,dw)$ has compact support, there exists a constant $C<\infty$ that may depend on $T$ such that

First, let’s address the mean-field term. Recall that $\sigma^{\prime}(\cdot)$ is bounded and that $\pi(dx,dy)$ has bounded marginal moments via Assumption 1.1. Therefore, for $0\leq t\leq T$,

We next bound the term \bigg{|}\int_{0}^{t}\int_{\mathcal{X}\times\mathcal{Y}}\left\langle c^{\prime}_{s}\sigma(w^{\prime}_{s}x),m^{2}-m^{1}\right\rangle\sigma(w_{s}^{1}\cdot x)\pi(dx,dy)ds\bigg{|}. We have that

Let the random variables $(c_{s}^{2,^{\prime}},w_{s}^{2,^{\prime}})$ have marginal distribution $m^{2}$ and $(c_{s}^{1,^{\prime}},w_{s}^{1,^{\prime}})$ have marginal distribution $m^{1}$. Then, for $0\leq s\leq T$,

where again Assumption 1.1 was used for the moment bounds of $\pi(dx,dy)$. The inequality holds for any joint distribution $\gamma(m^{1},m^{2})$ of $m^{1}$ and $m^{2}$. Then, the auxiliary calculations provided in Appendix A show that (4.6) can be bounded in terms of $D_{s,4}(m^{1},m^{2})$:

Thus, we overall get that there is a constant $C<\infty$ such that

Similar calculations also give the necessary bound for $\parallel w^{1}_{t}-w^{2}_{t}\parallel$. For completeness, the details are provided in Appendix B.

Hence, for $0\leq s\leq T$, we have the bound

Setting for notational convenience $Z_{s}=|c_{s}^{2}-c_{s}^{1}|+\parallel w_{s}^{2}-w_{s}^{1}\parallel$, the latter relation gives

where we used the fact that $u\mapsto D_{u,2}(m^{1},m^{2})$ is monotonically increasing.

Now, note that $|c_{s}^{1}|$ can be bounded in terms of $|c_{0}^{1}|$, the initial condition. In particular, using the boundedness of $\sigma(\cdot)$, the bound on $|\left\langle G_{s,x},m^{1}\right\rangle|$, the moments bounds for the distribution $\pi(dx,dy)$, and the fact that $0\leq t\leq T<\infty$, we get for some constant $C<\infty$ that changes from line to line

This bound holds for any $0\leq t\leq T$. Then,

Raising this to the power four gives for some constants $C_{1},C_{2}<\infty$ different than before

Next, we take expectation and apply the Cauchy-Schwartz inequality on the right hand side. We obtain

The latter concludes the proof due to Assumption 1.1.

By Assumption 1.1 we have that there exists a $0<q<\infty$ such that the moment generating function exists, i.e.

Lemma 4.3 immediately proves there is a contraction on the interval $[0,T_{0}]$. Indeed,

Then, choose $T_{0}$ such that $C_{2}^{1/4}C_{M}^{1/8}T_{0}<1$, $C_{1}T_{0}<q$,and $T_{0}\leq T_{1}$, where $T_{1}$ is from Lemma 4.2. That is, we choose $T_{0}<\min\left\{\frac{1}{C_{2}^{1/4}C_{M}^{1/8}},\frac{q}{C_{1}},T_{1}\right\}.$

If $T_{0}<\min\left\{\frac{1}{C_{2}^{1/4}C_{M}^{1/8}},\frac{q}{C_{1}},T_{1}\right\}$, then $D_{T_{0},4}(H(m^{1}),H(m^{2}))\leq kD_{T_{0},4}(m^{1},m^{2})$ for $k<1$ and we have proven uniqueness on the sub-interval $[0,T_{0}]$ (via the Banach fixed-point theorem).

In fact, this directly proves our next Lemma 4.4 regarding uniqueness of the limit point over the entire interval $[0,T]$.

Let $T<\infty$. The mapping $H_{T}=(F\circ F)_{T}$ has a unique fixed point.

By Lemmas 4.2, 4.3 and the Banach fixed-point theorem we readily obtain that there is $0<T_{0}<\infty$ such that $H_{T_{0}}(m)$ will be a contraction map leading to (4.5) having a unique solution on $[0,T_{0}]$. We then extend this construction to the whole interval $[0,T]$ by dividing the interval $[0,T]$ into sub-intervals $[0,T_{0}],[T_{0},2T_{0}],\ldots,[T-T_{0},T]$. In each sub-interval, it can be shown that the solution is unique by proving a contraction as was done in Lemma 4.3, which can be done as $T_{0}$ can be always taken to be of the same magnitude, i.e. it does not depend on which sub-interval is being examined. This concludes the proof. ∎

Proof of the Main Results

We now collect the results to prove Theorem 1.2, Corollary 1.4, and Theorem 1.6.

Let $\pi^{N}$ be the probability measure corresponding to $\mu^{N}$. Each $\pi^{N}$ takes values in the set of probability measures \mathcal{M}\big{(}D_{E}([0,T])\big{)}. Relative compactness, proven in Section 2, implies that every subsequence $\pi^{N_{k}}$ has a further sub-sequence $\pi^{N_{k_{m}}}$ which weakly converges. Section 3 proves that any limit point $\pi$ of $\pi^{N_{k_{m}}}$ will satisfy the evolution equation (1.7). Section 4 proves that the solution of the evolution equation (1.7) is unique. Therefore, by Prokhorov’s Theorem, $\pi^{N}$ weakly converges to $\pi$, where $\pi$ is the distribution of $\bar{\mu}$, the unique solution of (1.7). That is, $\mu^{N}$ converges in distribution to $\bar{\mu}$. ∎

The result follows from applying integration by parts to (1.7) using the assumption that $p(t,c,w)\rightarrow 0$ as $|c|,\parallel w\parallel\rightarrow\infty$. We also note that if a solution exists to (1.4), then it is unique due to the uniqueness of (1.7). ∎

Conclusion

In this paper we develop a law of large numbers result for neural networks with a single hidden layer as the number of hidden units and stochastic gradient descent iterations grow. The limiting distribution of the parameters is rigorously shown to satisfy an explicitly stated first-order nonlinear deterministic PDE, in the form of a measure evolution equation. The limiting PDE is a function of the inputs to the model, such as the learning rate, activation function, and distribution of the observed data. A numerical study on the well-known MNIST dataset illustrates the theoretical results of this paper. In related work which builds upon the results in this paper, a central limit theorem has been proven for single-layer neural networks in and a law of large numbers has been proven for deep neural networks in .

Appendix A Proof of (4.7)

We will show that (4.6) can be bounded in terms of $D_{s,4}(m^{1},m^{2})$. For notational convenience, define $Z_{s}\vcentcolon=|c_{s}^{2,^{\prime}}-c_{s}^{1,^{\prime}}|+\parallel w^{2,^{\prime}}_{s}-w^{1,^{\prime}}_{s}\parallel$.

The sixth line uses the Cauchy-Schwartz inequality and Young’s inequality. The seventh line uses the facts that $m^{1}$ and $m^{2}$ have bounded fourth order moments. The eighth line uses Young’s inequality. We have also used the facts that $(z\wedge 1)^{4}=z^{4}\wedge 1$ and $(Kz)\wedge 1\leq K(z\wedge 1)$ when $z\geq 0$ and $K\geq 1$.

Since this inequality holds for any joint distribution $\gamma(m^{1},m^{2})$, we have that (4.7) holds.

Appendix B Proof of (4.8)

Due to $m^{1}$ having bounded moments, $|\left\langle G_{s,x},m^{1}\right\rangle|<C$ (see Section 4 for similar calculations). Due to Assumption 1.1 on $\sigma$,

Using these inequalities and the bounded moments of $\pi(dx,dy)$, we can calculate the upper bound

Using the same approach as in the bound for (4.6), see Appendix A, we have the bound

References