Provable Bounds for Learning Some Deep Representations

Introduction

Can we provide theoretical explanation for the practical success of deep nets? Like many other ML tasks, learning deep neural nets is NP-hard, and in fact seems “badly NP-hard”because of many layers of hidden variables connected by nonlinear operations. Usually one imagines that NP-hardness is not a barrier to provable algorithms in ML because the inputs to the learner are drawn from some simple distribution and are not worst-case. This hope was recently borne out in case of generative models such as HMMs, Gaussian Mixtures, LDA etc., for which learning algorithms with provable guarantees were given [HKZ12, MV10, HK13, AGM12, AFH+12]. However, supervised learning of neural nets even on random inputs still seems as hard as cracking cryptographic schemes: this holds for depth-$5$ neural nets [JKS02] and even ANDs of thresholds (a simple depth two network) [KS09].

However, modern deep nets are not “just”neural nets (see the survey [Ben09]). The underlying assumption is that the net (or some modification) can be run in reverse to get a generative model for a distribution that is a close fit to the empirical input distribution. Hinton promoted this viewpoint, and suggested modeling each level as a Restricted Boltzmann Machine (RBM), which is “reversible”in this sense. Vincent et al. [VLBM08] suggested using many layers of a denoising autoencoder, a generalization of the RBM that consists of a pair of encoder-decoder functions (see Definition 1). These viewpoints allow a different learning methodology than classical backpropagation: layerwise learning of the net, and in fact unsupervised learning. The bottom (observed) layer is learnt in unsupervised fashion using the provided data. This gives values for the next layer of hidden variables, which are used as the “data” to learn the next higher layer, and so on. The final net thus learnt is also a good generative model for the distribution of the bottom layer. In practice the unsupervised phase is followed by supervised trainingRecent work suggests that classical backpropagation-based learning of neural nets together with a few modern ideas like convolution and dropout training also performs very well [KSH12], though the authors suggest that unsupervised pretraining should help further..

This viewpoint of reversible deep nets is more promising for theoretical work because it involves a generative model, and also seems to get around cryptographic hardness. But many barriers still remain. There is no known mathematical condition that describes neural nets that are or are not denoising autoencoders. Furthermore, learning even a a single layer sparse denoising autoencoder seems at least as hard as learning sparse-used overcomplete dictionaries (i.e., a single hidden layer with linear operations), for which there were no provable bounds at all until the very recent manuscript [AGM13]The parameter choices in that manuscript make it less interesting in context of deep learning, since the hidden layer is required to have no more than $\sqrt{n}$ nonzeros where $n$ is the size of the observed layer —in other words, the observed vector must be highly compressible..

The current paper presents both an interesting family of denoising autoencoders as well as new algorithms to provably learn almost all models in this family. Our ground truth generative model is a simple multilayer neural net with edge weights in $$and simple threshold (i.e.,$>0$) computation at the nodes. A$k$-sparse$0/1$ assignment is provided at the top hidden layer, which is computed upon by successive hidden layers in the obvious way until the “observed vector”appears at the bottommost layer. If one makes no further assumptions, then the problem of learning the network given samples from the bottom layer is still harder than breaking some cryptographic schemes. (To rephrase this in autoencoder terminology: our model comes equipped with a decoder function at each layer. But this is not enough to guarantee an efficient encoder function—this may be tantamount to breaking cryptographic schemes.)

So we make the following additional assumptions about the unknown “ground truth deep net”(see Section 2): (i) Each feature/node activates/inhibits at most $n^{\gamma}$ features at the layer below, and is itself activated/inhibited by at most $n^{\gamma}$ features in the layer above, where $\gamma$ is some small constant; in other words the ground truth net is not a complete graph. (ii) The graph of these edges is chosen at random and the weights on these edges are random numbers in $$.

Our algorithm learns almost all networks in this class very efficiently and with low sample complexity; see Theorem 1. The algorithm outputs a network whose generative behavior is statistically indistinguishable from the ground truth net. (If the weights are discrete, say in $\left\{-1,1\right\}$ then it exactly learns the ground truth net.)

Along the way we exhibit interesting properties of such randomly-generated neural nets. (a) Each pair of adjacent layers constitutes a denoising autoencoder in the sense of Vincent et al.; see Lemma 2. Since the model definition already includes a decoder, this involves showing the existence of an encoder that completes it into an autoencoder. (b) The encoder is actually the same neural network run in reverse by appropriately changing the thresholds at the computation nodes. (c) The reverse computation is stable to dropouts and noise. (d) The distribution generated by a two-layer net cannot be represented by any single layer neural net (see Section 8), which in turn suggests that a random t-layer network cannot be represented by any $t/2$-level neural netFormally proving this for $t>3$ is difficult however since showing limitations of even 2-layer neural nets is a major open problem in computational complexity theory. Some deep learning papers mistakenly cite an old paper for such a result, but the result that actually exists is far weaker..

Note that properties (a) to (d) are assumed in modern deep net work: for example (b) is a heuristic trick called “weight tying”. The fact that they provably hold for our random generative model can be seen as some theoretical validation of those assumptions.

Context. Recent papers have given theoretical analyses of models with multiple levels of hidden features, including SVMs [CS09, LSSS13]. However, none of these solves the task of recovering a ground-truth neural network given its output distribution.

Though real-life neural nets are not random, our consideration of random deep networks makes some sense for theory. Sparse denoising autoencoders are reminiscent of other objects such as error-correcting codes, compressed sensing, etc. which were all first analysed in the random case. As mentioned, provable reconstruction of the hidden layer (i.e., input encoding) in a known autoencoder already seems a nonlinear generalization of compressed sensing, whereas even the usual (linear) version of compressed sensing seems possible only if the adjacency matrix has “random-like” properties (low coherence or restricted isoperimetry or lossless expansion). In fact our result that a single layer of our generative model is a sparse denoising autoencoder can be seen as an analog of the fact that random matrices are good for compressed sensing/sparse reconstruction (see Donoho [Don06] for general matrices and Berinde et al. [BGI+08] for sparse matrices). Of course, in compressed sensing the matrix of edge weights is known whereas here it has to be learnt, which is the main contribution of our work. Furthermore, we show that our algorithm for learning a single layer of weights can be extended to do layerwise learning of the entire network.

Does our algorithm yield new approaches in practice? We discuss this possibility after sketching our algorithm in the next section.

Definitions and Results

Algorithmic ideas. We are unable to analyse existing algorithms. Instead, we give new learning algorithms that exploit the very same structure that makes these random networks interesting in the first place i.e., each layer is a denoising autoencoder. The crux of the algorithm is a new twist on the old Hebbian rule [Heb49] that “Things that fire together wire together.” In the setting of layerwise learning, this is adapted as follows: “Nodes in the same layer that fire together a lot are likely to be connected (with positive weight) to the same node at the higher layer.” The algorithm consists of looking for such pairwise (or $3$-wise) correlations and putting together this information globally. The global procedure boils down to the graph-theoretic problem of reconstructing a bipartite graph given pairs of nodes that are at distance $2$ in it (see Section 6). This is a variant of the GRAPH SQUARE ROOT problem which is NP-complete on worst-case instances but solvable for sparse random (or random-like) graphs.

Note that current algorithms (to the extent that they are Hebbian, roughly speaking) can also be seen as leveraging correlations. But putting together this information is done via the language of nonlinear optimization (i.e., an objective function with suitable penalty terms). Our ground truth network is indeed a particular local optimum in any reasonable formulation. It would be interesting to show that existing algorithms provably find the ground truth in polynomial time but currently this seems difficult.

Can our new ideas be useful in practice? We think that using a global reconstruction procedure to leverage local correlations seems promising, especially if it avoids the usual nonlinear optimization. Our proof currently needs that the hidden layers are sparse, and the edge structure of the ground truth network is “random like”(in the sense that two distinct features at a level tend to affect fairly disjoint-ish sets of features at the next level). Finally, we note that random neural nets do seem useful in so-called reservoir computing, so perhaps they do provide useful representational power on real data. Such empirical study is left for future work.

Throughout, we need well-known properties of random graphs with expected degree $d$, such as the fact that they are expanders; these properties appear in the appendix. The most important one, unique neighbors property, appears in the next Section.

Each layer is a Denoising Auto-encoder

As mentioned earlier, modern deep nets research often assumes that the net (or at least some layers in it) should approximately preserve information, and even allows easy going back/forth between representations in two adjacent layers (what we earlier called “reversibility”). Below, $y$ denotes the lower layer and $h$ the higher (hidden) layer. Popular choices of $s$ include logistic function, soft max, etc.; we use simple threshold function in our model.

(Denoising autoencoder) An autoencoder consists of a decoding function $D(h)=s(Wh+b)$ and an encoding function $E(y)=s(W^{\prime}y+b^{\prime})$ where $W,W^{\prime}$ are linear transformations, $b,b^{\prime}$ are fixed vectors and $s$ is a nonlinear function that acts identically on each coordinate. The autoencoder is denoising if $E(D(h)+\eta)=h$ with high probability where $h$ is drawn from the distribution of the hidden layer, $\eta$ is a noise vector drawn from the noise distribution, and $D(h)+\eta$ is a shorthand for “$D(h)$ corrupted with noise $\eta$.” The autoencoder is said to use weight tying if $W^{\prime}=W^{T}$.

We show a single-layer random network is a denoising autoencoder if the input layer is a random $\rho n$ sparse vector, and the output layer has density $\rho d/2<1/20$.

If $\rho d<0.1$ (i.e., the assignment to the observed layer is also fairly sparse) then the single-layer network above is a denoising autoencoder with high probability (over the choice of the random graph and weights), where the noise distribution is allowed to flip every output bit independently with probability $0.1$. It uses weight tying.

The proof of this lemma highly relies on a property of random graph, called the strong unique-neighbor property.

For any node $u\in U$ and any subset $S\subset U$ , let $\mathit{UF}(u,S)$ be the sets of unique neighbors of $u$ with respect to $S$,

In a bipartite graph $G(U,V,E,w)$, a node $u\in U$ has $(1-\epsilon)$-unique neighbor property with respect to $S$ if

The set $S$ has $(1-\epsilon)$-strong unique neighbor property if for every $u\in U$, $u$ has $(1-\epsilon)$-unique neighbor property with respect to $S$.

When we just assume $\rho d\ll n$, this property does not hold for all sets of size $\rho n$. However, for any fixed set $S$ of size $\rho n$, this property holds with high probability over the randomness of the graph.

Now we sketch the proof for Lemma 2 (details are in Appendix).For convenience assume the edge weights are in $\left\{-1,1\right\}$.

First, the decoder definition is implicit in our generative model: $y=\operatorname{sgn}(Wh)$. (That is, $b=\vec{0}$ in the autoencoder definition.) Let the encoder be $E(y)=\operatorname{sgn}(W^{T}y+b^{\prime})$ for $b^{\prime}=0.2d\times\vec{1}$.In other words, the same bipartite graph and different thresholds can transform an assignment on the lower level to the one at the higher level.

To prove this consider the strong unique-neighbor property of the network. For the set of nodes that are $1$ at the higher level, a majority of their neighbors at the lower level are adjacent only to them and to no other nodes that are $1$. The unique neighbors with a positive edge will always be 1 because there are no $-1$ edges that can cancel the $+1$ edge (similarly the unique neighbors with negative edge will always be 0). Thus by looking at the set of nodes that are $1$ at the lower level, one can easily infer the correct $0/1$ assignment to the higher level by doing a simple threshold of say $0.2d$ at each node in the higher layer.

Learning a single layer network

Our algorithm, outlined below (Algorithm 1), learns the network layer by layer starting from the bottom. Thus the key step is that of learning a single layer network, which we now focus on.Learning the bottom-most (real valued) layer is mildly different and is done in Section 7. This step, as we noted, amounts to learning nonlinear dictionaries with random dictionary elements. The algorithm illustrates how we leverage the sparsity and the randomness of the support graph, and use pairwise or 3-wise correlations combined with our graph recovery procedure of Section 6. We first give a simple algorithm and then outline one that works with better parameters.

For simplicity we describe the algorithm when edge weights are $\left\{-1,1\right\}$, and sketch the differences for real-valued weights at the end of this section.

The hidden layer and observed layer each have $n$ nodes, and the generative model assumes the assignment to the hidden layer is a random $0/1$ assignment with $\rho n$ nonzeros.

Say two nodes in the observed layer are related if they have a common neighbor in the hidden layer to which they are attached via a $+1$ edge.

Step 1: Construct correlation graph: This step is a new twist on the classical Hebbian rule (“things that fire together wire together”).

Claim In a random sample of the output layer, related pairs $u,v$ are both $1$ with probability at least $0.9\rho$, while unrelated pairs are both $1$ with probability at most $(\rho d)^{2}$.

(Proof Sketch): First consider a related pair $u,v$, and let $z$ be a vertex with $+1$ edges to $u$, $v$. Let $S$ be the set of neighbors of $u$, $v$ excluding $z$. The size of $S$ cannot be much larger than $2d$. Under the choice of parameters, we know $\rho d\ll 1$, so the event $h_{S}=\vec{0}$ conditioned on $h_{z}=1$ has probability at least 0.9. Hence the probability of $u$ and $v$ being both $1$ is at least $0.9\rho$. Conversely, if $u,v$ are unrelated then for both $u,v$ to be $1$ there must be two different causes, namely, nodes $y$ and $z$ that are $1$, and additionally, are connected to $u$ and $v$ respectively via $+1$ edges. The chance of such $y,z$ existing in a random sparse assignment is at most $(\rho d)^{2}$ by union bound.

Thus, if $\rho$ satisfies $(\rho d)^{2}<0.1\rho$, i.e., $\rho<0.1/d^{2}$, then using $O(\log n/\rho^{2})$ samples we can recover all related pairs whp, finishing the step.

Step 2: Use graph recover procedure to find all edges that have weight $+1$. (See Section 6 for details.)

Step 3: Using the $+1$ edges to encode all the samples $y$.

Although we have only recovered the positive edges, we can use PartialEncoder algorithm to get $h$ given $y$!

If support of $h$ satisfies $11/12$-strong unique neighbor property, and $y=\operatorname{sgn}(Gh)$, then Algorithm 3 outputs $h$ with $\theta=0.3d$.

This uses the unique neighbor property: for every $z$ with $h_{z}=1$, it has at least $0.4d$ unique neighbors that are connected with $+1$ edges. All these neighbors must be $1$ so $[(E^{+})^{T}y]_{z}\geq 0.4d$. On the other hand, for any $z$ with $h_{z}=0$, the unique neighbor property (applied to supp$(h)\cup\{z\}$) implies that $z$ can have at most $0.2d$ positive edges to the $+1$’s in $y$. Hence $h=\operatorname{sgn}((E^{+})^{T}y-0.3d\vec{1})$.

Now consider many pairs of $(h,y)$, where $h$ is found using Step 3. Suppose in some sample, $y_{u}=1$ for some $u$, and exactly one neighbor of $u$ in the $+1$ edge graph (which we know entirely) is in supp$(h)$. Then we can conclude that for any $z$ with $h_{z}=1$, there cannot be a $-1$ edge $(z,u)$, as this would cancel out the unique $+1$ contribution.

Given $O(\log n/(\rho^{2}d))$ samples of pairs $(h,y)$, with high probability (over the random graph and the samples) Algorithm 4 outputs the correct set $E^{-}$.

To prove this lemma, we just need to bound the probability of the following event for any non-edge $(x,u)$: $h_{x}=1$, $\left|\operatorname{supp}(h)\cap B^{+}(u)\right|=1$, $\operatorname{supp}(h)\cap B^{-}(u)=\emptyset$ ($B^{+},B^{-}$ are positive and negative parents). These three events are almost independent, the first has probability $\rho$, second has probability $\approx\rho d$ and the third has probability almost 1.

The above sketch used pairwise correlations to recover the $+1$ weights when $\rho<1/d^{2}$, roughly. It turns out that using $3$-wise correlations allow us to find correlations under a weaker requirement $\rho<1/d^{3/2}$. Now call three observed nodes $u,v,s$ related if they are connected to a common node at the hidden layer via $+1$ edges. Then we can prove a claim analogous to the one above, which says that for a related triple, the probability that $u,v,s$ are all $1$ is at least $0.9\rho$, while the probability for unrelated triples is roughly at most $(\rho d)^{3}$. Thus as long as $\rho<0.1/d^{3/2}$, it is possible to find related triples correctly. The graph recover algorithm can be modified to run on $3$-uniform hypergraph consisting of these related triples to recover the $+1$ edges.

The end result is the following theorem. This is the learner used to get the bounds stated in our main theorem.

Suppose our generative neural net model with weights $\left\{-1,1\right\}$ has a single layer and the assignment of the hidden layer is a random $\rho n$-sparse vector, with $\rho\ll 1/d^{3/2}$. Then there is an algorithm that runs in $O(n(d^{3}+n))$ time and uses $O(\log n/\rho^{2})$ samples to recover the ground truth with high probability over the randomness of the graph and the samples.

When weights are real numbers.

We only sketch this and leave the details to the appendix. Surprisingly, steps 1, 2 and 3 still work. In the proofs, we have only used the sign of the edge weights – the magnitude of the edge weights can be arbitrary. This is because the proofs in these steps relies on the unique neighbor property, if some node is on (has value $1$), then its unique positive neighbors at the next level will always be on, no matter how small the positive weights might be. Also notice in PartialEncoder we are only using the support of $E^{+}$, but not the weights.

After Step 3 we have turned the problem of unsupervised learning of the hidden graph to a supervised one in which the outputs are just linear classifiers over the inputs! Thus the weights on the edges can be learnt to any desired accuracy.

Correlations in a Multilayer Network

(outline) The first step is to show that for a vertex $u$ in level $i$, $\Pr[h^{(i)}(u)=1]$ is at least $3\rho_{i}/4$ and at most $5\rho_{i}/4$. This is shown by an inductive argument (details in the full version). (This is the step where we crucially use the randomness of the underlying graph.)

Now suppose $u,v$ have a common neighbor $z$ with $+1$ edges to both of them. Consider the event that $z$ is $1$ and none of the neighbors of $u,v$ with $-1$ weight edges are $1$ in layer $h^{(2)}$. These conditions ensure that $h^{(1)}(u)=h^{(1)}(v)=1$; further, they turn out to occur together with probability at least $\rho_{2}/2$, because of the bound from the first step, along with the fact that $u,v$ combined have only $2d$ neighbors (and $2d\rho_{2}n\ll n$), so there is good probability of not picking neighbors with $-1$ edges.

If $u,v$ are not related, it turns out that the probability of interest is at most $2\rho_{1}^{2}$ plus a term which depends on whether $u,v$ have a common parent in layer $h^{(3)}$ in the graph restricted to $+1$ edges. Intuitively, picking one of these common parents could result in $u,v$ both being $1$. By our choice of parameters, we will have $\rho_{1}^{2}<\rho_{2}/20$, and also the additional term will be $<\rho_{2}/10$, which implies the desired conclusion.

Then as before, we can use graph recovery to find all the $+1$ edges in the graph at the bottom most layer. This can then be used (as in Step 3) in the single layer algorithm to encode $h^{(1)}$ and obtain values for $h^{(2)}$. Now as before, we have many pairs $(h^{(2)},h^{(1)})$, and thus using precisely the reasoning of Step 4 earlier, we can obtain the full graph at the bottom layer.

This argument can be repeated after ‘peeling off’ the bottom layer, thus allowing us to learn layer by layer.

Graph Recovery

Graph reconstruction consists of recovering a graph given information about its subgraphs [BH77]. A prototypical problem is the Graph Square Root problem, which calls for recovering a graph given all pairs of nodes whose distance is at most $2$. This is NP-hard.

Let $G_{1}(U,V,E_{1})$ be an unknown random bipartite graph between $|U|=n$ and $|V|=n$ vertices where each edge is picked with probability $d/n$ independently.

Given: Graph $G(V,E)$ where $(v_{1},v_{2})\in E$ iff $v_{1}$ and $v_{2}$ share a common parent in $G_{1}$ (i.e. $\exists u\in U$ where $(u,v_{1})\in E_{1}$ and $(u,v_{2})\in E_{1}$).

Some of our algorithms (using $3$-wise correlations) need to solve analogous problem where we are given triples of nodes which are mutually at distance $2$ from each other, which we will not detail for lack of space.

We let $F(S)$ (resp. $B(S)$) denote the set of neighbors of $S\subseteq U$ (resp. $\subseteq V$) in $G_{1}$. Also $\Gamma(\cdot)$ gives the set of neighbors in $G$. Now for the recovery algorithm to work, we need the following properties (all satisfied whp by random graph when $d^{3}/n\ll 1$):

For any $v_{1},v_{2}\in V$, $\left|(\Gamma(v_{1})\cap\Gamma(v_{2}))\backslash(F(B(v_{1})\cap B(v_{2})))\right|<d/20$.

For any $u_{1},u_{2}\in U$, $\left|F(u_{1})\cup F(u_{2})\right|>1.5d$.

For any $u\in U$, $v\in V$ and $v\not\in F(u)$, $\left|\Gamma(v)\cap F(u)\right|<d/20$.

For any $u\in U$, at least $0.1$ fraction of pairs $v_{1},v_{2}\in F(u)$ does not have a common neighbor other than $u$.

The first property says “most correlations are generated by common cause”: all but possibly $d/20$ of the common neighbors of $v_{1}$ and $v_{2}$ in $G$, are in fact neighbors of a common neighbor of $v_{1}$ and $v_{2}$ in $G_{1}$.

The second property basically says the sets $F(u)$’s should be almost disjoint, this is clear because the sets are chosen at random.

The third property says if a vertex $v$ is not related to the cause $u$, then it cannot have correlation with all many neighbors of $u$.

The fourth property says every cause introduces a significant number of correlations that is unique to that cause.

In fact, Properties 2-4 are closely related from the unique neighbor property.

When graph $G_{1}$ satisfies Properties 1-4, Algorithm 5 successfully recovers the graph $G_{1}$ in expected time $O(n^{2})$.

We first show that when $(v_{1},v_{2})$ has more than one unique common cause, then the condition in the if statement must be false. This follows from Property 2. We know the set $S$ contains $F(B(v_{1})\cap B(v_{2}))$. If $\left|B(v_{1})\cap B(v_{2})\right|\geq 2$ then Property 2 says $\left|S\right|\geq 1.5d$, which implies the condition in the if statement is false.

Then we show if $(v_{1},v_{2})$ has a unique common cause $u$, then $S^{\prime}$ will be equal to $F(u)$. By Property 1, we know $S=F(u)\cup T$ where $\left|T\right|\leq d/20$.

For any vertex $v$ in $F(u)$, it is connected to every other vertex in $F(u)$. Therefore $\left|\Gamma(v)\cap S\right|\geq\left|\Gamma(v)\cap F(u)\right|\geq 0.8d-1$, and $v$ must be in $S^{\prime}$.

For any vertex $v^{\prime}$ outside $F(u)$, by Property 3 it can only be connected to $d/20$ vertices in $F(u)$. Therefore $\left|\Gamma(v)\cap S\right|\leq\left|\Gamma(v)\cap F(u)\right|+|T|\leq d/10$. Hence $v^{\prime}$ is not in $S^{\prime}$.

Following these arguments, $S^{\prime}$ must be equal to $F(u)$, and the algorithm successfully learns the edges related to $u$.

The algorithm will successfully find all vertices $u\in U$ because of Property 4: for every $u$ there are enough number of edges in $G$ that is only caused by $u$. When one of them is sampled, the algorithm successfully learns the vertex $u$.

Finally we bound the running time. By Property 4 we know that the algorithm identifies a new vertex $u\in U$ in at most $10$ iterations in expectation. Each iteration takes at most $O(n)$ time. Therefore the algorithm takes at most $O(n^{2})$ time in expectation.

Learning the lowermost (real-valued) layer

Note that in our model, the lowest (observed) layer is real-valued and does not have threshold gates. Thus our earlier learning algorithm cannot be applied as is. However, we see that the same paradigm – identifying correlations and using Graph recover – can be used.

The first step is to show that for a random weighted graph $G$, the linear decoder $D(h)=Gh$ and the encoder $E(y)=\operatorname{sgn}(G^{T}y+b)$ form a denoising autoencoder with real-valued outputs, as in Bengio et al. [BCV13].

If $G$ is a random weighted graph, the encoder $E(y)=\operatorname{sgn}(G^{T}y-0.4d\vec{1})$ and linear decoder $D(h)=Gh$ form a denoising autoencoder, for noise vectors $\gamma$ which have independent components, each having variance at most $O(d/\log^{2}n)$.

The next step is to show a bound on correlations as before. For simplicity we state it assuming the layer $h^{(1)}$ has a random $0/1$ assignment of sparsity $\rho_{1}$. In the full version we state it keeping in mind the higher layers, as we did in the previous sections.

When $\rho_{1}d=O(1)$, $d=\Omega(\log^{2}n)$, with high probability over the choice of the weights and the choice of the graph, for any three nodes $u,v,s$ the assignment $y$ to the bottom layer satisfies:

Two layers cannot be represented by one layer

In this section we show that a two-layer network with $\pm 1$ weights is more expressive than one layer network with arbitrary weights. A two-layer network $(G_{1},G_{2})$ consists of random graphs $G_{1}$ and $G_{2}$ with random $\pm 1$ weights on the edges. Viewed as a generative model, its input is $h^{(3)}$ and the output is $h^{(1)}=\operatorname{sgn}(G_{1}\operatorname{sgn}(G_{2}h^{(3)}))$. We will show that a single-layer network even with arbitrary weights and arbitrary threshold functions must generate a fairly different distribution.

For almost all choices of $(G_{1},G_{2})$, the following is true. For every one layer network with matrix $A$ and vector $b$, if $h^{(3)}$ is chosen to be a random $\rho_{3}n$-sparse vector with $\rho_{3}d_{2}d_{1}\ll 1$, the probability (over the choice of $h^{(3)}$) is at least $\Omega(\rho_{3}^{2})$ that $\operatorname{sgn}(G_{1}\operatorname{sgn}(G_{1}h^{(3)}))\neq\operatorname{sgn}(Ah^{(3)}+b)$.

The idea is that the cancellations possible in the two-layer network simply cannot all be accomodated in a single-layer network even using arbitrary weights. More precisely, even the bit at a single output node $v$ cannot be well-represented by a simple threshold function.

First, observe that the output at $v$ is determined by values of $d_{1}d_{2}$ nodes at the top layer that are its ancestors. It is not hard to show in the one layer net $(A,b)$, there should be no edge between $v$ and any node $u$ that is not its ancestor. Then consider structure in Figure 2. Assuming all other parents of $v$ are 0 (which happen with probability at least $0.9$), and focus on the values of $(u_{1},u_{2},u_{3},u_{4})$. When these values are $(1,1,0,0)$ and $(0,0,1,1)$, $v$ is off. When these values are $(1,0,0,1)$ and $(0,1,1,0)$, $v$ is on. This is impossible for a one layer network because the first two ask for $\sum_{A_{u_{i},v}}+2b_{v}\leq 0$ and the second two ask for $\sum_{A_{u_{i},v}}+2b_{v}<0$.

Conclusions

Rigorous analysis of interesting subcases of any ML problem can be beneficial for triggering further improvements: see e.g., the role played in Bayes nets by the rigorous analysis of message-passing algorithms for trees and graphs of low tree-width. This is the spirit in which to view our consideration of a random neural net model (though note that there is some empirical work in reservoir computing using randomly wired neural nets).

The concept of a denoising autoencoder (with weight tying) suggests to us a graph with random-like properties. We would be very interested in an empirical study of the randomness properties of actual deep nets learnt in real life. (For example, in [KSH12] some of the layers use convolution, which is decidedly nonrandom. But other layers do backpropagation starting with a complete graph and may end up more random-like.)

Network randomness is not so crucial for single-layer learning. But for provable layerwise learning we rely on the support (i.e., nonzero edges) being random: this is crucial for controlling (i.e., upper bounding) correlations among features appearing in the same hidden layer (see Lemma 6). Provable layerwise learning under weaker assumptions would be very interesting.

Acknowledgments

We would like to thank Yann LeCun, Ankur Moitra, Sushant Sachdeva, Linpeng Tang for numerous helpful discussions throughout various stages of this work. This work was done when the first, third and fourth authors were visiting EPFL.

References