Dimension reduction by random hyperplane tessellations
Yaniv Plan, Roman Vershynin
Introduction
The main result of this paper is a bound on the minimal number of hyperplanes that provide a uniform tessellation of a set . It turns out that for a fixed accuracy , an almost optimal estimate on depends only on one global parameter of , namely the mean width. Recall that the Gaussian mean width of is defined as
Consider a subset and let . Let
Theorem 1.2 has an equivalent formulation in the context of metric embeddings. It yields that every subset can be almost isometrically embedded into the Hamming cube with .
To explain this statement, let us recall a few standard notions. An -isometry (or almost isometry) between metric spaces and is a map which satisfies
and such that for every one can find satisfying . A map is an -isometric embedding of into if the map is an -isometry between and the subspace . It is not hard to show that can be -isometrically embedded into (by means of a suitable map ) if has the Gromov-Haussdorff distance at most from some subset of . Conversely, if there is an -isometry between and then the Gromov-Haussdorff distance between and is bounded by .
where denotes the vector of signs of the coordinates of . The fraction of the hyperplanes that separate points and thus equals
Then looking back at the definition of uniform tessellations, we observe the following fact:
Consider a -uniform tessellation of a set by hyperplanes. Then the set (with the induced geodesic distance) can be -isometrically embedded into the Hamming cube . The sign map provides such an embedding. ∎
This allows us to state Theorem 1.2 as follows:
Consider a subset and let . Let
Then can be -isometrically embedded into the Hamming cube .
Moreover, let be an random matrix with independent entries. Then with probability at least , the sign map
2. Almost isometry of K𝐾K and the tessellation graph.
The image of the sign map in (1.3) has a special meaning. When the Hamming cube is viewed as a graph (in which two points , are connected if they differ in exactly one coordinate), the image of defines a subgraph of , which is called the tessellation graph of . The tessellation graph has a vertex for each cell and an edge for each pair of adjacent cells, see Figure 2. Notice that the graph distance in the tessellation graph equals the number of hyperplanes that separate the two cells. Therefore the definition of a uniform tessellation yields:
Consider a -uniform tessellation of a set . Then is -isometric to the tessellation graph of . ∎
Hence we can read the conclusion of Theorem 1.2 as follows: is -isometric to the graph of its tessellation by random hyperplanes, where .
3. Computing mean width
Powerful methods to estimate the mean width have been developed in connection with stochastic processses. These methods include Sudakov’s and Dudley’s inequalities which relate to the covering numbers of in the Euclidean metric, and the sharp technique of majorizing measures (see ).
Mean width has a simple (and known) geometric interpretation. By the rotational invariance of the Gaussian random vector in (1.2), one can replace with a random vector that is uniformly distributed on , as follows:
Here are numbers that depend only on and such that and . We may refer to as the spherical mean width of . Let us assume for simplicity that is symmetric with respect to the origin. Then is the width of in the direction , which is the distance between the two supporting hyperplanes of whose normals are . The spherical mean width is then twice the average width of over all directions.
4. Dimension reduction
Our results are already non-trivial in the particular case . Since , Theorems 1.2 and 1.5 hold with . But more importantly, many interesting sets satisfy and therefore make our results hold with . In such cases, one can view the sign map in Theorem 1.5 as a dimension reduction mechanism that transforms an -dimensional set into a subset of .
A heuristic reason why dimension reduction is possible is that the quantity measures the effective dimension of a set . The effective dimension of a set is always bounded by the algebraic dimension, but it may be much smaller and it is robust with respect to perturbations of . In this regard, the notion of effective dimension is parallel to the notion of effective rank of a matrix from numerical linear algebra (see e.g. ). With these observations in mind, it is not surprising that the “true”, effective dimension of would be revealed (and would be the only obstruction according to Theorem 1.5) when is being squeezed into a space of smaller dimension.
Let us illustrate dimension reduction on the example of finite sets . Since (see e.g. [16, (3.13)]), Theorem 1.5 holds with , and we can state it as follows.
Let be a finite set. Let and . Then can be -isometrically embedded into the Hamming cube . ∎
Like the Johnson-Lindenstrauss lemma, Corollary 1.7 can be proved directly by combining concentration inequalities for with a union bound over pairs . In fact, this method of proof allows for the weaker requirement . However, as we discuss later, this argument cannot be generalized in a straightforward way to prove Theorem 1.5 for general sets . The Hamming distance is highly discontinuous, which makes it difficult to extend estimates from points in an -net of to nearby points.
5. Cells of uniform tessellations
We mentioned two nice features of uniform tessellations in Facts 1.4 and 1.6. Let us observe one more property: all cells of a uniform tessellation have small diameter. Indeed, iff points are in the same cell, so by (1.1) we have:
Every cell of a -uniform tessellation has diameter at most . ∎
With this, Theorem 1.2 immediately implies the following:
Consider a tessellation of a subset by random hyperplanes. Then, with probability at least , all cells of the tessellation have diameter at most .
This result has also a direct proof, which moreover gives a slightly better bound . We present this “curvature argument” in Section 3.
Here denotes the fraction of the affine hyperplanes that separate and .
7. Optimality
The main object of our study is , the smallest number of hyperplanes that provide a -uniform tessellation of a set . One has
where denotes the covering number of , i.e. the smallest number of balls of radius that cover . The upper bound in (1.5) is the conclusion of Theorem 1.2. The lower bound holds because a -uniform tessellation provides a decomposition of into at most cells each of which lies in a ball of radius by Fact 1.8.
To compare the upper and lower bounds in (1.5), recall Sudakov’s inequality [16, Theorem 3.18] that yields
While Sudakov’s inequality cannot be reversed in general, there are many situations where it is sharp. Moreover, according to Dudley’s inequality (see [16, Theorem 11.17] and [18, Lemma 2.33]), Sudakov’s inequality can always be reversed for some scale and up to a logarithmic factor in . (See also for a discussion of sharpness of Sudakov’s inequality.) So the two sides of (1.5) are often close to each other, but there is in general some gap. We conjecture that the optimal estimate is
so the mean width of seems to be completely responsible for the uniform tessellations of .
Note that the lower bound in (1.5) holds in greater generality. Namely, it is not possible to have for any decomposition of into pieces of diameter at most . However, from the upper bound we see that with a slightly larger value , an almost best decomposition of is achieved by a random hyperplane tessellation.
In this paper we have not tried to optimize the dependence of on . This interesting problem is related to the open question on the optimal dependence on distortion in Dvoretzky’s theorem. We comment on this in Section 3.2.
8. Related work: embeddings of K𝐾K into normed spaces
This result also follows from Lemma 2.1 below.
9. Related work: one-bit compressed sensing
The problem of one-bit compressed sensing was introduced by Boufounos and Baraniuk . Jacques, Laska, Boufounos and Baraniuk realized a connection of this problem to uniform tessellations of the set of sparse signals , and to almost isometric embedding of into the Hamming cube . For this set , they proved Corollary 1.9 with and a version of Theorem 1.5 for . The authors of the present paper analyzed in a bigger set of “compressible” signals and proved for a version of Corollary 1.9 with . Since the mean widths of both sets and are of the order , Theorem 1.5 holds for these sets with . In other words, apart from the dependence of (which is an interesting problem), the prior results follow as partial cases from Theorem 1.5.
It is important to note that Theorem 1.5 addresses only the theoretical aspect of one-bit compressed sensing problem, which guarantees that the quantized measurement map well preserves the geometry of signals. But one also faces an algorithmic challenge – how to efficiently recover from , and specifically in polynomial time. We will not touch on this algorithmic aspect here but rather refer the reader to and to our forthcoming work which is based on the results of this paper.
10. Related work: locality-sensitive hashing
11. Overview of the argument
We instead attempt to prove Theorem 1.2 by an -net argument, which typically proceeds as follows: (a) show that holds for a fixed pair with high probability; (b) take the union bound over all pairs in an finite -net of ; (c) extend the estimate from to by approximation. Unfortunately, as we indicate in Section 4 the approximation step (c) must fail due to the discontinuity of the Hamming distance .
A solution proposed in was to choose so small that none of the random hyperplanes pass near points with high probability. This strategy was effective for the set because the covering number of this specific set has a mild (logarithmic) dependence on , namely . However, adapting this strategy to general sets would cause our estimate on to increase by a factor of .
12. Notation
(b) The following deviation inequality holds:
By the rotational invariance of the Gaussian distribution, is distributed identically with where . Therefore
Thus has Lipschitz constant bounded by . We may now bound the deviation probability for using the Gaussian concentration inequality (see [16, Equation 1.6]) as follows:
One can state Lemma 2.1 in terms of random matrices. Indeed, let be an random matrix with independent entries. Then its rows satisfy the assumption of Lemma 2.1, and we can express as
Let be the random matrix as in Remark 2.2. Using Lemma 2.1 for and noting the form of in (2.3), we conclude that the following event holds with probability at least :
The above argument shows in fact that Corollary 2.3 holds for
As we noticed in Remark 1.11, the quantity more accurately reflects the geometric meaning of the mean width than .
Note that for the subspace we have from (2.3) that . Then Lemma 2.1 implies that
Proof of Corollary 1.9 by a curvature argument
In this section we give a short argument that leads to a version of Corollary 1.9 with a slightly better dependence of on .
Consider a subset and let . Let
The argument is based on Lemma 2.1. If points belong to the same cell, then the midpoint also belongs to the same cell (after normalization). Using Lemma 2.1 one can then show that . Due to the curvature of the sphere, this forces the length of the interval to be small, which means that the diameter of the cell is small. The formal argument is below.
Assume that the event (3.1) holds. Consider a pair of points that belong to the same cell of the tessellation, which means that
To complete the proof is suffices to show that . This will give desired diameter in the Euclidean metric. Furthermore, since for small the Euclidean and the geodesic distances are equivalent, the conclusion will hold for the geodesic distance as well.
We shall use (3.1) for and for the midpoint . Clearly , hence
By the parallelogram law, we conclude that
Unfortunately, the curvature argument does not lend itself to proving the more general result, Theorem 1.2 on uniform tessellations. To see why, suppose do not belong to the same cell but instead for some small . Consider the set of mismatched signs
These signs create an additional error term in the right hand side of (3.2), which is
By analogy with Lemma 2.1, we can expect that this term should be approximately equal . If this is true, then (3.2) becomes in our situation , which leads as before to . Ignoring , we see that the best estimate the curvature argument can give is rather than that is required in Theorem 1.2.
The weak point of this argument is that it takes into account the size of but ignores the nature of . For every , the hyperplane passes through the arc connecting and . If the length of the arc is small, this creates a strong constraint on . Conditioning the distribution of on the constraint that creates a bias toward smaller values of and . As a result, the conditional expected value of the error term (3.3) should be smaller than . Computing this conditional expectation is not a problem for a given pair , but it seems to be difficult to carry out a uniform argument over where the (conditional) distribution of depends on .
We instead propose a different and somewhat more conceptual way to deduce Theorem 1.2 from Lemma 2.1. This argument will be developed in the rest of this paper.
2. Dvoretzky theorem and dependence on δ𝛿\delta
The unusual dependence in Theorem 3.1 is related to the open problem of the optimal dependence on distortion in the Dvoretzky theorem.
Toward Theorem 1.2: a soft Hamming distance
Our proof of Theorem 1.2 will be based on a covering argument. A standard covering argument of geometric functional analysis would proceed in our situation as follows:
Show that with high probability for a fixed pair . This can be done using standard concentration inequalities.
Prove that uniformly for all in a finite -net of . Sudakov’s inequality can be used to estimate the cardinality of via the mean width . The conclusion will follow from step 1 by the union bound over .
Extend the estimate from to by approximation.
Both positive and negative may be considered. For positive the soft Hamming distance counts the hyperplanes that separate well enough; for negative it counts the hyperplanes that separate or nearly separate .
Clearly is a non-increasing function of . Moreover,
The soft Hamming distance for a fixed is as discontinuous as the usual (hard) Hamming distance. However, some version of continuity emerges when we allow to vary slightly:
Consider the events from the definition of the soft Hamming distance (4.2). By the assumptions, we have , for all . This implies by the triangle inequality that
We are ready to state a stronger version of Theorem 1.2 for the soft Hamming distance.
Consider a subset and let . Let
Note that if we take in the above theorem, we recover Theorem 1.2. However, we find it easier to prove the result for general , since in our argument we will work with different values of the for the soft Hamming distance.
Theorem 4.4 is proven in the next section.
Proof of Theorem 4.4 on the soft Hamming distance
We will follow the covering argument outlined in the beginning of Section 4, but instead of we shall work with the soft Hamming distance .
The first inequality follows from (5.1) and Jensen’s inequality. To prove the second inequality, we use the events and from Equations (4.1), (4.2) defining the hard and soft Hamming distances, respectively. It follows that
Now we upgrade Lemma 5.1 to an concentration inequality:
A standard Chernoff bound for binomial random variables states that
see e.g. [7, Corollary A.1.7]. The triangle inequality completes the proof. ∎
2. Concentration of distance over an ε𝜀\varepsilon-net
Let us fix a small whose value will be determined later. Let be an -net of in the Euclidean metric. By Sudakov’s inequality (see [16, Theorem 3.18]), we can arrange the cardinality of to satisfy
We can decompose every vector into a center and a tail so that
We first control the centers by taking a union bound in Lemma 5.2 over the net :
Let a random Gaussian matrix be as in Theorem 4.4. Let be a subset of whose cardinality satisfies (5.2). Let , and assume that
By Lemma 5.3 and a union bound over the set of pairs , we obtain
where the last inequality follows by (5.2) and (5.4). The proof is complete. ∎
3. Control of the tails
Now we control the tails in decomposition (5.3).
Consider a subset and let . Let
Consider independent random vectors . Then with probability at least , one has
Let us apply Lemma 2.1 for the set instead of , and for . Since , we obtain that the following holds with probability at least :
Note that . So using the assumption on we conclude that the quantity in (5.5) is bounded by , as claimed. ∎
4. Approximation
Now we establish a way to transfer the distance estimates from an -net to the full set . This is possible by a continuity property of the soft Hamming distance, which we outlined in Lemma 4.3. This result requires the perturbation to be bounded in norm. However, in our situation the perturbations are going to be bounded only in norm due to Lemma 5.4. So we shall prove the following relaxed version of continuity:
Consider the events from the definition of the soft Hamming distance (4.2). By the assumptions, we have
This proves the first inequality in (5.6). The proof of the second inequality is similar. ∎
5. Proof of Theorem 4.4.
Now we are ready to combine all the pieces and prove Theorem 4.4. To this end, consider the set , numbers , , , and the random matrix as in the theorem. Choose and .
Consider an -net of as we described in the beginning of Section 5.2. Let us apply Lemma 5.3 that controls the distances on along with Lemma 5.4 that controls the tails. By the assumption on in the theorem and by our choice of , both requirements on in these lemmas hold. By a union bound, with probability at least the following event holds: for every and , one has
Let . As we described in (5.3), we can decompose the vectors as
The bounds in (5.8) guarantee that the continuity property (5.6) in Lemma 5.5 holds. This gives
Finally, by the choice of and we obtain
This completes the proof of Theorem 4.4. ∎
Fix a large number whose value will be chosen later and consider the set
where the last inequality holds because by (6.1).
Consider arbitrary vectors and in and the corresponding vectors and in . Let us relate the distances between and appearing in (6.3) to corresponding distances between and .
Next we analyze the normalized geodesic distance , which satisfies
Denoting and and using the triangle inequality, we obtain
Note that (6.1) yields that . It follows that and the same bound holds for the other two similar terms in (6.6). Using this and (6.1) we conclude that . Putting this into (6.5) and using the triangle inequality twice, we obtain
Finally, we use this bound and (6.4) in (6.3), which gets us
Now we can assign the values and so the right hand side of (6.7) is bounded by , as required. Note that the condition that we used above in order to apply Theorem 1.2 is satisfied by (6.2). This completes the proof of Theorem 1.10. ∎