A semidefinite program for unbalanced multisection in the stochastic block model

Introduction

The stochastic block model is among the most actively studied models for networks with latent community structure. Such models lend themselves to the statistical task of recovering the community structure given a single such graph, a task known as community detection. This task manifests itself as that of finding human communities in social networks, and that of determining sets of proteins that function together in protein–protein interaction networks; more generally, community detection is the analogue of clustering for network data.

Since the introduction of the stochastic block model by Holland [HLL83], a wide range of algorithmic approaches have been deployed. Such algorithms are generally compared through the range of model parameters in which they succeed in recovering information; whether they achieve exact recovery of the underlying community structure, or only partial recovery of community structure correlated with the truth; and their computational efficiency.

In recent years, the block model has received a surge of interest as a meeting point of machine learning, algorithms, and statistical mechanics. A series of conjectures by Decelle et al. [DKMZ11] predicted a sharp threshold, separating a range of model parameters in which partial recovery is possible from a range in which it is impossible, using non-rigorous techniques from statistical physics. Some of these threshold effects have been proven [MNS14, Mas14], along with analogues for exact recovery [ABH16, AS15a], and among the first algorithms to achieve exact recovery all the way up to these thresholds were those based on the powerful algorithmic tools of semidefinite programming (SDP). Indeed, the maximum likelihood estimation problems for block models amount to cut problems on graphs, which have a rich history of study through SDPs (for example [GW95, FJ94]); it is natural for these to appear among the most powerful tools for community detection.

In recent work of Abbe and Sandon [AS15a], alternative methods of spectral clustering and local refinement have taken the lead in exact recovery, providing very general and sharp algorithmic results. It is natural to wonder how well SDP approaches can match these results, particularly in the light of strong robustness properties that convex programs enjoy. In this paper, we extend the frontier of optimal results on SDPs for exact recovery to the case of multiple communities of unequal sizes. We further address the relationship of SDPs to semirandom models, highlighting their robustness as an advantage over spectral techniques, but also exposing how this same robustness can pose fundamental limits to SDP approaches.

2 Models

The stochastic block model is a generative model for graphs, in which we suppose a vertex set of size $n$ has been partitioned into $r$ disjoint ‘communities’ $S_{1},\ldots,S_{r}$. An undirected graph is randomly generated from this hidden partition as follows: every unordered pair $(u,v)$ of distinct vertices is independently connected by an edge with probability $Q_{ij}$ where $u,v$ belong to communities $i,j$ respectively. Here $Q$ is a symmetric $r\times r$ matrix. Given a graph sampled from this model, the goal is to recover the underlying partition.

Many papers specialize to the planted partition model, the case where $Q_{ii}=p$ for all $i$ and $Q_{ij}=q$ for all $i\neq j$. We will largely work with this specialization, and occasionally discuss the more general model.

2.2 Semirandom models

Much work in community detection focuses on the Erdős–Rényi-style random models defined above. These models are mathematically convenient, owing to the independence of edges, yet most real-world networks do not take this form. Moreover, there is cause for concern that some algorithms developed for the models above are highly brittle and do not generalize to other graph models: as illustrated in [RJM16], one spectral method with strong theoretical guarantees degrades very rapidly when a sparse planted partition model is perturbed by adding small cliques, which are otherwise very infrequent in this random model, but do occur frequently in many real-world graphs.

Extending beyond the random models above, Blum and Spencer [BS95] introduced the notion of semirandom models for graph problems, as an intermediate model between average-case and worst-case performance. The idea, extended by Feige and Kilian [FK01], is to generate a preliminary graph according to a random model, but then allow a monotone adversary to make unbounded, arbitrarily structured changes of a nature that should only help the algorithm by further revealing the ground truth. Although these changes may seem to make the problem easier, they may significantly alter the distribution of observations revealed to the algorithm, breaking statistical assumptions made about the observations. Semirandom models are thus a means of penalizing brittle algorithms that are over-tuned to particular random models. Semirandom models are no easier than their random counterparts, as the adversary may opt to make no changes.

Following [FK01], we define the semirandom planted partition model as follows. A graph is first generated according to the planted partition model, and then a monotone adversary may arbitrarily add edges within communities, and remove edges between communities. The semirandom model can simulate aspects of real-world graphs and other graph models, such as a wide range of degree or subgraph profiles, while ensuring that the true community structure remains present in the graph. Thus the semirandom model aims to capture the unpredictable nature of real data.

An algorithm is called robust to monotone adversaries if, whenever it succeeds on a sample from the random model, it also succeeds after arbitrary monotone changes to that sample. Algorithms based on semidefinite programming (SDP) are typically robust, or can be modified to be robust, and essentially all known robust algorithms are based on convex programs such as SDPs. This property guarantees some ability of such algorithms to generalize to other models; by contrast, algorithms that over-exploit precise statistics of a random model will typically fail against similar random models with different statistics, and will also typically fail against a semirandom model.

2.3 Regimes

The stochastic block model admits two major regimes of study, along with other variants. The main distinction is between partial recovery, in which the goal is only to recover a partition that is reliably correlated with the true partition better than random guessing, versus exact recovery, in which the partition must be recovered perfectly. In partial recovery, one tends to take the within-community edge probability $p$ and the between-community edge probability $q$ to be $\Theta(1/n)$, whereas in exact recovery one takes them to be $\Theta(\log n/n)$. In these asymptotic regimes one observes a sharp threshold behavior: within some range of parameters $p$ and $q$, the problem is statistically impossible, and outside of that range one can find algorithms that succeed with high probability. For partial recovery, this is established in [MNS14, MNS13, Mas14], and for exact recovery, the most general result on this threshold is established in [AS15a].

We specialize to the assortative planted partition model, where $p>q$; some techniques may transfer to the dissortative $p<q$ case, by judicious negations, but we do not elaborate on this.

3 Contributions and prior work

The main result of this paper is that two variants of a certain SDP achieve exact recovery, up to the information-theoretic threshold, against the planted partition model with multiple communities of potentially different sizes. These SDPs are furthermore robust to monotone adversaries.

There has been considerable prior work on the development of algorithms and lower bounds for the stochastic block model, with algorithms making use of a wide range of techniques; see for instance the introduction to [ABH16] and works cited there. The use of semidefinite programming for exact recovery originated with [FK01], who achieved robust exact recovery in the case of two communities of equal sizes, falling slightly short of the optimal performance threshold. More recently, semidefinite algorithms have been found especially effective on sparse graphs [CSX12, ABH16], and have been proven to match lower bounds for the planted partition model in the case of two equal-sized communities [Ban15, HWX15a], two different-sized communities [HWX15b], and multiple equal-sized communities [HWX15b, ABKK15], but the case of recovering multiple communities of different sizes through SDPs remained unresolved until now. The SDP that we consider in this paper has appeared before in the literature [GV15, ABKK15], and in fact Agarwal et al. [ABKK15] conjectured that it achieves exact recovery up to the threshold; the main result of this paper resolves this conjecture affirmatively.

Abbe and Sandon [AS15a] established the information-theoretic threshold for the general stochastic block model with individual community-pair probabilities $Q_{ij}$, which we visit in Section 3. In addition to proving a sharp lower bound, they analyzed an algorithm that succeeds with high probability up to their lower bound, thus precisely determining the statistical limits for exact recovery. Their result may appear strictly more general than ours, applying to the general stochastic block model rather than the more specific planted partition model. However, their algorithm involves a highly-tuned spectral clustering step that is likely not robust to monotone adversaries or other forms of perturbation (see Section 4 for discussion), and so our work can be seen as an improvement from the robustness standpoint.

In fact, there are barriers against more general semirandom results. We show in Section 4 that no algorithm robust to monotone adversaries can handle the case of fully unknown parameters, nor can it extend from the planted partition model to the slightly more general strongly assortative block model. Thus our SDP-based approach is already essentially as general as one could hope for without compromising the strength of its robustness.

Although we are primarily interested in the exact recovery regime, we will take a brief detour and survey the parallel line of work on partial recovery. In contrast to exact recovery, none of the partial recovery algorithms that succeed down to the information-theoretic threshold are based on SDPs, and they have no robustness guarantees. There are robust SDP methods known [GV15, MPW16, MMV16] but they fall short of the threshold; this is essential, as it has been shown that no algorithm can robustly achieve the partial recovery threshold [MPW16]. Although robust algorithms for partial recovery cannot reach the threshold, there is an SDP that is conjectured to come quite close [JMR16] and is known to achieve the threshold in the limit of large average degree [MS16]; however, this analysis does not come with a robustness guarantee. By contrast, this paper shows that robust recovery is efficiently achievable up to the threshold in the exact recovery setting.

Community detection in semirandom models was previously discussed in [FK01, CSX14, AL14], and further since the first appearance of this paper, in [MPW16, MMV16, HWX15a]. Other works [CL15, MMV16] discuss robustness to some quantity of corruptions or outlier nodes.

4 Overview of techniques

In this section we summarize the arguments by which we prove that our SDPs achieve exact recovery against the semirandom planted partition model. More precisely, we show that with high probability, the unique SDP optimum is a matrix that exactly encodes the correct community structure; in contrast to some SDP-based algorithms (both for community detection and otherwise), there is no ‘rounding’ or post-processing procedure required.

The first step is to show robustness in the sense that if the SDP succeeds against a particular graph, then it will continue to succeed if that graph is modified by monotone changes. This follows from a simple argument in [FK01] which essentially observes that the SDP optimizes over a space of solutions, and monotone changes improve the objective value of the true solution more than they improve the objective value of any other solution.

In light of the above argument, it suffices to show that our SDP succeeds with high probability against the random model. As in previous work on SDPs for exact recovery, our proof proceeds by constructing a dual certificate. The idea here is that the SDP is a maximization problem for which the true solution is feasible; what we need to show is that no other solution has a larger objective value than the true one, which can be done by finding a matching solution to the dual SDP. This “dual certificate” that we construct depends on the random graph and can be shown to be dual-feasible with high probability. As is typical, the construction of the dual certificate is guided by complementary slackness, which provides a set of necessary conditions. However, these necessary conditions are not enough to uniquely determine the dual certificate and so some creativity is necessary here in order to find an optimal certificate that gets all the way to the threshold. Part of what makes the general problem harder than the special cases considered previously [HWX15b] is that the general case has more dual variables that are not automatically determined by complementary slackness but that nonetheless need to be chosen carefully. The crucial step in the construction of our dual certificate is that by making a change of variables we are able to find a connection between the complementary slackness conditions and the non-negativity of differences of certain binomial random variables, which in turn are closely related to the information-theoretic threshold. It then becomes clear which parts of the dual solution need to crucially be set a particular way, and which ones have “wiggle room.” As is typical, showing that our dual matrix is positive semidefinite relies in part on the spectral concentration of the adjacency matrix, e.g. Theorem 5.2 in [LR15].

5 Organization of this paper

In Section 2 we present our two closely-related semidefinite programs for exact recovery. In Section 3 we discuss the information-theoretic threshold determined in [AS15a]. In Section 4 we show that our SDPs are robust to the semirandom model, and we give some impossibility results for robust algorithms. In Section 5 we prove our main result: our SDPs achieve exact recovery up to the information-theoretic threshold.

Semidefinite algorithms and results

In this section we derive semidefinite programs for exact recovery, by taking convex relaxations of maximum likelihood estimators. Throughout we will use the letters $u,v$ for vertices and the letters $i,j$ for communities.

Given an observed $n$-vertex graph, a natural statistical approach to recovering the community structure is to compute a maximum-likelihood estimate (MLE). We begin by stating the log-likelihood of a candidate partition into communities:

Here $\sim$ denotes adjacency in the observed graph. We can represent a partition by its $n\times n$ partition matrix

In terms of $X$ and the observed $(0,1)$-adjacency matrix $A$, we can write

where $I$ is the identity matrix, $J$ is the all-ones matrix, and $\langle\cdot,\cdot\rangle$ denotes the Frobenius (entry-wise) inner product of matrices. Expanding, and discarding terms that do not depend on $X$, including $\langle X,I\rangle=n$:

The assumption $p>q$ implies that $\alpha$ and $\beta$ are positive.

At this stage it is worth distinguishing the two cases of known and unknown sizes. In the first form, the block sizes $s_{i}$ are known; then the second term above is a constant, and the MLE amounts to a minimum fixed-sizes multisection problem, and is NP-hard for worst-case $A$.

In the second form of the problem, the block sizes $s_{i}$ are unknown (but $r$ is known). Now the MLE requires knowledge of $p$ and $q$, and the resulting regularized minimum multisection problem is also likely computationally hard in general.

Although it is not obvious from the definition, we can think of $\omega$ as a sort of average of $p$ and $q$; the proof is deferred to Appendix A:

In order to compute either MLE in polynomial time, we will pass to a relaxation and show that the computation succeeds with high probability.

2 Semidefinite algorithms

The seminal work of [GW95] began a successful history of semidefinite relaxations for cut problems. Proceeding roughly in this vein to write a convex relaxation for the true feasible region of partition matrices of given sizes, one might reasonably arrive at the following relaxation of the known-sizes MLE:

Here $X\geq 0$ indicates that $X$ is entry-wise nonnegative, and $X\succeq 0$ indicates that $X$ is positive semidefinite (and, in particular, symmetric).

This SDP appears in [AL14] (under the name SDP-2’). Under the assumption of equal-sized communities, a stronger form involving row sum constraints appears in [AL14, HWX15b, ABKK15], and the latter two papers find that this strengthening achieves exact recovery up to the information-theoretic lower bound in the case of equal-sized communities.

In the case of unequal community sizes, it is more difficult to pursue this line of strengthening. The authors have analyzed the weaker SDP above for multiple unbalanced communities, and found that it does achieve exact recovery within some parameter range in the logarithmic regime, but its threshold for exact recovery is strictly worse than the information-theoretic threshold. (We have opted to omit these proofs from this paper, in view of our stronger results on the programs below.)

Instead, we revisit the somewhat arbitrary decision to encode the partition matrix with entries $0$ and $1$. Indeed, SDPs for the unbalanced two-community case tend to use entries $-1$ and $1$ [FK01, ABH16, HWX15b], with success up to the information-theoretic lower bound [HWX15b]. Some choices of entry values will result in a non-PSD partition matrix, so we opt for the choice of entries for which the partition matrix is only barely PSD: namely, we define the centered partition matrix

Aiming to recover this matrix, we reformulate our SDP as follows:

This SDP bears a strong similarity to classical SDPs for maximum $r$-cut [FJ94], and [ABKK15] conjectured that it achieves exact recovery for unbalanced multisection up to the information-theoretic lower bound. Our main result resolves this conjecture affirmatively.

Other than the natural motivation of following classical approaches to $r$-cut problems, the change from partition matrices to centered partition matrices buys us something mathematically concrete: the intended primal solution now has rank $r-1$ instead of rank $r$, which through complementary slackness will entail one less constraint on a candidate dual optimum.

We can write down a very similar relaxation for the MLE in the case of unknown sizes but known $p$ and $q$:

Here $\omega$ is as defined in (1). Our main assertion is that these SDPs achieve exact recovery up to the lower bounds in [AS15a]:

To reiterate, the known sizes SDP (Program 4) requires knowledge of the sum of squares of community sizes (equivalently $\sum_{i}\pi_{i}^{2}$) and the number $r$ of communities, but $p$ and $q$ can be unknown; the unknown sizes SDP (Program 5) requires knowledge of $p$, $q$, and $r$ (or at least $\omega$ and $r$), but the community proportions $\pi$ can be unknown.

Program 5 can tolerate some mis-specification in the value of $\omega$, though this tolerance may shrink to zero as the problem parameters approach the information-theoretic threshold. This tolerance is implicit in the proof of Theorem 2.

Lower bounds

In this section we visit the general information-theoretic lower bounds established by [AS15a], and we specialize them to the planted partition model, recovering bounds that directly generalize those of [HWX15b] for the case of two unequal communities. Meanwhile, we recall operational interpretations of the lower bounds, which will be our main concrete handle on them.

For $i\neq j$ define the Chernoff–Hellinger divergence

Then exact recovery is information-theoretically solvable if

Within this range, there exist efficient algorithms that succeed with probability $1-o(1)$.

Conversely, exact recovery is impossible if there exists a pair $(i,j)$ with $D_{+}(i,j)<1$. Within this range, any estimator for the community structure (even knowing the model parameters) will fail with probability $1-o(1)$.

[AS15a] also shows that the borderline case $D_{+}(i,j)=1$ remains solvable; for simplicity we neglect this case.

Note that in our regime (logarithmic average degree), we have

The divergence expression in Proposition 4 strongly resembles the lower bound proven in [HWX15b] for the case of two communities of different sizes. Indeed, in the notation of Lemma 2 of [HWX15b], we recognize our expression as

From that lemma, the following operational definition of the CH-divergence is immediate for the planted partition model:

Let $i\neq j$ be communities, and let $v$ be a vertex in community $i$. Let $E(v,j)$ denote the number of edges from $v$ to vertices in community $j$. Suppose that $T(n)=\tau(\pi_{i}-\pi_{j})\log n+o(\log n)$. The probability of the tail event $E(v,i)-E(v,j)\leq T(n)$ is $n^{-D_{+}(i,j)+o(1)}$.

By a naive union bound, when $D_{+}(i,j)>1$ for all pairs $i,j$, then we can assert with high probability that none of these tail events occur, over all vertices and communities.

A similar operational interpretation is given directly in [AS15a], phrased in terms of hypothesis testing between multivariate Poisson distributions. This result keeps more complete track of the $o(1)$ term, so as to guarantee the union bound even when $D_{+}(i,j)=1$.

Lastly we note the following monotonicity property of the divergence, with a proof deferred to Appendix C:

Thus, when determining whether exact recovery is feasible in the planted partition model for some set of parameters, it suffices to check the CH-divergence between the two smallest communities.

Semirandom robustness and its consequences

Let $\hat{X}$ be the ground truth partition of the vertices into communities.

Following [FK01] we define a monotone adversary to be a process which takes as input a graph (for instance, a random graph sampled from the stochastic block model with ground truth $\hat{X}$) and makes any number of monotone changes of the following types:

The adversary can add an edge between two vertices in the same community of $\hat{X}$.

The adversary can remove an edge between two vertices in different communities of $\hat{X}$.

These monotone changes appear to only strengthen the presence of the true community structure $\hat{X}$ in the observed graph, yet they may destroy statistical properties of the random model. The semirandom model is designed to penalize brittle algorithms that over-rely on specific stochastic models. It does not intend to mimic any real-world adversarial scenario, but it does intend to model the inherent unpredictability of real-world data.

It may help to consider examples of how such an adversary could break an algorithm:

Many algorithms perform PCA on the adjacency matrix [LR15]. The adversary could plant a slightly denser sub-community structure in a community, thus splitting one cluster of vertices into several nearby sub-clusters in the PCA, and introducing doubt as to which granularity of clustering is appropriate.

An adversary could introduce a noise distribution that changes the shape of clusters of vertices in the PCA or spreads them out, resulting in either a failure to cluster correctly, or else a failure in subsequent steps of estimating parameters and improving the community structure (as in [AS15b]).

These are extreme examples, but they correspond to realistic concerns:

Real community structure is sometimes hierarchical, e.g. tight friend groups within a larger social community.

Many real networks have hubs, or a degree distribution that is nowhere near Gaussian, so hypothesis tests designed for distributions of roughly Gaussian shape may have trouble generalizing.

2 Robustness

In this section, we establish that our SDPs are robust to monotone adversaries. We first elaborate on the definitions discussed in the introduction.

Suppose $f$ is a (deterministic) algorithm for recovery in the stochastic block model, namely $f$ takes in an adjacency matrix $A$ and outputs a partition $f(A)$ of the vertices. We say $f$ is robust to monotone adversaries if: for any $A$ such that $f(A)=\hat{X}$, we have $f(A^{\prime})=\hat{X}$ for any $A^{\prime}$ obtained from $A$ via a sequence of monotone changes.

We modify this definition slightly for SDPs in order to deal with the fact that an SDP may not have a unique optimum. By abuse of notation, let $\hat{X}$ also refer to the centered partition matrix corresponding to the partition $\hat{X}$.

Suppose $P_{A}$ is a semidefinite program which depends on the adjacency matrix $A$ (for instance, Program 4 or Program 5). We say $P_{A}$ is robust to monotone adversaries if: for any $A$ such that $\hat{X}$ is the unique optimum to $P_{A}$, we have that $\hat{X}$ is the unique optimum to $P_{A^{\prime}}$ for any $A^{\prime}$ obtained from $A$ via a sequence of monotone changes.

SDPs tend to possess such robustness properties. We will now show that our SDPs are no exception, following roughly the same type of argument as [FK01].

Programs 4 (known sizes) and 5 (unknown sizes) are robust to monotone adversaries.

Let $P_{A}$ be either Program 4 or Program 5 (the proof is identical for both cases). Suppose the true centered partition matrix $\hat{X}$ is the unique optimum for $P_{A}$. By induction it is sufficient to show that $\hat{X}$ is the unique optimum for $P_{A^{\prime}}$ where $A^{\prime}$ is obtained from $A$ via a single monotone change. Note that $P_{A}$ and $P_{A^{\prime}}$ have the same feasible region because $A$ only affects the objective function. Let $P_{A}(X)$ denote the objective value of a candidate solution $X$ for $P_{A}$, namely $P_{A}(X)$ is $\langle A,X\rangle$ for Program 4 and $\langle A,X\rangle-\omega\langle J,X\rangle$ for Program 5. First consider the case where $A^{\prime}$ is obtained from $A$ via a single monotone edge-addition step. Since the added edge lies within a community of $\hat{X}$ we have $P_{A^{\prime}}(\hat{X})=P_{A}(\hat{X})+2$. For any matrix $X$ feasible for $P_{A}$ (equivalently, feasible for $P_{A^{\prime}}$), we have $P_{A^{\prime}}(X)\leq P_{A}(X)+2$; this follows from $X\leq 1$ (entry-wise), which is implied by the constraints $X_{vv}=1$ and $X\succeq 0$. If $X\neq\hat{X}$ we have $P_{A^{\prime}}(\hat{X})=P_{A}(\hat{X})+2>P_{A}(X)+2\geq P_{A^{\prime}}(X)$ and so $\hat{X}$ is the unique optimum of $P_{A^{\prime}}$. Similarly, for the case where $A^{\prime}$ is obtained from $A$ via a single monotone edge-removal, we have $P_{A^{\prime}}(\hat{X})=P_{A}(\hat{X})+\frac{2}{r-1}$ and $P_{A^{\prime}}(X)\leq P_{A}(X)+\frac{2}{r-1}$ (using the constraint $X\geq\frac{-1}{r-1}$) and the result follows. ∎

Recall that in the semirandom planted partition model, a random graph is generated according to the random (planted partition) model and then a monotone adversary is allowed to make monotone changes. Once we have established our main result (Theorem 2) on the success of our SDPs against the random model, it is an immediate corollary of robustness that our SDPs also succeed against the semirandom model.

Programs 4 (known sizes) and 5 (unknown sizes) achieve exact recovery against the semirandom planted partition model, with probability $1-o(1)$, up to the information-theoretic threshold for the random model (given in Theorem 3).

3 BM-ordering and strongly assortative block models

Programs 4 and 5 achieve exact recovery with high probability against any strongly assortative block model that dominates a planted partition model lying within the information-theoretically feasible range.

4 Difficulties with general block models

Most natural SDPs tend to be robust to monotone adversaries. This strength of semidefinite approaches – their ability to adapt to other random models following BM-ordering – can be used to also reveal their limitations. We will show in this section that it is impossible for an algorithm robust to monotone changes to match the information-theoretic lower bound of [AS15a] in general, and even for strongly assortative block models.

This example shows how monotone changes become subtly unhelpful in the strongly assortative block model: the first two communities become harder to distinguish under these monotone changes because their interactions with the third community become more similar. Arguably this makes the semirandom model inappropriate for such a general block model. Nonetheless, monotone robustness is a property of our SDPs, and also of all prior SDPs in the community detection literature (at least after minor strengthening), and so the limitations below apply at least to these specific SDPs. Thus we are able to learn about the limitations of these SDPs by studying their robustness properties.

The argument above applies to any algorithm that is robust to the semirandom model; this means no robust algorithm can achieve the threshold. This motivates us to conjecture a different “monotone threshold” for general block models, which we believe captures the information-theoretic limits in the general semirandom model. Define the monotone divergence

Note that $D_{+}^{m}(i,j)$ is simply the value of $D_{+}(i,j)$ after setting $Q_{ik}=Q_{jk}$ for all $k\notin\{i,j\}$; this is a change in model that the monotone adversary can simulate (for instance set $Q_{ik}=Q_{jk}=0$), and it is in fact the best change-in-model that the adversary can simulate if it wants to decrease $D_{+}(i,j)$ as much as possible for a specific $(i,j)$ pair. It follows that if $D_{+}^{m}(i,j)<1$ for some $i\neq j$ then there does not exist a robust algorithm achieving exact recovery. We conjecture that conversely, if $D_{+}^{m}(i,j)\geq 1$ for all $i\neq j$, and if the block model is furthermore weakly assortative ($Q_{ii}>Q_{ij}$ for all $i\neq j$), then there exists a robust algorithm achieving exact recovery against this block model.

5 Difficulties with unknown parameters

Non-semidefinite techniques in [AS15b] achieve exact recovery up to the threshold without knowing any of the model parameters. One might ask whether it is possible for a robust algorithm (such as our SDPs) to achieve this; we now argue that this is not possible in general even in the planted partition model.

In effect, these two planted partition models have zero “monotone total variation distance”, though we do not formalize this notion here. It is necessary to know some model parameters in advance in order for robust algorithms to distinguish such models. A few approaches are available to overcome this drawback:

One could statistically estimate some or all of the parameters before running the SDP, as in Appendix B of [HWX15b]. However, this statistical approach relies on the specific random model and spoils our robustness guarantees.

One could try running the SDP several times on a range of possible input parameters, ignoring any returned solutions that are not partition matrices. A close reading of Section 5 reveals that, when running Program 5 (unknown sizes), mis-guessing the parameter $\omega$ by any $1-o(1)$ factor does not affect whether one succeeds with high probability.

This approach may return several valid solutions. In the example above, this approach will recover both of the given planted partition models, with high probability. In general this approach recovers the type of hierarchical community structure that the above example exhibits.

Proof of exact recovery

In this section we prove our main result (Theorem 2) which states that our SDPs achieve exact recovery against the planted partition model, up to the information-theoretic limit. Specifically, we show that if the divergence condition (3) holds, then with high probability, the true centered partition matrix $\hat{X}$ is the unique optimum for our SDPs. The main idea of the proof is to construct a solution to the dual SDP in order to bound the value of the primal.

2 Weak duality

Following the standard dual certificate approach, we begin by writing down duality and complementary slackness for our SDP. This will lead us to a set of sufficient conditions, outlined in Proposition 10 below.

We first write down the dual of Program 5:

Here the $n$-vector $\nu$ and the $n\times n$ matrix $\Gamma$ (both indexed by vertices) are dual variables, and $\omega$ is as defined in (1). We can now state weak duality in this context:

This implies weak duality $\langle A,X\rangle-\omega\langle J,X\rangle\leq\sum_{v}\nu_{v}+\frac{1}{r-1}\langle J,\Gamma\rangle$ (the primal objective value is at most the dual objective value) because $\langle\Gamma,X+\frac{1}{r-1}J\rangle\geq 0$ (since $\Gamma\geq 0$, $X+\frac{1}{r-1}J\geq 0$) and $\langle\Lambda,X\rangle\geq 0$ (since $\Lambda\succeq 0,X\succeq 0$).

3 Complementary slackness

Although we have only considered Program 5 so far, everything we have done also applies to Program 4. The dual of Program 4 is identical to Program 6, except that $\omega$ is replaced by a dual variable, and there is a corresponding term in the objective. By deterministically choosing this dual variable to take the value $\omega$, we arrive at a dual program with the same feasible region and complementary slackness conditions as Program 6. From this point onward, the same arguments apply to both Programs 4 and 5.

Let $\hat{X}$ be the true centered partition matrix with $(1,\frac{-1}{r-1})$ entries. The following proposition gives a sufficient condition for $\hat{X}$ to be the unique optimum for Programs 4 and 5.

Suppose there exists a dual solution $(\nu,\Gamma)$ satisfying:

$\Gamma_{S_{i}S_{j}}>0$ (entry-wise) for all $i\neq j$,

Then $\hat{X}$ is the unique optimum for Programs 4 and 5. (Here $\Lambda$ is defined as $\Lambda=\diag(\nu)+\omega J-A-\Gamma$ as in Program 6.)

We note that Proposition 10 is not novel in that all the arguments we have made so far are standard in the dual certificate approach.

4 Construction of dual certificate – overview

We now explore the space of dual certificates that will satisfy the conditions of Proposition 10, so as to sound out how to construct such a certificate. The main result of this section is to rewrite the problem in terms of a new set of variables $\gamma_{u}$. We believe this change of variables is novel, and it is crucial to our approach because it will allow us to make a connection between between complementary slackness and certain differences of binomial variables that are closely related to the information-theoretic threshold (see Lemma 5).

Using the definition $\Lambda=\diag(\nu)+\omega J-A-\Gamma$, this can be rewritten as the two equations

We can disregard the equations (6) because they are implied by the equations (5) via subtraction. From (5) we have that, for any fixed $u\in S_{i}$, the quantity $\omega s_{j}-E(u,j)-\sum_{v\in S_{j}}\Gamma_{uv}$ must be independent of $j$ (for $j\neq i$). Hence let us define

where $R_{uj}$ is shorthand for the row sum $\sum_{v\in S_{j}}\Gamma_{uv}$. Since $\Gamma$ is symmetric, $R_{uj}$ must be equal to the column sum $\sum_{v\in S_{j}}\Gamma_{vu}$, so we need for any $i\neq j$,

or equivalently, there needs to exists a constant $c$ such that

5 Intervals for γv\gamma_{v}

In this section we find necessary bounds for $\gamma_{v}$, which will guide our choice of these dual variables and of $c$. This is where the crucial connection between $\gamma_{v}$ and the information-theoretic threshold will become apparent. Let $v\in S_{i}$. For a lower bound on $\gamma_{v}$, we have that $\Lambda\succeq 0$ implies $\Lambda_{vv}\geq 0$ implies $\nu_{v}+\omega\geq 0$ which by (8) implies $\gamma_{v}\geq\omega(s_{i}-1)-E(v,i)$. For an upper bound, for any $j\neq i$ we must have that $\Gamma_{S_{i}S_{j}}>0$ implies $R_{vj}>0$ which by (9) implies $\gamma_{v}<\omega s_{j}-E(v,j)$. Therefore, $\gamma_{v}$ must lie in the interval

Our approach in choosing $\gamma_{v}$ will be to first make a preliminary guess $\gamma^{\prime}_{v}$ and then add an adjustment term to ensure that (10) holds. In order to absorb this adjustment, we will aim for $\gamma^{\prime}_{v}$ to lie in the slightly smaller interval

Here $\epsilon_{1}$ and $\epsilon_{2}$ are small $o(\log n)$ error terms which we will choose later.

The non-emptiness of these intervals is the crux of the proof, and provides the connection to the information-theoretic threshold:

If the divergence condition (3) holds, then $\alpha_{v}<\beta_{v}$, for all $v$, with high probability.

By Lemma 5, for each $v\in S_{i}$ and all $j\neq i$, the probability of the tail event

is $n^{-D_{+}(i,j)+o(1)}$. Thus, when $D_{+}(i,j)>1$ for all pairs $(i,j)$, we can take a union bound over all $n(r-1)$ such events, to find that $\beta_{v}-\alpha_{v}\geq 0$ with probability $1-o(1)$. ∎

By summing (11) over all $v\in S_{i}$ (roughly following (10), although $\gamma_{v}$ and $\gamma_{v}^{\prime}$ are slightly different), we obtain a target interval for $c$:

The endpoints of the interval (12) for $c$ will turn out to be highly concentrated near a pair of deterministic quantities, namely:

6 Choice of cc and γv\gamma_{v}

We have now made the key insight that $c$ and $\gamma_{v}$ must lie in certain intervals in order to get all the way to the information-theoretic threshold. However, as long as we fulfill these requirements, we have some “wiggle room” in choosing the dual variables. We will make what seems to be the simplest choices. We can deterministically take

using the definitions (13) of $\bar{\alpha}_{i},\bar{\beta}_{i}$ along with the facts $\epsilon_{1},\epsilon_{2}=o(\log n)$ and $q<\omega<p$ (Lemma 1). Our specific choice of $c$ is not crucial; we can in fact pick any deterministic $0<c<\bar{\beta}_{i}$ provided that $c=\Theta(n\log n)$ and $\bar{\beta}_{i}-c=\Theta(n\log n)$ for all $i$.

Recall that our goal is to choose each $\gamma_{v}$ to lie in (or close to) the interval $[\alpha_{v},\beta_{v}]$ subject to the condition $\sum_{v\in S_{i}}\gamma_{v}=c$ required by (10). To achieve this, define the deterministic quantity

Note that we expect $c$ to lie roughly $\kappa_{i}$ fraction of the way through the interval $[\alpha_{i},\beta_{i}]$ (which is the sum over $v\in S_{i}$ of the intervals $[\alpha_{v},\beta_{v}]$). Mirroring this, we make a rough initial choice $\gamma_{v}^{\prime}$ (for $v\in S_{i}$) that is $\kappa_{i}$ fraction of the way through the interval $[\alpha_{v},\beta_{v}]$:

However, these do not satisfy $\sum_{v\in S_{i}}\gamma_{v}^{\prime}=c$ on the nose – rather, there is some error that is on the order of the difference between $\alpha_{i}$ and $\bar{\alpha}_{i}$. We thus introduce an additive correction term $\delta_{i}$ chosen to guarantee $\sum_{v\in S_{i}}\gamma_{v}=c$. For $v\in S_{i}$,

Recall that our goal was for $\gamma_{v}$ to lie within some $o(\log n)$ error from the interval $[\alpha_{v},\beta_{v}]$. By construction we have $\gamma_{v}^{\prime}\in[\alpha_{v},\beta_{v}]$ and so we will have succeeded if we can show $\delta_{i}=o(\log n)$. This will be one of the goals of the next section.

7 High-probability bounds for random variables

In this section we establish bounds on various variables in the dual certificate that will hold with high probability $1-o(1)$. We can treat the failure of these bounds as a low-probability failure event for the algorithm.

First recall the following version of the Bernstein inequality:

If $X_{1},\ldots,X_{k}$ are independent zero-mean random variables with $|X_{i}|\leq 1$, then for any $t>0$,

Note that by replacing $X_{i}$ with $-X_{i}$ we get the same bound for $\Pr\left[\,\sum_{i}X_{i}\leq-t\,\right]$.

Our motivation for defining the quantity $\Delta_{v}$ is its appearance in the following bounds:

where the third bound makes use of the first two.

Taking $t=\log n\log\log n$ and union bounding over all $v$, we see that, with high probability, $\Delta_{v}\leq\log n\log\log n$ for all $v$. But this will not quite suffice for the bounds we need. Instead, taking $t=\log n/(\log\log n)^{2}$, we see that $\Delta_{v}\leq\log n/(\log\log n)^{2}$ for most values of $v$, with a number of exceptions that, with high probability, does not exceed $n^{1-1/(\log\log n)^{5}}$; for these exceptions, we fall back to the bound of $\log n\log\log n$ above. Above we have used the following consequence of Markov’s inequality: if there are $n$ bad events, each occurring with probability $\leq p$, then $\Pr[\text{at least }k\text{ bad events occur}]\leq\frac{np}{k}$.

For the sake of quickly abstracting away this two-tiered complication, we make the following three computations up front:

where the sums range over all vertices $u$,$v$. Note that in each case, the non-exceptional vertices dominate the bound. (One can easily compare two terms in the above calculations by computing the logarithm of each.)

Now we can show that $\delta_{i}$, the correction term from the previous section, is small. For any $i$ we have

We will be interested in defining the quantity $\Delta^{\prime}_{v}=\Delta_{v}+|\delta_{i}|$ (where $v\in S_{i}$) due to its appearance in the following bounds:

Using the identity $(x+y)^{2}\leq 2(x^{2}+y^{2})$ along with (20), (21) and (22), we have with high probability

8 Bounds on νv\nu_{v} and Rv​jR_{vj}

We can now prove two key results that we will need later: with high probability,

These results should not come as a surprise because they were more or less the motivation for defining the interval $[\alpha_{v},\beta_{v}]$ for $\gamma_{v}$ in the first place. Since the $\nu_{v}$ values lie on the diagonal of $\Lambda$, the bound on $\nu_{v}$ is important for proving $\Lambda\succeq 0$ which we need for dual feasibility. Since $R_{vj}$ are the row sums of $\Gamma$, the bound on $R_{vj}$ is important for achieving $\Gamma_{S_{i}S_{j}}>0$ which we need for Proposition 10. The specific quantity $\frac{\log n}{\log\log n}$ is not critical – anything would suffice that is $o(\log n)$ yet large enough to dominate some error terms in a later calculation.

In the previous section we showed (22) we showed $|\delta_{i}|=o(\log n)$ with high probability, and so we can choose the error terms $\epsilon_{1},\epsilon_{2}$ from the definition of $\alpha_{v},\beta_{v}$ (which, recall, are required to be $o(\log n)$) to absorb $\delta_{i}$. Specifically, let

Recall that by Lemma 11 the intervals $[\alpha_{v},\beta_{v}]$ are all nonempty with high probability, and by construction we have $\gamma^{\prime}_{v}\in[\alpha_{v},\beta_{v}]$.

Now we will show (27): $\nu_{v}\geq\frac{\log n}{\log\log n}$. For $v\in S_{i}$ we have

Now we show (28): $R_{vj}>0$. For $v\in S_{i}$ and $j\neq i$ we have

9 Choice of Γ\Gamma

We have shown how to choose strictly positive row sums $R_{uj}$ and column sums $R_{vi}$ of $\Gamma_{S_{i}S_{j}}$ (for $i\neq j$). There is still considerable freedom in choosing the individual entries, but we will make what seems to be the simplest choice: we take $\Gamma_{S_{i}S_{j}}$ to be the unique rank-one matrix satisfying these row and column sums, namely

where $T_{ij}$ is the total sum of all entries of $\Gamma_{S_{i}S_{j}}$,

We showed earlier that this last equality is guaranteed by (10). As the row sums $R_{uj}$ are all positive with high probability (28), it follows that $\Gamma_{S_{i}S_{j}}>0$.

10 PSD calculation for Λ\Lambda

We will bound the three terms in (30) separately. In particular, we will show that (with high probability):

Once we have this, we can (for sufficiently large $n$) rewrite (30) as

for some positive constants $C_{1},C_{2},C_{3}$. For sufficiently large $n$ we have

where (a) expands $\sum_{i,j}T_{ij}$ using (29),(9),(10), and (b) expands $\sum_{v\in S_{i}}\nu_{v}$ using (8),(10). Therefore

Since in (14) we chose $c$ to be $\Theta(n\log n)$, we have $y^{\top}\Lambda y=\Theta(\log n)$ as desired.

10.2 Lower bound for z⊤​Λ​yz^{\top}\Lambda y

Let $(\Lambda y)^{Z}$ denote the projection of the vector $\Lambda y$ onto the subspace $Z$. For $v\in S_{i}$ we have

10.3 Lower bound for z⊤​Λ​zz^{\top}\Lambda z

Recall that $\Gamma_{S_{i}S_{j}}$ has row sums $R_{uj}$ and total sum (of all entries)

By applying Bernstein’s inequality (Lemma 12) to $E(i,j)$ we get a high-probability bound for $T_{ij}$:

where $u\in S_{i}$, $v\in S_{j}$, $i\neq j$.

where in step (a), we appeal to the bounds (24), (33), causing cancellations in the high-order terms; and in (b) we have used the choice of $c$ (14) to check that the denominator is $\Theta(n^{3}\log n)$.

where (a) uses the triangle inequality for the Euclidean norm and (b) uses the high-probability bounds (25),(26) for expressions involving $\Delta^{\prime}_{v}$.

Acknowledgements

The authors are indebted to Ankur Moitra for suggesting the problem, for providing guidance throughout the project, and for several enlightening discussions on semirandom models. We would also like to thank David Rolnick for a helpful discussion on MLEs for the block model, and Roxane Sayde for comments on a draft of this paper.

References

Appendix A Proof of Lemma 1

In this appendix we verify, for all $0<q<p<1$, that $q<\omega<p$, where

The proof is an elementary computation using the bound: $\frac{x-1}{x}\leq\log x\leq x-1$ for all $x>0$, with both inequalities strict unless $x=1$.

For the lower bound, we proceed as follows:

and for the upper bound, we proceed similarly:

so that the result follows by taking reciprocals.

Appendix B Proof of Proposition 4

In this appendix, we establish a closed form for the CH-divergence in the planted partition model.

The CH-divergence is defined in [AS15a] as

where $\tau$ is as defined in Proposition 4.

As the supremand is smooth and concave in $u$, we can set the derivative in $u$ equal to zero in order to maximize it:

which is quadratic in $e^{u}$, and can be solved via the quadratic formula:

To obtain the simplest possible form for $D_{+}$, it is worth also solving for $e^{-u}$:

Dividing these two expressions, we obtain

Substituting back into the divergence, we obtain

Appendix C Proof of Proposition 6

As $\eta$ is smooth, we will show that $\frac{\partial}{\partial s}\eta\geq 0$. We will show this, in turn, by showing that $\lim_{s\to 0}\frac{\partial}{\partial s}\eta\geq 0$ and that $\frac{\partial^{2}}{\partial s^{2}}\eta\geq 0$.

To see that the second partial is non-negative, it suffices to see that the numerator is non-negative:

Both sides are non-negative, so it is equivalent to compare their squares:

To see that the limit $\lim_{s\to 0}\frac{\partial}{\partial s}\eta$ is non-negative, we first compute it: