Center-based Clustering under Perturbation Stability

Introduction

Problems of clustering data arise in a wide range of different areas – clustering proteins by function, clustering documents by topic, and clustering images by who or what is in them, just to name a few. In this paper we focus on the popular class of center based clustering objectives, such as $k$-median, $k$-center and $k$-means. Under these objectives we not only partition the data into $k$ subsets, but we also assign $k$ special points, called the centers, one in each cluster. The quality of a solution is then measured as a function of the distances between the data points and their centers. For example, in the $k$-median objective, the goal is to minimize the sum of distances of all points from their nearest center, and in the $k$-means objective, we minimize the sum of the same distances squared. As these are NP-hard problems , there has been substantial work on approximation algorithms with both upper and lower bounds on approximability of these and other objective functions. Note that we are especially interested in the case that $k$ is part of the input and not a constant.

Recently, Bilu and Linial , focusing on the Max-Cut problem , proposed considering instances where the optimal clustering is optimal not only under the given metric, but also under any bounded multiplicative perturbation of the given metric. This is motivated by the fact that in practice, distances between data points are typically just the result of some heuristic measure (e.g., edit-distance between strings or Euclidean distance in some feature space) rather than true “semantic distance” between objects. Thus, unless the optimal solution on the given distances is correct by pure luck, it likely is correct on small perturbations of the given distances as well. Bilu and Linial analyze Max-Cut instances of this type and show that for instances that are stable to perturbations of multiplicative factor roughly $O(n^{1/2})$, one can retrieve the optimal Max-Cut in polynomial time. However, they conjecture that stability up to only constant magnitude perturbations should be enough to solve the problem in polynomial time. In this paper we show that this conjecture is indeed true for $k$-median and $k$-means objectives and in fact for any well-behaved center-based objective function (see Definition 1.3).

First, let us formally define the notion due to of stability under multiplicative perturbations, stated in this context.

Note that $d^{\prime}$ may be any non-negative function, and need not be a metric.

Suppose we have a clustering instance composed of $n$ points residing in a metric $(S,d)$ and an objective function $\Phi$ we wish to optimize. We call the clustering instance $\alpha$-perturbation resilient for $\Phi$ if for any $d^{\prime}$ which is an $\alpha$-perturbation of $d$, the (only) optimal clustering of $(S,d^{\prime})$ under $\Phi$ is identical, as a partition of points into subsets, to the optimal clustering of $(S,d)$ under $\Phi$.

We will in particular be concerned with separable, center-based clustering objectives $\Phi$ (which include $k$-median, $k$-means, and $k$-center among others).

A clustering objective is center-based if the optimal solution can be defined by $k$ points $c^{*}_{1},\ldots,c^{*}_{k}$ in the metric space called centers such that every data point is assigned to its nearest center. Such a clustering objective is separable if it furthermore satisfies the following two conditions:

The objective function value of a given clustering is either a (weighted) sum or the maximum of the individual cluster scores.

Given a proposed single cluster, its score can be computed in polynomial time.

Our main result is that we can efficiently find the optimal clustering for perturbation-resilient instances of separable center-based clustering objectives. In particular, we get an efficient algorithm for 3-perturbation-resilient instances when the metric $S$ is defined only over data points, and for $(2+\sqrt{3})$-perturbation-resiliant instances for general metrics.

For $\alpha\geq 3$ (in the case of finite metrics defined only over the data) or $\alpha\geq 2+\sqrt{3}$ (for general metrics), there is a polynomial-time algorithm that finds the optimal clustering of $\alpha$-perturbation resilient instances for any given separable center-based clustering objective.

The algorithm, described in Section 2.2, turns out to be quite simple. As a first step, it runs the classic single-linkage algorithm, but unlike the standard approach of halting when $k$ clusters remain, it runs the algorithm until all points have been merged into a single cluster and keeps track of the entire tree-on-clusters produced.The example depicted in Figure 3 proves that indeed, halting the Single-Linkage algorithm once $k$ clusters are formed may fail on certain $\alpha$-perturbation resilient instances. Then, the algorithm’s second step is to apply dynamic programming to this hierarchical clustering to identify the best $k$-clustering that is present within the tree. Using a result of Balcan et al. we show that the resulting clustering obtained is indeed the optimal one. Albeit being very different, our approach resembles, in spirit, the work of Bartal , Abraham et al and Räcke in the sense that we reduce the problem of retrieving an optimal solution from a general instance to a tree-like instance (where it is poly-time solvable).

Our algorithms use only a weaker property, which we call center-proximity (see Section 2.1), that is implied by perturbation-resilience. We then complement these results with a lower bound showing that for the problem of $k$-median on general metrics, for any $\epsilon>0$, there exist NP-hard instances that satisfy $(3-\epsilon)$-center proximity.We note that while our belief was that allowing Steiner points in the lower bound was primarily a technicality, Balcan et al. (M.F. Balcan, personal communication) have recently shown this is not the case, giving a clever algorithm that finds the optimal clustering for $k$-median instances in finite metrics when $\alpha=1+\sqrt{2}$.

2 Related work

There have been a number of investigations of different notions of stability for the problem of clustering. For example, Ostrovsky et al. consider a $k$-means instance to be stable if the optimal $k$-clustering is substantially cheaper than the optimal $(k-1)$-clustering under this objective. They present an efficient algorithm for finding near-optimal $k$-means clusterings when this gap is large, and these results were subsequently strengthened to apply to smaller gaps in . Balcan et al. consider instead a clustering instance to be stable if good approximations to the given objective are guaranteed to be close, as clusterings, to a desired ground-truth partitioning. This is motivated by the fact that when the true goal is to match some unknown correct answer (e.g., to correctly cluster proteins by their function), this is an implicit assumption already being made when viewing approximation ratio as a good performance measure. Balcan et al. show that in fact this condition can be used to bypass approximation hardness results for a number of clustering objectives including $k$-median and $k$-means. Here they show that if all $(1+\alpha)$-approximations to the objective are $\delta$-close to the desired clustering in terms of how points are partitioned, then one can efficiently get $O(\delta/\alpha)$-close to the desired clustering. Ben-David et al. consider a notion of stability of a clustering algorithm, which is called stable if it outputs similar clusters for different sets of $n$ input points drawn from the same distribution. For $k$-means, the work of Meila discusses the opposite direction – classifying instances where an approximated solution for $k$-means is close to the target clustering.

Proof of Main Theorem

We begin by deriving other properties which every $\alpha$-perturbation resilient clustering instance must satisfy.

Let $p\in S$ be an arbitrary point, let $c^{*}_{i}$ be the center $p$ is assigned to in the optimal clustering, and let $c^{*}_{j}\neq c^{*}_{i}$ be any other center in the optimal clustering. We say a clustering instance satisfies the $\alpha$-center proximity property if for any $p$ it holds that

If a clustering instance satisfies the $\alpha$-perturbation resilience property, then it also satisfies the $\alpha$-center proximity property.

Let $C^{*}_{i}$ and $C^{*}_{j}$ be any two clusters in the optimal clustering and pick any $p\in C^{*}_{i}$. Assume we blow up all the pairwise distances within cluster $C^{*}_{i}$ by a factor of $\alpha$. As this is a legitimate perturbation of the metric, it still holds that the optimal clustering under this perturbation is the same as the original optimum. Hence, $p$ is still assigned to the same cluster. Furthermore, since the distances within $C^{*}_{i}$ were all changed by the same constant factor, $c_{i}^{*}$ will still remain an optimal center of cluster $i$. The same holds for cluster $C^{*}_{j}$. It follows that even in this perturbed metric, $p$ prefers $c^{*}_{i}$ to $c^{*}_{j}$. Hence $\alpha d(p,c^{*}_{i})=d^{\prime}(p,c^{*}_{i})<d^{\prime}(p,c^{*}_{j})=d(p,c^{*}_{j})$. ∎

For every point $p$ and its center $c^{*}_{i}$, and for every point $p^{\prime}$ from a different cluster, it follows that $d(p,p^{\prime})>(\alpha-1)d(p,c^{*}_{i})$.

Denote by $c^{*}_{j}$ the center of the cluster that $p^{\prime}$ belongs to. Now, consider two cases. Case (a): $d(p^{\prime},c^{*}_{j})\geq d(p,c^{*}_{i})$. In this case, by traingle inequality we get that $d(p,p^{\prime})\geq d(p^{\prime},c^{*}_{i})-d(p,c^{*}_{i})$. Since the data instance is stable to $\alpha$-perturbations, Fact 2.2 gives us that $d(p^{\prime},c_{i}^{*})>\alpha d(p^{\prime},c_{j}^{*})$. Hence we get that $d(p,p^{\prime})>\alpha d(p^{\prime},c^{*}_{j})-d(p,c^{*}_{i})$ $\geq(\alpha-1)d(p,c^{*}_{i})$. Case (b): $d(p^{\prime},c^{*}_{j})<d(p,c^{*}_{i})$. Again by traingle inequality we get that $d(p,p^{\prime})\geq d(p,c^{*}_{j})-d(p^{\prime},c^{*}_{j})>\alpha d(p,c^{*}_{i})-d(p^{\prime},c^{*}_{j})>(\alpha-1)d(p,c^{*}_{i})$. ∎

A key ingredient in the proof of Theorem 1.4 is the tree-clustering formulation of Balcan et. al . In particular, we prove that if an instance satisfies $\alpha$-center proximity for $\alpha\geq 3$ (in the case of finite metrics without Steiner points) or for $\alpha\geq 2+\sqrt{3}$ (for general metrics) then it also satisfies the “min-stability property” (defined below). The min-stability property, as shown in , is sufficient (and necessary) for the Single-Linkage algorithm to produce a tree such that the optimal clustering is some pruning of this tree. In order to define the “min-stability” property, we first introduce the following notation. For any two subsets $A,B\subset S$, we denote the minimum distance between $A$ and $B$ as $d_{\min}(A,B)=\min\{d(a,b)\ |\ a\in A,b\in B\}$.

A clustering instance satisfies the min-stability property if for any two clusters $C$ and $C^{\prime}$ in the optimal clustering, and any subset $A\subsetneq C$, it holds that $d_{\min}(A,C\setminus A)\leq d_{\min}(A,C^{\prime})$.

In words, the min-stability property means that for any set $A$ that is a strict subset of some cluster $C$ in the optimal clustering, the closest point to $A$ is a point from $C\setminus A$, and not from some other cluster. The next two lemmas lie at the heart of our algorithm.

A clustering instance in which centers must be data points that satisfies $\alpha$-center proximity for $\alpha\geq 3$ (for a center-based clustering objective), also satisfies the min-stability property.

Let $C^{*}_{i},C^{*}_{j}$ be any two clusters in the target clustering. Let $A$ and $A^{\prime}$ be any two subsets s.t. $A\subsetneq C^{*}_{i}$ and $A^{\prime}\subseteq C^{*}_{j}$. Let $p\in A$ and $p^{\prime}\in A^{\prime}$ be the two points which obtain the minimum distance $d_{\min}(A,A^{\prime})$. Let $q\in C^{*}_{i}\setminus A$ be the nearest point to $p$. Also, denote by $c^{*}_{i}$ and $c^{*}_{j}$ the centers of clusters $C^{*}_{i}$ and $C^{*}_{j}$ respectively.

For the sake of contradiction, assume that $d_{\min}(A,C^{*}_{i}\setminus A)\geq d_{\min}(A,A^{\prime})$. Suppose $c^{*}_{i}\notin A$. This means that $d(p,p^{\prime})=d_{\min}(A,A^{\prime})\leq d_{\min}(A,C^{*}_{i}\setminus A)\leq d(p,c^{*}_{i})$. As $\alpha\geq 3$, this contradicts Corollary 2.3.

Thus we may assume $c^{*}_{i}\in A$. It follows that $d(q,c^{*}_{i})\geq d(p,p^{\prime})>(3-1)d(p,c^{*}_{i})=2d(p,c^{*}_{i})$, so $d(p,c^{*}_{i})<d(q,c^{*}_{i})/2$. We therefore have that $d(p^{\prime},c^{*}_{i})\leq d(p,p^{\prime})+d(p,c^{*}_{i})\leq 3d(q,c^{*}_{i})/2$. This implies that $d(p^{\prime},c^{*}_{j})$ $<$ $d(p^{\prime},c^{*}_{i})/\alpha$ $<$ $d(q,c^{*}_{i})/2$, and thus $d(q,c^{*}_{j})\leq d(q,c^{*}_{i})+d(c^{*}_{i},p)+d(p,p^{\prime})+d(p^{\prime},c^{*}_{j})<3d(q,c^{*}_{i})\leq\alpha d(q,c^{*}_{i})$. This contradicts Fact 2.2. ∎

A clustering instance in which centers need not be data points that satisfies $\alpha$-center proximity for $\alpha\geq 2+\sqrt{3}$ (for a center-based clustering objective), also satisfies the min-stability property.

As in the proof of Lemma 2.5, let $C^{*}_{i},C^{*}_{j}$ be any two clusters in the target clustering and let $A$ and $A^{\prime}$ be any two subsets s.t. $A\subsetneq C^{*}_{i}$ and $A^{\prime}\subseteq C^{*}_{j}$. Let $p\in A$ and $p^{\prime}\in A^{\prime}$ be the two points which obtain the minimum distance $d_{\min}(A,A^{\prime})$ and let $q\in C^{*}_{i}\setminus A$ be the nearest point to $p$. Also, as in the proof of Lemma 2.5, let $c^{*}_{i}$ and $c^{*}_{j}$ denote the centers of clusters $C^{*}_{i}$ and $C^{*}_{j}$ respectively (though these need not be datapoints).

By definition of center-proximity, we have the following inequalities:

Multiplying the first inequality by $1-\frac{1}{\alpha+1}-\frac{1}{\alpha-1}$, the second by $\frac{1}{\alpha+1}$, the third by $\frac{1}{\alpha-1}$, and summing them together we get

which for $\alpha=2+\sqrt{3}$ implies $d(p,p^{\prime})>d(q,p)$ as desired. ∎

2 The Algorithm

As mentioned, Balcan et al proved (Theorem $2$) that if an instance satisfies min-stability, then the tree on clusters produced by the single-linkage algorithm contains the optimal clustering as some $k$-pruning of it. I.e., the tree produced by starting with $n$ clusters of size $1$ (viewed as leaves), and at each step merging the two clusters $C$,$C^{\prime}$ minimizing $d_{\min}(C,C^{\prime})$ (viewing the merged cluster as their parent) until only one cluster remains. Given the structural results proven above, our algorithm (see Figure 1) simply uses this clustering tree and finds the best $k$-pruning using dynamic programming.

By Lemmas 2.5 and 2.6, the data satisfies the min-stability property, which as shown in is sufficient to guarantee that some pruning of the single-linkage hierarchy is the target clustering. We then find the optimal clustering using dynamic programming by examining $k$-partitions laminar with the single-linkage clustering tree. The optimal $k$-clustering of a tree-node is either the entire subtree as one cluster (if $k=1$), or the minimum over all choices of $k_{1}$-clusters over its left subtree and $k_{2}$-clusters over its right subtree (if $k>1$). Here $k_{1},k_{2}$ are positive integers, such that $k_{1}+k_{2}=k$. Therefore, we just traverse the tree bottom-up, recursively solving the clustering problem for each tree-node. By assumption that the clustering objective is separable, so each step including the base-case can be performed in polynomial time. For the case of $k$-median in a finite metric, for example, one can maintain a $n\times O(n)$ table for all possible centers and all possible clusters in the tree, yielding a running time of $O(n^{2}+nk^{2})$. For the case of $k$-means in Euclidean space, one can compute the cost of a single cluster by computing the center as just the average of all its points. In general, the overall running time is $O(n(k^{2}+T(n)))$, where $T(n)$ denotes the time it takes to compute the cost of a single cluster. ∎

3 Some Natural Barriers

We complete this section with a discussion of barriers of our approach. First, our algorithm indeed fails on some finite metrics that are $(3-\epsilon)$-perturbation resilient. For example, consider the instance shown in Figure 2. In this instance, the clustering tree produced by single-linkage is not laminar with the optimal $k$-median clustering. It is easy to check that this instance is resilient to $\alpha$-perturbations for any $\alpha<3$.

Second, observe that our analysis, though emanating from perturbation resilience, only uses center proximity. We next show that for general metrics, one cannot hope to solve (in poly-time) $k$-median instances satisfying $\alpha$-center proximity for $\alpha<3$. This is close to our upper bound of $2+\sqrt{3}$ for general metrics.

For any $\alpha<3$, the problem of solving $k$-median instances over general metrics that satisfy $\alpha$-center proximity is NP-hard.

The proof of Theorem 2.7 follows from the classical reduction of Max-$k$-Coverage to $k$-median. In this reduction, we create a bipartite graph where the right-hand side vertices represent the elements in the ground set; the left-hand side vertices represent the given subsets; and the distance between the set-vertex and each element-vertex is $1$, if the set contains that element. Using shortest-path distances, it follows that the distance from any element-vertex to a set-vertex to which it does not belong to is at least $3$. Using the fact that the NP-hardness results for Max-$k$-Coverage holds for disjoint sets (i.e. the optimal solution of Yes-instances is composed of $k$ disjoint sets, see ), the $\alpha$-center proximity property follows. ∎

Lastly, we comment that using Single-Linkage in the usual way (namely, stopping when there are $k$ clusters remaining) is not sufficient to produce a good clustering. We demonstrate this using the example shown in Figure 3. Observe, in this instance, since $C$ contains significantly less points than $A$,$B$, or $D$, this instance is stable – even if we perturb distances by a factor of $3$, the cost of any alternative clustering is higher than the cost of the optimal solution. However, because $d(A,C)>d(B,D)$, it follows that the usual version of Single-Linkage will unite $B$ and $D$, and only then $A$ and $C$. Hence, if we stop the Single-Linkage algorithm at $k=3$ clusters, we will not get the desired clustering.

Open Problems

There are several natural open questions left by this work. First, can one reduce the perturbation factor $\alpha$ needed for efficient clustering? As mentioned earlier, recently Balcan et al. (M.F. Balcan, personal communication) have given a very interesting algorithm that reduces the $\alpha=3$ factor needed by our algorithm for finite metrics to $1+\sqrt{2}$. Can one go farther, perhaps by using further implications of perturbation-resilience beyond center-proximity? Alternatively, if one cannot find the optimal clustering for small values of $\alpha$, can one still find a near-optimal clustering, of approximation ratio better than what is possible on worst-case instances?

In a different direction, one can also consider relaxations of the perturbation-resilience condition. For example, Balcan et al. (personal communication) also consider instances that are “mostly resilient” to $\alpha$-perturbations: under any $\alpha$-perturbation of the underlying metric, no more than a $\delta$-fraction of the points get mislabeled under the optimal solution. For sufficiently large constant $\alpha$ and sufficiently small constant $\delta$, they present algorithms that get good approximations to the objective under this condition. A different kind of relaxation would be to consider a notion of resilience to perturbations on average: a clustering instance whose optimal clustering is likely not to change, assuming the perturbation is random from some suitable distribution. Can this weaker notion be used to still achieve positive guarantees?

References