Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Introduction

Fitting most machine learning models involves solving some sort of optimization problem. Gradient descent, and variants of it like coordinate descent and stochastic gradient, are the workhorse tools used by the field to solve very large instances of these problems. In this work we consider the basic problem of minimizing a smooth function and the convergence rate of gradient descent methods. It is well-known that if $f$ is strongly-convex, then gradient descent achieves a global linear convergence rate for this problem (Nesterov, 2004). However, many of the fundamental models in machine learning like least squares and logistic regression yield objective functions that are convex but not strongly-convex. Further, if $f$ is only convex, then gradient descent only achieves a sub-linear rate.

This situation has motivated a variety of alternatives to strong convexity (SC) in the literature, in order to show that we can obtain linear convergence rates for problems like least squares and logistic regression. One of the oldest of these conditions is the error bounds (EB) of Luo and Tseng (1993), but four other recently-considered conditions are essential strong convexity (ESC) (Liu et al., 2014), weak strong convexity (WSC) (Necoara et al., 2015), the restricted secant inequality (RSI) (Zhang and Yin, 2013), and the quadratic growth (QG) condition (Anitescu, 2000). Some of these conditions have different names in the special case of convex functions. For example, a convex function satisfying RSI is said to satisfy restricted strong convexity (RSC) (Zhang and Yin, 2013). Names describing convex functions satisfying QG include optimal strong convexity (OSC) (Liu and Wright, 2015), semi-strong convexity (SSC) (Gong and Ye, 2014), and (confusingly) WSC (Ma et al., 2015). The proofs of linear convergence under all of these relaxations are typically not straightforward, and it is rarely discussed how these conditions relate to each other.

Polyak-Łojasiewicz Inequality

We first focus on the basic unconstrained optimization problem

and we assume that the first derivative of $f$ is $L$-Lipschitz continuous. This means that

for all $x$ and $y$. For twice-differentiable objectives this assumption means that the eigenvalues of $\nabla^{2}f(x)$ are bounded above by some $L$, which is typically a reasonable assumption. We also assume that the optimization problem has a non-empty solution set $\mathcal{X}^{*}$, and we use $f^{*}$ to denote the corresponding optimal function value. We will say that a function satisfies the PL inequality if the following holds for some $\mu>0$,

This inequality simply requires that the gradient grows faster than a quadratic function as we move away from the optimal function value. Note that this inequality implies that every stationary point is a global minimum. But unlike SC, it does not imply that there is a unique solution. Linear convergence of gradient descent under these assumptions was first proved by Polyak (1963). Below we give a simple proof of this result when using a step-size of $1/L$.

Consider problem (1), where $f$ has an $L$-Lipschitz continuous gradient (2), a non-empty solution set $\mathcal{X}^{*}$, and satisfies the PL inequality (3). Then the gradient method with a step-size of $1/L$,

By using update rule (4) in the Lipschitz inequality condition (2) we have

Now by using the PL inequality (3) we get

Re-arranging and subtracting $f^{*}$ from both sides gives us $f(x_{k+1})-f^{*}\leq\left(1-\frac{\mu}{L}\right)(f(x_{k})-f^{*})$. Applying this inequality recursively gives the result. ∎

Note that the above result also holds if we use the optimal step-size at each iteration, because under this choice we have

A beautiful aspect of this proof is its simplicity; in fact it is simpler than the proof of the same fact under the usual SC assumption. It is certainly simpler than typical proofs which rely on the other conditions mentioned in Section 1. Further, it is worth noting that the proof does not assume convexity of $f$. Thus, this is one of the few general results we have for global linear convergence on non-convex problems.

As mentioned in the Section 1, several other assumptions have been explored over the last 25 years in order to show that gradient descent achieves a linear convergence rate. These typically assume that $f$ is convex, and lead to more complicated proofs than the one above. However, it is rarely discussed how the conditions relate to each other. Indeed, all of the relationships that have been explored have only been in the context of convex functions (Bolte et al., 2015; Liu and Wright, 2015; Necoara et al., 2015; Zhang, 2015). In Appendix 0.A, we give the precise definitions of all conditions and also prove the result below giving relationships between the conditions.

For a function $f$ with a Lipschitz-continuous gradient, the following implications hold:

If we further assume that $f$ is convex then we have

Note the equivalence between EB and PL is a special case of a more general result by Bolte et al. (2015, Theorem 5), while Zhang (2016) independently also recently gave the relationships between RSI, EB, PL, and QG. Drusvyatskiy and Lewis (2016) is a recent work discussing the relationships among many of these conditions for non-smooth functions. This result shows that QG is the weakest assumption among those considered. However, QG allows non-global local minima so it is not enough to guarantee that gradient descent finds a global minimizer. This means that, among those considered above, PL and the equivalent EB are the most general conditions that allow linear convergence to a global minimizer. Note that in the convex case QG is called OSC or SSC, but the result above shows that in the convex case it is also equivalent to EB and PL (as well as RSI which is known as RSC in this case).

2 Invex and Non-Convex Functions

While the PL inequality does not imply convexity of $f$, it does imply the weaker condition of invexity. A function is invex if it is differentiable and there exists a vector valued function $\eta$ such that for any $x$ and $y$ in ${\rm I\!R}^{n}$, the following inequality holds

We obtain convex functions as the special case where $\eta(x,y)=y-x$.

Invexity was first introduced by Hanson (1981), and has been used in the context of learning output kernels (Dinuzzo et al., 2011). Craven and Glover (1985) show that a smooth $f$ is invex if and only if every stationary point of $f$ is a global minimum. Since the PL inequality implies that all stationary points are global minimizers, functions satisfying the PL inequality must be invex. It is easy to see this by noting that at any stationary point $\bar{x}$ we have $\nabla f(\bar{x})=0$, so we have

where the last inequality holds because $\mu>0$ and $f(x)\geq f^{*}$ for all $x$. This implies that $f(\bar{x})=f^{*}$ and thus any stationary point must be a global minimum.

Theorem 2.2 shows that all of the previous conditions (except QG) imply invexity. The function $f(x)=x^{2}+3\sin^{2}(x)$ is an example of an invex but non-convex function satisfying the PL inequality (with $\mu=1/32$). Thus, Theorem 2.1 implies gradient descent obtains a global linear convergence rate on this function.

Unfortunately, many complicated models have non-optimal stationary points. For example, typical deep feed-forward neural networks have sub-optimal stationary points and are thus not invex. A classic way to analyze functions like this is to consider a global convergence phase and a local convergence phase. The global convergence phase is the time spent to get “close” to a local minimum, and then once we are “close” to a local minimum the local convergence phase characterizes the convergence rate of the method. Usually, the local convergence phase starts to apply once we are locally SC around the minimizer. But this means that the local convergence phase may be arbitrarily small: for example, for $f(x)=x^{2}+3\sin^{2}(x)$ the local convergence rate would not even apply over the interval $x\in$. If we instead defined the local convergence phase in terms of locally satisfying the PL inequality, then we see that it can be much larger ($x\in{\rm I\!R}$ for this example).

3 Relevant Problems

If $f$ is $\mu$-SC, then it also satisfies the PL inequality with the same $\mu$ (see Appendix 0.B). Further, by Theorem 2.2, $f$ satisfies the PL inequality if it satisfies any of ESC, WSC, RSI, or EB (while for convex $f$, QG is also sufficient). Although it is hard to precisely characterize the general class of functions for which the PL inequality is satisfied, we note one important special case below.

Strongly-convex composed with linear: This is the case where $f$ has the form $f(x)=g(Ax)$ for some $\sigma$-SC function $g$ and some matrix $A$. In Appendix 0.B, we show that this class of functions satisfies the PL inequality, and we note that this form frequently arises in machine learning. For example, least squares problems have the form

and by noting that $g(z)\triangleq\|z-b\|^{2}$ is SC we see that least squares falls into this category. Indeed, this class includes all convex quadratic functions.

In the case of logistic regression we have

This can be written in the form $g(Ax)$, where $g$ is strictly convex but not SC. In cases like this where $g$ is only strictly convex, the PL inequality will still be satisfied over any compact set. Thus, if the iterations of gradient descent remain bounded, the linear convergence result still applies. It is reasonable to assume that the iterates remain bounded when the set of solutions is finite, since each step must decrease the objective function. Thus, for practical purposes, we can relax the above condition to “strictly-convex composed with linear” and the PL inequality implies a linear convergence rate for logistic regression.

Convergence of Huge-Scale Methods

In this section, we use the PL inequality to analyze several variants of two of the most widely-used techniques for handling large-scale machine learning problems: coordinate descent and stochastic gradient methods. In particular, the PL inequality yields very simple analyses of these methods that apply to more general classes of functions than previously analyzed. We also note that the PL inequality has recently been used by Garber and Hazan (2015a) to analyze the Frank-Wolfe algorithm. Further, inspired by the resilient backpropagation (RPROP) algorithm of Riedmiller and Braun (1992), in Appendix 0.C we also give a convergence rate analysis for a sign-based gradient descent method.

Nesterov (2012) shows that randomized coordinate descent achieves a faster convergence rate than gradient descent for problems where we have $d$ variables and it is $d$ times cheaper to update one coordinate than it is to compute the entire gradient. The expected linear convergence rates in this previous work rely on SC, but in this section we show that randomized coordinate descent achieves an expected linear convergence rate if we only assume that the PL inequality holds.

To analyze coordinate descent methods, we assume that the gradient is coordinate-wise Lipschitz continuous, meaning that for any $x$ and $y$ we have

for any coordinate $i$, and where $e_{i}$ is the $i$th unit vector.

Consider problem (1), where $f$ has a coordinate-wise $L$-Lipschitz continuous gradient (5), a non-empty solution set $\mathcal{X}^{*}$, and satisfies the PL inequality (3). Consider the coordinate descent method with a step-size of $1/L$,

If we choose the variable to update $i_{k}$ uniformly at random, then the algorithm has an expected linear convergence rate of

By using the update rule (6) in the Lipschitz condition (5) we have

By taking the expectation of both sides with respect to $i_{k}$ we have

By using the PL inequality (3) and subtracting $f^{*}$ from both sides, we get

Applying this recursively and using iterated expectations yields the result. ∎

As before, instead of using $1/L$ we could perform exact coordinate optimization and the result would still hold. If we have a Lipschitz constant $L_{i}$ for each coordinate and sample proportional to the $L_{i}$ as suggested by Nesterov (2012), then the above argument (using a step-size of $1/L_{i_{k}}$) can be used to show that we obtain a faster rate of

where $\bar{L}=\frac{1}{d}\sum_{j=1}^{d}L_{j}$.

2 Greedy Coordinate Descent

Nutini et al. (2015) have recently analyzed coordinate descent under the greedy Gauss-Southwell (GS) rule, and argued that this rule may be suitable for problems with a large degree of sparsity. The GS rule chooses $i_{k}$ according to the rule $i_{k}=\hbox{argmax}_{j}|\nabla_{j}f(x_{k})|$. Using the fact that

it is straightforward to show that the GS rule satisfies the rate above for the randomized method.

However, Nutini et al. (2015) show that a faster convergence rate can be obtained for the GS rule by measuring SC in the $1$-norm. Since the PL inequality is defined on the dual (gradient) space, in order to derive an analogous result we could measure the PL inequality in the $\infty$-norm,

Because of the equivalence between norms, this is not introducing any additional assumptions beyond that the PL inequality is satisfied. Further, if $f$ is $\mu_{1}$-SC in the $1$-norm, then it satisfies the PL inequality in the $\infty$-norm with the same constant $\mu_{1}$. By using that $|\nabla_{i_{k}}f(x_{k})|=\|\nabla f(x_{k})\|_{\infty}$ when the GS rule is used, the above argument can be used to show that coordinate descent with the GS rule achieves a convergence rate of

when the function satisfies the PL inequality in the $\infty$-norm with a constant of $\mu_{1}$. By the equivalence between norms we have that $\mu/d\leq\mu_{1}$, so this is faster than the rate with random selection.

Meir and Rätsch (2003) show that we can view some variants of boosting algorithms as implementations of coordinate descent with the GS rule. They use the error bound property to argue that these methods achieve a linear convergence rate, but this property does not lead to an explicit rate. Our simple result above thus provides the first explicit convergence rate for these variants of boosting.

3 Stochastic Gradient Methods

Stochastic gradient (SG) methods apply to the general stochastic optimization problem

where the expectation is taken with respect to $i$. These methods are typically used to optimize finite sums,

Here, each $f_{i}$ typically represents the fit of a model on an individual training example. SG methods are suitable for cases where the number of training examples $n$ is so large that it is infeasible to compute the gradient of all $n$ examples more than a few times.

Stochastic gradient methods use the iteration

If instead we use a constant $\alpha_{k}=\alpha<\frac{1}{2\mu}$, then we obtain a linear convergence rate up to a solution level that is proportional to $\alpha$,

By using the update rule (9) inside the Lipschitz condition (2), we have

Taking the expectation of both sides with respect to $i_{k}$ we have

Decreasing step size: With $\alpha_{k}=\frac{2k+1}{2\mu(k+1)^{2}}$ in (10) we obtain

where the second line follows from $\frac{2k+1}{k+1}<2$. Summing up this inequality from $k=0$ to $k$ and using the fact that $\delta_{f}(0)=0$ we get

which gives the stated rate. Constant step size: Choosing $\alpha_{k}=\alpha$ for any $\alpha<1/2\mu$ and applying (10) recursively yields

where the last line uses that $\alpha<1/2\mu$ and the limit of the geometric series. ∎

The $O(1/k)$ rate for a decreasing step size matches the convergence rate of stochastic gradient methods under SC (Nemirovski et al., 2009). It was recently shown using a non-trivial analysis that a stochastic Newton method could achieve an $O(1/k)$ rate for least squares problems (Bach and Moulines, 2013), but our result above shows that the basic stochastic gradient method already achieves this property (although the constants are worse than for this Newton-like method). Further, our result does not rely on convexity. Note that if we are happy with a solution of fixed accuracy, then the result with a constant step-size is perhaps the more useful strategy in practice: it supports the often-used empirical strategy of using a constant size for a long time, then halving the step-size if the algorithm appears to have stalled (the above result indicates that halving the step-size will at least halve the sub-optimality).

4 Finite Sum Methods

In the setting of (8) where we are minimizing a finite sums, it has recently been shown that there are methods that have the low iteration cost of stochastic gradient methods but that still have linear convergence rates for SC functions (Le Roux et al., 2012). While the first methods that achieved this remarkable property required a memory of previous gradient values, the stochastic variance-reduced gradient (SVRG) method of Johnson and Zhang (2013) does not have this drawback. Gong and Ye (2014) show that SVRG has a linear convergence rate without SC under the weaker assumption of QG plus convexity (where QG is equivalent to PL). We review how the analysis of Johnson and Zhang (2013) can be easily modified to give a similar result in Appendix 0.D. A related result appears in Garber and Hazan (2015b), who assume that $f$ is SC but do not assume that the individual functions are convex. More recent analyses by Reddi et al. (2016a, b) have considered these types of methods under the PL inequality without convexity assumptions.

Proximal-Gradient Generalization

Proximal-gradient methods apply to problems of the form

We call this the proximal-PL inequality, and we note that if $g$ is constant (or linear) then it reduces to the standard PL inequality. Below we show that this inequality is sufficient for the proximal-gradient method to achieve a global linear convergence rate.

Consider problem (11), where $f$ has an $L$-Lipschitz continuous gradient (2), $F$ has a non-empty solution set $\mathcal{X}^{*}$, $g$ is convex, and $F$ satisfies the proximal-PL inequality (12). Then the proximal-gradient method with a step-size of $1/L$,

converges linearly to the optimal value $F^{*}$,

By using Lipschitz continuity of the gradient of $f$ we have

which uses the definition of $x_{k+1}$ and $\mathcal{D}_{g}$ followed by the proximal-PL inequality (12). This subsequently implies that

which applied recursively gives the result. ∎

While other conditions have been proposed to show linear convergence rates of proximal-gradient methods without SC (Kadkhodaie et al., 2014; Bolte et al., 2015; Zhang, 2015; Li and Pong, 2016), their analyses tend to be more complicated than the above. Further, in Appendix 0.G we show that the proximal-PL condition is in fact equivalent to the KL condition, which itself is known to be equivalent to a proximal-gradient variant on the EB condition (Bolte et al., 2015). Thus, the proximal-PL inequality includes the standard scenarios where existing conditions apply.

As with the PL inequality, we now list several important function classes that satisfy the proximal-PL inequality (12). We give proofs that these classes satisfy the inequality in Appendix 0.F and 0.G.

The inequality is satisfied if $f$ satisfies the PL inequality and $g$ is constant. Thus, the above result generalizes Theorem 2.1.

The inequality is satisfied if $f$ is SC. This is the usual assumption used to show a linear convergence rate for the proximal-gradient algorithm (Schmidt et al., 2011), although we note that the above analysis is much simpler than standard arguments.

The inequality is satisfied if $f$ has the form $f(x)=h(Ax)$ for a SC function $h$ and a matrix $A$, while $g$ is an indicator function for a polyhedral set.

The inequality is satisfied if $F$ is convex and satisfies the QG property.

The inequality is satisfied if $F$ satisfies the proximal-EB condition or the KL inequality.

By the equivalence shown in Appendix 0.G, the proximal-PL inequality also holds for other problems where a linear convergence rate has been show like group L1-regularization (Tseng, 2010), sparse group L1-regularization (Zhang et al., 2013), nuclear-norm regularization (Hou et al., 2013), and other classes of functions (Zhou and So, 2015; Drusvyatskiy and Lewis, 2016).

2 Least Squares with L1-Regularization

where $\lambda>0$ is the regularization parameter. This problem has been studied extensively in machine learning, signal processing, and statistics. This problem structure seems well-suited to using proximal-gradient methods, but the first works analyzing proximal-gradient methods for this problem only showed sub-linear convergence rates (Beck and Teboulle, 2009). Subsequent works show that linear convergence rates can be achieved under additional assumptions. For example, Gu et al. (2013) prove that their algorithm achieves a linear convergence rate if $A$ satisfies a restricted isometry property (RIP) and the solution is sufficiently sparse. Xiao and Zhang (2013) also assume the RIP property and show linear convergence using a homotopy method that slowly decreases the value of $\lambda$. Agarwal et al. (2012) give a linear convergence rate under a modified restricted strong convexity and modified restricted smoothness assumption. But these problems have also been shown to satisfy proximal variants of the KL and EB conditions (Tseng, 2010; Bolte et al., 2015; Necoara and Clipici, 2016), and Bolte et al. (2015) in particular analyzes the proximal-gradient method under KL while giving explicit bounds on the constant. This means any L1-regularized least squares problem also satisfies the proximal-PL inequality. Thus, Theorem 4.1 gives a simple proof of global linear convergence for these problems without making additional assumptions or making any modifications to the algorithm.

3 Proximal Coordinate Descent

It is also possible to adapt our results on coordinate descent and proximal-gradient methods in order to give a linear convergence rate for coordinate-wise proximal-gradient methods for problem (11). To do this, we require the extra assumption that $g$ is a separable function. This means that $g(x)=\sum_{i}g_{i}(x_{i})$ for a set of univariate functions $g_{i}$. The update rule for the coordinate-wise proximal-gradient method is

We state the convergence rate result below.

Assume the setup of Theorem 4.1 and that $g$ is a separable function $g(x)=\sum_{i}g_{i}(x_{i})$, where each $g_{i}$ is convex. Then the coordinate-wise proximal-gradient update rule (16) achieves a convergence rate

when $i_{k}$ is selected uniformly at random.

The proof is given in Appendix 0.H and although it is more complicated than the proofs of Theorems 3.2 and 4.1, it is arguably still simpler than existing proofs for proximal coordinate descent under SC (Richtárik and Takáč, 2014), KL (Attouch et al., 2013), or QG (Zhang, 2016). It is also possible to analyze stochastic proximal-gradient algorithms, and indeed Reddi et al. (2016c) use the proximal-PL inequality to analyze finite-sum methods in the proximal stochastic case.

4 Support Vector Machines

Another important model problem that arises in machine learning is support vector machines,

for a particular positive semi-definite matrix $M$ and constant $U$. This convex function satisfies the QG property and thus Theorem 4.2 implies that coordinate optimization achieves a linear convergence rate in terms of optimizing the dual objective. Further, note that Hush et al. (2006) show that we can obtain an $\epsilon$-accurate solution to the primal problem with an $O(\epsilon^{2})$-accurate solution to the dual problem. Thus this result also implies we can obtain a linear convergence rate on the primal problem by showing that stochastic dual coordinate ascent has a linear convergence rate on the dual problem. Global linear convergence rates for SVMs have also been shown by others (Tseng and Yun, 2009; Wang and Lin, 2014; Ma et al., 2015), but again we note that these works lead to more complicated analyses. Although the constants in these convergence rate may be quite bad (depending on the smallest non-zero singular value of the Gram matrix), we note that the existing sublinear rates still apply in the early iterations while, as the algorithm begins to identify support vectors, the constants improve (depending on the smallest non-zero singular value of the block of the Gram matrix corresponding to the support vectors).

Discussion

We believe that this work provides a unifying and simplifying view of a variety of optimization and convergence rate issues in machine learning. Indeed, we have shown that many of the assumptions used to achieve linear convergence rates can be replaced by the PL inequality and its proximal generalization. While we have focused on sufficient conditions for linear convergence, another recent work has turned to the question of necessary conditions for convergence (Zhang, 2016). Further, while we’ve focused on non-accelerated methods, Zhang (2016) has recently analyzed Nesterov’s accelerated gradient method without strong convexity. We also note that, while we have focused on first-order methods, Nesterov and Polyak (2006) have used the PL inequality to analyze a second-order Newton-style method with cubic regularization. They also consider a generalization of the inequality under the name gradient-dominated functions.

Throughout the paper, we have pointed out how our analyses imply convergence rates for a variety of machine learning models and algorithms. Some of these were previously known, typically under stronger assumptions or with more complicated proofs, but many of these are novel. Note that we have not provided any experimental results in this work, since the main contributions of this work are showing that existing algorithms actually work better on standard problems than we previously thought. We expect that going forward efficiency will no longer be decided by the issue of whether functions are SC, but rather by whether they satisfy a variant of the PL inequality.

We would like to thank Simon LaCoste-Julien, Martin Takáč, Ruoyu Sun, Hui Zhang, and Dmitriy Drusvyatskiy for valuable discussions. We would like to thank Ting Kei Pong and Zirui Zhou for pointing out an error in the first version of this paper, to Ting Kei Pong for discussions that lead to the addition of Appendix 0.G, to Jérôme Bolte for an informative discussion about the KL inequality and pointing us to related results that we had missed, to Liam Madden and Stephen Becker for pointing out an error (and the fix) in our “PL implies QG” proof, and to Boris Polyak for providing an English translation of his original work. This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC RGPIN-06068-2015). Julie Nutini is funded by a UBC Four Year Doctoral Fellowship (4YF) and Hamed Karimi is support by a Mathematics of Information Technology and Complex Systems (MITACS) Elevate Fellowship.

Appendix 0.A Relationships Between Conditions

We start by stating the different conditions. All of these definitions involve some constant $\mu>0$ (which may not be the same across conditions), and we’ll use the convention that $x_{p}$ is the projection of $x$ onto the solution set $\mathcal{X}^{*}$.

Strong Convexity (SC): For all $x$ and $y$ we have

Essential Strong Convexity (ESC): For all $x$ and $y$ such that $x_{p}=y_{p}$ we have

Weak Strong Convexity (WSC): For all $x$ we have

Restricted Secant Inequality (RSI): For all $x$ we have

If the function $f$ is also convex it is called restricted strong convexity (RSC).

Polyak-Łojasiewicz (PL): For all $x$ we have

Quadratic Growth (QG): For all $x$ we have

If the function $f$ is also convex it is called optimal strong convexity (OSC) or semi-strong convexity or sometimes WSC (but we’ll reserve the expression WSC for the definition above).

Below we prove a subset of the implications in Theorem 2.2. The remaining relationships in Theorem 2.2 follow from these results and transitivity.

$\bf SC\rightarrow ESC$: The SC assumption implies that the ESC inequality is satisfied for all $x$ and $y$, so it is also satisfied under the constraint $x_{p}=y_{p}$.

$\bf ESC\rightarrow WSC$: Take $y=x_{p}$ in the ESC inequality (which clearly has the same projection as $x$) to get WSC with the same $\mu$ as a special case.

$\bf WSC\rightarrow RSI$: Re-arrange the WSC inequality to

Since $f(x)-f^{*}\geq 0$, we have RSI with $\frac{\mu}{2}$.

$\bf RSI\rightarrow EB$: Using Cauchy-Schwartz on the RSI we have

and dividing both sides by $\|x-x_{p}\|$ (assuming $x\not=x_{p}$) gives EB with the same $\mu$ (while EB clearly holds if $x=x_{p}$).

$\bf EB\rightarrow PL$: By Lipschitz continuity we have

and using EB along with $f(x_{p})=f^{*}$ and $\nabla f(x_{p})=0$ we have

which is the PL inequality with constant $\frac{\mu}{L}$.

$\bf PL\rightarrow EB$: Below we show that PL implies QG with the same constant. Using this result in PL we get

which implies that EB holds with the same constant.

$\bf QG+Convex\rightarrow RSI$: By convexity we have

which is RSI with constant $\frac{\mu}{2}$.

$\bf PL\rightarrow QG$: Our argument that this implication holds is similar to the argument used in related works (Bolte et al., 2015; Zhang, 2015) Define the function

If we assume that $f$ satisfies the PL inequality then for any $x\not\in\mathcal{X}^{*}$ we have

By the definition of $g$, to show QG it is sufficient to show that

As $f$ is assumed to satisfy the PL inequality we have that $f$ is an invex function and thus by definition $g$ is a positive invex function ($g(x)\geq 0$) with a closed optimal solution set $\mathcal{X}^{*}$ such that for all $y\in\mathcal{X}^{*}$, $g(y)=0$. For any point $x_{0}\not\in\mathcal{X}^{*}$, consider solving the following differential equation:

for $x(t)\not\in\mathcal{X}^{*}$. (This is a flow orbit starting at $x_{0}$ and flowing along the gradient of $g$.) By (20), $\nabla g$ is bounded from below, and as $g$ is a positive invex function $g$ is also bounded from below. Thus, by moving along the path defined by (• ‣ 0.A) we are sufficiently reducing the function and will eventually reach the optimal set. Thus there exists a $T$ such that $x(T)\in\mathcal{X}^{*}$ (and at this point the differential equation ceases to be defined). We can show this by using the steps

As $g(x_{t})\geq 0$, this shows we need to have $T\leq 2g(x_{0})/\mu$, so there must be a $T$ with $x(T)\in\mathcal{X}^{*}$.

The length of the orbit $x(t)$ starting at $x_{0}$, which we’ll denote by $\mathcal{L}(x_{0})$, is given by

where $x_{p}$ is the projection of $x_{0}$ onto $\mathcal{X}^{*}$ and the inequality follows because the orbit is a path from $x_{0}$ to a point in $\mathcal{X}^{*}$ (and thus it must be at least as long as the projection distance).

Starting from the line marked $(*)$ above we have

As $g(x_{T})=0$, this yields our result (21), or equivalently

Appendix 0.B Relevant Problems

Strongly-convex: By minimizing both sides of the SC inequality with respect to $y$ we get

which implies the PL inequality holds with the same value $\mu$. Thus, Theorem 2.1 exactly matches the known rate for gradient descent with a step-size of $1/L$ for a $\mu$-SC function.

Strongly-convex composed with linear: To show that this class of functions satisfies the PL inequality, we first define $f(x):=g(Ax)$ for a $\sigma$-strongly convex function $g$. For arbitrary $x$ and $y$, we define $u:=Ax$ and $v:=Ay$. By the strong convexity of $g$, we have

By our definitions of $u$ and $v$, we get

where we can write the middle term as $(A^{T}\nabla g(Ax))^{T}(y-x)$. By the definition of $f$ and its gradient being $\nabla f(x)=A^{T}\nabla g(Ax)$ by the multivariate chain rule, we obtain

Using $x_{p}$ to denote the projection of $x$ onto the optimal solution set $\mathcal{X}^{*}$, we have

In the second line we use that $\mathcal{X}^{*}$ is polyhedral, and use the theorem of Hoffman (1952) to obtain a bound in terms of $\theta(A)$ (the smallest non-zero singular value of $A$). This derivation implies that the PL inequality is satisfied with $\mu=\sigma\theta(A)$.

Appendix 0.C Sign-Based Gradient Methods

The learning heuristic RPROP (Resilient backPROPagation) is a classic iterative method used for supervised learning problems in feedforward neural networks (Riedmiller and Braun, 1992). The general update for some vector of step sizes $\alpha_{k}\in{\rm I\!R}^{d}$ is given by

where the $\circ$ operator indicates coordinate-wise multiplication. Although this method has been used for many years in the machine learning community, we are not aware of any previous convergence rate analysis of such a method. Here we give a convergence rate when the individual step-sizes $\alpha_{i}^{k}$ are chosen proportional to $1/\sqrt{L_{i}}$, where the $L_{i}$ are constants such that the gradient is 1-Lipschitz continuous in the norm defined by

Formally, we assume that the $L_{i}$ are set so that for all $x$ and $y$ we have

and where the dual norm of the $\|\cdot\|_{L^{-1}}$ norm above is given by the $\|\cdot\|_{L[\infty]}$ norm,

We note that such $L_{i}$ always exist if the gradient is Lipschitz continuous, so this is not adding any assumptions on the function $f$. The particular choice of the step-sizes $\alpha_{i}^{k}$ that we will analyze is

which yields a linear convergence rate for problems where the PL inequality is satisfied.

The coordinate-wise iteration update under this choice of $\alpha_{i}^{k}$ is given by

Defining a diagonal matrix $\Lambda$ with $1/\sqrt{L_{i}}$ along the diagonal, the update can be written as

Consider the function $g(\tau)=f(x+\tau(y-x))$ with $\tau\in{\rm I\!R}$. Then

where the second inequality uses the Lipschitz assumption, and in the first inequality we’ve used the Cauchy-Schwarz inequality and that the dual norm of the $L^{-1}$ norm is the $L[\infty]$ norm. The above gives an upper bound on the function in terms of this $L[\infty]$-norm,

Subtracting $f^{*}$ from both sides yields

Applying the PL inequality with respect to the ${L^{-1}}$-norm (which, if the PL inequality is satisfied, holds for some $\mu_{L[\infty]}$ by the equivalence between norms),

Appendix 0.D Linear Convergence Rate of SVRG Method

In this section, we look at the SVRG method for the finite-sum optimization problem,

To minimize functions of this form, the SVRG algorithm of Johnson and Zhang (2013) uses iterations of the form

where $i_{t}$ is chosen uniformly from $\{1,2,\dots,n\}$ and we assume the step-size satisfies $\alpha<2/L$. In this algorithm we start with some $x^{0}$ and initially set $\mu^{0}=\nabla f(x^{0})$ and $x_{0}=x^{0}$, but after every $m$ steps we set $x^{s+1}$ to a random $x_{t}$ for $t\in\{ms+1,\dots,m(s+1)\}$, then replace $\mu^{s}$ with $\nabla f(x^{s})$ and $x^{t}$ with $x^{s+1}$. Analogous to Johnson and Zhang (2013) for the SC case, we now show tnat SVRG has a linear convergence rate if each $f_{i}$ is a convex function with a Lipschitz-continuous gradient and $f$ satisfies the PL inequality.

Following the same argument as Johnson and Zhang (2013), for any solution $x^{*}$ the assumptions on the $f_{i}$ mean that the “outer” SVRG iterations $x^{s}$ satisfy

Choosing the particular $x^{*}$ that is the projection of $x^{s-1}$ onto the solution set and using QG (which is equivalent to PL in this convex setting) we have

Dividing both sides by $2\alpha(1-2L\alpha)m$ we get

which is a linear convergence rate for sufficiently large $m$ and sufficiently small $\alpha$.

Appendix 0.E Proximal-PL Lemma

In this section we give a useful property of the function $\mathcal{D}_{g}$.

For any differentiable function $f$ and any convex function $g$, given $\mu_{2}\geq\mu_{1}>0$ we have

We’ll prove Lemma 1 as a corollary of a related result. We first restate the definition

and we note that we require $\lambda>0$. By completing the square, we have

Notice that if $g=0$, then $\mathcal{D}_{g}(x,\lambda)=||\nabla f(x)||^{2}$ and the proximal-PL inequality reduces to the PL inequality. We’ll the define the proximal residual function as the second part of the above equality,

If $g$ is convex then for any $x$ and $a$, and for $0<\lambda_{1}\leq\lambda_{2}$ we have

Without loss of generality, assume $x=0$. Then we have

where in the second line we used a changed of variables $\bar{y}=\lambda y$ (note that we are minimizing over the whole space of ${\rm I\!R}^{n}$). By the convexity of $g$, for any $\alpha\in$ and $z\in{\rm I\!R}^{n}$ we have

By using $0<\lambda_{1}/\lambda_{2}\leq 1$ and using the choices $\alpha=\frac{\lambda_{1}}{\lambda_{2}}$ and $z={\bar{y}/\lambda_{1}}$ we have

Adding $||\bar{y}+a||^{2}$ to both sides, we get

Taking the minimum over both sides with respect to $\bar{y}$ yields Lemma 2 due to (29). ∎

For any differentiable function $f$ and convex function $g$, given $\lambda_{1}\leq\lambda_{2}$, we have

By using $\mathcal{D}_{g}(x,\lambda)=||\nabla f(x)||^{2}-\mathcal{R}_{g}(\lambda,x,\nabla f(x))$, Corollary 1 is exactly Lemma 1.

Appendix 0.F Relevant Problems

In this section we prove that the three classes of functions listed in Section 4.1 satisfy the proximal-PL inequality condition. Note that while we prove these hold for $\mathcal{D}_{g}(x,\lambda)$ for $\lambda\leq L$, by Lemma 1 above they also hold for $\mathcal{D}_{g}(x,L)$.

$f(x)$, where $f$ satisfies the PL inequality ($g$ is constant): As $g$ is assumed to be constant, we have $g(y)-g(x)=0$ and the left-hand side of the proximal-PL inequality simplifies to

Thus, the proximal PL inequality simplifies to $f$ satisfying the PL inequality,

$F(x)=f(x)+g(x)$ and $f$ is strongly convex: By the strong convexity of $f$ we have

Since $\mathcal{X}$ is polyhedral, it can be written as a set $\{x:Bx\leq c\}$ for a matrix $B$ and a vector $c$. As before, assume that $x_{p}$ is the projection of $x$ onto the optimal solution set $\mathcal{X}^{*}$ which in this case is $\{x:Bx\leq c,Ax=z\}$ for some $z$.

where we’ve used the notation that $\{\cdot\}_{+}=\max\{0,\cdot\}$, the fourth equality follows because $x$ was projected onto $\mathcal{X}$ in the previous iteration (so $Bx-c\leq 0$), and the line after that uses Hoffman’s bound (Hoffman, 1952).

$F(x)=f(x)+g(x)$, $f$ is convex, and $F$ satisfies the quadratic growth (QG) condition: A function $F$ satisfies the QG condition if

The third line follows from the convexity of $f$, and the last inequality uses the QG condition of $F$. Multiplying both sides by $-2\lambda$, we have

This is true for any $\lambda>0$, and by choosing $\lambda=\mu/2$ we have

$F$ satisfies the KL inequality or the proximal-EB inequality: In the next section we show that these are equivalent to the proximal-PL inequality.

Appendix 0.G Equivalence of Proximal-PL with KL and EB

The equivalence of the KL condition and the proximal-gradient variant of the Luo-Tseng EB condition is known for convex $f$, see (Drusvyatskiy and Lewis, 2016, Corollary 3.6) and the proof of (Bolte et al., 2015, Theorem 5). Here we prove the equivalence of these conditions with the proximal-PL inequality for non-convex $f$. First we review the definitions of the three conditions:

Proximal-PL: There exists a $\mu>0$ such that

Proximal-EB: There exists $c>0$ such that we have

where $\partial F(x)$ is the Frechet subdifferential. In particular, if $F:H\rightarrow\mathcal{R}$ is a real-valued function then we say that $s\in H$ is a Frechet subdifferential of $F$ at $x\in\hbox{dom}\ F$ if

Note that for differentiable $f$ the Frechet subdifferential only contains the gradient, $\nabla f(x)$. In our case where $F(x)=f(x)+g(x)$ with a differentiable $f$ and a convex $g$ we have

The KL inequality is an intuitive generalization of the PL inequality since, analogous to the gradient vector in the smooth case, the negation of the quantity $\mathop{\hbox{argmin}}_{s\in\partial F(x)}~{}\!\|s\|$ points in the direction of steepest descent (see Bertsekas et al., 2003, Section 8.4)

We first derive an alternative representation of ${\cal D}_{g}(x,L)$ in terms of the so-called forward-backward envelope $F_{\frac{1}{L}}$ of $F$ (see Stella et al., 2016, Definition 2.1). Indeed,

It follows from the definition of $F_{\frac{1}{L}}(x)$ that we have

where the second line uses that we are taking the minimizer and the last line uses the Lipschitz continuity of $\nabla f$ as follows,

Proximal-EB $\rightarrow$ proximal-PL: we have that

for some constants $C_{0}$ and $C_{1}$, where the second inequality uses the proximal-EB and the last inequality follows from Stella et al. (2016, Proposition 2.2(i)). Now by using the fact that $F(x)-F_{\frac{1}{L}}(x)={1\over 2L}{\cal D}_{g}(x,L)$, the function satisfies the proximal-PL inequality.

Proximal-PL $\rightarrow$ KL: It’s sufficient to prove that ${\cal D}_{g}(x,\mu)\leq\min_{s\in\partial F(x)}\|s\|^{2}$ for any $x$ and $\mu$. First we observe that for any subgradient $\xi\in\partial g(x)$ we have

where the inequality follows from the definition of a subgradient. Now by minimizing both sides over $y$ we have

Multiplying both sides with $-2\mu$ we get

Since this holds for any $\xi\in\partial g(x)$, then it holds for any $\zeta=\nabla f(x)+\xi\in\partial F(x)$, it also holds for the minimium-norm subgradient of $F$.

KL $\rightarrow$ Proximal-EB: A function $h(x)$ is called “semiconvex” if there exist an $\alpha>0$ such that $h(x)+\alpha\|x\|^{2}$ is convex (see Bolte et al., 2010, Definition 10). Note that Lipschitz-continuity of $\nabla f$ implies semi-convexity of $f$ in light of (49) and Bolte et al. (2010, Remark 11(iii)). It follows from convexity of $g$ that $F$ is semi-convex. From Bolte et al. (2010, Theorem 13), for any $x\in\hbox{dom}F$ there exist a subgradient curve $\chi_{x}:[0,\infty]\rightarrow\hbox{dom}F$ that satisfies

where $F(\chi_{x}(t))$ is non-increasing and Lipschitz continuous on $[\eta,\infty]$ for any $\eta>0$. By using these facts let’s define the function $r(t)=\sqrt{F(\chi_{x}(t))-F^{*}}$. It is easy to see that

for the second line we used the definition of subgradient curve and for the third line we used KL inequality condition and the fact that $\dot{\chi_{x}}(t)\in-\partial F(\chi_{x}(t))$. Now we have

where we used the bound on the derivative of $r(t)$ above and that the length of the curve connecting any two points is less than the Euclidean distance between them. We’re now going to take the limit of $T\rightarrow\infty$, while using the facts that $r(\infty)=0$ (which we prove below) and using $r(0)=\sqrt{F(x)-F^{*}}$. This gives

From this inequality and also KL condition 45, we proved that there exist a $C>0$ such that

Now let’s show that $r(\infty)=0$ or $\chi_{x}(\infty)\in\mathcal{X}$. From equation 56 we have

where for the first inequality we used the KL property, and for the second inequality we used the fact that $F(\chi_{x}(t))$ is non-increasing, which means $F(\chi_{x}(T))\leq F(\chi_{x}(t))$ for any $t\in[0,T]$. This inequality gives a bound on $r(T)$,

now by taking the limit of $T\rightarrow\infty$ , we get $r(T)\rightarrow 0$.

Now by using 58 we can show that the proximal-EB condition is satisfied. Let’s define $\hat{x}={\rm prox}_{\frac{1}{L}g}\left(x-\frac{1}{L}\nabla f(x)\right)$. From the optimality of $\hat{x}$ we have $-\nabla f(x)-L(\hat{x}-x)\in\partial g(\hat{x})$, using this we get

Denoting the particular subgradient of $g$ that achieves this by $\xi$, we have

where the second inequality uses $\|a-b\|\leq\|a\|+\|b\|$, and for the last line we used Lipschitz continuity of of $\nabla f$. We finally get the proximal-EB condition using

where the first inequality follows from the triangle inequality, the second line uses 58, and the third inequality uses 60.

Appendix 0.H Proximal Coordinate Descent

Here we show linear convergence of randomized coordinate descent for $F(x)=f(x)+g(x)$ assuming that $F$ satisfies the proximal PL inequality, $\nabla f$ is coordinate-wise Lipschitz continuous, and $g$ is a separable convex function ($g(x)=\sum_{i}g_{i}(x_{i})$).

From coordinate-wise Lipschitz continuity of $\nabla f$ and separability of $g$, we have

Given a coordinate $i$ the coordinate descent step chooses $y_{i}$ to minimize this upper bound on the improvement in $F$,

We next use an argument similar to Richtárik and Takáč (2014) to relate the expected improvement (with random selection of the coordinates) to the function $\mathcal{D}_{g}$,

(Note that separability allows us to exchange the summation and minimization operators.) By using this and taking the expectation of (62) we get

Subtracting $F^{*}$ from both sides and applying the proximal-PL inequality yields a linear convergence rate of $\left(1-\frac{\mu}{nL}\right)$.

References

References