On Lower Complexity Bounds for Large-Scale Smooth Convex Optimization

Introduction

Huge sizes of convex optimization problems arising in some modern applications (primarily, in big-data-oriented signal processing and machine learning) are beyond the “practical grasp” of the state-of-the-art Interior Point Polynomial Time methods with their computationally demanding iterations. Indeed, aside of rare cases of problems with “extremely favourable” structure, the arithmetic cost of an interior point iteration is at least cubic in the design dimension $n$ of the instance; with $n$ in the range of $10^{4}$ – $10^{6}$, as is the case in the outlined applications, this makes a single iteration “lasting forever.” The standard techniques for handling large-scale convex problems – those beyond the practical grasp of Interior Point methods – are First Order methods (FOM’s). Under favorable circumstances, iterations of FOM’s are much cheaper than those of interior point methods, and the convergence rate, although just sublinear, is fully or nearly dimension-independent, which makes FOM’s the methods of choice when medium-accuracy solutions to large-scale convex programs are sought. Now, as a matter of fact, all known FOM’s are “black-box-oriented” – they “learn” the problem being solved solely via the local information (values and (sub)gradients of the objective and the constraints) accumulated along the search points generated by the algorithm. As a result, “limits of performance” of FOM’s are governed by Information-Based Complexity Theory. Some basic results in this direction have been established in the literature ; in particular, we know well enough what is the Information-Based Complexity of natural families of convex minimization problems $\min_{x\in X}f(x)$ with nonsmooth Lipschitz continuous objectives $f$ and how the complexity depends on the geometry and the dimension of the domain $X$. In the smooth case, our understanding is somehow limited; essentially, tight lower complexity bounds are known only in the case when $X$ is Euclidean ball and $f$ is convex function with Lipschitz continuous gradient. Lower bounds here come from least-squares problems , and the underlying techniques for generating “hard instances” heavily utilize the rotational invariance of a Euclidean ball.

Our first contribution is a unified framework to prove lower bounds for a variety of domains and different smoothness parameters of the objective with respect to a norm (for consistency we use the norm induced by the domain). In order to construct hard instances for lower bounds we need the normed space under consideration to satisfy a “smoothing property.” Namely, we need the existence of a “smoothing kernel” – a convex function with Lipschitz continuous gradient and “fast growth.” These properties guarantee that the inf-convolution of a Lipschitz continuous convex function $f$ and the smoothing kernel is smooth, and its local behaviour depends only on the local behavior of $f$. A novelty here, if any, stems from the fact that we need Lipschitz continuity of the gradient w.r.t. a given, not necessarily Euclidean, norm, while the standard Moreau envelope technique is adjusted to the case of the Euclidean norm)It well may happen that the extensions of the classical Moreau results which we present in Section 2 are known, so that the material in this section does not pretend to be novel. This being said, at this point in time we do not have at our disposal references to the results on smoothing we need, and therefore we decided to augment these simple results with their proofs, in order to make our presentation self-contained.)

We establish lower bounds on complexity of smooth convex minimization for general spaces satisfying the smoothing property. Our proof mimics the construction of hard instances for nonsmooth convex minimization , which now are properly smoothed by the inf-convolution.

With this general result, we are able to provide a unified analysis for lower bounds for smooth convex minimization over $n$-dimensional $\|\cdot\|_{p}$-balls, $1\leq p\leq\infty$. We show that in the large-scale case, our lower complexity bounds match, within at worst a logarithmic in $n$ factor, the upper complexity bounds associated with Nesterov’s fast gradient algorithms . When $p=\infty$, this result implies near optimality of the Conditional Gradient algorithm.

As a final application, we point out how our lower bounds extend to matrix optimization under Schatten norm constraints.

2 Related work

Oracle Complexity: The analysis of convex optimization algorithms via oracle complexity and lower complexity bounds were first studied in . Other standard references are . The oracle complexity of smooth convex optimization over Euclidean domains was studied in .

For optimal methods under non-Euclidean domains for smooth spaces and $p$-norms, where $2\leq p<\infty$ we refer to (for the case $p=2$ there is an interesting new algorithm that adapts itself to the smoothness parameter in the objective ).

It should be mentioned that for the case $2\leq p<\infty$ the lower bounds in this paper were announced in (and proved by the second author of this paper); however, aside of the very special case of $p=2$, the highly technical original proofs of the bounds were never published. For this reason, we recently have revisited the original proofs and were able to simplify them dramatically, thus making them publishable.

The Conditional Gradient algorithm and complexity under Linear Optimization oracles: The recent body of work on the Conditional Gradient algorithm is enormous. For upper bounds on its complexity we refer to . Interestingly, the last two references include results on linear convergence of the Conditional Gradient method for the strongly convex case, accelerated methods based on Linear Optimization oracles, and applications to stochastic and online convex programming.

Besides these accuracy upper bounds, there are some interesting lower bounds for algorithms based on a Linear Optimization oracle (whose only assumption is that the Linear Optimization oracle returns a solution that is a vertex of the domain): some of these contributions can be found in . Observe that a Linear Optimization oracle is in general less powerful than an arbitrary local oracle (in particular the first-order one) considered in our paper, and thus their lower bounds do not imply ours. However, our result for $p=\infty$ improves on their lower bounds (disregarding logarithmic factors).

3 Notation and preliminaries

Algorithms and Complexity: In the black-box oracle complexity model for convex optimization we are interested in solving problems of the form

where $X$ is a given convex compact subset of a normed space $({\mathbf{E}},\|\cdot\|)$, and $f$ is known to belong to a given family ${\cal F}$ of continuous convex functions on ${\mathbf{E}}$. This defines the family of problems ${\cal P}({\cal F},X)$ comprised of problems $(P_{f,X})$ with $f\in{\cal F}$. We assume that the family ${\cal F}$ is equipped with an oracle ${\cal O}$ which, formally, is a function ${\cal O}(f,x)$ of $f\in{\cal F}$ and $x\in{\mathbf{E}}$ taking values in some information space ${\cal I}$; when solving $(P_{f,X})$, an algorithm at every step can sequentially call the oracle at a query point $x\in{\mathbf{E}}$, obtaining the value ${\cal O}(f,x)$. In the sequel, we always assume the oracle to be local, meaning that for all $x\in{\mathbf{E}}$ and $f,g\in{\cal F}$ such that $f(\cdot)=g(\cdot)$ in a neighbourhood of $x$, we have ${\cal O}(f,x)={\cal O}(g,x)$. The most common example of oracle is the first-order oracle, which returns the value and a subgradient of $f$ at $x$. However, observe that when the subdifferential is not a singleton not every such oracle satisfies the local property, and we need to further restrict it to satisfy locality.

A $T$-step algorithm ${\cal M}$, utilizing oracle ${\cal O}$, for the family ${\cal P}({\cal F},X)$ is a procedure as follows. As applied to a problem $(P_{f,X})$ with $f\in{\cal F}$, ${\cal M}$ generates a sequence $x_{t}=x_{t}({\cal M},f)$, $1\leq t\leq T$ of search points according to the recurrence

where the search rules $X_{t}(\cdot)$ are deterministic functions of their arguments; we can identify ${\cal M}$ with the collection of these rules. Thus, $x_{1}$ is specified by ${\cal M}$ and is independent of $f$, and all subsequent search points are deterministic functions of the preceding search points and the information on $f$ provided by ${\cal O}$ when queried at these points. We treat $x_{T}=x_{T}({\cal M},f)$ as the approximate solution generated by the $T$-step solution method ${\cal M}$ applied to $(P_{f,X})$, and define the minimax risk associated with the family ${\cal P}({\cal F},X)$ and oracle ${\cal O}$ as the function of $T$ defined by

where the right hand side infinum is taken over all $T$-step solution algorithms ${\cal M}$ utilizing oracle ${\cal O}$ and such that $x_{T}({\cal M},f)\in X$ for all $f\in{\cal F}$. The inverse to the risk function

for $\varepsilon>0$ is called the information-based (or oracle) complexity of the family ${\cal P}({\cal F},X)$ with respect to oracle ${\cal O}$.

Geometry and Smoothness: Let ${\mathbf{E}}$ be an $n$-dimensional Euclidean space, and $\|\cdot\|$ be a norm on ${\mathbf{E}}$ (not necessarily the Euclidean one). Let, further, $X$ be a nonempty closed and bounded convex set in ${\mathbf{E}}$. Given a positive real $L$ and $\kappa\in(1,2]$, consider the family ${\cal F}_{\|\cdot\|}(\kappa,L)$ of all continuously differentiable convex functions $f:{\mathbf{E}}\to{\mathbf{R}}$ which are $(\kappa,L)$-smooth w.r.t. $\|\cdot\|$, i.e. satisfy the relation

where $\|\cdot\|_{*}$ is the norm conjugate to $\|\cdot\|$. We associate with $\|\cdot\|,X,\kappa,L$ the family of convex optimization problems ${\cal P}={\cal P}({\cal F}_{\|\cdot\|}(\kappa,L),X)$.

We assume the family ${\cal F}_{\|\cdot\|}(\kappa,L)$ is equipped with a local oracle ${\cal O}$. To avoid extra words, we assume that this oracle is at least as powerful as the First Order oracle, meaning that $f(x),\nabla f(x)$ is a component of ${\cal O}(f,x)$.

Our goal is to establish lower bounds on the risk ${\hbox{\rm Risk}}(T)$, taken w.r.t. the oracle ${\cal O}$, of the just defined family of problems ${\cal P}$. In the sequel, we focus solely on the ‘large-scale’ case $n\geq T$, and the reason is as follows: it is known that when $T\gg n$, ${\hbox{\rm Risk}}(T)$ “basically forgets the details specifying ${\cal P}$” and is upper-bounded by $O(\exp\{-CT/n\})$, where $C$ is an absolute constant, with the data $\|\cdot\|,X,L,\kappa$ of ${\cal P}$ affecting only the hidden factor in the outer $O(\cdot)$ and thus irrelevant when $T\gg n$. In contrast to this, in the large-scale regime $T\leq n$, ${\hbox{\rm Risk}}(T)$ is (at least in the cases we are about to consider) nearly independent of $n$ and goes to 0 sublinearly as $T$ grows, and its behavior in this range heavily depends on ${\cal P}$. In what follows, we focus solely on the large-scale regime.

Local Smoothing

In this section we introduce the main component of our technique, a Moreau-type approximation of a nonsmooth convex function $f$ by a smooth one. The main feature of this smoothing, instrumental for our ultimate goals, is that it is local – the local behaviour of the approximation at a point depends solely on the restriction of $f$ onto a neighbourhood of the point, the size of the neighbourhood being under our full control.

Let $({\mathbf{E}},\langle\cdot,\cdot\rangle)$ be a finite-dimensional Euclidean space, $\|\cdot\|$ be a norm on ${\mathbf{E}}$ (not necessarily induced by $\langle\cdot,\cdot\rangle$), and ${\cal C}_{\|\cdot\|}$ be the set of all Lipschitz continuous, with constant 1 w.r.t. $\|\cdot\|$, convex functions on ${\mathbf{E}}$. Let also $\phi(\cdot)$ (“smoothing kernel”) be a twice continuously differentiable convex function defined on an open convex set ${\hbox{\rm Dom}\,}\phi\subset{\mathbf{E}}$ with the following properties:

$0\in{\hbox{\rm Dom}\,}\phi$ and $\phi(0)=0$, $\phi^{\prime}(0)=0$;

There exists a compact convex set $G\subseteq{\hbox{\rm Dom}\,}\phi$ such that $0\in{\hbox{\rm int}\,}G$ and $\phi(x)>\|x\|$ for all $x\in\partial G$.

Note that A and B imply that for all $f\in{\cal C}_{\|\cdot\|}$, the function $f(x)+\phi(x)$ attains its minimum on the set ${\hbox{\rm int}\,}G$. Indeed, for every $x\in\partial G$ we have $f(x)+\phi(x)\geq f(0)-\|x\|+\phi(x)>f(0)+\phi(0)$, so that the (clearly existing) minimizer of $f+\phi$ on $G$ is a point from ${\hbox{\rm int}\,}G$. As a result, for every $f\in{\cal C}_{\|\cdot\|}$ and $x\in{\mathbf{E}}$ one has

and the right hand side minimum is achieved.

Given a function $f\in{\cal C}$, we refer to the function

as to the smoothing of $f$. Observe that by our assumptions on $\phi$ we have

${\cal S}[f](x)=f(x+h(x))+\phi(h(x))$, where $h(x)\in{\hbox{\rm int}\,}G$ is such that

for properly selected $f^{\prime}(x+h(x))\in\partial f(x+h(x))$;

$f(x)\geq{\cal S}[f](x)\geq f(x)-\rho_{\|\cdot\|}(G)$, where

indeed, by A we have $\phi(h)\geq\phi(0)=0$, so that $f(x)=f(x)+\phi(0)\geq{\cal S}[f](x)=f(x+h(x))+\phi(h(x))\geq f(x+h(x))\geq f(x)-\|h(x)\|$ (recall that $f\in{\cal C}_{\|\cdot\|}$), while $h(x)\in G$.

For a proof of (5) see Section A.1 in the Appendix.

2 Approximating a function by smoothing

For $\chi>0$ and $f\in{\cal C}_{\|\cdot\|}$, let

Observe that ${\cal S}[f]_{\chi}(\cdot)$ can be obtained as follows:

We associate with $f\in{\cal C}_{\|\cdot\|}$ the function $f_{\chi}(x)=\chi^{-1}f(\chi x)$; observe that this function belongs to ${\cal C}_{\|\cdot\|}$ along with $f$;

The latter relation combines with (5) to imply that

As bottom-line, if we can find a function $\phi$ as described above we have that for any convex function $f:{\mathbf{E}}\to{\mathbf{R}}$ with Lipschitz constant 1 w.r.t. $\|\cdot\|$ and every $\chi>0$ there exists a smooth (i.e., with Lipschitz continuous gradient) approximation ${\cal S}_{\chi}[f]$ that satisfies:

${\cal S}_{\chi}[f]$ is convex and Lipschitz continuous with constant 1 w.r.t. $\|\cdot\|$ and has a Lipschitz continuous gradient, with constant $M_{\phi}/\chi$, w.r.t. $\|\cdot\|$:

$\sup_{x\in E}|f(x)-{\cal S}_{\chi}[f](x)|\leq\chi\rho_{\|\cdot\|}(G)$. Moreover, $f(x)\geq{\cal S}_{\chi}[f](x)\geq f(x)-\chi\rho_{\|\cdot\|}(G)$.

${\cal S}_{\chi}[f]$ depends on $f$ in a local fashion: the value and the derivative of ${\cal S}_{\chi}[f]$ at $x$ depends only on the restriction of $f$ onto the set $x+\chi G$.

3 Example: p𝑝p-norm smoothing

Let $n>1$ and $p\in[2,\infty]$, and consider the case of ${\mathbf{E}}={\mathbf{R}}^{n}$, endowed with the standard inner product, and $\|\cdot\|=\|\cdot\|_{p}$. Assume for a moment that $p>2$, and let $r$ be a real such that $2<r\leq p$. Let us select $\theta>1$ such that $2\theta/r<1$ and set

Observe that $\phi$ is twice continuously differentiable on ${\hbox{\rm Dom}\,}\phi={\mathbf{R}}^{n}$ function satisfying A. Besides this, $r\leq p$ ensures that $\sum_{j}|x_{j}|^{r}\geq 1$ whenever $\|x\|_{p}=1$, so that $\phi(x)>\|x\|_{p}$ when $x\in\partial G$, which implies B. Besides, by choosing $r=\min[p,3\ln n]$ and $\theta>1$ close enough to 1, C is satisfied for $M_{\phi}=O(1)\min[p,\ln n]$ (for a proof we refer to Section A.2 in the Appendix).

For the case of $p=2$, we can set $\phi(x)=2\|x\|_{2}^{2}$ and, as above, $G=\{x:\|x\|_{2}\leq 1\}$, clearly ensuring A, B, and the validity of $C$ with $M_{\phi}=1$.

Applying the results of the previous section, we get

Let $p\in[2,\infty]$ and $f:{\mathbf{R}}^{n}\to{\mathbf{R}}$ be a Lipschitz continuous, with constant $1$ w.r.t. the norm $\|\cdot\|_{p}$, convex function. For every $\chi>0$, there exists a convex continuously differentiable function ${\cal S}_{\chi}[f](x):{\mathbf{R}}^{n}\to{\mathbf{R}}$ with the following properties:

(i) $f(x)\geq{\cal S}_{\chi}[f](x)\geq f(x)-\chi$, for all $x$;

(ii) $\|\nabla{\cal S}_{\chi}[f](x)-\nabla{\cal S}_{\chi}[f](y)\|_{{p\over p-1}}\leq O(1)\min[p,\ln n]\chi^{-1}\|x-y\|_{p}$ for all $x,y$;

(iii) For every $x$, the restriction of ${\cal S}_{\chi}[f](\cdot)$ on a small enough neighbourhood of $x$ depends solely on the restriction of $f$ on the set

Lower complexity Bounds for Smooth Convex Minimization

In this section we utilize Proposition 1 to prove our main result, namely, a general lower bound on the oracle complexity of smooth convex minimization, and then specify this result for the case of minimization over $\|\cdot\|_{p}$ balls, where $2\leq p\leq\infty$.

$\|\cdot\|$ be a norm on ${\mathbf{R}}^{n}$ and $X$ be a nonempty convex set containing the unit ball of $({\mathbf{R}}^{n},\|\cdot\|)$;

$T$ be a positive integer and $\Delta$ be a positive real with the following property: One can point out $T$ linear forms $\langle\omega_{i},\cdot\rangle$ on ${\mathbf{R}}^{n}$, $1\leq i\leq T$, such that

(a) $\|\omega_{i}\|_{*}\leq 1$ for $i\leq T$, and

(b) for every collection $\xi^{T}=(\xi_{1},...,\xi_{T})$ with $\xi_{i}\in\{-1,1\}$, it holds

$M$ and $\rho$ be positive reals such that for properly selected convex twice continuously differentiable on an open convex set ${\hbox{\rm Dom}\,}\phi\subset{\mathbf{R}}^{n}$ function $\phi$ and a convex compact subset $G\subset{\hbox{\rm Dom}\,}\phi$ the triple $(\phi,G,M_{\phi}=M)$ satisfies properties A, B, C from Section 2.1 and $\rho_{\|\cdot\|}(G)\leq\rho$.

Then for every $L>0$, $\kappa\in(1,2]$, every local oracle ${\cal O}$ and every $T$-step method ${\cal M}$ associated with this oracle there exists a problem $(P_{f,X})$ with $f\in{\cal F}_{\|\cdot\|}(\kappa,L)$ such that

20. Given a permutation $i\mapsto\sigma(i)$ of $\{1,...,T\}$ and a collection $\xi^{T}\in\{-1,1\}^{T}$, we associate with these data the functions

Observe that all these functions belong to ${\cal C}_{\|\cdot\|}$ due to $\|\omega_{j}\|_{*}\leq 1$, for $j\leq T$, so that the smoothed functions

(see Section 2.2) are well defined continuously differentiable convex functions on ${\mathbf{R}}^{n}$ which, by item S.1 in Section 2.2, satisfy that for all $x$, $y$ in $X$

Recalling the definition of $\beta$, we conclude that $f^{\sigma(\cdot),\xi^{T}}(\cdot)\in{\cal F}_{\|\cdot\|}(\kappa,L)$.

30. Given a local oracle ${\cal O}$ and an associated $T$-step method ${\cal M}$, let us define a sequence $x_{1},\ldots,x_{T}$ of points in ${\mathbf{R}}^{n}$, a permutation $\sigma(\cdot)$ of $\{1,...,T\}$ and a collection $\xi^{T}\in\{-1,1\}^{T}$ by the following $T$-step recurrence:

Step 1: $x_{1}$ is the first point of the trajectory of ${\cal M}$ (this point depends solely on the method and is independent of the problem the method is applied to). We define $\sigma(1)$ as the index $i$, $1\leq i\leq T$, that maximizes $|\langle\omega_{i},x_{1}\rangle|$, and specify $\xi_{1}\in\{-1,1\}$ in such a way that $\xi_{1}\langle\omega_{\sigma(1)},x_{1}\rangle=|\langle\omega_{\sigma(1)},x_{1}\rangle|$. We set

Step $t$, $2\leq t\leq T$: At the beginning of this step, we have at our disposal the already built points $x_{\tau}\in{\mathbf{R}}^{n}$, distinct from each other integers $\sigma(\tau)\in\{1,...,T\}$ and quantities $\xi_{\tau}\in\{-1,1\}$, for $1\leq\tau<t$. At step $t$, we build $x_{t}$, $\sigma(t)$, $\xi_{t}$, as follows. We set

thus getting a function from ${\cal C}_{\|\cdot\|}$, and define its smoothing $f^{t-1}(x)=\beta{\cal S}_{\chi}[g^{t-1}](x)$ which, same as above, belongs to ${\cal F}_{\|\cdot\|}(\kappa,L)$. We further define

$x_{t}$ as the $t$-th point of the trajectory of ${\cal M}$ as applied to $f^{t-1}$,

$\sigma(t)$ as the index $i$ that maximizes $|\langle\omega_{i},x_{t}\rangle|$, over $i\leq T$ distinct from $\sigma(1),...,\sigma(t-1)$,

$\xi_{t}\in\{-1,1\}$ such that $\xi_{t}\langle\omega_{\sigma(t)},x_{t}\rangle=|\langle\omega_{\sigma(t)},x_{t}\rangle|$

After $T$ steps of this recurrence, we get at our disposal a sequence $x_{1},\ldots,x_{T}$ of points from ${\mathbf{R}}^{n}$, a permutation $\sigma(\cdot)$ of indexes $1,\ldots,T$ and a collection $\xi^{T}=(\xi_{1},...,\xi_{T})\in\{-1,1\}^{T}$; these entities define the functions

40. We claim that $x_{1},\ldots,x_{T}$ is the trajectory of ${\cal M}$ as applied to $f^{T}$. By construction, $x_{1}$ indeed is the first point of the trajectory of ${\cal M}$ as applied to $f^{T}$. In view of this fact, taking into account the definition of $x_{t}$ and the locality of the oracle ${\cal O}$, all we need to support our claim is to verify that for every $t$, $2\leq t\leq T$, the functions $f^{T}$ and $f^{t-1}$ coincide in some neighbourhood of $x_{t-1}$. By construction, we have that for $t\leq s\leq T$

Since both $g^{t-1}$ and $g_{t}$ belong to ${\cal C}_{\|\cdot\|}$, it follows that $g^{t-1}(x)\geq g_{t}(x)$ in the $\|\cdot\|$-ball $B$ of radius $\delta/2$ centered at $x_{t-1}$, whence, by (12),

From $\chi\rho=\delta/2$ we have that $g^{t-1}\in{\cal C}_{\|\cdot\|}$ and $g^{T}\in{\cal C}_{\|\cdot\|}$ coincide on the set $x_{t-1}+\chi G$, whence, as we know from item S.3 in Section 2.2, $f^{t-1}(\cdot)=\beta{\cal S}_{\chi}[g^{t-1}](\cdot)$ and $f^{T}(\cdot)=\beta{\cal S}_{\chi}[g^{T}](\cdot)$ coincide in a neighbourhood of $x_{t-1}$, as claimed.

whence, by item S.2 in Section 2.2, ${\cal S}_{\chi}[g^{T}](x_{T})\geq-(T-1)\delta-\chi\rho\geq-T\delta=-\Delta/2$, implying that

On the other hand, by (7) there exists $x_{\ast}\in X$ such that $g^{T}(x_{\ast})\leq\max_{1\leq i\leq T}\xi_{i}\langle\omega_{\sigma(i)},x_{\ast}\rangle\leq-\Delta$, whence ${\cal S}_{\chi}[g^{T}](x_{\ast})\leq g^{T}(x_{\ast})\leq-\Delta$ and thus ${\hbox{\rm Opt}}(f^{T})\leq f^{T}(x_{\ast})\leq-\beta\Delta$. Since, as we have seen, $x_{1},\ldots,x_{T}$ is the trajectory of ${\cal M}$ as applied to $f^{T}$, $x_{T}$ is the approximate solution generated by ${\cal M}$ as applied to $f^{T}$, and we see that the inaccuracy of this solution, in terms of the objective, is at least ${\beta\Delta\over 2}={\Delta^{\kappa}\over 2^{\kappa+1}(\rho M)^{\kappa-1}}\cdot{L\over T^{\kappa-1}},$ as required. Besides this, $f^{T}$ is of the form $f^{\sigma(\cdot),\xi^{T}}$, and we have seen that all these functions belong to ${\cal F}_{\|\cdot\|}(\kappa,L)$. ∎

Note that the previous result immediately implies the lower bound

on the complexity of the family ${\cal P}({\cal F}_{\|\cdot\|}(\kappa,L),X)$, provided $X$ contains the unit $\|\cdot\|$-ball. Note that this bound is independent of the local oracle ${\cal O}$.

The case where $X$ contains a $\|\cdot\|$-ball of radius $R>0$ instead of the unit $\|\cdot\|$ ball can be reduced to the latter case by scaling instances $f(\cdot)\mapsto f(\cdot/R)$, which corresponds to the transformation $(\kappa,L)\mapsto(\kappa,\bar{L}:=LR^{\kappa})$ of the smoothness parameters. Thus, assuming that $X$ contains $\|\cdot\|$-ball of radius $R$, we have

Case of ∥⋅∥=∥⋅∥p\|\cdot\|=\|\cdot\|_{p}

In this section we provide lower complexity bounds for smooth convex optimization over $\|\cdot\|_{p}$-balls for the case when $\|\cdot\|=\|\cdot\|_{p}$. In section 4.1 we show that Proposition 2 implies nearly tight optimal complexity bounds for the range $2\leq p\leq\infty$; moreover, for fixed and finite $p$, the bound is tight within a factor depending solely on $p$. For the case $p=\infty$, our lower bound matches the approximation guarantees of the Conditional Gradient algorithm, up to a logarithmic factor, proving near-optimality of the algorithm.

In section 4.2 we study the range $1\leq p<2$. Here we prove nearly optimal complexity bounds by using nearly-Euclidean sections of the $\|\cdot\|_{p}$-ball, together with the $p=\infty$ lower bound.

Consider the case when $\|\cdot\|$ is the norm $\|\cdot\|_{p}$ on ${\mathbf{R}}^{n}$, $2\leq p\leq\infty$. Given positive integer $T\leq n$, let us specify $\omega_{i}$, $1\leq i\leq T$, as the first $T$ standard basic orths, so that for every collection $\xi^{T}\in\{-1,1\}^{T}$ one clearly has

Invoking the results from Section 2.3 (cf. Proposition 1), we see that when $X\subset{\mathbf{R}}^{n}$ is a convex set containing the unit $\|\cdot\|_{p}$-ball, Assumptions II and III in Proposition 2 are satisfied with $M=O(1)\min[p,\ln n]$, $\rho=1$ and $\Delta=T^{-1/p}$. Applying Proposition 2, we arrive at

Let $2\leq p\leq\infty$, $\kappa\in(1,2]$, $L>0$, and let $X\subset{\mathbf{R}}^{n}$ be a convex set containing the unit ball w.r.t. $\|\cdot\|_{p}$. Then, for every $T\leq n$ and every local oracle ${\cal O}$, the minimax risk of the family of problems ${\cal P}({\cal F},X)$ with ${\cal F}={\cal F}_{\|\cdot\|_{p}}(\kappa,L)$ admits the lower bound

independent of the local oracle ${\cal O}$ in use.

Let us discuss some interesting consequences of the above result.

A. Complexity of smooth minimization over the box: Corollary 1 implies that when $X$ is the unit $\|\cdot\|_{\infty}$-ball in ${\mathbf{R}}^{n}$, the $T$-step minimax risk ${\hbox{\rm Risk}}_{{\cal F},X,{\cal O}}(T)$ of minimizing over $X$ of objectives from the family ${\cal F}={\cal F}_{\|\cdot\|_{\infty}}(\kappa,L)$ in the range $T\leq n$ is lower-bounded by $\Omega(1/\ln n)L/T^{\kappa-1}$. On the other hand, from the standard efficiency estimate of Conditional Gradient algorithm (see, e.g., ) it follows that when applying the method to minimizing over $X$ a function $f\in{\cal F}_{\|\cdot\|_{\infty}}(\kappa,L)$ over a convex compact domain $X$ of $\|\cdot\|_{\infty}$-diameter $2R$, the inaccuracy after $T=1,2,...$ steps does not exceed

We see that when $X$ is in-between two $\|\cdot\|_{\infty}$-balls with ratio of sizes $\theta$, the lower complexity bound coincides with the upper one within the factor $O(1)\theta^{\kappa}\ln^{\kappa-1}(n)$. In particular, when minimizing functions $f\in{\cal F}_{\|\cdot\|_{\infty}}(\kappa,L)$ over $n$-dimensional unit box $X$, the performance of the Conditional Gradient algorithm, as expressed by its minimax risk, cannot be improved by more than $O(\ln^{\kappa-1}(n))$ factor, for any local oracle in use. In fact, the same conclusion remains true when $\|\cdot\|_{\infty}$ and the unit box $X$ are replaced with $\|\cdot\|_{p}$ and the unit $\|\cdot\|_{p}$-ball with “large” $p$, specifically, $p\geq\Omega(1)\ln n$.

B. Tightness: In fact, in the case of $2\leq p<\infty$ the lower complexity bounds for smooth convex minimization over $\|\cdot\|_{p}$-balls established in Corollary 1, are tight: it is shown in , see also [11, Section 2.3] that a properly modified Nesterov’s algorithm ${\cal N}$ for smooth convex optimization via the first-order oracle, as applied to problems of minimizing functions $f$ from ${\cal F}_{\|\cdot\|_{p}}(\kappa,L)$ over the $n$-dimensional unit $\|\cdot\|_{p}$-ball $X$, for any number $T\geq 1$ of steps ensures that

with $C(p)$ depending solely on $p$, which is in full accordance with (15).

2 Smooth Convex Minimization over ∥⋅∥p\|\cdot\|_{p}-balls, 1≤p≤21𝑝21\leq p\leq 2

We have obtained lower complexity bounds for smooth convex minimization over $\|\cdot\|_{p}$-balls, where $2\leq p\leq\infty$. Now we consider the case $1\leq p<2$. We will build nearly tight bounds by reducing to the case of $p=\infty$.

Let $1\leq p\leq 2$, $\kappa\in(1,2]$, $L>0$, and let $X\subset{\mathbf{R}}^{n}$ be a convex set containing the unit $\|\cdot\|_{p}$-ball. For properly selected absolute constant $\alpha\in(0,1)$ and for every $T\leq\alpha n$, the minimax risk of the family of problems ${\cal P}({\cal F},X)$ with ${\cal F}={\cal F}_{\|\cdot\|_{p}}(\kappa,L)$ admits the lower bound

independent of the local oracle ${\cal O}$ in use.

Proof. 10. By Dvoretzky’s Theorem for the $\|\cdot\|_{p}$-ball [20, Theorem 4.15], there exists an absolute constant $\alpha\in(0,1)$, such that for any positive integer $T\leq\alpha n$ there is a subspace $M\subseteq{\mathbf{R}}^{n}$ of dimension $T$, and a centered at the origin ellipsoid $E\subseteq M$, such that

Let $\{\gamma_{i}(\cdot):\,i=1,\ldots,T\}$ be linear forms on $M$ such that $E=\{y\in M:\,\sum_{i=1}^{T}\gamma_{i}^{2}(y)\leq 1\}$. By the second inclusion in (17), for every $i$, the maximum of the linear form $\gamma_{i}(\cdot)$ over $B_{M}$ does not exceed $1$, whence, by the Hahn-Banach Theorem, the form $\gamma_{i}(\cdot)$ can be extended from $M$ to a linear form on the entire ${\mathbf{R}}^{n}$ to have the maximum over $B:=\{x:\|x\|_{p}\leq 1\}$ not exceeding $1$. In other words, we can point out vectors $g_{i}\in{\mathbf{R}}^{n}$, $1\leq i\leq T$ such that $\gamma_{i}(y)=\langle g_{i},y\rangle$ for every $y\in M$ and $\|g_{i}\|_{{p\over p-1}}\leq 1$, for all $1\leq i\leq T$. Now consider the linear mapping

20.Observe that an optimization problem of the form

and when the objective of the former problem belongs to ${\cal F}^{T}:={\cal F}^{T}_{\|\cdot\|_{\infty}}(\kappa,L)$, the objective of the latter problem belongs to ${\cal F}^{n}={\cal F}^{n}_{\|\cdot\|_{p}}(\kappa,L)$. It is intuitively clear that the outlined reducibility implies that the complexity of solving problems from the family $\Phi_{n}:=\{(P_{f,X}):f\in{\cal F}^{n}\}$ cannot be smaller than the complexity of solving problems from the family $\Phi_{T}:=\{(P_{f,Y}):f\in{\cal F}^{T}\}$. Taking this claim for granted (for a proof, see section A.3), let us derive from it the desired result. To this end, observe that from the first inclusion in (17) it follows that $Y$ contains the centered at the origin $\|\cdot\|_{\infty}$-ball of radius $R={1\over 2\sqrt{T}}$ (indeed, by construction this ball is already contained in the image of ${1\over 2}E\subset X$). By Corollary 1 as applied to $p=\infty$ and to $Y$ in the role of $X$, the worst-case, w.r.t. problems from the family $\Phi_{T}$, inaccuracy of any $T$-step method based on a local oracle is at least

see (13). According to our claim, the latter quantity lower-bounds the worst-case, w.r.t. problems from the family $\Phi_{n}$, inaccuracy of any $T$-step method based on a local oracle, and (16) follows. ∎

Finally, we remark that the lower complexity bound stated in Proposition 3 in the smooth case $\kappa>1$ is, to the best of our knowledge, new (the nonsmooth case $\kappa=1$ was considered already in ). This lower bound matches, up to logarithmic in $n$ factors, the upper complexity bound for the family in question, see .

3 Matrix case

We have proved lower bounds for smooth optimization over $\|\cdot\|_{p}$-balls for all $1\leq p\leq\infty$. Now we show how these bounds can be used for proving lower complexity bounds on smooth convex minimization over Schatten norm balls in the spaces of matrices. Recall that the Shatten $p$-norm $\|x\|_{{\hbox{\tiny\rm Sch}},p}$ of an $n\times n$ matrix $x$ is, by definition the $p$-norm of the vector of singular values of $x$. The problems we are interested in now are of the form

where $f\in{\cal F}_{\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}}(\kappa,L)$.

Observe that Corollary 1 remains true when replacing in it the embedding space ${\mathbf{E}}={\mathbf{R}}^{n}$ of $X$ with the space ${\mathbf{E}}={\mathbf{R}}^{n\times n}$ of $n\times n$ matrices, the norm $\|\cdot\|_{p}$ on ${\mathbf{R}}^{n}$ with the Schatten norm $\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}$, and the requirement “$X\subset{\mathbf{R}}^{n}$ is a convex set containing the unit ball of $\|\cdot\|_{p}$” with the requirement “$X\subset{\mathbf{R}}^{n\times n}$ is a convex set containing the unit ball of $\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}$.” This claim is an immediate consequence of the fact that when restricting an $n\times n$ matrix onto its diagonal, we get a linear mapping of ${\mathbf{R}}^{n\times n}$ onto ${\mathbf{R}}^{n}$, and the factor norm on ${\mathbf{R}}^{n}$ induced, via this mapping, by $\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}$ is nothing but the usual $\|\cdot\|_{p}$-norm. Consequently, minimizing a function from ${\cal F}_{\|\cdot\|_{p}}(\kappa,L)$ over the unit $\|\cdot\|_{p}$ ball $X$ of ${\mathbf{R}}^{n}$ reduces to minimizing a convex function of exactly the same smoothness, as measured w.r.t. $\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}$, over the unit Schatten $p$-norm ball $X^{+}$ of ${\mathbf{R}}^{n\times n}$. As a result, every universal (i.e., valid for every local oracle) lower bound on the minimax risk for the problem class ${\cal P}({\cal F}_{\|\cdot\|_{p}}(\kappa,L),X)$ automatically is a universal lower bound on the minimax risk for the problem class ${\cal P}({\cal F}_{\|\cdot\|_{{\hbox{\tiny\rm Sch}},p}}(\kappa,L),X^{+})$.

Note, however, that the “matrix extension” of our lower complexity bounds is not “completely costless” – the resulting bounds are applicable when $T\leq n$ ($p\geq 2$) of $T\leq O(1)n$ ($1\leq p\leq 2$), and $n$ now is the square root of the actual dimension of $x$. Thus, in the matrix case our lower complexity bounds are applicable in relatively more narrow range of values of $T$.

References

Appendix A Appendix

In order to prove (5), by the standard approximation argument, it suffices to establish this relation in the case when, in addition to the inclusion $f\in{\cal C}_{\|\cdot\|}$ and the assumptions A – C on $\phi$, $f$ and $\phi$ are C∞ smooth and $\phi$ is strongly convex. By (4),

where $h:E\to G$ is well defined and solves the nonlinear system of equations

We have ${\partial F(x,h)\over\partial h}=f^{\prime\prime}(x+h)+\phi^{\prime\prime}(h)\succ 0$, implying by the Implicit Function Theorem that $h(x)$ is smooth. Differentiating the identity $F(x,h(x))\equiv 0$, we get

On the other hand, differentiating (19), we get

As a result, for all $e$, $x$, we have, taking into account that $P$, $Q$ are symmetric positive definite,

A.2 Proof for section 2.3

(we used that $2\theta/r<1$ in (21) and the Hölder inequality in (A.2)). By continuity, the resulting inequality holds true when $x=0$ as well.

Now let us set $r=\min[p,3\ln n]$. When $p\leq 3\ln n$, we have $r=p$, and (23) reads

expressing the fact that $\phi$, $G$ satisfy assumption C with $M_{\phi}=5p$, provided that $1<\theta\leq 5/4$. When $p>3\ln n$, we have $r=3\ln n$, whence $\|e\|_{r}\leq n^{{1\over r}-{1\over p}}\|e\|_{p}\leq\exp\{1/3\}\|e\|_{p}$, so that for $\theta>1$ close enough to 1 (what is “close enough”, depends solely on $n$) (23) reads

expressing the fact that $\phi$, $G$ satisfy assumption C with $M_{\phi}=15\exp\{2/3\}\ln(n)$. Thus, $\phi$, $G$ satisfy assumption C with $M_{\phi}=O(1)\min[p,\ln(n)]$.

A.3 Item 20 of the proof of Proposition 3

Observe, first, that the claim we intend to justify indeed needs a justification: we cannot just argue that solving “lifted” problems – those from the family $\Phi_{n}^{+}=\{(P_{f^{+},X}):f\in{\cal F}^{T}_{\|\cdot\|_{\infty}}(\kappa,L)\}\subset\Phi_{n}$ – cannot be simpler than solving problems from $\Phi_{T}$ due to the fact that the problems from the latter family can be reduced to those from the former one; we should specify the local oracles associated with the families in question, and to ensure that “lifting” does not simplify problems just because the oracle for the “lifted” family is more informative than the oracle for the original family. The justification here is as follows: observe that among local oracles for families of real-valued functions on ${\mathbf{R}}^{m}$ there is the “most informative” one, let us call it maximal; when queried about a function $f$ at a point $x$, the maximal oracle returns the class $\widetilde{f}$ of $f$ w.r.t. the equivalence relation “$f$ is equivalent to $g$ if and only if $f$ and $g$ coincide with each other in some (perhaps depending on $f$ and $g$) neighbourhood of $x$.” Clearly the maximal oracle allows to mimic any other local oracle, so that for every family of problems, the lower complexity bounds valid for the maximal oracle are valid for any other local oracle. Now, it is easily seen that the maximal oracle for the family of functions ${\cal F}^{T}$ induces the maximal oracle for the lifted family $\{f^{+}:f\in{\cal F}^{T}\}$; with this in mind, it is immediately seen that any maximal-oracle-based $T$-step method ${\cal M}^{+}$ for solving problems from the family $\Phi_{n}^{+}$ induces a maximal-oracle-based $T$-step method ${\cal M}$ for solving problems from the family $\Phi^{T}$ in such a way that the trajectory $x_{1},x_{2},\ldots$ of ${\cal M}$ on a problem $(P_{f,Y})$ is linked to the trajectory $x_{1}^{+},x_{2}^{+},\ldots$ of ${\cal M}^{+}$ on $(P_{f_{+},X})$ by the relation $x_{t}=Gx_{t}^{+}$. Consequently, when the maximal oracles are used, a lower bound on the $T$-step minimax risk of $\Phi_{T}$ automatically is a lower bound on the same quantity for the family $\Phi_{n}^{+}=\{(P_{f_{+},X}):f\in{\cal F}^{T}\}$, and therefore for the larger family $\Phi_{n}$. In particular, the quantity (18), which by Corollary 1 lower-bounds the maximal-oracle-based $T$-step minimax risk when solving problems from $\Phi_{T}$, lower-bounds the similar quantity for $\Phi_{n}$, and thus - the $T$-step minimax risk of $\Phi_{n}$ taken w.r.t. any local oracle, as claimed.