DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

Introduction

We study the fundamental optimization problem

As models become larger and more complex, the cost and environmental impact of training have rapidly grown as well (Sharir et al., 2020; Patterson et al., 2021). Therefore, it is vital that we develop more efficient and effective methods of solving machine learning optimization tasks. One of the chief challenges in applying gradient-based methods is that they often require tuning one or more stepsize parameters (Goodfellow et al., 2016), and the choice of stepsize can significantly influence a method’s convergence speed as well as the quality of the obtained solutions, especially in deep learning (Wilson et al., 2017).

The cost and impact of hyperparameter tuning on the optimization process have led to significant research activity in designing parameter-free and adaptive optimization methods in recent years, see e.g. (Orabona and Cutkosky, 2020; Carmon and Hinder, 2022) and the references therein.

We say an algorithm is universal if it adapts to many different problem geometries or regularity conditions on the function $f$ (Nesterov, 2014; Levy et al., 2018; Grimmer, 2022). In this work, we focus on two regularity conditions: (a) $f$ Lipschitz and (b) $f$ smooth. Lipschitz functions have a bounded rate of change, that is, there exists some $G>0$ such that for all $x,y\in\mathcal{X}$ we have $\absolutevalue{f(x)-f(y)}\leq G\norm{x-y}$. The Lipschitz property is beneficial for the convergence of gradient-based optimization algorithms. They converge even faster on smooth functions, which have continuous derivatives; that is, there exists some $L>0$ such that for all $x,y\in\mathcal{X}$ we have $\norm{\nabla f(x)-\nabla f(y)}\leq L\norm{x-y}$. Smoothness leads to faster convergence of gradient-based methods than the Lipschitz property. Universality is a highly desirable property because in practice the same optimization algorithms are often used for both smooth and nonsmooth optimization (e.g. optimizing both ReLU and smooth networks).

The main question of our work is as follows:

Can we design a universal, parameter-free gradient descent method for (OPT)?

Existing universal variants of gradient descent either rely on line search (Nesterov, 2014; Grimmer, 2022), bisection subroutines (Carmon and Hinder, 2022), or are not parameter-free (Hazan and Kakade, 2019; Levy et al., 2018; Kavis et al., 2019). Line search algorithms are theoretically strong, achieving the optimal convergence rates in both the nonsmooth and smooth settings with only an extra log factor. Through an elegant application of bisection search, Carmon and Hinder (2022) design a parameter-free method whose convergence is only double-logarithmically worse than gradient descent with known problem parameters. However, this method requires resets, i.e. restarting the optimization process many times, which can be very expensive in practice. Therefore, we seek a universal, parameter-free gradient descent method for (OPT) with no search subroutines.

Our contributions. We provide a new algorithm that meets the above requirements. Our main contribution is a new universal, parameter-free gradient descent method with no search subroutines. Building upon the recently proposed Distance-over-Gradients (DoG) algorithm (Ivgi et al., 2023), we develop a new method, DoWG (Algorithm 1), that uses a different stepsize with adaptively weighted gradients. We show that DoWG automatically matches the performance of gradient descent on (OPT) up to logarithmic factors with no stepsize tuning at all. This holds in both the nonsmooth setting (Theorem 3) and the smooth setting (Theorem 4). Finally, we show that DoWG is competitive on real machine learning tasks (see Section 4).

Related Work

There is a lot of work on adaptive and parameter-free approaches for optimization. We summarize the main properties of the algorithms we compare against in Table 1. We enumerate some of the major approaches below:

Polyak stepsize. When $f_{*}=f(x_{*})$ is known, the Polyak stepsize (Polyak, 1987) is a theoretically-grounded, adaptive, and universal method (Hazan and Kakade, 2019). When $f_{*}$ is not known, Hazan and Kakade (2019) show that an adaptive re-estimation procedure can recover the optimal convergence rate up to a log factor when $f$ is Lipschitz. Loizou et al. (2021) study the Polyak stepsize in stochastic non-convex optimization. Orvieto et al. (2022) show that a variant of the Polyak stepsize with decreasing stepsizes can recover the convergence rate of gradient descent, provided the stepsize is initialized properly. Unfortunately, this initialization requirement makes the method not parameter-free.

The doubling trick. The simplest way to make an algorithm parameter-free is the doubling-trick. For example, for gradient descent for $L$-smooth and convex optimization, the stepsize $\eta=\frac{1}{L}$ results in the convergence rate of

where $D_{0}=\norm{x_{0}-x_{\ast}}$. We may therefore start with a small estimate $L_{0}$ of the smoothness constant $L$, run gradient descent for $T$ steps, and return the average point. We restart and repeat this for $N$ times, and return the point with the minimum function value. So long as $N\geq\log\frac{L}{L_{0}}$, we will return a point with loss satisfying eq. 1 at the cost of only an additional logarithmic factor. This trick and similar variants of it appear in the literature on prediction with expert advice and online learning (Cesa-Bianchi et al., 1997; Cesa-Bianchi and Lugosi, 2006; Hazan and Megiddo, 2007). It is not even needed to estimate $N$ in some cases, as the restarting can be done adaptively (Streeter and McMahan, 2012). In practice, however, the performance of doubling trick suffers from restarting the optimization process and throwing away useful that could be used to guide the algorithm.

Parameter-free methods. Throughout this paper, we use the term “parameter-free algorithms” to describe optimization algorithms that do not have any tuning parameters. We specifically consider only the deterministic setting with a compact domain. As mentioned before, Carmon and Hinder (2022) develop an elegant parameter-free and adaptive method based on bisection search. Bisection search, similar to grid search, throws away the progress of several optimization runs and restarts, which may hinder their practical performance. Ivgi et al. (2023); Defazio and Mishchenko (2023) recently developed variants of gradient descent that are parameter-free when $f$ is Lipschitz. However, D-Adaptation (Defazio and Mishchenko, 2023) has no known guarantee under smoothness, while DoG (Ivgi et al., 2023) was only recently (after the initial release of this paper) shown to adapt to smoothness. We compare against the convergence guarantees of DoG in Section 3.2. For smooth functions, Malitsky and Mishchenko (2020) develop AdGD, a method that efficiently estimates the smoothness parameter on-the-fly from the training trajectory. AdGD is parameter-free and matches the convergence of gradient descent but has no known guarantees for certain classes of Lipschitz functions. A proximal extension of this method has been proposed by Latafat et al. (2023).

Parameter-free methods in online learning. In the online learning literature, the term “parameter-free algorithms” was originally used to describe another class of algorithms that adapt to the unknown distance to the optimal solution (but can still have other tuning parameters such as Lipschitz constant). When the Lipschitz parameter is known, approaches from online convex optimization such as coin betting (Orabona and Pál, 2016), exponentiated gradient (Streeter and McMahan, 2012; Orabona, 2013), and others (McMahan and Orabona, 2014; Orabona and Cutkosky, 2020; Orabona and Pál, 2021; Orabona and Tommasi, 2017) yield rates that match gradient descent up to logarithmic factors. Knowledge of the Lipschitz constant can be removed either by using careful restarting schemes (Mhammedi et al., 2019; Mhammedi and Koolen, 2020), or adaptive clipping on top of coin betting (Cutkosky, 2019). For optimization in the deterministic setting, it is later clarified that, by leveraging the normalization techniques developed in (Levy, 2017), the aforementioned online learning algorithms can be used without knowing other tuning parameters (i.e., achieve “parameter-free” in the sense of this paper) for optimizing both Lipschitz functions (Orabona and Pál, 2021) and smooth functions (Orabona, 2023). Concretely, as shown in Orabona (2023) (which appears after the initial release of this paper), combining algorithms in Streeter and McMahan (2012); Orabona and Pál (2016) with normalization techniques (Levy, 2017) yields new algorithms that are also search-free, parameter-free (in the sense of this paper), and universal. However, these algorithms are rather different from DoWG in algorithmic style: these algorithms only use normalized gradients while DoWG does use the magnitudes of the gradients; DoWG falls in the category of gradient descent algorithms with adaptive learning rate, while these algorithms do not.

Line search. As mentioned before, line-search-based algorithms are universal and theoretically grounded (Nesterov, 2014) but are often expensive in practice (Malitsky and Mishchenko, 2020).

AdaGrad family of methods. Li and Orabona (2019) study a variant of the AdaGrad stepsizes in the stochastic convex and non-convex optimization and show convergence when the stepsize is tuned to depend on the smoothness constant. Levy et al. (2018) show that when the stepsize is tuned properly to the diameter of the domain $\mathcal{X}$ in the constrained convex case, AdaGrad-Norm adapts to smoothness. Ene et al. (2021) extend this to AdaGrad and other algorithms, and also to variational inequalities. Ward et al. (2019); Traoré and Pauwels (2021) show the convergence of AdaGrad-Norm for any stepsize for non-convex (resp. convex) optimization, but in the worst case the dependence on the smoothness constant is worse than gradient descent. Liu et al. (2022) show that AdaGrad-Norm converges in the unconstrained setting when $f$ is quasi-convex, but their guarantee is worse than gradient descent. We remark that all AdaGrad-style algorithms mentioned above require tuning stepsizes, and are thus not parameter-free.

Alternative justifications for normalization. There are other justifications for why adaptive methods work outside of universality. Zhang et al. (2020a) study a generalized smoothness condition and show that in this setting tuned clipped gradient descent can outperform gradient descent. Because the effective stepsize used in clipped gradient descent is only a constant factor away from the effective stepsize in normalized gradient descent, (Zhang et al., 2020a), also show that this improvement holds for NGD. Zhang et al. (2020b) observe that gradient clipping and normalization methods outperform SGD when the stochastic gradient noise distribution is heavy-tailed. However, Kunstner et al. (2023) later observe that adaptive methods still do well even when the effect of the noise is limited.

Algorithms and theory

In this section we first review the different forms of adaptivity in gradient descent and normalized gradient descent, and then introduce our proposed algorithm DoWG. The roadmap for the rest of the paper is as follows: we first review the convergence of gradient descent in the Lipschitz and smooth settings, and highlight the problem of divergence under stepsize misspecification, and how normalization fixes that. Then, we introduce our main new algorithm, DoWG, and give our main theoretical guarantees for the algorithm. Finally, we evaluate the performance of DoWG on practical machine learning problems.

We start our investigation with the standard Gradient Descent (GD) algorithm:

Suppose that $f$ is convex with minimizer $x_{\ast}$. Let $f_{\ast}=f(x_{\ast})$. Let $D_{0}\overset{\text{def}}{=}\norm{x_{0}-x_{\ast}}$ be the initial distance to the optimum. Denote by $\hat{x}_{T}=\frac{1}{T}\sum_{t=0}^{T-1}x_{t}$ the average iterate returned by GD. Then:

(Bubeck, 2015) If $f$ is $G$-Lipschitz, the average iterate satisfies for any stepsize $\eta>0$:

(Nesterov, 2018) If $f$ is $L$-smooth, then for all $\eta<\frac{2}{L}$ the average iterate satisfies

Minimizing eq. 2 over $\eta$ gives $f(\bar{x}_{T})-f_{*}=\mathcal{O}\left(\frac{D_{0}G}{\sqrt{T}}\right)$ with $\eta=\frac{D_{0}}{G\sqrt{T}}$. We have several remarks to make about this rate for gradient descent. First, the optimal stepsize depends on both the distance to the optimum $D_{0}$ and the Lipschitz constant $G$, and in fact, this rate is in general optimal (Nesterov, 2018, Theorem 3.2.1). Moreover, if we misspecify $D_{0}$ or $G$ while tuning $\eta$, this does not in general result in divergence but may result in a slower rate of convergence. On the other hand, for the smooth setting the optimal stepsize is $\eta=\frac{1}{L}$ for which $f(x_{T})-f_{*}\leq\mathcal{O}\left(\frac{LD_{0}^{2}}{T}\right)$. Unfortunately, to obtain this rate we have to estimate the smoothness constant $L$ in order to choose a stepsize $\eta<\frac{2}{L}$, and this dependence is hard: if we overshoot the upper bound $\frac{2}{L}$, the iterations of gradient descent can diverge very quickly, as shown by Figure 1. Therefore, GD with a constant stepsize cannot be universal: we have to set the stepsize differently for smooth and nonsmooth objectives.

Normalized Gradient Descent (NGD) (Shor, 2012) consists of iterates of the form

Under the same conditions as Theorem 1, the iterations generated by generated by (NGD) satisfy after $T$ steps satisfy:

(Nesterov, 2018) If $f$ is $G$-Lipschitz, the minimal function suboptimality satisfies

where $D_{0}\overset{\text{def}}{=}\norm{x_{0}-x_{*}}$.

(Levy, 2017; Grimmer, 2019) If $f$ is $L$-Lipschitz, the minimal function suboptimality satisfies

Tuning eq. 4 in $\eta$ gives $\eta=\frac{D_{0}}{\sqrt{T}}$, and the stepsize is also optimal for eq. 5. This gives a convergence rate of $\frac{D_{0}G}{\sqrt{T}}$ when $f$ is Lipschitz and $\frac{D_{0}^{2}L}{T}$ when $f$ is smooth. Observe that NGD matches the dependence of gradient descent on $G$ and $L$ without any knowledge of it. Furthermore that, unlike GD where the optimal stepsize is $\frac{1}{L}$ in the smooth setting and $\frac{D_{0}}{G\sqrt{T}}$ in the nonsmooth setting. The optimal stepsize for NGD is the same in both cases. Therefore, NGD is universal: the same method with the same stepsize adapts to nonsmooth and smooth objectives. Moreover, misspecification of the stepsize in NGD does not result in divergence, but just slower convergence. Another interesting property is that we only get a guarantee on the best iterate: this might be because NGD is non-monotonic, as Figure 2 (a) shows.

Theorem 2 offers an alternative, global, and less explicit explanation of this phenomenon: NGD matches the optimal gradient descent rate, and in order to do so it must drive the effective stepsize to be large. Specifically, suppose that we use the optimal stepsize $\eta=\frac{D_{0}}{\sqrt{T}}$, and call the best iterate returned by NGD $x_{\tau}$. Then $x_{\tau}$ satisfies $f(x_{\tau})-f_{\ast}\leq\frac{D_{0}^{2}L}{T}$ and therefore by smoothness

2 DoWG

We saw in the last section that NGD adapts to both the Lipschitz constant $G$ and the smoothness $L$, but we have to choose $\eta$ to vary with the distance to the optimum $D_{0}=\norm{x_{0}-x_{*}}$. In this section, we develop a novel algorithm that adaptively estimates the distance to the optimum, and attains the optimal convergence rate of gradient descent for constrained convex and smooth optimization up to a logarithmic factor. Our algorithm builds upon the recently proposed Distance over Gradients (DoG) algorithm developed by Ivgi et al. (2023). We call the new method DoWG (Distance over Weighted Gradients), and we describe it as Algorithm 1 below.

DoWG uses the same idea of estimating the distance from the optimum by using the distance from the initial point as a surrogate, but instead of using the square root of the running gradient sum $G_{t}=\sum_{k=0}^{t}{\left\lVert\nabla f(x_{k})\right\rVert}^{2}$ as the normalization, DoWG uses the square root of the weighted gradient sum $v_{t}=\sum_{k=0}^{t}\bar{r}_{k}^{2}{\left\lVert\nabla f(x_{k})\right\rVert}^{2}$. Observe that because the estimated distances $\bar{r}_{t}$ are monotonically increasing, later gradients have a larger impact on $v_{t}$ than earlier ones compared to $G_{t}$. Therefore, we may expect this to aid the method in adapting to the local properties of the problem once far away from the initialization $x_{0}$. We note that using a weighted sum of gradients is not new: AcceleGrad (Levy et al., 2018) uses time-varying polynomial weights and Adam (Kingma and Ba, 2015) uses exponentially decreasing weights. The difference is that DoWG chooses the weights adaptively based on the running distance from the initial point. This use of distance-based weighted averaging is new, and we are not aware of any previous methods that estimate the running gradient sum in this manner.

Nonsmooth analysis. The next theorem shows that DoWG adapts to the Lipschitz constant $G$ and the diameter $D$ of the set $\mathcal{X}$ if the function $f$ is nonsmooth but $G$-Lipschitz. We use the notation $\log_{+}x=\log x+1$ following (Ivgi et al., 2023).

(DoWG, Lipschitz $f$). Suppose that the function $f$ is convex, $G$-Lipschitz, and has a minimizer $x_{*}\in\mathcal{X}$. Suppose that the domain $\mathcal{X}$ is a closed convex set of (unknown) diameter $D>0$. Let $r_{\epsilon}<D$. Then the output of Algorithm 1 satisfies for some $t\in\left\{0,1,\ldots,T-1\right\}$

where $\bar{x}_{t}\overset{\text{def}}{=}\frac{1}{\sum_{k=0}^{t-1}\overline{r}_{k}^{2}}\sum_{k=0}^{t-1}\overline{r}_{k}^{2}x_{k}$ is a weighted average of the iterates returned by the algorithm.

Discussion of convergence rate. DoWG matches the optimal $\mathcal{O}\left(\frac{DG}{\sqrt{T}}\right)$ rate of tuned GD and tuned NGD up to an extra logarithmic factor. We note that the recently proposed algorithms DoG (Ivgi et al., 2023) and D-Adaptation (Defazio and Mishchenko, 2023) achieve a similar rate in this setting.

Comparison with DoG. As we discussed before, DoWG uses an adaptively weighted sum of gradients for normalization compared to the simple sum used by DoG. In addition, DoG uses the stepsize $\frac{\bar{r}_{t}}{\sqrt{\sum_{k=0}^{t}{\left\lVert\nabla f(x_{k})\right\rVert}^{2}}}$, whereas the DoWG stepsize is pointwise larger: since $\bar{r}_{k}^{2}$ is monotonically increasing in $k$ we have

Of course, the pointwise comparison may not reflect the practical performance of the algorithms, since after the first iteration the sequence of iterates $x_{2},x_{3},\ldots$ generated by the two algorithms can be very different. We observe in practice that DoWG is in general more aggressive, and uses larger stepsizes than both DoG and D-Adaptation (see Section 4).

Smooth analysis. Our next theorem shows that DoWG adapts to the smoothness constant and the diameter $D$ of the set $\mathcal{X}$.

(DoWG, Smooth $f$). Suppose that the function $f$ is $L$-smooth, convex, and has a minimizer $x_{*}\in\mathcal{X}$. Suppose that the domain $\mathcal{X}$ is a closed convex set of diameter $D>0$. Let $r_{\epsilon}<D$. Then the output of Algorithm 1 satisfies for some $t\in\left\{0,1,\ldots,T-1\right\}$

where $\bar{x}_{t}\overset{\text{def}}{=}\frac{1}{\sum_{k=0}^{t-1}\overline{r}_{k}^{2}}\sum_{k=0}^{t-1}\overline{r}_{k}^{2}x_{k}$ is a weighted average of the iterates returned by the algorithm.

The proof of this theorem and all subsequent results is relegated to the supplementary material. We note that the proof of Theorem 4 uses the same trick used to show the adaptivity of NGD to smoothness: we use the fact that $\norm{\nabla f(x)}\leq\sqrt{2L(f(x)-f_{*})}$ for all $x\in\mathcal{X}$ applied to a carefully-chosen weighted sum of gradients.

Comparison with GD/NGD. Both well-tuned gradient descent and normalized gradient descent achieve the convergence rate $\mathcal{O}\left(\frac{LD_{0}^{2}}{T}\right)$ where $D_{0}=\norm{x_{0}-x_{*}}\leq D$ for the constrained convex minimization problem. Theorem 4 shows that DoWG essentially attains the same rate up to the difference between $D_{0}$ and $D$ and an extra logarithmic factor. In the worst case, if we initialize far from the optimum, we have $D_{0}\simeq D$ and hence the difference is not significant. We note that DoG (Ivgi et al., 2023) suffers from a similar dependence on the diameter $D$ of $\mathcal{X}$, and can diverge in the unconstrained setting, where $\mathcal{X}$ is not compact. This can be alleviated by making the stepsize smaller by a polylogarithmic factor. A similar reduction of the stepsize also works for DoWG, and we provide the proof in Section 7 in the supplementary.

Comparison with DoG. After the initial version of this paper, Ivgi et al. (2023) reported a convergence guarantee for the unweighted average $\hat{x}_{T}=\frac{1}{T}\sum_{k=0}^{T-1}x_{k}$ returned by DoG. In particular, Proposition 3 in their work gives the rate

where $D_{0}=\norm{x_{0}-x_{\ast}}$, and where in the second step we used the bound $D_{0}\leq D$ and $\bar{r}_{T}\leq D$. This rate is the same as that achieved by the weighted average of the DoWG iterates up to an extra logarithmic factor $\log_{+}\frac{D}{\bar{r}_{\epsilon}}$. We note that DoG also has a guarantee in the stochastic setting, provided the gradients are bounded locally with a known constant, while in this work we have focused exclusively on the deterministic setting.

Edge of Stability. Like NGD, DoWG also tends to increase the stepsize and train at the edge of stability. The intuition from NGD carries over: in order to preserve the convergence rate of GD, DoWG tends to drive the stepsize larger. However, once it overshoots, the gradients quickly diverge, forcing the stepsize back down. Figure 3 shows the performance of DoWG and its stepsize on the same regularized linear regression problem as in Figure 2. Comparing the two figures, we observe that DoWG is also non-monotonic and trains close to the edge of stability, but its stepsize oscillates less than NGD’s effective stepsize.

Universality. Theorems 4 and 3 together show that DoWG is universal, i.e. it almost recovers the convergence of gradient descent with tuned stepsizes in both the smooth and nonsmooth settings. As the optimal stepsize for gradient descent can differ significantly between the two settings, we believe that achieving both rates simultaneously without any parameter-tuning or search procedures is a significant strength of DoWG.

Comparison with other universal methods. Nesterov (2014) developed a universal method for convex optimization based on a modified line search that is almost parameter-free, requiring only a small initial estimate of the smoothness or Lipschitz constants. Grimmer (2022) later extended the analysis of Nesterov’s method to minimizing finite-sums. Levy et al. (2018) showed that properly-tuned AdaGrad-Norm and an accelerated variant of it are universal. However, neither AdaGrad-Norm nor its accelerated variant is parameter-free. Hazan and Kakade (2019) show that the Polyak stepsize is universal, provided $f_{*}$ is known. Kavis et al. (2019) also develop a universal method, UniXGrad, but it requires estimating the distance to the optimum $D_{0}=\norm{x_{0}-x_{*}}$. Carmon and Hinder (2022) also develop a universal method that is almost-optimal using bisection search. However, their method requires an initial stepsize $\eta_{0}$ to satisfy $\eta_{0}\leq\frac{1}{2L}$, though we can get around this requirement by choosing $\eta_{0}$ to be very small, only paying a $\log\log\frac{1}{\eta_{0}}$ penalty. In contrast, the only initialization DoWG requires is choosing $r_{\epsilon}\leq D$. This can be done by choosing $r_{\epsilon}\leq\norm{x-y}$ for any two $x\neq y\in\mathcal{X}$.

Experimental results

We compare DoWG to DoG, L-DoG from Ivgi et al. (2023), for all of which we also report performance of the polynomially-averaged iterate with power 8 as recommended by Ivgi et al. (2023). We also add comparison against Adam (Kingma and Ba, 2015) with cosine annealing and the standard step size $10^{-3}$. All methods are used with batch size 256 with no weight decay on a single RTX3090 GPU. We plot the results in Figure 4 with the results averaged over 8 random seeds. We train the VGG11 (Simonyan and Zisserman, 2015) and ResNet-50 (He et al., 2016) neural network architectures on CIFAR10 (Krizhevsky, 2009) using PyTorch (Paszke et al., 2019), and implementhttps://github.com/rka97/dowg DoWG on top of the DoG codehttps://github.com/formll/dog. Unsurprisingly, DoWG’s estimates of the step size are larger than that of DoG and D-Adapt-norm, which also makes it less stable on ResNet-50. While the last iterate of DoWG gives worse test accuracy than Adam, the average iterate of DoWG often performs better.

Finally, we note that while both neural networks tested are generally nonsmooth, recent work shows local smoothness can significantly influence and be influenced by a method’s trajectory (Cohen et al., 2022; Pan and Li, 2022). We believe this adaptivity to smoothness might explain the empirical difference between DoWG and DoG, but leave a rigorous discussion of adaptivity to local smoothness to future work.

References

Supplementary material

In this section we collect different results that are algorithm-independent, the first is a consequence of smoothness:

Because $f$ is lower bounded by $f_{*}$ we thus have

Rearranging gives ${\left\lVert\nabla f(x)\right\rVert}^{2}\leq 2L\left(f(x)-f_{*}\right)$. ∎

The next two results are helpful algebraic identities that will be useful for the proof of DoWG.

[Ivgi et al., 2023, Lemma 4]. Let $a_{0},..,a_{t}$ be a nondecreasing sequence of nonnegative numbers. Then

This is [Ivgi et al., 2023, Lemma 4]. We include the proof for completeness:

([Ivgi et al., 2023, Lemma 3], similar to [Defazio and Mishchenko, 2023, Lemma 11]). Let $s_{0},s_{1},\ldots,s_{T}$ be a positive increasing sequence. Then

where $\log_{+}x\overset{\text{def}}{=}\log x+1$.

This is [Ivgi et al., 2023, Lemma 3]. We include the proof for completeness: Define $K=\lceil\log\frac{s_{T}}{s_{0}}\rceil$ and $n=\left\lfloor\frac{T}{K}\right\rfloor$. Then,

Rearranging and using $K=\lceil\log\frac{s_{T}}{s_{0}}\rceil$ gives

Proofs for DoWG

This section collects proofs for DoWG. First, we give the following lemma, which holds under convexity alone (regardless of whether $f$ is smooth or Lipschitz).

Suppose that $f$ is convex and has minimizer $x_{*}$. For the iterations generated by Algorithm 1, we have

This proof follows the proof of DoG [Ivgi et al., 2023, Lemma 1], itself a modification of the standard proof for adaptive cumulative gradient normalization methods [Gupta et al., 2017] incorporating insights from [Carmon and Hinder, 2022]. We specifically modify the proof to handle the weighting scheme we use in DoWG. By the nonexpansivity of the projection we have

Rearranging and dividing by $2\eta_{k}$ we get

Multiplying both sides by $\bar{r}_{k}^{2}$ we get

Summing up as $k$ varies from to $t-1$ we get

We shall now bound each of the terms (A) and (B). We have

where eq. 8 holds by definition of the DoWG stepsize $\eta_{k}$, eq. 9 holds by telescoping, eq. 10 holds because $v_{k}=v_{k-1}+\bar{r}_{k}^{2}{\left\lVert\nabla f(x_{k})\right\rVert}^{2}\geq v_{k-1}$ and hence $\sqrt{v_{k}}\geq\sqrt{v_{k-1}}$, and $d_{k}^{2}\leq\bar{d}_{t}^{2}$ by definition. Equation 11 just follows by telescoping. Finally observe that $\bar{d}_{t}^{2}-d_{t}^{2}=d_{s}^{2}-d_{t}^{2}$ for some $s\in[t]$, and $d_{s}^{2}-d_{t}^{2}=(d_{s}-d_{t})(d_{s}+d_{t})$. Then by the triangle inequality and that the sequence $\bar{r}_{k}$ is monotonically nondecreasing we have

Therefore $d_{s}^{2}-d_{t}^{2}\leq(\bar{r}_{s}+\bar{r}_{t})(d_{s}+d_{t})\leq 4\bar{r}_{t}\bar{d}_{t}$. This explains eq. 12.

where eq. 13 is by Lemma 1. Plugging eqs. 12 and 14 in eq. 7 gives

We now prove the convergence of DoWG under smoothness. In particular, we shall use Fact 1 and the DoWG design to bound the weighted cumulative error $S_{t}=\sum_{k=0}^{t-1}\overline{r}_{k}^{2}\left[f(x_{k})-f_{*}\right]$ by its square root multiplied by a problem-dependent constant. We note that a similar trick is used in the analysis of AdaGrad-Norm [Levy et al., 2018], in reductions from online convex optimization to stochastic smooth optimization [Orabona and Cutkosky, 2020], and in the method of [Carmon and Hinder, 2022]. However, in all the mentioned cases, the unweighted error $M_{t}=\sum_{k=0}^{t-1}\left[f(x_{k})-f_{*}\right]$ is bounded by its square root. Here, DoWG’s design allows us to bound the weighted errors $S_{t}$ instead.

We start with Lemma 3. Let $t\in[T]$. By eq. 6 we have

Using this to lower bound the left-hand side of eq. 15 gives

If $f(x_{k})-f_{*}=0$ for some $k\in[t-1]$ then the statement of the theorem is trivial. Otherwise, we can divide both sides by the latter square root to get

Dividing both sides by $\sum_{k=0}^{t-1}\bar{r}_{k}^{2}$ we get

By Lemma 2 applied to the sequence $s_{k}=\bar{r}_{k}^{2}$ we have that for some $t\in[T]$

Because $\mathcal{X}$ has diameter $D$ we have $\bar{r}_{T}^{2}\leq D^{2}$ and therefore

If $T\geq 2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$ then $\frac{T}{\log\frac{D^{2}}{\bar{r}_{0}^{2}}}-1\geq\frac{T}{2\log\frac{D^{2}}{\bar{r}_{0}^{2}}}$ and we use this in eqs. 19 and 18 to get

Observe that because $\mathcal{X}$ has diameter at most $D$ we have $\bar{d}_{t}+\bar{r}_{t}\leq 2D$, therefore

If $T<2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$, then $1<\frac{2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}}{T}$. Let $t\in[T]$. Using smoothness and this fact we have

Observe $\bar{x}_{t},x_{*}\in\mathcal{X}$ and $\mathcal{X}$ has diameter $D$, hence ${\left\lVert\bar{x}_{t}-x_{*}\right\rVert}^{2}\leq D^{2}$ and we get

Thus in both cases we have that $f(\bar{x}_{t})-f_{*}=\mathcal{O}\left(\frac{LD^{2}}{T}\log_{+}\frac{D}{\bar{r}_{0}}\right)$, this completes our proof. ∎

2 Nonsmooth case

We now give the proof of DoWG’s convergence when $f$ is Lipschitz.

We start with Lemma 3. Let $t\in[T]$. By eq. 6 we have

Using this to lower bound the left-hand side of eq. 20 gives

We have by the fact that $f$ is $G$-Lipschitz that ${\left\lVert\nabla f(x)\right\rVert}^{2}\leq G^{2}$ for all $x\in\mathcal{X}$. Therefore,

Taking square roots and plugging into eq. 21 gives

Dividing both sides by $\sum_{k=0}^{t-1}\bar{r}_{k}^{2}$ we get

By Lemma 2 applied to the sequence $s_{k}=\bar{r}_{k}^{2}$ we have that for some $t\in[T]$

Because $\bar{r}_{T}\leq D$ we further have

If $T\geq 2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$: then $\frac{T}{\log\frac{\bar{r}_{T}^{2}}{\bar{r}_{0}^{2}}}-1\geq\frac{T}{2\log\frac{\bar{r}_{T}^{2}}{\bar{r}_{0}^{2}}}$. We can use this in eq. 22 alongside eq. 23 and the fact that $\log_{+}x^{2}=\max(\log x^{2},1)=\max(2\log x,1)\leq 2\log_{+}x$ to get

Because the diameter of $\mathcal{X}$ is bounded by $D$ we have $\bar{r}_{T}\leq D$ and $\bar{d}_{t}\leq D$, using this and convexity we get

If $T<2\log_{+}\frac{D^{2}}{\bar{r}_{0}^{2}}$: then

Because $f$ is $G$-Lipschitz then $\norm{\nabla f(\bar{x}_{t})}\leq G$ and because $\mathcal{X}$ has diameter $D$ we have $\norm{\bar{x}_{t}-x_{*}}\leq D$. Using this and eq. 24 in eq. 25 gives

Now because $T\geq 1$ we have $\sqrt{T}\leq T$ and hence

In both cases, we have that $f(\bar{x}_{t})-f_{*}=\mathcal{O}\left(\frac{4DG\log_{+}\frac{D}{\bar{r}_{0}}}{\sqrt{T}}\right)$, and this completes our proof. ∎

Unconstrained domain extension

In this section we consider the case where the domain set is unbounded, and we seek dependence only on $d_{0}=\norm{x_{0}-x_{\ast}}$. We use the same technique for handling the unconstrained problem as [Ivgi et al., 2023] in this section and consider DoWG iterates with the reduced stepsizes

We prove that with this stepsize, the iterates do not venture far from the initialization. The proof follows [Ivgi et al., 2023].

(Stability). For the iterates $x_{t+1}=x_{t}-\eta_{t}\nabla f(x_{t})$ following the stepsize scheme given by (26) we have $\bar{d}_{t}^{2}\leq 12d_{0}$ and $\bar{r}_{t}^{2}\leq 32d_{0}^{2}$ provided that $r_{0}\leq d_{0}$.

Summing up as $t$ varies from $k=1$ to $k=t$ and using [Ivgi et al., 2023, Lemma 6]

Therefore we have $d_{t}^{2}\leq d_{1}^{2}+\frac{\bar{r}_{t}^{2}}{4}$. Now suppose for the sake of induction that $\bar{r}_{t}^{2}\leq 8d_{1}^{2}$, then applying the last equation we get $d_{t}^{2}\leq 3d_{1}^{2}$. Taking square roots gives $d_{t}\leq\sqrt{3}d_{1}$. By the triangle inequality we then get

Squaring both sides gives ${\left\lVert x_{t+1}-x_{0}\right\rVert}^{2}\leq(1+\sqrt{3})^{2}d_{1}^{2}\leq 8d_{1}^{2}$. This completes our induction and we have $\bar{r}_{t}^{2}\leq 8d_{1}^{2}$ for all $t$. Finally, observe that

It follows that $\bar{r}_{t}^{2}\leq 8d_{1}^{2}\leq 32d_{0}^{2}$ for all $t$. Finally, we have $d_{t}^{2}\leq d_{1}^{2}+\frac{\bar{r}_{t}^{2}}{4}\leq 3d_{1}^{2}\leq 12d_{0}^{2}$. This completes our proof. ∎

Therefore the iterates stay bounded. The rest of the proof then follows Theorems 4 and 3 and is omitted for simplicity.. In both cases it gives the same results with $D_{0}=\norm{x_{0}-x_{\ast}}$ instead of $D$, up to extra constants and polylogarithmic factors.