Performance of Bayesian linear regression in a model with mismatch

Introduction and set-up

Linear regression with random covariates in the high-dimensional regime, where both the number of data points and their dimension scale proportionally, is a paradigmatic problem for modern inference. It has been studied mostly from two complementary perspectives: $(i)$ In Bayesian statistics where, due to important simplifications that occur in this setting, analyses have focused on the case of the estimator obtained as the expectation of the “true” posterior distribution of the regression coefficients . This is usually referred to as the Bayesian-optimal estimator. But also on sparse Bayesian regression, see and the section below on related works. $(ii)$ Another large body of literature has studied instead the performance of optimization procedures in the context of robust statistics and M-estimation, where an estimator is obtained as the mode of a log-concave posterior measure or, equivalently, as the minimizer of a convex cost function .

Our work focuses on Bayesian inference outside the optimal setting, for which much less is known. The lack of perfect knowledge of the model generating the data forces one to make certain inaccurate prior assumptions on the distribution of the regression coefficients as well as on the form of the model. This is, in a sense, more realistic and of practical interest. One natural setting for this, is to consider Bayesian linear regression with some sources of mismatch. Due to the lack of certain simplifying symmetries inherent to the optimal setting of inference, known as the “Nishimori identities” in physics , even a small mismatch can result in several theoretical challenges.

In the present work, we aim to study a mismatched linear regression model which is as follows. Let $x^{*}=(x_{i}^{*})_{i\leqslant N}$ be an unknown ground-truth of regression coefficients. Assume that

for some deterministic $\gamma\geqslant 0$ and fixed $C$ independent of $N$. Let $\alpha>0$ be a fixed parameter and let $M=\lfloor\alpha N\rfloor$To be more precise, we might take $M=\alpha N+o(N)$. But tracking this correction makes no difference.; $\|\,\cdot\,\|$ is the usual $L_{2}$ norm. Consider a matrix $G=(g_{ki})_{k\leqslant M,i\leqslant N}$ with i.i.d. standard normal entries. The columns in $G$ are understood as random covariate vectors, while the rows correspond to independent samples. The responses $y=(y_{k})_{k\leqslant M}$ are obtained according to the linear model

for $S_{k}(x^{*}):=(\bar{G}x^{*})_{k},$ where $\bar{G}:=N^{-1/2}G$ and the variables $z=(z_{k})_{k\leqslant M}\stackrel{{\scriptstyle\rm{i.i.d.}}}{{\sim}}\mathcal{N}(0,\Delta_{*})$ for some $\Delta_{*}>0$ which gives the strength of the noise; $\mathcal{N}(m,\sigma^{2})$ is the normal law of mean $m$ and variance $\sigma^{2}$. Here we will consider the regression task of recovering $x^{*}$ from the data $\mathcal{D}:=(y,\bar{G})$ with two sources of mismatch:

The distribution of $x^{*}$ is assumed to be unknown. Thus, we take the prior to be that of a gaussian vector with independent coordinates of mean and variance $1/(2\kappa)>0$.

Under these sources of mismatch, the Bayesian posterior distribution is given by

2 Main results and contributions

Our main contributions are establishing the asymptotics of the log-normalizing constant and the performance of estimator (4). Informally, we prove that, whenever the four-dimensional system (17)–(20) presented below admits a unique solution, the log-normalizing constant converges to some non-random quantity $F(q,\rho)$, that is

and the mean-square error converges to some non-random quantity $q$, that is

Here $(q,\rho)$ is the unique critical point of the real-valued function $F$ with domain over the set $\{(q,\rho):0\leqslant q\leqslant\rho<\infty\}$ defined in (21) below.

There are two major findings that follow from our results. First of all, somewhat surprisingly (5) implies that the performance of the Bayesian estimator with gaussian prior is insensitive to the statistical properties of the coefficients $x^{*}$ other than its empirical second moment. The second major finding regards our main running example for our results which will play a special role given its tractability in the analysis and its importance in statistics. This example will correspond to the model where the function $u(s)$ is of the form $-s^{2}/(2\Delta)$ for some $\Delta>0$. Notice that this model is still mismatched since $\Delta$ is not necessarily equal to the $\Delta_{*}$ appearing in the model generating the data. When we convert the posterior distribution into a Gibbs measure parametrized by the inverse temperature $\beta$ we discover that, for quadratic $u$, the mean-square error is independent of the inverse temperature $\beta$ in the high-dimensional limit. This means that when the performance is evaluated using a square loss, Bayesian regression with gaussian prior and ridge regression have the same performance. In other words, sampling and optimizing are equivalent in this setting. One may argue that the closedness between the mean and the mode of the posterior distribution with gaussian prior is mainly due to the isotropy of the covariates. This intuition makes perfect sense in the classical regime, where a lot of data is accessible $M\gg N$ when maximum likelihood estimation is optimal . But in the high-dimensional regime where both $M$ and $N$ are large and comparable, this feature becomes much less evident. To the best of our knowledge, this fact was not rigorously obtained prior to our work. Exploring the generality of this equivalence between optimization and sampling procedures is an interesting open direction for future research. See for related connections between M-estimation and Bayesian inference from statistical mechanics heuristics.

The approach in this paper relies primarily on a connection between our regression task and a generalized version of the Shcherbina-Tirozzi (ST) mean-field spin glass model . We study the limits of the corresponding log-partition function and some critical quantities called the overlaps. By utilizing the so-called smart path method, analogous results were intensively explored by Talagrand assuming that $u$ grows at most linearly. Whereas the machinery therein relies on quite heavy analysis, the advantage of our approach is methodological. In addition to being able to relax the growth condition for $u$ from linear to quadratic, our analysis is also rather simple and self-contained. It is based on methods coming from the mathematical physics of mean-field spin glasses: mainly a rigorous version of the cavity method also called the leave-one-out method in statistics (see Sections 5 and 6 for details). We expect for the simplicity of our approach to allow us to extend the analysis to generalized linear models in mismatched settings, for which results similar to the present ones exist only in the Bayesian-optimal setting of inference . This is left for future work.

3 Related works

In recent years, many works on mismatched regression have appeared: on non-rigorous statistical mechanics techniques such as the replica method like, e.g., in ; on the analysis of approximate message-passing algorithms, like in ; and on Gordon’s convex min-max theorem . All these approaches strongly rely on the fact that the estimator is obtained as a unique minimizer of a convex function and therefore do not generalize easily to mismatched Bayesian settings. The leave-one-out method was also used in but, like in the aforementioned references, only for the analysis of (non-Bayesian) M-estimators.

Complementary to our setup is a recent work on the mismatched Bayesian linear regression that derives conditions under which the “naive mean-field approximation” to the log-partition function is valid, and provides an infinite-dimensional variational formula for it (see for a related setting). Last but not least, Bayesian linear regression has also been considered in some different high-dimensional scaling regimes than the one presented here: in settings with sparse ground-truth vectors of regression coefficients with a possibly vanishing fraction of non-zero entries. Reference studies in detail the properties of the posterior distribution with sparsity inducing priors in this case, see also for related prior works on sparse linear regression. It has recently been understood that in the very sparse regime, and with the number of sampled responses much smaller than the covariates’ dimension, intriguing “all-or-nothing” phase transitions occur where the recovery error jumps from almost zero to its maximum value at a certain threshold . For a recent review on the Bayesian approach to high-dimensional inference, see .

4 Organization

The paper is organized as follows. In Section 2, we explain the link between the high-dimensional regression problem presented here and the ST spin glass model. In Section 3, we first state two results on the main quantities of interest in spin glass models, namely, the free energy (i.e., log-partition function) and overlaps (that will later be related to the mean-square error) in the ST model. We then state our main results on the regression task with a particular emphasis on the classical example of the Bayesian regression with quadratic loss and gaussian prior. Sections 4–6 are devoted to establishing results concerning the ST model. More explicitly, Section 4 establishes a number of concentration properties for the overlaps and free energy. Section 5 shows how to obtain certain fixed-point equations for the “order parameters” of the model based on the concentration results in Section 4 by utilizing the cavity method and simple gaussian integration by parts. Section 6 proves a closed-form expression for the limiting log-partition function of the posterior in a straightforward manner thanks to the convergence results for the order parameters. Finally, our results on the regression task are established in Section 7.

Connection to the Shcherbina-Tirozzi model

and the corresponding log-partition function, also called free energy, by

In the terminology of the large deviation theory , it is the scaled cumulant log-moment generating function associated with the energy per variable $H_{N}(\sigma)/N$. When $z_{k}=0$ and $u$ grows at most linearly, the Hamiltonian $H_{N}$ is known as the original ST model extensively studied in . In the setting of the present paper, we allow $u$ to grow quadratically (see Assumption 1 below for details).

Two key quantities for our analysis will be the overlaps between replicas defined according to

Additionally, we also define a pair of “conjugate overlaps” in terms of the following two auxiliary quantities

As one shall see, the vector $(R_{1,1},R_{1,2},Q_{1,1},Q_{1,2})$ of overlaps order parameters is the “correct” one, in the sense that all the statistical analysis can be asymptotically expressed solely in terms of the limits of these quantities and the model at the “macroscopic level” is completely determined by them.

In view of our regression task, the posterior measure (3) can be thought of as a Gibbs measure of the spin glass model with Hamiltonian

As in the ST model, we define the free energy by

In particular, from (4), we readily see that the Bayesian estimator

The connection between the ST and our models relies on a number of change of variables. First of all, by letting $w:=x-x^{*}$, we may write

To simplify this integral, take an orthogonal matrix $O\in\mathcal{O}(N)$ such that $Ox^{*}=N^{-1/2}{\|x^{*}\|}\bm{1}_{N}$. Note that conditionally on $x^{*}$,

which implies that $GO^{\intercal}\sim G$ conditionally on $x^{*}$. Note also that $\langle x^{*},O^{\intercal}a\rangle=N^{-1/2}\|x^{*}\|\langle\bm{1}_{N},a\rangle$ and $\|O^{\intercal}a\|=\|a\|.$ From these and using the change of variable $\sigma:=Ow$,

which implies, from the rotational invariance of $G$ mentioned above and the symmetry of $z_{k},$ that

for $\gamma_{N}:=\|x^{*}\|^{2}/N.$ If now we randomize $h=h_{N}=2\kappa\sqrt{\gamma_{N}}$ in the Hamiltonian (6) of the ST model independently of all other randomness, then we readily get

Consequently, to understand our regression problem, we may study the limiting free energy and overlaps corresponding to the generalized ST model first. This characterization is part of our main results and is stated in the next section.

Main results

The first part of our main results consists of an expression for the limiting free energy and the convergence of the overlaps in the generalized ST model defined in (6). Throughout the rest of this paper, the following assumption on the function $u$ is in force.

The function $u(s)$ is concave, non-positive, and there exists a constant $d>0$ such that

The concavity of $u$ is conceptually an important assumption, which ensures that our posterior distribution is a log-concave measure and, as a result, the overlaps are concentrated under the Gibbs measure $G_{N}$. While the other assumptions are purely technical, overall they weaken the settings considered in . In particular, in this reference $u$ is allowed to grow at most linearly and thus its results do not apply to our main example, namely, a quadratic $u$.

for $0\leqslant q<\rho$ and $0\leqslant\bar{r}\leqslant r$, where $\widetilde{z},\xi$ are independent random variables with normal law $\mathcal{N}(0,1)$. If we let $\theta=\widetilde{z}\sqrt{\Delta_{*}+q}+\xi\sqrt{\rho-q}$, basic gaussian integration by parts with respect to $\widetilde{z}$ and $\xi$ implies that the critical points of the function $\bar{F}$ must satisfy the following system of equations:

Observe that if $(q,\rho,r,\bar{r})$ satisfies (17)–(20), then $\bar{F}(q,\rho,r,\bar{r})$ and $F(q,\rho)$ match each other as can be seen by rewriting (17) and (18) as

and plugging these into the second line in (16) after writing $rq-\bar{r}\rho=-r(\rho-q)+(r-\bar{r})\rho$. Here, if we express $r,\bar{r}$ in terms of $q,\rho$ in (19)–(20), the pair $(q,\rho)$ is also a critical point of $F$. Our first main result states that the limiting free energy in the ST model is equal to $F(q,\rho)$ as long as (17)–(20) have a unique solution.

The next result establishes the convergence of the overlaps (9) and (11).

From these and with the help of the identities (14) and (15), we can now compute the limiting free energy and the mean-square error associated to our regression task.

Assume the convergence (1) for the norm of the ground-truth coefficients towards some deterministic $\sqrt{\gamma}\geqslant 0$. For $\kappa>0$, $h=2\kappa\sqrt{\gamma}$, and $\Delta_{*}\geqslant 0$, if (17)–(20) have a unique solution $(q,\rho,r,\bar{r})$, then

Let $u(s)=-{s^{2}}/(2\Delta),$ where $\Delta>0$ is not necessarily equal to the true gaussian noise variance $\Delta_{*}.$ Obviously $u$ satisfies Assumption 1. More importantly, it can be checked that (17)–(20) have a unique solution given by

To see this, note that (19) and (20) can be written as

Take $a=\rho-q$ and $a^{\prime}=r-\bar{r}$. From the difference between (17) and (18) and that of (19) and (20), we arrive at

from which we can eliminate $a^{\prime}$ to get a single equation for $a$,

It is easy to see that the only non-negative solution to this equation is $a=c$ and thus, $a^{\prime}=\alpha/(\Delta+c).$ Finally, using these values of $a$ and $a^{\prime}$ and the second equation of (25) leads to

Note that $q$ is positive since (26) implies that $aa^{\prime}<1$ and $aa^{\prime}<\alpha$, which ensures that $(\Delta+c)^{2}/c^{2}=(aa^{\prime})^{2}<\alpha$. From this $q$, $(\rho,r,\bar{r})$ can then be uniquely determined through the second to the fourth equations in (24).

Surprisingly, if we parametrize the posterior distribution by an “inverse temperature” hyperparameter $\beta>0$, namely, the original Gibbs measure $\widetilde{G}_{N}$ in (12) becomes

meaning that $\Delta$ and $\kappa$ are replaced by $\Delta/\beta$ and $\beta\kappa$ in the original model, then the resulting constant $q_{\beta}$, i.e., the mean-square error, remains the same as the original $q$ for any inverse temperature $\beta>0$. This can be easily seen from the above equations (for quadratic $u$). At the same time, formula (27) matches the one for ridge regression at zero temperature (see, e.g., equation (26) in ). This non-trivial fact is due to the rotational invariance of the model following from the isotropy of the covariates. But as mentioned already in the introduction, this is not a priori evident in the high-dimensional regime we consider here, where $M$ and $N$ are comparable. It would be interesting to explore in future research for which settings posterior mean and mode (i.e., sampling and optimizing) yield the same reconstruction performance.

Concentration

The last assumption (G-3) will be used in an equivalent form,

Also, the assumptions (29), (30) may look redundant given (31), but we state them for convenience.

We will later specialize these assumptions for the following two specific choices:

For $n=1$: $\phi(s)=s^{2}$ and $\psi(s)=u^{\prime\prime}(s)+u^{\prime}(s)^{2}$,

For $n=2$: $\phi(s_{1},s_{2})=s_{1}s_{2}$ and $\psi(s_{1},s_{2})=u^{\prime}(s_{1})u^{\prime}(s_{2})$.

One can easily see that Assumption 1 on $u$ implies (G-1)–(G-3) in both cases, by possibly increasing the value of $d$.

for functions $\phi$ and $\psi$ verifying assumptions (G-1)–(G-3). Define the (non-averaged) free energy associated to $X_{N,\lambda,\eta}$ by

Note that assumptions (28), (29), and (32) as well as $|\lambda|,|\eta|\leqslant\min(\eta_{0},\kappa/(2d))$ ensure that this quantity is a.s. finite in $G$ and $z=(z_{k})_{k\leqslant M}$ Denote by $\langle\,\cdot\,\rangle_{\lambda,\eta}$ the expectation with respect to the Gibbs measure proportional to $\exp X_{N,\lambda,\eta}(\vec{\sigma})$:

Under the above assumptions, there exists some $K>0$ such that

Moreover, for any $k\geqslant 1$, there exists some $C_{k}>0$ such that

In the rest of this section, we will establish Proposition 4.1. To fix our notation, for any real $M\times N$-matrix $A$, define the Frobenius and operator norms, respectively, as

Recall $\|\cdot\|$ applied to a vector is the $L_{2}$ norm and note that $\|A\|_{op}\leqslant\|A\|$.

Let $\bar{z}:=z/\sqrt{N}$ and recall also that $\bar{G}:=G/\sqrt{N}$.

There exist $\varepsilon>0$ and $K\geqslant 1$ such that if $t,|\lambda|,|\eta|\leqslant\varepsilon$ then

By (37), if $t+d|\lambda|\leqslant\kappa/2$, then

which, together with (32) for $|\eta|\leqslant\eta_{0}$ implies that,

Combining this with (37) and (38), we obtain that

which together with the fact that $\ln x\leqslant x+1$ for all $x>0$ implies that

Our assertion follows by putting this and (39) together.

We let $I(A)$ be the indicator function for event $A$.

There exist constants $\varepsilon>0$ and $K,K^{\prime}>0$ such that, for any $|\lambda|,|\eta|\leqslant\varepsilon$,

Proof. Recall the constants $\varepsilon$ and $K$ from the statement of Lemma 4.2. If $t\leqslant\varepsilon$, then for any $a>0$,

Our proof is completed by taking $a=\sqrt{{2t^{-1}KN(\|\bar{z}\|^{2}+\|\bar{G}\|_{op}^{2}+1+h^{2})}}.$

Let $k\geqslant 1.$ There exist $0<\varepsilon<1$ and $K\geqslant 1$ such that, for any $|\lambda|,|\eta|\leqslant\varepsilon$,

For any non-negative random variable $X$, by Lemma 3.1.8 in ,

Using this inequality with $X=t\|\vec{\sigma}\|^{2}$, we have

2 Convexity and pseudo-Lipschitz property of the overlaps

If $|\lambda|+\|\bar{G}\|_{op}^{2}|\eta|\leqslant\kappa/(dn)$ then the following function is convex:

Proof. For simplicity, for $\vec{\sigma}=(\sigma^{1},\ldots,\sigma^{n})$, we denote $\sigma(i)=(\sigma_{i}^{1},\ldots,\sigma_{i}^{n}).$ From (29),

where in the third line we used the Minkowski inequality. Again, by (29),

we can again use the Minkowski inequality to obtain

For any $k\geqslant 1$ there exists some $C_{k}>0$ such that

Proof. Let us denote by $\vec{s}:=\vec{\sigma}/\sqrt{N}.$ Note that from Lemma 4.6,

where we used the assumption (30). Using Lemma 4.4 with $\lambda=\eta=0$, we see that there exists some $C$ such that

3 Concentration of the free energy

For clarity, here we will now denote $F_{N}(\lambda,\eta)$ by $F(G,z)$. Let us start with the following result.

There exist constants $\varepsilon>0$ and $K>0$ such that for all $|\lambda|,|\eta|\leqslant\varepsilon$,

Using that $|u^{\prime}(s)|\leqslant d(1+|s|)$ by (28) (below the constant $K$ amy change from place to place),

where the last inequality used (41). Note that from (31), there exists a constant $d^{\prime}$ such that

Using this inequality, we can argue similarly that

In similar manner, we readily compute that

for some $\delta>0$, which implies Lemma 4.8.

There exist absolute constants $\varepsilon>0$ and $K>0$ such that, for all $|\lambda|,|\eta|\leqslant\varepsilon$,

Proof. For simplicity of notation, denote

If we define $\bar{y}:=y/\sqrt{N}$, then the above Lemma proves that

Let $L>0$ be a constant to be chosen later and $L_{h}:=L(1+h^{2})$. Since, for $r(y),r(y^{\prime})\leqslant L_{h}$,

and we have equality for $y^{\prime}=y$, if we define

then $\Phi(y)=f(y)$ for all $y$ such that $r(y)\leqslant L_{h}$. From this definition, it can be seen from the reverse triangle inequality that

By (44) and the Gaussian-Poincaré concentration inequality,

Next, using (45) and the Cauchy-Schwarz inequality, we control

4 Proof of Proposition 4.1

We will only consider the case of the overlap $R(\vec{\sigma})$, since the proof for $Q(\vec{\sigma})$ is almost identical. In this section we will denote by $K$ a constant that depends only on the parameters of the model, but can vary from one occurrence to the next. Let us consider the event

By (43), we can and will choose $K\geqslant 1$ so that the probability of the event $E_{N}$ is at least $1-4e^{-N}$. From Lemma 4.7,

Let $F_{N}^{\prime}$ be the $\lambda$-derivative of $F_{N}$. In the remainder we will control the same expectation on the event $E_{N}$, and we will use the convexity of $F_{N}(\lambda,\eta)$ in $\lambda$, which implies that

Take $\varepsilon>0$ such that, for $|\lambda|,|\eta|\leqslant\varepsilon$, the inequalities in Lemmas 4.3, 4.4 and 4.9 hold. By Lemma 4.9, the expectation of the square of the term between the parentheses above can be bounded by $K/(N\lambda^{2})$.

In order to control the first term $F_{N}^{\prime}(\lambda,0)-F_{N}^{\prime}(-\lambda,0)$ above, our goal will be to control

For simplicity of notation we denote $\vec{s}:=\vec{\sigma}/\sqrt{N}$. Since, for $|\lambda|,|\eta|\leqslant\varepsilon$, the inequalities in Lemmas 4.3 and 4.4 hold, on the event $E_{N}$, for some $L\geqslant 1$,

where we also denoted $\vec{s}^{\prime}:=\vec{\sigma}^{\prime}/\sqrt{N}$.

We now proceed as in the proof of Lemma 4.9. Take $L$ as in (49) and $D$ as in (50), denote $r(\vec{\sigma}):=\|\vec{s}\|$ and let

Then, from (50), $\rho(\vec{\sigma})=R(\vec{\sigma})$ for all $r(\vec{\sigma})\leqslant L$,

Using (52) and the Cauchy-Schwarz and Minkowski inequalities,

where in the last line we used the first equation in (49). Since $R(0)=\phi(0)$, by (50)

Therefore, the second equation in (49) implies that

where the last inequality used (51). Together with (53), this implies that, on the event $E_{N}$,

assuming as above that $|\lambda|,|\eta|\leqslant\varepsilon$ and $|\lambda|+K|\eta|\leqslant\kappa/(dn)$. In particular,

and \bigl{|}F_{N}^{\prime}(\lambda,0)-F_{N}^{\prime}(-\lambda,0)\bigr{|}\leqslant K\lambda. Plugging this into (48) and, as we mentioned above, using Lemma 4.9 for the second term,

by optimizing over $\lambda$. Using (54) on the event $E_{N}$ and then combining with (47) gives

This proves the first inequality in (36). The proof of the second inequality is identical.

Fixed point equations

This section establishes the proof of Theorem 3.3. The key ingredient here is to show that equations (17)–(20) are satisfied by the subsequential limits of the overlaps, based on a cavity/leave-one-out argument and gaussian integration by parts.

For every $k\geqslant 1$, there exists some $C_{k}>0$ such that, uniformly on $N$,

Proof. Here we will denote $u(S_{1}(\sigma)+z_{1})$ by $u_{1}\leqslant 0$. Because $|u_{1}|^{k}$ is an increasing function of $|u_{1}|$ and $\exp(-|u_{1}|)$ is decreasing, the Harris inequality implies $\langle|u_{1}|^{k}\exp(-|u_{1}|)\rangle_{c}\leqslant\langle|u_{1}|^{k}\rangle_{c}\langle\exp(-|u_{1}|)\rangle_{c}$. As a consequence we have that

Furthermore, because the standard gaussian vector $g_{1}:=(g_{11},\dots,g_{1N})$ is independent of $\sigma$ under the cavity measure, by the rotational invariance of $g_{1}$ we have that $\langle g_{1},\sigma\rangle\sim g\|\sigma\|$ where $g$ is an independent standard gaussian variable, so under the cavity measure $S_{1}(\sigma)\sim g\|\sigma\|/\sqrt{N}=g\sqrt{R_{1,1}}$. Thus, by Assumption 1

Finally, by the independence of $R_{1,1}$ from $g$ and $z_{1}$ and Proposition 4.1, the right-hand side of this last equation is easily seen to be smaller than some constant that does not depend on $N$.

There exist constants $K,K^{\prime}>0$ such that

Proof. As in the previous lemma, we will denote $u(S_{1}(\sigma)+z_{1})$ by $u_{1}$. Let

for $t\in$, and denote by $\langle\,\cdot\,\rangle_{t}$ the expectation with respect to the associated Gibbs measure. We then clearly have that

for some constants $K,K^{\prime}>0$ that do not depend on $t\in$. Here, the bound for the first factor follows from the Brascamp-Lieb inequality, while the bound on the second term is a generalization of Lemma 5.1. The proof for $R_{1,2}$ is similar.

2 Consistency equations for the overlaps

Throughout this subsection, we will repeatedly use the concentration results for the generalized overlaps $Q$ and $R$ from Section 4 that are applicable to the overlaps $Q_{1,1}$ using $\psi(s)=u^{\prime\prime}(s)+u^{\prime}(s)^{2}$, $Q_{1,2}$ using $\psi(s_{1},s_{2})=u^{\prime}(s_{1})u^{\prime}(s_{2})$, $R_{1,1}$ using $\phi(s)=s^{2}$, and $R_{1,2}$ using $\phi(s_{1},s_{2})=s_{1}s_{2}$. Denote

By Proposition 4.1, there exists some compact set $K$ such that for all $N\geqslant 1$, $(\rho_{N},q_{N},\bar{r}_{N},r_{N})$ belongs to $K$. Thus, there exists a subsequence $(N_{m})_{m\geqslant 1}$ of system sizes along which

for some $(\rho,q,\bar{r},r)$ in $K$. For the rest of this section, we work with this convergent subsequence and we aim to show that $(\rho,q,\bar{r},r)$ satisfies (17)–(20) in Propositions 5.4 and 5.5. To begin with, we establish

By Proposition 4.1 and Lemma 5.2, $\Sigma_{m}\xrightarrow{L^{2}}\Sigma$ as $m\to+\infty$ where

The vector $(\rho,q,\bar{r},r)$ satisfies (19) and (20).

By equation (55), the overlap definition (11), the spin symmetry and the geometric series (recall $u(s)\leqslant 0$) we have that

Note that by Assumption 1 the first term is the mean of a continuous and bounded function of $a_{1},\dots,a_{K}$. Then by Lemma 5.3 we have that

The second term can be bounded in the following way

where in the first line we used Assumption 1, in the second Cauchy-Schwarz and Jensen’s inequalities, and in the last one that by Lemma 5.1 there exists a constant $C>0$ that bounds the first factor in the square root. Now, because $(1-\exp{u(a_{1})})^{{2K}}$ is a continuous and bounded function of $a_{1}$, by Lemma 5.3 we have that

By taking the limit $m\to+\infty$ in equation (58) and using the convergence of the overlap $Q_{1,1}$ along the subsequence $(N_{m})_{m\geqslant 1}$, this implies that

Consider the two terms in $f_{k}$ separately. Taking the limit $K\to+\infty$, by the monotone convergence theorem,

and since $|u^{\prime\prime}(a_{1})|\leqslant d$ by Assumption 1, by the dominated convergence theorem, we also have

To see that the second term goes to , note that $1-\exp{u(\theta)}<1$. The limit then follows by the dominated convergence theorem. This proves $\bar{r}=\bar{\Phi}(q,\rho)$, i.e., equation (20).

The proof that $r=\Phi(q,\rho)$ (equation (19)) follows in a similar way. Let $K\geqslant 1$. For each $k\leqslant K$ define

By Assumption 1, all these are continuous and bounded functions of $a_{1},\dots,a_{2k}$. As before, by the cavity formula and the geometric series we have that

Finally, the conclusion follows in a similar manner as for the first equation: Lemma 5.3 is used for the sum while the second term can be seen to go to when $K\to+\infty$.

To establish the second pair of fixed point equations, a feasible approach would be to perform a cavity argument over the variables indexed by $i=1,\ldots,N$ analogous to the proof of Proposition 5.4 (this “cavity in $N$” is in general more involved than the “cavity in $M$”). Nevertheless, by the virtue of concentration of the overlaps, we show that they can also be obtained through an elementary derivation via gaussian integration by parts.

The vector $(\rho,q,\bar{r},r)$ satisfies (17) and (18).

Proof. To see this, note that by the term $-\kappa(\sigma_{1}^{1})^{2}$ in the Hamiltonian (6) we can use gaussian integration by parts with respect to $\sigma_{1}^{1}$ and the gaussian variables $(g_{k1})_{k\leqslant M}$. In particular

and, denoting the modified Hamiltonian $\widehat{H}_{N}(\sigma):=\sum_{k\leqslant M}u(S_{k}(\sigma)+z_{k})-h\sum_{i\leqslant N}\sigma_{i}$,

These identities imply (recall the definitions (10))

The concentrations of $R_{1,1}$, $R_{1,2}$, $Q_{1,1}$, and $Q_{1,2}$ (see Proposition 4.1) then implies that

Sending $m\to+\infty$, these lead to $q=\Psi(r,\bar{r})$ and $\rho=\bar{\Psi}(r,\bar{r})$ (equations (17) and (18)) and complete our proof.

Propositions 5.4 and 5.5 together conclude that $(\rho,q,\bar{r},r)$ satisfies (17)–(20). We are now in position to prove Theorem 3.3 for the ST spin glass model.

3 Proof of Theorem 3.3.

Consider any subsequence of system sizes. As explained above Lemma 5.3, there exists some further subsequence $(N_{m})_{n\geqslant 1}$ such that

for some $(\rho,q,\bar{r},r)$ in a compact set $K$. By Propositions 5.4 and 5.5 and the continuity of the functions $\bar{\Phi}$, $\Phi$, $\bar{\Psi}$, and $\Psi$, we see that $(\rho,q,\bar{r},r)$ is a solution of (17)–(20). Since this system of equations has only one unique solution as ensured by the given assumption, it follows that any convergent subsequence will share the same limit and this implies the assertion.

Proof of Theorem 3.2

In this section, we establish the convergence of $F_{N}$ using a basic interpolation in this section. Let $\alpha,\Delta_{*},\Delta,\kappa>0$ be fixed. For $t\in$, consider an auxiliary free energy

For any $t_{0}\in(0,1],$ there exist constants $K>0$ and $\delta>0$ such that for all $t\in[t_{0}-\delta,t_{0}+\delta]\cap(0,1]$ and $N\geqslant 1$ we have

Sending $t\downarrow 0$ in (17)–(20) also yields that $\lim_{t\downarrow 0}(\rho_{t},q_{t},\bar{r}_{t},r_{t})=(\rho_{0},q_{0},\bar{r}_{0},r_{0})$. Hence, $\rho_{t},q_{t},\bar{r}_{t},r_{t}$ are continuous on $$.

where $\theta_{t}:=\widetilde{z}\sqrt{\Delta_{*}+q_{t}}+\xi\sqrt{\rho_{t}-q_{t}}$ with i.i.d. standard gaussian $\widetilde{z},\xi$. Because the fixed point equations satisfy the critical point conditions

and $\rho_{t},q_{t},\bar{r}_{t},r_{t}$ are locally Lipschitz on $(0,1),$ we conclude that $df/dt=\partial f/\partial t$. To compute this partial derivative, let $\theta_{t}^{\prime}:=\widetilde{z}\sqrt{\Delta_{*}+q_{t}}+\xi^{\prime}\sqrt{\rho_{t}-q_{t}}$ for $\xi^{\prime}$ an independent copy of $\xi$. Using gaussian integration by parts with respect to $\widetilde{z}$ and $\xi$ yields

Proof of Theorem 3.4

The free energy and overlap for the ST model with a random $h_{N}$ and fixed $h$ are asymptotically the same:

For $0\leqslant t\leqslant 1,$ consider the interpolated free energy,

Let $H_{N,t}(\sigma)$ be the Hamiltonian defined by the above exponent. The associated Gibbs mean is denoted by $\langle\,\cdot\,\rangle_{t}$ and is defined by

From these, the mean value theorem and the Hölder inequality, there exists $t\in$ such that

The same proof as of Lemma 4.4 yields that for any $k\geqslant 1$ there exists some $K>0$ such that for all $t\in$ and $N\geqslant 1,$

Since the expectation of every term in the bracket is bounded by a universal constant, we obtain (62) by using (7) and taking $k=1.$

where $K$ is a constant independent of $w,N,t.$ Therefore, letting $w=(1,\ldots,1)$ implies

we can apply (68) with the choices $w=\sigma^{2}$ and $w=\langle\sigma^{1}\rangle_{t}$ to get

Putting these together and applying (67) with $k=2$, we obtain that for all $0\leqslant t\leqslant 1$ and $N\geqslant 1,$

for some constant $K^{\prime}$ independent of $t,N.$ Plugging (69) and (70) into (7) validates (63). ∎

Now we proceed to establish (22) and (23). From Theorem 3.3 and (15),

As for (22), from (LABEL:add:eq-5.1), we can write

and $O$ is an orthonormal matrix satisfying $Ox^{*}=N^{-1/2}{\|x^{*}\|}\bm{1}_{N}.$ Write

Here the first term on the right-hand side vanishes in $L^{2}$ norm by the given assumption. For the second term, observe that conditionally on $x^{*}$, $F_{N}(x^{*})$ is the same as $F_{N}(h_{N})$ in distribution. Lemma 4.9 (with $n=1$ and $\lambda=\eta=0$) allows us to write that with probability one,

where $K$ is independent of $N.$ Hence, we can bound the second term by

We bound the third term in (71) by using the Jensen inequality,

W.-K.C. was partly supported by NSF DMS-17-52184. D.P. was partially supported by NSERC and Simons Fellowship.

References