Bayesian fractional posteriors

Introduction

The usage of fractional likelihoods has generated renewed attention in Bayesian statistics in recent years, where one raises a likelihood function to a fractional power, and combines the resulting fractional likelihood with a prior distribution via the usual Bayes formula to arrive at a power posterior or fractional posterior distribution. Applications of fractional posteriors has been diverse, ranging from fractional Bayes factors in objective Bayesian model selection , data-dependent priors for sparse estimation , to marginal likelihood approximation and posterior simulation . The fractional posteriors are a special instance of Gibbs posteriors or quasi-posteriors , where the negative exponent of a loss function targeted towards a specific parameter of interest is used as a surrogate for the likelihood function; see for a general framework for updating of prior beliefs using Gibbs posteriors.

The recent surge of interest in fractional posteriors can be largely attributed to its empirically demonstrated robustness to misspecification . For correctly specified, or well-specified (non)parametric models, there is now a rich body of literature guaranteeing concentration of the posterior distribution around minimax neighborhoods of the true data generating distribution. While it can be argued that the primary objective of Bayesian nonparametric models is to relax parametric assumptions to capture finer aspects of the data, susceptibility to model misspecification remains a potent concern. First, in many practical situations, it may be unreasonable to assume that all aspects of data generating distribution can be captured adequately via a probabilistic model, and fitting models of increasingly high complexity additionally carries the risk of overfitting. Second, even though Bayesian nonparametric models can be arbitrarily flexible, they still rely on parametric building blocks, for example, component specific distributions in mixture models, and small perturbations to these assumptions can lead to fairly drastic differences in the inference drawn from the model fit .

There is a comparatively smaller literature on large sample behavior of nonparametric Bayesian procedures under misspecification , where the general aim is to establish sufficient conditions under which the usual posterior distribution concentrates around the nearest Kullback–Leibler (KL) point to the truth inside the parameter space. However, these conditions are considerably more stringent than those in case of well-specified models, so that verification can be fairly nontrivial, along with comparatively limited scope of applicability. In fact, empirically demonstrate through a detailed simulation study that even convergence to the nearest KL point may not take place in misspecified models. They instead recommend using a fractional posterior, with a data-driven approach to choose the fractional power; see also . More recently, proposed a coarsened posterior approach to combat model misspecification, where one conditions on neighborhoods of the empirical distribution rather than on the observed data to apply Bayes formula. When the neighborhood of the empirical distribution is chosen based on the KL divergence, exhibited that the resulting coarsened posterior essentially takes the form of a fractional posterior.

These observations compel us to systematically study the concentration properties of fractional posteriors. While established consistency of power posteriors for well-specified models; see also for certain rate results; we derive rates of convergence for the fractional posterior for general non-i.i.d. models in a misspecified model framework. The sufficient conditions for the fractional posterior to concentrate at the nearest KL point turn out to be substantially simpler compared to the existing literature on misspecified models. We state our concentration results for a class of Rényi divergence measures in a non-asymptotic environment, which in particular, imply Hellinger concentration in properly specified settings. The effect of flattening the likelihood shows up in the leading constant in the rate. The sub-exponential nature of the posterior tails allow us to additionally derive posterior moment bounds.

As one of our contributions, we show that the contraction rate of the fractional posterior is entirely determined by the prior mass assigned to appropriate KL neighborhoods of the true distribution, bypassing the construction of sieves and testing arguments in the existing theory . One practically important consequence is that concentration results can be established for the fractional posterior for a much broader class of priors compared to the regular posterior. We provide several examples on usage of heavy tailed hyperpriors in density estimation and regression, where the fractional posterior provably concentrates at a (near) minimax rate, while the regular posterior has inconclusive behavior. Another novel application of our result lies in shape constrained function estimation. Obtaining metric entropy estimates in such problems pose a stiff technical challenge and constitutes an active area of research . The fractional posterior obviates the need to obtain such entropy estimates en route to deriving concentration bounds.

As a second contribution, we develop oracle inequalities for the fractional posterior based on a new PAC-Bayes inequality in a fully general Bayesian model. Many previous results on PAC-Bayes type inequalities are specifically tailored to classification (bounded loss, ) or regression (squared loss, ) problems. Moreover, in the machine learning literature, a PAC-Bayes inequality is primarily used as a computational tool for controlling the generalization error by optimizing its upper bound over a restricted class of “posterior” distributions . There is a need to develop a general PAC-Bayes inequality and an accompanied general theory for analyzing the Bayesian risk that can be applied to a broader class of statistical problems. In this paper, we derive an oracle type inequality for Bayesian procedures, which will be referred to as a Bayesian oracle inequality (BOI), based on a new PAC-Bayes inequality. Similar to the local Rademacher complexity or local Gaussian complexity in a frequenstist oracle inequality (FOI) for penalized empirical risk minimization procedures , a BOI also involves a penalty term, which we refer to as local Bayesian complexity, that characterizes the local complexity of the parameter space. Roughly speaking, the local Bayesian complexity is defined as the inverse sample size times the negative logarithm of the prior mass assigned to certain Kullback-Leibler neighbourhood around the (pseudo) true parameter. In the special case when the prior distribution is close to be “uniform” over the parameter space, the local Bayesian complexity becomes the inverse sample size times a local covering entropy, and our BOI recovers the convergence rates derived from local covering conditions . Moreover, our BOI naturally leads to sharp oracle inequalities when the model is misspecified. For example, when applied to convex regression, we derive a sharp oracle inequality with minimax-optimal (up to $\log n$ factors) excess risk bound that extends the recent sharp oracle inequality obtained in from dimension one to general dimension $d\geq 1$.

Last but not the least, our analysis reveals several potential advantages of averaging based Bayesian procedures over optimization based frequentist procedures. First, due to the averaging nature of a Bayesian procedure, our averaging case analysis leading to a BOI is significantly simpler than a common worst case analysis leading to a FOI. For example, a local average type excess risk bound from a Bayesian procedure allows us to use simple probability tools, such as the Markov inequality and Chebyshev’s inequality, to obtain a high probability bound for the excess risk, since the expectation operation exchanges with the averaging (integration) operation. This is different from a local supremum type excess risk from a optimization procedure, where more sophisticated empirical process tools are exploited to obtain a high probability bound for excess risk , due to the non-exchangeability between the expectation operation and the supremum operation. For further details about the comparison between BOI and FOI, please refer to Section 3.3. Second, a Bayesian procedure naturally leads to adaptation to unknown hyperparameters or tuning parameters. We show that by placing a hyper-prior that distributes proper weights to different levels of the hyperparameter, a BOI adaptively leads to the optimal rate corresponding to the best choice of the hyperparameter.

The rest of the paper is organized as follows. The main results of the paper are stated in §3, with contraction results in §3.2, and the PAC-Bayesian inequality and Bayesian oracle inequality in §3.3. Applications to convex regression, Gaussian process regression and density estimation are discussed in §4. All proofs are deferred to §5.

Preliminaries

We begin by introducing notation, and then briefly review Rényi divergences as our key metric characterizing the contraction of fraction posteriors.

2 Rényi divergences

Let $P$ and $Q$ be probability measures on a common probability space with a dominating measure $\mu$, and let $p=dP/d\mu,q=dQ/d\mu$. The Hellinger distance $h^{2}(p,q)=(1/2)\int(\sqrt{p}-\sqrt{q})^{2}d\mu=1-A(p,q)$, where $A(p,q)=\int\sqrt{pq}\,d\mu$ denotes the Hellinger affinity. Let $D(p,q)=\int p\log(p/q)d\mu$ denote the Kullback–Leibler (KL) divergence between $p$ and $q$. For any $\alpha\in(0,1)$, let

denote the Rényi divergence of order $\alpha$. Let us also denote $A_{\alpha}(p,q)=\int p^{\alpha}q^{1-\alpha}d\mu=e^{-(1-\alpha)D_{\alpha}(p,q)}$, which we shall refer to as the $\alpha$-affinity. When $\alpha=1/2$, the $\alpha$-affinity equals the Hellinger affinity. We recall some important inequalities relating the above quantities; additional details and proofs can be found in .

(R1) $0\leq A_{\alpha}(p,q)\leq 1$ for any $\alpha\in(0,1)$, which in particular implies that $D_{\alpha}(p,q)\geq 0$ for any $\alpha\in(0,1)$.

(R2) $D_{1/2}(p,q)=-2\log A(p,q)=-2\log\{1-h^{2}(p,q)\}\geq 2h^{2}(p,q)$ using the inequality $\log(1+t)<t$ for $t>-1$.

(R3) For fixed $p,q$, $D_{\alpha}(p,q)$ is increasing in the order $\alpha\in(0,1)$. Moreover, the following two-sided inequality shows the equivalence of $D_{\alpha}$ and $D_{\beta}$ for $0<\alpha\leq\beta<1$:

(R4) By an application of L’Hospital’s rule, $\lim_{\alpha\to 1_{-}}D_{\alpha}(p,q)=D(p,q)$.

Contraction and Bayesian oracle inequalities for fractional posteriors

In this section, we present our main results. To begin with, we introduce the background including the definition of fractional posterior distributions in Bayesian procedures in §3.1. Then we present our results on the contraction of fraction posterior distributions in §3.2, and Bayesian oracle inequalities based on PAC-Bayes type bounds in §3.3.

We will present our theory on the large sample properties of fractional posteriors in its full generality by allowing the model to be misspecified and the observations, denoted by $X^{(n)}=(X_{1},X_{2},\ldots,X_{n})$, to be neither identically nor independently distributed (abbreviated as non-i.i.d.) . Our non-i.i.d. result can be applied to models with nonindependent observations such as Gaussian time series and Markov processes, or models with independent, nonidentically distributed (i.n.i.d.) observations such as Gaussian regression and density regression.

Let $\Pi_{n,\alpha}(\cdot)$ denote the posterior distribution obtained by combining the fractional likelihood $L_{n,\alpha}$ with the prior $\Pi_{n}$, that is, for any measurable set $B\in\mathcal{B}$,

where $r_{n}(\theta,\theta^{\dagger}):\,=\log\{p^{(n)}_{\theta^{\dagger}}(X^{(n)})/p^{(n)}_{\theta}(X^{(n)})\}$ is the negative log-likelihood ratio between $\theta$ and any other fixed parameter value $\theta^{\dagger}$. For example, we may choose $\theta^{\dagger}$ as the parameter $\theta_{0}$ associated with the true data generating distribution, abbreviated as the true parameter. Clearly, $\Pi_{n,1}$ denotes the usual posterior distribution.

plays the role of $\theta_{0}$ in well-specified models . In fact, we will show that the fractional posterior distribution $\Pi_{n,\alpha}$ tends to contract towards $\theta^{\ast}$ as $n\to\infty$. We use the divergence

is an $\alpha$-affinity between $\theta$ and $\theta^{\ast}$ with respect to $\theta_{0}$.

Remark: In the well-specified case where $\theta^{\ast}=\theta_{0}\in\Theta$, $A^{(n)}_{\theta_{0},\alpha}$ reduces to the usual $\alpha$-affinity defined in §2.2, and $D^{(n)}_{\theta_{0},\alpha}$ becomes the Rényi divergence of order $\alpha$ between $p^{(n)}_{\theta}$ and $p^{(n)}_{\theta_{0}}$:

If $\{p_{\theta}^{(n)}:\theta\in\Theta\}$ is convexGiven any $\theta,\theta^{\prime}\in\Theta$, and $\omega\in(0,1)$, there exists $\bar{\theta}\in\Theta$ such that $p_{\bar{\theta}}^{(n)}=\omega p_{\theta}^{(n)}+(1-\omega)p_{\theta^{\prime}}^{(n)}$ or $\theta^{*}$ is an interior point of $\Theta$, then $0<A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})\leq 1$ for any $\alpha\in(0,1)$. Therefore, $D^{(n)}_{\theta_{0},\alpha}$ defines a divergence that satisfies $D^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})\geq 0$ for $\theta\in\Theta$ and $D^{(n)}_{\theta_{0},\alpha}(\theta^{\ast},\theta^{\ast})=0$.

We will primarily focus on the following two cases in this paper.

where $p_{\theta}$ is the common density indexed by $\theta\in\Theta$. The negative log-likelihood ratio $r_{n}(\theta,\theta^{\dagger})=\sum_{i=1}^{n}\log\{p_{\theta^{\dagger}}(X_{i})/p_{\theta}(X_{i})\}$ becomes the sum of individual log density ratios. Moreover, the $\alpha$-affinity and divergence can be simplified as A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})=\big{\{}A_{\theta_{0},\alpha}(\theta,\theta^{\ast})\big{\}}^{n} and $D^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})=n\,D_{\theta_{0},\alpha}(\theta,\theta^{\ast})$, where $A_{\theta_{0},\alpha}(\theta,\theta^{\ast})$ and $D_{\theta_{0},\alpha}(\theta,\theta^{\ast})$ respectively are the $\alpha$-affinity and divergence for $n=1$.

Independent observations:

and the negative log-likelihood ratio $r_{n}(\theta,\theta^{\dagger})=\sum_{i=1}^{n}\log\{p_{\theta^{\dagger},i}(X_{i})/p_{\theta,i}(X_{i})\}$. The $\alpha$-affinity and divergence can be decomposed, respectively, as $A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})=\prod_{i=1}^{n}A_{\theta_{0},\alpha,i}(\theta,\theta^{\ast})$ and $D^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})=\sum_{i=1}^{n}D_{\theta_{0},\alpha,i}(\theta,\theta^{\ast})$, where $A_{\theta_{0},\alpha,i}(\theta,\theta^{\ast})$ and $D_{\theta_{0},\alpha,i}(\theta,\theta^{\ast})$ are the $\alpha$-affinity and divergence associated with the $i$th observation $X_{i}$.

2 General concentration bounds

In this subsection, we consider the asymptotic behavior of fractional posterior distributions and corresponding Bayes estimators based on non-i.i.d. observations under the general misspecified framework. We give general results on the rate of contraction of the fractional posterior measure towards the KL minimizer $\theta^{\ast}$ relative to the $\alpha$-divergence $D^{(n)}_{\theta_{0},\alpha}$.

For any $\theta^{\dagger}$, we define a specific kind of KL neighborhood of $\theta^{\dagger}$ with radius $\varepsilon$ as

It is standard practice to make assumptions on the prior mass assigned to such KL neighborhoods to obtain the rate of posterior concentration in misspecified models . With these notations, we present a nonasymptotic upper bound for the posterior probability assigned to complements of $\alpha$-divergence neighborhoods of $\theta^{\ast}$ with respect to $\theta_{0}$.

Fix $\alpha\in(0,1)$. Recall $\theta^{*}$ from (4). Assume that $\varepsilon_{n}$ satisfies $n\,\varepsilon_{n}^{2}\geq 2$ and

Theorem 3.2 characterizes the contraction of the fractional posterior measure where the posterior of $D^{(n)}_{\theta_{0},\alpha}(\theta,\,\theta^{\ast})$ exhibits a sub-exponentially decaying tail. As a direct consequence, we have the following corollary that characterizes the fractional posterior moments of $D^{(n)}_{\theta_{0},\alpha}(\theta,\,\theta^{\ast})$.

Under the conditions of Theorem 3.2, we have that for any $k\geq 1$,

Implications for well-specified models. While Theorem 3.2 and Corollary 3.3 apply generally to the misspecified setting, it is instructive to first consider their implications in the well-specified setting, i.e., when the data generating parameter $\theta_{0}\in\Theta$. Setting $t=1$ in Theorem 3.2 implies that the fractional posterior increasingly concentrates on $\varepsilon_{n}$-sized $D_{\theta_{0},\alpha}^{(n)}$ neighborhoods of the true parameter $\theta_{0}$. In particular, given (R2) and (R3), Theorem 3.2 implies that for any $\alpha\in(0,1)$, the rate of concentration of the fractional posterior $\Pi_{n,\alpha}$ in the Hellinger metric is $\varepsilon_{n}$. Similar concentration results for the usual posterior distribution in the Hellinger metric were established in for the i.i.d. case, and in for the non-i.i.d. case. Since the prior mass condition (10) appears as one of the sufficient conditions there as well, the fractional posterior achieves the same rate of concentration as the usual posterior (albeit up to constants) in all the examples considered in these works, which is typically minimax up to a logarithmic term for appropriately chosen priors. In addition to the prior mass condition (10), the sufficient conditions of additionally require the construction of sieves $\mathcal{F}_{n}\subset\Theta$ whose $\varepsilon_{n}$-entropy in the Hellinger metric is stipulated to grow in the order $\lesssim e^{Cn\varepsilon_{n}^{2}}$, and at the same time, the prior probability assigned to the complement of the sieve is required to be exponentially small, i.e., $\Pi_{n}(\mathcal{F}_{n}^{c})\leq e^{-C^{\prime}n\varepsilon_{n}^{2}}$. The existence of such sieves with suitable control over their metric entropy is a crucial ingredient of their theory, as it guarantees existence of exponentially consistent test functions to test the true density against complements of Hellinger neighborhoods of the form \{\theta\in\mathcal{F}_{n}:h^{2}\big{(}p_{\theta}^{(n)},p^{(n)}_{\theta_{0}}\big{)}\geq M\varepsilon_{n}^{2}\}.

An important distinction for the fractional posterior in Theorem 3.2 is that the prior mass condition alone is sufficient to guarantee optimal concentration. This is important for at least two distinct reasons. First, the condition of exponentially decaying prior mass assigned to the complement of the sieve implies fairly strong restrictions on the prior tails and essentially rules out heavy-tailed prior distributions on hyperparameters. On the other hand, a much broader class of prior choices lead to provably optimal posterior behavior for the fractional posterior. Second, obtaining tight bounds on the metric entropy in non-regular parameter spaces, for example, in shape-constrained regression problems, can be a substantially nontrivial exercise , which is entirely circumvented using the fractional posterior approach. Specific examples of either kind are provided in §4.

While it may be argued that the conditions on the entropy and complement probability of the sieve are only sufficient conditions, a counterexample from suggests that some control on the complexity of the parameter space is also necessary to ensure the consistency of a regular posterior when the model space is well-specified. Specifically, in their example, the posterior tends to put all its mass on a set of distributions that are $\sqrt{2-\sqrt{2}}$ away from the true data generating distribution with respect to the Hellinger metric, even though the prior assigns positive probability over any $\varepsilon$-KL ball around the true parameter. As an implication, the fractional posterior can still achieve a certain rate of contraction for this problem even though the regular posterior is not consistent. In fact, the rate of concentration of the fractional posterior $\varepsilon_{n}=(1-\alpha)^{-1}n^{-1/3}$ for this problem, since their prior satisfies $\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))\geq e^{-C\,\varepsilon^{-1}}$ for some constant $C>0$. Therefore, a combination of Theorem 3.2 and the counterexample in shows that the fractional posterior has an annealing effect that can flatten the potential peculiar spikes in the regular posterior that are far away from the true parameter. However, this additional flexibility of the fractional posterior comes at a price—when the regular posterior contracts, then the $\alpha$-fractional posterior will sacrifice a factor of $(1-\alpha)^{-1}$ in the rate of contraction.

The following theorem shows that for fixed $n$, the fractional posterior will almost surely converges to the regular posterior ($\alpha=1$) as $\alpha\to 1_{-}$.

This theorem implies that although for a fixed $\alpha\in(0,1)$, the fraction posterior has the annealing effect of flattening the posterior, it will eventually convergence to the regular posterior as $\alpha\to 1_{-}$ almost surely. This observation also justifies the empirical observation that parallel tempering can boost the convergence of the posterior when the posterior contracts. However, when the posterior is ill-behaved—does not have consistency or has multimodality, then we need a very fine grid for the design of $\alpha$ as $\alpha\to 1_{-}$ in the parallel tempering algorithm, since otherwise all factional posteriors will only exhibit the one big mode around $\theta^{\ast}$ and miss the rest.

A key reference for Bayesian asymptotics in infinite-dimensional misspecified models is , where sufficient conditions analogous to the well-specified case were provided for the posterior to concentrate around $\theta^{*}$. The primary technical difficulty in showing such a result compared to the well-specified case is the construction of test functions, for which proposed a novel solution. Akin to the well-specified case for the regular posterior, the sufficient conditions of constitute of a prior thickness condition as in Theorem 3.2, and conditions on entropy numbers. However, the entropy number conditions (equations (2.2) and (2.5) in ) for the misspecified case are substantially harder to verify. In their Lemma 2.1, a simpler sufficient condition related their entropy number condition to ordinary entropy numbers. Further, in their Lemma 2.3, exploiting convexity of the parameter space, they established that the sufficient conditions of their Lemma 2.1 are satisfied by a weighted Hellinger distance

which then amounts to obtaining entropy numbers in the weighted Hellinger metric. Such an exercise typically requires further assumptions on the behavior of $p_{\theta}^{(n)}/p_{\theta^{*}}^{(n)}$. For example, if $\sup_{\theta}\left|p_{\theta}^{(n)}/p_{\theta^{*}}^{(n)}\right|$ is finite, the ordinary Hellinger metric dominates the weighted Hellinger metric and it suffices to obtain covering numbers with respect to the ordinary Hellinger metric. Under this assumption, the authors proceeded to derive convergence rates for the regular posterior in a density estimation problem using Dirichlet process mixture priors. However, this assumption precludes the true density $p_{\theta_{0}}^{(n)}$ to have heavier tails than that prescribed by the model. For example, if the true density is heavier that the class of densities specified by the model, the assumption $\sup\left|p_{\theta}^{(n)}/p_{\theta^{*}}^{(n)}\right|<\infty$ is not satisfied. Typically, in the misspecified case, controlling the prior mass (10) in Theorem 3.2 requires certain tail conditions on $f_{0}$. However, Theorem 3.2 obviates the need to verify any entropy conditions for the fractional posterior. It thus avoids the need to assume $\sup\left|p_{\theta}^{(n)}/p_{\theta^{*}}^{(n)}\right|<\infty$, unless required to verify the prior mass condition.

For $\alpha=1/2$, our divergence measure $D_{1/2}({\theta}^{(n)},{\theta^{*}}^{(n)})$ dominates the weighted Hellinger distance in which derive their convergence rate for the density estimation problem in Theorem 3.1. This can be readily seen from

where the last inequality follows from $\log x\leq x-1$ and the penultimate inequality follows from Lemma 3.1.

3 PAC-Bayes bounds and Bayesian oracle inequalities

In many problems, the performance of a (pseudo) Bayesian approach can be characterized via PAC-Bayes type inequalities . A typical PAC-Bayes inequality takes the form as

where $R$ is a statistical risk function, $\kappa_{n}$ is some tuning parameter, Rem is some remainder term, and $S_{n}$ is some function that measures the discrepancy between $\theta$ and $\theta_{0}$ on the support of $\rho$. We present a PAC-Bayes inequality for the fractional posterior distribution, where the risk function $R$ is a multiple of the $\alpha$-Rényi divergence $D^{(n)}_{\alpha}$ in (6), and $S_{n}(\theta,\theta_{0})$ a multiple of the negative log-likelihood ratio $r_{n}(\theta,\theta_{0})$.

Fix $\alpha\in(0,1)$. Then, for any $\varepsilon\in(0,1)$,

Theorem 3.5 immediately implies an oracle type inequality for the Bayes estimator $\widehat{\theta}_{B}:\,=\int_{\Theta}\theta\,\Pi_{n,\alpha}(d\theta\,|\,X^{(n)})$ by using the convexity of $D^{(n)}_{\alpha}(\cdot,\theta_{0})$ and applying Jensen’s inequality,

for all probability measure $\rho\ll\Pi_{n}$. We call this inequality a Bayesian oracle inequality.

Under this notation, a typical FOI takes a form as

up to some other remainder terms, where $r_{n}$ is a critical radius obtained as the fixed point of certain function depending on $\Psi_{n}$.

Now let us look at the BOI (12), which can be rewritten as

Because we can exchange the expectation with integration, this local average form allows us to use simple probability tools, such as Markov’s inequality and Chebyshev’s inequality, to obtain bounds for the excess risk. This is different from the local supremum form (15), where expectation does not exchange with supremum, and we need much more sophisticated empirical process tools such as chaining and peeling techniques to bound the excess risk (see, for example, ).

As a simple illustration of applying Chebyshev’s inequality to BOI or inequality (11) in Theorem 3.5 to obtain an explicit risk bound for the Bayes estimator, we present the following corollary. Recall the definition of the KL neighorhood $B_{n}(\theta_{0},\varepsilon;\theta_{0})$ defined in (9).

In particular, if we let $\varepsilon_{n}$ to be the Bayesian critical radius that is a stationary point of

The main idea of the proof is to choose the probability measure $\rho$ as $\Pi_{n}(\cdot\cap B_{n}(\theta_{0},\varepsilon;\theta_{0}))/\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))$; the restriction of the prior $\Pi_{n}$ to $B_{n}(\theta_{0},\varepsilon;\theta_{0})$. Under this choice, we have $D(\rho\,||\,\Pi_{n})=-\log\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))$, and $\int r_{n}(\theta,\,\theta_{0})\,\rho(d\theta)$ can be bounded by applying Chebyshev’s inequality. If higher moment constraints on the likelihood ratio $r_{n}(\theta,\,\theta_{0})$ is also included into the definition of $B_{n}(\theta_{0},\varepsilon;\theta_{0})$ in (9), then the probability bound for (17) to hold can be boosted (for details, see Section 2 in ).

According to Corollary 3.6, the overall risk bound in (17) is a balance between two terms: an approximation error term $\varepsilon_{n}^{2}$ and a local complexity measure term $-\frac{1}{n}\log\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))$. For this reason, we will refer to the second term as the local Bayesian complexity. The local Bayesian complexity reflects the compatibility between the prior distribution and the parameter space: if $\Pi_{n}$ is close to a uniform distribution over $\Theta$, then -\log\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))=\log\big{\{}1/\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0})\} is roughly the logarithm of the number of $\varepsilon_{n}$-balls needed to cover a neighborhood of $\theta_{0}$, and therefore is related to the local covering entropy. On the other hand, if some prior knowledge about $\theta_{0}$ is available, then we can combine these knowledge to increase the prior mass around $\theta_{0}$, which may significantly boost the rate of convergence of the Bayes estimator. This observation is consistent with our previous intuition that averaging based (average case analysis) Bayesian approaches sometimes can be better than optimization based (worst case analysis) frequentist approaches. For example, when certain hyperparameter or tuning parameter, such as the regularity of a function class or sparsity level of a regression model, is unknown, then a Bayesian procedure naturally achieves adaptation to those unknown parameters by placing a prior on them that distributes proper weights to different levels of the hyperparameter (see our examples in Section 4). In contrast, a common way to select a tuning parameter in frequentist methods is via cross-validation or data-splitting. These approaches only uses some proportion of data to do estimation, after learning the tuning parameter via the rest, which may not be the most efficient way to use data.

Although Theorem 3.5 is useful for obtaining an BOI, when transformed into form (14) the resulting leading constant $c$ of the approximation error term in the BOI is typically strictly larger than $1$, resulting in a non-sharp oracle inequality. Here, we call an oracle inequality sharp if the leading constant $c$ in (14) is $1$; see, for example, . To solve this issue for the PAC-Bayes inequality in Theorem 3.5, we consider a second class of PAC-Bayes inequalities that directly characterizes the closeness between $\theta$ and the best approximation $\theta^{\ast}$ of $\theta_{0}$ from $\Theta$.

Fix $\alpha\in(0,1)$. Then, for any $\varepsilon\in(0,1)$,

Similar to Corollary 3.6 for a concrete Bayesian risk bound for characterizing the closeness between $\theta$ and $\theta_{0}$, we have the following counterpart for $\theta$ and $\theta^{\ast}$.

In particular, if we let $\varepsilon_{n}$ to be the Bayesian critical radius that is a stationary point of

Suppose we are interested in certain metric $d_{n}$, the square of which is weaker than the average $\alpha$-divergence $\frac{1}{n}D^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})$, that is

where $c_{\alpha}$ is some positive constant that may depend on $\alpha$. For simplicity, we assume that $\theta^{\ast}$ is also the minimizer of $d_{n}(\theta,\theta_{0})$ over $\theta\in\Theta$. If this is not the case, then we can always add an extra remainder term to the upper bound that characterizes the difference between $\theta^{\ast}$ and the best approximation of $\theta_{0}$ from $\Theta$ relative to $d_{n}$. Under these assumptions, Corollary 3.8 implies that with high probability, the Bayes estimator $\widehat{\theta}_{B}$ satisfies

where $\varepsilon_{n}$ is the Bayesian critical radius. Now adding $d_{n}(\theta^{\ast},\,\theta_{0})$ to both sides of this inequality and applying the triangle inequality, we obtain

which is a sharp oracle inequality. Sometimes, we may be interested in obtaining an oracle inequality for the squared loss $d_{n}^{2}$, when $\Theta$ is a vector space and $d_{n}$ is induced by an inner product, denoted by $\langle\cdot,\cdot\rangle_{n}$. This is a more intricate problem, as the trivial bound $d_{n}^{2}(\widehat{\theta}_{B},\theta_{0})\leq 2[d_{n}^{2}(\widehat{\theta}_{B},\theta^{\ast})+d_{n}^{2}(\theta^{\ast},\theta_{0})]$ renders the oracle inequality non-sharp. However, it is usually true when $\Theta$ is a convex set that

For example, this inequality holds for regression with fixed design, where $d_{n}$ is the $L_{2}$ empirical norm (details can be found in Section 4.1). Again, by applying Corollary 3.8 and adding $d_{n}^{2}(\theta^{\ast},\,\theta_{0})$ to both sides of this inequality, we obtain

which is a sharp oracle inequality for the squared loss $d_{n}^{2}$. As an application of this technique, we derive a sharp oracle inequality for estimating a convex function in Theorem 4.2 when $f_{0}$ is not necessarily convex.

The most relevant PAC-Bayes type result to ours, such as Theorem 3.5, is the Theorem 1 in , which focus on the regression setting $Y_{i}=f(x_{i})+w_{i}$, where $\theta=f$ is the unknown regression function to be estimated, $x_{i}$’s are the fixed design points and $w_{i}$’s are the i.i.d. zero mean noise, corresponding to the i.n.i.d. observations. They propose to use the posterior mean of the following quasi-likelihood function as the estimator,

where according to their terms, $\beta$ is a temperature parameter. In the special case when $w_{i}\sim N(0,\sigma^{2})$ and $\beta=\sigma^{2}$, this function reduces to the likelihood function. They establish a PAC-Bayes inequality

Examples

In this section, we demonstrate the salient features of our theory through a number of illustrative examples. All results in this section are stated in the form of PAC-Bayes bounds derived from Corollary 3.6. We note that one could alternatively use Theorem 3.2 to obtain similar conclusions. Our first example illustrates the efficacy of the fractional posterior approach in shape-constrained estimation. In the well-specified case, we demonstrate the optimal concentration in estimating a convex function, where we bypass the need to compute the covering entropy of the convex function space. To best of our knowledge, such a result is not available in the Bayesian literature. In the model misspecified case, we derive a sharp Bayesian oracle inequality that extends the recent sharp oracle inequality for one-dimensional convex regression obtained in to general dimension $d\geq 1$. The next two examples concern the classical nonparametric regression and nonparametric density estimation, where we show that the fractional posterior optimally and adaptively concentrates at the true parameter value with substantially relaxed assumptions on the prior compared to existing theory. These two examples also theoretically justify the adaptation nature of integration based Bayesian approaches.

We start with a general regression problem. Consider the following nonparametric regression model with fixed design

where $\psi=\sigma/\sqrt{\alpha}$ and $\phi_{\sigma}$ stands for the multivariate normal density with zero mean and covariance matrix $\sigma^{2}I_{n}$. Hence the fractional posterior for (20) is essentially a standard posterior with a different variance parameter in the likelihood.

therefore, the function $\mu^{\ast}\in\mathcal{F}$ that minimizes the KL divergence is

Moreover, straightforward calculations show

With some simple algebra, $T=(1-\alpha)\|\widetilde{\mu}-\widetilde{\mu}^{\ast}\|^{2}+2\langle\widetilde{\mu}-\widetilde{\mu}^{\ast},\widetilde{\mu}^{\ast}-\widetilde{\mu}_{0}\rangle\geq 0$ by the above inequality. Hence, $D_{\mu_{0},\alpha}^{(n)}(\mu,\,\mu^{\ast})$ defines a valid divergence measure if $K$ is convex. It is straightforward to verify that $K$ is convex if the class of functions $\mathcal{F}$ is monotone or convex.

The observations in the previous paragraph suggest the applicability of our framework to shape-constrained regression problems. We provide an illustration via convex regression, where $\mathcal{F}$ is the function space of all $d$-dimensional convex functions over $^{d}$. Our fractional posterior framework becomes especially attractive in such problems, since it obviates the need to compute entropy numbers in restricted spaces, which can be a challenging exercise in itself .

It is recent practice in the frequentist literature to avoid additional smoothness assumptions on convex functions while studying rates of convergence . To that end, let $\partial\mu(x)$ denote the sub-gradient of the function $\mu$ at the point $x$, that is,

As in , define the class of convex, sub-differentiable, uniformly Lipschitz functions on $^{d}$ as

We model $\mu$ as a maximum of hyperplanes , with a prior distribution for the number of affine functions over which the maximum is taken. Specifically, we let

The following theorem shows that in the well-specified case where $\mu_{0}\in\mathcal{F}$, with no additional smoothness condition on $\mu_{0}$, we obtain a Bayes risk bound of the order $n^{-2/(4+d)}$ up to logarithmic terms, which coincides with the minimax risk under any $d\geq 1$ .

where $\varepsilon_{n}=n^{-2/(4+d)}\log^{t}n$ with $t=2/(4+d)$, and $C$ is some constant independent of $\alpha$.

Now we consider the misspecified case, where $\mu_{0}$ maybe a non-convex function. In this case, $\mu^{\ast}$ is the projection of $\mu_{0}$ into $\mbox{Co}_{L}^{d}$ relative to the $\|\cdot\|_{2,n}$ norm. We obtain the following sharp Bayesian oracle inequality, which generalizes the result of one-dimensional convex regression obtained in to general dimension $d\geq 1$.

where $\varepsilon_{n}$ is given in Theorem 4.1, and $C$ is some constant independent of $\alpha$.

This sharp oracle inequality implies some geometric structure of the fractional posterior that cannot be obtained via a non-sharp one. For example, as an immediate consequence, if we let $\mu^{\ast}$ be any minimizer of $\left\|\mu-\mu_{0}\right\|_{2,n}^{2}$ over $\mbox{Co}_{L}^{d}$, then (24) implies

Since $\langle\mu-\mu^{\ast},\mu^{\ast}-\mu_{0}\rangle_{n}$ is nonnegative for all $\mu\in\mbox{Co}_{L}^{d}$ due to the convexity of $\mbox{Co}_{L}^{d}$ (see, for example, ), this display suggests that the fractional posterior distribution of $\mu$ tends to concentrate on the narrow cone with vertex $\mu^{\ast}$ consisting of points such that the angle between vectors $\mu-\mu^{\ast}$ and $\mu^{\ast}-\mu_{0}$ is of order

That is, with large fractional posterior probability, $\mu-\mu^{\ast}$ is almost perpendicular to $\mu^{\ast}-\mu_{0}$.

We conjecture that the current technique will allow us to find sharp oracle inequalities for Bayes estimators in monotone function estimation as well. Sharp oracle inequalities for isotonic regression for $d=1$ has been recently established in , improving on a previous risk bound by . We leave this as a topic for future research.

Nonparametric GP regression:

With this assumption, we show that the fractional posterior concentrates at the minimax rate (up to logarithmic terms) adaptively over \mu_{0}\in C^{\beta}^{d}, where $\beta$ is the unknown smoothness level of $\mu_{0}$. To obtain the same result for the usual posterior, require the prior on $a$ to additionally satisfy an upper bound of the same order as the lower bound in (25), once again, ruling out heavy tailed priors.

Consider the model (20), with a conditional GP prior $\mu\mid a\sim\mbox{GP}(0,c_{a})$ and suppose $a\sim g(\cdot)$ satisfies (25). If the true function \mu_{0}\in C^{\beta}^{d}, then (24) is satisfied with $\varepsilon_{n}=n^{-\beta/(2\beta+d)}(\log n)^{t}$, where $t=\{(1+d)\vee q\}/(2+d/\alpha)$.

Theorem 4.3 can be extended to other kernels in a straightforward manner.

2 Nonparametric density estimation

Assumptions on the true density: We assume $f_{0}\in\mathcal{C}^{\beta,L,\tau_{0}}$ and for all $k\leq\lfloor\beta\rfloor$ and some $\varepsilon>0$,

Moreover, for all sufficiently large $y\in\mathcal{Y}$ and some positive $(c,b,\tau)$,

We model the density $f$ of i.i.d. observations $X_{1},\ldots,X_{n}$ via a mixture of finite mixtures (MFM; ), which is a finite mixture model with a prior on the number of mixture components. With some minor modifications, the results can be adapted to infinite mixture models, such as Dirichlet process mixtures . Our concentration results can accommodate heavy tailed prior distributions on the component specific means.

Prior: We model the unknown density by a location mixture of normal densities

We assign a Dirichlet$(a/m,\ldots,a/m)$ prior for $(\alpha_{1},\ldots,\alpha_{m})$ given $m$, where $a>0$ is a fixed constant, and let

Finally, we assume that the $\mu_{j}$s are independent from other parameters and across $j$, with a prior density satisfying

Let $p_{\theta_{0}}^{(n)}=\prod_{i=1}^{n}f_{0}(X_{i})$ and $p_{\theta}^{(n)}=\prod_{i=1}^{n}p(y_{i}\mid\psi,m)$, where $\theta=\{(\mu_{j},\alpha_{j})_{j=1}^{\infty},\sigma,m\}$. Then $\frac{1}{n}D^{(n)}_{\alpha}(\theta,\theta_{0})=D_{\alpha}(f_{0},p(\cdot\mid\psi,m))$, abbreviated by $D_{\alpha}(f_{0},p)$.

where $\varepsilon_{n}=Cn^{-\beta/(2\beta+d)}(\log n)^{t}$ with $t>t_{0}+\max\{0,(1-\tau_{1})/2\}$, $t_{0}=(ds+\max\{\tau_{1},1,\tau_{2}/\tau\})/(2+d/\beta)$, and $s=1+1/\beta+1/\tau$ for some constant $C>0$.

Inverse-gamma, exponential and half-Cauchy families of densities satisfy (29). In addition to (29), also require two-sided prior tail bounds

which impose additional restrictions for the prior choice on $\sigma$, in particular, ruling out heavy-tailed priors. Assumption (31) allows heavy-tailed prior distributions on the component specific means $\mu_{j}$s, which is not permissible in the existing theory due to the additional requirement on the tail behavior

for some $a_{13},\tau_{3}>0$ and all sufficiently large $r>0$.

Proofs

In this section, we present proofs of our results in the main sections.

It is immediate from the definition of $A^{(n)}_{\theta_{0},\alpha}$ that $A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})>0$ for all $\theta\in\Theta$. We prove the second part that $A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})\leq 1$. Let

It follows that $g^{\prime}(0+)=1-A^{(n)}_{\theta_{0},1}(\theta,\theta^{\ast})$. Using Jensen’s inequality, we have A^{(n)}_{\theta_{0},\alpha}(\theta,\theta^{\ast})\leq\big{\{}A^{(n)}_{\theta_{0},1}(\theta,\theta^{\ast})\big{\}}^{\alpha}. Therefore, it suffices to show that $g^{\prime}(0+)\geq 0$.

For $x\geq 0$, $h_{x}$ monotonically increases to $x$ as $\phi\downarrow 0$. We have the decomposition

Now, we can apply the monotone convergence theorem to the first term and bounded convergence theorem to the second term in this display to obtain that $g^{\prime}(0+)=\lim_{\phi\to 0+}\{g(\phi)-g(0)\}/\phi$ exists and is greater than or equal to zero.

2 Proof of Theorem 3.2

Then, we can express the desired posterior probability as

Let us first consider the numerator. By the definition of the $\alpha$-divergence $D^{(n)}_{\theta_{0},\alpha}(\theta,\,\theta^{\ast})$, we have

Now integrating both side with respect to the prior $\Pi$ over $U_{n}$ and applying Fubini’s theorem, we can get

where the last step follows from the definition of $U_{n}$. An application of the Markov inequality yields the following high probability bound for the numerator on the right hand side of (34),

Next, we consider the denominator on the right hand side of (34). We always have the lower bound

We invoke the following result (Lemma 8.1, ), which is a high probability lower bound to $\int_{B_{n}(\theta^{\ast},\varepsilon_{n};\theta_{0})}e^{-\alpha\,r_{n}(\theta,\,\theta^{\ast})}\Pi_{n}(d\theta)$ (which has been adapted to the $\alpha$-fractional likelihood): for any $D>1$, we have

Now combining (34), (36) and (37), we obtain that with probability at least $1-2/\{(D-1+t)^{2}n\varepsilon_{n}^{2}\}$,

3 Proof of Corollary 3.3

An application of a union probability bound to Theorem 3.2 yields that

Therefore, using this bound and applying Fubini’s theorem, we obtain

4 Proof of Theorem 3.4

Recall that the density of $\Pi_{n,1}$ with respect to the prior $\Pi_{n}$ is

Since $e^{-\alpha\,s_{n}(f)}$ as a function of $\alpha$ is monotonically increasing in $A_{n}(X^{(n)})$ and bounded by $1$ in $B_{n}(X^{(n)})$, by applying the monotone convergence theorem for the first term and bounded convergence theorem for the second in the preceding display, we obtain

Similarly, for any measurable set $B\in\mathcal{B}$, we have the decomposition

Therefore, by the monotone convergence theorem and the bounded convergence theorem, we have

Combining (38) and (39), we obtain that for any $X^{(n)}\in E_{n}$,

Since this is true for any measurable set $B\in\mathcal{B}$, we proved the theorem.

5 Proof of Theorem 3.5

We first state a key variational lemma that plays a critical role in the proof.

Let $\mu$ be a probability measure and $h$ a measurable function such that $e^{h}\in L_{1}(\mu)$. Then,

Further, the supremum on the right hand side is attained when

Fix $\rho\ll\mu$ and let $f_{\rho}=d\rho/d\mu$. Without loss of generality, we may assume $\mu\ll\rho$ so that $d\mu/d\rho=1/f_{\rho}$, since otherwise we can always find a common dominating measure. Now, we have, by applying Jensen’s inequality to the convex function $e^{x}$, that

Further, we have equality when $\log f_{\rho}-h$ is constant, or equivalently, when $d\rho/d\mu\propto e^{h}$. ∎

Return to the proof of the theorem. Recall the definition of the $\alpha$-Renyi divergence and $\alpha$-affinity that

Thus, for any $\varepsilon\in(0,1)$, we have

Integrating both side of this inequality with respect to $\Pi_{n}$ and interchanging the integrals using Fubini’s theorem, we obtain

If we choice $\rho(\cdot)=\Pi_{n,\alpha}(\cdot\,|\,X^{(n)})$ as the fractional posterior distribution in the preceding display, then

Noting the relation between $r_{n}(\theta,\theta_{0})$ and $\Pi_{n,\alpha}(\cdot\,|\,X^{(n)})$ by (3) with $\theta^{\dagger}=\theta_{0}$, we can apply Lemma 5.1 with $h=-\alpha\,r_{n}(\theta,\theta_{0})$ to obtain that

The second part is a direct consequence by combining the results in the two preceding displays.

6 Proof of Corollary 3.6

The second part is a direct consequence of the first part, which we are going to prove. Pick $\varepsilon=e^{-\alpha n\varepsilon^{2}}$ in Theorem 3.5 and let $\mathcal{A}\in\sigma(X_{1},\ldots,X_{n})$ denote the set on which (11) holds. Picking $\rho$ as $\rho^{\ast}=\Pi_{n}(\cdot\cap B_{n}(\theta_{0},\varepsilon;\theta_{0}))/\Pi_{n}(B_{n}(\theta_{0},\varepsilon;\theta_{0}))$, the restriction of the prior $\Pi_{n}$ to $B_{n}(\theta_{0},\varepsilon;\theta_{0})$. Let $\mathcal{A}^{\prime}\in\sigma(X_{1},\ldots,X_{n})$ denote the event in which

Thus, by applying Cauchy-Schwarz inequality, Chebyshev’s inequality and Fubini’s theorem, we have

where in the last step we used the definition of $B_{n}(\theta_{0},\varepsilon;\theta_{0})$.

7 Proof of Theorem 3.7

The proof follows the same lines as the proof of Theorem 3.5 in Section 5.5. The only difference is that we use (35) instead of (40) when applying the variational lemma to obtain the PAC-Bayes inequality. Due to space constraints, we omit the proof.

8 Proof of Corollary 3.8

The proof follows the same lines as the proof of Corollary 3.6 in Section 5.6. The only difference is that we use $B_{n}(\theta^{\ast},\varepsilon;\theta_{0})$ instead of $B_{n}(\theta_{0},\varepsilon;\theta_{0})$ in the definition of $\rho$ and in the application of Chebyshev’s inequality. Due to space constraints, we omit the proof.

9 Proof of Theorem 4.1

In the well-specified case, we have $\mu^{\ast}=\mu_{0}$. We choose $\alpha=1/2$ so that $D^{(n)}_{\theta_{0},1/2}(\theta,\theta_{0})=\left\|\mu-\mu_{0}\right\|_{2,n}^{2}/(4\sigma^{2})$ and

To lower bound $\Pi(\left\|\mu-\mu_{0}\right\|_{\infty}<C\varepsilon)$, we consider the sum over $k\in[\lceil(\varepsilon/2)^{-d/2}\rceil,2\lceil(\varepsilon/2)^{-d/2}\rceil]$. Using $\pi(k)\geq\exp\{-Ck\log k\}$, we have $\Pi(\left\|\mu-\mu_{0}\right\|_{\infty}<C\varepsilon)\geq e^{-C_{3}\varepsilon_{n}^{-d/2}\log(1/\varepsilon)}$ for sufficiently small $\epsilon$. Hence $\Pi(\left\|\mu-\mu_{0}\right\|_{\infty}<C\varepsilon_{n})\geq e^{-C_{4}n\varepsilon_{n}^{2}}$ is satisfied for $\varepsilon_{n}=n^{-2/(4+d)}(\log n)^{t}$ for $t=2/(4+d)$.

10 Proof of Theorem 4.2

We apply Corollary 3.8 to obtain that with probability tending to one,

By the convexity of $\mbox{Co}_{L}^{d}$ and the definition of $\mu^{\ast}$ as the projection, we have that for any $\mu\in\mbox{Co}_{L}^{d}$,

Therefore, inequality (42) in particular implies

Plugging this back into (42), and noting that $\|\mu^{\ast}-\mu_{0}\|_{2,n}^{2}$ is independent of $\mu$, we obtain

for some constant $C_{1}$ independent of $n$. Now use the prior concentration results derived in the proof of Theorem 4.1, we obtain

for some constant $C_{2}>0$. Putting pieces together, we obtain by choosing $\varepsilon=\varepsilon_{n}$ that

implying the desired Bayesian oracle inequality.

11 Proof of Theorem 4.3

Similar to the proof of Theorem 4.1, it suffices to obtain a lower bound on $\Pi(\left\|\mu-\mu_{0}\right\|_{\infty}<C_{1}\varepsilon)$. From Section 5.1 (pp. 2671) of , it can be shown that for $\varepsilon$ sufficiently small

Therefore, $\Pi(\left\|\mu-\mu_{0}\right\|_{\infty}<C_{1}\varepsilon_{n})\geq e^{-n\varepsilon_{n}^{2}}$ is satisfied with $\varepsilon_{n}\geq n^{-\alpha/(2\alpha+d)}(\log n)^{t}$ where $t=\{(1+d)\vee q\}/(2+d/\alpha)$. From Corollary 3.6 with $\alpha=1/2$ and $\varepsilon_{n}=n^{-\alpha/(2\alpha+d)}(\log n)^{t}$, we have,

12 Proof of Theorem 4.4

The proof of Theorem 4.4 follows in a straightforward manner from with the following modifications. To apply Corollary 3.6, we need to prove that for the $\varepsilon$ stated in the statement of Theorem 4.4, we have $-\log\Pi_{n}(B_{n}(\theta_{0},\varepsilon_{n};\theta_{0}))\leq n\varepsilon_{n}^{2}$. For $\sigma_{n}=\varepsilon_{n}/\log(1/\varepsilon_{n})]^{1/\beta}$, $\varepsilon$ defined in (26), a sufficiently small $\delta>0$, $b$ and $\tau$ defined in (27), $a_{0}=\{(8\beta+4\varepsilon+16)/(b\delta)\}^{1/\tau}$, $a_{\sigma_{n}}=a_{0}\{\log(1/\sigma_{n})\}^{1/\tau}$, and $b_{1}>\max\{1,1/2\beta\}$ satisfying $\varepsilon_{n}^{b_{1}}\{\log(1/\varepsilon_{n})\}^{5/4}\leq\varepsilon_{n}$, the proof of Theorem 4 in implies the following three claims. First, there exists a partition of $\{y\in\mathcal{Y}:,y,\leq a_{\sigma_{n}}\}$, $\{U_{j},j=1,\ldots,K\}$ such that for $j=1,\ldots,N$, $U_{j}$ is a ball with diameter $\sigma_{n}\varepsilon_{n}^{2b_{1}}$ and center $y_{j}$; for $j=N+1,\ldots,K$, $U_{j}$ is a set with a diameter bounded above by $\sigma_{n}$; $1\leq N<K\leq C_{2}\sigma_{n}^{-d}\{\log(1/\varepsilon_{n})\}^{d+d/\tau}$, where $C_{2}>0$ does not depend on $n$. Second, there exist $\theta^{\star}=\{\mu_{j}^{\star},\alpha_{j}^{\star},j=1,2,\ldots;\sigma_{n}\}$ with $\alpha_{j}^{\star}=0$ for $j>N$, $\mu_{j}^{\star}=z_{j}$ for $j=1,\ldots,N$, and $\mu_{j}^{\star}\in U_{j}$ for $j=N+1,\ldots,K$ such that for $m=K$ and a positive constant $C_{3}$,

Third, there exists constant $B_{0}>0$ such that

For $\theta\in S_{\theta^{\star}}$, where,

For $j=1,\ldots,N$, $\left\|\phi_{\mu_{j}^{\star},\sigma_{n}}-\phi_{\mu_{j},\sigma_{n}}\right\|_{1}\leq,\mu_{j}^{\star}-\mu_{j},/\sigma_{n}\leq\varepsilon_{n}^{2b_{1}}$. Also,

where the penultimate inequality follows from the fact that $\left|\log x-x+1\right|\leq C_{4}\left|x-1\right|^{2}$ for $x$ in a neighborhood of 1 and some $C_{4}>0$. Hence, $d_{h}(p(\cdot|\theta,m),p(\cdot|\theta^{\star},m))\leq C_{5}\sigma_{n}^{\beta}$for some $C_{5}>0$, all $\theta\in S_{\theta^{\star}}$, and $m=K$.

Next, for $\theta\in S_{\theta^{\star}}$, let us consider a lower bound on the ratio $p(y|\theta,m)/f_{0}(y)$. Note that $\sup_{y}f_{0}(y)<\infty$ and $p(y|\theta,m)\geq\sigma^{d}p(y|\theta,m)$. For $y\in\mathcal{Y}$ with $\left\|z\right\|\leq a_{\sigma_{n}}$, there exists $J\leq K$ for which $,y-\mu_{J},\leq\sigma_{n}$. Thus, for all sufficiently large $n$ such that $\sigma_{n}^{2}/\sigma^{2}\leq 2$, $p(y|\theta,m)\geq\min_{j}\alpha_{j}\cdot\phi_{\mu_{J},\sigma}(y)\geq[\varepsilon_{n}^{4db_{1}}/2]\cdot\sigma_{n}^{-d}e^{-1}/(2\pi)^{d/2}$ and

For $y\in\mathcal{Y}$ with $\left\|y\right\|>a_{\sigma_{n}}$, $\left\|z-\mu_{j}\right\|^{2}\leq 2(\left\|z\right\|^{2}+\left\|\mu\right\|^{2})\leq 4\left\|y\right\|^{2}$ for all $j=1,\ldots,K$. Thus, for all sufficiently large $n$, $p(y|\theta,m)\geq\sigma_{n}^{-d}\exp(-4\left\|y\right\|^{2}/\sigma_{n}^{2})/(2\pi)^{d/2}$ and

Denote the lower bound in (47) by $\lambda_{n}$ and consider all sufficiently large $n$ such that $\lambda_{n}<e^{-1}$. For any $\theta\in S_{\theta^{\star}}$,

for some constant $C_{8}$. The last inequality follows from (45) and tail condition in (27). Also note that

From Lemma 4 of , both $E_{0}(\log(f_{0}(Y)/p(Y|\theta,m)))$ and $E_{0}([\log(f_{0}(Y)/p(Y|\theta,m))]^{2})$ are bounded by $C_{9}\log(1/\lambda_{n})^{2}\sigma_{n}^{2\beta}\leq A\varepsilon_{n}^{2}$ for some constant $A$. Finally, we calculate a lower bound on the prior probability of $m=K$ and $\{\theta\in S_{\theta^{\star}}\}$. By (30), for some $C_{10}>0$,

From Lemma 10 of , for some constants $C_{11},C_{12}>0$ and all sufficiently large $n$,

For $\pi_{\mu}$ denoting the prior density of $\mu_{j}^{y}$ and some $C_{13},C_{14}>0$, (31) implies

Assumption (29) on the prior for $\sigma$, implies

It follows from (48) - (51), that for all sufficiently large $n$, $s=1+1/\beta+1/\tau$, and some $C_{16}>0$

The last expression of the above display is bounded below by $\exp\{-Cn\varepsilon_{n}^{2}\}$ for any $C>0$, $\varepsilon_{n}=n^{-\beta/(2\beta+d)}(\log n)^{t}$, any $t>(ds+\max\{\tau_{1},1,\tau_{2}/\tau\})/(2+d/\beta)$, and all sufficiently large $n$. Since the inequality in the definition of $t$ is strict, the claim of the theorem follows immediately.

References