Sharpening Jensen's Inequality

Introduction

Jensen’s inequality is a fundamental inequality in mathematics and it underlies many important statistical proofs and concepts. Some standard applications include derivation of the arithmetic-geometric mean inequality, non-negativity of Kullback and Leibler divergence, and the convergence property of the expectation-maximization algorithm (Dempster et al., 1977). Jensen’s inequality is covered in all major statistical textbooks such as Casella and Berger (2002, Section 4.7) and Wasserman (2013, Section 4.2) as a basic mathematical tool for statistics.

Let $X$ be a random variable with finite expectation and let $\varphi(x)$ be a convex function, then Jensen’s inequality (Jensen, 1906) establishes

We have incorporated the materials in this paper in our classroom teaching. With only slightly increased technical level and lecture time, we are able to present a much sharper version of the Jensen’s inequality that significantly enhances students’ understanding of the underlying concepts.

Main result

Let $X$ be a one-dimensional random variable with mean $\mu$, and $P(X\in(a,b))=1$, where $-\infty\leq a<b\leq\infty$. Let $\varphi(x)$ is a twice differentiable function on $(a,b)$, and define function

Let $F(x)$ be the cumulative distribution function of $X$. Applying Taylor’s theorem to $\varphi(x)$ about $\mu$ with a mean-value form of the remainder gives

where $g(x)$ is between $x$ and $\mu$. Explicitly solving for $\varphi^{\prime\prime}(g(x))/2$ gives $\varphi^{\prime\prime}(g(x))/2=h(x;\mu)$ as defined above. Therefore

and the result follows because $\inf_{x\in(a,b)}h(x;\mu)\leq h(x;\mu)\leq\sup_{x\in(a,b)}h(x;\mu)$. ∎

Theorem 1 also holds when $\inf h(x;\mu)$ is replaced by $\inf\varphi^{\prime\prime}(x)/2$ and $\sup h(x;\mu)$ replaced by $\sup\varphi^{\prime\prime}(x)/2$ since

These less tight bounds are implied in the economics working paper Becker (2012). Our lower and upper bounds have the general form $J\cdot\text{var}(X)$, where $J$ depends on $\varphi$. Similar forms of bounds are presented in Abramovich and Persson (2016); Dragomir (2014); Walker (2014), but our $J$ in Theorem 1 is much simpler and applies to a wider class of $\varphi$.

Inequality (2) implies Jensen’s inequality when $\varphi^{\prime\prime}(x)\geq 0$. Note also that Jensen’s inequality is sharp when $\varphi(x)$ is linear, whereas inequality (2) is sharp when $\varphi(x)$ is a quadratic function of $x$.

In some applications the moments of $X$ present in (2) are unknown, although a random sample $x_{1},\ldots,x_{n}$ from the underlying distribution $F$ is available. A version of Theorem 1 suitable for this situation is given in the following corollary.

Let $x_{1},\ldots,x_{n}$ be any $n$ datapoints in $(-\infty,\infty)$, and let

where $a=\min\{x_{1},\ldots,x_{n}\}$ and $b=\max\{x_{1},\ldots,x_{n}\}$.

Consider the discrete random variable $X$ with probability distribution $P(X=x_{i})=1/n$, $i=1,\ldots,n$. We have $E[X]=\bar{x}$, $E[\varphi(X)]=\overline{\varphi_{x}}$, and $\text{var}(X)=S^{2}$. Then the corollary follows from application of Theorem 1. ∎

If $\varphi^{\prime}\left(x\right)$ is convex, then $h(x;\mu)$ is monotonically increasing in $x$, and if $\varphi^{\prime}\left(x\right)$ is concave, then $h(x;\mu)$ is monotonically decreasing in $x$.

We prove that $h^{\prime}(x;\mu)\geq 0$ when $\varphi^{\prime}(x)$ is convex. The analogous result for concave $\varphi^{\prime}(x)$ follows similarly. Note that

Without loss of generality we assume $x>\mu$. Convexity of $\varphi^{\prime}(x)$ gives

Lemma 1 makes Theorem 1 easy to use as the follow results hold:

Note the limits of $h(x;\mu)$ can be either finite or infinite. The proof of Lemma 1 borrows ideas from Bennish (2003). Examples of functions $\varphi(x)$ for which $\varphi^{\prime}$ is convex include $\varphi(x)=\exp(x)$ and $\varphi(x)=x^{p}$ for $p\geq 2$ or $p\in(0,1]$. Examples of functions $\varphi(x)$ for which $\varphi^{\prime}$ is concave include $\varphi(x)=-\log x$ and $\varphi(x)=x^{p}$ for $p<0$ or $p\in$.

Examples

For any random variable $X$ supported on $(a,b)$ with a finite variance, we can bound the moment generating function $E[e^{tX}]$ using Theorem 1 to get

For $t>0$ and $(a,b)=(-\infty,\infty)$, we have

So Theorem 1 provides no improvement over Jensen’s inequality. However, on a finite domain such as a non-negative random variable with $(a,b)=(0,\infty)$, a significant improvement in the lower bound is possible because

The less sharp lower bound using $\inf\varphi^{\prime\prime}(x)/2$ is 0.125. Utilizing elaborate approximations and numerical optimizations Walker (2014) yielded a more accurate lower bound of 0.271.

Let $X$ be a positive random variable on interval $(a,b)$ with mean $\mu$. Note that $-\log(x)$ is convex whose derivative is concave. Applying Theorem 1 and Lemma 1 leads to

Now consider a sample of $n$ positive data points $x_{1},\ldots,x_{n}$. Let $\bar{x}$ be the arithmetic mean and $\bar{x}_{g}=(x_{1}x_{2}\cdots x_{n})^{\frac{1}{n}}$ be the geometric mean. Applying Corollary 1.1 gives

where $a$, $b$, $S^{2}$ are as defined in Corollary 1.1. To give some numerical results, we generated 100 random numbers from uniform distribution on . For these 100 numbers, the arithmetic mean $\bar{x}$ is 54.830 and the geometric mean $\bar{x}_{g}$ is 47.509. The above inequality becomes

which are fairly tight bounds. Replacing $h(x_{n};\bar{x})$ by $\varphi^{\prime\prime}(x_{n})/2$ and $h(x_{1};\bar{x})$ by $\varphi^{\prime\prime}(x_{1})/2$ leads to a less accurate lower bound 1.0339 and upper bound 21.698.

Let $X$ be a positive random variable on a positive interval $(a,b)$ with mean $\mu$. For any real number $s\not=0$, define the power mean as

Jensen’s inequality establishes that $M_{s}(X)$ is an increasing function of $s$. We now give a sharper inequality by applying Theorem 1. Let $r\not=0$, $Y=x^{r}$, $\mu_{y}=EY$, $p=s/r$ and $\varphi(y)=y^{p}$. Note that $EX^{s}=E\{\varphi(Y)\}$. Applying Theorem 1 leads to

To apply Lemma 1, note that $\varphi^{\prime}(y)$ is convex for $p\geq 2$ or $p\in(0,1]$ and is concave for $p<0$ or $p\in$ as noted in Section 2.

Applying the above result to the case of $r=1$ and $s=-1$, we have $Y=X$, $p=-1$. Therefore

For the same sequence $x_{1},\ldots,x_{n}$ generated in Example 2, we have $\bar{x}_{\text{harmonic}}=39.113$. Applying Corollary 1.1 leads to

Note that the upper bound 48.905 is much smaller than the arithmetic mean $\bar{x}=54.830$ by the Jensen’s inequality. Replacing $h(b;\bar{x})$ by $\varphi^{\prime\prime}(b)/2$ and $h(a;\bar{x})$ by $\varphi^{\prime\prime}(a)/2$ leads to a less accurate lower bound 0.8298 and 51.0839.

In a recent article published in the American Statistician, de Carvalho (2016) revisited Kolmogorov’s formulation of generalized mean as

where $\varphi$ is a continuous monotone function with inverse $\varphi^{-1}$. The Example 2 corresponds to $\varphi(x)=-\log(x)$ and Example 3 corresponds to $\varphi(x)=x^{s}$. We can also apply Theorem 1 to bound $\varphi^{-1}(E\varphi(X))$ for a more general function $\varphi(x)$.

Rao-Blackwell theorem (Theorem 7.3.17 in Casella and Berger, 2002; Theorem 10.42 in Wasserman, 2013) is a basic result in statistical estimation. Let $\hat{\theta}$ be an estimator of $\theta$, $L(\theta,\hat{\theta})$ be a loss function convex in $\hat{\theta}$, and $T$ a sufficient statistic. Then the Rao-Blackwell estimator, $\hat{\theta}^{*}=E[\hat{\theta}\mid T]$, satisifies the following inequality in risk function

We can improve this inequality by applying Theorem 1 to $\varphi(\hat{\theta})=L(\theta,\hat{\theta})$ with respect to the conditional distribution of $\hat{\theta}$ given $T$:

where function $h$ is defined as in Theorem 1 for $\varphi(\hat{\theta})$ and $P(\hat{\theta}\in(a,b)\mid T)=1$. Further taking expectations over $T$ gives

In particular for square-error loss, $L(\theta,\hat{\theta})=(\hat{\theta}-\theta)^{2}$, we have

Using the original Jensen’s inequality only establishes the cruder inequality in Equation (4).

Improved bounds by partitioning

As discussed in Example 1 above, Theorem 1 does not improve on Jensen’s inequality if $\inf h(x;\mu)=0$. In such cases, we can often sharpen the bounds by partitioning the domain $(a,b)$ following an approach used in Walker (2014). Let

$I_{j}=\left[x_{j-1},x_{j}\right)$, $\eta_{j}=P(X\in I_{j})$, and $\mu_{j}=E(X\mid X\in I_{j})$. It follows from the law of total expectation that

Let $Y$ be a discrete random variable with distribution $P(Y=\mu_{j})=\eta_{j},j=1,2,\ldots,m$. It is easy to see that $EY=EX$. It follows by Theorem 1 that

We can also apply Theorem 1 to each $E[\varphi(X\mid X\in I_{j})]-\varphi(\mu_{j})$ term:

Combining the above two equations, we have

Replacing $\inf$ by $\sup$ in the righthand side gives the upper bound.

The Jensen gap on the left side of (5) is positive if any of the $m+1$ terms on the right is positive. In particular, the Jensen gap is positive if there exists an interval $I\subset(a,b)$ that satisfies $\inf_{x\in I}\varphi^{\prime\prime}(x)>0$, $P(X\in I)>0$ and $\text{var}(X\mid X\in I)>0$. Note that a finer partition does not necessarily lead to a sharper lower bound in (5). The focus of the partition should therefore be on isolating the part of interval $(a,b)$ in which $\varphi^{\prime\prime}(x)$ is close to 0.

Consider example $X{\sim}N\left(\mu,\sigma^{2}\right)$ with $\mu=0$ and $\sigma=1$ and $\varphi(x)=e^{x}$. We divide $(-\infty,\infty)$ into three intervals with equal probabilities. This gives

The actual Jensen gap is $e^{\mu+\frac{\sigma}{2}}-e^{\mu}=0.649$. The lower bound from (5) is 0.409, which is a huge improvement over Jensen’s bound of 0. The upper bound $\infty$, however, provides no improvement over Theorem 1.

To summarize, this paper proposes a new sharpened version of the Jensen’s inequality. The proposed bound is simple and insightful, is broadly applicable by imposing minimum assumptions on $\varphi(x)$, and provides fairly accurate result in spite of its simple form. It can be incorporated in any calculus-based statistical course.

References