James-Stein Type Center Pixel Weights for Non-Local Means Image Denoising

I Introduction

Image noise commonly exists in the image acquisition, quantization, transmission and many other processing stages. A digital image contaminated by noises leads to visible loss in image quality and can impact many advanced image processing and computer vision tasks like tracking, recognition, classification, etc. The importance of image denoising is therefore self-explanatory.

Conventional image denoising methods more or less related to filters , like moving average filters, Wiener filters, and wavelet filter banks. These filter-based image denoising techniques are commonly of low complexity and can be easily implemented in hardware. However, their performance is not always adequate. With the help of the increased computational capacity powered by digital processors, many advanced denoising techniques are now feasible. Among these techniques, the Non-Local Means (NLM) method has attracted significant attention in recent years. The NLM denoises an image pixel as the weighted sum of its noisy neighbors, where each weight reflects the similarity between the local patch centered at the noisy pixel to be denoised and the patch centered at the neighbor pixel. In this way, NLM adapts the denoising process for each pixel and thus outperforms conventional techniques .

Many improvements on the original NLM have been proposed in recent years. These discussions mainly focus on three questions: 1) how to pick NLM parameters heuristically or automatically ; 2) how to accelerate the NLM or save computations without loss of denoising performance ; and 3) how to adjust the NLM framework to achieve better performance . In , the closed-form of Stein’s risk estimator is derived for the NLM and allows prediction of the risk of a denoised image without knowing the clean image. In , the integral image scheme is adapted to the NLM and greatly reduces the computational costs. In , multiple-patches are discussed to eliminate artifacts in the NLM.

The importance of the center pixel weight (CPW) has been noticed for a long time , and various heuristic weights are designed and used . However, the methods proposed are non-ideal as they do not consider all aspects of CPW problem (see details in Section III-A). Thus, new CPWs need to be designed. In this letter, we discuss the CPW problem in NLM and propose new solutions based on the James-Stein estimator . The rest of the letter is organized as: Sec. II reviews the NLM and related works on CPWs; Sec. III discusses the new formulation of the CPW problem and the new James-Stein type CPWs; Sec. IV shows the experimental comparisons; and we conclude the letter in Sec. V.

II A Brief Review

II-B Existing Central Pixel Weights

In addition to this CPW, several other CPWs have been proposed and used in the NLM community to enhance performance. These include the zero CPW (Eq. (5)), the Stein CPW (Eq. (6)), the max CPW (Eq. (7)) and the heuristic CPW (Eq. (8)). In the rest of the letter, we use $v_{l}$ to denote these existing CPWs.

These CPWs can be classified into two groups: global CPWs (Eqs. (4),(5) and (6)) and local CPWs (Eqs. (7) and (8)). The global CPWs use a constant CPW for all pixels, while the local CPWs use different CPWs for pixels. In the next section, we will show that these CPWs fail to take all variables into full consideration and therefore oversimplify the CPW problem.

III Shrinkage based Center Pixel Weights

To fully reveal the CPW problem, we separate the contributions of the non-center and of the center pixels in the NLM denoised pixel $\widehat{x_{l}}$ (Eq. (2))

where $W_{l}$ is the summation of all non-center weights

and $\widehat{z_{l}}$ is the denoised pixel by using all non-center weights.

As one can see, a CPW $v_{l}$ is contained in the coefficients of both $\widehat{z_{l}}$ and $y_{l}$ and thus influences the final denoised pixel $\widehat{x_{l}}$ directly. If we are given an optimal $\widehat{x_{l}}$ and solve for $v_{l}$, then it is clear that the optimal $v_{l}$ should be a function of $W_{l}$, $\widehat{z_{l}}$ and $y_{l}$. In other words, a CPW $v_{l}$ that does not consider all these variables is incomplete. It is noticeable that the global CPWs $v_{l}^{one}$ $v_{l}^{zero}$ and $v_{l}^{stein}$ neglects all three, while the local CPWs $v_{l}^{max}$ and $v_{l}^{heur.}$ neglect $y_{l}$.

Let $p_{l}$ be the fraction ($p_{l}\in$) of the contribution of the center pixel $y_{l}$ in $\widehat{x_{l}}$, namely

$p_{l}$ is then a normalized version of $v_{l}$. Consequently, the NLM-CPW problem in (9) can be rewritten as

and is a so-called shrinkage estimator, which improves an existing estimator by using the raw data. In the context of the NLM, the existing estimator is $\widehat{z_{l}}$ and the raw data is the noisy pixel $y_{l}$. The effect of the CPW is to tune the final denoised pixel $\widehat{x_{l}}$ in somewhere between $\widehat{z_{l}}$ and $y_{l}$, or equivalently to shrink $y_{l}$ towards to $\widehat{z_{l}}$.

III-B The James-Stein Center Pixel Weight

One important result in shrinkage estimators is the James-Stein estimator . It states that for an unknown parameter vector $\mathbf{a}$ and its observations of $\mathbf{b}$ with the relation,

there exists a James-Stein estimator that shrinks towards an arbitrary vector $\mathbf{c}$ in the form that

This James-Stein estimator is a classic solution to minimize the risk of estimation in terms of the mean square error $E[\|\mathbf{a}-\mathbf{\widehat{a}}\|^{2}]$ , where $\|.\|$ denotes the $L^{2}$-norm.

In the context of NLM-CPW problem, the James-Stein based CPW (JSCPW) has the weight of form (17),

III-C Local Adapted James-Stein Center Pixel Weights

Although the proposed JSCPW considers all $W_{l}$, $\widehat{x_{l}}$ and $y_{l}$, it is still a global CPW, which gives the weight to each pixel unbiasedly. However, we know the denoising process is always biased rather than unbiased for each pixel. For example, pixels in a homogeneous region are commonly better denoised than edge pixels. Therefore, ideally we want a locally adapted CPW for each pixel as $p_{l}$ in the form of (20). One natural idea is to replace ${\|\mathbf{y}-\mathbf{\widehat{z}}\|^{2}}$ in (20) with ${\|y_{l}-{\widehat{z_{l}}}\|^{2}}$, but it does not lead to a stable solution, because ${\|y_{l}-{\widehat{z_{l}}}\|^{2}}$ is commonly noisy. Alternatively, we view each image block as a small image and thus the JSCPW (17) computed for a local block gives a local CPW adapted to each pixel.

In this way, we construct a local CPW for each pixel, and thus the denoised pixel by using LJSCPW can be written as

Intuitively, this LJSCPW helps eliminate the influence of remote image pixels and tunes the optimization locally.

III-D Implementation

The computational cost of the LJSCPW can be only 5 operations/pixel more than the existing CPWs. Specifically, we construct the integral image $\mathbf{II}$ with 2 operations/pixel for the pixel-wise mean square error between $\mathbf{\widehat{z}}$ and $\mathbf{y}$. Each pixel is the summation of form (21).

In a real case, one may see $p_{l}^{*LJS}\!<\!0$ which conflicts with our assumption that $p_{l}^{*LJS}\in$. This case happens when $\mathbf{\widehat{zr}}_{l}$ is a slightly denoised version of $\mathbf{yr}_{l}$. So it is reasonable to use $\widehat{z_{l}}$ rather than $y_{l}$, implying $p_{l}^{*LJS}=0$. Therefore, we use the positive part of $p_{l}^{*LJS}$ in Eq. (22), where $(.)^{+}=\max(.,0)$.

Thus $p_{l}^{*LJS+}$ ranges from $$. Because$p_{l}^{*LJS+}=1$would indicate the raw data is used, i.e. the pixel is not denoised, it may prove useful in some applications to limit$p_{l}^{*LJS+}$ to a user-defined value less than unity. In this letter, however, we allow the shrinkage operator to operate over the full range.

IV Simulation Results

These results are summarized in Table I. As one can see, compared to the PSNR performance of the zero CPW, the proposed $p_{l}^{*LJS}$ (LJSCPW) and $p^{*JS}$ (JSCPW) are the only two in all CPWs that always improve the denoising performance in terms of a higher mean PSNR with a smaller variance, regardless of patch sizes, test images and noise levels. This implies that James-Stein type CPWs are more efficient than other CPWs.

V Conclusion

In this letter, we reviewed the CPW problem in NLM and proposed two new solutions JSCPW and LJSCPW based on the James-Stein estimator. We showed that the NLM-CPW problem can be viewed as the well studied statistical shrinkage estimator problem. This novel formulation opens a new door to the CPW problem and allows us to use the James-Stein shrinkage estimator for the NLM-CPW problem directly, that is the global JSCPW. To further enhance denoising performance, we propose a locally adapted James-Stein type CPW for each pixel. In this way, the denoising performance is tuned with respect to each local pixel rather than an entire image. Our experimental results show that the proposed James-Stein type CPWs help the NLM to achieve better overall performance in terms of a higher average PSNR with a more robust performance in terms of a smaller PSNR variance. By using these new CPWs, the NLM algorithm is then less sensitive to the temperature parameter $h$ and is better able to retain weak edges.

References