Diffusion Models in Low-Level Vision: A Survey

Introduction

Serving as an essential part of computer vision, low-level vision tasks have been extensively explored for enhancing the low-quality data degraded by complex scenarios and thereby encompass wide and practical applications, including but not limited to image super-resolution , deblurring , dehazing , inpainting , fusion , compression sensing, low-light enhancement , and remote sensing cloud removal . See Fig. 1 for visual results.

Traditional approaches formulated the problem as a variational optimization challenge and employed handcrafted algorithms to solve the proximity constraints related to certain image properties or degradation priors . Those methods, however, fail to cope with the complex degradation for lacking generalizability. With the advent of deep learning, convolutional neural networks (CNNs) and transformers are widely used in low-level vision tasks for their powerful feature extraction capacities. Besides, the collection of abundant datasets, e.g., DIV2K in super-resolution and Rain800 in deraining, further promotes their generalizability. Although achieving promising results, especially in distortion-based metrics like PSNR and SSIM, those techniques suffer from unsatisfactory texture generation, limiting the application in real-world scenarios.

In response to this limitation, deep generative models, especially generative adversarial networks (GANs) , have been introduced to the field of low-level vision. Benefiting from their powerful generative capacities, these networks are expected to synthesize realistic texture details and thus can be extended to real-world scenarios. However, those strategies still encounter several critical challenges: (1) The training process is susceptible to mode corruption and unstable optimization, necessitating intricate hyperparameter tuning during training. (2) The generated results can still exhibit artifacts and counterfactual details, undermining global coherence and limiting their applicability range.

A novel deep generative model, termed diffusion models (DMs) , has recently emerged as a hot topic in computer vision for its remarkable generative capacities and training stability. DMs, characterized by a forward diffusion stage and a reverse diffusion stage, systematically perturb data by introducing noise and subsequently learn to reverse this process for sample generation. Operating within the category of likelihood-based models, DMs formulate their training objective as a re-weighted variational lower bound, garnering acclaim for their extensive distribution coverage, stationary training objective, and straightforward scalability.

Leveraging the above advantages, DMs have exhibited significant success across multiple domains, encompassing data generation, image content comprehension, and low-level vision. Within the realm of low-level vision, DMs mainly concentrate on the restoration of low-quality data, ensuring the reconstruction of high-quality data with precise semantic information and realistic texture details, even in real-world scenarios characterized by complex and severe degradation. As depicted in Fig. 1, numerous DM-based algorithms have demonstrated promising outcomes across various low-level vision tasks. However, the techniques employed in different tasks exhibit considerable diversity and complexity, rendering them challenging to comprehend and enhance, thereby posing obstacles to future development and the introduction of a general-purpose reconstruction model. In light of this, there is a pressing need for a well-structured and comprehensive DM-based survey within low-level vision tasks. Nonetheless, most existing DM-based surveys focus on foundational theoretical models and the development of those generation-based techniques. Only a few surveys concentrate on one specific problem or a few limited tasks within natural image scenarios in low-level vision tasks.

To address the needs of the field and mitigate the above drawbacks, we propose the first DM-based survey tailored for low-level vision tasks (see Figs. 2 and 3). This survey comprises a detailed theoretical introduction, wide application scopes, thorough experimental analysis, and extensive future perspectives. In specific, we commence with a comprehensive introduction to the fundamentals of diffusion models in Sec. 2, elucidating the connections and interrelations between DMs and other deep generative models. Then, we summarize existing cutting-edge DM-based natural low-level vision methods in Sec. 3, categorizing them based on the underlying framework and the target task, including six widely-used tasks. We then expand our scope in Sec. 4 to encompass broader scenarios, including medical, remote sensing, and video scenarios, aiming to provide a comprehensive overview with wide application ranges. Furthermore, Sec. 5 compiles collections of over 30 commonly used benchmarks and more than 10 fundamental evaluation metrics in low-level vision tasks. Abundant experiments on DM-based methods across three prominent tasks, including super-resolution, image deblurring, and low-light image enhancement, are provided in their corresponding settings. Finally, we identify the limitations of existing DM-based methods in low-level vision tasks and propose three main potential directions for future research and improvements in Sec. 6, and summarize the survey in Sec. 7.

We aspire that this DM-based survey, aimed at advancing the understanding of the low-level vision field, will inspire further interest in the computer vision community and promote related research endeavors.

Note. In formulating our search strategy, we executed a comprehensive exploration across diverse databases, including DBLP, Google Scholar, and ArXiv Sanity Preserver. Our emphasis was particularly placed on reputable sources, such as TPAMI, IJCV, and TIP, as well as esteemed conferences like CVPR, ICCV, and ECCV. Preference was accorded to studies that furnished official codes, thereby augmenting reproducibility, as well as those garnering higher citations and Github stars, indicative of broader acknowledgment and adoption within the academic community. Subsequent to this initial screening, our literature selection process entailed a meticulous evaluation of each paper’s novelty, contribution, and significance, along with assessing whether it constituted seminal work in the field. While acknowledging the potential omission of noteworthy papers, our endeavor was to present a comprehensive overview of the most influential and impactful research, thereby fostering research advancement and suggesting potential future avenues.

A Walk-through of diffusion models

Diffusion models constitute a category of likelihood-based models. They are characterized by a shared principle of progressively perturbing data through a random noise process known as “diffusion” and then removing the noise to produce samples (see Fig. 4). These models are typically classified into three subcategories: denoising diffusion probabilistic models (DDPMs), noise-conditional score networks (NCSNs), and stochastic differential equations (SDEs).

DDPMs and their variants have garnered significant attention owing to their straightforward algorithmic flow and the ease of integrating conditional controls. In contrast, NCSNs and SDEs are often subject to detailed mathematical analysis, given their potential for more efficient sampling and enhancements in task generalization. In the following subsections, we will delve into these three formulations, elucidating the noise introduction and reversal process, and outlining the generation of new samples during inference.

given the hyperparameters ${\alpha}_{t}:=1-{{\beta}_{t}}$, ~{}{{\bar{\alpha}}_{t}}:=\mathchoice{\mathop{\vbox{\hbox{\scalebox{0.8}{\displaystyle\prod}}}}}{\mathop{\vbox{\hbox{\scalebox{0.8}{\textstyle\prod}}}}}{\mathop{\vbox{\hbox{\scalebox{0.8}{\scriptstyle\prod}}}}}{\mathop{\vbox{\hbox{\scalebox{0.8}{\scriptscriptstyle\prod}}}}}_{s=1}^{t}{{{\alpha}_{s}}}\text{ }. The above equations can be reformulated as

By reparameterizing Eq. 3, ${x}_{t}$ can be calculated as

While the latter process ${{p}_{\theta}}({{x}_{0}})=\int{{{p}_{\theta}}({{x}_{0:T}})d{{x}_{1:T}}}$ reverses the former from $p({{x}_{T}})=\mathcal{N}({{x}_{T}};0,\bf I)$.

where learnable Gaussian transitions kernels with $\theta$ are parameterized by deep neural networks under the training objects of minimizing the Kullback-Leibler (KL) divergence between $q({x_{0}},{{x}_{1}},\cdot\cdot\cdot,{{x}_{T}})$ and ${{p}_{\theta}}({{x}_{0}},{{x}_{1}},\cdot\cdot\cdot,{{x}_{T}})$.

The optimization principle is as follows: To generate $x_{0}$ in the reverse process, we sample from the noise vector $x_{T}\sim p(x_{T})$ to obtain $x_{T-1},x_{T-2},\ldots,x_{1},x_{0}$ using the learnable transition kernel. The key to this sampling process is training the reverse Markov chain to match the actual time reversal of the forward Markov chain. This requires adjusting $\theta$ to align the joint distribution of the reverse Markov chain $p_{\theta}(x_{0},x_{1},\ldots,x_{T})$ closely with that of the forward process $q(x_{0},x_{1},\ldots,x_{T})$. We use the KL divergence to characterize the gap between these two distributions. $\theta$ can be trained by minimizing the KL divergence:

For better sample quality, a simplified form of loss function is proposed as the optimization target of the model:

where $\lambda(t)$ is a positive weighting function computed from Eq. 4. $\mathcal{U}\left[\!\left[1,T\right]\!\right]$ represents a uniform distribution over the set $\{1,2,\ldots,T\}$. ${{\epsilon}_{\theta}}$ is a deep network with parameters $\theta$ that predicts the noise vector $\epsilon$ given ${{x}_{t}}$ and $t$.

2 Noise Conditioned Score Networks

NCSNs are designed to estimate the probabilistic distribution of the target data from the score function, which guides the sampling process progressively toward the center of the data distribution. The score function for a specific data density $p(x)$ is defined as the gradient of the log-density function, ${{\nabla}{x}}\log p\left(x\right)$, which defines a vector field over the entire space that data $x$ inhabits, pointing towards the directions along which the probability density function has the largest growth rate. The directions provided by these gradients are utilized by the Langevin dynamics algorithm to iteratively shift from a random prior sample ${{x}_{0}}$ to samples ${{x}_{T}}$ in regions with high density. By learning the score function of a real data distribution, it is possible to generate samples from any point in the same space by iteratively following the score function until a peak is reached. Specifically, the process can be represented as:

where $t\sim\mathcal{U}\left[\!\left[1,T\right]\!\right]$. $\gamma$ controls the updating magnitude in the direction of the score, akin to the learning rate in stochastic gradient descent. The noise ${{\epsilon}_{t}}\sim\mathcal{N}\left(0,\mathbf{I}\right)$ represents random normal Gaussian noise at time step $t$, introducing random perturbations into the recursive process to address the issue of getting stuck in local minima. As the time step $T\to\infty$ and $\gamma\to 0$, the distribution $p\left({{x}_{T}}\right)$ approaches the original data distribution $p(x)$. Hence, a generative model can utilize the above method to sample from $p(x)$ after estimating the score with a network ${{s}_{\theta}}\left(x,\text{ }t\right)\approx{{\nabla}_{x}}\log\text{ }{p}\left(x\right)$. This network can be trained via score matching to optimize the objective function presented as follows:

where $t\sim\mathcal{U}\left[\!\left[1,T\right]\!\right],{{x}_{0}}\sim p({{x}_{0}}),{x}_{t}\sim{{p}}({x}_{t}|{x}_{0})$. In practice, because $\nabla_{x_{t}}\log p(x_{t}|x_{0})$ is unknown, Eq. 9 can only be solved by those score matching-based methods rather than be directly solved, limiting the generalization to real data. According to the manifold hypothesis, conventional score function estimation methods, including denoising score matching and sliced score matching , when combined with Langevin dynamics, can lead the resulting distribution to collapse to a low-dimensional manifold and thus bring inaccurate score estimation in the low-density region. To address this issue, annealed Langevin dynamics perturbs the data with Gaussian noise at different scales and further proposes an optimization objective under a monotonically decreasing noise strategy $({{\sigma}_{t}})_{t=1}^{T}$:

where ${{{{x}_{t}}\sim{{p}_{{{\sigma}_{t}}}}({{x}_{t}}\left|x)\right.}}$. In inference, one can initiate with white noise and apply Eq. 8 for a predetermined $T$. Once ${{\theta}^{*}}$ is acquired through optimizing the objective conditioned on $T$, as shown in Eq. 10, one can use the approximation ${{\nabla}_{x_{t}}}\log{p}\left({{x}_{t}}\right)\approx{{s}_{{{\theta}^{*}}}}\left({{x}_{t}},\text{ }t\right)$ as a plug-in estimate to replace the score function used in the stochastic differential equations . As iterative processes continue, the final sample is derived from the output obtained at $t=0$.

3 Stochastic Differential Equations

As a continuous extension of NCSNs, SDE and reverse-time SDE can corresponding model the forward diffusion process and reverse diffusion process, respectively. The forward diffusion process modeled by SDE is formulated as

where $d\bar{\omega}$ denotes the infinitesimal negative time step, defining the standard Wiener process running backward in time. Solutions to the reverse-time SDE are diffusion processes that gradually convert noise to data. Note that the reverse SDE defines the generative process through the score function ${{\nabla}_{x}}\log p(x)$, a shared concept in Sec. 2.2.

During both train and inference phases, SDE-based methods rely on practical numerical sampling techniques. Alongside numerical solutions discussed in Sec. 2.2, methodologies like Euler-Maruyama discretization and Ordinary Differential Equations (ODEs) are effective, with the latter offering better sample efficiency advantages.

If the score function ${{\nabla}_{x}}\log p(x)$ is known, we can solve the reverse-time SDE easily. By generalizing the score-matching optimization objective in NCSNs to continuous time, we parameterize a time-dependent score model $s_{\theta}(x_{t},t)$ to estimate the score function in reverse-time SDE, bringing the same optimization objective as Eq. 9.

Comparing the expansion result of the score function that uses Bayes’ rule with the noise result obtained from Eq. 4, it is easy to observe that the training objectives for DDPMs and NCSNs are equivalent, as shown in Eq. 13. Namely, the optimization learning objectives of both methods only differ by a fixed scaling factor:

Moreover, when generalizing to the case of infinite time steps or noise levels, both DDPMs and NCSNs can be considered as discrete numerical solutions of SDEs in practical applications. For example, the Variance Preserving (VP) form of the SDE can be perceived as the continuous version of DDPM , and the corresponding SDE is

where $\beta(\frac{t}{T})=T{{\beta}_{t}}$ as $T$ goes to infinity. NCSNs with annealed Langevin dynamics are equivalent to the discrete version of Variance Exploding (VE) SDE , which is

where $\sigma(\frac{t}{T})={{\sigma}_{t}}$ as $T$ goes to infinity.

4 Comparisons With Other Deep Generative Models

In this subsection, we explore the connections between diffusion models and other commonly-used generative models, whose flowcharts are illustrated in Fig. 5.

Both DMs and variational autoencoders (VAEs) involve mapping data to a latent space, where the generative process learns to transform the latent representations back into data. Additionally, in both cases, the objective function can be derived as a lower bound of the data likelihood. However, while the latent representation in VAEs contains compressed information about the original image, classical assumptions suggest that DMs completely destroy the data after the final step of the forward process. Furthermore, the latent representations in diffusion models have the same dimensions as the original data, whereas VAEs tend to perform better with reduced dimensions. Drawing inspiration from these similarities, some existing work has explored the use of diffusion models on the latent space of a VAE to build more efficient models , or to construct hybrid models that fully leverage the advantages of both models.

Normalizing flows (NFs) transform a simple Gaussian distribution into a complex data distribution through a series of invertible functions with easily computable Jacobian determinants. The only similarity between DMs and NFs lies in their mapping of the data distribution to Gaussian noise. However, the learnable forward process of NFs, unlike that of DMs, imposes additional constraints on the architecture due to its requirement for invertible and differentiable properties. DiffFlow , serving as a bridge between these two generative algorithms, extends both diffusion models and normalizing flows to enable trainable stochastic forward and reverse processes.

GANs drive the fake data distribution towards the real one through adversarial learning on the generator and the discriminator, ensuring that the sampled data resembles real data. Consequently, GANs are extensively utilized for generating photo-realistic high-resolution images (e.g., PGGAN and StyleGAN series ). However, GANs are notorious for their challenging training process due to their adversarial objective and often suffer from mode collapse. In contrast, DMs exhibit a stable training process and offer greater diversity as they are likelihood-based. Despite these advantages, DMs are less efficient than GANs as they require multiple iterative steps during inference.

The distinctions between GANs and DMs also manifest in their ability to manipulate semantic properties within the latent space. GANs’ latent space has been observed to contain subspaces associated with visual attributes, enabling attribute manipulation through changes in the latent space and thus facilitating more precise control over generated images. However, DMs manipulate semantic properties of the latent space in a more implicit and less controllable manner. Fortunately, Song et al. demonstrate that DMs’ latent space exhibits a well-defined structure. Nonetheless, the exploration of DMs’ latent space has been less extensive compared to GANs, indicating the need for further research.

Diffusion models for natural image processing in low-level vision

To begin with, we first give the definition of ”natural images”, which depict common scenes and objects encountered in daily life, serving as the foundational input data in model training and evaluation, particularly for image restoration algorithms. In this section, ”images” encompass the ordinary and general notion of natural images.

Low-level vision tasks primarily focus on various ill-posed inverse problems in the image restoration domain. These tasks aim to restore degraded and noisy low-quality (LQ) images to high-quality (HQ) images. The general form of the forward model can be stated as

Through rapid development, DM-based models have achieved significant progress in this domain. Unlike random sample generation methods such as vanilla DDPM in Sec. 2, here the degraded LQ images are used as conditional inputs to guide the latent variables during inference. The models are expected to learn a parametric approximation to the unknown conditional distribution, posterior $p\left(x|y\right)$, through a stochastic iterative refinement process.

After conducting a comprehensive review of approximately 100 relevant DM-based works, we classify them from two perspectives. First, according to whether the DMs require training, we classify those methods into supervised and zero-shot DM-based methods. Additionally, we categorize them depending on the task being solved, e.g., super-resolution and low-light image enhancement.

DM-based models typically employ two training methods: supervised and zero-shot. Research has flourished in both paradigms, demonstrating the versatility of DMs as frameworks for various low-level vision tasks.

Supervised DM-based methods. Supervised DM-based methods tend to specialize in addressing specific degradation scenarios. They employ the well-designed conditional mechanism to incorporate distorted images as guidance during the reverse process, enabling them to tackle several extreme challenges, such as dehazing, deraining, and desnowing, that cannot be effectively modeled using the form of Eq. 16. However, despite yielding promising texture generation performance, these approaches require training the diffusion model from scratch using paired clean and distorted images from a particular degradation scenario. This results in costly data acquisition and limits the algorithm’s generalization to other degradation scenarios.

Zero-shot DM-based methods. Zero-shot DM-based techniques, leveraging the image priors extracted from pre-trained diffusion models, offer an appealing alternative as they are plug-and-play without retraining on a specific dataset. The underlying concept is based on the understanding that pre-trained generative models, constructed using extensive real-world datasets such as ImageNet , can serve as a repository of structure and texture. A key challenge of zero-shot DM-based methods lies in extracting the corresponding perceptual priors while preserving the underlying data structure from distorted images. Consequently, these zero-shot DM methods are often applied to degradation scenarios simplified as linear reverse problems, such as super-resolution and inpainting. Given the simplicity of the application process, which only requires replacing the forward measurement operator, evaluating performance on linear inverse problems has become a common practice to assess the generalization of newly proposed conditional diffusion models. However, these works are frequently categorized under multi-task alongside other high-level tasks in existing surveys, without receiving systematic analysis and summarization. Therefore, we devote a specific subsection to introducing these DM-based inverse problem solvers for general-purpose image restoration in Sec. 3.2.

2 DM-based methods with different application goals

To provide a comprehensive overview, we categorize existing DM-based methods into the general-purpose image restoration task, encompassing most zero-shot methods and several supervised methods and six distinct specific tasks, including super-resolution (SR), inpainting, deblurring, dehazing, low-light image enhancement, and image fusion.

General-purpose image restoration. Notably, most methods in this subsection presuppose prior knowledge of the forward operator $H(\cdot)$ in Eq. 16, thereby confining their scope to non-blind inverse problems. To adhere to specific assumptions, further constraints are occasionally imposed to convert them into linear inverse problems, as shown in Fig. 6. Nonetheless, the mapping $y\to x_{0}$ remains many-to-one, rendering it impossible to precisely recover $x_{0}$.

Focusing on sampling from the posterior $p(x|y)$, the relationship can be formally established with the Bayes’ rule: $p(x|y)=p(y|x)p(x)/p(y)$. However, apart from $p(y|{{x}_{0}})\sim\mathcal{N}(y|A({{x}_{0}}),\text{ }{{\sigma}^{2}}\bf I)$, there does not exist explicit dependency between $y$ and ${{x}_{t}}$, where ${{x}_{t}}$ denotes the noisy results at time step $t$. To solve the intractability of the posterior distribution, Song et al. propose conditional denoising estimator (CDiffE) ${{s}_{\theta}}\left(x,\text{ }y,\text{ }t\right)$. The condition $y$ is added to the input of the estimator to learn an approximation to the posterior score function ${{\nabla}_{{{x}_{t}}}}\text{log }p\left({{x}_{t}}|y\right)$ without altering the training object. Instead of direct learning, the conditional diffusive estimator jointly diffuses $x$ and $y$ and then learns the posterior approximated from the joint distribution $p\left({{x}_{t}},\text{ }{{y}_{t}}\right)$ using denoising score matching. Batzolis et al. rigorously prove the effect of the above two methods theoretically and analyze the errors caused by the imperfections.

To enhance consistency, and guide the gradient towards high-density regions by conditioning it through projections on the measurement subspace without directly learning the posterior score function. Chung et al. introduce the manifold constraint after the update step, which corrects deviations from the perfect data consistency. Utilizing pre-trained unconditional DDPM, Choi et al. propose Iterative Latent Variable Refinement (ILVR). As shown in Fig. 7, ILVR is a learning-free method adopting low-frequency information from $y$ to guide the generation process towards a narrow data manifold. However, such approaches are limited to those noiseless inverse problems.

Besides the above learning-free methods, plug-and-play posterior sampling provides another favorable choice. According to the Bayesian framework, the posteriors are decomposed into likelihood functions and plug-and-play priors. The prior term is typically satisfied by generative diffusion models capable of learning robust assumptions about the underlying structure of images (e.g., the distribution of natural images), while the likelihood term, also known as the data term, is derived from the degradation model.

In plug-and-play sampling methods, Graikos et al. first showcase the viability of directly using pre-trained DDPMs as plug-and-play modules that involve other differentiable constraints. Kawar et al. propose the Denoising Diffusion Restoration Models (DDRM) to reconstruct the missing information in $y$ within the spectral space of $H(\cdot)$ using Singular Value Decomposition (SVD). Leveraging the power of pre-trained DMs, DDRM demonstrates versatility across several tasks, including SR, deblurring, inpainting, and colorization, even in the presence of varying levels of measurement noise. However, only linear inverse problems are viable in this framework due to the limitations of SVD.

Zhu et al. decouple the data term and the prior term with Half-Quadratic-Splitting and propose DiffPIR, handling a wide range of degradation models with different degradation operators $H(\cdot)$, as the data term can be independently solved. Chung et al. introduce a Laplacian approximation-based method named Diffusion Posterior Sampling for posterior sampling, applicable to nonlinear inverse problems with complex noise. Similarly utilizing DM as a prior term, Wang et al. propose to solve zero-shot image restoration using Denoising Diffusion Null-space Model (DDNM). The pseudo-inverse based on SVD computes the low-dimensional image representation, then decomposed into its range and null-space components. By refining only the null-space during the reverse process, DDNM can learn missing information in image inverse problems while accommodating only linear operators.

Methods based on Schrödinger bridges, i.e., InDI and I2SB, revisit DMs’ assumptions and depart from commencing the reverse diffusion process from Gaussian noise, ensuring efficiency. Chung et al. demonstrate the equivalence of the above methods and propose the Consistent Direct Diffusion Bridge (CDDB), incorporating a previously overlooked data consistency module, to realize the generalization of Schrödinger bridges on low-level vision tasks. Similarly, Chung et al. use a feed-forward network (FFN) to initialize the diffusion process and reduce sampling steps to generate images from pure noise, shedding light on combining existing FFNs for inverse problems with DMs.

To mitigate the computational overhead, researchers have shifted DMs from the image level to the vector level. Rombach et al. propose latent diffusion models (LDMs), where both the forward and reverse processes occur in the latent space obtained through an auto-encoder. DiffIR further compress latent variables to an extremely low-dimensional representation to guide the transformer in image reconstruction. To balance latent disentanglement and high-quality reconstructions, Pandey et al. integrate VAEs within the diffusion model and propose DiffuseVAE, offering novel conditional parameterizations for DMs and providing a promising alternative for hybrid modeling.

Due to prevalent limitations stemming from various presuppositions, these models are frequently applied to relatively simple degradation scenarios that can be abstracted and simplified as linear inverse problems. Consequently, they are less effective in real-world blind tasks compared to task-specific methods.

Super-resolution (SR). Diffusion models have shown prowess in generating high-quality outputs with intricate details, addressing over-smoothing and artifacts commonly encountered for high-resolution SR. SRDiff is the pioneering single-image SR model based on diffusion models, utilizing a pre-trained low-resolution encoder and a conditional noise predictor to produce diverse and realistic SR predictions. This approach effectively addresses over-smoothing and large footprint issues in previous methods. Saharia et al. also leverage a conditional diffusion network, employing low-resolution images as conditional inputs to resolve SR tasks, particularly for human faces.

Cascaded Diffusion Models (CDM) propose to arrange multiple DMs sequentially. In specific, the initial model generates low-resolution images based on image classes while subsequent models progressively generate images with higher resolutions, facilitating SR at arbitrary magnifications but increasing training costs. Gao et al. propose implicit DMs for continuous SR (in Fig. 8). They introduce a scale-adaptive conditioning mechanism to regulate resolution and adjust the ratio of realistic data and use implicit neural representation to capture complex structures across continuous resolutions. Niu et al. first utilize a pre-trained SR model to generate high-resolution conditional inputs. Besides, they propose a $n^{th}$ order sampler to perform a deterministic denoising process, greatly reducing the iteration number. Wang et al. propose StableSR to leverage prior knowledge encapsulated in pre-trained text-to-image DMs for blind SR. By utilizing a time-aware encoder, StableSR achieves promising restoration results without modifying the pre-trained synthesis model, preserving generative priors and minimizing training costs. Similarly leveraging generative diffusion priors, Lin et al. design DiffBIR for blind image SR, decoupling the restoration process into two stages. Sunet al. propose CoSeR, which leverages generative reference images from a pretrained LDM as implicit priors. It combines these generated results with low-resolution image priors and CLIP’s semantic priors as conditions to control the diffusion process, achieving cognitive SR. AlHalawani et al. introduce DMs into the surveillance domain, verifying the superiority in license plate SR. Meanwhile, Wang et al. propose the two-stage DR2 focusing on facial data, achieving desirable results.

Inpainting. As a probabilistic generative model, diffusion models exhibit robust generalization across different masks and effectively handle large missing regions while producing visually plausible results. RePaint employs an enhanced denoising strategy involving resampling iterations to better condition images in Fig. 9. The model first uses a deep network to generate a rough estimate, which is then refined by a diffusion model with a Markov random field. To specify a desired inpainted object, Gebre et al. input an extra target image to guide the generation of the masked region, providing valuable exploration in the controllable generation. Zhang et al. employ both image and text as multimodal guidance. By integrating the inverse diffusion process with CLIP, semantic reference information is better encoded, thus enhancing performance and controllability.

Spatial diffusion model employs a Markov random field and a network to estimate the missing pixels, which considers surrounding contexts and thus inpaints large missing regions. Saharial et al. introduce Palette to explore diverse optimization objectives and underscore the significance of self-attention. BrushNet is a plug-and-play model engineered to embed pixel-level masked features into any pre-trained DMs by separating masked image features and noisy latent into separate branches. Grechka et al. propose a training-free DM-based method GradPaint for gradient-guided inpainting. GradPaint aims to improve the coherence and realism of generated images by using custom loss and gradient guidance, demonstrating promising results when generalized to various datasets.

Deblurring. Because realistic blur is often complex and non-uniform, modeling specific degradation prior can result in poor performance. Consequently, DMs in these domains often rely on hand-designed networks. Wang et al. is the first to introduce diffusion models (as illustrated in Fig. 10) into the deblurring task, proposing a “predict-and-refine” conditional diffusion model. This model architecture comprises a deterministic data-adaptive predictor and a stochastic sampler, which refines the output through residual modeling. Ren et al. introduce multiscale structure guidance in image-conditioned DPMs (icDPMs) for deblurring. Their guidance module projects the input image into a multiscale representation, where auxiliary priors provide information about edges. This guidance is integrated into the network’s intermediate layers as an implicit bias, enhancing the robustness by informing the coarse structure of the sharp image. Hierarchical Integration Diffusion Model (HI-Diff) in leverages the diffusion model in a compact latent space to generate priors and fuse these priors into the regression-based model through a cross-attention mechanism, enabling generalization in complex blurry scenarios.

Laroche et al. propose a DM-based blind image deblurring method. This method integrates DMs with the Expectation-Minimization (EM) estimation to jointly estimate restored images and the unknown blur kernel. Spetlik et al. propose a DDPM-based method, SI-DDPM-FMO, for single-image object deblurring and trajectory recovery of fast-moving objects, getting competitive results to multi-frame methods. DiffEvent firstly introduces DMs into the event deblurring task and designs diffusion priors for the image and the residual. To better adapt to real-world scenes, DiffEvent builds an Event-Blur Residual Degradation (EBRD) to provide pseudo-inverse guidance, enhancing subtle details and handling unknown degradation. Luo et al. deviate from classical SDE and propose a novel approach called Image Restoration Stochastic Differential Equation (IR-SDE). The core is a mean-reverting SDE with a maximum likelihood objective for training. This ensures that the entire SDE will diffuse towards the mean $\mu$, representing the low-quality images with specific Gaussian noise, thus directly simulating the degradation process .

Dehazing, deraining, and desnowing. Unlike linear inverse problems such as super-resolution and inpainting, real-world degradations like dehazing and deraining are complex and cannot be effectively modeled by a prior operator $H(\cdot)$. Consequently, they pose challenges for incorporation into general-purpose image restoration frameworks.

Özdenizci et al. present a patch-based image restoration algorithm based on DDPMs called WeatherDiffusion (illustrated in Fig. 11). This approach facilitates size-agnostic image restoration by employing a guided denoising process with smoothed noise estimates across overlapping patches during inference, effectively mitigating the drawbacks of merging artifacts from independently restored intermediate results. WeatherDiffusion achieves superior performance on both weather-specific and multi-weather image restoration tasks, including dehazing, desnowing, deraining, and raindrop removal. Building upon the IR-SDE as the base diffusion framework, Luo et al. further enhance it to perform restoration in a low-resolution latent space, which constitutes a resolution-agnostic architecture. This enhancement offers another viable option for handling large-size images. Additionally, they propose a latent-replacing pretraining strategy to improve training stability. Wang et al. propose a Frequency Compensation block (FCB), equipped with a bank of filters that collectively amplify the mid-to-high frequencies of an input signal, aiming to enhance the reconstruction of image details in dehazing tasks and improve generalization to real haze scenarios.

Low-light image enhancement. Due to the intrinsic relationship between low-light image enhancement (LLIE) and the human visual perception of light intensity, various interpretive prior models have emerged, inspiring multiple approaches for integrating the advantages of physical models and generative networks. Consequently, compared to the relatively monolithic black-box design in other tasks, a plethora of research related to DMs has emerged in LLIE.

Zhu et al. are pioneers in introducing DMs into LLIE within space-based visible cameras. This method effectively reduces computational complexity by diffusing processes on grayscale images and supplementing features with RGB images. Wu et al. focus on restoring pure black images, providing a robust generative network for enhancing low-light images with diverse outputs. DiD , designed for downstream low-light text recognition, utilizes a bootstrap diffusion model to learn the distribution of light-enhancement curve parameters, preserving high-frequency details in dark conditions and alleviating the computational burden. Panagiotou et al. employ DMs as a postprocessing technique, estimating the amount of noise present in an enhanced image in one pass through the model. Zhou et al. propose the Pyramid Diffusion model named PyDiff (illustrated in Fig. 12) for LLIE, which progressively increases the resolution during the reverse process, effectively reducing computational burden. Jiang et al. introduce a wavelet-based conditional diffusion model (WCDM), which leverages wavelet transform to reduce computation. Moreover, a high-frequency restoration branch module is proposed to supplement the diagonal information with vertical and horizontal details. Wang et al. integrate DMs with a physics-based exposure model in the raw image space, where the reverse process can directly start from a noisy image instead of pure noise, boasting a small parameter size and fast inference speed. Lv et al. input low-light images into a latent space for the diffusion process, also effectively reducing computation.

Some methods that integrate DMs with other advanced techniques have yielded superior results. LLDiffusion proposes a degradation-aware learning scheme to integrate degradation representations from pre-trained models and proposes a dynamic diffusion module that considers both color maps and degradation. Hou et al. regularize the ODE-trajectory and introduce a global structure-aware regularization term to constrain the intrinsic structures, along with an uncertainty-guided regularization to relax constraints on extreme situations. Diff-Retinex decomposes the image into illumination and reflectance maps based on the Retinex theory and then uses multi-path DMs to estimate the light distribution and restore degradation. Adopting the opposite strategy, He et al. propose a Retinex-based LDM to extract reflectance and illumination priors, and then perform decomposition and enhancement using a Retinex-guided transformer, ultimately achieving superior results. Yin et al. achieve an interactive and controllable LLIE model based on a conditional DM. Users can customize the brightness level and enhance specific target regions with the assistance of the Segment Anything Model (SAM). To fully utilize the CLIP-based model prior, Xue et al. introduce multimodal visual-language information and propose a novel approach named CLIP-Fourier Guided Wavelet Diffusion (CFWD). CFWD combines the strengths of wavelet transform, Fourier transform, and CLIP to guide the DM-based enhancement process in a multiscale visual-language manner, demonstrating the immense potential of integrating semantic features from CLIP and high-frequency detail recovery from the Fourier transform.

Image fusion. Image fusion is an often overlooked low-level vision task among researchers. However, its significance lies in its capacity to amalgamate complementary data from multiple images, elevating the overall visual quality and facilitating diverse downstream applications. Yue et al. propose the first DM-based method for image fusion named Dif-Fusion (see in Fig. 13) to enhance color fidelity in infrared and visible image fusion. By creating a multi-channel data distribution using a denoising network, Dif-Fusion facilitates multi-source information aggregation and maintains high-quality color representation. The forward and reverse diffusion processes in the latent space enable the extraction of diffusion features for effective fusion, leading to superior results compared to existing methods.

Zhao et al. propose DDFM for Multi-Modality Image Fusion. The conditional generation problem is divided into an unconditional DDPM for utilizing image generation priors and a maximum likelihood sub-problem for preserving cross-modal information of source images. The authors innovatively model the latter using a hierarchical Bayesian approach and integrate its EM algorithm-based solution into the unconditional DDPM, ensuring the generation of visually fidelity results. Cao et al. propose the utilization of diffusion models for general image fusion tasks and devise two conditional injection modulation modules to introduce coarse-grained style information and fine-grained frequency information into the diffusion UNet. This method achieves state-of-the-art results with strong generalization performance, showcasing the potential of diffusion models in image fusion tasks. Li et al. apply the DDPM model to the multi-focus image fusion task, showcasing the excellent performance and robustness of DM-based methods in terms of noise resistance and few-shot capabilities.

Extended diffusion models

Beyond natural image processing scenarios, certain specialized domains—such as medical imaging, remote sensing, and video analysis—exhibit unique characteristics that necessitate the development of specifically tailored DM-based methods to address their distinct requirements.

Compared with natural data, medical data acquisition typically involves more intricate and precise physical imaging processes , resulting in poor image quality due to equipment limitations and usage conditions (e.g., hospital throughput requirements, patient examination time constraints, and radiation dosage limits). Leveraging the robust learning capacity of DMs, researchers suggest that these models can implicitly capture knowledge related to imaging physics from dataset distributions. Hence, DM-based reconstruction methods have been introduced to address low-quality medical images degraded by imaging limitations, including limited-angle computed tomography (CT) and accelerated magnetic resonance imaging (MRI).

In addition to enhancing low-quality data, another significant application of DM-based algorithms is the generation of missing modalities. In disease diagnosis, the combination of multimodal data assists doctors in making more accurate diagnoses. However, the types of actual modal data collected are often limited, and aligning the acquired multimodal data can be challenging. DM-based image translation methods offer a potential solution in this regard. Furthermore, certain rarer medical images (e.g., Positron Emission Computed Tomography (PET) and Optical Coherence Tomography (OCT)) unavoidably contain speckle noise that traditional methods struggle to eliminate. Due to the nature of generative models in detail reconstruction, diffusion models are well-suited for addressing such issues.

To provide a multi-perspective categorization, we will classify methods according to their imaging modalities, covering MRI, CT, multi-modal, and other modalities.

Magnetic resonance imaging. Due to its inherent physics, MRI involves a time-consuming imaging process, where patient movement leads to various artifacts in the images. Hence, medical image reconstruction is necessary to achieve faster acquisition speed and solve this ill-posed inversion problem. Jalal et al. initiate the primary study in MRI reconstruction using Compressed Sensing with Generative Models (CSGM). CSGM involves training score-based generative models on MRI images to provide prior information for inverting under-sampled MRI into realistic data through Langevin dynamics in a posterior sampling scheme.

Chung et al. design a score-based framework for accelerated MRI reconstruction, illustrated in Fig. 14. Initially, they train a time-dependent score function using denoising score matching on magnitude images. They employ the VE SDE for sampling from the pre-trained score model distribution conditioned on the measurement. The reconstruction process involves splitting the image into real and imaginary components and applying data consistency mapping. Iterative steps include image division, correction, and further data consistency mapping for enhancement. It effectively handles multi-coil images and exhibits robust generalization, unaffected by subsampling patterns, unlike traditional models requiring retraining for each new sampling scheme.

CT. Similar to MRI, limited-angle CT reconstruction has been a primary focus in CT research, aiming to reduce patient radiation exposure and enhance examination throughput. DM-based methods have shown remarkable performance in this reconstruction task. For example, Liu et al. introduce DOLCE, a method specifically designed for limited-angle CT reconstruction within a DDPM framework. Conventionally, the Filtered Back Projection (FBP) algorithm is employed to map CT images from sinograms, leveraging the Fourier slice theorem. However, limited-angle measurements lead to Fourier measurement loss and subsequently degraded reconstruction outcomes.

Due to the ill-posed nature, directly using DDPM presents challenges. Following the design in inpainting tasks, DOLCE integrates the FBP output on limited sinograms as prior information to condition the diffusion model (Fig. 15). Additionally, DOLCE enforces a consistency term in the denoising iteration to ensure reconstruction consistency through iterative refinement using proximal mapping in the inference step to meet the consistency conditions presented by sinograms. Evaluation on C4KC-KiTS verifies DOLCE’s effectiveness in generating high-quality CT images. Moreover, the reconstruction performance is further evaluated in downstream tasks such as 3D Segmentation.

Multi-modal medical data. MRI and CT are the two most widely used medical imaging modalities. MRI shows soft tissues such as vessels and organs in rich contrast while CT is preferred for imaging hard tissues such as bones and interfaces. Due to their complementary characteristics, multi-modality imaging with MRI and CT is often used in clinical practice. Therefore, the development of a simultaneous CT-MRI device is currently a hot research topic, and various studies have been carried out to propose advanced designs for such a device. To translate MR to CT images, Lyu et al. examine conditional DDPM and SDE models, employing three different sampling methods.

As shown in Fig. 16, Meng et al. introduce a Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to complete a wider range of missing modality images. This model is presented in a conditional format of SDE, leveraging a novel multi-in multi-out conditional score network (mm-CSN) module, to learn cross-modal conditional distributions. Experiments on BraTS19 indicate that the method can generate missing-modality images with high fidelity and clear brain structural information.

Due to the inter-modality differences and lacking medical data, training powerful DM-based models in a zero-shot manner for image translation is not feasible. However, this approach remains viable for translation tasks with lower difficulties, such as the CBCT-to-CT image translation and cross-institutional MRI image translation tasks. For example, Li et al. propose the Frequency-Guided Diffusion Model (FGDM), which uses frequency-domain filters to preserve structure during image translation. FGDM enables zero-shot learning and exclusive training on target domain data, allowing direct deployment for source-to-target domain translation. This verifies significant advantages in zero-shot medical image translation over existing methods.

Other modalities. PET, crucial for cancer screening, faces challenges related to low SNR and resolution due to the limited beam count radiation during scans. To mitigate the oversmoothing in previous PET denoising methods, Gong et al. introduce a DDPM-based framework for PET denoising, termed PET-DDPM. PET-DDPM explores the collaboration of diverse modalities to learn noise distribution through PET images. The MR image, serving as the prior, is seamlessly integrated as the input for the denoising network. Experiments reveal that employing MR prior as the input while embedding PET images as a data-consistency constraint during inference achieves the best performance.

Hu et al. apply a DDPM to address speckle noise in OCT volumetric retina data using an unsupervised technique called DenoOCT-DDPM. OCT imaging encounters challenges due to restricted spatial-frequency bandwidth, resulting in images with speckle noise that hampers ophthalmologist diagnosis and tissue visibility, and conventional methods like averaging multiple B-scans struggle to handle this. DenoOCT-DDPM exploits DDPM’s adaptability to noise patterns instead of real-data patterns. It incorporates self-fusion as a preprocessing step, feeding the DDPM with a clear reference image for training the parameterized Markov chain (refer to Fig. 17). The qualitative results support the efficacy of DMs in eliminating speckle noise while preserving detailed features like small vessels.

2 Diffusion models for remote sensing data

The versatility of diffusion models makes them well-suited for remote sensing data processing. Their applications span a spectrum of challenges encountered in the analysis of diverse remote sensing modalities, including visible-light images, hyperspectral imaging (HSI), and Synthetic Aperture Radar (SAR). These tasks encompass but are not limited to super-resolution , despeckling, cloud removal, multi-modal fusion, and cross-modal image translation.

We continue to categorize these works based on the imaging modality, examining the significant impact and advancements brought about by diffusion models.

Visible-light remote sensing data. Compared with natural images, DM-based methods in remote sensing remain underdeveloped. Visible-light Remote Sensing Images share a high similarity with natural images. In this case, Sebaq et al. employ techniques similar to Imagen for low-resolution result generation and referenced the cascaded SR pipeline of CDM , ultimately constructing a powerful framework for high-resolution satellite imagery generation.

Given that RS images suffer from detail loss, Liu et al. propose the first DM for Remote Sensing Super-Resolution (RSSR) and introduce a detailed supplement inpainting task through random masking, aiming to enhance the recovery ability for specific small objects and complex scenes. Besides, they introduce a joint loss to suppress the undesirable excessive diversity. Considering that RS images often have higher resolution and exhibit unusual sizes, Huang et al. introduce an Adaptive Region-Based DM (in Fig. 18) capable of addressing arbitrary RS image dehazing tasks. They employ the cyclic shift strategy to eliminate inconsistent color and artifacts.

Hyperspectral imaging. HSI is a crucial modality in remote sensing with widespread applications. However, due to the limitations of imaging devices, HSIs suffer from data-hungry, noise corruption, and low spatial resolution. Zhang et al. train the first DM conditioned on RGB natural images for HSI generation. The authors employ a spectral folding technique to achieve spectral-to-spatial mapping, addressing the convergence challenges associated with the large spatial sampling space of HSI images due to their high channel count. Deng et al. propose a DM-based model for HSI denoising, utilizing random masking, resembling the one in , to balance the importance of spatial and spectral information for performance improvement.

As shown in Fig. 19, Miao et al. introduce an innovative self-supervised DM, DDS2M, for HSI restoration, addressing the data-hungry issue. DDS2M leverages the variational spatio-spectral module, comprising two untrained networks, each focusing on the spatial and spectral dimensions, to exploit the intrinsic structural information of the underlying HSIs. By introducing prior information, DDS2M can learn the posterior distribution solely using the degraded HSI without extra training data. Experiments on HSI denoising, noisy HSI completion, and super-resolution verify the superiority of DDS2M. DDS2M provides new insight into integrating existing DMs with untrained networks and offers a promising solution for HSI restoration.

The spatial and spectral resolutions of spectral images are often challenging to balance due to limitations in imaging technology. Wu et al. propose HSR-Diff, the first diffusion model for HSI Super-resolution (HSR). The model fuses high-resolution multispectral images (MSI) with low-resolution hyperspectral images (LR-HSI) to obtain HR-HSI. The conditional DDPM uses the Conditional Denoising Transformer (CDFormer), which replaces the time embedding with noise embedding, designing spatio-spectral transformer layers designed for HSI characteristics. Shi et al. employ a similar approach and demonstrate the effect of DM-based models on multiple remote sensing datasets.

Synthetic Aperture Radar. Tuel et al. pioneer the use of diffusion models for radar remote sensing imagery, achieving tasks related to SAR image generation and denoising. This method highlights, due to limited data, the lack of powerful feature extractors specific to remote sensing data as a major bottleneck for high-quality generation. Speckle, a type of signal-dependent multiplicative noise affecting coherent imaging modalities including SAR images, is addressed by Perera et al., who introduce DDPM to SAR despeckling. Besides, a new inference strategy based on cycle spinning is proposed to further improve performance. Xiao et al. transform multiplicative noise into traditional additive noise through operations in the logarithmic domain for DM-based denoising. This method introduces patch shifting and averaging-based algorithm similar to to adapt to inputs of arbitrary resolutions, further enhancing performance, albeit increasing computational burden.

Muti-modal remote sensing data. SAR images are robust to weather conditions but are hard to interpret, lacking intuitive visual clarity. In the remote sensing domain, SAR often collaborates with other modalities to accomplish cloud removal tasks. Similarly, in DM-based models, compared to simply modeling cloud removal tasks as inpainting tasks, results with SAR as auxiliary input often exhibit higher credibility. Jing et al. introduce an innovative approach for cloud removal in optical satellite images with DDPM Feature-Based Network for Cloud Removal (DDPM-CR). This model incorporates auxiliary SAR data and multilevel features from the DDPM architecture to recover missing information across various scales. The cloud removal head, equipped with an attention mechanism, recovers missing information, and a cloud-oriented loss balances information recovery in cloud-covered and cloud-free regions. Zhao et al. further integrate multi-temporal sequence information into DM-based models in Fig. 20, amalgamating two mainstream cloud removal concepts in a single framework.

Seo et al., employing a self-supervised denoiser in the latent space, train the Brownian-Bridge diffusion model to achieve SAR to Electro-Optical image translation tasks. Experiments conducted around flood datasets verify enhanced visual information and downstream segmentation task performance for the translated images.

3 Diffusion models for video processing

The latest research endeavors aim to extend the exploration of DMs into higher-dimensional data, particularly in video tasks . However, compared with image, video processing requires temporal consistency across video frames. Currently, the number of DM-based video models are relatively few, only applied in several fundamental tasks, including video frame prediction , interpolation , super-resolution , and restoration .

Video frame prediction and interpolation. Renowned for remarkable generative capacities, DM-based models are especially suitable for video prediction and interpolation. Yang et al. first use DMs in autoregressive video prediction. Aiming to model residuals in video compression, the two-stage hybrid model (see in Fig. 21) initially utilizes RNNs to obtain deterministic predictions for the next frame, providing sequential priors for the DM. Then the DM focuses on modeling residuals, whose effect is verified with various metrics perceptually and probabilistically.

By employing different mask manners for time-series, masked conditional DMs can be trained for prediction and interpolation. Höppe et al. introduce conditions through a randomized masking schedule, also allowing the model to be trained conditionally with only slight modifications to the unconditionally trained models. Voleti et al. employ a similar masking concept but further propose a blockwise autoregressive conditioning procedure to facilitate coherent long-term generation. They also incorporate temporal information into the U-Net architecture. In contrast to direct modifications of DDPM, Danier et al. first use LDM in video frame interpolation. They design a vector-quantized autoencoding model for LDM, better recovering high-frequency details and achieving perceptual superiority.

Video super-resolution. Early DM-based video works merely tailor the classical framework to meet data dimensionality of input-output sequences and train the models from scratch, resulting in an undeniable computational burden. Given the tremendous success of DMs , one approach is to leverage off-the-shelf pre-trained models and endow them with temporal modeling capacities by integrating temporal layers into the U-Net architecture. Inspired by such works , Yuan et al. propose an efficient DM-based method for text-to-video super-resolution. By inflating text-to-image (T2I) model weights into the video generation framework and incorporating an attention-based temporal adapter for coherence across frames, this method achieves high-quality and temporally consistent results.

Striving for Spatial Adaptation and Temporal Coherence (SATeCo), Chen et al. propose a novel video SR approach SATeCo (see in Fig. 22), which freezes pre-trained parameters and optimizes spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules. Specifically, SFA modulates the pixel-level high-resolution features for spatial adaptation through learning affine parameters guided by low-resolution videos. TFA conducts self-attention to enhance feature interaction and further performs cross-attention between counterparts of different resolutions to guide temporal feature alignment learning. Experiments validate the effect of the modules in preserving spatial fidelity and enhancing temporal feature alignment.

Video restoration. Limited DM-based algorithms focus on video restoration, showing a promising future direction. Yang et al. propose a novel Diffusion Test-Time Adaptation (Diff-TTA) method for all-in-one adverse weather removal in videos. At the training stage, a novel temporal noise model is introduced to exploit frame-correlated information in degraded video clips. During inference, the authors first introduce test-time adaptation to DM-based methods by proposing a novel proxy task named Diffusion Tubelet Self-Calibration (Diff-TSC). This allows the model to adapt in real-time without modifying the training process and achieve restoration under unseen weather conditions.

EXPERIMENTS

Large-scale datasets for model pre-training. Constructing DMs from scratch has a rigorous training process, requiring great computation capacities and high-memory hardware that are beyond the reach of many research labs . Hence, using pre-trained models is a common practice in generative modeling tasks, despite the associated risk of potential data leakage. Several commonly used large-scale datasets for pre-training are compiled as:

ImageNet is a large-scale dataset with over 14 million natural images spanning over 21k classes, termed ImageNet21K. ImageNet1k, serving as a subset of ImageNet21K, has 1k classes with about 1k images per class.

CelebA has 200k facial images, each annotated with 40 attributes, where CelebA-HQ is a subset having 30k high-resolution facial images.

LSUN includes 10 scene categories and 20 object categories, each having about 1 million labeled images, typically compressed to JPEG image quality of 75.

FFHQ comprises 70k high-resolution facial images with diverse distributions. Existing methods based on pre-trained DMs undergo training on FFHQ and evaluation on CelebA-HQ to showcase their generalizability.

Low-level vision datasets for model training. Various datasets are tailored to address diverse low-level vision tasks, aiming to accommodate various degradation modes. Due to space limitations, we provide a summary of commonly used datasets for several classical natural low-level vision tasks in Table I, including their scales, sources, modalities, and remarks. For more information about datasets in different scenarios, please refer to our repository. In practice, DM-based models are typically pre-trained on large-scale datasets to learn general features and structures, before being fine-tuned on specific low-level vision datasets to address the specific degradation issues.

2 Evaluation metrics

PSNR (Peak Signal to Noise Ratio) quantifies the pixel-wise disparity between a corrupted image and its clean image by computing their mean squared error.

SSIM (Structural Similarity ), aiming to accommodate human visual perception, assesses the likeness between distorted and clean images across three aspects, including contrast, brightness, and structure.

2.2 Inception-based metrics

LPIPS (Learned Perceptual Image Patch Similarity ), is a learning-based metric that leverages the pre-trained AlexNet as a feature extractor and adjusts the linear layer to emulate human perception.

FID (Fréchet inception distance ) assesses the fidelity and diversity of generated images by computing the Fréchet distance of their reference images.

KID (Kernel Inception Distance ), similar to FID, employs maximum mean discrepancy with a polynomial kernel to measure the distance, showing greater stability in the zero-shot and few-shot conditions.

NIQE (Natural Image Quality Evaluator ), a no-reference metric, evaluates the distance between the natural scene statistics of distorted images and natural images modeled with a multivariate Gaussian model.

2.3 Human-centric evaluations

Human-centric evaluation serves as a primary subjective assessment method, where participants select the image verifying the most effective performance from a set of images. To ensure fairness, anonymizing the method and randomizing the order of images within each set are essential practices. Typically, human assessment scores are calculated using the Mean Opinion Score (MOS) derived from a pool of participants. A higher MOS indicates superior perceptual quality as perceived by humans. However, evaluating via MOS can be costly, and the results may be biased due to subjective perceptual differences. Besides, the time-consuming nature of the procedure makes it suitable for small-scale assessments, such as user studies, but challenging to employ for evaluation during training and broader comparisons.

2.4 Downstream application-based evaluations

Apart from improving visual quality, generating those enhanced images that can facilitate high-level vision tasks, such as object recognition and image segmentation, is also a significant object. Hence, the evaluation of various methods extends to examining the impact of low-level vision methods on real-world vision-based applications.

3 Experimental results

We provide quantitative comparisons for DM-based methods on three commonly investigated tasks.

Results on super-resolution. The results for DM-based models on 4$\times$ image super-resolution, tested on DIV2k and Urban100 , are listed in Table III. We find that IDM and DiffIR perform well on LPIPS. They leverage preprocessed images or features as conditional input, which are demonstrated to enhance perceptual quality. In contrast, Resdiff performs well on PSNR and SSIM. This is because Resdiff uses DM to estimate residual information, ensuring the salient consistency of the restored image with the high-resolution image.

Results on deblurring. We evaluate five DM-based methods on the motion deblurring task using the Gopro and HIDE datasets and report The results in Table III. We can find that DiffEvent and HI-Diff achieve competitive performance on PSNRs and SSIMs. DiffEvent is enabled to achieve both low-light recovery and image deblurring by introducing a learnable decomposer. HI-Diff achieves good generalization performance in complex fuzzy scenarios by using LDM to generate a highly-compressed prior. Moreover, MSGD introduces a multi-scale structural bootstrap to better sample from the target condition distribution, hence the best performance on perceptual metrics.

Results on low-light image enhancement. We validate the performance of five DM-based methods on the low-light image enhancement task using the LOLv1 , LOLv2 Real , and LOLv2 Syn datasets. The results, presented in Table IV, demonstrate that GSAD shows superior performance in terms of PSNR. Pydiff performs exceptionally well on LOLv1 Dataset with a roughly 6% improvement over other diffusion models in terms of SSIM. Reti-Diff achieves competitive performance in terms of LPIPS . In terms of model parameters and computational complexity, GSAD significantly outperforms other methods on parameters, but introduces a huge computational complexity. Benefiting from using LDM within a low-dimensional compact space, Reti-Diff has the second smallest parameter count and the lowest MACs. Noting that GSAD and PyDiff employ the ”gt mean” strategy, which involves fine-tuning the brightness of the generated results using the ground truth, thus producing much more impressive results than others in PSNR and SSIM.

Future directions

Compared to other generative models, DMs exhibit the capability to generate high-fidelity images with complex details, rendering DMs widely applied in low-level vision tasks. However, considerable room for advancement remains in both DMs and low-level vision tasks.

In this section, we primarily focus on three avenues to enhance the influence of DMs in low-level vision tasks: (1) Mitigating the limitations of DMs, (2) Amalgamating the strengths of DMs with the traits of low-level vision, and (3) Tackling the inherent challenges of low-level vision.

Due to the high computational overhead, DMs encounter barriers to be applied in low-level vision tasks. Two viable ways are listed and discussed to mitigate this challenge.

Reducing sample steps. Various efforts, extending beyond low-level vision, have been undertaken to enhance the sampling efficiency of DM: (1) Modeling the diffusion process with a non-Markov Chain, such as DDIM. (2) Designing efficient ODE solvers . (3) Using knowledge distillation to reduce sampling steps. (4) Performing DMs on compressed latent spaces . (5) Introducing cross-modality priors with conditional mechanisms.

With these efforts, the number of sampling steps has been notably reduced to 10-20 steps, ensuring faster reconstruction. For instance, DDRM achieves an inference time reduction to 8 seconds for a single $224\times 224$ image by using the sampling strategy of DDIM . Besides, some studies initialize networks by sampling from low-quality images or one-step reconstruction results of baseline networks, streamlining the learning target.

However, despite notable progress, the overall computational cost remains high, particularly for high-resolution images, presenting a substantial gap from real-time applications. Addressing this challenge remains a longstanding and crucial direction for accelerating diffusion models.

Compressing model consumption. The deployment of DM-based models in low-resource environments, such as edge devices, faces challenges due to their immense parameter size and computational complexity. Apart from employing fewer-step inference, researchers can explore architectural optimizations to address this issue, including model quantization, pruning, and knowledge distillation. Zhang et al. combine automated layer pruning with normalized feature distillation to compress models. Castells et al. propose EdgeFusion, an optimized model for deploying SDMs on Neural Processing Units, which leverages advanced distillation techniques and model-level tiling to facilitate rapid inference. However, current methods primarily focus on generation tasks. In the future, these techniques are expected to be extended to low-level vision tasks, leveraging specific properties of each task for model compression.

2 Amalgamating the strengths of DMs with the traits of low-level vision

The greatest trait of low-level vision lies in the diversity of evaluation criteria, including visual fidelity, content invariance, and downstream task-based evaluations. DM-based methods, generating visual fidelity results, also should ensure the content invariance of the original one and the generated result and facilitate downstream tasks.

Perception-distortion trade-off. DM-based methods generate visually appealing results and excel in inception-based metrics, such as LPIPS and FID. However, their high diversity often leads to challenges in maintaining content consistency, resulting in suboptimal performance in those distortion-based metrics such as PSNR and SSIM.

One potential solution involves designing hybrid models that integrate DMs with CNN-based or Transformer-based frameworks . These hybrid models have shown promising results, particularly in improving distortion-based metrics. Besides, Pareto-frontiers are introduced as a comprehensive indicator to evaluate both perception and distortion and have proven the positive shift of the multiscale guidance mechanism that enhances coarse sharp image structures (in Fig. 23 (a)). However, breakthrough progress has not yet been made and further explorations about novel mixed structure and new metrics are expected.

Downstream task-friendly designs. Enabling reconstructed images to better serve downstream tasks is a continuous endeavor in low-level vision research. This pursuit manifests in three primary approaches with DMs.

First, as shown in Fig. 23 (b), several strategies adopt bi-level optimization to jointly optimize the networks of both the low-level vision task and the downstream task, such as image segmentation and object detection. By jointly optimizing the enhancement network with constraints from both itself and the downstream task, these methods aim to produce visually appealing results while enhancing downstream performance. Besides, Ju et. al propose feature-level information aggregation between low-level vision tasks and downstream tasks instead of the previous image-level manner, improving performance with deep constraints. Inspired by the adversarial attacks, which introduce slight perturbations to cause original methods to fail, Sun et. al propose adding slight noise to dehazed images. This strategy enhances downstream detection performance without altering the visual outcome. However, these methods are often tailored to specific downstream tasks. There remains a need for a unified strategy, especially DM-based solutions that can generate visually friend results, to optimize generated images for a wide range of downstream tasks, which awaits further exploration.

3 Tackling the inherent challenges of low-level vision

Low-level vision tasks have several inherent challenges, including generalizability, data volume and controllability.

Real-world image restoration. Two ways can help DM-based methods to address real-world scenarios, i.e., distortion invariant learning (DIL) and distortion estimation (DE).

DIL, renowned for its degradation-invariant representation and structural information preservation, can enhance DM-based methods by incorporating a distortion-invariant noise predictor and condition. This enables these methods to generalize effectively to diverse and even unknown degradations. Pioneering efforts have focused on redesigning the condition module to achieve distortion-invariant conditions, as demonstrated in works such as DifFace and DR2 . Notably, the effectiveness of such conditions also relies on DIL, warranting further research.

Moreover, DE techniques, extracting prior knowledge of degradation processes, are also urgently needed to extend the zero-shot diffusion models to real-world applications. Even though explicit results cannot be obtained, the powerful image synthesis capability of DMs can be utilized to convert synthetic datasets into real-world paired datasets, which will be discussed in detail in the following subsection.

Data generation for data-hungry fields. Data hungry is a prevalent challenge in low-level tasks, often stemming from limitations inherent in imaging devices and scenarios.

While the unsupervised training is one avenue, many existing approaches resort to data generation strategies to create pseudo image pairs. These pairs typically consist of generated degraded low-quality images paired with their corresponding original high-quality counterparts. This is a promising way for DM-based methods, although with limited explorations, for their powerful generation capacity. Moreover, certain extreme tasks suffer from severely limited data availability due to the difficulty or costliness of data acquisition, as seen in Photoacoustic data and Cryo-electron microscopy data. He et. al propose leveraging existing data to generate more training data with GAN and thus enhance the generalizability of the method. This strategy aligns well with the DM-based methods, offering stable training conditions. Furthermore, controllable data generation, facilitated by user interaction, presents a promising approach to filtering out negative data that could otherwise affect stable performance.

Controllable and interactive low-level vision. Enhancing the controllability of low-level vision methods, enabling them to discern what and where users desire recovery, is of paramount importance. This focus has persisted over time, with efforts including the integration of human perception-related loss functions and interactive guidance priors . Recently, the utilization of vision prompts facilitated by Vision-Language models has provided a means for existing low-level vision methods to explicitly incorporate and interact with prompts within their networks, thereby achieving improved control and restoration effects . Given that these vision prompts can act as interactive priors to curb the excessive diversity inherent in DM-based methods, leveraging Vision-Language models to develop controllable and interactive DM-based methods shows promise.

Moreover, future efforts should address real-world scenarios that involve multiple degradations. Zheng et al. introduce a novel DM-based method named DiffUIR, employing a selective hourglass mapping technique. Instead of relying on prompt learning methods, DiffUIR combines shared distribution mapping and robust conditional guidance based on Residual Denoising Diffusion Models to improve image restoration performance. Improving the internal mechanisms of deep learning to better learn the distribution of multi-task degradations represents a promising direction for future DM-based explorations.

Conclusions

In this survey, we offer an extensive examination of denoising diffusion models applied in low-level vision tasks, a gap overlooked in previous surveys. Our review covers both advances and practical implementations in this domain. Firstly, we identify and discuss various generic diffusion modeling frameworks. We then propose a comprehensive categorization of diffusion models utilized in low-level vision tasks from multiple perspectives. Lastly, we highlight the limitations of existing diffusion models and suggest intriguing avenues for future research and explorations. Advancements in low-level vision tasks using these models are emerging in more complex and higher-dimensional mediums, including 3D objects, locomotion, and 4D scenes, warranting sustained attention in future research.

References