Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Introduction

Single image super-resolution (SR) is an active research topic, which aims at reconstructing a high-resolution (HR) image from its low-resolution (LR) counterpart. Since the pioneering work of SRCNN , deep convolution neural network (CNN) approaches have brought prosperous developments in the SR field. However, most approaches assume an ideal bicubic downsampling kernel, which is different from real degradations. This degradation mismatch makes those approaches unpractical in real-world scenarios.

Blind super-resolution , on the contrary, aims to restore low-resolution images suffering from unknown and complex degradations. Existing approaches can be roughly categorized into explicit modeling and implicit modeling, according to the underlying degradation process. Classical degradation model , which consists of blur, downsampling, noise and JPEG compression (more details in Sec. 3.1), is widely adopted in explicit modeling methods . However, the real-world degradations are usually too complex to be modeled with a simple combination of multiple degradations. Thus, these methods will easily fail in real-world samples. Implicit modeling methods utilize data distribution learning with Generative Adversarial Network (GAN) to obtain the degradation model. Yet, they are limited to the degradations within training datasets, and could not generalize well to out-of-distribution images. Readers are encouraged to refer to a recent blind SR survey for a more comprehensive taxonomy.

In this work, we aim to extend the powerful ESRGAN to restore general real-world LR images by synthesizing training pairs with a more practical degradation process. The real complex degradations usually come from complicate combinations of different degradation processes, such as imaging system of cameras, image editing, and Internet transmission. For example, when we take a photo with our cellphones, the photos may have several degradations, such as camera blur, sensor noise, sharpening artifacts, and JPEG compression. We then do some editing and upload to a social media app, which introduces further compression and unpredictable noises. The above process becomes more complicated when the image is shared several times on the Internet.

This motivates us to extend the classical “first-order” degradation model to “high-order” degradation modeling for real-world degradations, i.e., the degradations are modeled with several repeated degradation processes, each process being the classical degradation model. Empirically, we adopt a second-order degradation process for a good balance between simplicity and effectiveness. A recent work also proposes a random shuffling strategy to synthesize more practical degradations. However, it still involves a fixed number of degradation processes, and whether all the shuffled degradations are useful or not is unclear. Instead, high-order degradation modeling is more flexible and attempts to mimic the real degradation generation process. We further incorporate $sinc$ filters in the synthesis process to simulate the common ringing and overshoot artifacts.

As the degradation space is much larger than ESRGAN, the training also becomes challenging. Specifically, 1) the discriminator requires a more powerful capability to discriminate realness from complex training outputs, while the gradient feedback from the discriminator needs to be more accurate for local detail enhancement. Therefore, we improve the VGG-style discriminator in ESRGAN to an U-Net design . 2) The U-Net structure and complicate degradations also increase the training instability. Thus, we employ the spectral normalization (SN) regularization to stabilize the training dynamics. Equipped with the dedicated improvements, we are able to easily train our Real-ESRGAN and achieve a good balance of local detail enhancement and artifact suppression.

To summarize, in this work, 1) we propose a high-order degradation process to model practical degradations, and utilize $sinc$ filters to model common ringing and overshoot artifacts. 2) We employ several essential modifications (e.g., U-Net discriminator with spectral normalization) to increase discriminator capability and stabilize the training dynamics. 3) Real-ESRGAN trained with pure synthetic data is able to restore most real-world images and achieve better visual performance than previous works, making it more practical in real-world applications.

Related Work

The image super-resolution field has witnessed a variety of developments since SRCNN . To achieve visually-pleasing results, generative adversarial network is usually employed as loss supervisions to push the solutions closer to the natural manifold . Most methods assume a bicubic downsampling kernel and usually fail in real images. Recent works also incorporate reinforcement learning or GAN prior to image restoration .

There have been several excellent explorations in blind SR. The first category involves explicit degradation representations and typically consists of two components: degradation prediction and conditional restoration. The above two components are performed either separately or jointly (iteratively) . These approaches rely on predefined degradation representations (e.g., degradation types and levels), and usually consider simple synthetic degradations. Moreover, inaccurate degradation estimations will inevitably result in artifacts.

Another category is to obtain/generate training pairs as close to real data as possible, and then train a unified network to address blind SR. The training pairs are usually 1) captured with specific cameras followed by tedious alignments ; 2) or directly learned from unpaired data with cycle consistency loss ; 3) or synthesized with estimated blur kernels and extracted noise patches . However, 1) the captured data is only constrained to degradations associated with specific cameras, and thus could not well generalize to other real images; 2) learning fine-grained degradations with unpaired data is challenging, and the results are usually unsatisfactory.

Degradation models. Classical degradation model is widely adopted in blind SR methods . Yet, real-world degradations are usually too complex to be explicitly modeled. Thus, implicit modeling attempts to learn a degradation generation process within networks . In this work, we propose a flexible high-order degradation model to synthesize more practical degradations.

Methodology

Blind SR aims to restore high-resolution images from low-resolution ones with unknown and complex degradations. The classical degradation model is usually adopted to synthesize the low-resolution input. Generally, the ground-truth image $\bm{y}$ is first convolved with blur kernel $\bm{k}$. Then, a downsampling operation with scale factor $r$ is performed. The low-resolution $\bm{x}$ is obtained by adding noise $\bm{n}$. Finally, JPEG compression is also adopted, as it is widely-used in real-world images.

where $\mathcal{D}$ denotes the degradation process. In the following, we briefly revisit these commonly-used degradations. The detailed settings are specified in Sec. 4.1. More descriptions and examples are in Appendix. A. Blur. We typically model blur degradation as a convolution with a linear blur filter (kernel). Isotropic and anisotropic Gaussian filters are common choices. For a Gaussian blur kernel $\bm{k}$ with a kernel size of $2t+1$, its $(i,j)\in[-t,t]$ element is sampled from a Gaussian distribution, formally:

where $\bm{\Sigma}$ is the covariance matrix; $\bm{C}$ is the spatial coordinates; $N$ is the normalization constant. The covariance matrix could be further represented as follows:

where $\sigma_{1}$ and $\sigma_{2}$ are the standard deviation along the two principal axes (i.e., eigenvalues of the covariance matrix); $\theta$ is the rotation degree. When $\sigma_{1}=\sigma_{2}$, $\bm{k}$ is an isotropic Gaussian blur kernel; otherwise $\bm{k}$ is an anisotropic kernel.

Discussion. Though Gaussian blur kernels are widely used to model blur degradation, they may not well approximate real camera blur. To include more diverse kernel shapes, we further adopt generalized Gaussian blur kernels and a plateau-shaped distribution. Their probability density function (pdf) are $\frac{1}{N}\exp(-\frac{1}{2}(\bm{C}^{T}\bm{\Sigma}^{-1}\bm{C})^{\beta}$, and $\frac{1}{N}\frac{1}{1+(\bm{C}^{T}\bm{\Sigma}^{-1}\bm{C})^{\beta}}$, respectively. $\beta$ is the shape parameter. Empirically, we find that including these blur kernels could produce sharper outputs for several real samples. Noise. We consider two commonly-used noise types – 1) additive Gaussian noise and 2) Poisson noise. Addictive Gaussian noise has a probability density function equal to that of the Gaussian distribution. The noise intensity is controlled by the standard deviation (i.e., sigma value) of the Gaussian distribution. When each channel of RGB images has independent sampled noise, the synthetic noise is color noise. We also synthesize gray noise by employing the same sampled noise to all three channels .

Poisson noise follows the Poisson distribution. It is usually used to approximately model the sensor noise caused by statistical quantum fluctuations, that is, variation in the number of photons sensed at a given exposure level. Poisson noise has an intensity proportional to the image intensity, and the noises at different pixels are independent. Resize (Downsampling). Downsampling is a basic operation for synthesizing low-resolution images in SR. More generally, we consider both downsamping and upsampling, i.e., the resize operation. There are several resize algorithms - nearest-neighbor interpolation, area resize, bilinear interpolation, and bicubic interpolation. Different resize operations bring in different effects - some produce blurry results while some may output over-sharp images with overshoot artifacts.

In order to include more diverse and complex resize effects, we consider a random resize operation from the above choices. As nearest-neighbor interpolation introduces the misalignment issue, we exclude it and only consider the area, bilinear and bicubic operations. JPEG compression. JPEG compression is a commonly used technique of lossy compression for digital images. It first converts images into the YCbCr color space and downsamples the chroma channels. Images are then split into $8\times 8$ blocks and each block is transformed with a two-dimensional discrete cosine transform (DCT), followed by a quantization of DCT coefficients. More details of JPEG compression algorithms can be found in . Unpleasing block artifacts are usually introduced by the JPEG compression.

The quality of compressed images is determined by a quality factor $q\in$, where a lower $q$ indicates a higher compression ratio and worse quality. We use the PyTorch implementation - $\mathtt{DiffJPEG}$ .

2 High-order Degradation Model

When we adopt the above classical degradation model to synthesize training pairs, the trained model could indeed handle some real samples. However, it still can not resolve some complicated degradations in the real world, especially the unknown noises and complex artifacts (see Fig. 3). It is because that the synthetic low-resolution images still have a large gap with realistic degraded images. We thus extend the classical degradation model to a high-order degradation process to model more practical degradations.

The classical degradation model only includes a fixed number of basic degradations, which can be regarded as a first-order modeling. However, the real-life degradation processes are quite diverse, and usually comprise a series of procedures including imaging system of cameras, image editing, Internet transmission, etc. For instance, when we want to restore a low-quality image download from the Internet, its underlying degradation involves a complicated combination of different degradation processes. Specifically, the original image might be taken with a cellphone many years ago, which inevitably contains degradations such as camera blur, sensor noise, low resolution and JPEG compression. The image was then edited with sharpening and resize operations, bringing in overshoot and blur artifacts. After that, it was uploaded to some social media applications, which introduces a further compression and unpredictable noises. As the digital transmission will also bring artifacts, this process becomes more complicated when the image spreads several times on the Internet.

Such a complicated deterioration process could not be modeled with the classical first-order model. Thus, we propose a high-order degradation model. An $n$-order model involves $n$ repeated degradation processes (as shown in Eq. 5), where each degradation process adopts the classical degradation model (Eq. 1) with the same procedure but different hyper-parameters. Note that the “high-order” here is different from that used in mathematical functions. It mainly refers to the implementation time of the same operation. The random shuffling strategy in may also include repeated degradation processes (e.g., double blur or JPEG). But we highlight that the high-order degradation process is the key, indicating that not all the shuffled degradations are necessary. In order to keep the image resolution in a reasonable range, the downsampling operation in Eq. 1 is replaced with a random resize operation. Empirically, we adopt a second-order degradation process, as it could resolve most real cases while keeping simplicity. Fig. 2 depicts the overall pipeline of our pure synthetic data generation pipeline.

It is worth noting that the improved high-order degradation process is not perfect and could not cover the whole degradation space in the real world. Instead, it merely extends the solvable degradation boundary of previous blind SR methods through modifying the data synthesis process. Several typical limitation scenarios can be found in Fig. 11.

3 Ringing and overshoot artifacts

Ringing artifacts often appear as spurious edges near sharp transitions in an image. They visually look like bands or “ghosts” near edges. Overshoot artifacts are usually combined with ringing artifacts, which manifest themselves as an increased jump at the edge transition. The main cause of these artifacts is that the signal is bandlimited without high frequencies. These artifacts are very common and usually produced by a sharping algorithm, JPEG compression, etc. Fig. 5 (Top) shows some real samples suffering from ringing and overshoot artifacts.

We employ the $sinc$ filter, an idealized filter that cuts off high frequencies, to synthesize ringing and overshoot artifacts for training pairs. The $sinc$ filter kernel can be expressed asWe use the implementation in this url.:

where $(i,j)$ is the kernel coordinate; $\omega_{c}$ is the cutoff frequency; and $J_{1}$ is the first order Bessel function of the first kind. Fig. 5 (Bottom) shows $sinc$ filters with different cutoff frequencies, and their corresponding filtered images. It is observed that it could well synthesize ringing and overshoot artifacts (especially introduced by over-sharp effects). These artifacts are visually similar to those in the first two real samples in Fig. 5 (Top).

We adopt $sinc$ filters in two places: the blurring process and the last step of the synthesis. The order of the last $sinc$ filter and JPEG compression is randomly exchanged to cover a larger degradation space, as some images may be first over-sharpened (with overshoot artifacts) and then have JPEG compression; while some images may first do JPEG compression followed by sharpening operation.

4 Networks and Training

ESRGAN generator. We adopt the same generator (SR network) as ESRGAN , i.e., a deep network with several residual-in-residual dense blocks (RRDB), as shown in Fig. 4. We also extend the original $\times 4$ ESRGAN architecture to perform super-resolution with a scale factor of $\times 2$ and $\times 1$. As ESRGAN is a heavy network, we first employ the pixel-unshuffle (an inverse operation of pixel-shuffle ) to reduce the spatial size and enlarge the channel size before feeding inputs into the main ESRGAN architecture. Thus, the most calculation is performed in a smaller resolution space, which can reduce the GPU memory and computational resources consumption.

U-Net discriminator with spectral normalization (SN). As Real-ESRGAN aims to address a much larger degradation space than ESRGAN, the original design of discriminator in ESRGAN is no longer suitable. Specifically, the discriminator in Real-ESRGAN requires a greater discriminative power for complex training outputs. Instead of discriminating global styles, it also needs to produce accurate gradient feedback for local textures. Inspired by , we also improve the VGG-style discriminator in ESRGAN to an U-Net design with skip connections (Fig. 6). The U-Net outputs realness values for each pixel, and can provide detailed per-pixel feedback to the generator.

In the meanwhile, the U-Net structure and complicate degradations also increase the training instability. We employ the spectral normalization regularization to stabilize the training dynamics. Moreover, we observe that spectral normalization is also beneficial to alleviate the over-sharp and annoying artifacts introduced by GAN training. With those adjustments, we are able to easily train the Real-ESRGAN and achieve a good balance of local detail enhancement and artifact suppression.

The training process is divided into two stages. First, we train a PSNR-oriented model with the L1 loss. The obtained model is named by Real-ESRNet. We then use the trained PSNR-oriented model as an initialization of the generator, and train the Real-ESRGAN with a combination of L1 loss, perceptual loss and GAN loss .

Experiments

Training details. Similar to ESRGAN, we adopt DIV2K , Flickr2K and OutdoorSceneTraining datasets for training. The training HR patch size is set to 256. We train our models with four NVIDIA V100 GPUs with a total batch size of 48. We employ Adam optimizer . Real-ESRNet is finetuned from ESRGAN for faster convergence. We train Real-ESRNet for $1000K$ iterations with learning rate $2\times 10^{-4}$ while training Real-ESRGAN for $400K$ iterations with learning rate $1\times 10^{-4}$. We adopt exponential moving average (EMA) for more stable training and better performance. Real-ESRGAN is trained with a combination of L1 loss, perceptual loss and GAN loss, with weights $\{1,1,0.1\}$, respectively. We use the $\{\mathtt{conv1},...\mathtt{conv5}\}$ feature maps (with weights $\{0.1,0.1,1,1,1\}$) before activation in the pre-trained VGG19 network as the perceptual loss. Our implementation is based on the BasicSR .

Degradation details. We employ a second-order degradation model for a good balance of simplicity and effectiveness. Unless otherwise specified, the two degradation processes have the same settings. We adopt Gaussian kernels, generalized Gaussian kernels and plateau-shaped kernels, with a probability of $\{0.7,0.15,0.15\}$. The blur kernel size is randomly selected from $\{7,9,...21$}. Blur standard deviation $\sigma$ is sampled from $[0.2,3]$ ($[0.2,1.5]$ for the second degradation process). Shape parameter $\beta$ is sampled from $[0.5,4]$ and $$for generalized Gaussian and plateau-shaped kernels, respectively. We also use$sinc$ kernel with a probability of 0.1. We skip the second blur degradation with a probability of 0.2.

We employ Gaussian noises and Poisson noises with a probability of $\{0.5,0.5\}$. The noise sigma range and Poisson noise scale are set to $$and$[0.05,3]$, respectively ($$and$[0.05,2.5]$for the second degradation process). The gray noise probability is set to 0.4. JPEG compression quality factor is set to$$. The final$sinc$ filter is applied with a probability of 0.8. More details can be found in the released codes.

Training pair pool. In order to improve the training efficiency, all degradation processes are implemented in PyTorch with CUDA acceleration, so that we are able to synthesize training pairs on the fly. However, batch processing limits the diversity of synthetic degradations in a batch. For example, samples in a batch could not have different resize scaling factors. Therefore, we employ a training pair pool to increase the degradation diversity in a batch. At each iteration, the training samples are randomly selected from the training pair poor to form a training batch. We set the pool size to 180 in our implementation.

Sharpen ground-truth images during training. We further show a training trick to visually improve the sharpness, while not introducing visible artifacts. A typical way of sharpening images is to employ a post-process algorithm, such as unsharp masking (USM). However, this algorithm tends to introduce overshoot artifacts. We empirically find that sharpening ground-truth images during training could achieve a better balance of sharpness and overshoot artifact suppression. We denote the model trained with sharped ground-truth images as Real-ESRGAN+ (comparisons are shown in Fig. 7).

2 Comparisons with Prior Works

We compare our Real-ESRGAN with several state-of-the-art methods, including ESRGAN , DAN , CDC , RealSR and BSRGAN . We test on several diverse testing datasets with real-world images, including RealSR , DRealSR , OST300 , DPED , ADE20K validation and images from Internet. Since existing metrics for perceptual quality cannot well reflect the actual human perceptual preferences on the fine-grained scale , we present several representative visual samples in Fig. 7. The quantitative results are also included in the Appendix. B for reference.

It can be observed from Fig. 7 that our Real-ESRGAN outperforms previous approaches in both removing artifacts and restoring texture details. Real-ESRGAN+ (trained with sharpened ground-truths) can further boost visual sharpness. Specifically, the first sample contains overshoot artifacts (white edges around letters). Directly upsampling will inevitably amplify those artifacts (e.g., DAN and BSRGAN). Real-ESRGAN takes such common artifacts into consideration and simulates them with $sinc$ filter, thus effectively removing ringing and overshoot artifacts. The second sample contains unknown and complicated degradations. Most algorithms can not effectively eliminate them while Real-ESRGAN trained with second-order degradation processes could. Real-ESRGAN is also capable of restoring more realistic textures (e.g., brick, mountain and tree textures) for real-world samples, while other methods either fail to remove degradations or add unnatural textures (e.g., RealSR and BSRGAN).

3 Ablation Studies

Second-order degradation model. We conduct ablation studies of degradations on Real-ESRNet, as it is more controllable and can better reflect the influence of degradations. We replace the second-order process in Real-ESRNet with the classical degradation model to generate training pairs. As shown in Fig. 8 (Top), models trained with classical first-order degradation model cannot effectively remove noise on the wall or blur in the wheat field, while Real-ESRNet can handle these cases.

sinc filters. If $sinc$ filters are not employed during training, the restored results will amplify the ringing and overshoot artifacts that existed in the input images, as shown in Fig. 8 (Bottom), especially around the text and lines. In contrast, models trained with $sinc$ filters can remove those artifacts.

U-Net discriminator with SN regularization. We first employ the ESRGAN setting including the VGG-style discriminator and its loss weights. However, we can observe from Fig. 9, this model cannot restore detailed textures (bricks and bushes) and even brings unpleasant artifacts in bush branches. Using a U-Net design could improve local details. Yet, it introduces unnatural textures and also increases training instability. SN regularization could improve restored textures while stabilizing training dynamics.

More complicated blur kernels. We remove the generalized Gaussian kernel and plateau-shaped kernel in blur synthesis. As shown in Fig. 10, on some real samples, the model cannot remove blur and recover sharp edges as Real-ESRGAN do. Nevertheless, on most samples, their differences are marginal, indicating that the widely-used Gaussian kernels with a high-order degradation process can already cover a large real blur space. As we can still observe slightly better performance, we adopt those more complicated blur kernels in Real-ESRGAN.

4 Limitations

Though Real-ESRGAN is able to restore most real-world images, it still has some limitations. As shown in Fig. 11, 1) some restored images (especially building and indoor scenes) have twisted lines due to aliasing issues. 2) GAN training introduces unpleasant artifacts on some samples. 3) It could not remove out-of-distribution complicated degradations in the real world. Even worse, it may amplify these artifacts. These drawbacks have great impact on the practical application of Real-ESRGAN, which are in urgent need to address in future works.

Conclusion

In this paper, we train the practical Real-ESRGAN for real-world blind super-resolution with pure synthetic training pairs. In order to synthesize more practical degradations, we propose a high-order degradation process and employ $sinc$ filters to model common ringing and overshoot artifacts. We also utilize a U-Net discriminator with spectral normalization regularization to increase discriminator capability and stabilize the training dynamics. Real-ESRGAN trained with synthetic data is able to enhance details while removing annoying artifacts for most real-world images.

Acknowledgement. This work is partially supported by National Natural Science Foundation of China (61906184), the Shanghai Committee of Science and Technology, China (Grant No. 21DZ1100800 and 21DZ1100100).

References

Appendix A Details of Classical Degradation Model

In this section, we provide more details (especially examples) of each degradation type used in the classical degradation model.

Isotropic and anisotropic Gaussian filters are the common choices for blur kernels. We show several Gaussian kernels and their corresponding blurry images in Fig. 12.

To include more diverse kernel shapes, we further adopt generalized Gaussian blur kernels and a plateau-shaped distribution. Fig. 13 shows how the shape parameter $\beta$ controls kernel shapes. Empirically, we found that including these blur kernels produces sharper outputs for several real samples.

A.2 Noise

Fig. 14 depicts the additive Gaussian noise and Poisson noise. Poisson noise has an intensity proportional to the image intensity, and the noises at different pixels are independent of one another. As shown in Fig. 14, the Poisson noise has low noise intensity in dark areas.

A.3 Resize

There are several resize algorithms. We compare the following resize operations: nearest-neighbor interpolation, area resize, bilinear interpolation and bicubic interpolation. We examine the different effects of these resize operations. We first downsample an image by a scale factor of four and then upsample to its original size. Different downsampling and upsampling algorithms are performed, and the results of different combinations are shown in Fig. 15. It is observed that different resize operations result in very different effects - some produce blurry results while some may output over-sharp images with overshoot artifacts.

A.4 JPEG compression

We use the PyTorch implementation - $\mathtt{DiffJPEG}$. We observe that the compressed images by $\mathtt{DiffJPEG}$ are a bit different from those compressed by the $\mathtt{cv2}$ package. Fig. 16 shows the typical JPEG compression artifacts and the difference caused by using different packages. Such a difference may bring an extra gap between synthetic and real samples. In this work, we only adopt $\mathtt{DiffJPEG}$ for simplicity, and this difference will be addressed later.

Appendix B Quantitative Comparisons

We provide the non-reference image quality assessment - NIQE for reference. Note that existing metrics for perceptual quality cannot well reflect the actual human perceptual preferences on the fine-grained scale .

We compare our Real-ESRGAN with several state-of-the-art methods, including ESRGAN , DAN , CDC , RealSR and BSRGAN . We test on several diverse testing datasets with real-world images, including RealSR , DRealSR , OST300 , DPED , ImageNet validation and ADE20K validation . The results are shown in Tab. 1. Though our Real-ESRGAN+ does not optimize for NIQE scores, it sill produces lower NIQE scores on most testing datasets.

Appendix C More Qualitative Comparisons

We show more qualitative comparisons with previous works. As shown in Fig. 17, our Real-ESRGAN outperforms previous approaches in both removing artifacts and restoring texture details. Real-ESRGAN+ (trained with sharpened ground-truths) can further boost visual sharpness. Other methods typically fail to remove complicated artifacts (the 1st sample) and overshoot artifacts (the 2nd, 3rd sample), or fail to restore realistic and natural textures for various scenes (the 4th, 5th samples).