NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results

Introduction

Single image Super-Resolution (SR) is the task of increasing the resolution of a given image by filling in additional high-frequency content. It has been a popular research topic for decades due to its many applications. The current trend addresses the ill-posed SR problem using deep Convolutional Neural Networks (CNNs). While initial methods focused on achieving high fidelity in terms of PSNR . Recent work has put further emphasis on generating perceptually more appealing predictions using for instance adversarial losses .

Deep learning based SR methods are known to consume large quantities of training data. Most current approaches rely on paired low and high-resolution images to train the network in a fully supervised manner. However, such image pairs are not available in real-world applications. To circumvent this fact, the conventional approach has been to downscale images, often with a bicubic kernel, to artificially generate corresponding LR images. This strategy significantly changes the low-level characteristics of the image, by e.g. severely reducing the sensor noise. Super-resolution networks trained on downscaled images therefore often struggle to generalize to natural images. The research direction of blind super-resolution does not fully address this setting since it often relies on paired data and constrained image formation models. In this challenge, the aim is instead to learn super-resolution from unpaired data and without any restricting assumptions on the input image formation. This scenario has recently attracted significant interest due to its high relevance in applications .

The NTIRE 2020 Challenge on Real-World Image Super-Resolution aims to stimulate research in the direction of real-world super-resolution. No paired reference HR images are available for training. Instead, the participants are only provided the source input images, along with an unpaired set of high-quality images that act as the target quality domain. The challenge consists of two tracks. The source images for Track 1 are generated by performing a degradation operation that is unknown to the participants. This degradation arise from image signal processing methods similar to those found on low-end devices (see Figure 1 an example). A synthetic degradation allows us to compute reference-based metrics for evaluation. Track 2 employs images taken from a low-quality smartphone camera, with no available ground-truth. For both tracks, the goal is to achieve perceptually pleasing results. The final ranking is therefore performed using a human study.

This challenge is one of the NTIRE 2020 associated challenges on: deblurring , nonhomogeneous dehazing , perceptual extreme super-resolution , video quality mapping , real image denoising , real-world super-resolution , spectral reconstruction from RGB image and demoireing .

NTIRE 2020 Challenge

The goals of the NTIRE 2020 Challenge on Real-World Image Super-Resolution is to (i) promote research into weak and unsupervised learning approaches for SR, that jointly enhance the image quality (ii) promote a benchmark protocol and dataset; and (iii) probe the current state-of-the-art in the field. The challenge contains two tracks. Both tracks have the goal of upscaling with factor $4\times$. The competition was organized using the Codalab platform.

This track employs the benchmarking strategy described in , which employs an artificial degradation operator to enable reference-based evaluation.

Degradation operator We employ an undisclosed degradation operator which generates structured artifacts commonly produced by the kind of image processing pipelines found on very low-end devices. This type of degradation operator is very different from what has been used in previous challenges . This operation is applied to all source domain images of train, validation and test. According to the rules of the challenge, the participants were not permitted to try to reverse-engineer or with hand-crafted algorithms construct similar-looking degradation artifacts. It was however allowed to try to learn the degradation operator using generic techniques (such as deep networks) that can be applied to any other sort of degradations or source of natural images. The reason is that the method as a whole needs to generalize to different types of degradations and input domains.

Data The dataset is constructed following the general strategy used for Track 2 in the previous edition of the challenge . We construct a dataset of source (i.e. input) domain training images $\mathcal{X}_{\text{train}}=\{x_{i}\}$ by applying the degradation operation to the 2650 images of the Flickr2K dataset, without performing any downsampling. The target domain for training $\mathcal{Y}_{\text{train}}=\{y_{j}\}$ are the original 800 clean high-quality training images from DIV2K. For validation and testing, we employ the corresponding splits from the DIV2K dataset. The source domain images $\mathcal{X}_{\text{val}}$ and $\mathcal{X}_{\text{test}}$ are obtained by first downscaling the images followed by the degradation. The Ground Truth images for validation $\mathcal{Y}_{\text{val}}^{\text{tr1}}$ and test $\mathcal{Y}_{\text{test}}^{\text{tr1}}$ are the original DIV2K images. A visual example for source and target images are provided in Figure 1.

2 Track 2: Smartphone Images

Here the task is to super-resolve real-world images obtained from a low-quality smartphone camera. The desired output quality is defined by set of clean high-quality images. We employ the iPhone3 images of the DPED dataset as source domain $\mathcal{X}_{\text{train}}$. For training and validation, we employ the corresponding predefined splits of DPED. As a ground truth to super-resolved images above sensor size does not exist, we use crops of the validation set of DPED for a human perception study. The target domain $\mathcal{Y}_{\text{train}}$ is the same as in Track 1. A visual example for source and target images are provided in Figure 1.

3 Challenge phases

The challenge had three phases: (1) Development phase: the participants got training images and the LR images of the validation set. (2) Validation phase: the participants had the opportunity to measure performance using the PSNR and SSIM metrics by submitting their results on the server for Track 1. A validation leaderboard was also available. (3) Final test phase: the participants got access to the LR test images and had to submit their super-resolved images along with description, code and model weights for their methods.

Challenge Results

Before the end of the final test phase, participating teams were required to submit results, code/executables, and factsheets for their approaches. From 292 registered participants in Track 1, 19 valid methods were submitted, stemming from 16 different teams. Track 2 had 251 registered participants, of which 15 valid methods were submitted from 14 different teams. Table 1 and 2 report the final results of Track 1 and 2 respectively, on the test data of the challenge. The methods of the teams that entered the final phase are described in Section 4 and the teams’ members and affiliations are shown in Section Appendix A.

Inspired by the results of the last challenge in AIM 2019 and on the success of recent approaches , most top methods pursued a two step approach. The first step aims to learn a network that can transfer clean images to the source domain. This network thus learns a degradation operator, adding the kind of noise and corruptions present in the source images. It is then used to generate paired training data for the second step, which involves learning the super resolution network itself. It is generally trained using pairs generated by first downscaling and then applying the learned degradation on images from the target domain set. Many works employed the DSGAN framework from the winner of the AIM 2019 challenge to learn the degradation operator in the first step.

Some of the top methods in this challenge proposed particularly notable alterations and extensions to the general idea described above for learning the degradation network. The AITA-Noah team (Sec. 4.2) employs an iterative approach for Track 1, alternating between learning the degradation and SR network. It also uses an explicit denoising algorithm and train a sharpening network to decrease the blurring effects from the former. Impressionism (Sec. 4.1) is the only team that aims to explicitly estimate the blur kernel in the image, for improved source data generation. For Track 2, it employs the KernelGAN for this purpose. It also aims to explicitly estimate the noise variance using source image patches. This approach led to superior sharpness and quality in the generated SR images for Track 2. There were also some alternative strategies proposed. In particular, the Samsung-SLSI-MSL team (Sec. 4.3) aim to train a robust SR network capable of handling different source domains by randomly sampling a variety of degradations during the training of the SR network.

For the Real-world Super-Resolution setting, the results in the challenge suggest that training strategy and careful degradation modelling is far more important than choice of SR architecture. For the latter, most top methods simply adopted popular architectures, such as the RRDB/ESRGAN and the RCAN . Most methods also included adversarial and perceptual VGG losses, often based on the ESRGAN framework. Brief descriptions of the methods submitted from each team is given in Sec. 4.

2 Baselines

We compare methods participating in the challenge with several baseline approaches.

Bicubic Standard bicubic upsampling using MATLAB’s imresize function.

RRDB PT The pre-trained RRDB , using the network weights provided by the authors. The network was trained with clean images using bicubic down-sampling for supervision. The only objective is the PSNR oriented L1 loss.

ESRGAN Supervised ESRGAN network that is fine-tuned in a fully supervised manner, by applying the synthetic degradation operation used in Track 1. The degradation was unknown for the participants. This method therefore serves as an upper bound in performance, allowing us to analyze the gap between supervised and unsupervised methods. We employ the source $\mathcal{X}_{\text{train}}$ and target $\mathcal{Y}_{\text{train}}$ domain train images respectively. Low-resolution training samples are constructed by first down-sampling the image using the bicubic method and then apply the synthetic degradation. The network is thus trained with real input and output data, which is otherwise inaccessible. As for previous baselines, the network is initialized with the pre-trained weights provided by the authors. Note that no supervised baseline is available for Track 2 since no ground-truth HR images exists.

3 Evaluation Metrics

The aim of the challenge is to pursue good image quality as perceived by humans. As communicated to the participants at the start of the challenge, the final ranking was therefore to be decided based on a human perceptual study.

Track 1 For Track 1, the fidelity-based Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity index (SSIM) was provided on the Codalab platform for quantitative feedback. These metrics are also reported here for the test set. Moreover, we report the LPIPS distance, which is a learned reference-based image quality metric computed as the $L^{2}$ distance in a deep feature space. The network itself has been fine-tuned based on image quality annotations, to correlate better with human perceptual opinions. However, this metric needs to be used with great care since many methods employ feature-based losses using ImageNet pre-trained VGG networks, which in its design is very similar to LPIPS. Moreover, some methods directly use the LPIPS distance as a loss of for hyper-parameter tuning. We treat LPIPS as an indication of perceptual quality, but not as a metric to decide final rankings.

To obtain a final ranking of the methods, we performed a user study on Amazon Mechanical Turk. For Track 1, where reference images are available, we calculate the Mean Opinion Score (MOS) in the following manner. The test candidates were shown a side-by-side comparison of a sample prediction of a certain method and the corresponding reference ground-truth. They were then asked to evaluate the quality of the SR image w.r.t. the reference image using the 6-level scale defined as: 0 - ’Perfect’, 1 - ’Almost Perfect’, 2 - ’Slightly Worse’, 3 - ’Worse’, 4 - ’Much Worse’, 5 - ’Terrible’. The images shown to the participants of the study were composed of zoomed crops, as shown in Figure 2. The human study was performed for the top 10 methods according to LPIPS distance, along with 4 baseline approaches.

Track 2 For Track 2, a ground truth reference does not exist due to the nature of the problem. Therefore we used several no-reference based image quality assessment (IQA) metrics. In particular, we report the NIQE , BRISQUE and PIQE , using their corresponding MATLAB implementations. Moreover, we report the learned NRQM IQA score. We also report two metrics that summarize the result of the computed IQA metrics. The Perceptual Index PI, previously employed in , is calculated as an adjusted mean of NIQE and NRQM. We also compute the mean IQA-Rank by taking the average image-wise rank achieved w.r.t. each of the four IQA metrics. In this case, taking the average rank is preferred over the average value, since the rank is not sensitive to the specific scaling or range of the particular metric.

Since no reference image exists in Track 2, the MOS score as defined for Track 1 cannot be computed. Instead, we compute the Mean Opinion Rank (MOR) by asking the study participants to rank the predictions of several methods in terms of image quality. For each question, the study participants were shown the SR results of all methods in the study for a particular image. These images were then ranked in terms of overall image quality. The MOR is then computed by averaging the assigned rank of each method, over all images and study participants. Since ranking too many entries at once is cumbersome and can lead to inaccurate results, we performed the human study on the top 5 approaches along with two baselines. As we did not find any of the IQA metrics previously discussed to correlate well with perceived image quality, the initial selection of top 5 methods was performed using a purely visual comparison performed by the challenge organizers. The top 5 methods were selected by assessing sharpness, noise, artifacts, and overall quality. The MOR scores were then computed using Amazon Mechanical Turk.

4 Track 1: Image Processing Artifacts

Here we present the results for Track 1. All experiments presented were conducted on the test set. The results are shown in Table 1. The Impressionism team achieves the best result, with a $9.5\%$ better MOS than the second entry, namely Samsung-SLSI-MSL. Both these teams take a more direct approach for simulating degradations for supervised SR learning. While Samsung-SLSI-MSL sample random noise distributions and down-scaling kernels, Impressionism aim to estimate the kernel and noise statistics. The following three approaches: BOE-IOT-AIBD, MSMers, and KU-ISPL, employ CycleGAN or DSGAN based methods to learn the degradation operator. Also the AITA-Noah team follows this general strategy, achieving impressive MOS results. However, their methods are not ranked in Track 1 since source domain images from the test set was used for training, which is against the rules of the challenge. Notable are also the results of ITS425, who achieve the second best PSNR and best SSIM, while preserving good perceptual quality. Also the third-ranked method BOE-IOT-AIBD achieves very impressive PSNR and SSIM.

When comparing with the previous edition of the challenge , the performance of the proposed method has improved substantially. In , most method achieved similar or worse results than simple Bicubic interpolation. Here, all top-10 approaches achieved better MOS than the Bicubic baseline. Moreover, while a large gap to supervised methods was reported in , in this year challenge, the winning Impressionism method even beats the ESRGAN baseline, which is trained with full supervision. While this can also be partly explained by other modifications and hyper-parameter settings, it clearly demonstrates that the performance gap to supervised SR methods is significantly narrower. Visual results for all methods in Figure 2.

5 Track 2: Smartphone Images

Quantitative results for Track 2 are reported in Table 2. In this track, the Impressionism method outperforms other approaches by a large margin in the human study (MOR). This is also confirmed in the visual examples shown in Figure 3. The generated images are superior in sharpness compared to those of other approaches. Moreover, the SR images contain almost no noise and few artifacts. While AITA-Noah and ITS425 also generate clean images, they lack the sharpness and detail of Impressionism. We believe this to be largely due to the kernel estimation performed in the latter approach, employing KernelGAN for this purpose. This allows the SR network to take the pointspread function of the specific camera sensor into account.

We observe that Impressionism also achieves the best average IQA-Rank. However, note that while the Relbmag-Eht team achieves a similar IQA-Rank, their result severely suffers from a structured noise pattern. This suggests that standard IQA metrics are not well suited as evaluation criteria in this setting and data. Interestingly, the Samsung-SLSI-MSL team employed the paired DSLR images provided by . This approach is therefore not ranked in this track. However, this approach does still not achieve close to the same level of sharpness as Impressionism.

Despite being the first challenge of its kind, the top participating teams achieved very impressive results in this difficult real-world setting, where no reference data is available. In particular, the Impressionism team achieves not only a higher resolution image, but also substantially better image quality than the source image taken by the camera.

Challenge Methods and Teams

This sections give brief descriptions of the participating methods. A summary of all participants is given in table 3.

The team Impressionism proposes a novel framework, introduced in , to improve the robustness of the super-resolution model on real images, which usually fails when trained on bicubic downsampled data. To generate more realistic LR images, they design a real-world degradation process that maintains important original attributes. Specifically, they focus on two aspects: 1) The blurry LR image is obtained by downsampling High-Resolution (HR) images with estimated kernels from real blurry images. 2) The real noise distribution is restored by injecting collected noise patches from real noisy images. From the real-world (source domain) dataset $\mathcal{X}$ and the clean HR (target domain) dataset $\mathcal{Y}$, the team thus aims to construct domain-consistent data $\{{\mathbf{I}}_{\text{LR}},{\mathbf{I}}_{\text{HR}}\}\in\{\mathcal{X},\mathcal{Y}\}$.

Clean-up Since bicubic downsampling can remove high-frequency noise, they directly do a bicubic downsampling on the image from $\mathcal{X}$ to obtain more HR images. Let ${\mathbf{I}}_{\text{src}}\in\mathcal{X}$, and ${\mathbf{k}_{\text{bic}}}$ be the ideal bicubic kernel. Then the image is downsampled with a clean-up scale factor $s$ as ${\mathbf{I}}_{\text{HR}}=({\mathbf{I}}_{\text{src}}*{\mathbf{k}_{\text{bic}}}){\downarrow}_{s}$. Then the images after downsampling are regarded as clean HR images, that is ${\mathbf{I}}_{\text{HR}}\in\mathcal{Y}$.

Downsampling The team performs downsampling on the clean HR images using the estimated kernels by KernelGAN . The downsampling process is a cross-correlation operation followed by sampling with stride $s$,

where ${\mathbf{I}}_{D}$ denotes the downsampled image, and ${\mathbf{k}}_{i}$ refers to the specific blur kernel.

Noise Injection Mere estimation of the blurry kernel cannot accurately model the degradation process of $\mathcal{X}$. By observing the real data, they find that the noise is usually combined with content of the image. In order to decouple noise and content, they design a filtering rule to collect noise patches $\{{\mathbf{n}}_{i},i\in\{1,2\cdots l\}\}$ with their variance in a certain range $\sigma^{2}({\mathbf{n}}_{i})<v$, where $\sigma^{2}(\cdot)$ denotes the variance, and $v$ is the maximum value of variance. Then these patches will be added to ${\mathbf{I}}_{D}$ as,

After downsampling HR images with the estimated kernels and injecting collected noise, they obtain ${\mathbf{I}}_{LR}\in\mathcal{X}$.

Network Details Based on ESRGAN , they train a super-resolution model on constructed paired data $\{{\mathbf{I}}_{LR},{\mathbf{I}}_{HR}\}\in\{\mathcal{X},\mathcal{Y}\}$. Three losses are applied to training including pixel loss $L_{1}$, perceptual loss $L_{per}$, and adversarial loss $L_{adv}$. Different from default setting, they use patch discriminator instead. Overall, the final training loss is as follows:

where ${\lambda}_{1}$, ${\lambda}_{\text{per}}$, and ${\lambda}_{\text{adv}}$ are set as 0.01, 1, and 0.005 empirically.

2 AITA-Noah

This method, which is detailed in , adopts the idea of learning the degradation operator in order to synthetically generate paired training data for SR network. For Track 1, an approach termed Iterative Domain Adaptation is developed. The source training data $\mathcal{X}_{tr}$ and downsampled target training data $\mathcal{Y}_{tr\downarrow}$ are first processed with a denoising algorithm (Non-local Means), denoted $D$. The sets $D(\mathcal{Y}_{tr\downarrow})$ and $\mathcal{Y}_{tr\downarrow}$ are then used to train a sharpening network $S$, in a fully supervised manner. When applied to the source data, $S(D(x_{tr}))$ generates images that are clean and sharp. This set can then be used to train a degradation operator $G$, using pairs from $S(D(\mathcal{X}_{tr}))$ and $\mathcal{X}_{tr}$. This is then used to train a super-resolution network $SR$ using pairs generated by $G(\mathcal{Y}_{tr\downarrow})$ and $\mathcal{Y}_{tr}$. The approach then proceeds by iteratively improving the degradation model $G$ using pairs $f(\mathcal{X}_{tr})$ generated by the current SR model $f$ and $\mathcal{X}_{tr}$, and improving the super-resolution model $f$ using pairs $G(\mathcal{Y}_{tr})$ generated by the current degradation operator $G$ and $\mathcal{Y}_{tr}$. In practice, the team used the 100 source validation images and 100 source test images as $\mathcal{X}_{tr}$. The team is not ranked in track 1, since according to the challenge rules, the test data should not be used during training, even in unpaired form.

For Track 2, the team adopts the CycleSR framework to generate degrade images. As illustrated in Fig. 5, this framework is composed of two stages: 1) unsupervised image translation between real LR images $\mathcal{X}_{tr}$ and synthetic LR images, i.e., $4\times$ bicubic downsampled HR images $\mathcal{Y}_{tr}$, denoted by $\mathcal{Y}_{tr\downarrow}$; 2) supervised super-resolution from degraded LR images $\hat{\mathcal{Y}}_{tr\downarrow}$ to get $\hat{\mathcal{Y}}_{tr}$. In detail, the approach first takes the unsupervised image translation model CycleGAN for mapping between domain $\mathcal{X}_{tr}$ and $\mathcal{Y}_{tr\downarrow}$. An SR module SRResNet is employed after CycleGAN to super-resolve $\hat{\mathcal{Y}}_{tr\downarrow}$ to get $\hat{\mathcal{Y}}_{tr}$ and compute the loss $L_{\text{SR}}$ with ground truth $\mathcal{Y}_{tr}$. Hence, with an image translation model and an SR module together and a joint training strategy, we are able to train a model that super-resolves real LR images to HR images with an indirect supervised path. Compared with degradation directly using original CycleGAN, benefiting from the pixel-wise feedback of the SR module, CycleSR can alleviate color and brightness changes during degradation.

In both tracks, the same super-resolution architecture, based on the ESRGAN is used. The team furthermore use an LR-conditional frequency-separation discriminator to train the model and employ AutoML to tune the loss weights, employing LPIPS and NIQE as objective. Two versions of this approach was submitted, with the significant differences as follows:

AITA-Noah-A For Track 1, this version uses the method described above. For Track 2, it includes an extra 400 images selected from DIV8K in the target domain set $\mathcal{Y}_{tr}$ to improve data diversity.

AITA-Noah-B For Track 1, this approach additionally uses an ensemble fusion strategy (i.e. running inference on the vertical flipped/horizontal flipped/transposed images of the original input, and then average the results), in addition to above. For Track 2, no extra data was used and no adversarial loss was used during training the ESRGAN model (i.e. only RRDBNet was used).

3 Samsung-SLSI-MSL

For Track 1, this team aims to train a generic SR model that is robust to various image degradations, which can therefore be applied in real-world scenarios without knowledge of the specific degradation operator. This is performed by sampling diverse degradation types during training. The strategy proposed in the blind denoising method is extended by adding downscaling and blur. The training set is generated by sampling different downscaling (e.g. bilinear, nearest neighbor or bicubic), blur kernels (Gaussian kernel with different sigma), and noise distributions (additive Gaussian, Poisson, Poisson-Gaussian with randomly sampled parameters). The SR model consists of the RCAN architecture, which is trained with a GAN loss while emphasizing the perceptual losses. To further improve the perceptual quality, they deploy an ensemble of two different GANs, and use cues from the image luminance and adjust to generate better HR images at low-illumination. The workflow is given in Fig. 6.

For Track 2, real world SR on images captured by mobile devices, the same GANs are trained by weak supervision on a mobile SR training set that they constructed to have LR-HR image pairs, from the DPED dataset which provides registered mobile-DSLR images at the same scale . They use the mobile images as LR, and apply the track 1 generic SR model on the paired DSLR images to create super resolved HR images with good perceptual quality. This method is considered as a kind of Supervised approach, and does not compete with the other participants in Track 2. Details about the proposed method can be found in .

4 MSMers

This method takes inspiration from , developing a two-stage approach. First, a degradation operator is learned in an unsupervised manner. This is then used to generate paired data for the second stage, in which the SR network is learned. Specifically, CycleGAN is adopted in the first stage to learn a mapping from bicubic downsampled HR to real LR. To keep the color consistent, the weight of the identity loss is increased in the setting. As for the second stage, RCAN is used to super-resolve the LR image, which is first trained on L1 loss. On top of that, perceptual loss and adversarial loss are added for better perceptual quality. Specifically, we use features of VGG19 relu5-1 layer to compute a perceptual loss and the WGAN-GP as adversarial loss. The method is visualized in Figure 7.

5 BOE-IOT-AIBD

This team aims to learn the degradation operator in order to generate paired SR training samples. To this end, it employs solution provided by DSGAN to artificially generate LR images, as shown in Figure 8a. These are then used to train an SR model. For this, it uses the modified MGBPv2 network, proposed in the winning solution of the AIM ExtremeSR challenge . It is adapted to $4\times$ upscaling by using a triple–V cycle (instead of the W–cycle) and adding multi–scale denoising modules as shown in Figure 8b. During inference, an overlapping patch approach is used to further allow upscaling of large images. The training strategy employs a multiscale loss, combining distortion and perception losses on the output images. Model selection was performed by selecting low NIQE results on validation set and human tests based on ITU–T P.910. An additional set of 739 collected images for training. The team only participated in Track 1.

6 InnoPeak-SR

This approach does not directly address the unavailability of paired training data. Instead, it aims to develop a robust architecture capable of generalizing to the degradations present in the real-world setting, while trained using standard strategies. The SR network consists of a residual channel attention generator, visualized in Figure 9. It mainly consists of four parts: shallow feature extraction, residual channel attention feature extraction, upscale module, and reconstruction. The discriminator network is implemented using four repeated $4\times 4$ convolution layers, followed by BatchNorm and ReLU. The networks are trained in a standard GAN fashion. The generator additionally uses $L_{1}$, VGG, SSIM and gradient losses. The authors additionally used 10000 images from the ImageNet dataset for training.

Details about the proposed method can be found in

7 ITS425

This team focus on improving the SR network architecture. The image degradation operator is first learned using an improved version of the DSGAN , by using a smaller generator model than that of the original work. Unlike other methods, this team also aims to improve the quality of the target domain images. This is performed by training denoising and detail enhancement models to improve the target domain HR training images. The SR models is based on the RDN architecture . It is modified by using the add operation instead of concatenate, which not only reduces the amount of calculations of the model but also reduces the high-level information that is passed back to the final layers.

8 MLP-SR

This team follow a two stage approach. First, a DSGAN (winner of the AIM2019 RWSR challenge) network and training strategy is employed to learn the image degradation mapping. This is then used to generate paired SR training data for the second stage. The team proposes a SR architecture, shown in Figure 10, inspired by a physical image formation model. It uses a encoder-decoder structure. The inner ResNet consists of 5 residual blocks with two pre-activation Conv layers. The pre-activation is the parametrized rectified linear unit (PReLU). The trainable projection layer inside Decoder computes the proximal map with the estimated noise standard deviation and handles the data fidelity and prior terms. The noise realization is estimated in the intermediate ResNet that is sandwiched between Encoder and Decoder. The estimated residual image after Decoder is subtracted from the LR input image. Reflection padding is also used before all Conv layers to ensure slowly-varying changes at the boundaries of the input images. The generator structure can also be described as the generalization of one stage TNRD and UDNet that have good reconstruction performance for image denoising problem. For the discriminator, it employs the architecture used in SRGAN , with the relativistic loss used in ESRGAN . In addition, $L_{1}$, Total-Variation and VGG losses are used.

9 KU-ISPL

This team propose an un-paired GAN-based framework . It consists of three generators, one SR model and three discriminators. The overall architecture is visualized in Figure 11. The generators $G_{1}$, $G_{2}$, and $G_{3}$ constitute a modified CinCGAN . Residual networks are used for these architectures. $G_{3}$ further downsamples the image by a factor of $4$. The SR model is based on ESRGAN . Bilinear upsampling is introduced into the architecture to preserve details and avoid checkerboard patterns induced by the transposed convolution module. The three discriminators $D_{N}$, $D_{C}$, and $D_{Y}$ are trained with different losses: adversarial noise loss, adversarial color loss, and adversarial texture loss respectively. The $D_{N}$ uses a raw image, which contains noise signal. The $D_{C}$ and $D_{Y}$ employ a Gaussian blurred image and a grayscale image, respectively, as in WESPE . To improve performance of the discriminators, source domain images are used when the discriminators are trained. Instead of classifying real or fake, the discriminator distinguishes between source and target domain images. The generator is trained to make target domain-like fake images and the discriminator is trained to classify fake images as a source domain image. The cycle consistency and identity loss each consist of three losses: a pixel-wise $L_{1}$ loss, a VGG perceptual loss, and an SSIM loss.

10 Webbzhou

This team aims to first learn the degradation process in order to generate data for a second-stage SR network training. The degradation learning is based on the frequency separation in DSGAN . Furthermore, in order to alleviate the color shift in degradation process, the team proposed a generator based on Color Attention Residual Block (CARB) . In addition, the team modified the discrimnator of ESRGAN which treats high frequency and low frequency separately. Finally, an EdgeLoss with Canny operator is constructed to further enhance details of edge.

11 SR-DL

The team propose a joint image denoising and super-resolution model by using generative Variational AutoEncoder (dSRVAE) . It includes two parts: a Denoising AutoEncoder (DAE) and a Super-Resolution Sub-Network (SRSN). With the absence of target images, a simple discriminator is trained together with the autoencoder to encourage the SR images to pick up the desired visual pattern from the reference images. During the training, Denoising AutoEncoder (DAE) is trained first by using source image training set. Then the Super-Resolution Sub-Network (SRSN) is attached as a small head to the DAE which forms the proposed dSRVAE to output super-resolved images. Together with dSRVAE, a simple convolutional neural network is used as a discriminator to distinguish whether generated SR images are close to the original input images.

The method is visualized in Figure 12. The proposed dSRVAE network first uses the encoder to learn the latent vector of the clean image. A Gaussian model randomly samples from the latent vector to the decoder. The input noisy LR image is also included as a conditional constraint to supervise the reconstruction of the decoder. Combining both noisy image features and latent features, the decoder learn the noise pattern. Finally, the estimated clean image is obtained by subtracting the estimated noise from the input noisy image. At the second stage, Super-Resolution Sub-Network (SRSN) is added to the end of the Denoising AutoEncoder to take both bicubic interpolated original clean and estimated denoised images as input to generation superresolution result. Since there is no ground truth of super-resolved images, a discriminator is trained to distinguish the super-resolution results and cropped reference image. The balance is achieved when the discriminator cannot distinguish between reference and denoised SR image.

12 TeamAY

This team proposes a simple but strong method for unsupervised SR (SimUSR). Their approach is based on the zero-shot super-resolution (ZSSR) which trains the image-specific network at runtime using only a single given test image $\mathbf{I}_{LR}$. The ZSSR enables to optimize the model even if high-resolution images are not accessible. However, ZSSR suffers from high runtime latency and inferior performance compared to the supervised SR methods. To mitigate such issues, this team first slightly relax the constraint of ZSSR and assumes that it is relatively easy to collect the LR images, $\{\mathbf{I}_{LR_{1}},\dots,\mathbf{I}_{LR_{N}}\}$. Thanks to this assumption, they can convert fully unsupervised SR into the supervised learning regime by generating multiple pseudo-pairs $\{(\mathbf{I}^{\prime}_{LR_{1}},\mathbf{I}^{\prime}_{HR_{1}}),\dots(\mathbf{I}^{\prime}_{LR_{N}},\mathbf{I}^{\prime}_{HR_{N}})\}$ by

where $\mathbf{I}_{LR}^{son}=\mathbf{I}_{LR}\downarrow_{s,k}$ and $\mathbf{I}_{LR}^{father}=\mathbf{I}_{LR}$.

Though this is a very simple correction, their modification brings several benefits: It allows their framework to exploit every benefit of supervised learning. For instance, unlike ZSSR, their SimUSR can utilize recently developed network architectures and training techniques that provide huge performance gains. In addition, since the online (runtime) training is not necessary, SimUSR can significantly reduce its runtime latency. For the NTIRE 2020 challenge, they use pretrained RCAN (on bicubic $\times$4 scale) as a backbone model of SimUSR. Also, they attach ad-hoc denoiser (BM3D ) before train the SimUSR method. Details about the proposed method can be found in .

13 Bigfeature-Camera

This method use DSGAN to learn the degradation, used for generating paired training data. In the second stage a RNAN based SR network is trained. It is modified to handle multiple scales and by adding a contrast channel attention layers along with local attention blocks.

14 BMIPL-UNIST-YH

This method focus on how to train on unpaired data. Similar to , a CycleGAN is used to learn the degradation. In the second stage, and RCAN SR architecture is trained on generated data.

15 SVNIT1

This team combines self- and unsupervised strategies to train the SR network without supervision. For the self-supervised part, the LR input is upsampled bicubically and used for a pixel-wise loss. The unsupervised losses consist of a Total-Variation loss and a deep image quality loss. For the latter loss, a pre-trained quality assessment network was used. Details about the proposed method can be found in . Two versions of this approach was submitted:

SVNIT1-A In addition to above, this version employs an adversarial loss on the SR output. The discriminator architecture is inspired by .

SVNIT1-B Instead of a descriminator, this variant Variational Encoder which follows the architectural guidelines in .

16 SVNIT2

This method uses cyclic consistency between an SR network and a downscaling network. Two generator are trained: the SR generator going from LR to HR and the downscaling generator going from HR to LR. In addition to cycle consistency, the VGG loss, GAN loss, and a learned image quality loss is employed.

17 KU-ISPL2

This team base their approach on SRGAN . This is extended with a multi-scale convolutional block, that combines the results of convolutions with different kernel sizes.

18 SuperT

This method uses a balanced Laplacian pyramid network for progressive image super-resolution. For training, both degraded and clean images are used with standard downsampling them for training data generation.

19 GDUT-wp

This method uses an ensemble of SRResNets trained on bicubic downsampled data. The idea is that by selecting the best from an ensemble, the effect of random artifacts can be reduced.

20 MLP-SR

This method is based on the DSGAN approach. The loss of the super-resolution method consists of a VGG, GAN, TV and L1 loss. To improve the fidelity, they further used a ensemble method at test time . Details about the proposed method can be found in .

21 Relbmag-Eht

Instead of generating ‘fake’ natural image as DSGAN , this team aims to improve this method to aggregate this paring procedure into the super-resolution model. To supervise this matching from HR or bicubic images to natural images, a module with discriminators both in the LR and HR phase is proposed. It allows the downsampling model to learn from upsampling results. The ESRGAN is used as SR model.

22 QCAM

This work fine-tunes a pretrained SR model on real data using only supervision in the low-resolution. That is, it aims to minimize the loss $\min_{\theta}\|D(f_{\theta}(x))-x\|^{2}$ for source images $x$. Here, $f_{\theta}$ is the SR model with parameters $\theta$ and $D$ is the bicubic downsampling operation.

Conclusions

This paper presents the setup and results of the NTIRE 2020 challenge on real world super-resolution. Contrary to conventional super-resolution, this challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of unpaired source and target input images were provided to the participants. The source images have unknown degradations, while the target images are clean, high quality images. The challenge contains two tracks, where the goal was to super-resolve images with Image Processing artifacts (Track 1) or low-quality smart-phone images (Track 2). The challenge had in total 22 teams competing in the final step. Most of the participating were influenced AIM 2019 and demonstrated interesting and innovative solutions. Our goal is that this challenge stimulates future research in the area of unsupervised learning for image super-resolution and other similar tasks, by serving as a standard benchmark and by the establishment of new baseline methods.

Acknowledgements

We thank the NTIRE 2020 sponsors: Huawei, Oppo, Voyage81, MediaTek, DisneyResearch$\mid$Studios, and Computer Vision Lab (CVL) ETH Zurich.

Appendix A Teams and affiliations

Members: Andreas Lugmayr (andreas.lugmayr@vision.ee.ethz.ch) Martin Danelljan (martin.danelljan@vision.ee.ethz.ch) Radu Timofte (radu.timofte@vision.ee.ethz.ch)

Affiliation: Computer Vision Lab, ETH Zurich

SR-DL

Title: Generative Variational AutoEncoder for Real Image Super-Resolution

Team Leader: Zhi-Song Liu (zhi-song.liu@inria.fr)

Members: Zhi-Song Liu, LIX - Computer science laboratory at the École polytechnique [Palaiseau] Li-Wen Wang, Center of Multimedia Signal Processing, The Hong Kong Polytechnic University Marie-Paule Cani, LIX - Computer science laboratory at the École polytechnique [Palaiseau] Wan-Chi Siu, Center of Multimedia Signal Processing, The Hong Kong Polytechnic University

MSMers

Title: Cycle-based Residual Channel Attention Network for Real-World Super-Resolution

Team Leader: Fuzhi Yang (yfzcopy0702@sjtu.edu.cn)

Members: Fuzhi Yang, Shanghai Jiao Tong University, Huan Yang, Microsoft Research Beijing, P.R. China, Jianlong Fu, Microsoft Research Beijing, P.R. China,

GDUT-wp

Title: Ensemble of ResNets for Image Restoration

Team Leader: Hao Li (2111903004@mail2.gdut.edu.cn)

Members: Hao Li, Guangdong University of Technology Yukai Shi, Guangdong University of Technology Junyang Chen, Guangdong University of Technology

KU-ISPL

Title: Unsupervised Real-World Super Resolution with Cycle-in-Cycle Generative Adversarial Network and Domain-Transfer Discriminator.

Team Leader: Gwantae Kim (gtkim@ispl.korea.ac.kr)

Members: Gwantae, Kim, Intelligent Signal Processing Laboratory, Korea University Kanghyu, Lee, Intelligent Signal Processing Laboratory, Korea University Jaihyun, Park, Intelligent Signal Processing Laboratory, Korea University Junyeop, Lee, Intelligent Signal Processing Laboratory, Korea University Jeongki, Min, Intelligent Signal Processing Laboratory, Korea University Bokyeung, Lee, Intelligent Signal Processing Laboratory, Korea University Hanseok, Ko, Intelligent Signal Processing Laboratory, Korea University

TeamAY

Title: SimUSR: A Simple but Strong Baseline for Unsupervised Image Super-resolution

Team Leader: Namhyuk Ahn (aa0dfg@ajou.ac.kr)

Members: Namhyuk, Ahn, Ajou University Jaejun, Yoo, EPFL Kyung-Ah, Sohn, Ajou University

MLP-SR

Title: Deep Generative Adversarial Residual Convolutional Networks for Real-World Super-Resolution

Team Leader: Rao Muhammad Umer (engr.raoumer943@gmail.com)

Members: Rao Muhammad Umer, University of Udine, Italy. Christian Micheloni, University of Udine, Italy.

BOE-IOT-AIBD

Title: DSGAN and Triple-V MGBPv2 for Real Super-Resolution

Team Leader: Pablo Navarrete Michelini (pnavarre@boe.com.cn)

Members: Pablo, Navarrete Michelini, BOE Technology Group Co. Ltd. Fengshuo, Hu, BOE Technology Group Co. Ltd. Yanhong, Wang, BOE Technology Group Co. Ltd. Yunhua, Lu, BOE Technology Group Co. Ltd.

SuperT

Title: Fast and Balanced Laplacian Pyramid Networks for Progressive image super-resolution

Team Leader: Tongtong Zhao (daitoutiere@gmail.com)

Members: Jinjia,Peng, Dalian Maritime University Huibing,Wang, Dalian Maritime University

BIGFEATURE-CAMERA

Title: Deep Residual Mix Attention Network for Image Super-Resolution

Team Leader: Kaihua Cheng (consonwm0909@gmail.com)

Members: Kaihua Cheng, Guangdong OPPO Mobile Telecommunications Corp., Ltd Haijie Zhuo, Guangdong OPPO Mobile Telecommunications Corp., Ltd

KU-ISPL2

Title: Modular generative adversarial network based super-resolution

Team Leader: Kanghyu Lee (khlee@ispl.korea.ac.kr)

Members: Gwantae Kim is with Department of Video Information Processing, Korea University Junyeop Lee is with School of Electrical Engineering, Korea University Jeongki Min is with School of Electrical Engineering, Korea University Bokyeung Lee is with School of Electrical Engineering, Korea University Jaihyun Park is with School of Electrical Engineering, Korea University Hanseok Ko is with School of Electrical Engineering, Korea University

Impressionism

Title: Real World Super-Resolution via Kernel Estimation and Noise Injection

Team Leader: Xiaozhong Ji (shawn_ji@163.com)

Members: Xiaozhong Ji, Tencent Youtu Lab Yun Cao, Tencent Youtu Lab Ying Tai, Tencent Youtu Lab Chengjie Wang, Tencent Youtu Lab Jilin Li, Tencent Youtu Lab Feiyue Huang, Tencent Youtu Lab

Relbmag Eht

Title: Network of Aggregated Downsampler-Upsampler with Dual Resolution Domain Matching

Team Leader: Timothy Haoning Wu (1700012826@pku.edu.cn)

ITS425

Title: Addptive Residual dense block network for real image super-resolution

Team Leader: Ziyao Zong (824924664@qq.com)

Members: Ziyao Zong, North China University of Technology Shuai Liu, North China University of Technology Biao Yang, North China University of Technology

AITA-Noah

Title: Real World Super-Resolution with Iterative Domain Adaptation, CycleSR and Conditional Freqency Separation GAN

Team Leader: Ziluan Liu (liuziluan@huawei.com), Xueyi Zou (zouxueyi@huawei.com)

Members: Xing Liu, Huawei Technologies Co., Ltd Shuaijun Chen, Huawei Technologies Co., Ltd Lei Zhao, Huawei Technologies Co., Ltd Zhan Wang, Huawei Technologies Co., Ltd Yuxuan Lin, Huawei Technologies Co., Ltd Xu Jia, Huawei Technologies Co., Ltd Ziluan Liu, Huawei Technologies Co., Ltd Xueyi Zou, Huawei Technologies Co., Ltd

Webbzhou

Title: Guided Frequency Separation Network for Real World Super-Resolution

Team Leader: Yuanbo Zhou (webbozhou@gmail.com)

Members: Yuanbo Zhou,Fuzhou University Tong Tong, Fuzhou University, Imperial Vision Technology Qinquan Gao, Fuzhou University, Imperial Vision Technology Wei Deng, Imperial Vision Technology

Samsung-SLSI-MSL

Title: Real-World Super-Resolution using Generative Adversarial Networks

Team Leader: Haoyu Ren, Amin Kheradmand (co-leader) (haoyu.ren@samsung.com)

Members: Haoyu Ren SOC R&D, Samsung Semiconductor, Inc., USA Amin Kheradmand SOC R&D, Samsung Semiconductor, Inc., USA Mostafa El-Khamy SOC R&D, Samsung Semiconductor, Inc., USA Shuangquan Wang SOC R&D, Samsung Semiconductor, Inc., USA Dongwoon Bai SOC R&D, Samsung Semiconductor, Inc., USA Jungwon Lee SOC R&D, Samsung Semiconductor, Inc., USA

BMIPL-UNIST-YH-1

Title: Unpaired Domain-Adaptive Image Super-Resolution using Cycle-Consistent Adversarial Networks

Team Leader: YongHyeok Seo (syh4661@unist.ac.kr)

Members: SeYoung Chun, Ulsan national institute of science and technology

SVNIT1

Title: Unsupervised Real-World Single Image Super-Resolution (SR) using Generative Adversarial Networks (GAN) and Variational Auto-Encoders (VAE)

Team Leader: Kalpesh Prajapati (kalpesh.jp89@gmail.com)

Members: Heena, Patel, Sardar Vallabhbhai National Institute Of Technology, Surat Vishal, Chudasama, Sardar Vallabhbhai National Institute Of Technology, Surat Kishor, Upla, Norwegian University of Science and Technology, Gjøvik, Norway Raghavendra, Ramachandra, Norwegian University of Science and Technology, Gjøvik, Norway Kiran, Raja, Norwegian University of Science and Technology, Gjøvik, Norway Christoph, Busch, Norwegian University of Science and Technology, Gjøvik, Norway

SVNIT2

Title: Unsupervised Real-World Single Image Super-Resolution (SR)

Team Leader: Kalpesh Prajapati (kalpesh.jp89@gmail.com)

Members: Heena, Patel, Sardar Vallabhbhai National Institute Of Technology, Surat Vishal, Chudasama, Sardar Vallabhbhai National Institute Of Technology, Surat Kishor, Upla, Norwegian University of Science and Technology, Gjøvik, Norway Raghavendra, Ramachandra, Norwegian University of Science and Technology, Gjøvik, Norway Kiran, Raja, Norwegian University of Science and Technology, Gjøvik, Norway Christoph, Busch, Norwegian University of Science and Technology, Gjøvik, Norway

InnoPeak-SR

Title: Deep Residual Channel Attention Generative Adversarial Networks for Image Super-Resolution and Noise Reduction

Team Leader: Jie Cai (jie.cai@innopeaktech.com)

Members: Jie Cai, InnoPeak Technology Zibo Meng, InnoPeak Technology Chiu Man Ho, InnoPeak Technology

References