SRFlow: Learning the Super-Resolution Space with Normalizing Flow

Introduction

Single image super-resolution (SR) is an active research topic with several important applications. It aims to enhance the resolution of a given image by adding missing high-frequency information. Super-resolution is therefore a fundamentally ill-posed problem. In fact, for a given low-resolution (LR) image, there exist infinitely many compatible high-resolution (HR) predictions. This poses severe challenges when designing deep learning based super-resolution approaches.

Initial deep learning approaches employ feed-forward architectures trained using standard $L_{2}$ or $L_{1}$ reconstruction losses. While these methods achieve impressive PSNR, they tend to generate blurry predictions. This shortcoming stems from discarding the ill-posed nature of the SR problem. The employed $L_{2}$ and $L_{1}$ reconstruction losses favor the prediction of an average over the plausible HR solutions, leading to the significant reduction of high-frequency details. To address this problem, more recent approaches integrate adversarial training and perceptual loss functions. While achieving sharper images with better perceptual quality, such methods only predict a single SR output, which does not fully account for the ill-posed nature of the SR problem.

We address the limitations of the aforementioned approaches by learning the conditional distribution of plausible HR images given the input LR image. To this end, we design a conditional normalizing flow architecture for image super-resolution. Thanks to the exact log-likelihood training enabled by the flow formulation, our approach can model expressive distributions over the HR image space. This allows our network to learn the generation of photo-realistic SR images that are consistent with the input LR image, without any additional constraints or losses. Given an LR image, our approach can sample multiple diverse SR images from the learned distribution. In contrast to conventional methods, our network can thus explore the space of SR images (see Fig. 1).

Compared to standard Generative Adversarial Network (GAN) based SR approaches , the proposed flow-based solution exhibits a few key advantages. First, our method naturally learns to generate diverse SR samples without suffering from mode-collapse, which is particularly problematic in the conditional GAN setting . Second, while GAN-based SR networks require multiple losses with careful parameter tuning, our network is stably trained with a single loss: the negative log-likelihood. Third, the flow network employs a fully invertible encoder, capable of mapping any input HR image to the latent flow-space and ensuring exact reconstruction. This allows us to develop powerful image manipulation techniques for editing the predicted SR or any existing HR image.

Contributions: We propose SRFlow, a flow-based super-resolution network capable of accurately learning the distribution of realistic HR images corresponding to the input LR image. In particular, the main contributions of this work are as follows: (i) We are the first to design a conditional normalizing flow architecture that achieves state-of-the-art super-resolution quality. (ii) We harness the strong HR distribution learned by SRFlow to develop novel techniques for controlled image manipulation and editing. (iii) Although only trained for super-resolution, we show that SRFlow is capable of image denoising and restoration. (iv) Comprehensive experiments for face and general image super-resolution show that our approach outperforms state-of-the-art GAN-based methods for both perceptual and reconstruction-based metrics.

Related Work

Single image SR: Super-resolution has long been a fundamental challenge in computer vision due to its ill-posed nature. Early learning-based methods mainly employed sparse coding based techniques or local linear regression . The effectiveness of example-based deep learning for super-resolution was first demonstrated by SRCNN , which further led to the development of more effective network architectures . However, these methods do not reproduce the sharp details present in natural images due to their reliance on $L_{2}$ and $L_{1}$ reconstruction losses. This was addressed in URDGN , SRGAN and more recent approaches by adopting a conditional GAN based architecture and training strategy. While these works aim to predict one example, we undertake the more ambitious goal of learning the distribution of all plausible reconstructions from the natural image manifold.

Stochastic SR: The problem of generating diverse super-resolutions has received relatively little attention. This is partly due to the challenging nature of the problem. While GANs provide an method for learning a distribution over data , conditional GANs are known to be extremely susceptible to mode collapse since they easily learn to ignore the stochastic input signal . Therefore, most conditional GAN based approaches for super-resolution and image-to-image translation resort to purely deterministic mappings . A few recent works address GAN-based stochastic SR by exploring techniques to avoid mode collapse and explicitly enforcing low-resolution consistency. In contrast to those works, we design a flow-based architecture trained using the negative log-likelihood loss. This allows us to learn the conditional distribution of HR images, without any additional constraints, losses, or post-processing techniques to enforce low-resolution consistency. A different line of research exploit the internal patch recurrence by only training the network on the input image itself. Recently employed this strategy to learn a GAN capable of stochastic SR generation. While this is an interesting direction, our goal is to exploit large image datasets to learn a general distribution over the image space.

Normalizing flow: Generative modelling of natural images poses major challenges due to the high dimensionality and complex structure of the underlying data distribution. While GANs have been explored for several vision tasks, Normalizing Flow based models have received much less attention. These approaches parametrize a complex distribution $p_{\mathbf{y}}(\mathbf{y}|{\boldsymbol{\theta}})$ using an invertible neural network $f_{\boldsymbol{\theta}}$, which maps samples drawn from a simple (e.g. Gaussian) distribution $p_{\mathbf{z}}(\mathbf{z})$ as $\mathbf{y}=f^{-1}_{\boldsymbol{\theta}}(\mathbf{z})$. This allows the exact negative log-likelihood $-\log p_{\mathbf{y}}(\mathbf{y}|{\boldsymbol{\theta}})$ to be computed by applying the change-of-variable formula. The network can thus be trained by directly minimizing the negative log-likelihood using standard SGD-based techniques. Recent works have investigated conditional flow models for point cloud generation as well as class and image conditional generation of images. The latter works adapt the widely successful Glow architecture to conditional image generation by concatenating the encoded conditioning variable in the affine coupling layers . The concurrent work consider the SR task as an example application, but only addressing $2\times$ magnification and without comparisons with state-of-the-art GAN-based methods. While we also employ the conditional flow paradigm for its theoretically appealing properties, our work differs from these previous approaches in several aspects. Our work is first to develop a conditional flow architecture for SR that provides favorable or superior results compared to state-of-the-art GAN-based methods. Second, we develop powerful flow-based image manipulation techniques, applicable for guided SR and to editing existing HR images. Third, we introduce new training and architectural considerations. Lastly, we demonstrate the generality and strength of our learned image posterior by applying SRFlow to image restoration tasks, unseen during training.

Proposed Method: SRFlow

We formulate super-resolution as the problem of learning a conditional probability distribution over high-resolution images, given an input low-resolution image. This approach explicitly addresses the ill-posed nature of the SR problem by aiming to capture the full diversity of possible SR images from the natural image manifold. To this end, we design a conditional normalizing flow architecture, allowing us to learn rich distributions using exact log-likelihood based training.

The goal of super-resolution is to predict higher-resolution versions $\mathbf{y}$ of a given low-resolution image $\mathbf{x}$ by generating the absent high-frequency details. While most current approaches learn a deterministic mapping $\mathbf{x}\mapsto\mathbf{y}$, we aim to capture the full conditional distribution $p_{\mathbf{y}|\mathbf{x}}(\mathbf{y}|\mathbf{x},{\boldsymbol{\theta}})$ of natural HR images $\mathbf{y}$ corresponding to the LR image $\mathbf{x}$. This constitutes a more challenging task, since the model must span a variety of possible HR images, instead of just predicting a single SR output. Our intention is to train the parameters ${\boldsymbol{\theta}}$ of the distribution in a purely data-driven manner, given a large set of LR-HR training pairs $\{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{M}$.

The core idea of normalizing flow is to parametrize the distribution $p_{\mathbf{y}|\mathbf{x}}$ using an invertible neural network $f_{{\boldsymbol{\theta}}}$. In the conditional setting, $f_{{\boldsymbol{\theta}}}$ maps an HR-LR image pair to a latent variable $\mathbf{z}=f_{{\boldsymbol{\theta}}}(\mathbf{y};\mathbf{x})$. We require this function to be invertible w.r.t. the first argument $\mathbf{y}$ for any LR image $\mathbf{x}$. That is, the HR image $\mathbf{y}$ can always be exactly reconstructed from the latent encoding $\mathbf{z}$ as $\mathbf{y}=f_{{\boldsymbol{\theta}}}^{-1}(\mathbf{z};\mathbf{x})$. By postulating a simple distribution $p_{\mathbf{z}}(\mathbf{z})$ (e.g. a Gaussian) in the latent space $\mathbf{z}$, the conditional distribution $p_{\mathbf{y}|\mathbf{x}}(\mathbf{y}|\mathbf{x},{\boldsymbol{\theta}})$ is implicitly defined by the mapping $\mathbf{y}=f_{{\boldsymbol{\theta}}}^{-1}(\mathbf{z};\mathbf{x})$ of samples $\mathbf{z}\sim p_{\mathbf{z}}$. The key aspect of normalizing flows is that the probability density $p_{\mathbf{y}|\mathbf{x}}$ can be explicitly computed as,

It is derived by applying the change-of-variables formula for densities, where the second factor is the resulting volume scaling given by the determinant of the Jacobian $\frac{\partial f_{{\boldsymbol{\theta}}}}{\partial\mathbf{y}}$. The expression (1) allows us to train the network by minimizing the negative log-likelihood (NLL) for training samples pairs $(\mathbf{x},\mathbf{y})$,

HR image samples $\mathbf{y}$ from the learned distribution $p_{\mathbf{y}|\mathbf{x}}(\mathbf{y}|\mathbf{x},{\boldsymbol{\theta}})$ are generated by applying the inverse network $\mathbf{y}=f_{{\boldsymbol{\theta}}}^{-1}(\mathbf{z};\mathbf{x})$ to random latent variables $\mathbf{z}\sim p_{\mathbf{z}}$.

In order to achieve a tractable expression of the second term in (2), the neural network $f_{{\boldsymbol{\theta}}}$ is decomposed into a sequence of $N$ invertible layers $\mathbf{h}^{n+1}=f_{{\boldsymbol{\theta}}}^{n}(\mathbf{h}^{n};g_{{\boldsymbol{\theta}}}(\mathbf{x}))$, where $\mathbf{h}^{0}=\mathbf{y}$ and $\mathbf{h}^{N}=\mathbf{z}$. We let the LR image to first be encoded by a shared deep CNN $g_{{\boldsymbol{\theta}}}(\mathbf{x})$ that extracts a rich representation suitable for conditioning in all flow-layers, as detailed in Sec. 3.3. By applying the chain rule along with the multiplicative property of the determinant , the NLL objective in (2) can be expressed as

We thus only need to compute the log-determinant of the Jacobian $\frac{\partial f_{{\boldsymbol{\theta}}}^{n}}{\partial\mathbf{h}^{n}}$ for each individual flow-layer $f_{{\boldsymbol{\theta}}}^{n}$. To ensure efficient training and inference, the flow layers $f_{{\boldsymbol{\theta}}}^{n}$ thus need to allow efficient inversion and a tractable Jacobian determinant. This is further discussed next, where we detail the employed conditional flow layers $f_{{\boldsymbol{\theta}}}^{n}$ in our SR architecture. Our overall network architecture for flow-based super-resolution is depicted in Fig. 2.

2 Conditional Flow Layers

The design of flow-layers $f_{{\boldsymbol{\theta}}}^{n}$ requires care in order to ensure a well-conditioned inverse and a tractable Jacobian determinant. This challenge was first addressed in and has recently spurred significant interest . We start from the unconditional Glow architecture , which is itself based on the RealNVP . The flow layers employed in these architectures can be made conditional in a straight-forward manner . We briefly review them here along with our introduced Affine Injector layer.

Conditional Affine Coupling: The affine coupling layer provides a simple and powerful strategy for constructing flow-layers that are easily invertible. It is trivially extended to the conditional setting as follows,

Here, $\mathbf{h}^{n}=(\mathbf{h}^{n}_{A},\mathbf{h}^{n}_{B})$ is a partition of the activation map in the channel dimension. Moreover, $\mathbf{u}$ is the conditioning variable, set to the encoded LR image $\mathbf{u}=g_{{\boldsymbol{\theta}}}(\mathbf{x})$ in our work. Note that $f_{{\boldsymbol{\theta}},\text{s}}^{n}$ and $f_{{\boldsymbol{\theta}},\text{b}}^{n}$ represent arbitrary neural networks that generate the scaling and bias of $\mathbf{h}^{n}_{B}$. The Jacobian of (4) is triangular, enabling the efficient computation of its log-determinant as $\sum_{ijk}f_{{\boldsymbol{\theta}},\text{s}}^{n}(\mathbf{h}^{n}_{A};\mathbf{u})_{ijk}$.

Invertible $1\times 1$ Convolution: General convolutional layers are often intractable to invert or evaluate the determinant of. However, demonstrated that a $1\times 1$ convolution $\mathbf{h}^{n+1}_{ij}=W\mathbf{h}^{n}_{ij}$ can be efficiently integrated since it acts on each spatial coordinate $(i,j)$ independently, which leads to a block-diagonal structure. We use the non-factorized formulation in .

Actnorm: This provides a channel-wise normalization through a learned scaling and bias. We keep this layer in its standard un-conditional form .

Squeeze: It is important to process the activations at different scales in order to capture correlations and structures over larger distances. The squeeze layer provides an invertible means to halving the resolution of the activation map $\mathbf{h}^{n}$ by reshaping each spatial $2\times 2$ neighborhood into the channel dimension.

Affine Injector: To achieve more direct information transfer from the low-resolution image encoding $\mathbf{u}=g_{{\boldsymbol{\theta}}}(\mathbf{x})$ to the flow branch, we additionally introduce the affine injector layer. In contrast to the conditional affine coupling layer, our affine injector layer directly affects all channels and spatial locations in the activation map $\mathbf{h}^{n}$. This is achieved by predicting an element-wise scaling and bias using only the conditional encoding $\mathbf{u}$,

Here, $f_{{\boldsymbol{\theta}},\text{s}}$ and $f_{{\boldsymbol{\theta}},\text{s}}$ can be any network. The inverse of (5) is trivially obtained as $\mathbf{h}^{n}=\exp(-f_{{\boldsymbol{\theta}},\text{s}}^{n}(\mathbf{u}))\cdot(\mathbf{h}^{n+1}-f_{{\boldsymbol{\theta}},\text{b}}^{n}(\mathbf{u}))$ and the log-determinant is given by $\sum_{ijk}f_{{\boldsymbol{\theta}},\text{s}}^{n}(\mathbf{u})_{ijk}$. Here, the sum ranges over all spatial $i,j$ and channel indices $k$.

3 Architecture

Our SRFlow architecture, depicted in Fig. 2, consists of the invertible flow network $f_{{\boldsymbol{\theta}}}$ and the LR encoder $g_{{\boldsymbol{\theta}}}$. The flow network is organized into $L$ levels, each operating at a resolution of $\frac{H}{2^{l}}\times\frac{W}{2^{l}}$, where $l\in\{1,\ldots,L\}$ is the level number and $H\times W$ is the HR resolution. Each level itself contains $K$ number of flow-steps.

Flow-step: Each flow-step in our approach consists of four different layers, as visualized in Fig. 2. The Actnorm if applied first, followed by the $1\times 1$ convolution. We then apply the two conditional layers, first the Affine Injector followed by the Conditional Affine Coupling.

Level transitions: Each level first performs a squeeze operation that effectively halves the spatial resolution. We observed that this layer can lead to checkerboard artifacts in the reconstructed image, since it is only based on pixel re-ordering. To learn a better transition between the levels, we therefore remove the conditional layers first few flow steps after the squeeze (see Fig. 2). This allows the network to learn a linear invertible interpolation between neighboring pixels. Similar to , we split off $50\%$ of the channels before the next squeeze layer. Our latent variables $(z_{l})_{l=1}^{L}$ thus model variations in the image at different resolutions, as visualized in Fig. 2.

Low-resolution encoding network $g_{{\boldsymbol{\theta}}}$: SRFlow allows for the use of any differentiable architecture for the LR encoding network $g_{{\boldsymbol{\theta}}}$, since it does not need to be invertible. Our approach can therefore benefit from the advances in standard feed-forward SR architectures. In particular, we adopt the popular CNN architecture based on Residual-in-Residual Dense Blocks (RRDB) , which builds upon . It employs multiple residual and dense skip connections, without any batch normalization layers. We first discard the final upsampling layers in the RRDB architecture since we are only interested in the underlying representation and not the SR prediction. In order to capture a richer representation of the LR image at multiple levels, we additionally concatenate the activations after each RRDB block to form the final output of $g_{{\boldsymbol{\theta}}}$.

Details: We employ $K=16$ flow-steps at each level, with two additional unconditional flow-steps after each squeeze layer (discussed above). We use $L=3$ and $L=4$ levels for SR factors $4\times$ and $8\times$ respectively. For general image SR, we use the standard 23-block RRDB architecture for the LR encoder $g_{{\boldsymbol{\theta}}}$. For faces, we reduce to 8 blocks for efficiency. The networks $f_{{\boldsymbol{\theta}},\text{s}}^{n}$ and $f_{{\boldsymbol{\theta}},\text{b}}^{n}$ in the conditional affine coupling (4) and the affine injector (5) are constructed using two shared convolutional layers with ReLU, followed by a final convolution.

4 Training Details

We train our entire SRFlow network using the negative log-likelihood loss (3). We sample batches of 16 LR-HR image pairs $(\mathbf{x},\mathbf{y})$. During training, we use an HR patch size of $160\times 160$. As optimizer we use Adam with a starting learning rate of $5\cdot 10^{-4}$, which is halved at $50\%,75\%,90\%$ and $95\%$ of the total training iterations. To increase training efficiency, we first pre-train the LR encoder $g_{{\boldsymbol{\theta}}}$ using an $L_{1}$ loss for $200$k iterations. We then train our full SRFlow architecture using only the loss (3) for $200$k iterations. Our network takes 5 days to train on a single NVIDIA V100 GPU. Further details are provided in the appendix.

Datasets: For face super-resolution, we use the CelebA dataset. Similar to , we pre-process the dataset by cropping aligned patches, which are resized to the HR resolution of $160\times 160$. We employ the full train split ($160$k images). For general SR, we use the same training data as ESRGAN , consisting of the train split of 800 DIV2k along with 2650 images from Flickr2K. The LR images are constructed using the standard MATLAB bicubic kernel.

Applications and Image Manipulations

The distribution $p_{\mathbf{y}|\mathbf{x}}(\mathbf{y}|\mathbf{x},{\boldsymbol{\theta}})$ learned by our SRFlow can be explored by sampling different SR predictions as $\mathbf{y}^{(i)}=f_{{\boldsymbol{\theta}}}^{-1}(\mathbf{z}^{(i)};\mathbf{x}),\,\mathbf{z}^{(i)}\!\sim p_{\mathbf{z}}$ for a given LR image $\mathbf{x}$. As commonly observed for flow-based models, the best results are achieved when sampling with a slightly lower variance . We therefore use a Gaussian $\mathbf{z}^{(i)}\!\sim\mathcal{N}(0,\tau)$ with variance $\tau$ (also called temperature). Results are visualized in Fig. 4 for $\tau=0.8$. Our approach generates a large variety of SR images, including differences in e.g. hair and facial attributes, while preserving consistency with the LR image. Since our latent variables $\mathbf{z}_{ijkl}$ are spatially localized, specific parts can be re-sampled, enabling more controlled interactive editing and exploration of the SR image.

2 LR-Consistent Style Transfer

3 Latent Space Normalization

4 Image Content Transfer

5 Image Restoration

Experiments

We perform comprehensive experiments for super-resolution of faces and of generic images in comparisons with current state-of-the-art and an ablative analysis. Applications, such as image manipulation tasks, are presented in Sec. 4, with additional results, analysis and visuals in the appendix.

Evaluation Metrics: To evaluate the perceptual distance to the Ground Truth, we report the default LPIPS . It is a learned distance metric, based on the feature-space of a finetuned AlexNet. We report the standard fidelity oriented metrics, Peak Signal to Noise Ratio (PSNR) and structural similarity index (SSIM) , although they are known to not correlate well with the human perception of image quality . Furthermore, we report the no-reference metrics NIQE , BRISQUE and PIQUE . In addition to visual quality, consistency with the LR image is an important factor. We therefore evaluate this aspect by reporting the LR-PSNR, computed as the PSNR between the downsampled SR image and the original LR image.

We evaluate SRFlow for face SR ($8\times$) using 5000 images from the test split of the CelebA dataset. We compare with bicubic, RRDB , ESRGAN , and ProgFSR . While the latter two are GAN-based, RRDB is trained using only $L_{1}$ loss. ProgFSR is a very recent SR method specifically designed for faces, shown to outperform several prior face SR approaches in . It is trained on the full train split of CelebA, but using a bilinear kernel. For fair comparison, we therefore separately train and evaluate SRFlow on the same kernel.

Results are provided in Tab. 1 and Fig. 7. Since our aim is perceptual quality, we consider LPIPS the primary metric, as it has been shown to correlate much better with human opinions . SRFlow achieves more than twice as good LPIPS distance compared to RRDB, at the cost of lower PSNR and SSIM scores. As seen in the visual comparisons in Fig. 7, RRDB generates extremely blurry results, lacking natural high-frequency details. Compared to the GAN-based methods, SRFlow achieves significantly better results in all reference metrics. Interestingly, even the PSNR is superior to ESRGAN and ProgFSR, showing that our approach preserves fidelity to the HR ground-truth, while achieving better perceptual quality. This is partially explained by the hallucination artifacts that often plague GAN-based approaches, as seen in Fig. 7. Our approach generate sharp and natural images, while avoiding such artifacts. Interestingly, our SRFlow achieves an LR-consistency that is even better than the fidelity-trained RRDB, while the GAN-based methods are comparatively in-consistent with the input LR image.

2 General Super-Resolution

Next, we evaluate our SRFlow for general SR on the DIV2K validation set. We compare SRFlow to bicubic, EDSR , RRDB , ESRGAN , and RankSRGAN . Except for EDSR, which used DIV2K, all methods including SRFlow are trained on the train splits of DIV2K and Flickr2K (see Sec. 3.3). For the $4\times$ setting, we employ the provided pre-trained models. Due to lacking availability, we trained RRDB and ESRGAN for $8\times$ using the authors’ code.

EDSR and RRDB are trained using only reconstruction losses, thereby achieving inferior results in terms of the perceptual LPIPS metric (Tab. 2). Compared to the GAN-based methods , our SRFlow achieves significantly better PSNR, LPIPS and LR-PSNR and favorable results in terms of PIQUE and BRISQUE. Visualizations in Fig. 8 confirm the perceptually inferior results of EDSR and RRDB, which generate little high-frequency details. In contrast, SRFlow generates rich details, achieving favorable perceptual quality compared to ESRGAN. The first row, ESRGAN generates severe discolored artifacts and ringing patterns at several locations in the image. We find SRFlow to generate more stable and consistent results in these circumstances.

3 Ablative Study

To ablate the depth and width, we train our network with different number of flow-steps $K$ and hidden layers in two conditional layers (9) and (5) respectively. Figure 9 shows results on the CelebA dataset. Decreasing the number of flow-steps $K$ leads to more artifacts in complex structures, such as eyes. Similarly, a larger number of channels leads to better consistency in the reconstruction. In Tab. 9 we analyze architectural choices. The Affine Image Injector increases the fidelity, while preserving the perceptual quality. We also observe the transitional linear flow steps (Sec. 3.3) to be beneficial.

Conclusion

We propose a flow-based method for super-resolution, called SRFlow. Contrary to conventional methods, our approach learns the distribution of photo-realistic SR images given the input LR image. This explicitly accounts for the ill-posed nature of the SR problem and allows for the generation of diverse SR samples. Moreover, we develop techniques for image manipulation, exploiting the strong image posterior learned by SRFlow. In comprehensive experiments, our approach achieves improved results compared to state-of-the-art GAN-based approaches.

Acknowledgements: This work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, a Google GCP grant, an Amazon AWS grant, and an Nvidia GPU grant.

References

A Architecture Details

In this section, we give additional details about our SRFlow architecture. The construction of a flow-based architecture requires the flow layers to be invertible and have a tractable Jacobian log-determinant. Since super-resolution of diverse images has to be able to cope with different input sizes, we also ensure that our architecture is fully convolutional. We can therefore train our network on smaller patches, and directly apply it to the full image during testing. The computational time of our approach is $1.13$ seconds for super-resolving one $256\times 256$ input LR image with a scale factor of $4\times$ on an Nvidia V100 GPU.

Our SRFlow network is conditioned on the encoding of the low-resolution image $\mathbf{u}=g_{{\boldsymbol{\theta}}}(\mathbf{x})$. To this end, we employ the RRDB-based architecture, described in the paper. It employs several RRDB-blocks with a channel dimension of 64, operating in the resolution of the input LR image. The final conditioning output $\mathbf{u}=g_{{\boldsymbol{\theta}}}(\mathbf{x})$ is achieved by concatenating the activations from 5 equally spaced RRDB blocks, resulting in a dimensionality of 320.

A.2 The Affine Injector Layer

Our affine injector layer provide a direct means of conditioning all dimensions of the flow feature-map $\mathbf{h}^{n}$ on the LR encoding as,

The scale and bias are extracted using non-invertible networks $f_{{\boldsymbol{\theta}},\text{s}}(\mathbf{u})$ and $f_{{\boldsymbol{\theta}},\text{b}}(\mathbf{u})$ respectively. The input $\mathbf{u}$ is first bilinearly resized to the resolution of the corresponding flow-level. A conv-ReLU block first reduces the dimensionality to 64. Another conv-ReLU block is then applied with 64-dimensional output. The output of $f_{{\boldsymbol{\theta}},\text{s}}(\mathbf{u})$ and $f_{{\boldsymbol{\theta}},\text{b}}(\mathbf{u})$ are then achieved by two separate conv-layers applied to the same 64-dimensional input. For these layers, we employ the zero-initialization strategy proposed in . All convolutions have a $3\times 3$ kernel.

A.3 Conditional Affine Coupling

This building block allows applying complex unconstrained conditional learned functions that act on the normalizing flow, without harming its invertibility. This is made possible by bypassing half of the activations and applying an affine transformation to the other half . This transformation depends on the bypassed half $\mathbf{h}^{n}_{A}$ and conditional features $\mathbf{u}$ as,

This expression can be easily inverted . The network architectures of $f_{{\boldsymbol{\theta}},\text{s}}$ and $f_{{\boldsymbol{\theta}},\text{b}}$ are similar to those of the Affine Injector, described above. The only difference is that the two inputs $\mathbf{h}^{n}_{A}$ and $\mathbf{u}$ are initially concatenated after $\mathbf{u}$ is resized to the resolution of $\mathbf{h}^{n}_{A}$.

A.4 Squeeze Operation

This layer reshapes the activation map to half the width and height. In order to preserve the locality, neighboring pixels are stacked as seen in Figure 10.

A.5 Activation Norm

The Activation Norm (Actnorm) is a normalization layer. Unlike Batchnorm, it does not require synchronization among the elements of a batch. It simply consists of a learned scaling and bias factor for each dimension of the feature map. Thus it helps distributed learning on multiple GPUs.

B Training Details

In this section, we give additional details about the training procedure for our SRFlow. We employ the Adam optimizer with a starting learning rate of $5\cdot 10^{-4}$. This learning rate is halved at $50\%,75\%,90\%$ and $95\%$ of the total number of training iterations. During the first $50\%$ of the training iterations, the pre-trained weights of the LR encoder $g_{{\boldsymbol{\theta}}}$ are frozen in a warm-up phase. In the latter $50\%$, all parameters of the SRFlow network, including $g_{{\boldsymbol{\theta}}}$, are optimized jointly with the same learning rate.

As has been observed in e.g. , adding slight random noise to the target image helps the training process and leads to better visual results. We therefore add Gaussian noise with a standard deviation of $\sigma=\frac{4}{\sqrt{3}}$ to the high-resolution image. In contrast to , we do not employ 5-bit quantization.

C Detailed Quantitative Analysis

In this section, we provide additional quantitative analysis of our approach.

Here, we analyze the impact of the sampling temperature $\tau$ used during inference. It controls the variance of the Gaussian latent variable used when sampling SR images as $\mathbf{y}=f_{{\boldsymbol{\theta}}}^{-1}(\mathbf{z};\mathbf{x}),\,\mathbf{z}\sim\mathcal{N}(0,\tau)$. As described in Section 4.1 of the main paper, a slightly reduced temperature $\tau<1$, increases the image quality. When further decreasing the temperature to $\tau=0$, the sampling process becomes deterministic. We analyze the effect of the sampling temperature $\tau$ on the main performance metrics, and on the sampling diversity itself. Results are shown in Figures 12, 13 and 14. A temperature $\tau=0$ generates predictions with high fidelity, in terms of PSNR and SSIM. However, the results are blurry, as seen in Figure 11, explaining the poor perceptual quality (LPIPS) for this setting. Increasing the temperature leads to a drastic improvements in perceptual quality in terms of LPIPS distance. This is also clearly seen in the visual results in Figure 11. We also plot how the sampling diversity improves with increased temperature $\tau$ in terms of pixel-wise variance.

C.2 Perception–Distortion analysis

Here, we analyze the perception–distortion trade-off provided by our SRFlow. This trade off is an important choice decision for super-resolution methods . While most techniques do not allow to influence the super-resolution process during inference, SRFlow provides an effective means of controlling this trade-off using the sampling temperature $\tau$. We analyze this by plotting the perceptual quality (LPIPS) vs. the distortion (PSNR) with respect to the ground-truth in Figure 15. We plot the results for different $\tau$ for SRFlow. Our approach provides different alternative trade-offs. It achieves similar PSNR compared to the $L_{1}$-loss trained RRDB for $\tau=0$. On the other hand, SRFlow provides similar or better perceptual quality compared to ESRGAN for $\tau\geq 0.8$, while achieving superior fidelity (PSNR).

C.3 Impact of LR-Encoder Initialization

To efficiently compare different variants of SRFlow, we reduced training time by pretraining the LR-Encoder $g_{{\boldsymbol{\theta}}}$. As shown in Table 4, the perceptual quality is comparable, while the fidelity is slightly higher, compared to using a randomly initalized LR-Encoder. The default SRFlow network was trained for 200k steps and uses a pretrained LR-Encoder, which was trained for 200k steps. The model without pretraining was trained for 300k iterations to make up for the missing pretraining. Since the main bottleneck during training is the calculation of the log determinant, this reduces training time.

C.4 Oracle Analysis of the Sampling Space

As opposed to other state-of-the-art super-resolution approaches, SRFlow can be used to sample many variants of plausible super-resolutions. To further demonstrate the potential of this property, we analyze the performance of our SRFlow when selecting the best result among $n$ random samples. Results, using a sampling temperature of $\tau=0.8$, are shown in Figure 17. The results are computed over the full CelebA test set of 5000 images. The best result w.r.t. the ground-truth in each plot is selected based on the corresponding performance metric for $n=1,\ldots,10$ samples. This results shows that the perceptual quality in particular benefits from the oracle selection. This might be explained by our temperature setting, which forces the model to prefer perceptual quality over fidelity. It demonstrates that SRFlow provides a rich and diverse space of super-resolved images, from which solutions can be sampled. It provides the opportunity for improving the predictions of SRFlow by rejecting lower quality samples. A visual example is shown in Figure 16, when selecting the best out of $n$ samples using the LPIPS distance.

C.5 Image Restoration

We provide additional quantitative and qualitative results for image restoration, described in Section 4.5. Table 5 shows quantitative results for the task of image denoising when using white Gaussian noise with standard deviation $\sigma=20$. We report performance metrics w.r.t. the clean ground-truth for the original noisy image, when just super-resolving the down-sampled image, and when using our restoration approach based on latent space normalization, as described in Section 4.5. Despite only being trained for the task of super-resolving clean images, our approach provides promising results for image denoising. This demonstrates the strong image posterior learned by our SRFlow. We show visual examples on CelebA and DIV2K in Figure 18 and Figure 19 respectively.

D Visual Results

In this section, we provide additional visual results.

Additional examples that compare SRFlow with state-of-the-art for face super-resolution on CelebA are shown in Figure 20. For fair comparison, we also show SRFlow results when trained and applied on the same bilinear downsampling kernel as ProgFSR . Our approach provides superior perceptual quality and better fidelity compared to the GAN-based methods.

D.2 State-of-the-Art General Super-Resolution

We provide more visual examples for the experiments on DIV2K, comparing SRFlow with with state-of-the-art super-resolution methods. In Figure 21 illustrates results for $4\times$. In addition, we provide results for DIV2K $8\times$ in Figure 22. SRFlow achieves perceptual quality similar or better than ESRGAN in most cases. Moreover, our approach do not suffer from the hallucination artifacts typically seen in GAN-based methods.

D.3 Stochastic Face Super-Resolution

Here we provide additional examples to show the variety when sampling SR images with our default temperature $\tau=0.8$ for CelebA. As seen for $8\times$ super-resolution sampling in Figure 23, the low resolution image still contains significant information about facial characteristics. This bounds the diversity of super-resolution in order to be consistent. On the other hand in Figure 24 we show $16\times$ super-resolution which is much more free while still being consistent to the low-resolution. Therefore one can observe a much higher variety.

D.4 Stochastic General Super-Resolution

In analogy to the visual sampling experiments for CelebA, we show results for the same procedure applied to DIV2K. An example for the variety of upscaling factor $4\times$ is shown in Figure 25. For example, one can observe that the door in the lower right sometimes looks more like an archway and other examples more square. In addition we show the results for $8\times$ upsampling in Figure 26. There it can be observed that the texture of the stones varies from being smooth to being rough.

D.5 Image Content Transfer

Additional examples for image content transfer are depicted in Figure 27. For this task we trained SRFlow with random shifts of 4px in HR to obtain a higher flexibility.