Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

Introduction

Single image super-resolution aims at recovering a high-resolution (HR) image from a single low-resolution (LR) one. The problem is ill-posed since a multiplicity of solutions exist for any given low-resolution pixel. To overcome this problem, contemporary methods such as those based on deep convolutional neural networks constrain the solution space through learning mapping functions from external low- and high-resolution exemplar pairs. To push the solution closer to the natural manifold, new losses are proposed to replace the conventional pixel-wise mean squared error (MSE) loss that tends to encourage blurry and overly-smoothed results. Specifically, perceptual loss is introduced to optimize a super-resolution model in a feature space instead of pixel space. Ledig et al. and Sajjadi et al. further propose adversarial loss to encourage the network to favor solutions that look more like natural images. With these loss functions the overall visual quality of reconstruction is significantly improved.

Though great strides have been made, texture recovery in SR remains an open problem. Examples are shown in Fig. 1. A variety of different HR patches could have very similar LR counterparts, as shown by the building and plant examples. Generating realistic textures faithful to the inherent class is non-trivial. The results obtained by using perceptual and adversarial losses (without prior) do add fine details to the reconstructed HR image. But if we examine closely, these details are not reminiscent of the textures one would usually observe. Without stronger prior information, existing methods struggle in distinguishing these LR patches and restoring natural and realistic textures thereon.

We believe that the categorical prior, which characterizes the semantic class of a region in an image (e.g., sky, building, plant), is crucial for constraining the plausible solution space in SR. We demonstrate the effectiveness of categorical prior using the same example in Fig. 1. Specifically, we try to restore the visually ambiguous plant and building pairs using two different CNN models, each of which is specially trained on a plant dataset and a building dataset. It is observed that generating realistic textures faithful to the inherent class can be better achieved by selecting the correct class-dedicated model. This phenomenon is previously documented by Timofte et al. . They train specialized models for each semantic category on exemplar-based methods and show that SR results can be improved by semantic priors.

In this study, we wish to investigate class-conditional image super-resolution with CNN. This problem is challenging especially when multiple segments of different classes and sizes co-exist in a single image. No previous work has investigated how categorical priors can be obtained and further incorporated into the reconstruction process. We make this attempt by exploring the possibility of using semantic segmentation maps as the categorical prior. Our experiments embrace this choice and show that segmentation maps encapsulate rich categorical prior up to pixel level. In addition, semantic segmentation results on LR images are satisfactory given a contemporary CNN that is fine-tuned on LR images. The remaining question is to find a formulation that allows factorized texture generation in an SR network conditioned on segmentation maps. This is a non-trivial problem. Training a separate SR model for each semantic class is neither scalable nor computationally efficient. Combining LR images with segmentation maps as inputs, or concatenating segmentation maps with intermediate deep features cannot make an effective use of segmentation.

In this work, we present a novel approach known as Spatial Feature Transform (SFT) that is capable of altering the behavior of an SR network through just transforming the features of some intermediate layers of the network. Specifically, an SFT layer is conditioned on semantic segmentation probability maps, based on which it generates a pair of modulation parameters to apply affine transformation spatially on feature maps of the network. The advantages of SFT are three-fold: (1) It is parameter-efficient. Reconstruction of an HR image with rich semantic regions can be achieved with just a single forward pass through transforming the intermediate features of a single network. (2) SFT layers can be easily introduced to existing SR network structures. The layers can be trained end-to-end together with the SR network using conventional loss functions. (3) It is extensible. While we consider categorical prior in our study, other priors such as depth maps can also be applied using the proposed SFT layer. We demonstrate the effectiveness of our approach, named as SFT-GAN, in Fig. 2. More results, a user study, and ablation experiments are provided in Sec. 4.

Related Work

Single image super-resolution. Many studies have introduced prior information to help address the ill-posed SR problem. Early methods explore a smoothing prior such as bicubic interpolation and Lanczos resampling . Image priors such as edge features , statistics and internal patch recurrence are employed to improve performance. Dong et al. train domain specific dictionaries to better recover local structures in a sparse representation framework. Sun et al. propose context-constrained super-resolution by learning from texturally similar training segments. Timofte et al. investigate semantic priors by training specialized models separately for each semantic category on exemplar-based methods . In contrast to these studies, we explore categorical priors in the form of segmentation probability maps in a CNN framework.

Contemporary SR algorithms are mostly learning-based methods, including neighbor embedding , sparse coding and random forest . As an instantiation of learning-based methods, Dong et al. propose SRCNN for learning the mapping of LR and HR images in an end-to-end manner. Later on, the field has witnessed a variety of network architectures, such as a deeper network with residual learning , Laplacian pyramid structure , residual blocks , recursive learning , and densely connected network . Multi-scale guidance structure has also been proposed for depth map super-resolution . Different losses have also been proposed. Pixel-wise loss functions, like MSE and Charbonnier penalty , encourage the network to find an average of many plausible solutions and lead to overly-smooth results. Perceptual losses are proposed to enhance the visual quality by minimizing the error in a feature space. Ledig et al. introduce an adversarial loss, generating images with more natural details. Sajjadi et al. develop a similar approach and further explore the local texture matching loss, partly reducing visually unpleasant artifacts. We use the same losses but encourage the network to find solutions under the categorical priors. Enforcing category-specific priors in CNN has been attempted in Xu et al. but they only focus on two classes of images, i.e., faces and text. Prior is assumed at image-level rather than pixel-level. We take a further step to assume multiple categorical classes to co-exist in an image, and propose an effective layer that enables an SR network to generate rich and realistic textures in a single forward pass conditioned on the prior provided up to the pixel level.

Network conditioning. Our work is inspired by previous studies on feature normalization. Batch normalization (BN) is a widely used technique to ease network training by normalizing feature statistics . Conditional Normalization (CN) applies a learned function of some conditions to replace parameters for feature-wise affine transformation in BN. Some variants of CN have proven highly effective in image style transfer , visual question answering and visual reasoning . Perez et al. develop a feature-wise linear modulation layer (FiLM), to exploit linguistic information for visual reasoning. This layer can be viewed as a generalization of CN. Perez et al. show that the affine transformation in CN needs not be placed after normalization. Features can be directly modulated. FiLM shows promising results in visual reasoning. Nonetheless, the formulation cannot handle conditions with spatial information (e.g., semantic segmentation maps) since FiLM accepts a single linguistic input and outputs one scaling and one shifting parameter for each feature map, agnostic to spatial location. Preserving spatial information is crucial for low-level tasks, e.g., SR, since these tasks usually require adaptive processing at different spatial locations of an image. Applying FiLM in SR will result in homogeneous spatial feature modulation, hurting SR quality. The proposed SFT layer addresses this shortcoming. It is capable of converting spatial conditions for not only feature-wise manipulation but also spatial-wise transformation.

Semantic guidance. In image generation , semantic segments are used as input conditions to generate natural images. Gatys et al. use semantic maps to control perceptual factors in neural style transfer. uses semantic segmentation for video deblurring. Zhu et al. propose an approach to generate new clothing on a wearer. It first generates human segmentation maps and then uses them to render plausible textures by enforcing region-specific texture rendering. Our work differs from these works mainly in two aspects. First, we use semantic maps to guide texture recovery for different regions in SR domain. Second, we utilize probability maps to capture delicate texture distinction instead of simple image segments.

Methodology

Given a single low-resolution image $\bm{x}$, super-resolution aims at estimating a high-resolution one $\hat{\bm{y}}$, which is as similar as possible to the ground truth image $\bm{y}$. Current CNN-based methods use feed-forward networks to directly learn a mapping function $G_{\bm{\theta}}$ parametrized by $\bm{\theta}$ as

In order to estimate $\hat{\bm{y}}$, a specific loss function $\mathcal{L}$ is designed for SR to optimize $G_{\bm{\theta}}$ on the training samples,

where $(\bm{x}_{i},\bm{y}_{i})$ are training pairs. Perceptual loss and adversarial loss are introduced to solve the regression-to-the-mean problem that is usually caused by conventional MSE-oriented loss functions. These new losses greatly improve the perceptual quality of reconstructed images. However, the generated textures tend to be monotonous and unnatural, as seen in Fig. 1.

We argue that the semantic categorical prior, i.e., knowing which region belongs to the sky, water, or grass, is beneficial for generating richer and more realistic textures. The categorical prior $\Psi$ can be conveniently represented by semantic segmentation probability maps $\mathbf{P}$,

where $P_{k}$ is the probability map of $k^{th}$ category and $K$ is the total number of considered categories. To introduce priors in SR, we reformulate Eqn. (1) as

where $\Psi$ defines the prior upon which the mapping function can condition. Note that apart from categorical priors, the proposed formulation is also applicable to other priors such as depth information, which could be helpful to the recovery of texture granularity in SR. And one could easily extend the formulation to consider multiple priors simultaneously. In the following section, we focus on categorical priors and the way we use them to influence the behavior of an SR network.

A Spatial Feature Transform (SFT) layer learns a mapping function $\mathcal{M}$ that outputs a modulation parameter pair $(\bm{\gamma},\bm{\beta})$ based on some prior condition $\Psi$. The learned parameter pair adaptively influences the outputs by applying an affine transformation spatially to each intermediate feature maps in an SR network. During testing, only a single forward pass is needed to generate the HR image given the LR input and segmentation probability maps.

More precisely, the prior $\Psi$ is modeled by a pair of affine transformation parameters $(\bm{\gamma},\bm{\beta})$ through a mapping function $\mathcal{M}:\Psi\mapsto(\bm{\gamma},\bm{\beta})$. Consequently,

After obtaining $(\bm{\gamma},\bm{\beta})$ from conditions, the transformation is carried out by scaling and shifting feature maps of a specific layer:

where F denotes the feature maps, whose dimension is the same as $\bm{\gamma}$ and $\bm{\beta}$, and $\odot$ is referred to element-wise multiplication, i.e., Hadamard product. Since the spatial dimensions are preserved, the SFT layer not only performs feature-wise manipulation but also spatial-wise transformation.

Figure 3 shows an example of implementing SFT layers in an SR network. We provide more details of the SR branch in Sec. 3.2. Here we focus on the conditioning part. The mapping function $\mathcal{M}$ can be arbitrary functions. In this study, we use a neural network for $\mathcal{M}$ so that it can be optimized end-to-end with the SR branch. To further share parameters among multiple SFT layers for efficiency, we use a small condition network to generate shared intermediate conditions that can be broadcasted to all the SFT layers. Meanwhile, we still keep few parameters inside each SFT layer to further adapt the shared conditions to the specific parameters $\bm{\gamma}$ and $\bm{\beta}$, providing fine-grained control to the features.

Segmentation probability maps as prior. We provide a brief discussion on the segmentation network we used. The details are provided in the supplementary material. The LR image is first upsampled to the desired HR size with bicubic interpolation. It is then fed into a segmentation network as the input. The network is pretrained on the COCO dataset and then fine-tuned on the ADE dataset with additional animal and mountain images. We train the network separately from the main SR network.

For a sanity check, we study the accuracy of segmentation maps obtained from LR image. In a typical setting of SR studies, LR images are downsampled with a scaling factor of $\times 4$ from HR images. We find that under this resolution, satisfactory segmentation results can still be obtained even on LR images given a modern CNN-based segmentation model . Some LR images and the corresponding segmentation results are depicted in Fig. 4. As can be observed in Fig. 4, LR segmentation is close to that of HR. We have not yet tried segmentation on small objects as this remains a challenging problem in the image segmentation community. During testing, classes that fall outside the pre-defined $K$ segmentation classes will be categorized as ‘background’ class. In this case, our method would still generate a set of default $\bm{\gamma}$ and $\bm{\beta}$, degenerating itself as SRGAN, i.e., treating all classes equally.

Discussion. There are alternative ways to introduce categorical priors to an SR network. For instance, one can concatenate the segmentation probability maps with the input LR image as a joint input to the network. We find that this method is ineffective in altering the behavior of CNN (see Sec. 4.3). Another method is to directly concatenate the probability maps with feature maps in the SR branch. This approach resembles the multi-texture synthesis network The approach in is not applicable in our work since it can only generate outputs with a fixed size due to the upsampling operation from one-hot vector.. This method, though not as parameter efficient as SFT, amounts to simply adding a post-layer for feature-wise conditional bias. It is thus a special case of SFT. Another more brute-force approach is to first decompose the LR image based on the predicted semantic class and process each region separately using a model trained specifically for that class. These models may share features to save computation. The final output is generated by combining the output of each model class-wise. This method is computationally inefficient as we need to perform forward passes with several CNN models for a single input image.

2 Architecture

Our framework is based on adversarial learning, inspired by . Specifically, it consists of one generator $G_{\bm{\theta}}$ and one discriminator $D_{\bm{\eta}}$, parametrized by $\bm{\theta}$ and $\bm{\eta}$ respectively. They are jointly trained with a learning objective given below:

where $p_{\text{HR}}$ and $p_{\text{LR}}$ are the empirical distributions of HR and LR training samples, respectively.

The architecture of $G_{\bm{\theta}}$ is shown in Fig. 3. It consists of two streams: a condition network and an SR network. The condition network takes segmentation probability maps as input, which are then processed by four convolution layers. It generates intermediate conditions shared by all the SFT layers. To avoid interference of different categorical regions in one image, we restrict the receptive field of the condition network by using $1\times 1$ kernels for all the convolution layers.

The SR network is built with 16 residual blocks with the proposed SFT layers, which take the shared conditions as input and learn $(\bm{\gamma},\bm{\beta})$ to modulate the feature maps by applying affine transformation. Skip connection is used to ease the training of deep CNN. We upsample features by using nearest-neighbor upsampling followed by a convolution layer. The upsampling operation is performed in the latter part of the network and thus most computation is done in the LR space. Although we have not tried other architectures for the SR network, we believe many contemporary models such as DRRN and MemNet are applicable and can equally be benefited from the SFT layer.

For discriminator $D_{\bm{\eta}}$, we apply a VGG-style network of strided convolutions to gradually decrease the spatial dimensions. The full architecture and details are provided in the supplementary material.

3 Loss Function

We draw inspiration from and apply perceptual loss and adversarial loss in our model. The perceptual loss measures the distance in a feature space. To obtain the feature maps, we use a pre-trained 19-layer VGG network , denoted as $\phi$,

Similar to , we use the feature maps obtained by the forth convolution before the fifth max-pooling layer and compute the MSE on their feature activations.

The adversarial loss $\mathcal{L}_{D}$ from GAN is also used to encourage the generator to favor the solutions in the manifold of natural images,

Experiments

Implementation details. In this work, we focus on outdoor scenes since their textures are rich and well-suited for our study. For example, the sky is smooth and lacks sharp edges, while the building is rich of geometric patterns. The water presents smooth surface with waves, while the grass has matted textures. We assume seven categories, i.e., sky, mountain, plant, grass, water, animal and building. A ‘background’ category is used to encompass regions that do not appear in the aforementioned categories.

Following , all experiments were performed with a scaling factor of $\times 4$ between LR and HR images. We initialized the SR network by parameters pre-trained with perceptual loss and GAN loss on ImageNet dataset. After removing low resolution images of size below 30kB, we obtained roughly 450k training images. During training, we followed existing studies to obtain LR images by downsampling HR images using MATLAB bicubic kernel. The mini-batch size was set to 16. The spatial size of cropped HR and LR sub-images were $96\times 96$ and $24\times 24$, respectively.

After initialization, with the same training setting, we fine-tuned our full network on outdoor scenes conditionally on the input segmentation probability maps. In particular, we collected a new outdoor dataset by querying images from search engines using the defined categories as keywords. The dataset was divided into training and test partitions (OutdoorSceneTrain and OutdoorSceneTest)For OutdoorSceneTrain, we cropped each image so that only one category exists, resulting in 1k to 2k images for each category. Background images were randomly sampled from ImageNet. The total number of training images is 10,324. The OutdoorSceneTest partition consists of 300 images and they are not pre-processed. Segmentation probability maps were generated by the segmentation network (Sec. 3.1).

For optimization, we used Adam with $\beta_{1}=0.9$. The learning rate was set to $1\times 10^{-4}$ and then decayed by a factor of 2 every 100k iterations. We alternately optimized the generator and discriminator until the model converged at about $5\times 10^{5}$ iterations. Inspired by , our discriminator not only distinguishes whether the input is real or fake, but also predicts which category the input belongs to. This is possible since our training images were cropped to contain only one category. This restriction was not applied on test images. We find this strategy facilitates the generation of images with more realistic textures.

We did not conduct our main evaluation on standard benchmarks such as Set5 , Set14 and BSD100 since these datasets are lack of regions with well-defined categories. Nevertheless, we will show that our method still perform satisfactorily on out-of-category examples in Sec. 4.3. Results (PSNR, SSIM) on standard benchmarks are provided in the supplementary material.

Figure 5 shows the qualitative results of different models including PSNR-oriented methods, such as SRCNN , VDSR , LapSRN , DRRN , MemNet , and GAN-based methods, such as SRGAN and EnhanceNet . More results are provided in the supplementary material. For SRGAN, we re-implemented their method and fine-tuned the model with the same setting as ours. We directly used the released test code of EnhanceNet since no training code is available. Despite of preserving sharp edges, PSNR-oriented methods always produce blurry textures. SRGAN and EnhanceNet largely improve the high-frequency details, however, they tend to generate monotonous and unnatural textures, like water waves in Fig. 5. Our method employs categorical priors to help capture the characteristics of each category, leading to more natural and realistic textures.

2 User Study

We performed a user study to quantify the ability of different approaches to reconstruct perceptually convincing images. To better compare our method against PSNR-oriented baselines and GAN-based approaches, we divided the evaluations into two sessions. In the first session, we focused on PSNR-oriented baselines. The users were requested to rank 4 versions of each image: SRCNN , MemNet (the state-of-the-art PSNR-oriented method), our SFT-GAN, and the original HR image according to their visual quality. We used 30 random images chosen from OutdoorSceneTest and all images were presented in a randomized fashion. In the second session, we focused on GAN-based methods so that the user can concentrate on the texture quality. The subjects were shown the super-resolved image pairs (enlarged texture patches were depicted to facilitate the comparison). Each pair consists of an image of the proposed SFT-GAN and the counterpart generated by SRGAN or EnhanceNet . The users were asked to pick the image with more natural and realistic textures. This session involved 96 randomly selected images in total.

We asked 30 users to finish our user study. The results of the first and second sessions are presented in Fig. 6 and Fig. 7, respectively. The results of the first session show that SFT-GAN outperforms the PSNR-oriented methods by a large margin. This is not surprising since PSNR-oriented methods always produce blurry results especially in texture regions. Our method sometimes generates good-quality images comparable to HR causing confusion within the users. In the second session, our method is ranked higher than SRGAN and EnhanceNet , especially in building, animal, and grass categories. Comparable performance is found on sky and plant categories.

3 Ablation Study

From segmentation probability maps to feature modulation parameters. Our method modulates intermediate features based on segmentation probability maps. We investigate the relationship between the probability and feature modulation parameters, as depicted in Fig. 9. All SFT layers exhibit similar behavior so that we only present its first layer for this analysis. In the top row, we show an image where building and grass co-exist. It is observed that modulation parameters $\bm{\gamma}$ and $\bm{\beta}$ are different for various categorical regions to exert meaningful spatial-wise transformation. From the heat map of $\bm{\gamma}$ and $\bm{\beta}$, we can see that the modulation parameter maps are closely related to the segmentation probability maps. The boundary of building and grass is clear with a sharp transition. With the guidance of the probability map, our model is able to generate building and grass textures simultaneously without interference of different categories.

Some classes, like plant and grass, may not be clearly distinguishable by their visual appearance. They interlace with each other without a clear boundary. Despite the ambiguity, the probability maps are still capable of capturing the semantics to certain extent and the SFT layers reflect the subtle differences between categories in its spatial transformation. In Fig. 9, the upper row shows the probability map and modulation parameters activated for plant while the lower row shows those for grass. Distinct activations with smooth transition can be observed. As a result, textures generated by SFT-GAN become more realistic.

Robustness to out-of-category examples. Our model mainly focuses on outdoor scenes and it is effective given segmentation maps of the pre-defined $K$ classes. Despite the assumption, it is also robust to other scenes where segmentation results are not available. As shown in Fig. 10, the SFT-GAN can still produce comparative results with SRGAN when all the regions are deemed as ‘background’. More results are provided in the supplementary material.

Comparison with other conditioning methods. We qualitatively compare with several alternatives for conditioning SR network, which are already discussed in Sec. 3.1.

1) Input concatenation – This method concatenates the segmentation probability maps with the LR image as a joint input to the network. This is equivalent to adding SFT conditional bias at the input layer.

2) Compositional mapping – This method is identical to Zhu et al. . It decomposes an LR image based on the predicted semantic classes and processes each region separately using a specific model for that class. Different models share parameters at lower layers.

3) FiLM – This method predicts one parameter for each feature map without spatial information and then uses these parameters to modulate the feature maps.

As can be observed from Fig. 11, the proposed SFT-GAN yields outputs that are perceptually more convincing. Naive input concatenation is not sufficient to exert the necessary condition for class-specific texture generation. Compositional mapping produces good results but it is not parameter efficient ($\times 2.5$ parameters as ours). It is also computationally inefficient as we need to forward several times for a single input image. FiLM cannot handle the situations where multiple categorical classes co-exist in an image since it predicts one parameter for each feature map, agnostic to spatial information. For example, in the first image of Fig. 11, the road and sky interfere with the building’s structure and thus noisy bricks are generated. Similarly in the second image, the animal’s fine texture is severely affected by the grass.

Discussion and Conclusion

We have explored the use of semantic segmentation maps as categorical prior for constraining the plausible solution space in SR. A novel Spatial Feature Transform (SFT) layer has been proposed to efficiently incorporate the categorical conditions into a CNN-based SR network. Thanks to the SFT layers, our SFT-GAN is capable of generating distinct and rich textures for multiple semantic regions in a super-resolved image in just a single forward pass. Extensive comparisons and a user study demonstrate the capability of SFT-GAN in generating realistic and visually pleasing textures, outperforming previous GAN-based methods .

Our work currently focuses on SR of outdoor scenes. Despite robust to out-of-category images, it does not consider priors of finer categories, especially for indoor scenes, e.g., furniture, appliance and silk. In such a case, it puts forward challenging requirements for segmentation tasks from an LR image. Future work aims at addressing these shortcomings. Furthermore, segmentation and SR may benefit from each other and jointly improve the performance.

Acknowledgement. This work is supported by SenseTime Group Limited and the General Research Fund sponsored by the Research Grants Council of the Hong Kong SAR (CUHK 14241716, 14224316. 14209217).

References