Image Synthesis From Reconfigurable Layout and Style

Introduction

Remarkable recent progress has been made on both unconditional and conditional image synthesis . The former aims to generate high-fidelity images from some random latent codes. The latter needs to do so with given conditions satisfied in terms of some consistency metrics. The conditions may take many forms such as categorical labels, desired attributes, descriptive sentences, scene graphs, and paired or unpaired images/semantic maps. From the perspective of generative learning, the solution space of the latter is much difficult to capture than that of the former. Conditional image synthesis, especially with coarse yet complicated and reconfigurable conditions, remains a long-standing problem. Once powerful systems are developed, they can facilitate to pave a way for computers to truly understand visual patterns via analysis-by-synthesis. They will also enable a wide range of practical applications, e.g., generating high-fidelity data for long-tail scenarios in different vision tasks such as autonomous driving.

In this paper, we are interested in conditional image synthesis from layout and style. The layout consists of labeled bounding boxes configured in an image lattice (e.g., $64\times 64$ or $128\times 128$). The style is represented by some latent code. Layout represents a sweet yet challenging spot for conditional image synthesis: First, layout is usually used as the intermediate representation for other conditional image synthesis such as text-to-image and scene-graph-to-image . Second, layout is more flexible, less constrained and easier to collect than semantic segmentation maps . Third, layout-to-image requires addressing challenging one-to-many mapping and consistent multi-object generation (e.g., occlusion handling for overlapped bounding boxes and uneven, especially long-tail distributions of objects).

Layout-to-image is a relatively new task with many new technical challenges for state-of-the-art image synthesis frameworks and only a few work have been proposed in the very recent literature . Recently, we have seen remarkable progress on the high-fidelity class-conditional image synthesis in ImageNet by the BigGAN , and on the amazing style control for specific objects (e.g., faces and cars) by the unconditional StyleGAN (which may be considered as implicitly conditional image synthesis since only one category is usually handled in training). Despite the big successes in generative learning, the problem considered in this paper is still more challenging since the solution space is much more difficult to capture and has much more complicated distributions. For example, we can use the BigGAN to generate a cat image, and as long as the generated image looks realistic and sharp, we think it does a great job. Similarly, we can use the StyleGAN to generate a face image, and we are happy (even shocked sometimes) if a realistic and sharp face image is generated with a natural style (e.g., smile or sad). Layout-to-image needs to tackle many spatial and semantic (combinatorial) relationships among multiple objects besides the naturalness.

In this paper, we further focus on image synthesis from reconfigurable layout and style. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of layout and style latent code (Figure 1). State-of-the-art methods on reconfigurable layout-to-image still mainly focus on low resolution ($64\times 64$) (which are, in part, due to computationally expensive designs in the pipelines such as convolutional LSTM used in ). Beside the resolution issue, another drawback of existing methods is that the diversity of generated images (i.e., style control) is not sufficiently high to preserve the intrinsic one-to-many mapping. We aim to improve both the resolution and the style diversity in reconfigurable layout-to-image.

2 Method Overview

To address the challenges in layout-to-image and inspired by the recent StyleGANs , we present a LayOut- and STyle-based architecture for GANs (termed LostGANs) in the paper (Figure 2).

First, since layout-to-image entails highly expressive neural architectures handling multi-object generation and their diverse occurrence and configurations in layouts. We utilize ResNet for both the generator and discriminator in the proposed LostGAN, as done in the projection-based cGAN and BigGAN .

Second, to account for the gap between bounding boxes in a layout and underlying object shapes, we introduce an encoder for layout to predict masks for each bounding box. As we will show in experiments, our LostGAN can predict reasonably good masks in a weakly-supervised manner. The masks help place objects in the generated images with fine-grained geometric properties. So, we address layout-to-image by computing layout-to-mask-to-image (Figure 3), which is motivated by impressive recent progress on conditional image synthesis from semantic label maps .

Third, to achieve instance-sensitive and layout-aware style control, we extend the Adaptive Instance Normalization (AdaIN) used in the StyleGAN to object instance-specific and layout-aware feature normalization (ISLA-Norm) for the generator for fine-grained spatially distributed multi-object style control. ISLA-Norm computes the mean and variance as done in BatchNorm , but computes object instance-specific and layout-aware affine transformations (i.e., gamma and beta parameters) separately for each sample in a min-batch as done in AdaIN (Figure 3). We utilize the projection-based approach proposed in . From the layout encoder, we compute object instance-specific style latent codes (gamma and beta parameters) via simple linear projection. Then, we place the projection-based latent codes in the corresponding predicted masks, and thus induce layout-aware affine transformations for recalibrating normalized feature responses.

Lastly, we utilize both image and object adversarial hinge losses as adopted in in the end-to-end training. Object adversarial loss follows the projection based method in which is the state-of-the-art approach for embedding labels.

We deliberately try to keep our LostGAN as simple as possible by exploiting the best practices in the literature of conditional image synthesis. We hope it can stimulate more exploration on this relatively new task, image synthesis from reconfigurable layout and style.

In experiments, our LostGAN is tested in the COCO-Stuff dataset and the Visual Genome (VG) dataset . It obtains state-of-the-art performance on both datasets in terms of the inception score , Frèchet Inception Distance , diversity score , and classification accuracy , which supports the effectiveness of our ILSA-Norm and LostGAN.

Related Works

Conditional Image Synthesis. Generative Adversarial Networks (GANs) have achieved great success in image synthesis conditioned on additional input information (i.e. class information , source image , text description , etc). How to feed conditional information to model has been studied in various ways. In vector encoded from conditional information concatenated with noise vector is passed as input to generator. In , conditional information is provided to generator by conditional gains and bias in BatchNorm layers. Concurrent work learns spatially adaptive normalization from well annotated semantic masks, while our proposed ISLA-Norm learns from coarse layout information. feed the conditional information into discriminator by naively concatenation with the input or intermediate feature vector. In , projection based way to incorporate conditional information to discriminator effectively improve the quality of class conditional image generation. In our proposed method, layout condition is adopted to generator with ISLA-Norm, and objects information is utilized in projection based discriminator as .

Image Synthesis from Layout. Spatial layout conditioned image generation has been studied in recent literature. In , layout and object information is utilized in text-to-image generation. controls location of multiple objects in text-to-image generation by adding an object pathway to both the generator and discriminator. performs text-to-image synthesis in two steps: semantic layout (class label and bounding boxes) generation from text first, and image synthesis conditioned on predicted semantic layout and text description. However, requires pixel-level instance segmentation annotation, which is labor intensive to collect, for training of shape generator, while our method does not require pixel-level annotation and can learn segmentation mask in a weakly-supervised manner. studied similar task with us, where variational autoencoders based network is adopted for scene image generation from layout.

Our Contributions. This paper makes the following main contributions to the field of conditional image synthesis.

It presents a layout- and style-based architecture for GANs (termed LostGANs) which integrates the best practices in conditional and unconditional GANs for a relatively new task, image synthesis from reconfigurable layout and style.

It presents an object instance-specific and layout-aware feature normalization scheme (termed ISLA-Norm) which is inspired by the projection-based conditional BatchNorm used in cGANs and the Adaptive Instance Normalization (AdaIN) used in StyleGAN . It explicitly accounts for the layout information in the affine transformations.

It shows state-of-the-art performance in terms of the inception score , Frèchet Inception Distance , diversity score and classification accuracy on two widely used datasets, the COCO-Stuff and the Visual Genome .

The Proposed Method

In this section, we first define the problem and then present details of our LostGAN and ISLA-Norm.

Image synthesis from layout and style is the problem of learning a generation function which is capable of synthesizing an image defined on $\lambda$ for a given input $(L,z_{img},Z_{obj})$,

where $\Theta_{G}$ represents the parameters of the generation function. Ideally, $G(\cdot)$ is expected to capture the underlying conditional data distribution $p(I|L,z_{img},Z_{obj})$ in the high-dimensional space.

Reconfigurability of $G(\cdot)$. We are interested in three aspects in this paper:

Image style reconfiguration: If we fix the layout $L$, is $G(\cdot)$ capable of generating images with different styles for different $(z_{img},Z_{obj})$?

Layout reconfiguration: Given a $(L,z_{img},Z_{obj})$, is $G(\cdot)$ capable of generating consistent images for different $(L^{+},z_{img},Z_{obj}^{+})$ where we can add a new object to $L^{+}$ or just change the bounding box location of an existing object? When a new object is added, we also sample a new $z_{obj}$ to add in $Z_{obj}^{+}$.

It is a big challenge to address the three aspects by learning a single generation function. It may be even difficult for well-trained artistic people to do so at scale (e.g., handling the 171 categories in the COCO-Stuff dataset). Due to the complexity that the generation function (Eqn. 1) needs to handle, it is parameterized (often over-parameterized) by powerful deep neural networks (DNNs). It is also well-known that training the DNN-based generation function individually is a extremely difficult task. Generative adversarial networks (GANs) are entailed which are formulated under two-player minmax game settings.

2 The LostGAN

As Figure 2 shows, our LostGAN follows the traditional GAN pipeline with the following modifications.

Figure 3 (a) shows the generator which utilizes the ResNet architecture as backbone. Consider generating $64\times 64$ images, the generator consists of 4 residual building blocks (ResBlocks). The image style latent code $z_{img}$ is a $d_{noise}$-dim vector ($d_{noise}=128$ in our experiments) whose elements are sampled from standard normal distribution under i.i.d. setting. Through a linear fully connected (FC) layer, $z_{img}$ is projected to a $4\times 4\times(16\times ch)$ dimensional vector which is then reshaped to $(4,4,16\times ch)$ (representing height, width and channels) where $ch$ is a hyperparameter to control model complexity (e.g., $ch=64$ for generating $64\times 64$ images). Then, each of the four ResBlocks upsamples its input with ratio $2$ and bilinear interpolation. In the meanwhile, the feature channel will be decreased by ratio $2$. For generating $128\times 128$ images, we use 5 ResBlocks with $ch=64$ and the same $d_{noise}=128$ for $z_{img}$.

2.2 The ISLA-Norm

Figure 3 (b) shows the detail of ResBlock and the proposed ISLA-Norm. The ResBlock uses the basic block design as adopted in the projection-based cGAN and BigGAN . Our ISLA-Norm first computes the mean and variance as done in BatchNorm , and then learns object instance-specific layout-aware affine transformation for each sample in a batch similar in spirit to the AdaIN used by the StyleGAN . So, the feature normalization is computed in a batch manner, and the affine transformation is recalibrated in a sample-specific manner.

Denote by $x$ the input 4D feature map of ISLA-Norm, and $x_{nhwc}$ the feature response at position $(n,h,w,c)$ (using the convention order of axes for batch, spatial height and width axis, and channel). We have $n\in[0,N-1],\,h\in[0,H-1],\,w\in[0,W-1]$ and $c\in[0,C-1]$ where $H,W,C$ depend on the stage of a ResBlock.

In training, ISLA-Norm first normalizes $x_{nhwc}$ by,

where the channel-wise batch mean $\mu_{c}=\frac{1}{N\cdot H\cdot W}\sum_{n,h,w}x_{nhwc}$ and standard deviation (std) $\sigma_{c}=\sqrt{\frac{1}{N\cdot H\cdot W}\sum_{n,h,w}(x_{nhwc}-\mu_{c})^{2}+\epsilon}$ ($\epsilon$ is a small positive constant for numeric stability). In standard BatchNorm , for the affine transformation, a channel-wise $\gamma_{c}$ and $\beta_{c}$ will be learned and shared with all spatial locations and all samples in a batch. our ISLA-Norm will learn object instance-specific and layout-aware affine transformation parameters, $\gamma_{nhwc}$ and $\beta_{nhwc}$, and then recalibrate the normalized feature responses by,

Computing $\gamma_{nhwc}$ and $\beta_{nhwc}$. Without loss of generality, we show how to compute the gamma and beta parameters for one sample, i.e., $\gamma_{hwc}$ and $\beta_{hwc}$. As shown in Figure 3 (b), we have the following four steps.

ii) Object instance-specific projection. With the final embedding matrix, we compute object instance-specific channel-wise $\gamma$ and $\beta$ via linear projection with a learnable $(d_{e}+d_{noise})\times 2C$ projection matrix where $C$ is the number of channels.

iii) Mask prediction. The $s\times s$ mask for each object instance (e.g., $s=16$ in our experiments) is predicted by a sub-network consisting of several up-sample convolution followed by sigmoid transformation. So, our predicted masks are not binary. Then, we resize the predicted masks to the sizes of corresponding bounding boxes.

iv) ISLA $\gamma$ and $\beta$ computation. We unsqueeze the object instance-specific channel-wise $\gamma$ and $\beta$ to their corresponding bounding boxes with the predicted mask weights multiplied. Then, we add them together with averaged sum used for overlapping regions.

2.3 The Discriminator

As shown in Figure 2, our discriminator consists of three components: the shared ResNet backbone, the image head classifier and the object head classifier.

The ResNet backbone has several ResBlocks (4 for 64$\times$64 and 5 for 128$\times$128) as in . The image head classifier consists of a ResBlock, a global average pooling layer and a fully-connected (FC) layer with one output unit, while object head classifier consists of ROI Align , a global average pooling layer and a FC layer with one output unit.

Following the projection-based cGANs and the practice in BigGANs , we learn a separate label embedding for computing object adversarial hinge loss.

Denote by $D(\cdot;\Theta_{D})$ the discriminator with parameters $\Theta_{D}$. Given an image $I$ (real or synthesized) and a layout $L$, the discriminator computes the prediction score for image and the average score for cropped objects, and we have,

2.4 The Loss Functions

To train $(\Theta_{G},\Theta_{D})$ in our LostGAN, we utilize the hinge version of the standard adversarial loss ,

where $t\in\{img,obj\}$. Let $l(I,L)=l_{img}(I,L)+\lambda\cdot l_{obj}(I,L)$ with $\lambda$ the trade-off parameter for controlling the quality between synthesized images and objects ($\lambda=1$ in our experiments). We have the expected losses for the discriminator and the generator,

where $p(I,L)$ represents all the real and fake (by the current generator) data and $p_{fake}(I,L)$ represents the fake data.

Experiments

We test our LostGAN in the COCO-Stuff dataset and the Visual Genome (VG) dataset . We evaluate it for generating images at two resolutions 64$\times$64 and 128$\times$128. In comparison, the state-of-the-art methods include the very recent Layout2Im method , the scene graph to image (sg2im) method and the pix2pix method .

The COCO-Stuff 2017 augments the COCO dataset with pixel-level stuff annotations. The annotation contains 80 thing classes (person, car, etc.) and 91 stuff classes (sky, road, etc.) Following settings of , objects covering less than 2% of the image are ignored, and we use images with 3 to 8 objects. The Visual Genome dataset . Following settings of to removing small and infrequent objects, we have 62,565 training, 5,506 val and 5,088 testing images with 3 to 30 objects from 178 categories in each image.

2 Evaluation Metrics

We evaluate quality and visual appearance of generated images by Inception Score (higher is better) and Frèchet Inception Distance (FID, lower is better) , which use pretrained Inception network to encourage recognizable objects within images and diversity across images. Diversity score computes perceptual similarity between two images (higher is better). We adopt LPIPS metric to compute perceptual similarity in feature space between two images generated from same layout as diversity score. We also evaluate our model by recently proposed Classification Accuracy Score (CAS) .

3 Quantitative results

Table 1, 2 summarizes comparisons between our model and state-of-the-art models with respect to inception score, FID, diversity score and classification accuracy. Our LostGAN outperforms the most recent Layout2Im in terms of both Inception score and Diversity score. For 64$\times$64 images, the improvement of Inception score, FID and classification accuracy indicates higher visual quality of image generated by our model. Diversity score is improved significantly which shows that our LostGAN can generate images with various appearance for a given layout. We also conduct experiments at the resolution of 128$\times$128, and our LostGAN obtains consistently better results.

4 Qualitative results

Figure 4 shows results of different models generating images from the same layout on both COCO-Stuff and VG. The input layouts are quite complex. Our LostGAN can generate visually more appealing images with more recognizable objects that are consistent with input layouts at resolution 64$\times$64, and is further capable of synthesizing images at $128\times 128$ resolution with better image quality.

We also conduct some ablation studies on the three aspects of reconfigurability and mask prediction.

Layout reconfiguration is demonstrated by adding object to or moving a bounding box in a layout (Figure 5). Our LostGAN shows better layout reconfigurability than the Layout2Im . When adding extra objects or moving bounding box of one instance, our model can generate reasonable objects at desired position while keeping existing objects unchanged as we keep the input style of existing objects fixed. When moving bounding box of one object, style of generated object in new position can also be kept consistent, like (f) and (g), the person is moved while keep style feature like pose and color of clothes unaffected.

Image style reconfiguration To assess diversity of generation, multiple images are sampled from our LostGAN for each input layout (Figure 6). Our model can synthesize images with different visual appearance for a given layout while preserving objects at desired location.

Object instance style reconfiguration Our LostGAN is also capable of controlling styles at object instance level. Figure 7 shows results of gradually morphing styles of one instance in different images. Top row shows how the style of sky gradually turns from blue to dusk while keeping styles of other objects unaltered. Bottom row displays how the style of grass transforms from green to withered.

Weakly-supervised mask prediction Figure 8 shows generated semantic label map when synthesizing images from given layouts. For pixels where bounding boxes of different objects overlap, their semantic labels are assigned by objects with the highest predicted mask weight. Unlike where ground truth masks is adopted to guide learning of shape generator, our model can learn semantic masks in a weakly-supervised manner. Even for objects with overlapped bounding box, like person and surfboard in (f), synthesized images and learned masks are consistent and semantically reasonable.

Conclusion

This paper presents a layout- and style-based architecture for generative adversarial networks (LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. The proposed LostGAN can learn fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and proposes the object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. State-of-the-art performance is obtained on COCO-Stuff and VG dataset. Qualitative results demonstrate the proposed model is capable of generating scene images with reconfigurable layout and instance-level style control.

Acknowledgement

The authors would like to thank the anonymous reviewers for their helpful comments. This work was supported in part by ARO grant W911NF1810295, NSF IIS-1909644, Salesforce Inaugural Deep Learning Research Grant (2018) and ARO DURIP grant W911NF1810209. The views presented in this paper are those of the authors and should not be interpreted as representing any funding agencies.

References