Be Your Own Prada: Fashion Synthesis with Structural Coherence
Shizhan Zhu, Sanja Fidler, Raquel Urtasun, Dahua Lin, Chen Change Loy
Introduction
Imagine that you could be your own fashion designer, and be able to seamlessly transform your current outfit in the photo into a completely new one, by simply describing it in words (Figure 1). In just minutes you could design and “try on” hundreds of different shirts, dresses, or even styles, allowing you to easily discover what you look good in. The goal of this paper is to develop a method that can generate new outfits onto existing photos, in a way that preserves structural coherence from multiple perspectives:
Retaining the body shape and pose of the wearer,
Producing regions and the associated textures that conform to the language description, and
Enforcing coherent visibility of body parts.
Meeting all these requirements at the same time is a very challenging task. First, the input image is the only source from which we can mine for the body shape information. With only a single view of the wearer, it is nontrivial to recover the body shape accurately. Moreover, we do not want the shape of the generated outfit to be limited by the original garments of the wearer. For example, replacing the original long-sleeve shirt with a short-sleeve garment would require the model to hallucinate the person’s arms and skin.
Conventional 2D non-parametric methods or 3D graphics approaches meet the first requirement through structural constraints derived from human priors. They can be in the form of accurate physical measurements (e.g. height, waist, hip, arm length) to create 3D virtual bodies ; manual manipulations of sliders such as height, weight and waist girth ; or indication of joint positions and a rough sketch outlining the human body silhouette . All these methods require explicit human interventions at test time, which would limit their applicability in practical settings. In addition, as these methods provide no obvious ways to incorporate textual descriptions to condition the synthesis process, it is non-trivial to fulfil the second requirement with existing methods. Lastly, they do not meet the third requirement as they do not support hallucination of the missing parts.
Generative Adversarial Networks (GAN) is an appealing alternative to conventional methods. In previous work, DCGAN , a GAN formulation combined with convolutional networks, has been shown to be an effective model to produce realistic images. Moreover, it allows for an end-to-end embedding of textual descriptions to condition the image generation. The task of clothing generation presents two significant challenges which are difficult to address with the standard DCGAN. First, it directly targets the pixel values and provides no mechanism to enforce structural coherence w.r.t. to the input. Second, it tends to average out the pixels , thus resulting in various artifacts, e.g. blurry boundaries, as shown by our experiments.
To tackle this problem, we propose an effective two-stage GAN framework that generates shape and textures in different stages. In the first stage, we aim to generate a plausible human segmentation map that specifies the regions for body parts and the upper-body garment. This stage is responsible for preserving the body shape and ensuring the coherent visibility of parts based on the description. In the second stage, the generator takes both the produced segmentation map and the textual description as conditions, and renders the region-specific texture onto the photograph.
To ensure the coherence in structure of the synthesized image with respect to the input image (i.e. preserving the body shape and pose of the wearer), we present an effective spatial constraint that can be derived from the input photograph. We formulate it carefully so that it does not contradict to the textual description when both of them are used to condition the first-stage GAN. In addition, we also introduce a new compositional mapping layer into the second-stage GAN to enforce region-specific texture rendering guided by the segmentation map. In contrast to existing GANs that perform non-compositional synthesis, the new mapping layer is capable of generating more coherent visibility of body parts with image region-specific textures.
To train our model, we extend the DeepFashion dataset by annotating a subset of upper-body images with sentence descriptions and human body annotationsThe data and code can be found at http://mmlab.ie.cuhk.edu.hk/projects/FashionGAN/.. Extensive quantitative and qualitative comparisons are performed against existing GAN baselines and 2D non-parametric approaches. We also conduct a user study in order to obtain an objective evaluation on both the shape and image generation results.
Related Work
Generative Adversarial Networks (GAN) have shown impressive results generating new images, e.g. faces , indoor scenes , fine-grained objects like birds , or clothes . Training GANs based on conditions incorporates further information to guide the generation process. Existing works have explored various conditions, from category labels , text to an encoded feature vector . Different from the studies above, our study aims at generating the target by using the spatial configuration of the input images as a condition. The spatial configuration is carefully formulated so that it is agnostic to the clothing worn in the original image, and only captures information about the user’s body.
There exist several studies to transfer an input image to a new one. Ledig et al. apply the GAN framework to super-resolve a low-resolution image. Zhu et al. use a conditional GAN to transfer across the image domains, e.g. from edge maps to real images, or from daytime images to night-time. Isola et al. change the viewing angle of an existing object. Johnson et al. apply GANs to neural style transfer. All these studies share a common feature - the image is transformed globally on the texture level but is not region-specific. In this study, we explore a new compositional mapping method that allows region-specific texture generation, which provides richer textures for different body regions.
There are several recent studies that explore improved image generation by stacking GANs. Our work is somewhat similar in spirit to – our idea is to have the first stage to create the basic composition, and the second stage to add the necessary refinements to the image generated in the first stage. However, the proposed FashionGAN differs from S2GAN in that the latter aims at synthesizing a surface map from a random vector in its first stage. In contrast, our goal is to generate a plausible mask whose structure conforms to a given photograph and language description, which requires us to design additional spatial constraints and design coding as conditions. Furthermore, these two conditions should not contradict themselves. Similarly, our work requires additional constraints which are not explored in . Compositional mapping is not explored in the aforementioned studies as well.
Yo et al. propose an image-conditional image generation model to perform domain transfer, e.g., generating a piece of clothing from an input image of a dressed person. Our work differs in that we aim at changing the outfit of a person into a newly designed one based on a textual description. Rendering new outfits onto photographs with unconstrained human poses bring additional difficulties in comparison with work that generates pieces of clothing in a fixed view-angle as in .
Methodology
Our framework is inspired by the generative adversarial network (GAN) proposed by Goodfellow et al. . We first provide a concise review of GAN, and then introduce our outfit generation framework. Generative Adversarial Network has shown a powerful capability in generating realistic natural images. A typical GAN contains a generator and a discriminator . They are jointly trained with a learning objective given below:
We define the problem as follows. We assume we have the original image of a wearer and a sentence description of the new outfit. An example of a description we envision is “a white blouse with long sleeves but without a collar”. Our goal is to produce a new image of the user wearing the desired outfit.
Our method requires training data in order to learn the mapping from one photo to the other given the description. We do not assume paired data where the same user is required to wear two outfits (current, and the described target outfit). Instead, we only require one photo per user where each photo has a sentence description of the outfit. Such data is much easier to collect (Sec. 3.5).
Since in our scenario we only have one (described) image per user, this image serves as both the input and the target during training. Thus, rather than working directly with the original image , we extract the person’s segmentation map, , which contains pixel-wise class labels such as hair, face, upper-clothes, pants/shorts, etc. The segmentation map is thus capturing the shape of the wearer’s body and parts, but not their appearance.
To capture further information about the wearer, we extract a vector of binary attributes, , from the person’s face, body and other physical characteristics. Examples of attributes include gender, long/short hair, wearing/not wearing sunglasses and wearing/not wearing hat. The attribute vector may additionally capture the mean RGB values of skin color, as well as the aspect ratio of the person, representing coarse body size. These are the properties that our final generated image should ideally preserve. Details of how we extract this information are given in Sec. 3.5.
As shown in Fig. 2, we decompose the overall generative process into two relatively easier stages, namely the human segmentation (shape) generation (corresponding to the desired/target outfit) and texture rendering. This decomposition can be expressed as follows:
We provide details of the two generators in our framework in Sec. 3.2 and Sec. 3.3, respectively.
We use a down-sampled version of as a constraint so as to weaken the correlation between the two conditions and , which can contradict each other. Specifically, while keeps the complete information of the wearer’s body shape, its internal partitioning of regions do not necessarily agree with the specifications conveyed in the design coding . If we were to directly feed the high-resolution segmentation map of the original image into the model, strong artifacts would appear when the textual description contradicts with the segmentation map, e.g., the model simultaneously receives the text description “to generate a long dress” and the segmentation map that indicates short upper clothes. Figure 3 shows such failure cases.
Compositional Mapping. Conventional GANs generate an image without enforcing region-specific texture rendering. In the proposed FashionGAN, we propose a new compositional mapping layer that generates the image with the guidance of the segmentation map. In comparison to non-compositional counterparts, the new mapping layer helps to generate textures more coherent to each region and maintain visibility of body parts.
4 Training
Our two GANs are trained separately due to the non-differentiable operation between the two steps. The training process needs one fashion image for each person in the training set, along with the textual description (represented by its designing coding ) and the segmentation map . In our first GAN, we derive the tuple from each training sample and train the networks, following the typical conditional GAN training procedure. In our second GAN, we derive the tuple from each training sample for training. We use the Adam optimizer in training. Discriminative networks only appear in the training phase. Similar to , we provide the conditions (design coding, segmentation maps) to the discriminative networks to enforce consistency between the conditions and the generated results.
5 Implementation Details and Dataset
We perform bicubic down-sampling to get , with a size of 88 (i.e. ). We keep the hair and face regions in our merged maps avoiding the need for the generator to generate the exact face as the original wearer (we replace the generated hair/face region with the original image ). It is hard and not necessary in practice.
To train our framework we extended the publicly available DeepFashion dataset with richer annotations (captions and segmentation maps). In particular, we selected a subset of 78,979 images from the DeepFashion attribute dataset, in which the person is facing toward the camera, and the background of the image is not severely cluttered. Training our algorithm requires segmentation maps and captions for each image. We manually annotated one sentence per photo, describing only the visual facts (e.g., the color, texture of the clothes or the length of the sleeves), avoiding any subjective assessments. For segmentation, we first applied a semantic segmentation method (VGG model fine-tuned on the ATR dataset ) to all the images, and then manually checked correctness. We manually relabeled the incorrectly segmented samples with GrabCut .
Experiments
We verify the effectiveness of FashionGAN through both quantitative and qualitative evaluations. Given the subjective nature of fashion synthesis, we also conduct a blind user study to compare our method with 2D nonparametric based method and other GAN baselines.
Benchmark. We randomly split the whole dataset (78,979 images) into a disjoint training set (70,000 images) and test set (8,979 images). All the results shown in this section are drawn from the test set. A test sample is composed of a given (original) image and a sentence description serving as the redressing condition.
Baselines. As our problem requires the model to generate a new image by keeping the person’s pose, many existing unconditional GAN-based approaches (e.g., DCGAN ) are not directly applicable to our task. Instead, we use the conditional variants to serve as the baseline approach in our evaluation. We compare with several baselines as follows:
One-Step-8-7: We use the down-sampled but not merged segmentation map (887) as the prior;
One-Step-8-4: We use the down-sampled merged segmentation map (884) as the prior (the same setting we used in our first stage GAN).
The architecture of the generator and discriminator used in these baselines are consistent to those used in our proposed method, i.e., containing 6 deconvolution and convolution layers in both the generator and discriminator.
(2) Non-Compositional: To demonstrate the effectiveness of the segmentation guidance, we build a baseline that generates an image as a whole, i.e., without using Eq. (4). In this baseline, we use two generative stages as in our proposed framework. In addition, the first stage generator of this baseline is still conditioned on the spatial constraint to ensure structure coherence to the wearer’s pose.
A well-generated fashion image should faithfully produce regions and the associated textures that conform to the language description. This requirement can be assessed by examining if the desired outfit attributes are well captured by the generated image. In this section, we conduct a quantitative evaluation of our approach to verify the capability of FashionGAN in preserving attribute and structural coherence with the input text.
Table 1 summarizes the attribute prediction results. It can be observed that attribute predictions yielded by FashionGAN are more accurate than other baselines. In particular, our approach outperforms one-step GANs that come without the intermediate shape generation, and two-stage GAN that does not perform compositional mapping. Moreover, the performance of FashionGAN is close to the upper-bound, which was provided by applying the attribute detector on image , where the text input originated from. The results suggest the superiority of FashionGAN in generating fashion images with structure coherence.
2 Qualitative Evaluation
Conditioning on the Same Wearer. Given an image, we visualize the output of FashionGAN with different sentence descriptions. We show all the intermediate results and final rendering step-by-step in Fig. 6, showcasing our generation process. A plausible segmentation map is generated first, and one can notice the variation in shape (e.g., the length of the sleeve). The image generated in the second step has consistent shape with the shape generated in the first step. The generated samples demonstrate variations in textures and colors, while the body shape and pose of the wearer are retained.
Conditioning on the Same Description. In this experiment, we choose photos of different wearers but use the same description to redress them. We provide results in Fig. 7. Regardless of the variations in the human body shapes and poses, our model consistently generates output that respects the provided sentence, further showing the capability of FashionGAN in retaining structural coherence.
Matrix Visualization. In this experiment, we visualize our results in an eight by eight matrix, where each row is generated by conditioning on the same original person, while each column is generated by conditioning on the same text description. We provide results in Fig. 8.
Walking through the embedding space. In this experiment, we generate the images by interpolating the embedding space (i.e. a concatenation of the input Gaussian noise and the text encoding), to show the gradual changes among the shapes and the textures of the generated clothes. We provide results in Fig. 9. For each row, the first and the last images are the two samples that we will make the interpolation. We gradually change the input from the left image. In the first row, we only interpolate the input to the first stage and hence the generated results only change in shapes. In the second row, we only interpolate the input to the second stage and hence the results only change in textures. The last row interpolate the input for both the first and second stages and hence the generated interpolated results transfer smoothly from the left to the right.
Comparison with One-Step GAN Baselines. We provide a qualitative comparison with One-Step variants in Fig. 10. As shown in the figure, our approach achieves better visual quality with fewer artifacts and more consistent human shape.
Comparison with the Non-Compositional Baseline. We show the results in Fig. 11. Our approach provides clearer clothing regions while much fewer visual artifacts and noise, outperforming the baseline approach.
Comparison with the 2D Non-Parametric Baseline. We compare with this conventional baseline by retrieving an exemplar from a large database by text and perform Poisson image blending to apply the new outfit on the wearer. Results are shown in Fig. 12. Due to shape inconsistency between the exemplar and wearer’s body, the rendering results are not satisfactory.
3 User Study
Evaluating the Generated Segmentation Maps. A total of 50 volunteers participated in our user study. Our goal is to examine the quality of the intermediate segmentation maps generated by the first stage of FashionGAN. To this end, we provided the human segmentation map of the original photograph and the generated map, i.e., a pair of maps for each test case, and asked participants to determine which map looked more realistic and genuine. A higher number of misclassified test cases implies a better quality of the generated maps. As only FashionGAN would produce such intermediate segmentation map, we thus only conduct this experiment with our approach. For the total of test cases, the participants misclassified of them (the misclassification rate is 42%). This is significant as our segmentation maps fooled most of the participants, whose ratings were close to random guessing.
Evaluating the Generated Photos. The same group of volunteers were asked to provide a ranking of the generated images produced by FashionGAN as well as the results from three baseline approaches, namely, ‘One-Step-8-7’, ‘One-Step-8-4’, and ‘Non-Compositional’. In addition, we also compared against the 2D non-parametric approach. During the user study, each participant was provided with the original image and the corresponding sentence description. The participants were asked to rank the quality of the generated images with respect to the relevance to the sentence description and the texture quality.
All the 8,979 test images were evenly and randomly assigned to these participants. We summarize various statistics in Table 2 and the frequency statistics for each rating in Fig. 13. For each approach, we computed the average ranking (where 1 is the best and 5 is the worst), standard deviation, and the frequency of being assigned with each ranking. We can observe that most of the high ranks go to our approach, which indicates that our solution achieves the best visual quality and relevance to the text input.
Conclusion
We presented a novel approach for generating new clothing on a wearer based on textual descriptions. We designed two task-specific GANs, the shape and the image generators, and an effective spatial constraint in the shape generator. The generated images are shown to contain precise regions that are consistent with the description, while keeping the body shape and pose of a person unchanged. Quantitative and qualitative results outperform the baselines.
The results generated are limited by the current database we adopted. Our training set contains images mostly with a plain background as they were downloaded from on-line shopping sites (i.e., http://www.forever21.com/). Hence the learning model is biased towards such a distribution. In fact we do not assume any constraints or post-processing of the background. We believe that our model can also render textured background if the training set contains more images with textured background. The background distribution will be captured by the latent vector .
Acknowledgement: This work is supported by SenseTime Group Limited and the General Research Fund sponsored by the Research Grants Council of the Hong Kong SAR (CUHK 14224316, 14209217).