Learning Residual Images for Face Attribute Manipulation

Wei Shen, Rujie Liu

Introduction

Considerable progresses have been made on face image processing, such as age analysis , emotion detection and attribute classification . Most of these studies concentrate on inferring attributes from images. However, we raise an inverse question on whether we can manipulate a face image towards a desired attribute value (i.e. face attribute manipulation). Some examples are shown in Fig. 1.

Generative models such as generative adversarial networks (GANs) and variational autoencoders (VAEs) are powerful models capable of generating images. Images generated from GAN models are sharp and realistic. However, they can not encode images since it is the random noise that is used for image generation. Compared to GAN models, VAE models are able to encode the given image to a latent representation. Nevertheless, passing images through the encoder-decoder pipeline often harms the quality of the reconstruction. In the scenario of face attribute manipulation, those details can be identity-related and the loss of those details will cause undesired changes. Thus, it is difficult to directly employ GAN models or VAE models to face attribute manipulation.

An alternative way is to view face attribute manipulation as a transformation process which takes in original images as input and then outputs transformed images without explicit embedding. Such a transformation process can be efficiently implemented by a feed-forward convolutional neural network (CNN). When manipulating face attributes, the feed-forward network is required to modify the attribute-specific area and keep irrelevant areas unchanged, both of which are challenging.

In this paper, we propose a novel method based on residual image learning for face attribute manipulation. The method combines the generative power of the GAN model with the efficiency of the feed-forward network (see Fig. 2). We model the manipulation operation as learning the residual image which is defined as the difference between the original input image and the desired manipulated image. Compared to learning the whole manipulated image, learning only the residual image avoids the redundant attribute-irrelevant information by concentrating on the essential attribute-specific knowledge. To improve the efficiency of manipulation learning, we adopt two CNNs to model two inverse manipulations (e.g. removing glasses as the primal manipulation and adding glasses as the dual manipulation, Fig. 2) and apply the strategy of dual learning during the training phase. Our contribution can be summarized as follows.

We propose to learn residual images for face attribute manipulation. The proposed method focuses on the attribute-specific face area instead of the entire face which contains many redundant irrelevant details.

We devise a dual learning scheme to learn two inverse attribute manipulations (one as the primal manipulation and the other as the dual manipulation) simultaneously. We demonstrate that the dual learning process is helpful for generating high quality images.

Though it is difficult to assess the manipulated images quantitatively, we adopt the landmark detection accuracy gain as the metric to quantitatively show the effectiveness of the proposed method for glasses removal.

Related Work

There are many techniques for image generation in recent years . Radford et al. applied deep convolutional generative adversarial networks (DCGANs) to learn a hierarchy of representations from object parts to scenes for general image generation. Chen et al. introduced an information-theoretic extension to the GAN that was able to learn disentangled representations. Larsen et al. combined the VAE with the GAN to learn an embedding in which high-level abstract visual features could be modified using simple arithmetic.

Our work is an independent work along with . In , Li et al. proposed a deep convolutional network model for identity-aware transfer of facial attributes. The differences between our work and are noticeable in three aspects. (1) Our method generates manipulated images using residual images which is different from . (2) Our method models two inverse manipulations within one single architecture by sharing the same discriminator while the work in treats each manipulation independently. (3) Our method does not require post-processing which is essential in .

Learning the Residual Image

The architecture of the proposed method is presented in Fig. 2. For each face attribute manipulation, it contains two image transformation networks $G_{0}$ and $G_{1}$ and a discriminative network $D$ . $G_{0}$ and $G_{1}$ simulate the primal and the dual manipulation respectively. $D$ classifies the reference images and generated images into three categories. The following sections will first give a brief introduction of the generative adversarial network and then the detailed description of the proposed method.

The generative adversarial network is introduced by Goodfellow et al. . It is an unsupervised framework containing a generative model $G$ and a discriminative model $D$ . The two models play a minimax two-player game in which $G$ tries to recover the training data and fool $D$ to make a mistake about whether the data is from realistic data distribution or from $G$ . Given a data prior $p_{data}$ on data $x$ and a prior on the input noise variables $p_{z}(z)$ , the formal expression of the minimax game is as follows:

The parameters of both $G$ and $D$ are updated iteratively during the training process. The GAN framework provides an effective way to learn data distribution of realistic images and makes it possible to generate images with desired attribute. Based on the GAN framework, we redesign the generator and the discriminator for face attribute manipulation in the following sections.

2 Image Transformation Networks

The motivation of our approach is that face attribute manipulation usually only needs modest modification of the attribute-specific face area while other parts remain unchanged. For example, when removing a pair of glasses from a face image, only the area of the glasses should be replaced with face skin or eyes while the change of other face parts such as mouth, nose, and hair should not be involved. Thus, we model the manipulation as learning the residual image targeted on the attribute-specific area.

In order to let the residual image be sparse, we apply an L-1 norm regularization as

3 The Discriminative Network

where $t$ is the label of the image and $p_{t}$ is the softmax probability of the $t$ -th label. Similar strategy for constructing the GAN loss is also adopted in .

Perceptual loss is widely used to measure the content difference between different images . We also apply this loss to encourage the transformed image to have similar content to the input face image. Let $\phi(x)$ be the activation of the third layer in $D$ . The perceptual loss is defined as:

Given the discriminative network $D$ , the GAN loss for the image transformation networks $G_{0}$ and $G_{1}$ is

4 Dual Learning

In addition to applying adversarial learning in the model training, we also adopt dual learning which has been successfully applied in machine translation . A brief introduction is as follows. Any machine translation has a dual task, i.e. the source language to the target language (primal) and the target language to the source language (dual). The mechanism of dual learning can be viewed as a two-player communication game. The first player translates a message from language A to language B and sends it to the second player. The second player checks if it is natural in language B and notifies the first player. Then he translates the message to language A and sends it back to the first player. The first player checks whether the received message is consistent with his original message and notifies the second player. The information feedback signals from both players can benefit each other through a closed loop.

5 Loss Function

Taking the loss functions all together, we have the following loss function for $G_{0}$ / $G_{1}$ :

where $\alpha$ and $\beta$ are constant weight for regularization terms.

Datasets

We adopt two datasets in our experiments, i.e. the CelebA dataset and the Labeled Faces in the Wild (LFW) dataset . The CelebA dataset contains more than 200K celebrity images, each with 40 binary attributes. We pick 6 of them, i.e. glasses, mouth_open, smile, no_beard, young, and male to evaluate the proposed method. The center part of the aligned images in the CelebA dataset are cropped and scaled to 128 $\times$ 128. Despite there are a large number of images in the dataset, the attribute labels are highly biased. Thus, for each attribute, 1,000 images from the attribute-positive class and 1,000 images from the attribute-negative class are randomly selected for test. From the rest images, we select all the images belong to the minority class and equal number of images from the majority class to make a balance dataset. The LFW dataset is used only for testing the generalization of our method. Note that there are no ground truth manipulated images in the CelebA dataset for training the transformation networks.

Implementation Details

The detailed architectures of $G_{0}$ , $G_{1}$ and $D$ are specified in Tab. 1. We keep $\beta=0.1\alpha$ and set $\alpha$ =5e-4 for local face attribute (i.e. glasses, no_beard, mouth_open, smile) manipulation and $\alpha$ =1e-6 for global face attribute (i.e. male, young) manipulation. The weights of all the networks are initialized from a zero-centered Normal distribution with standard deviation 0.02. The Adam optimizer is used in the training phase. The learning rates for both the transformation networks and the discriminators are the same 2e-4. Both $G_{0}$ and $G_{1}$ are trained at the same time without any staging.

Experiments

Among the six attributes, we group glasses, mouth_open, smile and no_beard as local attributes since the manipulations only operate on local face area. The other two attributes male and young are treated as global attributes. We compare the results of our method and those of the state-of-the-art method VAE-GAN on the CelebA dataset (Fig. 4). The results on the LFW dataset are presented in Fig. 5.

We firstly give an overall observation of the results. As shown in Fig. 4, the VAE-GAN model changes many details, such as hair style, skin color and background objects. In contrast, the results from our method remain most of the details. Comparing the original images in the first row and the transformed images in the third row, we can find that details in original face images mostly remain the same in their manipulated counterparts except the areas corresponding to the target attributes. This observation is also proved by the residual images in the last row. For local attribute manipulation, the strong responses on the residual images mainly concentrate in local areas. For example, when adding sun glasses to the face image, the most strong response on the residual image is the black sun glasses. Similarly, removing glasses will cause the residual image to enhance the eyes and remove any hint of glasses that is presented in the original face image.

Local face attribute manipulations are straightforward and obvious to notice. Further, we investigate more interesting tasks such as mouth_open and smile manipulations. Both manipulations will cause the “movement” of the chin. From Fig. 4(c,d), we can observe that the “movement” of the chin is captured by the image transformation networks. When performing the mouth_open manipulation, the network “lowers” the chin and when performing mouth_close manipulation, the network “lifts” the chin.

The most challenging task would be manipulating global attributes young and male. The networks have to learn subtle changes such as wrinkles, hair color, beard etc. In Fig. 4(f), changing from young to old will cause more wrinkles and the dual operation will darken the hair color. From the residual images in Fig. 4(e), we observe that the main difference between the male and the female are the beard, the color of the lips and the eyes. The strong responses in the residual images for these two manipulations are scattered over the entire images rather than restricted within a local area.

2 Ablation Study

Our model consists of two pivotal components: residual image learning and dual learning. In this section, we further validate their effectiveness. We modify the proposed model to obtain two more models. One breaks the identity mapping in the transformation networks to enforce the networks to learn to generate the entire image. The other breaks the data-feed loop (i.e. the output of $G_{0}$ and $G_{1}$ will not be fed to each other). Other network settings are kept the same as those of the proposed model. We use the manipulation of glasses as an example and the results are shown in Fig. 6. We observe that without residual image learning, the model produces much lower quality images in terms of introducing much noise and some correlated features (e.g. wrongly added beard in the second and third column), which indicates the task has become challenging. The drop of dual learning also deteriorates image quality. We notice some change in hair color which is caused by the performance degradation of the transformation networks. The effectiveness of the dual learning can be explained from two aspects. 1) Images generated from both generators increase the number of training samples. 2) During the dual learning phase, the ground truth images for $G_{1}(G_{0}(x_{0}))$ and $G_{0}(G_{1}(x_{1}))$ are known, which eases the training of both generators. Thus, we argue that combining residual image learning with dual learning will lead to better manipulation results.

3 Visual Feature Decorrelation

Training a classifier in an end-to-end way does not ensure the classifier can precisely identify the target visual features especially when the dataset is highly biased. Blind spots of predictive models are observed in . For example, if the training data consist only of black dog images, and white cat images. A predictive model trained on these data will incorrectly label a white dog as a cat with high confidence in the test phase. When analyzing the CelebA dataset, we find that male and no_beard is highly correlated (the Pearson correlation is -0.5222). It is not surprising since only male faces have beard.

Classifiers trained with correlated features may also propagate the blind spots back to the generative models which may cause the generators to produce correlated visual features. To demonstrate that the proposed method can learn less correlated features, we choose to add beard to female face images. Since there are no images containing female faces wearing beard in the training data, this manipulation may introduce other correlated male features. We compare the manipulation results from our method with those from the VAE-GAN model in Fig. 7. We show the VAE-GAN reconstruction results of the original images to ensure that the VAE-GAN model do learn well about the original images. However, the hair length in the manipulated images is significantly shorter than that in the original images. This could be explained by the fact that most male faces wear short hair and this visual feature is correlated with the beard feature. The VAE-GAN model incorrectly treats the short-hair feature as the evidence of the presence of the beard. However, the proposed method successfully decorrelates these two features. The hair in the transformed images is almost the same as that in the original images. We owe this appealing property to residual image learning and dual learning that help the method concentrate on attribute-specific area.

4 Landmark Detection with Glasses Removal

Besides visually inspecting the manipulation results, we quantify the effectiveness of glasses removal by the performance gain of face landmark detection accuracy. The landmark detection algorithm is the ERT method implemented with Dlib . We trained the detection model on the 300-W dataset . Three test sets are included into consideration. They are dataset $D1$ containing images wearing glasses, dataset $D0$ containing images without glasses and dataset $D1_{m}$ containing the same images as $D1$ while those images are processed with glasses removal using the proposed method. Note that $D0$ and $D1$ are the same test sets as those in the experiment of glasses manipulation.

The detection results are reported in Tab. 2 and illustrated in Fig. 8. Although the identities in $D0$ and $D1$ are different, we can still find that wearing glasses affects the landmark detection. Comparing the results of the eye landmark detections between the first and the second column, we find the detection error increases. However, the error on $D1_{m}$ is much lower than that on $D1$ which demonstrates the benefit of applying glasses removal for eye landmark detection. Comparing the results in the second row in Tab. 2, we observe that the errors on $D1$ and $D1_{m}$ are almost the same which indicates that the rest parts of the face remain almost unchanged.

Discussion

The hyper-parameters $\alpha$ and $\beta$ are manually set in this work. We empirically found $\alpha$ =5e-6 and $\alpha$ =5e-4 are appropriate for global and local attribute manipulation respectively. Larger $\alpha$ and $\beta$ will prevent the generative models from simulating the manipulations and will collapse the models to 0. The adoption of only one discriminator in our work is mostly for improving the computation efficiency. There could be identity overlaps between the training set and the test set since the number of identities is much smaller than the number of all images and images for testing are also randomly selected. However, even the face identities could be the same, the face poses, illumination conditions, and the background objects can hardly be the same. Therefore, there is little information leak caused by identity overlaps to the test set. In Fig. 9, we show some failure cases in glasses removal. Those cases are mainly due to large poses.

Conclusion

In this work, we present a GAN based method to tackle the task of face attribute manipulation. We adopt both residual image learning and dual learning in our framework. Those strategies allow image transformation networks to focus on attribute-specific area and learn from each other. Experiments demonstrate that the proposed method can successfully manipulate face images while remain most details in attribute-irrelevant areas unchanged.