3D Guided Fine-Grained Face Manipulation

Introduction

Face manipulation, a problem involving changing the facial expressions in images enables many creative applications. Until very recently, this problem was mainly addressed from a graphical perspective in which a 3D Morphable Model (3DMM) was first fitted to the image and then re-rendered with a different facial expression. Such techniques jointly model both the shape and the appearance and are typically trained using spatially aligned 3D scans of people . A desired facial expression can then be generated by combining graphical primitives called blendshapes . The blendshapes often correspond to the Facial Action Coding System (FACS) which defines a set of anatomically related muscle activations. Unfortunately, due to the Gaussian assumption, 3DMMs often produce blurry shapes and appearances, preventing realistic face rendering.

Deep generative techniques offer a different way of solving the face manipulation problem. In contrast to 3DMMs, they learn an internal representation that jointly models the shape and the appearance of the faces. Manipulation is then performed by conditioning the decoder on expression labels or latent vectors . This solution is sub-optimal in several respects. First, neural networks have been recently shown to have difficulties in generating simple geometric transformations , whereas face manipulation involves many such transformations, such as mouths opening, eyes closing and other transformations. Second, their models require many examples of such transformations along with their intensities at the training time, which becomes even more problematic for less common expressions such as sad-smile or negative-surprise. Third, each model supports only a small set of manipulation operations, not allowing fine-grained 3D manipulation.

In this paper we present a novel method that combines 3DMMs and deep generative techniques in a single framework for fine-grained face manipulation. Randomly selected qualitative samples produced by our method are given in Fig. 1. Given a face image, we first fit a 3D face model on the image to obtain the texture and the shape. The shape is further represented as identity and expression coefficients using a bilinear model . This way we disentangle the shape and the texture spaces and use separate branches in our pipeline to apply transformations in these spaces.

The texture branch consists of a convolutional neural network and assumes the texture and the desired expression as inputs, producing a new texture which corresponds to the desired expression. Due to the difficulties of the convolutional networks in generating geometric transformations, we propose conditioning the texture branch on the UV maps that describes target geometry information instead of directly concatenating the labels as in or coefficients as in . To better preserve texture-expression consistency and the identities in the generated images, we design corresponding loss functions for improved results.

The shape branch is implemented using a fully connected neural network taking the identity and the expression coefficients as inputs and outputting shape deformation necessary to accurately match the desired expression. Notably, a common problem in fitting a morphable model to the face is its inability to fully capture the face shape given only a 2D RGB input image sparsely labeled with 2D landmarks. This is often called face shape hallucination . At training time, to improve 3D reconstruction, we additionally supervise the shape branch using the available depth data in the FaceWarehouse dataset .

The proposed approach has a number of benefits. First, we disentangle the texture and shape shapes to make it easier to learn for each branch. In the texture space, faces tend to be more similar despite significant variance in the image space caused by different poses and expressions. Therefore, the texture branch only focuses on the appearance details such as wrinkles, shadows and shading. Similarly, the shape branch focuses on the geometric details only. Second, since we represent expressions as a combination of Face Action Unit coefficients , rather than discrete labels, our approach can generate infinite number of target expressions. Third, we further distinguish identity and expression coefficients, to better preserve subject-specific features by only changing the expression components in the shape space.

We compare the proposed method to the most recent face manipulation methods and show that our approach is superior in all the experiments. In the user studies that we conducted, the presented method is preferred more than 85% of the time when compared with the existing works. When compared to the ground truth testing images, our method is preferred in 53% of cases, supporting that it is difficult for a human to distinguish real images from those generated by our method.

Related Work

We review relevant geometry based methods and deep generative methods for face manipulation.

Geometry-based methods. A pioneering work of Blanz and Vetter presented the first public 3D Morphable Model (3DMM). They densely captured surface geometry and color data of 200 identities and created a linear model to represent the face variations of different subjects using principal component analysis (PCA). Vlasic et al. proposed a multilinear model of facial expressions for tracking and re-targeting. Cao et al. proposed FaceWarehouse, an extensive facial expression database, which contains 47 different facial expressions for each of the 150 subjects. This dataset later became one of the most adopted datasets for 3D face fitting and animation .

In , they first fitted a 3D face shape to match the input image, and then changed the expression coefficients to perform animation by warping the image to a new expression. Thies et al. presented Face2Face for real-time video-to-video facial expression re-targeting. They first fit a 3DMM together with lighting parameters and re-render it in the target video. Although these geometry-based methods produce convincing results of large-scale motions, they are unable to model parts not existing in the source image, such as teeth when the mouth is closed, and resort to rendering such parts using conventional graphics approaches. Therefore, these methods often fail to achieve realistic results, as humans are especially sensitive to non-realistic artifacts in faces.

Deep generative methods. Face manipulation can be viewed as the unpaired image-to-image translation problem . Until very recently, one had to train a separate model, attribute-by-attribute to perform face manipulation . Lample et al. proposed to additionally control the intensity of the attribute. Their work can change two attributes at the same time, but only at the cost of reduced image quality. Choi et al. used conditional image-to-image translation to allow multiple attributes to be trained together in an unsupervised fashion. These attributes can include gender, age, hair color, expression and so on. Despite the impressive results, their approach is still limited to a finite number of attributes, preventing fine-grained manipulation. Several video generation methods for face animation were proposed. Given a face image, such methods perform video prediction or motion transfer to manipulate faces. Recently, Pumarola et al. presented a work performing anatomically-aware face animation. Similarly to us, they animate faces according to Facial Action Units.

The method presented in this paper is different than geometry-based and deep generative methods in that it combines the benefits of both lines of work in a single end-to-end trainable framework. As opposed to purely 3DMM-based methods and similarly to deep generative works, our framework features high quality face texture synthesis. In contrast to deep generative works, and similarly to 3DMMs-based methods, our approach can generate arbitrary number of facial expressions. A key difference with Pumarola et al. is that we learn to explicitly disentangle shape and appearance into different branches. This enables learning a rich face prior from our shape branch, and allows the texture branch to focus on synthesizing realistic images.

Method

Our pipeline is shown in Fig. 2. The approach requires a face image and the desired expression encoded by coefficients. We first fit the 3D face shape and camera projection matrix from the image, with which we extract textures (Sec. 3.1). Then, we input the texture and the target expression to the texture branch and generate the target texture containing the details of the desired expression (Sec. 3.2). As the 3DDM-based shape representations are often inaccurate, we use a fully connected network in the shape branch to predict a more accurate shape for improved synthesis quality (Sec. 3.3). The predicted texture and shape are then combined and rendered to obtain a target image. We then use the global branch network on the target image to further improve the quality (Sec. 3.4).

Face fitting is the process of estimating the 3D face shape and the camera projection matrix given an input face image. Following , we represent the 3D face shape using a bilinear model as:

The inaccuracy in the fitting process causes the extracted textures to be misaligned and thus introduces additional variance for neural network to learn. To tackle this, we make use of the depth data when it is available. For the input image with a depth map (Fig. 3c), we minimize the L2 distance between the shape vertices and its closest 3D depth points and then refine the shape using (Fig. 3d). When the depth is not available, we deform the shape to further reduce the landmark errors as in .

We define a 2D UV coordinate for each 3D shape vertex, consistent across the dataset. The textures are extracted with the UV coordinates, camera projection and fitted 3D shape using the standard rasterization pipeline. (Fig. 3e).

2 Texture Branch

Input format. Typically the generator $G$ is modeled as a convolutional neural network. In our case, the generator needs to take both the texture image $\mathbf{T}$ and the expression coefficients $\mathbf{e}$ as input. A straightforward approach to combine these different formats is to concatenate each element of $\mathbf{e}$ as a separate feature map to the input image $\mathbf{T}$ as in . We argue that converting the geometry information of $\mathbf{e}$ into a spatial representation, such as a UV-map, helps better utilize local convolutional operations learned by the texture branch.

In our implementation, this information includes object space normals, deformation, curvature, position difference, normal difference and semantic labels. We show examples in Fig. 4. Normal determines the local surface orientation which is considered important in shading. Deformation is determined by the ratio of the one-ring area near each vertex in the target and neutral expressions, where a small deformation value means compression and can be associated with wrinkles. Curvature differentiates bumped regions from flat regions. Position and normal differences imply similarities between source and target expressions near each vertex, indicating the likelihood of the output pixel resembling the input pixel at the same location. Furthermore, to address the translational equivariance issue of convolutions , semantic labels are used to indicate different facial components which should be synthesized differently. These labels include eyes, eyebrows, nose, lips and inner mouth and others. As all the shapes have the fixed layout in the UV space, we manually define the labels on the 3D mesh and rasterize them to get the semantic map. We then use this semantic map for all the samples. We evaluate the effectiveness of our input format in Sec. 5.1.

3 Shape Branch

The 3D face shape $\mathbf{S}$ is a non-linear function of the expression coefficients due to the complex interaction of muscles, flesh and bones. Previous works model this complex interaction linearly. Although this method is simple and widely adopted, we argue that these limited expression models can only represent the large-scale motion, and struggle to capture the fine-grained details.

4 Global Branch

We use the predicted texture $\hat{\mathbf{T}}$ and shape $\hat{\mathbf{S}}$ to render the predicted face on the image. The goal of the global branch is to blend this face into the background seamlessly. We show the process in Fig. 6. We first make the artificial margin between the rendered face and the background and train a network to hallucinate in between. The margin is computed using a dilation approach with kernel size 12. To fill in the margin, one could use image inpainting techniques . We have a simpler problem since the input image is usually similar to the background image. Therefore, we use the global network that takes the input image, the rendered face and the region outside of the margin as input. The network then learns to blend the generated face and the background together. Occasionally this still produces artifacts near the boundary. Therefore at test time, we apply image blending with the input image as a post-processing step. We describe this step in more details in the supplemental materials.

Implementation Details

Datasets. Our datasets include FaceWarehouse and Chicago Face Dataset (CFD) . For the training set, we use 493 identities from FaceWarehouse, each with at least 20 different expressions and 152 identities from CFD, each with at least 5 expressions. Among this data, 140 identities in Facewarehouse have depth. For the test set, we use 87 identities from Facewarehouse and 5 identities from CFD. Our datasets span different genders and skin colors. We use $256\times 256$ for our image and texture resolution. To further increase the resolution multistage generative models can be employed .

Network architecture. The texture and global branch generators adopt pix2pix architecture with attention maps . We change the transposed convolutions to upsampling layers followed by 3x3 convolutions. Similarly our discriminators adopt the pix2pix discriminator architectures. See the supplement for more details.

Training. We use Adam optimizer with a learning rate of 0.0001, $\beta_{1}=0.5$, $\beta_{2}=0.9$. We first train the texture branch and the shape branch. Then we fix their weights and train the global branch. We use a single NVIDIA Tesla V100 GPU and we train for 5 days to get the best results.

Experiments

In this section, we first conduct an ablation study to evaluate the design choices in our system. Next, we compare our approach with other approaches both qualitatively and quantitatively. Finally, we show additional qualitative results.

Texture branch input format. We compare our proposed input format with directly concatenating expression coefficients to the input of the neural network as in . We show that our approach generalizes better by transferring an image from CFD to a rare expression that CFD rarely covers in Fig. 7. The model (top row) which appends expression coefficients directly as input fails to generate the correct appearance for regions like the inner mouth, cheeks near mouth corners and lips. This occurs since the generator has rarely seen the combination of this face skin color with these specific coefficients in the training dataset. The proposed approach, which conditions on the texture branch on the spatial representation of geometry information, generalizes better. We believe our approach better uses the local convoluational structure of the neural network.

Shape branch. We demonstrate that our shape branch generates more realistic shapes than the linear blendshapes both quantitatively and qualitatively. We first compute the root mean square error (RMSE) between the generated face mesh and our ground truth fitted face mesh in Table 1. After being deformed by our shape branch, the predicted mesh is closer to the ground truth.

We also show an example demonstrating the change of the mesh in Fig. 9. Without the shape branch, the fitted linear blendshapes tend to open the jaw more widely, which looks less natural, while our shape branch learns to close the jaw, such that the shape gets closer to the ground truth.

Note that despite the obvious benefits of the texture shape decoupling, our carefully designed input format, loss functions and shape branch are necessary for best results.

2 Comparisons

We compare our face manipulation results to the direct texture mapping approach , StarGAN and GANimation . The method in is a linear model combined with a computer graphics rendering approach, which also separates the texture and the shape but does not alter the texture. The latter two train on image space only concatenating the attributes or action units directly to the input. To evaluate the effectiveness of different methods at handling wide ranges of extreme expressions, we choose FaceWarehouse as our training set because it contains many challenging expressions other datasets do not normally cover. We use the 87 identities in our test set as mentioned in Sec. 4 and we do not include CFD for easier comparisons. We trained StarGAN using $20$ different expressions as attributes. We implemented GANimation with all the attention mechanisms and loss terms, except that we replaced the regressor with a classifer in their discriminator, which tends to give better results on our dataset. For all the comparison experiments, we transfer neutral expressions to different expressions. We use the real captured data from as the ground truth.

Qualitative study. We show several examples in Fig. 10. Direct texture mapping is not able to generate wrinkles in the smiling expression, teeth in mouth opening expression or correct shading details in mouth blowing expression. For StarGAN and GANimation, we observe that they tend to produce more artifacts in expressions that have larger scale facial movements like mouth opening and mouth sloping. We hypothesize that this is because the competing approaches need to learn a complex model with all the rigid pose, shape and appearance variance together, while our fitting process and shape branch take the first two away, leaving a simpler function for the texture branch to learn. We also find that GANimation sometimes leaves the details from the input image in the output. Interested readers can magnify the lips region on the 4th column and eyebrows region in the 5th column to see the artifacts. We hypothesize that this is a problem caused by the attention mechanism in the image space. Our approach has a fixed texture layout and thus does not have this problem.

Note that our synthesized images have different camera poses than the ground truth. This is because FaceWarehouse is captured with different head poses and our images are cropped differently based on the face sizes. Also note that the synthesized images look different from the ground truth. This is because there are numerous ways that a person can perform an expression and our method only generates a possible realization of that expression.

Quantitative study. We adopt Average Content Distance (ACD) from to evaluate how well identities are preserved using different methods. We extract feature vectors from each synthesized image and compute the $\mathcal{L}_{2}$ distance to the feature vector of the input image. We show the results in Table 2. Our method gives the best results besides ground truth. Note that we do not optimize with respect to any pretrained face recognition networks at training time. We attribute our lower ACD to our disentanglement representation of texture and shape, which makes it easier to preserve identities.

User study. We perform a user study on Amazon Mechanical Turk (AMT), where each worker is presented with the reference image, an image synthesized by our method and an image synthesized by a competing method. We ask the turkers to evaluate the synthesized images based on their quality, realism of the expression, and similarity to the reference image. Since faces in ground truth images have different poses, for comparison with ground truth we only ask the subjects to evaluate based on the quality of the image and expression, eliminating other irrelevant factors as much as possible. For each comparison, we have $1,740$ pairs of images and each pair is evaluated by 3 workers. We only accepted turkers with a lifetime HIT approval rate $\geq 95\%$. We show the results in Table 2. Users prefer our methods over all other methods. We get a slightly higher preference score than ground truth. This proves that it is difficult for humans to distinguish between the images generated by our method and the ground truth.

3 More Results

Different input expressions. Our method can handle input expressions that are not neutral. We show synthesized images using the same person with different expressions as input in Fig. 11. Although our input images are different, our synthesized images with the same target expressions still look similar. We also note that the method can generate a different version of each expression for each subject.

Images in the wild. We show examples of our method applied to images in-the-wild in Fig. 1 and refer the reader to the supplementary materials for more in-the-wild results. Due to the decoupled face representation and separate texture and shape branches, our method is robust to different identities, expressions, head poses or lighting.

Conclusion

We presented a 3D guided fine grained face manipulation approach to transfer from one arbitrary expression to another arbitrary expression. The method decomposes an image into shape and texture spaces, followed by processing of these spaces with separate branches. We showed the benefits of such a scheme. Conditioning the pipeline on the spatial representation of important geometry information is advantageous over the straightforward approach of directly appending expression coefficients. To further boost the quality, we introduced several of the loss functions accounting for the pairwise consistency and identity. Our ablation studies supported the proposed framework. Furthermore, our method showed a significantly better ACD score as well as a preference by human annotators when compared to the competing approaches. Finally, when compared to the real images, the annotators were not able to distinguish our generated images from the real images, fully supporting the benefits of the presented method.

Acknowledgements. This work was mainly done when the first author Zhenglin Geng, who interned at Snap Inc. We also thank Davis Rempe, Rahul Sheth and Aletta Hiemstra for their help with the paper revision.

References

Appendices

Appendix A More Results From CFD

We present more results from CFD test set in Figure 12.

Appendix B More Results From FaceWarehouse

We present more results from Facewarehouse test set in Figure 14.

Appendix C More Results from Images in the Wild

Though our training set only contains images captured in a lab setting with frontal faces and uniform lighting, we show that our trained model can work on more challenging in-the-wild images in Figure 15.

Appendix D More Results on Continuous Editing

As we use expression coefficients as conditions, we can trivially manipulate faces continuously. We show more results on continuous editing in the submitted video.

where we multiply the first term by 2 to prevent the discriminator from simply producing a small value.

Appendix F Network Architectures

For our texture branch generator, we use pix2pix and attention map . For our texture branch input, we concatenate input texture(3), normal(3), area deformation(1), curvature(1), normal difference(3), position different(3) and noise (1) together, thus the total number of channels for our input is 15. For our output, we use a separate attention map for each R,G,B channel, therefore the number of channels for our output is 6. To avoid being saturated in the gradients, we do not use any sigmoid or tanh activations. For the hidden layers in the middle, we use the UNet structure with skip link. For the encoder, we use convolutional layers with filter size 4, stride 2 and padding 1 for downsampling. For the decoder, we use bilinear upsampling followed by a convolutional layer with filter size 3, stride 1 and padding 1 for upsampling. Following the notation from , we use $Ck$ denote Convolution-BatchNorm-ReLU layer with $k$ filters. encoder: $C32-C64-C64$ decoder: $C64-C32-C6$ All ReLUs in the encoder are leaky, with slope 0.2. All ReLUs in the decoder are not leaky.

Appendix G More Details in the Global Branch

The goal of the global branch is to blend the rendered image seamlessly into the background. We first generate an margin by calling the OpenCV dilate function with a kernel size of 12. The our global branch takes the rendered face, input image and regions outside of the margin as input to hallucinate inside. Sometimes there is still an observable boundary in which case we apply image blending. We blend the image based on the vertex distance $d$ from the source expression mesh to the target expression mesh. The blending alpha is determined heuristically as $\exp(d^{2}/4)$.

Appendix H More Comparison with Texture Mapping

We show more examples of the difference between our approach and direct texture mapping approach. To manipulate expression in the image, one can change only the underlying shape without substantially changing the texture like . However this can result in many artifacts, especially when the source and target expressions significantly differ. For example, in Fig. 13, if one directly uses the texture extracted from the source image and renders it with a smiling face shape, the missing crease and teeth and image distortion make the result less realistic. Our texture branch learns to reconstruct these missing parts and change the local appearance near the eyes, which makes the resulting image look natural.