Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks

Introduction

Reconstructing the 3D facial shape and texture from a single image is a vital problem in computer vision and graphics. The seminal work of Blanz and Vetter illustrates the power of the 3D Morphable Model (3DMM), which is a statistical model of facial shape and texture built from hundreds of scanned faces. The 3DMM and its variants make it possible to recover the facial shape and albedo from a single image . However, the fidelity of the texture recovered from the 3DMM coefficients is still not high enough. It is mainly because that the texture computed from 3DMM cannot capture the face details of the input image, especially for the images in-the-wild.

In order to tackle the problem, recently there have been some work trying to reconstruct high-quality textures from 2D images. For example, Deng et al. learn a generative model to complete the self-occluded regions in the facial UV map. Gecer et al. first utilize GANs to train a generator of facial textures in UV space and then use non-linear optimization to find the optimal latent parameters. While they can achieve high fidelity textures, their methods require a large-scale database of high-resolution UV maps, which is not publicly available. Besides, capturing such a database is rather laborious, which is infeasible for ordinary users.

In this paper, we seek to reconstruct the 3D facial shape with high fidelity texture from a single image, without the need to capture a large-scale face texture database. To achieve this goal, we make a key observation that, though lacking details, the 3DMM texture model can provide a globally-reasonable color for the whole face mesh. We can further refine this initial texture by introducing the facial details from the image to the face mesh. To this end, we propose to reconstruct the detailed colors for the mesh vertices instead of reconstructing the UV map. In particular, we utilize graph convolutional networks (GCN) to decode the image features and propagate the detailed RGB values to the vertices of the face mesh.

Our reconstruction framework works in a coarse-to-fine manner, based on a combination of a 3DMM model and GCNs. A convolutional neural network (CNN) is trained to regress 3DMM coefficients (identity/expression/texture) and rendering parameters (pose/lighting) from a 2D image. With the 3DMM model, the face shape and initial coarse texture can be computed with an affine model. At a key stage, we utilize a pre-trained CNN to extract face features from the image and feed them to a graph convolutional network to produce detailed colors for the mesh vertices. Our framework adopts a differentiable rendering layer to enable self-supervised training, and further improves the results with a GAN loss .

To summarize, this paper makes the following contributions:

We propose a coarse-to-fine framework for reconstructing 3D faces with high fidelity textures from single images, without requiring to capture a large scale of high-resolution face texture data.

To the best of our knowledge, we are the first to use graph convolutional networks to generate high fidelity face texture, which is able to generate detailed colors for the mesh vertices from the image.

We compare our results with state-of-the-art methods, and ours outperform others in both qualitative and quantitative comparison.

Related Work

Blanz and Vetter introduced the first 3D morphable face model twenty years ago. Since then, there have been multiple variations of 3DMM . Those models produce low-dimensional representations for the facial identity, expression and texture from multiple face scans using PCA. One of the most widely used, publicly available variants of 3DMM is the Basel Face Model (BFM) . It registers a template mesh to the scanned faces with the Optimal Step Nonrigid ICP algorithm, then employs PCA for dimensionality reduction to construct the model. We will use this model as our 3DMM model in our experiments.

In a 3DMM, given the identity coefficients $c_{i}$, expression coefficients $c_{e}$ and texture coefficients $c_{t}$, the face shape $S$ and the texture $T$ can be represented as:

where $S_{mean}$ and $T_{mean}$ are the mean face shape and texture, and $I_{base}$, $E_{base}$ and $T_{base}$ are the PCA bases of identity, expression and texture respectively. We use $S_{mean}$, $T_{mean}$, $I_{base}$ and $T_{base}$ from BFM , and $E_{base}$ built from FaceWarehouse .

2 Fitting a Morphable Face Model

A classical approach for 3D face reconstruction from a single image is to iteratively fit a template model to the input 2D image. However, this approach is sensitive to the lighting, expression, and pose of the 2D face image. Some improvements have been made to improve the fitting stability and accuracy, but they don’t perform well on images in the wild.

Deep learning based methods directly regress the 3DMM coefficients from images. To obtain paired 2D-3D data for supervised learning, Richardson et al. generate synthetic data by random sampling from the morphable face model, but this approach may not perform well when dealing with complex lighting, occlusion, and other in-the-wild conditions. Tran et al. do not generate synthetic data directly. Instead, they create the ground truth using an iterative optimization to fit a large number of face images. Nevertheless, the problem of being delicate in uncontrolled environments still remains.

Recently, differentiable renderers has been introduced. It renders a 3D face mesh to a 2D image based on the face shape, texture, lighting and other related parameters, and compare the rendered image with the input image to compute the loss in terms of image differences. With such a fixed, differentiable rendering layer, unsupervised or weakly-supervised training is enabled without requiring the training pairs. We also follow this strategy. Specially, we adopt the differentiable renderer from Genova et al. , which is known as ”tf-mesh-renderer”, as our differentiable rendering layer.

3 Texture of Morphable Face Models

Recent deep learning based methods aim to reconstruct the facial textures from single images by regressing the 3DMM texture coefficients. For example, Deng et al. proposed a method to simultaneously predict the 3DMM shape and texture coefficients, which employs an illumination and rendering model during training, and conducts image-level and perception-level losses, leading to a better result than others. However, the texture they generated is still inherently limited by the 3DMM texture model. In our method, we deploy their scheme to train a 3DMM coefficient regression model, to get a globally reasonable face texture. Then we utilize graph convolutional networks to refine the texture with the image details.

4 Graph Convolutional Networks

To enable convolutional operations in non-Euclidean structured data, Bruna et al. utilize the graph Laplacian and the Fourier basis to make the first extension of CNNs on graphs. However, it is computationally expensive and ignores the local features. Defferrard et al. propose the ChebyNet, which approximates the spectral filters by truncated Cehbyshev polynomials, avoiding the computation of the Fourier basis. CoMA introduces mesh downsampling and mesh upsampling layers, which constructs an autoencoder to learn a latent embedding of 3D face meshes. Motivated by CoMA , our method learns a latent vector representing the detailed face color from a 2D image, then decodes it to produce detailed colors for the face mesh vertices with graph convolutional networks.

Approach

We propose a coarse-to-fine approach for 3D face reconstruction. As shown in Fig. 2, our framework is composed of three modules. The feature extraction module includes a Regressor for regressing the 3DMM coefficients, face pose, and lighting parameters, and a FaceNet for extracting image features for the subsequent detail refinement and identity-preserving. The texture refinement module consists of three graph convolutional networks: a GCN Decoder to decode the features extracted from FaceNet and producing detailed colors for mesh vertices, a GCN Refiner to refine the vertex colors generated from the Regressor, and a combiner to combine the two colors to produce final vertex colors. The Discriminator attempts to improve the output of the texture refinement module via adversarial training.

Moreover, we utilize a pre-trained FaceNet to extract a feature vector from the face image. The extracted feature serves two purposes. First, it can be treated as an image feature embedding for our Decoder to generate detailed albedo colors for the mesh vertices, which is described in Sec. 3.3. Second, it can be used to measure the identity distance in the identity-preserving loss in Sec. 3.4.

2 Differentiable Rendering

In order to train our networks, we conduct a self-supervised approach with a differentiable rendering layer. That is, we render the face mesh to a 2D image with the predicted parameters, and compute the losses based on the differences between the rendered image and the input image. We adopt the differentiable rendering layer from , which introduced a general-purpose, differentiable rasterizer based on a deferred shading model. The rasterizer computes screen-space buffers with triangle IDs and barycentric coordinates for the pixels. The colors and normals from the mesh are interpolated at the pixels. During training, the vertex normals is computed as the average of surrounding triangle normals.

Specifically, with the shape $S$, texture $T$ and pose generated from the Regressor, we can compute the face albedo projected on the input image. As the projected face albedo is not the final result, we further illuminate the face mesh with the estimated lighting and render it to get the final image $R$, which is compared with the input image to compute the loss. An example of illumination and rendering is shown in Fig. 3.

3 Texture Refinement Module

Our Decoder takes as input the feature vector from FaceNet and produces the albedo RGB values for each vertex. The Decoder architecture is built following the idea of residual networks, which consists of four spectral residual blocks. The spectral upsampling layers are placed between each two spectral residual blocks. Each spectral residual block contains two Chebyshev convolutional layers and one shortcut layer. Every Chebyshev convolutional layer uses $K=6$ Chebyshev polynomials and is followed by a biased ReLU layer .

The Refiner refines the vertex colors from the 3DMM model with spectral convolutional layers. It also contains spectral residual blocks similar to the Decoder. However, only one downsampling layer and one upsampling layer are deployed in the bottom and at the top of the network separately.

To produce the final vertex colors with details, the combination net concatenates the outputs of the Decoder and Refiner along the channel axis, and feed them into a graph convolutional layer, followed by a tanh activation layer.

We also adopt the same differentiable renderer to self-supervise the training of our texture refinement module as in Sec. 3.2. Specifically, we render the 3D face mesh to $R^{\prime}$ with illumination and compare it with the input image. To make the final texture own higher fidelity, we further utilize an adversarial training strategy. Since we do not use the real 3D face data, the Discriminator is deployed on the input 2D images and the rendered images from the reconstructed 3D face meshes. Our Discriminator contains 6 convolutional layers with kernel size 3, each of which is followed by a max-pooling layer. During training, we follow the procedure described in Wasserstein GANs with gradient penalty .

4 Losses

A straightforward objective is to minimize the differences between the input images and the rendered images. However, as the input face image may have occlusions (e.g., self-occlusion, glasses, and masks), we only compute the Euclidean distance between certain face regions $M_{face}$. The face regions are acquired by a pre-trained face segmentation networks following on Halen dataset , and we use the regions of the face, eyebrows, eyes, nose, and mouth to compute the loss. Some examples are illustrated in Fig. 4.

where $x$ is the input image, $x^{\prime}$ is the rendering image with illumination, and $M_{proj}$ denotes the region that the face mesh can be projected onto.

4.2 Identity-Preserving Loss

Using the pixel-level loss could generate a generally good result. However, the reconstructed 3D face might not ”look like” the input 2D face image, especially under extreme circumstances. To tackle this issue, we define a loss function at the face feature level, which is called the identity-preserving loss. The loss requires the features between the input image and rendered image, extracted by the FaceNet, should be close to each other. We define the identity-preserving loss as a cosine distance:

where $x$ is the input image, $x^{\prime}$ is the rendered image, and $F(\cdot)$ is the feature extraction function by FaceNet. $<F(x),F(x^{\prime})>$ is the inner product.

4.3 Vertex-wise Loss

When training the refinement module, the graph convolutional networks may not learn the texture RGB values properly from the image due to the occluded regions on the face. Therefore we construct a vertex-wise loss based on the texture predicted by the Regressor as assistance at the early stage of refinement module training, and then reduce the weights of the vertex-wise loss gradually. In order to obtain the small details from a human face, we also retrieve the vertex colors $T_{p}$ by projecting the face vertices to the 2D image, then feed them to the Refiner along with 3DMM texture RGB values. Taking those two into consideration, our vertex-wise loss can be defined as:

where $N$ is the number of vertices. $x_{i}$ is the albedo value $T$ generated by the Regressor or the retrieved vertex color $T_{p}$, and $x^{\prime}_{i}$ is the refined albedo $T^{\prime}$ from the refinement module or the illuminated $T^{\prime}$, denoted as $\widetilde{T^{\prime}}$.

4.4 Adversarial Loss

For the adversarial training, we adopt the paradigm of Improved Wasserstein GAN , whose adversarial loss is defined as:

Implementation Details

Before training, every face image has been aligned following the method of . Then we use the networks from pre-trained on the Halen dataset for face segmentation, generating a face region mask for every face image. The face image dataset we use is CelebA , and the 3D morphable face model is Basel Face Model . The Regressor network is adopted from , which predicts the 3DMM coefficients, pose and lighting parameters, and generates the face shape and coarse texture via the 3DMM model. We set the input image size to $224\times 224$ and the number of vertices to 35,709, the same as .

After getting the face shape and coarse texture, the next step is to train the texture refinement module and discriminator. The training loss is defined as:

where $\sigma_{2}=0.2$ and $\sigma_{3}=0.001$ are fixed during training. As for $\sigma_{1}$ and $\sigma_{4}$, we train the texture refinement module starting with a ”warm up” manner. Initially, we set $\sigma_{1}=0$ and $\sigma_{4}=1$. After one epoch of warm up, $\sigma_{1}$ is gradually increased to 1 and $\sigma_{4}$ is decreased to 0 accordingly.

During inference, our network can process about 50 images every second in parallel with an NVIDIA 1080Ti GPU.

Experimental Results

In this section, we demonstrate our results of the proposed framework and compare the reconstructed 3D faces with prior works. We also uploaded a video rotating the 3D face to YouTubehttps://youtu.be/ZGdkfhsd_Hw to show our results at large poses.

Fig. 5 shows our results compared to the most recent methods . The first row shows the input images and the second row shows our results. The remaining rows demonstrate the results of Deep Facial Details Nets , Deep 3D Face Reconstruction , GANFIT , and the method of Genova et al. which introduced a differentiable renderer.

It is worth to mention that in , they used their own captured data, which is not publicly available. Specifically, Chen et al. captured 366 high-quality 3D scans from 122 different subjects, and Gecer et al. trained a progressive GAN to model the distribution of UV representations of 10,000 high-resolution textures, but we only make use of the in-the-wild 2D face image dataset.

The method of used UNets to produce displacement maps for facial details synthesis, and the networks are trained semi-supervised, with labeled 3D face exploited. However, their work mostly focused on local face shape details, the global shape of the reconstructed 3D face are not as good as the locals.

With a large scale UV map dataset, can generate high fidelity textures, but the results are lack of diversity in the respect of lights and holistic colors, as the predicted lights and colors are not so accurate.

Genova et al. introduced an end-to-end, unsupervised training procedure, to learn to regress 3DMM coefficients. Deng et al. predicted the lights and face pose simultaneously with 3DMM coefficients, which generated a better result under complex lighting situations. Still, our results have more detailed information since theirs are produced by a 3DMM model, which is not able to handle in-the-wild situations very well.

2 Quantitative Comparison

To quantitatively evaluate our reconstructed 3D face, we employ the metrics on 2D images, since we do not have 3D ground-truth data. The metrics are computed between the input images and the projected images from face meshes, specifically, when the face mesh is projected to a 2D image, only the face region is projected, Fig. 6 shows an example.

Firstly, we use the $L_{1}$ distance, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), which measure the results in pixel level. Then we evaluate the results in perception level by calculating the cosine similarity of feature vectors, which are extracted by two state-of-the-art pretrained face recognition networks, LightCNN and evoLVe . We do not use FaceNet because it appears in our training pipeline, making it unsuitable as an evaluation metric. Since most 3D face reconstruction methods focus on the shape of the face, not the texture, making it difficult to find enough metrics to compare with others. Additionally, we also compare the earth mover’s distance between distributions of VGG-Face feature vector similarity from same and different identities on MICC datasets . The quantitative comparison is shown in Tab. 1.

Our results are better than others in multiple evaluation metrics. A lower $L_{1}$ distance and higher PSNR and SSIM indicate that our reconstructed 3D face textures is more close to the input images in pixel level. And both state-of-the-art face recognition networks believe that our results and the input images are more likely to be the same person.

In terms of MICC datasets, the facial meshes used in contain ears and neck region, while ours not. And we follow their procedure, only use the rendered images (third image in Fig. 6 demonstrates an example) for computing similarity score. The above factors may lead to lower scores of our method.

3 Ablation Study

Fig. 7 demonstrates the ablation study of our approach, where our full model presents a more detailed and realistic texture than its variants. Tab. 2 shows the quantitative metrics. The coarse texture is generated by the Regressor and 3DMM model, which is able to produce a basic shape and texture in general, however, the details such as lentigo or eyes are not faithfully predicted.

Starting by $L_{pix}$ and $L_{id}$, the networks are no longer restricted to a 3DMM model, and predicts higher fidelity face skin and eyes. With the help of adversarial training, the results become less blurry and more realistic. Finally, we construct our full model by adding the $L_{vert}$, the predictions contain more details, and looked very similar to the input image.

We also replace GCNs with fully-connected layers or convolutional layers on unwrapped UV space, and the performance are no better than GCNs, as shown in Fig. 7 and Tab. 2. We reach to the similar conclusion as , that using FCs or CNNs on UV spaces leads to a large number of parameters in the network and does not utilize the spatial information of the 3D facial structure.

To improve the results, we train and test our proposed networks on a higher resolution image dataset, CelebA-HQ , as well. Higher resolution helps to reduce the checkerboard-like artifacts and produce better results. A comparison is shown in Fig. 8.

Conclusions

In this paper, we present a novel 3D face reconstruction method, which produces face shapes with high fidelity textures from single-view images. To the best of our knowledge, we are the first to use graph convolutional networks to generate high fidelity face textures from the single images. Compared with other methods, our method does not require capturing high-resolution face texture datasets but can generate realistic face textures with just a 3DMM model. In the future, we will try to minimize the checkerboard-like artifacts and generate more detailed face shapes and expressions using graph convolutional networks as well.

Acknowledgments

We thank anonymous reviewers for their valuable comments. The work was supported by National Key R&D Program of China (2018YFB1004300), NSF China (No. 61772462, No. 61572429, No. U1736217) and the 100 Talents Program of Zhejiang University.

References