Generative Neural Articulated Radiance Fields

Introduction

Unsupervised learning of 3D-aware generative adversarial networks (GANs) using large-scale datasets of unstructured single-view images is an emerging research area. Such 3D GANs have recently been demonstrated to enable photorealistic and multi-view consistent generation of radiance fields representing human faces . These approaches, however, have not been shown to work with human bodies, partly because learning the body pose distribution is much more challenging given the significantly higher diversity in articulations compared to facial expressions.

Yet, generative 3D models of photorealistic humans have significant utility in a wide range of applications, including visual effects, computer vision, and virtual or augmented reality. In these scenarios, it is critical that the generated people are editable to support interactive applications, which is not necessarily the case for existing 3D GANs. While variations of linear blend skinning have been adopted to articulate radiance fields for single-scene scenarios , it is unclear how to efficiently apply such deformation methods to generative models.

With our work, dubbed generative neural articulated radiance fields or GNARF, we propose solutions to both of these challenges. Firstly, we demonstrate generation of high-quality 3D (i.e., multi-view consistent and geometry aware) human bodies using a GAN that is trained in an unsupervised manner on datasets containing single-view images. To this end, we adopt the recently proposed tri-plane feature representation , which is extremely efficient for training and rendering radiance fields, while being compatible with conventional 2D CNN–based generators, such as StyleGAN . While this framework has been successfully demonstrated for faces in prior work, we are the first to adapt it to generating radiance fields of full human bodies. Secondly, we tackle the editability of the generated radiance fields by introducing an explicit radiance field deformation step as part of our GAN training procedure. This step ensures that the generator synthesizes radiance fields of people in a canonical body pose, which is then explicitly warped according to the body pose distribution of the training data. We show that this new approach generates high-quality, editable, multi-view-consistent human bodies and that our approach can also be applied to editing faces, increasing the controllability of existing generative models for this task (see Fig. 1).

To summarize, the contributions of our approach are:

We present a 3D-aware GAN framework for the generation of editable radiance fields of human bodies. To our knowledge, this is the first approach of its kind.

Our framework introduces an efficient neural representation for articulated objects, including bodies and heads, that combines the recently proposed tri-plane feature volume representation with an explicit feature volume deformation that is guided by a template shape.

We demonstrate high-quality results for unconditional generation and animation of human bodies using the SURREAL and AIST++ datasets and faces using the FFHQ dataset.

Related Work

Parametric shape templates are one of the most common types of articulated 3D representations adopted in recent neural scene representation and rendering approaches. These templates, including faces , bodies , hands , or a combination of these parts , and even animals , have been widely utilized for pose estimation and reconstruction, e.g. . Volume deformation is also commonly used in computer graphics, for example using mean value coordinates (MVC) or biharmonic coordinates for shape deformation and editing . GNARF combines parametric template shapes, such as FLAME for heads and SMPL for bodies, with an intuitive surface-driven volume deformation approach that is both computationally efficient and qualitatively comparable to or better than both skinning and MVC-based deformation. This provides intuitive editing control for articulated radiance field deformation.

Neural Radiance Fields.

Coordinate networks, also known as neural fields , have emerged as a powerful tool that enable differentiable representations of 3D scenes and learning view-dependent neural radiance fields . While initial proposals have focused on static scenarios, recent work has demonstrated successful representations of dynamic scenes . Articulated neural radiance fields further extend these approaches by providing editability for neural representations of human heads and bodies or animals , often by deforming the underlying radiance fields using traditional 3D morphable models or skeleton-based parameterizations, or alternatively conditioning the radiance field decoder with pose-related parameters. A more detailed survey of static, dynamic, and articulated neural radiance fields can be found in the recent state-of-the-art report by Tewari et al. . Note that all of these techniques are supervised with scene-specific multi-image data and focus on representing, i.e., “overfitting”, a single scene. Therefore, it is not easily possible to train these models using unstructured 2D image data and then apply them to generate and edit new and unseen objects or humans.

Generative 3D-aware Radiance Fields.

Building on the success of 2D image-based GANs , recent efforts have focused on training 3D-aware multi-view consistent GANs from collections of single-view 2D images in an unsupervised manner. Achieving this challenging goal requires a combination of a neural scene representation and differentiable rendering algorithm. Recent work in this domain builds on representations using meshes , dense or sparse voxel grids, multiple planes , fully implicit networks , or a combination of low-resolution voxel grids combined with 2D CNN-based image upsampling layers . Our 3D GAN architecture is most closely related to the recent work by Chan et al. , which uses an efficient tri-plane-based volume representation combined with neural volume rendering. We extend this work by including an explicit deformation field in our GAN architecture to model diverse articulations, which allows the generator to synthesize radiance fields of human bodies or heads in a canonical pose while being supervised by 2D image collections that contain arbitrary pose distributions. Explicitly disentangling radiance field generation and deformation enables us to drastically improve the quality of generated human bodies and faces when their poses or facial expressions are edited.

HeadNeRF is related to our approach in that they generate 3D heads conditioned on various attributes. Both HeadNeRF and GNARF condition on identity and facial expression independently, with the former also conditioning on illumination and albedo. However, to disentangle individual attributes they need to acquire training images of the same person performing various expressions in different lighting conditions. In contrast, and similar to other 3D GANs, our approach only requires single-view images of different people and can therefore work with readily available 2D image collections. The recent work by Grigorev et al. also generates human bodies. However, their work proposes a 2D GAN that generates textures which are used in combination with a standard articulated template mesh whereas we aim at generating and editing radiance fields using a 3D GAN. The concurrent work of Noguchi et al. is closest to ours as the method also includes a radiance field deformation step in a tri-plane-based 3D GAN. Our evaluation shows superior generation quality and, more importantly, while their approach ties the network architecture to the specific choice of skeleton, our surface-driven volume deformation is agnostic to the particular choice of template and can be used with human bodies, faces, or other object types.

Generative Articulated Neural Radiance Fields

GNARF is a novel general framework to train 3D-aware GANs for deformable objects that have a parametric template mesh, e.g. human bodies and faces. It builds on the efficient tri-plane feature representation for the generated neural radiance field, but additionally applies an explicit deformation which alleviates the requirement for the generator to learn a complicated distribution of articulations. As a result, the generator automatically learns to generate radiance fields of objects in the canonical pose, which are then warped explicitly to produce target body poses and facial expressions in a fully controllable and interpretable manner.

We first discuss our approach to modeling and rendering articulated radiance fields, before describing how this is integrated into the 3D GAN in Sec. 3.2.

Articulated deformation.

There are many possible choices for how to specify the deformation field in an intuitive manner. For example, linear blend skinning can be used to deform the entire volume “rigged” by a skeleton (see e.g. ). While skinning is popular for human body articulations, it cannot explain subtle deformation due to varying facial expressions. Another option is to use the object-specific template mesh as a cage and apply cage-based deformation for the entire volume using mean value coordinates (MVC) . However, the high computational cost of evaluating MVCs on the full-resolution grid (see Tab. 1) is prohibitive for GAN training and, more critically, this approach generally leads to severe artifacts when the template mesh (accidentally) includes self-intersections (see supplement).

To alleviate these problems, we use an intuitive surface-driven deformation method, which we label as the Surface Field (SF) method. This method only requires canonical and target template meshes with correspondences, which are readily available for faces and bodies . These template shapes, in turn, can be driven using skeletons, manual editing, or using keypoints or landmarks that could be detected in and transfered from videos of other people. Therefore, the SF method is generally sufficient to apply to different body parts and it can be intuitively edited in a number of ways, resulting in accurate volume deformation for our class of volumetric models.

The SF approach is very fast to compute and mitigates artifacts from self-intersections of the template shape, thereby combining the benefits of linear blend skinning and MVC-based approaches for the task of radiance field deformation.

Rendering deformed radiance fields.

We render the radiance field using (neural) volume rendering . For this purpose, the aggregated feature $F(\mathbf{r})$ of a ray $\mathbf{r}$ is computed by integrating the volumetric features $\mathbf{f}$ and density $\sigma$ as

where $t_{n}$ and $t_{f}$ indicate near and far bounds along the ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ pointing from its origin $\mathbf{o}$ into direction $\mathbf{d}$. The volume rendering equation (eq. 3) is typically approximated using numerical methods, such as the quadrature rule .

2 3D GAN Framework

An overview of our 3D pipeline is shown in Fig. 2. Several components, including the StyleGAN generator, the tri-plane representation, the volume rendering, the CNN-based image super-resolution module, and (dual) discrimination are directly adopted from the EG3D framework .

Instead of directly generating the radiance field with the target body pose or facial expression, however, GNARF is unique in generating the radiance field in a canonical pose and then applying the deformation field discussed in the previous section to warp the feature volume. We additionally remove the pose conditioning on the generator, and only use camera pose and body pose conditioning in the discriminator. This removes the ability for the generator to incorporate any knowledge about the final view or pose in the canonical radiance field generation, ensuring that the generated results will be robustly animatable beyond just the image rendered at training time. Thus, the generator depends only on the latent code controlling identity, which is input into a StyleGAN2 generator. This architectural choice takes advantage of the state-of-the-art 2D generative model architectures by using them to generate the tri-plane 3D representation. The discriminator having access to the camera and body poses ensure that the GAN learns to generate warping accurate to a target pose rather than just being in the correct distribution. Finally, we adopt a radiance field rendering strategy which samples along each ray inside of an expanded template mesh. This ensures that the integration samples are taken in regions of the radiance field with the most detail and not taken in empty space, simultaneously improving the quality of the generated results and speeding up training.

Additional implementation details, source code, and pre-trained models can be found in the supplement or on our website.

Experiments

We first evaluate the proposed deformation field by overfitting a single representation on a single dynamic full body scene. Then we apply this deformation method in a GAN training pipeline for both bodies (AIST++ and SURREAL ) and faces (FFHQ) . Training details and hyper-parameters are discussed in the supplement.

AIST++ is a large dataset consisting of 10.1M images capturing 30 performers in dance motion. Each frame is annotated with a ground truth camera and fitted SMPL body model. SURREAL contains 6M images of synthetic humans created using SMPL body models in various poses rendered in indoor scenes. FFHQ is a large dataset of high-resolution images of human faces collected from Flickr. All images have licenses that allow free use, redistribution, and adaptation for non-commencial use.

We select a multi-view video sequence from the AIST++ dataset and optimize tri-plane features in the canonical pose using a subset of the views and frames for supervision. We then evaluate the quality of the estimated radiance field warped into these training views and poses but also into held-out test views and poses. We apply several modifications to the tri-plane architecture to reduce overfitting; details regarding these changes as well as the selection of training and evaluation views are provided in the supplemental material.

To speed up MVC and SF computation, we decimate the source and deformed SMPL mesh using Quadric Error Metric Decimation in the Open3D library from the original 13,776 faces to 1,376 faces, while tracking the correspondence between the source and deformed meshes. Nonetheless computing MVC for each deformed pose is still prohibitively expensive for online training (3.7 s per example). We thus precompute the deformation for training and test body poses on a fixed $16^{3}$ grid and retrieve the deformation for arbitrary sampling points using trilinear interpolation.

As shown in Tab. 1, our SF method outperforms the others for both training and test images. MVC performs worst, partially due to the grid approximation, which is essential in practice. The skinning method is comparable to SF in terms of image quality but it is 3$\times$ slower. Moreover, skinning cannot sufficiently deform subtle facial expressions. Therefore, the SF approach is the most flexible among these deformation methods by being compatible with different human body parts while also offering computational and memory efficiency.

2 Human Body Generation and Animation

We now use our SF approach as the deformation method for feature volumes generated by GNARF. Our method is trained and evaluated on the captured AIST++ and the synthetic SURREAL datasets. For both datasets, our method generates high-quality multi-view consistent human bodies in diverse poses that closely match the target pose.

Because GNARF is the first method to learn a generative model of radiance fields representing bodies, we propose a baseline where we use the original EG3D trained without deformation to generate a feature volume (not in the canonical pose) then warp it into various target poses during inference time using the proposed SF deformation method. Without the feature volume deformation, the generator is forced to learn to model both identity and pose in its latent space. As a result, the tri-plane features no longer represent a human body in a consistent canonical pose, but rather match the distribution of poses in the dataset. The animation of generated bodies is similar to our proposed approach, except that the generated (arbitrarily posed) human body is used as the canonical pose, for which we obtain a SMPL mesh by applying the human shape reconstruction method SPIN . Additionally, we included the concurrent work ENARF-GAN and its variant ENARF-VAE in the evaluation whenever a fair and standardized comparison is possible.

We compare FID scores to evaluate the quality and diversity of generated images as well as the Percentage of Correct Keypoints (PCK) metric to evaluate the quality of the animation. PCK computes the percentage of 2D keypoints detected on a generated rendered image that are within an error threshold (half the size of the head in the case of PCKh@0.5) of keypoints on a ground truth image in the same pose and view. We use an off-the-shelf body keypoint estimator trained on MPII publically available on the MMPose Project to estimate keypoints from a corresponding ground truth and generated image.

AIST++.

AIST++ is a challenging dataset as the body poses are extremely diverse. We collect 30 frames per video as our training data after filtering out frames whose camera distance is above a threshold or the human bounding box is partially outside the image. Then we extract the human body by cropping a $600\times 600$ patch centered at the pelvis joint, and resize these frames to $256\times 256$. Since this dataset does not provide ground truth segmentation masks, we use a pre-trained segmentation model to remove backgrounds to stabilize the GAN training. To speed up training, we use GAN transfer learning . Rather than initializing our network weights randomly, we begin training from a pre-trained EG3D model. Fine-tuning allows for quicker convergence and saves computational resources during training.

As shown in Tab. 2, we can see that our method outperforms the naive reanimation of EG3D by a large margin (see the 4th row vs. the 5th row). Furthermore, our animated method also generates better images than the EG3D baseline that does not support animations. This is likely due to GNARF allowing the generator to focus on generating a specific identity in a canonical pose instead of learning both identity and complex pose distribution in a combined latent space. Similarly, our model enables high-quality re-animation, as demonstrated by the PCKh@0.5 metric. In Fig. 3, our method produces significantly better qualitative results than those generated by the baseline. The baseline results using re-warped EG3D are significantly degraded, since it is difficult to accurately estimate the SMPL mesh from the generated images, which we use as the canonical pose in the deformation function. Additionally, floating artifacts which exist in the radiance field outside of camera views and make no difference in the conventionally rendered images become visible after being warped can cause false occlusions. In Fig. 1, we show that our method generates bodies in a canonical pose with diverse identities. Additionally, we show that by changing the SMPL parameters, we can drive each radiance field to a desired target pose and render at an arbitrary novel view.

SURREAL.

We also test our method on the SURREAL dataset. The training data is extracted from the first frame of each video in SURREAL’s training split. Each frame is cropped based on the provided segmentation mask from head-to-toe, and resized to $128\times 128$. We filter out the backgrounds and set them to be black. The SURREAL dataset provides ground truth SMPL parameters and camera intrinsics and extrinsics for each frame. Similarly to AIST++, we use transfer learning from a pre-trained EG3D model at the appropriate resolution.

In Tab. 2, we see that our method trained with deformation produces improved FID scores as the version trained without SF feature volume deformation (5th vs 3rd row). Attempting to deform generated radiance fields results in an immense degradation in quality, shown by both the FID and PCKh@0.5 metrics (4th row).

Additional qualitative results and evaluations are included on the website.

3 Human Face Generation and Editing

Thanks to the surface-driven deformation method, SF, our method can directly utilize expressive parametric face models to drive subtle deformations. We use the FLAME head model and apply the state-of-the-art facial reconstruction method DECA to estimate the flame parameters (100D identity vectors, 6D joint position vectors and 50D expression vectors) from the training dataset. Since DECA does not account for eyeball movement, we remove the eyeballs from the original FLAME template. Additionally, we add triangles to close the holes at the neck and mouth area in the FLAME template, which improves the consistency of the SF deformation. These preprocessing steps produce a mesh with 3,741 vertices and 7,478 faces. Finally we apply the same decimation method as in the body experiments to obtain a coarse mesh (2,500 triangles), which we use during the GAN training for faster SF deformation.

To train GNARF, we start with a conventional EG3D model pre-trained using the FFHQ dataset (see for details on the training details and data processing). We then fine-tune this model using the proposed framework, which includes the SF deformation module. Note that we apply a global scaling and translation to the FLAME template mesh to roughly align it to the faces generated by the pre-trained EG3D model.

For evaluation, our first baseline is the original EG3D model with re-warping at inference time, as described in Sec. 4.2. We also include two state-of-the-art facial reenactment methods, PIRenderer and 3D GAN inversion . Several metrics are used by our quantitative evaluation. FID (500) follows the evaluation protocol of Lin et al. , where the ground truth dataset consists of 500 randomly sampled identities and the test dataset is constructed by animating the ground truth using randomly sampled target poses. FID (50k) follows the protocol in EG3D, where the entire FFHQ dataset is treated as the ground truth and the test dataset includes 50k generated images using randomly sampled latent vectors, camera poses and FLAME facial parameters. Following , we evaluate the fidelity of the animation with the Average Expression Distance (AED) and the Average Pose Distance (APD) computed using DECA estimation, as well as the identity consistency based on a face recognition model .

As shown in Tab. 3, our method is superior to the baseline methods in both FIDs and in the edited fidelity, and comparable to the baseline EG3D in terms of identity consistency. The advantage of our method in editing ability is clearly demonstrated in Fig. 5: our method reproduces the target expression more accurately when compared to the face reenactment methods and mitigates the warping artifacts present in the baseline EG3D. In Fig. 6, we further demonstrate the quality of our results. Notice that even though the FLAME model does not include teeth, the SF deformation guides the neural radiance field to construct teeth consistently at the correct location.

Discussion

Our work is not without limitations. The level of detail in the generated bodies, for example, is relatively low. This is partly due to the limited resolution of the training data in the SURREAL and AIST++ datasets, but also due to the limited resolution that the tri-plane representation offers for any one body part, such as the face. An interesting avenue of future work could include the exploration of adaptive radiance field resolution for human bodies, allocating more resolution to salient parts (see e.g. ). The image quality of the generated bodies is currently on par with simpler approaches that only need to generate an RGB texture on the SMPL mesh . Yet, details in faces and hair cannot be handled by a texture-generating approach and we expect the quality of generative radiance fields to surpass that of simpler alternatives with increasing and perhaps adaptive radiance field resolutions. Another bottleneck of our current framework is the challenge of being able to work with large-scale datasets showing a diversity of visible humans. In-the-wild datasets, such as MSCOCO , do contain this diversity but also contain a significant amount of occlusion and detailed backgrounds, requiring the generator to spend capacity on modeling the distribution of backgrounds and occlusions themselves. An interesting avenue of future work consists of explicitly modeling occlusion and background in the GAN training pipeline independently from identity and pose. Currently, the background is not modeled in the generative framework and requires pre-processing of the dataset to separate foreground from background. Additionally, pose estimation from in-the-wild images is still not very accurate, degrading the quality of our deformation function. On the other hand, large-scale custom curated datasets such as the one used in InsetGAN are not publicly available. Finally, the deformation we use as part of the 3D GAN training is limiting in several ways: it does not allow for topology changes, it does not prevent solid parts like teeth or eyeglasses from being stretched, and the deformation quality degrades for points far from the surface. In general, using the surface of a parametric mesh to guide an volume may not be the optimal choice for more complex volumetric scenes. More advanced deformation methods, perhaps including kinematic constraints, could be an interesting direction of future research.

Ethical considerations.

GANs could be misused for generating edited imagery of real people. Such misuse of image synthesis techniques poses a societal threat, and we do not condone using our work with the intent of spreading disinformation. We also recognize a potential lack of diversity in our results, stemming from implicit biases of the datasets we process.

Conclusion.

Our work takes important steps towards photorealistic 3D-aware image synthesis of articulated human bodies and faces with applications in visual effects, virtual or augmented reality, and teleconferencing among others.

Acknowledgments and Disclosure of Funding

Alexander W. Bergman was supported by a Stanford Graduate Fellowship. Gordon Wetzstein was supported by NSF Award 1839974, Samsung, Stanford HAI, and a PECASE from the ARO. We thank Connor Lin for helping standardize comparisons to . We thank Atsuhiro Noguchi and the authors of for helping in standardization of the comparisons of our methods.

References

Checklist

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] See Section 5.

Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [N/A]

Did you include complete proofs of all theoretical results? [N/A]

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We plan to release the code completely, but have not yet with submission. Details on the architecture, training, and data are available in the supplemental material.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplemental document and sections 4.2 and 4.3.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] GNARF models require a significant amount of computational resources (see supplemental document), and thus averaging training over many seeds would be computationally infeasible. However, we evaluate our trained models with a large number of generated images, demonstrating the robustness of our method.

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental document.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes]

Did you mention the license of the assets? [Yes] See Section 4.

Did you include any new assets either in the supplemental material or as a URL? [No]

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]