Efficient Geometry-aware 3D Generative Adversarial Networks

Introduction

Generative adversarial networks (GANs) have seen immense progress, with recent models capable of generating high-resolution, photorealistic images indistinguishable from real photographs . Current state-of-the-art GANs, however, operate in 2D only and do not explicitly model the underlying 3D scenes.

Recent work on 3D-aware GANs has begun to tackle the problem of multi-view-consistent image synthesis and, to a lesser extent, extraction of 3D shapes without being supervised on geometry or multi-view image collections. However, the image quality and resolution of existing 3D GANs have lagged far behind those of 2D GANs. Furthermore, their 3D reconstruction quality, so far, leaves much to be desired. One of the primary reasons for this gap is the computational inefficiency of previously employed 3D generators and neural rendering architectures.

In contrast to 2D GANs, 3D GANs rely on a combination of a 3D-structure-aware inductive bias in the generator network architecture and a neural rendering engine that aims at providing view-consistent results. The inductive bias can be modeled using explicit voxel grids or neural implicit representations . While successful in single-scene “overfitting” scenarios, neither of these representations is suitable for training a high-resolution 3D GAN because they are simply too memory inefficient or slow. Training a 3D GAN requires rendering tens of millions of images, but state-of-the-art neural volume rendering at high-resolutions with these representations is computationally infeasible. CNN-based image upsampling networks have been proposed to remedy this , but such an approach sacrifices view consistency and impairs the quality of the learned 3D geometry.

We introduce a novel generator architecture for unsupervised 3D representation learning from a collection of single-view 2D photographs that seeks to improve the computational efficiency of rendering while remaining true to 3D-grounded neural rendering. We achieve this goal with a two-pronged approach. First, we improve the computational efficiency of 3D-grounded rendering with a hybrid explicit–implicit 3D representation that offers significant speed and memory benefits over fully implicit or explicit approaches without compromising on expressiveness. These advantages enable our method to skirt the computational constraints that have limited the rendering resolutions and quality of previous approaches and forced over-reliance on image-space convolutional upsampling . Second, although we use some image-space approximations that stray from the 3D-grounded rendering, we introduce a dual-discrimination strategy that maintains consistency between the neural rendering and our final output to regularize their undesirable view-inconsistent tendencies. Moreover, we introduce pose-based conditioning to our generator, which decouples pose-correlated attributes (e.g., facial expressions) for a multi-view consistent output during inference while faithfully modeling the joint distributions of pose-correlated attributes inherent in the training data.

As an additional benefit, our framework decouples feature generation from neural rendering, enabling it to directly leverage state-of-the-art 2D CNN-based feature generators, such as StyleGAN2, to generalize over spaces of 3D scenes while also benefiting from 3D multi-view-consistent neural volume rendering. Our approach not only achieves state-of-the-art qualitative and quantitative results for view-consistent 3D-aware image synthesis, but also generates high-quality 3D shapes of the synthesized scenes due to its strong 3D-structure-aware inductive bias (see Fig. 1).

We introduce a tri-plane-based 3D GAN framework, which is both efficient and expressive, to enable high-resolution geometry-aware image synthesis.

We develop a 3D GAN training strategy that promotes multi-view consistency via dual discrimination and generator pose conditioning while faithfully modeling pose-correlated attribute distributions (e.g., expressions) present in real-world datasets.

We demonstrate state-of-the-art results for unconditional 3D-aware image synthesis on the FFHQ and AFHQ Cats datasets along with high-quality 3D geometry learned entirely from 2D in-the-wild images.

Related work

Emerging neural scene representations use differentiable 3D-aware representations that can be optimized using 2D multi-view images via neural rendering . Explicit representations, such as discrete voxel grids (Fig. 2b), are fast to evaluate but often incur heavy memory overheads, making them difficult to scale to high resolutions or complex scenes . Implicit representations, or coordinate networks (Fig. 2a), offer potential advantages in memory efficiency and scene complexity compared to discrete voxel grids by representing a scene as a continuous function (e.g., ). In practice, these implicit architectures use large fully connected networks that are slow to evaluate as each query requires a full pass through the network. Therefore, fully explicit and implicit representations provide complementary benefits.

Local implicit representations and hybrid explicit–implicit representations combine the benefits of both types of representations by offering computationally and memory-efficient architectures. Inspired by these ideas, we design a new hybrid explicit–implicit 3D-aware network that uses a memory-efficient tri-plane representation to explicitly store features on axis-aligned planes that are aggregated by a lightweight implicit feature decoder for efficient volume rendering (Fig. 2c). Our representation bears some resemblance to previous plane-based hybrid architectures , but it is unique in its specific design. Our representation is key to enabling the high 3D GAN image quality that we demonstrate through efficient training comparable (in time scales) to modern 2D GANs .

Generative 3D-aware image synthesis.

Generative adversarial networks have recently achieved photorealistic image quality for 2D image synthesis . Extending these capabilities to 3D settings has started to gain momentum as well. Mesh-based approaches build on the most popular primitives used in computer graphics, but lack the expressiveness needed for high-fidelity image generation . Voxel-based GANs directly extend the CNN generators used in 2D settings to 3D . The high memory requirements of voxel grids and the computational burden of 3D convolutions, however, make high-resolution 3D GAN training difficult. Low-resolution 3D volume generation can be remedied with 2D CNN-based image upsampling layers , but without an inductive 3D bias the results often lack view consistency. Block-based sparse volume representations overcome some of these issues, but are applicable to mostly empty scenes and difficult to generalize across scenes. As an alternative, fully implicit representation networks have been proposed for 3D scene generation , but these architectures are slow to query, which makes the GAN training inefficient, limiting the quality and resolution of generated images.

One of the primary insights of our work is that an efficient 3D GAN architecture with 3D-grounded inductive biases is crucial for successfully generating high-resolution view-consistent images and high-quality 3D shapes. Our framework achieves this in several ways. First, unlike most existing 3D GANs, we directly leverage a 2D CNN-based feature generator, i.e., StyleGAN2 , removing the need for inefficient 3D convolutions on explicit voxel grids. Second, our tri-plane representation allows us to leverage neural volume rendering as an inductive bias, but in a much more computationally efficient way than fully implicit 3D networks . Similar to , we also employ 2D CNN-based upsampling after neural rendering, but our method introduces dual discrimination to avoid view inconsistencies introduced by the upsampling layers. Unlike existing StyleGAN2-based 2.5D GANs, which generate images and depth maps , our method works naturally for steep camera angles and in 360∘ viewing conditions.

The concurrently developed 3D-aware GANs StyleNeRF and CIPS-3D demonstrate impressive image quality. The central distinction between these and ours is that while StyleNeRF and CIPS-3D operate primarily in image-space, with less emphasis on the 3D representation, our method operates primarily in 3D. Our approach demonstrates greater view consistency, and is capable of generating high-quality 3D shapes. Furthermore, our experiments report superior FID image scores on FFHQ and AFHQ.

Tri-plane hybrid 3D representation

Training a high-resolution GAN requires a 3D representation that is both efficient and expressive. In this section, we introduce a new hybrid explicit–implicit tri-plane representation that offers both of these advantages. We introduce the representation in this section for a single-scene overfitting (SSO) experiment, before discussing how it is integrated in our GAN framework in the next section.

The primary advantage of this hybrid representation is efficiency—by keeping the decoder small and shifting the bulk of the expressive power into the explicit features, we reduce the computational cost of neural rendering compared to fully implicit MLP architectures without losing expressiveness. To validate that the tri-plane representation is compact yet sufficiently expressive, we evaluate it with a common novel-view synthesis setup. For this purpose, we directly optimize the features of the planes and the weights of the decoder to fit $360^{\circ}$ views of a scene from the Tanks & Temples dataset (Fig. 3). In this experiment, we use feature planes of resolution $N=512$ and channels $C=48$, paired with an MLP of four layers of 128 hidden units each and a Fourier feature encoding. We compare the results against a dense feature volume of equal capacity. For reference, we include comparisons to a state-of-the-art fully implicit 3D representation . Fig. 3 and Tab. 1 demonstrate that the tri-plane representation is capable of representing this complex scene, albeit without view-dependent effects, outperforming dense feature volume representations and fully implicit representations in terms of PSNR and SSIM, while offering considerable advantages in computation and memory efficiency. For a side length of $N$ features, tri-planes scale with $O(N^{2})$ rather than $O(N^{3})$ as dense voxels do, which means for equal capacity and memory, the tri-plane representation can use higher resolution features and capture greater detail. Finally, our tri-plane representation has one other key advantage over these alternatives: the feature planes can be generated with an off-the-shelf 2D CNN-based generator, enabling generalization across 3D representations using the GAN framework discussed next.

3D GAN framework

Armed with an efficient and expressive 3D representation, we train a 3D GAN for geometry-aware image synthesis from 2D photographs, without any explicit 3D or multi-view supervision. We associate each training image with a set of camera intrinsics and extrinsics using off-the-shelf pose detectors ; see the supplement for details.

Fig. 4 gives an overview of our network architecture. We use the tri-plane representation introduced in the last section to efficiently render images through neural volume rendering, but make a number of modifications to adapt this representation to the 3D GAN setting. Unlike in the SSO experiment, where the features of the planes were directly optimized from the multiple input views, for the GAN setting we generate the tri-plane features, each containing $32$ channels, with the help of a 2D convolutional StyleGAN2 backbone (Sec. 4.1). Instead of producing an RGB image, in the GAN setting our neural renderer aggregates features from each of the $32$-channel tri-planes and predicts 32-channel feature images from a given camera pose. This is followed by a “super-resolution” module to upsample and refine these raw neurally rendered images (Sec. 4.2). The generated images are critiqued by a slightly modified StyleGAN2 discriminator (Sec. 4.3). The entire pipeline is trained end-to-end from random initialization, using the non-saturating GAN loss function with R1 regularization , following the training scheme in StyleGAN2 . To speed training, we use a two-stage training strategy in which we train with a reduced ($64^{2}$) neural rendering resolution followed by a short fine-tuning period at full ($128^{2}$) neural rendering resolution. Additional experiments found that regularization to encourage smoothness of the density field helped reduce artifacts in 3D shapes. The following sections discuss major components of our framework in detail. For additional descriptions, implementation details, and hyperparameters, please see the supplement.

The features of the tri-plane representation, when used in our GAN setting, are generated by a StyleGAN2 CNN generator. The random latent code and camera parameters are first processed by a mapping network to yield an intermediate latent code which then modulates the convolution kernels of a separate synthesis network.

We change the output shape of the StyleGAN2 backbone such that, rather than producing a three-channel RGB image, we produce a $256\times 256\times 96$ feature image. This feature image is split channel-wise and reshaped to form three 32-channel planes (see Fig. 4). We choose StyleGAN2 for predicting the tri-plane features because it is a well-understood and efficient architecture achieving state-of-the-art results for 2D image synthesis. Furthermore, our model inherits many of the desirable properties of StyleGAN: a well-behaved latent space that enables style-mixing and latent-space interpolation (see Sec. 5 and supplement).

We sample features from the tri-planes, aggregate by summation, and process the aggregated features with a lightweight decoder, as described in Sec. 3. Our decoder is a multi-layer perceptron with a single hidden layer of $64$ units and softplus activation functions. The MLP does not use a positional encoding, coordinate inputs, or view-direction inputs. This hybrid representation can be queried for continuous coordinates and outputs a scalar density $\sigma$ as well as a 32-channel feature, both of which are then processed by a neural volume renderer to project the 3D feature volume into a 2D feature image.

Volume rendering is implemented using two-pass importance sampling as in . Following , volume rendering in our GAN framework produces feature images, rather than RGB images, because feature images contain more information that can be effectively utilized for the image-space refinement described next. For the majority of the experiments reported in this manuscript, we render 32-channel feature images $I_{F}$ at a resolution of $128^{2}$, with 96 total depth samples per ray.

2 Super resolution

Although the tri-plane representation is significantly more computationally efficient than previous approaches, it is still too slow to natively train or render at high resolutions while maintaining interactive framerates. We thus perform volume rendering at a moderate resolution (e.g., $128^{2}$) and rely upon image-space convolutions to upsample the neural rendering to the final image size of $256^{2}$ or $512^{2}$.

Our super resolution module is composed of two blocks of StyleGAN2-modulated convolutional layers that upsample and refine the 32-channel feature image $I_{F}$ into the final RGB image $I^{+}_{RGB}$. We disable per-pixel noise inputs to reduce texture sticking and reuse the mapping network of the backbone to modulate these layers.

3 Dual discrimination

As in standard 2D GAN training, the resulting renderings are critiqued by a 2D convolutional discriminator. We use a StyleGAN2 discriminator with two modifications.

First, we introduce dual discrimination as a method to avoid multi-view inconsistency issues observed in prior work . For this purpose, we interpret the first three feature channels of a neurally rendered feature image $I_{F}$ as a low-resolution RGB image $I_{RGB}$. Intuitively, dual discrimination then ensures consistency between $I_{RGB}$ and the super-resolved image $I^{+}_{RGB}$. This is achieved by bilinearly upsampling $I_{RGB}$ to the same resolution as $I^{+}_{RGB}$ and concatenating the results to form a six-channel image (see Fig. 4). The real images fed into the discriminator are also processed by concatenating each of them with an appropriately blurred copy of itself. We discriminate over these six-channel images instead of the three-channel images traditionally seen in GAN discriminators.

Dual discrimination not only encourages the final output to match the distribution of real images, but also offers additional effects: it encourages the neural rendering to match the distribution of downsampled real images; and it encourages the super-resolved images to be consistent with the neural rendering (see Fig. 5). The second point importantly allows us to leverage effective image-space super-resolution layers without introducing view-inconsistency artifacts.

Second, we make the discriminator aware of the camera poses from which the generated images are rendered. Specifically, following the conditional strategy from StyleGAN2-ADA , we pass the rendering camera intrinsics and extrinsics matrices (collectively $\mathbf{P}$) to the discriminator as a conditioning label. We find that this conditioning introduces additional information that guides the generator to learn correct 3D priors. We provide additional studies in the supplement showing the effect of this discriminator conditioning and the robustness of our framework to high levels of noise in the input camera poses.

4 Modeling pose-correlated attributes

Most real-world datasets like FFHQ include biases that correlate camera poses with other attributes (e.g., facial expressions), and naively handling them leads to view inconsistent results. For example, the camera angle with respect to a person’s face is correlated with smiling (see supplement). While faithfully modeling such attribute correlations inherent in the dataset is important for reproducing the best image quality, such unwanted attributes need to be decoupled during inference for multi-view consistent synthesis. Related work has been successful at being view consistent or modeling pose-appearance correlations , but cannot achieve both simultaneously.

We introduce generator pose conditioning as a means to model and decouple correlations between pose and other attributes observed in the training images. To this end, we provide the backbone mapping network not only a latent code vector $z$, but also the camera parameters $\mathbf{P}$ as input, following the conditional generation strategy in . By giving the backbone knowledge of the rendering camera position, we allow the target view to influence scene synthesis.

During training, pose conditioning allows the generator to model pose-dependent biases implicit to the dataset, allowing our model to faithfully reproduce the image distributions in the dataset. To prevent the scene from shifting with camera pose during inference, we condition the generator on a fixed camera pose when rendering from a moving camera trajectory. We noticed that always conditioning the generator with the rendering camera pose can lead to degenerate solutions where the GAN produces 2D billboards angled towards the camera (see supplement). To prevent this, we randomly swap the conditioning pose in $\mathbf{P}$ with another random pose with 50% probability during training.

Experiments and results

We compare methods on the task of unconditional 3D-aware generation with FFHQ , a real-world human face dataset, and AFHQv2 Cats , a small, real-world cat face dataset. We augment both datasets with horizontal flips and use off-the-shelf pose estimators to extract approximate camera extrinsics. For all methods on AFHQv2, we apply transfer learning from corresponding FFHQ checkpoints; for our method on AFHQv2 $512^{2}$, we additionally use adaptive data augmentation. For more results, please see the accompanying video.

1 Comparisons

We compare our methods against three state-of-the-art methods for 3D-aware image synthesis: $\pi$-GAN , GIRAFFE , and Lifting StyleGAN .

Qualitative results.

Fig. 6 presents selected examples synthesized by our model with FFHQ and AFHQ at a resolution of $512^{2}$, highlighting the image quality, view-consistency, and diversity of outputs produced by our method. Fig. 7 provides a qualitative comparison against baselines. While GIRAFFE synthesizes high-quality images, reliance on view-inconsistent convolutions produces poor-quality shapes and identity shift—note the hairline inconsistency between rendered views. $\pi$-GAN and Lifting StyleGAN generate adequate shapes and images but both struggle with photorealism and in capturing detailed shapes.

Our method synthesizes not only images that are higher quality and more view-consistent but also higher-fidelity 3D geometry as seen in the detailed glasses and hair strands.

Quantitative evaluations.

Table 2 provides quantitative metrics comparing the proposed approach against baselines. We measure image quality with Fréchet Inception Distance (FID) between 50k generated images and all available real images. We evaluate shape quality by calculating MSE against pseudo-ground-truth depth-maps (Depth) and poses (Pose) estimated from synthesized images by ; a similar evaluation was introduced by . We assess multi-view facial identity consistency (ID) by calculating the mean Arcface cosine similarity score between pairs of views of the same synthesized face rendered from random camera poses. Additional evaluation details are provided in the supplement. Our model demonstrates significant improvements in FID across both datasets, bringing the 3D GAN to near the same level as StyleGAN2 $512^{2}$ (2.97 for FFHQ and 2.99 for Cats ) while also maintaining state-of-the-art view consistency, geometry quality, and pose accuracy.

Runtime.

Table 3 compares rendering speed at inference running on a single NVIDIA RTX 3090 GPU. Our end-to-end approach achieves real-time framerates at $512^{2}$ final resolution with $128^{2}$ neural rendering resolution and 96 total depth samples per ray, suitable for applications such as real-time visualization. When rendering consecutive frames of a static scene, we need not regenerate the tri-plane features every frame; caching the generated features is a simple tweak that improves render speed. The proposed approach is significantly faster than fully implicit methods like $\pi$-GAN . Although it is not as fast as Lifting StyleGAN and GIRAFFE , we believe major improvements in image quality, geometry quality, and view-consistency outweigh the increased compute cost.

2 Ablation study

Without dual discrimination, generated images can include multi-view inconsistencies due to the unconstrained image-space super-resolution layers. We measure this effect quantitatively by extracting smile-related Facial Action Coding System (FACS) coefficients from videos produced by models with and without dual discrimination, using a proprietary facial tracker. We measure the standard deviation of smile coefficients for the same scene across video frames. A view-consistent scene should exhibit little expression shift and thus produce little variation in smile coefficients. This is validated in Table 4 showing that introducing dual discrimination (second row) reduces the smile coefficient variation versus the naive model (first row), indicating improved expression consistency. However, dual discrimination also reduces image quality as seen by the slightly worse FID score, perhaps because the model is restricted from reproducing the pose-correlated attribute biases in the FFHQ dataset. By adding generator pose conditioning (third row), we allow the generator to faithfully model pose-correlated attributes while decoupling them at inference, leading to both the best FID score and view-consistent results.

3 Applications

Since our 3D representation is designed with the StyleGAN2 backbone from the ground up, it inherits the well-studied properties of the StyleGAN2 latent space, allowing us to do semantic image manipulations. Fig. 8 shows our method’s results for style mixing .

Single-view 3D reconstruction.

Fig. 9 shows the application of our learned latent space for single-view 3D reconstruction. We use pivotal tuning inversion (PTI) to fit test images. The learned 3D prior over FFHQ enables surprisingly high-quality single-view geometry recovery. Further exploration of few-shot 3D reconstruction and novel-view-synthesis may prove a fruitful avenue for future work.

Discussion

Although our shapes show significant improvements over those generated by previous 3D-aware GANs, they may still contain artifacts and lack finer details, such as individual teeth. To further improve the quality of the learned shapes, we could instill a stronger geometry prior or regularize the density component of the radiance field following methods proposed by .

Our model requires knowledge of the camera pose distribution of the dataset. Although prior work has proposed learning the pose distribution on the fly , others have noticed such methods can diverge , so it would be fruitful to explore this direction further. Pose conditioning aids the generator in decoupling appearance from pose, but still does not fully disentangle the two. Furthermore, ambiguities that can be explained by geometry remain unresolved. For example, by creating concave eye sockets, the generator creates the illusion of eyes that “follow” the camera, an incorrect interpretation, though the renderings are view-consistent and reflect the underlying geometry.

We used StyleGAN 2, but other 2D backbones may find success in our framework. Alternative backbones, such as as image-to-image translation or Transformer-based models, could enable new applications in conditional synthesis.

Ethical considerations.

The single-view 3D reconstruction or style mixing applications could be misused for generating edited imagery of real people. Such misuse of image synthesis techniques poses a societal threat, and we do not condone using our work with the intent of spreading misinformation or tarnishing reputation. We also recognize a potential lack of diversity in our faces results, stemming from implicit biases of the datasets we process.

Conclusion.

By combining an efficient explicit–implicit neural representation with an expressive pose-aware convolutional generator and a dual discriminator, our approach takes significant steps towards photorealistic 3D-aware image synthesis and high-quality unsupervised shape generation. This may enable rapid prototyping of 3D models, more controllable image synthesis, and novel techniques for shape reconstruction from temporal data.

Acknowledgements

We thank David Luebke, Jan Kautz, Jaewoo Seo, Jonathan Granskog, Simon Yuen, Alex Evans, Stan Birchfield, Alexander Bergman, and Joy Hsu for feedback on drafts, Alex Chan, Giap Nguyen, and Trevor Chan for help with diagrams, and Colette Kress and Bryan Catanzaro for allowing use of their photographs. This project was in part supported by Stanford HAI and a Samsung GRO. Koki Nagano and Eric Chan were partially supported by DARPA’s Semantic Forensics (SemaFor) contract (HR0011-20-3-0005). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. Distribution Statement “A” (Approved for Public Release, Distribution Unlimited).

References