PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

Introduction

High-fidelity human digitization is the key to enabling a myriad of applications from medical imaging to virtual reality. While metrically accurate and precise reconstructions of humans is now possible with multi-view systems , it has remained largely inaccessible to the general community due to its reliance on professional capture systems with strict environmental constraints (e.g., high number of cameras, controlled illuminations) that are prohibitively expensive and cumbersome to deploy. Increasingly, the community has turned to using high capacity deep learning models that have shown great promise in acquiring reconstructions from even a single image . However, the performance of these methods currently remains significantly lower than what is achievable with professional capture systems.

The goal of this work is to achieve high-fidelity 3d reconstruction of clothed humans from a single image at a resolution sufficient to recover detailed information such as fingers, facial features and clothing folds (see Fig. 1). Our observation is that existing approaches do not make full use of the high resolution (e.g., 1k or larger) imagery of humans that is now easily acquired using commodity sensors on mobile phones. This is because the previous approaches rely on holistic reasoning to map between the 2D appearance of an imaged human and their 3D shape, where, in practice, down-sampled images are used due to the prohibitive memory requirements . Although local image patches have important cues for detailed 3D reconstruction, these are rarely leveraged in the full high-resolution inputs due to the memory limitations of current graphics hardware.

Approaches that aim to address this limitation can be categorized into one of two camps. In the first camp, the problem is decomposed in a coarse-to-fine manner, where high-frequency details are embossed on top of low-fidelity surfaces. In this approach, a low image resolution is used to obtain a coarse shape. Then, fine details represented as surface normal or displacements are added by either a post-process such as Shape From Shading or composition within neural networks. The second camp employs high-fidelity models of humans (e.g., SCAPE ) to hallucinate plausible detail. Although both approaches result in reconstructions that appear detailed, they often do not faithfully reproduce the true detail present in the input images.

In this work, we introduce an end-to-end multi-level framework that infers 3D geometry of clothed humans at an unprecedentedly high 1k image resolution in a pixel-aligned manner, retaining the details in the original inputs without any post-processing. Our method differs from the coarse-to-fine approaches in that no explicit geometric representation is enforced in the coarse levels. Instead, implicitly encoded geometrical context is propagated to higher levels without making an explicit determination about geometry prematurely. We base our method on the recently introduced Pixel-Aligned Implicit Function (PIFu) representation . The pixel-aligned nature of the representation allows us to seamlessly fuse the learned holistic embedding from coarse reasoning with image features learned from the high-resolution input in a principled manner. Each level incrementally incorporates additional information missing in the coarse levels, with the final determination of geometry made only in the highest level.

Finally, for a complete reconstruction, the system needs to recover the backside, which is unobserved in any single image. As with low resolution input, missing information that is not predictable from observable measurements will result in overly smooth and blurred estimates. We overcome this problem by leveraging image-to-image translation networks to produce backside normals, similar to . Conditioning our multi-level pixel-aligned shape inference with the inferred back-side surface normal removes ambiguity and significantly improves the perceptual quality of our reconstructions with a more consistent level of detail between the visible and occluded parts.

The main contributions in this work consists of:

an end-to-end trainable coarse-to-fine framework for implicit surface learning for high-resolution 3D clothed human reconstruction at 1k image resolution.

a method to effectively handle uncertainty in unobserved regions such as the back, resulting in complete reconstructions with high detail.

Related Work

Single-view 3D human reconstruction is an ill-posed problem due to the fundamental depth ambiguity along camera rays. To overcome such ambiguity, parametric 3D models are often used to restrict estimation to a small set of model parameters, constraining the solution space to a specifically chosen parametric body model . However, the expressiveness of the resulting models is limited by using a single template mesh as well as by the data on which the model is built (often comprised mainly of minimally clothed people). While using a separate parameteric model can alleviate the limited shape variation , large deformations and topological changes are still non-trivial to handle with these shape representations.

Researchers have also proposed methods that do not use parametric models, but rather directly regress “free-form” 3D human geometry from single views. These approaches vary their directions based on the input and output representation that each algorithm uses. Some methods represent the 3D output world via a volumetric representation . Of particular relevance to this work is the DeepHuman approach of Zheng et al., where a discretized volumetric representation is produced by the network in increasing resolution and detail. Additional details using surface normals are embossed at the final level. While this method obtains impressive results, the cubic memory requirement imposed by the discrete voxel representation prevents obtaining high resolution simply by naively scaling the input resolution. Alternative methods consider additional free-form deformation on top of a parametric model space , and there exist also multiple approaches that predict depth maps of the target people as output .

The recently introduced Pixel-Aligned Implicit Function (PIFu) does not explicitly discretize the output space representation but instead regresses a function which determines the occupancy for any given 3D location. This approach shows its strength in reconstructing high-fidelity 3D geometry without having to keep a discretized representation of the entire output volume in memory simultaneously. Furthermore, unlike implicit surface representations using a global feature vector , PIFu utilizes fully convolutional image features, retaining local details present in an input image.

High-Resolution Synthesis in Texture Space

A number of recent approaches pursue reconstructing high-quality 3D texture or geometry by making use of a texture map representation on which to estimate geometric or color details. Particularly, the Tex2Shape approach of Alldieck et al. aims to reconstruct high quality 3D geometry by regressing displacements in an unwrapped UV space. However, this type of approach is ultimately limited by the topology of the template mesh (exhibiting problems when representing different topologies, such as required by different hair styles or skirts) and the topology chosen for the UV parameterization (e.g., visible seam artifacts around texture seams). Recent approaches leverage neural network models to predict intermediate texture or depth representations that are then used to reconstruct final 3D geometry output .

Our work is also related to approaches that produce high quality or high resolution synthetic human images. Recent methods consider producing high quality synthetic faces to overcome limitations of original GAN-based approaches . Similar trade-offs are pursued in semantic segmentation tasks .

Method

Our method builds on the recently introduced Pixel-aligned Implicit Function (PIFu) framework of , which takes images with resolution of 512$\times$512 as input and obtains low-resolution feature embeddings (128$\times$128). To achieve higher resolution outputs, we stack an additional pixel-aligned prediction module on top of this framework, where the fine module takes as input higher resolution images (1024$\times$1024) and encodes into high-resolution image features (512$\times$512). The second module takes the high-resolution feature embedding as well as the 3D embeddings from the first module to predict an occupancy probability field. To further improve the quality and fidelity of the reconstruction, we first predict normal maps for the front and back sides in image space, and feed these to the network as additional input. See Fig. 2 for an overview of the method.

where $Z=\mathbf{X}_{z}$ is the depth along the ray defined by the 2D projection $\mathbf{x}$. Note that all 3D points along the same ray have exactly the same image features $\Phi\left(\mathbf{x},I\right)$ from the same projected location $\mathbf{x}$, and thus the function $g$ should focus on the varying input depth $Z$ to disambiguate the occupancy of 3D points along the ray. In , a Convolutional Neural Network (CNN) architecture is used for the 2D feature embedding function $\Phi$ and a Multilayer Perceptron (MLP) for the function $g$.

A large scale dataset synthetically generated by rendering hundreds of high quality scanned 3D human mesh models is used to train the function $f$ in an end-to-end fashion. Unlike voxel-based methods, PIFu does not produce a discretized volume as output, so training can be performed by sampling 3D points and computing the occupancy loss at the sampled locations, without generating 3D meshes. During inference, 3D space is uniformly sampled to infer the occupancy and the final iso-surface is extracted with a threshold of 0.5 using marching cubes .

The input size as well as the image feature resolution of PIFu and other existing work are limited to at most 512$\times$512 and 128 $\times$ 128 in resolution respectively, due to memory limitations in existing graphics hardware. Importantly, the network should be designed such that its receptive field covers the entire image so that it can employ holistic reasoning for consistent depth inference—thus, a repeated bottom-up and top-down architecture with intermediate supervision plays an important role to achieve robust 3D reconstruction with generalization ability. This prevents the method from taking higher resolution images as input and keeping the resolution in the feature embeddings, even though this would potentially allow the network to leverage cues about detail present only at those higher resolutions. We found that while in theory the continuous representation of PIFu can represent 3D geometry at an arbitrary resolution, the expressiveness of the representation is bounded by the feature resolution in practice. Thus, we need an effective way of balancing robustness stemming from long-range holistic reasoning and expressiveness by higher feature embedding resolutions.

2 Multi-Level Pixel-Aligned Implicit Function

We present a multi-level approach towards higher fidelity 3D human digitization that takes 1024$\times$1024 resolution images as input. Our method is composed of two levels of PIFu modules: (1) a coarse level similar to PIFu , focusing on integrating global geometric information by taking the downsampled $512\times 512$ image as input, and producing backbone image features of $128\times 128$ resolution, and (2) a fine level that focuses on adding more subtle details by taking the original 1024$\times$1024 resolution image as input, and producing backbone image features of 512$\times$512 resolution (four times higher resolution than the implementation of ). Notably, the fine level module takes 3D embedding features extracted from the coarse level instead of the absolute depth value. Our coarse level module is defined similar to PIFu, but as a modification (Sect 3.3) it also takes predicted frontside and backside normal maps:

Because the fine level takes these features from the first pixel-aligned MLP as a 3d embedding, the global reconstruction quality should not be degraded, and should improve if the network design can properly leverage the increased image resolution and network capacity. Additionally, the fine network doesn’t need to handle normalization (i.e., producing a globally consistent 3D depth) and therefore doesn’t need to see the entire image, allowing us to train it with image crops. This is important to allow high-resolution image inputs without being limited by memory.

3 Front-to-Back Inference

Predicting the accurate geometry of the back of people is an ill-posed problem because it is not directly observed in the images. The backside must therefore be inferred entirely by the MLP prediction network and, due to the ambiguous and multimodal nature of this problem, the 3D reconstruction tends to be smooth and featureless. This is due in part to the occupancy loss (Sect. 3.4) favoring average reconstructions under uncertainty, but also because the final MLP layers need to learn a complex prediction function.

We found that if we instead shift part of this inference problem into the feature extraction stage, the network can produce sharper reconstructed geometry. To do this, we predict normal maps as a proxy for 3D geometry in image space, and provide these normal maps as features to the pixel-aligned predictors. The 3D reconstruction is then guided by these maps to infer a particular 3D geometry, making it easier for the MLPs to produce details. We predict the backside and frontal normals in image space using a pix2pixHD network, mapping from RGB color to normal maps. Similarly to recent approaches , we find that this produces plausible outputs for the unseen backside for sufficiently constrained problem domains, such as clothed humans.

4 Loss Functions and Surface Sampling

The specifics of the loss functions used can have a strong effect on the details recovered by the final model. Rather than use an average L1 or L2 loss as in , we use an extended Binary Cross Entropy (BCE) loss at a set of sampled points,

where $\mathcal{S}$ denotes the set of samples at which the loss is evaluated, $\lambda$ is the ratio of points outside surface in $\mathcal{S}$, $f^{*}(\cdot)$ denotes the ground truth occupancy at that location, and $f^{\{L,H\}}(\cdot)$ are each of the pixel-aligned implicit functions of Sect. 3.2. As in , we sample points using a mixture of uniform volume samples and importance sampling around the surface using Gaussian perturbation around uniformly sampled surface points. We found that this sampling scheme produces sharper results than sampling points proportionally to the inverse of distance from the surface. In fact, a mixture of Gaussian balls on the surface has higher sampling density near regions with high curvature (up to the inverse of Gaussian ball radius). Since curvature is the second-order derivative of surface geometry, importance sampling based on curvature significantly enhances details and fidelity.

Experimental Results

To obtain high-fidelity 3D geometry and corresponding images, we use RenderPeople data , which consists of commercially available 500 high-resolution photogrammetry scans. We split the dataset into a training set of 450 subjects and a test set of 50 subjects and render the meshes with precomputed radiance transfer using 163 second-order spherical harmonics from HDRI Havenhttps://hdrihaven.com/. Each subject is rendered from every other degree in yaw axis with an elevation fixed with $0\degree$. Unlike , where clean segmentation mask is required, we augment random background images using COCO dataset, removing the need of segmentation as pre-process.

Implementation Details.

The surface normal inference uses a network architecture proposed by , consisting of $9$ residual blocks with $4$ downsampling layers. We train two networks that predict frontside and backside normals individually with the following objective functions:

where $\mathcal{L}_{VGG}$ is the perceptual loss proposed by Johnson et al. , and $\mathcal{L}_{l1}$ is the $l1$ distance between the prediction and ground truth normals. The relative weight $\lambda_{l1}$ is set to $5.0$ in our experiments. We use the aforementioned $450$ RenderPeople training set to generate synthetic ground truth front and backside normals together with the corresponding input images. We use Adam optimizer with learning rate of $2.0\times 10^{-4}$ until the convergence.

1 Evaluations

We evaluate our multi-level pixel aligned implicit functions with several alternatives to assess the factors that contribute to achieving high-fidelity reconstructions. First, we assess the importance of 3D embedding that accounts for holistic context for high-resolution inference. To support larger input resolution at inference time, we train the network with random cropping of $512\times 512$ from $1024\times 1024$ images, similar to 2D computer vision tasks (e.g., semantic segmentation). We found that if our fine level module is conditioned on the absolute depth value instead of the learned 3D embedding, training with sliding window significantly degrades both training and test accuracy (see Fig. 3). This illustrates that 3D reconstruction using high-resolution features without holistic reasoning severely suffers from depth ambiguity and is unable to generalize with input size discrepancy between training and inference. Thus, the combination of holistic reasoning and high-resolution images features is essential for high-fidelity 3D reconstruction.

Second, we evaluate our design choice from both robustness and fidelity perspective. To achieve high-resolution reconstruction, it is important to keep feature resolution large enough while maintaining the ability to reason holistically. In this experiment, we implement 1) a pixel-aligned implicit function using only our fine-level image encoder by processing the full resolution as input during training, 2) conditioning 1) with jointly learned global feature using ResNet34 as a global feature encoder in spirit to , 3) a single PIFu (i.e., our coarse-level image encoder) by resizing input to $512\times 512$, 4) our proposed multi-level PIFu (two levels) by training all networks jointly (ML-PIFu, end-to-end), and 5) ours with alternate training of the coarse and fine modules (ML-PIFu, alternate).

Figure 4 and Table 1 show our qualitative and quantitative evaluation using RenderPeople and BUFF dataset. We compute point-to-surface distance, Chamfer distance, and surface normal consistency using ground truth geometry. Large spatial resolution of feature embeddings ($512\times 512$) greatly enhance local details compared with a single-level PIFu whose backbone feature resolution ($128\times 128$) is spatially $4$ times smaller. On the other hand, due to the limited design choices for high resolution input, using local feature suffers from overfitting and robustness and generalization becomes challenging. While adding global context helps a network reason more precise geometry, resulting in sharper reconstruction, lack of precise spatial information in the global feature deteriorates the robustness. This problem becomes more critical in case of non-rigid articulated objects . Also, we found that alternatively training the coarse and fine modules results in higher accuracy than jointly training them in an end-to-end manner.

We also evaluate the importance of inferring backside normal to recover detail on the occluded region. Figure 5 shows that PIFu that takes only input image suffers from blurred reconstruction on the missing regions due to ambiguity. On the other hand, directly providing guidance by facilitating image-to-image translation networks significantly improves the reconstruction accuracy on both front and back side with more realistic wrinkles. Since the pixel-aligned implicit functions are computationally expensive to differentiablly render on a image plane, solving sub-problems in the image domain is a practical solution to address the completion task with plausible details.

2 Comparisons

We qualitatively compare our method with state-of-the-art 3D human reconstruction methods with various shape representations on the publicly available People Snapshot dataset . The shape representations include multi-scale voxel (DeepHuman) , pixel-aligned implicit function (PIFu) , and a human parametric model with texture mapping using displacements and surface normals (Tex2shape) . While Tex2shape and DeepHuman adopt a coarse-to-fine strategy, the results show that the effect of refinement is marginal due to the limited representation power of the base shapes (See Fig. 6). More specifically, a voxel representation limits spatial resolution, and a template-based approach has difficulty handling varying topology and large deformations. Although the template-based approach retains some distinctive shapes such as wrinkles, the resulting shapes lose the fidelity of the input subject due to the imperfect mapping from image space to the texture parameterization using the off-the-shelf human dense correspondences map . In contrast, our method fully leverages the expressive shape representation for both base and refined shapes and directly predicts 3D geometry at a pixel-level, retaining all the details that are present in the input image. More qualitative results can be found in Figure 7.

Discussion and Future Work

We present a multi-level framework that performs joint reasoning over holistic information and local details to arrive at high-resolution 3D reconstructions of clothed humans from a single image without any additional post processing or side information. Our multi-level Pixel-Aligned Implicit Function achieves this by incrementally propagating global context through a scale pyramid as an implicit 3D embedding. This avoids making premature decisions about explicit geometry that has limited prior approaches. Our experiments demonstrate that it is important to incorporate such 3D-aware context for accurate and precise reconstructions. Furthermore, we show that circumventing ambiguity in the image-domain greatly increases the consistency of 3D reconstruction detail in occluded regions.

Since the multi-level approach relies on the success of previous stages in extracting 3D embeddings, improving the robustness of our baseline model is expected to directly merit our overall reconstruction accuracy. Future work may include incorporating human specific priors (e.g., semantic segmentation, pose, and parametric 3D face models) and adding 2D supervision of implicit surface to further support in-the-wild inputs.

References