Video Based Reconstruction of 3D People Models

Introduction

A personalized realistic and animatable 3D model of a human is required for many applications, including virtual and augmented reality, human tracking for surveillance, gaming, or biometrics. This model should comprise the person-specific static geometry of the body, hair and clothing, alongside a coherent surface texture.

One way to capture such models is to use expensive active scanners. But size and cost of such scanners prevent their use in consumer applications. Alternatively, multi-view passive reconstruction from a dense set of static body pose images can be used . However, it is hard for people to stand still for a long time, and so this process is time-consuming and error-prone. Also, consumer RGB-D cameras can be used to scan 3D body models , but these specialized sensors are not as widely available as video. Further, all these methods merely reconstruct surface shape and texture, but no rigged animation skeleton inside. All aforementioned applications would benefit from the ability to automatically reconstruct a personalized movable avatar from monocular RGB video. Despite remarkable progress in reconstructing 3D body models or free-form surface from depth data, 3D reconstruction of humans in clothing from monocular video (without a pre-recorded scan of the person) has not been addressed before. In this work, we estimate the shape of people in clothing from a single video in which the person moves. Some methods infer shape parameters of a parametric body model from a single image , but the reconstruction is limited to the parametric space and can not capture personalized shape detail and clothing geometry. To estimate geometry from a video sequence, we could jointly optimize a single free-form shape constrained by a body model to fit a set of $F$ images. Unfortunately, this requires to optimize $F$ poses at once and more importantly it requires storing $F$ models in memory during optimization which makes it computationally expensive and unpractical.

The key idea of our approach is to generalize visual hull methods to monocular videos of people in motion. Standard visual hull methods capture a static shape from multiple views. Every camera ray through a silhouette point in the image casts a constraint on the 3D body shape. To make visual hulls work for monocular video of a moving person it is necessary to “undo” the human motion and bring it to a canonical frame of reference. In this work, the geometry of people (in wide or tight clothing) is represented as a deviation from the SMPL parametric body model of naked people in a canonical T-pose; this model also features a pose-dependent non-rigid surface skinning. We first estimate an initial body shape and 3D pose at each frame by fitting the SMPL model to 2D detections similar to . Given such fits, we associate every silhouette point in every frame to a 3D point in the body model. We then transform every projection ray according to the inverse deformation model of its corresponding 3D model point; we call this operation unposing (Fig. 2). After unposing the rays for all frames we obtain a visual hull that constrains the body shape in a canonical T-pose. We then jointly optimize body shape parameters and free-form vertex displacements to minimize the distance between 3D model points and unposed rays. This allows us to efficiently optimize a single displacement surface on top of SMPL constrained to fit all frames at once, which requires storing only one model in memory (Fig. 1). Our technique allows for the first time extracting accurate 3D human body models, including hair and clothing, from a single video sequence of the person moving in front of the camera such that the person is seen from all sides. Our results on several 3D datasets show that our method can reconstruct 3D human shape to a remarkable accuracy of 4.5 mm (even higher 3.1 mm with ground truth poses) despite monocular depth ambiguities. We provide our dataset and source code of our method for research purposes .

Related Work

Shape reconstruction of humans in clothing can be classified according to two criteria: (1) the type of sensor used and (2) the kind of template prior used for reconstruction. Free-form methods typically use multi-view cameras, depth cameras or fusion of sensors and reconstruct surface geometry quite accurately without using a strong prior on the shape. In more unconstrained and ambiguous settings, such as in the monocular case, a parametric body model helps to constrain the problem significantly. Here we review free-form and model-based methods and focus on methods for monocular images.

methods reconstruct the moving geometry by deforming a mesh or using a volumetric representation of shape . The advantage of these methods is that they allow reconstruction of general dynamic shapes provided that a template surface is available initially. While flexible, such approaches require high-quality multi-view input data which makes them impractical for many applications. Only one approach showed reconstruction of human pose and deforming cloth geometry from monocular video using a pre-captured shape template . Using a depth camera, systems like KinectFusion allow reconstruction of 3D rigid scenes and also appearance models by incrementally fusing geometry in a canonical frame. A number of methods adapt KinectFusion for human body scanning . The problem is that these methods require separate shots at different time instances. The person thus needs to stand still while the camera is turned around, or subtle pose changes need to be explicitly compensated. The approach in generalized KinectFusion to non-rigid objects. The approach performs non-rigid registration between the incoming depth frames and a concurrently updated, initially incomplete, template. While general, such template-free approaches are limited to slow and careful motions. One way to make fusion and tracking more robust is by using multiple kinects or multi-view ; such methods achieve impressive reconstructions but do not register all frames to the same template and focus on different applications such as streaming or remote rendering for telepresence, e.g., in the holoportation project . Pre-scanning the object or person to be tracked reduces the problem to tracking the non-rigid deformations. Some works are in-between free-form and model-based methods. In they pre-scan a template and insert a skeleton and in they use a skeleton to regularize dynamic fusion. Our work is also related to the seminal work of where they align visual hulls over time to improve shape estimation. In the articulated case, they need to segment and track every body part separately and then merge the information together in a coarse voxel model; more importantly, they need multi-view input. In they compensate for small motions of captured objects by de-blurring occupancy images but no results are shown for moving humans. In they reconstruct the shape of clothed humans in outdoor environments from RGB video, requiring the subject to stand still. All these works use either multi-view systems, depth cameras or do not handle moving humans. In contrast, we use a single RGB video of a moving person, which makes the problem significantly harder as geometry can not be directly unwarped as it is done in depth fusion papers.

Model-based.

Several works leverage a parametric body model for human pose and shape estimation from images . Early models in computer vision were based on simple primitives . Recent ones are learned from thousands of scans of real people and encode pose, and shape deformations . Some works reconstruct the body shape from depth data sequences exploiting the temporal information. Typically, a single shape and multiple poses are optimized to exploit the temporal information. Using multi-view some works have shown performance capture outdoors by leveraging a sum of Gaussians body model or using a pre-computed template . A number of works are restricted to estimating the shape parameters of a body model from multiple views or single images with manually clicked points; silhouettes shading cues and color have been used for inference. Some works fit a body model to images using manual intervention with the goal of image manipulation. Shape and clothing from a single image is recovered in but the user needs to click points in the image and select the clothing types from a database. In they obtain shape from contour drawings. The advance in 2D pose detection has made 3D pose and shape estimation possible in challenging scenarios. In they fit a 3D body model to 2D detections; since only model parameters are optimized and these methods heavily rely on 2D detections, results tend to be close to the shape space mean. In they add a silhouette term to reduce this effect.

Shape Under Clothing.

The aforementioned methods ignore clothing or treat it as noise, but a number of works explicitly reason about clothing. Typically, these methods incorporate constraints such as the body should lie inside the clothing silhouette. In they estimate body shape under clothing by optimizing model parameters for a set of images of the same person in different clothing. In they exploit temporal sequences of scans to estimate shape under clothing. Results are usually restricted to the (naked) model space. In they estimate detailed shape under clothing from scan sequences by optimizing a free-form surface constrained by a body model. The approach in jointly captures clothing geometry and body shape using separate meshes but requires 3D scan sequences as input. DoubleFusion reconstructs clothing geometry and inner body shape from a single depth camera in real time.

Learning based.

Only very few works predict human shape from images using learning methods since images annotated with ground truth shape, pose and clothing geometry are hardly available. A few exceptions are the approach of that predicts shape from silhouettes using a neural network and that predicts garment geometry from a single image. Predictions in are restricted to model shape space and tend to look over-smooth; only garments seen in the dataset can be recovered in . Recent works leverage 2D annotations to train networks for the task of 3D pose estimation . Such works typically predict a stick figure or bone skeleton only, and can not estimate body shape or clothing.

Method

Given a single monocular RGB video depicting a moving person, our goal is to generate a personalized 3D model of the subject, which consists of the shape of body, hair and clothing, a personalized texture map, and an underlying skeleton rigged to the surface. Non-rigid surface deformations in new poses are thus entirely skeleton-driven. Our method consists of 3 steps: 1) pose reconstruction (Sec. 3.2) 2) consensus shape estimation (Sec. 3.3) and 3) frame refinement and texture map generation (Sec. 3.4). Our main contribution is step 2), the consensus shape estimation; step 1) builds on previous work and step 3) to obtain texture and time-varying details is optional.

In order to estimate the consensus shape of the subject, we first calculate the 3D pose in each frame (Sec. 3.2). We extend the method of to make it more robust and enforce better temporal coherence and silhouette overlap. In the second step, the consensus shape is calculated as detailed in Sec. 3.3. The consensus shape is efficiently optimized to maximally explain the silhouettes at each frame instance. Due to time-varying cloth deformations the posed consensus shape might be slightly misaligned with the frame silhouettes. Hence, in order to compute texture and capture time-varying details, in step 3) deviations from the consensus shape are optimized per frame in a sliding window approach (Sec. 3.4). Given the refined frame-wise shapes we can compute the texture map. Our method relies on a foreground segmentation of the images. Therefore, we adopt the CNN based video segmentation method of and train it with 3-4 manual segmentations per sequence. In order to counter ambiguities in monocular 3D human shape reconstruction, we use the SMPL body model as starting point. In the following, we briefly explain how we adapt original SMPL body model for our problem formulation.

SMPL is a parameterized model of naked humans that takes $72$ pose and $10$ shape parameters and returns a triangulated mesh with $N=6890$ vertices. The shape $\boldsymbol{\beta}$ and pose $\boldsymbol{\theta}$ deformations are applied to a base template $\mathbf{T}$, which in the original SMPL model corresponds to the statistical mean shape in the training scans $\mathbf{T}_{\mu}$:

Such offsets $\mathbf{D}$ allow us to deform the model to better explain details and clothing. Offsets are optimized in step 2.

2 Pose Reconstruction

The approach in optimizes SMPL model parameters to fit a set of 2D joint detections in the image. As with any monocular method, scale is an inherent ambiguity. To mitigate this effect, we take inspiration from and extend such that it jointly considers $P=5$ frames and optimizes a single shape and $P=5$ poses. Note that optimizing many more frames would become computationally very expensive and many models would have to be simultaneously stored in memory. Our experiments reveal that even when optimizing over $P=5$ poses the scale ambiguity prevails. The reason is that pose differences induce additional 3D ambiguities which cannot be uniquely decoupled from global size, even on multiple frames . Hence, if the height of the person is known, we incorporate it as constraint during optimization. If height is not known the shape reconstructions of our method are still accurate up to a scale factor (height estimation is roughly off by 2-5 cm). The output of initialization are SMPL model shape parameters $\boldsymbol{\beta}_{0}$ that we keep fixed during subsequent frame-wise pose estimation. In order to estimate 3D pose more reliably, we extend by incorporating a silhouette term:

with the silhouette image of the rendered model $\mathbf{I}_{rn}(\boldsymbol{\theta})$, distance transform of observed image mask $\mathbf{C}$ and its inverse $\bar{\boldsymbol{C}}$, weights $\mathbf{w}$. To be robust to local minima we optimize at 4 different levels of a Gaussian pyramid $G$. We further update the method to use state of the art 2D joint detections and a single-modal A-pose prior. We train the prior from SMPL poses fitted against body scans of people in A-pose. Further, we enforce a temporal smoothness and initialize the pose in a new frame with the estimated pose $\boldsymbol{\theta}$ in the previous frame. If the objective error gets too large, we re-initialize the tracker by setting the pose to zero. While optimization in batches of frames would be beneficial it slows down computation and we have not found significant differences in pose accuracy. The output of this step is a set of poses $\{\boldsymbol{\theta}_{p}\}_{p=1}^{F}$ for the $F$ frames in the sequence.

3 Consensus Shape

Given the set of estimated poses we could jointly optimize a single refined shape matching all original $F$ poses, which would yield a complex, non-convex optimization problem. Instead, we merge all the information into an unposed canonical frame, where refinement is computationally easier. At every frame a silhouette places a new constraint on the body shape; specifically, the set of rays going from the camera to the silhouette points define a constraint cone, see Fig. 2. Since the person is moving, the pose is changing. Our key idea is to unpose the cone defined by the projection rays using the estimated poses. Effectively, we invert the SMPL function for every ray. In SMPL, every vertex $\mathbf{v}$ deforms according to the following equation:

Doing this for every ray effectively unposes the silhouette cone and places constraints on a canonical T-pose, see Fig. 2. Unposing removes blend-shape calculations from the optimization problem and significantly reduces the memory foot-print of the method. Without unposing the vertex operations and the respective Jacobians would have to be computed for every frame at every update of the shape. Given the set of unposed rays for $F$ silhouettes (we use $F=120$ in all experiments), we formulate an optimization in the canonical frame

and minimize it with respect to shape parameters $\boldsymbol{\beta}$ of a template model and the vertex offsets $\mathbf{D}$ defined in Eq. 3. The objective $E_{\text{cons}}$ consists of a data term $E_{\text{data}}$ and three regularization terms $E_{\text{lp}},E_{\text{var}},E_{\text{sym}}$ with weights $w_{*}$ that balance its influence.

measures the distance between vertices and rays. Point to line distances can be efficiently computed expressing rays using Plucker coordinates ($\mathbf{r}=\mathbf{r}_{m},\mathbf{r}_{n})$. Given a set of correspondences $(\mathbf{v}_{i},\mathbf{r})\in\mathcal{M}$ the data term equals

Laplacian Term.

We enforce smooth deformation by adding the Laplacian mesh regularizer :

where $\delta=L(\mathbf{v}(\boldsymbol{\beta}_{0},\mathbf{0}))$ and $L$ is the Laplace operator. The term forces the Laplacian of the optimized mesh to be similar to the Laplacian of the mesh at initialization (where offsets $\mathbf{D}=\mathbf{0}$).

Body Model Term.

We penalize deviations of the reconstructed free-form vertices $\mathbf{v}(\boldsymbol{\beta}_{0},\mathbf{D})$ from vertices explained by the SMPL model $\mathbf{v}(\boldsymbol{\beta},\mathbf{0})$:

Symmetry Term.

Humans are usually axially symmetrical with respect to the Y-axis. Since the body model is nearly symmetric, we add a constraint on the offsets alone that enforces a symmetrical shape:

where $\mathcal{S}$ contains all pairs of Y-symmetric vertices. We phrase this as a soft-constraint to allow potential asymmetries in clothing wrinkles and body shapes. Since the refined consensus shape still has the mesh topology of SMPL, we can apply the pose-based deformation space of SMPL to simulate surface deformation in new skeleton poses.

Implementation Details.

Body regions that are typically unclothed or where silhouettes are noisy (face, ears, hands, and feet) are more regularized towards the body model using per-vertex weights $\boldsymbol{\tau}$. We optimize $E_{\text{cons}}$ using a “dog-leg” trust region method using the chumpy auto-differentiation framework. We alternate minimizing $E_{\text{cons}}$ with respect to model parameters and offsets and finding point to line correspondences. We also re-initialize $E_{\text{lp}}$, $E_{\text{var}}$, $E_{\text{sym}}$. More implementation details and runtime metrics are given in the supplementary material.

4 Frame Refinement and Texture Generation

After calculating a global shape for the given sequence, we aim to capture the temporal variations. We adapt the energy in Eq. 7 to process frames sequentially. The optimization is initialized with the preceding frame and regularized with neighboring frames:

where $\psi_{j}=1$ for $j=k$ and $\psi_{j}=w_{\text{neigh}}<1$ for neighboring frames. Hence, $w_{\text{neigh}}$ defines the influence of neighboring frames and $E_{\text{last}}$ regularizes the reconstruction to the result of the preceding frame. To create the texture, we warp our estimated canonical model back to each frame, back-project the image color to all visible vertices, and finally generate a texture image by calculating the median of the most orthogonal texels from all views. An example of keyframes we use for texture mapping and the resulting texture image is shown in Fig. 3.

Experiments

We study the effectiveness of our method, qualitatively and quantitatively, in different scenarios. For quantitative evaluation, we used two publicly available datasets consisting of 3D scan sequences of humans in motion: with minimal clothing (MC) (DynamicFAUST ) and with clothing (BUFF ). Since these datasets were recorded without RGB sensors we simply render images of the scans using a virtual camera and use them as input. In order to evaluate our method on more varied clothing and backgrounds, we captured a new test dataset (People-Snapshot dataset), and present qualitative results. To the best of our knowledge, our method is the first approach that enables detailed human body model reconstruction in clothing from a single monocular RGB video without requiring a pre-scanned template or manually clicked points. Thus, there exist no methods with the same setting as ours. Hence, we provide a quantitative comparison to the state-of-the-art RGB-D based approach KinectCap on their dataset. The image sequences and ground truth scans were provided by the authors of . While reconstruction from monocular videos is much harder than from depth videos, a comparison is still informative. In all experiments, the method’s parameters are set to two constant values, one set for clothed and one set for people in MC, which are empirically determined.

We take all 9 sequences of 5 different subjects in the BUFF dataset and all 9 sequences of 9 subjects from the DynamicFaust dataset performing “Hip” movements, featuring strong fabric movement or soft tissue dynamics respectively. Each dynamic sequence consists of 300-800 frames. To simulate the subject rotating in front of a camera, we create a virtual camera at 2.5 meters away from the 3D scans of the subject. We rotate the camera in a circle around the person moving one time per sequence. The foreground masks are easily obtained from the alpha channel of the rendered images. For BUFF we render images with real dynamic textures; for DynamicFAUST since textures are not available we rendered shaded models.

In Fig. 5, we show some examples of our reconstruction results on image sequences rendered from BUFF and DynamicFAUST scans. The complete results of all 9 sequences are provided in the supplementary material. To be able to quantitatively evaluate the reconstruction quality, we adjust the pose and scale of our reconstruction to match the ground truth body scans following . Then, we compute a bi-directional vertex to surface distance between our reconstruction and the ground truth geometry. Per-vertex errors (in millimeters) on all sequences are provided in Tab. 1. The heatmaps of per-vertex errors are shown in Fig. 5. As can be seen, our method yields accurate reconstruction on all sequences including personalized details. To study the importance of the pose estimation component, we report the accuracy of our method using ground truth poses versus using estimated poses full method. Ground truth poses were obtained by registering SMPL to the 3D scans. The results of the ablation evaluation are also shown in Fig. 5 and Tab. 1. We can see that our complete pipeline achieved comparable accuracy with the one using ground truth poses which demonstrates robustness. Results show that there is still room for improvement in 3D pose reconstruction.

2 Qualitative Results on RGB Images

We also evaluate our method on real image sequences. The People-Snapshot dataset consists of 24 sequences of $11$ subjects varying a lot in height and weight. The sequences are captured with a fixed camera, and we ask the subjects to rotate while holding an A-pose. To cover a variety of clothing, lighting conditions and background, the subjects were captured with varying sets of garments and with three different background scenes: in the studio with green screen, outdoor, and indoor with complex dynamic background. Some examples of our reconstruction results are shown in Fig. 6 and Fig. LABEL:fig:teaser. We show more example in the supplementary material and in the video. We can see that our method yields detailed reconstructions of similar quality as the results on rendered sequences, which demonstrates that our method generalizes well on the real world scenarios. The benefits of our method are further evidenced by overlaying the re-posed final reconstruction on to the input images. As shown in Fig. 8, our reconstructions precisely overlay the body silhouettes in the input images.

3 Comparison with KinectCap

We compare our method to on their collected dataset. Subjects were captured in both A-pose and T-poses in this dataset. Since T-poses (zero-pose in SMPL) are rather unnatural, they are not well captured in our general pose-prior. Hence, we adjust our pose prior to contain also T-poses. Note that their method relies on depth data, while ours only uses the RGB images. Notably, our method obtains comparable results qualitatively and quantitatively despite solving a much more ill-posed problem. This is further evidenced by the per-vertex errors in Tab. 1.

4 Surface Refinement Using Shading

As mentioned before, our method captures both body shape and medium level surface geometry. In contrast to pure model-based methods, we already add significant details (Fig. 4). Using existing shape from shading methods the reconstruction can be further improved by adding the finer level details of the surface, e.g. folding and wrinkles. Fig. 9 shows an example result of applying the shape from shading method of to our reconstruction. This application further demonstrates the accuracy of our reconstruction, since such good result cannot be obtained without an accurate model-to-image alignment.

Discussion and Conclusions

We have proposed the first approach to reconstruct a personalized 3D human body model from a single video of a moving person. The reconstruction comprises personalized geometry of hair, body, and clothing, surface texture, and an underlying model that allows changes in pose and shape. Our approach combines a parametric human body model extended by surface displacements for refinement, and a novel method to morph and fuse the dynamic human silhouette cones in a common frame of reference. The fused cones merge the shape information contained in the video, allowing us to optimize a detailed model shape. Our algorithm not only captures the geometry and appearance of the surface, but also automatically rigs the body model with a kinematic skeleton enabling approximate pose-dependent surface deformation. Quantitative results demonstrate that our approach can reconstruct human body shape with an accuracy of 4.5mm and an ablation analysis shows robustness to noisy 3D pose estimates.

The presented method finds its limits in appearances that do not share the same topology as the body: long open hair or skirts can not be modeled as an offset from the body. Furthermore, we can only capture surface details that are seen on the outline of at least one view. This means especially concave regions like armpits or inner thighs are sometimes not well handled. Strong fabric movement caused by fast skeletal motions will additionally result in decreased level of detail. In future work, we plan to incorporate illumination and material estimation alongside with temporally varying textures in our method to enable realistic rendering and video augmentation.

For the first time, our method can extract realistic avatars including hair and clothing from a moving person in a monocular RGB video. Since cameras are ubiquitous and low cost, people will be able to digitize themselves and use the 3D human models for VR applications, entertainment, biometrics or virtual try-on for online shopping. Furthermore, our method precisely aligns models with the images, which opens up many possibilities for image editing.

Acknowledgments The authors gratefully acknowledge funding by the German Science Foundation from project DFG MA2555/12-1. We would like to thank Rudolf Martin and Juan Mateo Castrillon Cuervo for great help in data collection and processing. Another thanks goes to Federica Bogo and Javier Romero for providing their results for comparison.

References