Monocular Total Capture: Posing Face, Body, and Hands in the Wild

Introduction

Human motion capture is essential for many applications including visual effects, robotics, sports analytics, medical applications, and human social behavior understanding. However, capturing 3D human motion is often costly, requiring a special motion capture system with multiple cameras. For example, the most widely used system VICON needs multiple calibrated cameras with reflective markers carefully attached to the subjects’ body. The actively-studied markerless approaches are also based on multi-view systems Gall-09; Liu-2013; Elhayek-15; joo2017panoptic; joo2018 or depth cameras Shotton2011; Baak2011. For this reason, the amount of available 3D motion data is extremely limited. Capturing 3D human motion from single images or videos can provide a huge breakthrough for many applications by increasing the accessibility of 3D human motion data, especially by converting all human-activity videos on the Internet into a large-scale 3D human motion corpus.

Reconstructing 3D human pose or motion from a monocular image or video, however, is extremely challenging due to the fundamental depth ambiguity. Interestingly, humans are able to almost effortlessly reason about the 3D human body motion from a single view, presumably by leveraging strong prior knowledge about feasible 3D human motions. Inspired by this, several learning-based approaches have been proposed over the last few years to predict 3D human body motion (pose) from a monocular video (image) Taylor2000; Ramakrishna2012; akhter2015pose; Tekin2016; Bogo2016; Mehta2017; martinez2017simple; zhou2017towards; kanazawa2018end; Moreno-noguer2017 using available 2D and 3D human pose datasets Andriluka-14; lin2014microsoft; CMUMocap; h36m_pami; joo2017panoptic. Recently, similar approaches have been introduced to predict 3D hand poses from a monocular view zimmermann2017learning; mueller2018ganerated; Cai_2018_ECCV. However, fundamental difficulty still remains due to the lack of available in-the-wild 3D body or hand datasets that provide paired images and 3D pose data; thus most of the previous methods only demonstrate results in controlled lab environments. Importantly, there exists no method that can reconstruct motion from all body parts including body, hands, and face altogether in a single view, although this is important to fully understand human behavior.

In this paper, we aim to reconstruct the 3D total motions joo2018 of a human using a monocular imagery captured in the wild. This ambitious goal requires solving challenging 3D pose estimation problems for different body parts altogether, which are often considered as separate research domains. Notably, we apply our method to in-the-wild situations (e.g., videos from YouTube), which has rarely been demonstrated in previous work. We use a 3D representation named Part Orientation Fields (POFs) to efficiently encode the 3D orientation of a body part in the 2D space. A POF is defined for each body part that connects adjacent joints in torso, limbs, and fingers, and represents relative 3D orientation of the rigid part regardless of the origin of 3D Cartesian coordinates. POFs are efficiently predicted by a Fully Convolutional Network (FCN), along with 2D joint confidence maps tompson2014joint; Wei2016; cao2017realtime. To train our networks, we collect a new 3D human motion dataset containing diverse body, hands, and face motions from 40 subjects. Separate CNNs are adopted for body, hand and face, and their outputs are consolidated together in a unified optimization framework. We leverage a 3D deformable model that is built for the total capture joo2017panoptic in order to exploit the shape and motion prior embedded in the model. In our optimization framework, we fit the model to the CNN measurements at each frame to simultaneously estimate the 3D motion of body, face, fingers, and feet. Our mesh output also enables us to additionally refine our motion capture results for better temporal coherency by optimizing the photometric consistency in the texture space.

This paper presents the first approach to monocular total motion capture in various challenging in-the-wild scenarios (e.g., Fig. ). We demonstrate that our single framework achieves comparable results to existing state-of-the-art 3D body or hand pose estimation methods on public benchmarks. Notably, our method is applied to various in-the-wild videos, which has rarely been demonstrated in either 3D body or hand estimation area. We also conduct thorough experiments on our newly collected dataset to quantitatively evaluate the performance of our method with respect to viewpoint and body pose changes. The major contributions of our paper are summarized as follows:

We present the first method to produce 3D total motion capture results from a monocular image or a video in various challenging in-the-wild scenarios.

We introduce an optimization framework to fit a deformable human model on 3D POFs and 2D keypoint measurements for total body pose estimation, and show comparable results to the state-of-the-art methods in both 3D body and 3D hand estimation benchmarks.

We present a method to enforce photometric consistency across time to reduce motion jitters.

We capture a new 3D human motion dataset with 40 subjects to provide training and evaluation data for monocular total motion capture.

Related Work

Single Image 2D Human Pose Estimation: Over the last few years, great progress has been made in detecting 2D human body keypoints from a single image toshev2014deeppose; tompson2014joint; bulat2016human; Wei2016; Newell-16; cao2017realtime by leveraging large-scale manually annotated datasets lin2014microsoft; Andriluka-14 with deep Convolutional Neural Network (CNN) framework. In particular, the major breakthrough is boosted by using the fully convolutional architectures to produce confidence scores for each joint with a heatmap representation tompson2014joint; Wei2016; Newell-16; cao2017realtime, which is known to be more efficient than directly regressing the joint locations with fully connected layers toshev2014deeppose. A recent work cao2017realtime similarly learns the connectivity between pairs of adjacent joints, called the Part Affinity Fields (PAFs) in the form of 2D heatmaps, to assemble 2D keypoints for different individuals in the multi-person 2D pose estimation problem.

Single Image 3D Human Pose Estimation: Early work Ramakrishna2012; akhter2015pose models the 3D human pose space as an over-complete dictionary learned from a 3D human motion database CMUMocap. More recent approaches rely on deep neural networks, which are roughly divided into two directions: two-stage methods and direct estimation. The two-stage methods take 2D keypoint estimation as input and focus on lifting 2D human poses to 3D independently without input image Bogo2016; Chen2017; martinez2017simple; Moreno-noguer2017; Nie2017; AAAI1816471. These methods ignore rich information in images that encodes 3D information, such as shading and appearance, and also suffer from sensitivity to 2D localization error. Direct estimation methods predict 3D human pose directly from images, in the form of direct coordinate regression Rogez2016; Sun2017; Sun_2018_ECCV, voxel prediction pavlakos2017coarse; luvizon20182d; varol2018bodynet or depth map prediction zhou2017towards. Similar to ours, a recent work uses 3D orientation fields orinet2018 as an intermediate representation for the 3D body pose. However, these models are usually trained on MoCap datasets, with limited ability to generalize to in-the-wild scenarios.

Due to the above limitations, some methods have been proposed to integrate prior knowledge about human pose for better in-the-wild performance. Some work pavlakos2018ordinal; relativeposeBMVC18; ijcai2018-136 proposes to use ordinal depth as additional supervision for CNN training. Additional loss functions are introduced in zhou2017towards; dabral2018learning to enforce constraints on predicted bone length and joint angles. Some work kanazawa2018end; yang20183d uses Generative Adversarial Networks (GAN) to exploit human pose prior in data-driven approaches.

Monocular Hand Pose Estimation: Hand keypoint estimation is often considered as independent research domain from body pose estimation. Most of previous work is based on depth image as input Oikonomidis-12; Sridhar-13; Sharp-15; Sridha-15; Tzionas-16; Ye-16, while RGB-based method is introduced recently, for 2D keypoint estimation simon2017hand and 3D pose estimation zimmermann2017learning; Cai_2018_ECCV; Iqbal_2018_ECCV.

3D Deformable Human Models: 3D deformable models are commonly used for markerless body anguelov2005scape; Loper2015; pons2015dyna and face motion capture blanz1999morphable; cao2014facewarehouse to restrict the reconstruction output to the parametric shape and motion spaces defined by the models. Although the outputs are limited by the expressive power of models (e.g., some body models cannot express clothing and some face models cannot express wrinkles), they greatly simplify the 3D motion capture problem. We can fit the models based on available measurements by optimizing cost functions with respect to the model parameters. Recently, a generative 3D model that can express body and hands is introduced by Romero et al. romero2017embodied; the Adam model is introduced by Joo et al. joo2018 to enable the total body motion capture (face, body and hands), which we adopt for monocular total capture.

Method Overview

Our method takes as input a sequence of images capturing the motion of a single person from a monocular RGB camera, and outputs the 3D total body motion capture (including the motion from body, face, hands, and feet) of the target person in the form of a deformable 3D human model Loper2015; joo2018 for each frame. Given a $N$-frame video sequence, our method produces the parameters of the 3D human body model joo2018, including body motion parameters $\{\boldsymbol{\theta}_{i}\}_{i=1}^{N}$, facial expression parameters $\{\boldsymbol{\sigma}_{i}\}_{i=1}^{N}$, and global translation parameters $\{\boldsymbol{t}_{i}\}_{i=1}^{N}$. The body motion parameter $\boldsymbol{\theta}$ includes hands and feet motions, as well as the global rotation of the body. Our method also estimates shape coefficients $\boldsymbol{\phi}$ shared among all frames in the sequence, while $\boldsymbol{\theta}$, $\boldsymbol{\sigma}$, and $\boldsymbol{t}$ are estimated for each frame respectively. The output parameters are defined by the 3D deformable human model Adam joo2018. Note that our method can be also applied to capture only a subset of total motions (e.g., body motion only with the SMPL model Loper2015 or hand motion only by separate hand model of Frankenstein in joo2018). We denote a set of all parameters $(\boldsymbol{\phi},\boldsymbol{\theta},\boldsymbol{\sigma},\boldsymbol{t})$ by $\boldsymbol{\Psi}$, and denote the result for the $i$-th frame by $\boldsymbol{\Psi}_{i}$.

Our method is divided into 3 stages, as shown in Fig. 1. In the first stage, each image is fed into a Convolutional Neural Network (CNN) obtain the joint confidence maps and the 3D orientation information of body parts, which we call the 3D Part Orientation Fields (POFs). In the second stage, we perform total body motion capture by fitting a deformable human mesh model joo2018 on the image measurements produced by the CNNs. We utilize the prior information embedded in the human body model for better robustness of results against the noise in CNN outputs. This stage produces the 3D pose for each frame independently, represented by parameters of the deformable model $\{\boldsymbol{\Psi}_{i}\}_{i=1}^{N}$. In the third stage, we additionally enforce temporal consistency across frames to reduce motion jitters. We define a cost function to ensure photometric consistency in the texture domain of mesh model, based on the initial fitting outputs of the second stage. This stage produces refined model parameters $\{\boldsymbol{\Psi}^{+}_{i}\}_{i=1}^{N}$. We demonstrate that this temporal refinement is crucial to obtain realistic body motion capture output.

Predicting 3D Part Orientation Fields

Note that the POF values are defined only for the pixel region belonging to the current target part $\mathbf{P}_{(m,n)}$ and we follow cao2017realtime to define the pixels belonging to the part as a rectangular (please refer to cao2017realtime for details). An example POF of a body part (right lower arm) is shown in Fig. 2.

Model-Based 3D Pose Estimation

Given the 2D joint confidence maps $\mathbf{S}^{B}$ predicted by our CNN for body, we obtain 2D keypoint locations $\{\mathbf{j}^{B}_{m}\}_{m=1}^{J}$ by taking the maximum in each channel of $\mathbf{S}^{B}$. Given the $\{\mathbf{j}^{B}_{m}\}_{m=1}^{J}$ and the other CNN output POFs $\mathbf{L}^{B}$, we compute the 3D orientation of each bone ${\hat{\mathbf{P}}}^{B}_{(m,n)}$, by averaging the values of $\mathbf{L}^{B}$ along the segment from $\mathbf{j}^{B}_{m}$ to $\mathbf{j}^{B}_{n}$, as in cao2017realtime. We obtain a set of mesh parameters $\boldsymbol{\theta}$, $\boldsymbol{\phi}$, and $\boldsymbol{t}$ that agree with these image measurements by minimizing the following objective function:

where $\mathcal{F}^{B}_{\text{2D}}$, $\mathcal{F}^{B}_{\text{\tiny POF}}$, and $\mathcal{F}^{B}_{p}$ are different constraints as defined below. The 2D keypoint constraint $\mathcal{F}^{B}_{\text{2D}}$ penalizes the discrepancy between network-predicted 2D keypoints and the projections of the joints in the human body model:

where $\mathbf{A}^{B}_{\theta}$ and $\boldsymbol{\mu}^{B}_{\theta}$ are poses prior learned from CMU Mocap dataset CMUMocap, and $w^{B}_{p}$ is a balancing weight. We use Levenberg-Marquardt algorithm ceres-solver to optimize Equation 3. The mesh fitting process is illustrated in Fig. 3.

2 Total Body Capture with Hands, Feet and Face

Given the network output of the hand network $\mathbf{S}^{LH},\mathbf{L}^{LH}$ and $\mathbf{S}^{RH},\mathbf{L}^{RH}$ for both hands, we can additionally fit the Adam model to satisfy the hand pose using similar optimization objectives:

$\mathcal{F}^{LH}$ is the objective function for left hand and each term is defined similarly to Equation 4, 5 and 6. The hand pose priors are learned from MANO dataset romero2017embodied. The objective function for the right hand $\mathcal{F}^{RH}$ is similarly defined.

Once we fit the body and hand parts of deformable model to the CNN outputs, the projection of the 3D model on the image is already well aligned to the target person. Then we can reconstruct other body parts by simply adding more 2D joint constraints using additional 2D keypoint measurements. In particular, we include 2D face and foot keypoints from the OpenPose detector. The additional cost function for toes is defined as:

Putting everything together, the final optimization objective is

where the balancing weights for all the terms are omitted for clarity. We optimize this final objective function in multiple stages to avoid local minima. We first fit the torso, then add limbs, and finally optimize the full objective function including all constraints. This stage produces 3D total body motion capture results for each frame independently in the form of Adam model parameters $\{\mathbf{\Psi}_{i}\}_{i=1}^{N}$.

Enforcing Photo-Consistency in Textures

In the previous stages, we perform per-frame processing, which is vulnerable to motion jitters. We propose to reduce the jitters using the pixel-level image cues given the initial model fitting results. The core idea is to enforce photometric consistency in textures of the model, extracted by projecting the fitted mesh models on the input images. Ideally, the textures should be consistent across frames, but in practice there exist discrepancies due to motion jitters. In order to efficiently implement this constraint in our optimization framework, we compute optical flows from projected texture to the target input image. The destination of each flow indicates the expected location of vertex projection. To describe our method, we define a function $\mathcal{T}$ which extracts a texture given an image and a mesh structure:

Given the input image $\mathbf{I}_{i}$ of the $i$-th frame and mesh determined by $\mathbf{\Psi}_{i}$, the function $\mathcal{T}$ extracts a texture map $\boldsymbol{\mathcal{T}}_{i}$ by projecting the mesh structure for $i$-th frame on the image for the visible parts. We ideally expect the texture for ($i$+$1$)-th frame $\boldsymbol{\mathcal{T}}_{i+1}$ to be the same as $\boldsymbol{\mathcal{T}}_{i}$. Instead of directly using this constraint for optimization, we use optical flow to compute the discrepancy between these textures for easier optimization. More specifically, we pre-compute the optical flow between the raw image $\mathbf{I}_{i+1}$ and the rendering of the mesh model at ($i$+$1$)-th frame with the $i$-th frame’s texture map $\boldsymbol{\mathcal{T}}_{i}$, which we call ‘synthetic image’:

where $M_{i+1}=M(\mathbf{\Psi}_{i+1})$ is the mesh for the ($i$+$1$)-th frame, and $\mathcal{R}$ is a rendering function that renders a mesh with a texture to an image. The function $f$ computes optical flows from the synthetic image to the input image $\mathbf{I}_{i+1}$. The output flow $\mathbf{f}_{i+1}:\mathbf{x}\longrightarrow\mathbf{x^{\prime}}$ maps a 2D location $\mathbf{x}$ to a new location $\mathbf{x^{\prime}}$ following the optical flow result. Intuitively, the computed flow mapping $\mathbf{f}_{i+1}$ drives the projection of 3D mesh vertices toward the directions for better photometric consistency in textures across frames. Based on this flow mapping, we define the texture consistency term:

where $\mathbf{v}^{+}_{n}(i+1)$ is the projection of the $n$-th mesh vertex as a function of model parameters $\boldsymbol{\Psi}^{+}_{i+1}$ under optimization. $\mathbf{v}^{\prime}_{n}(i+1)=\mathbf{f}_{i+1}(\mathbf{v}_{n}(i+1))$ is the destination of each optical flow, where $\mathbf{v}_{n}(i+1)$ is the projection of $n$-th mesh vertex of mesh $M_{i+1}$. Note that $\mathbf{v}^{\prime}_{n}(i+1)$ is pre-computed and constant during the optimization. This constraint is defined in image space, and thus it mainly reduces the jitters in $x,y$ directions. Since there is no image clue to reduce the jitters along $z$ direction, we just enforce a smoothness constraint for $z$-components of 3D joint locations:

where $\mathbf{J}^{+z}_{m}(i+1)$ is $z$-coordinate of the $m$-th joint of the mesh model as a function of parameters under optimization, and $\mathbf{J}^{z}_{m}(i)$ is the corresponding value in previous frame as a fixed constant. Finally, we define a new objective function:

where the balancing weights are omitted. We minimize this function to obtain the parameter of the ($i$+1)-th frame $\mathbf{\Psi}^{+}_{i+1}$, initialized from output of last stage $\mathbf{\Psi}_{i+1}$. Compared to the original full objective Equation 11, this new objective function is simpler since this optimization starts from a good initialization. Most of the 2D joint constraints are replaced by $\mathcal{F}_{\text{tex}}$, while we found that the POF term and face keypoint term are still needed to avoid error accumulation. Note that this optimization is performed recursively—we use the updated parameters of the $i$-th frame $\boldsymbol{\Psi}^{+}_{i}$ to extract the texture $\boldsymbol{\mathcal{T}}_{i}$ in Equation 12, and update the model parameters at the ($i$+1)-th frame from $\boldsymbol{\Psi}_{i+1}$ to $\boldsymbol{\Psi}^{+}_{i+1}$ with this optimization. Also note that the shape parameters $\{\boldsymbol{\phi}^{+}_{i}\}$ should be the same across the sequence, so we take $\boldsymbol{\phi}^{+}_{i+1}=\boldsymbol{\phi}^{+}_{i}$ and fix it during optimization. We also freeze facial expression and does not optimize it in this stage.

Results

We quantitatively evaluate the performance of our method on public benchmarks for 3D body pose estimation and hand pose estimation. We also thoroughly evaluate our method on view point changes and human pose changes in our newly collected multi-view human pose dataset. For all quantitative experiments, we use the camera intrincics provided by the datasets. We finally show our total motion capture results in various challenging videos recorded by us or obtained from YouTube. Our qualitative results are best shown in our supplementary videos.

Body Pose Dataset: Human3.6M h36m_pami is a large-scale indoor marker-based human MoCap dataset, and currently the most commonly used benchmark for 3D body pose estimation. We quantitatively evaluate the body part of our algorithm on it. We follow the standard training-testing protocol as in pavlakos2017coarse.

Hand Pose Dataset: Stereo Hand Pose Tracking Benchmark (STB) zhang20163d is a 3D hand pose dataset consisting of 30K images for training and 6K images for testing. Dexter+Object (D+O) sridhar2016real is a hand pose dataset captured by an RGB-D camera, providing about 3K testing images in 6 sequences. Only the locations of finger tips are annotated.

Newly Captured Total Motion Dataset: We use the Panoptic Studio joo2015panoptic; joo2017panoptic to capture a new dataset for 3D body and hand pose in a markerless way joo2018. We use 31 HD cameras to capture 40 subjects. Each subject makes a wide range of motion in body and hand under the guidance of a video for 2.5 minutes. After cleaning out the erroneous frames, we obtain about 834K body images and 111K hand images with corresponding 3D pose data. We split this dataset into training and testing set such that no subject appears in both. This dataset will be publicly shared.

2 Quantitative Comparison with Previous Work

Comparison on Human3.6M. We compare the performance of our single-frame body pose estimation method with previous state-of-the-arts. Our network is initialized from the 2D body pose estimation network of OpenPose. We train the network using COCO dataset lin2014microsoft, our new 3D body pose dataset, and Human3.6M for 165k iterations with a batch size of 4. During testing time, we fit Adam model joo2018 onto the network output. Since Human3.6M has a different joint definition from Adam model, we build a linear regressor to map Adam mesh vertices to 17 joints in Human3.6M definition using the training set, as in kanazawa2018end. For evaluation, we follow pavlakos2017coarse to rescale our output to match the size of an average skeleton computed from the training set. The Mean Per Joint Position Error (MPJPE) after aligning the root joint is reported as in pavlakos2017coarse.

The experimental results are shown in Table 1. Our method achieves competitive performance; in particular, we show the lowest pose estimation error among all methods that demonstrate their results on in-the-wild videos (marked with ‘*’ in the table). We argue that this is important because methods are in general prone to overfitting to this specific dataset. As an example, our result with pose prior shows increased error compared to our result without prior, although we find that pose priors helps to produce good surface structure and joint angles in the wild.

Ablation Studies. We investigate the importance of each dataset through ablation studies on Human3.6M. We compare the reconstruction error by training networks with: (1) Human3.6M; (2) Human3.6M and our captured dataset; and (3) Human3.6M, our captured dataset, and COCO. Note that setting (3) is the method we used for the previous comparison. We follow the same evaluation protocol and metric as in Table 1. The result is shown in Table 2. First, it is worth noting that with only Human3.6M as training data, we already achieve the best results among results marked with ‘*’ in Table 1. Second, comparing (2) with (1), our new dataset provides an improvement despite the difference in background, human appearance and pose distribution between our dataset and Human3.6M. This verifies the value of our new dataset. Third, we see a drop in error when we add COCO to the training data, which suggests that our framework can take advantage of this dataset with only 2D human pose annotation for 3D pose estimation.

2.2 3D Hand Pose Estimation.

We evaluate our method on the Stereo Hand Pose Tracking Benchmark (STB) and Dexter+Object (D+O), and compare our result with previous methods. For this experiment we use the separate hand model of Frankenstein in joo2018.

STB. Since the STB dataset has a palm joint rather than the wrist joint used in our method, we convert the palm joint to wrist joint as in zimmermann2017learning to train our CNN. We also learn a linear regressor using the training set of STB dataset. During testing, we regress back the palm joint from our model fitting output for comparison. For the evaluation, we follow the previous work zimmermann2017learning and compute the error after aligning the position of root joint and global scale with the ground truth, and report the Area Under Curve (AUC) of the Percentage of Correct Keypoints (PCK) curve in the 20mm-50mm range. The results are shown in the left of Fig. 5. Our performance is on par with the state-of-the-art methods that are designed particularly for hand pose estimation. We also point out that the performance on this dataset has almost saturated, because the percentage is already above $90\%$ even at the lowest threshold.

D+O. Following mueller2018ganerated and Iqbal_2018_ECCV, we report our results using a PCK curve and the corresponding AUC, as shown in the right of Fig. 5. Since previous methods are evaluated by estimating the absolute 3D depth of 3D hand joints, we follow them by finding an approximate hand scale using a single frame in the dataset, and fix the scale during the evaluation. In this case, our performance (AUC=$0.70$) is comparable with the previous state-of-the-art Iqbal_2018_ECCV (AUC=$0.71$). However, since there is fundamental depth-scale ambiguity for single-view pose estimation, we argue that aligning the root with the ground truth depth is a more reasonable evaluation setting. In this setting, our method (AUC=$0.84$) outperforms the previous state-of-the-art method mueller2018ganerated (AUC=$0.70$) in the same setting, and even achieves better performance than an RGB-D based method sridhar2016real (AUC=$0.81$).

3 Quantitative Study for View and Pose Changes

Our new 3D pose data contain multi-view images with the diverse body postures. This allows us to quantitatively study the performance of our method in view changes and body pose changes. We compare our single view 3D body reconstruction result with the ground truth. Due to the scale-depth ambiguity of monocular pose estimation, we align the depth of root joint to the ground truth by scaling our result along the ray directions from the camera center, and compute the Mean Per Joint Position Error (MPJPE) in centimeter. The average MPJPE for all testing samples is $6.30$ cm. We compute the average errors per each camera viewpoint, as shown in the top of Fig. 6. Each camera viewpoint is represented by azimuth and elevation with respect to the subjects’ initial body location. We reach two interesting findings: first, the performance worsens in the camera views with higher elevation due to the severe self-occlusion and foreshortening; second, the error is larger in back views compared to the frontal views because limbs are occluded by torso in many poses. At the bottom of Fig. 6, we show the performance for varying body poses. We run k-means algorithm on the ground truth data to find body pose groups (the center poses are shown in the figure), and compute the error for each cluster. Body poses with more severe self-occlusion or foreshortening tend to have higher errors.

4 The Effect of Mesh Tracking

To demonstrate the effect of our temporal refinement method, we compare the result of our method before and after this refinement stage using Panoptic Studio data. We plot the reconstructed left shoulder joint in Fig. 7. We find that the result after tracking (in blue) tends to be more temporally stable than that before tracking (in green), and is often closer to the ground truth (in red).

5 Qualitative Evaluation

Qualitative Results on Images: In this section we present qualitative results of our method on individual images in Fig. 8. We show results on images with various background, human appearance and poses. Our method works well for both indoor Mocap images (the first row in Fig. 8) and in-the-wild images (the latter 2 rows).

Qualitative Results on Video Sequences: We show results of our method on video sequences. We test our method on two kinds of videos. First, we take videos of human motion using camera by ourselves; second, we use videos downloaded from Youtube. The results are presented in our supplementary video. For videos where only the upper body of the target person is visible, we assume that the orientation of torso and legs is vertically downward in Equation 5.

Discussion

In this paper, we present a method to simultaneously reconstruct 3D total motion of a single person from an image or a monocular video. We thoroughly evaluate the robustness of our method on various benchmarks and demonstrate monocular 3D total motion capture results on in-the-wild videos. There are some limitations with our method. First, we observe failure cases when a significant part of the target person is invisible (out of image boundary or occluded by other objects) due to erroneous network prediction. Second, our hand pose detector fails in the case of insufficient resolution or severe motion blur. Third, our CNN requires bounding boxes for body and hands as input, and cannot handle multiple bodies or hands simultaneously. Solving these problems points to interesting future directions.

References

Appendix A New 3D Human Pose Dataset

In this section, we provide more details of the new 3D human pose dataset that we collect.

We randomly recruit 40 volunteers on campus and capture their motion in a multi-view system joo2015panoptic; joo2017panoptic. During the capture, all subjects follow the motion in the same video of around 2.5 minutes recorded in advance.

We use multi-view 3D reconstruction algorithms joo2015panoptic; joo2017panoptic; simon2017hand to reconstruct 3D body, hand and face keypoints.

We run filters on the reconstruction results. We compute the average lengths of all bones for every subject, and discard a frame if the difference between the length of any bone in the frame and the average length is above a certain threshold. We further manually verify the correctness of hand annotations by projecting the skeletons onto 3 camera views and checking the alignment between the projection and images.

A.2 Statistics and Examples

To train our networks, we use our captured 3D body data and hand data, include a total of 834K image-annotation pairs for bodies and 111K pairs for hands. Example data are shown in Fig. 9 and our supplementary video.

Appendix B Network Skeleton Definition

Appendix C Deformable Human Model

C.2 3D Keypoints Definition

In this section we specify the correspondences between the keypoints predicted by our networks and Adam keypoints.

In order to get facial expression, we also directly fit Adam vertices using the 2D face keypoints predicted by OpenPose (Equation 9 in the main paper). Note that although OpenPose provides 70 face keypoints, we only use 41 keypoints on eyes, nose, mouth and eyebrows, ignoring those on the face contour. The Adam vertices used for fitting are illustrated in Fig. 11 (right).

Appendix D Implmentation Details

In this section, we provide details about the parameters we use in our implementation.

In Equation 4 and 5 of the main paper, we use

We have similarly defined weights for left and right hands omitted in Equation 7, for which we use

Weights for Equation 10 (omitted in the main paper) are

In Equation 15, a balancing weight is omitted for which we use

In Equation 16, $\mathcal{F}_{\text{POF}}$ consists of POF terms for body, left hands and right hands, i.e., $\mathcal{F}_{\text{POF}}=\mathcal{F}^{B}_{\text{POF}}+\mathcal{F}^{LH}_{\text{POF}}+\mathcal{F}^{RH}_{\text{POF}}$. We use weights $25,1,1$ to balance these 3 terms.