C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

Introduction

3D reconstruction of static scenes is mature, but the problem remains challenging when objects can deform due to articulation and intra-class variations. In some cases, deformations can be avoided by capturing multiple simultaneous images of the object. However, this requires expensive hardware comprising several imaging sensors and only provides instantaneous 3D reconstructions of the objects without modelling their deformations. Extracting deformation models requires establishing correspondences between the instantaneous 3D reconstructions, which is often done by means of physical markers. Modern systems such as the Panoptic Studio [joo2015panoptic] can align 3D reconstructions without markers, but require complex specialized hardware, making them unsuitable for use outside a specialized laboratory.

In this paper, we thus consider the problem of reconstructing and modelling 3D deformable objects given only unconstrained monocular views and keypoint annotations. Traditionally, this problem has been regarded as a generalization of static scene reconstruction, and approached by extending Structure from Motion (SFM) techniques. Due to their legacy, such Non-Rigid SFM (NR-SFM) methods have often focused on the geometric aspects of the problem, but the quality of the reconstructions also depends on the ability to model statistically the object shapes and deformations.

We argue that modern deep learning techniques may be used in NR-SFM to capture much better statistical models of the data than the simple low-rank constraints employed in traditional approaches. We thus propose a method that reconstructs the object in 3D while learning a deep network that models it. This network is inspired by recent approaches [kudo2018unsupervised, kanazawa2018learning, pavlakos2018learning, drover20183dpose, kar2015category] that accurately lift 2D keypoints to 3D given a single view of the object. The difference is that our network does not require 3D information for supervision, but is instead trained jointly with 3D reconstruction from 2D keypoints.

Our model, named C3DPO, has two important innovations. First, it performs 3D reconstruction by factoring the effects of viewpoint changes and object deformations. Hence, it reconstructs the 3D object in a canonical frame that registers the overall 3D rigid motion and leaves as residual variability only the motion “internal” to the object.

However, achieving this factorization correctly is non-trivial, as noted extensively in the NR-SFM literature [xiao2004dense]. Our second innovation is a solution to this problem. We observe that, if two 3D reconstructions overlap up to a rigid motion, they must coincide (since the reconstruction network should remove the effect of a rigid motion). Hence, any class of 3D shapes equivalent up to a rigid motion must contain at most one canonical reconstruction. If so, there exits a “canonicalization” function that maps elements in each equivalent class to this canonical reconstruction. We exploit this fact by learning, together with the reconstruction network, a second network that performs this canonicalization, which regularizes the solution.

Empirically, we show that these innovations lead to a very effective and robust approach to non-rigid reconstruction and modelling of deformable 3D objects from unconstrained 2D keypoint data. We compare C3DPO against several traditional NR-SFM baselines as well as other approaches that use deep learning [kanazawa2018learning, kudo2018unsupervised]. We test on a number of benchmarks, including Human3.6M, PASCAL3D+, and Synthetic Up3D, showing superior results for methods that make no use of ground-truth 3D information.

Related work

There are several lines of work which address the problem of 3D shape and viewpoint recovery of a deforming object from 2D observations. This section covers relevant work in NR-SFM and recent deep-learning based methods.

There are several solutions to the NR-SFM problem which can recover the viewpoint and 3D shape of a deforming object from 2D keypoints across multiple frames [akhter2009nonrigid, bregler2000recovering, akhter2011trajectory, dai2014simple], the majority of which are based on Bregler’s factorization framework [bregler2000recovering]. However the NR-SFM problem is severely under constrained as both the camera and 3D object are moving along with the object deforming. This poses a challenge in correctly factoring the viewpoint and shape [xiao2004dense], and additional problems with missing values in the observations. Priors about the shape and the camera motion are employed to improve conditioning of the problem, including the use of low-rank subspaces in the spatial domain [agudo2018image, fragkiadaki2014grouping, dai2014simple, zhu2014complex], temporal domain, for example, fitting 2D keypoint trajectories to a set of predefined DCT basis functions [akhter2009nonrigid, akhter2011trajectory], spatio-temporal domain [agudo2017dust, gotardo2011non, kumar2018scalable, kumar2017spatial], multiple unions of low-rank subspaces [zhu2014complex, agudo2018deformable], learning an overcomplete dictionary of basis shapes from 3D motion capture data and imposing an L1 penalty on basis coefficients [zhou2016sparseness, zhou2016sparse] and imposing Gaussian priors on the shape coefficients [torresani2008nonrigid].

Many of these approaches however, as we have empirically verified, are not scalable and can only reliably reconstruct datasets of few thousands of images and hundreds of keypoints. Furthermore, many of them require keypoint correspondences for the same instance from multiple images from a monocular view or from multi-view cameras. Finally, in contrast to our method, using the listed approaches it is difficult or computationally expensive to reconstruct new test samples after training on a fixed collection of training shapes.

Also related are methods that reconstruct shapes of a visual object category, such as cars or birds. [cashman2013shape] is an early work that learns a morphable model of dolphins from 2D keypoints and segmentation masks. Using similar supervision, Vicente et al. [vicente2014reconstructing, carreira2015virtual] reconstruct the categories of PASCAL VOC. An important part of the pipeline is an initial SFM algorithm which returns a mean shape and camera matrices of each object category. Similarly, Kar et al. [kar2015category] utilize an NR-SFM method for reconstructing categories from PASCAL3D+. [novotny17learning] proposed the first purely image-driven method for single-view reconstruction of rigid object categories. Most recently, Kanazawa et al. [kanazawa2018learning] train a deep network capable of learning both shape and texture of deformable objects. The commonality among the aforementioned methods is their reliance on the initial SFM/NR-SFM step which can often fail. Our method overcomes this problem by learning a monocular shape predictor in a single step without any additional, potentially unreliable, preprocessing steps.

Our approach is related to weakly supervised methods that lift 2D human skeleton keypoints to 3D given a single input view. Besides the fully supervised methods [martinez2017simple, moreno20173d], several works have explored multi-view supervision [kocabas2019self, pavlakos2017harvesting, rhodin2018learning], ordinal depth supervision [pavlakos2018ordinal], unpaired 2D-3D data [pavlakos2018learning, tung2017adversarial, zhou2016sparseness, kanazawa2018end] or videos [kanazawa2019learning] to alleviate the need for full 2D-3D annotations. While these auxiliary sources of supervision allow for compelling 3D predictions, in this work we use only inexpensive 2D keypoint labels.

Closer to our supervisory scheme, [kudo2018unsupervised, drover20183dpose] recently proposed a method that rotates the 3D-lifted keypoints into new views and validates the resulting projections with an adversarial network that learns the distribution of plausible 2D poses. However, both methods require all keypoints to be visible in every frame. This restricts their use to ‘multi-view’ datasets such as Human3.6M. In addition to the 2D keypoints, [drover20183dpose] use the intrinsic camera parameters, and 3D ground truth data to generate new synthetic 2D views, which leads to substantially better quantitative results at the cost of a greater level of supervision.

To conclude, our contribution differs from prior work as it 1) recovers both 3D canonical shape and viewpoint using only 2D keypoints in a single image at test time, 2) uses a novel self-supervised constraint to correctly factorize 3D shape and viewpoint, 3) can handle occlusions and missing values in the observations, 4) works effectively across multiple object categories.

Method

We start by summarizing some background facts about SFM and NR-SFM and then we introduce our method.

Hence, SFM can be formulated as factoring the views $Y$ into viewpoints $M$ and structure $X$. This factorization is not unique, resulting in a mild reconstruction ambiguity, as discussed in supplementary material.

2 Non-rigid structure from motion

Given multiple views of the points, the goal of NR-SFM is to recover the views, the poses, and the shape basis from observations $y_{nk}=\Pi(R_{n}\sum_{d=1}^{D}\alpha_{nd}S_{dk}+T_{n}).$ As in SFM, for orthographic projection the translation can be removed from the equation by centering, and NR-SFM can be expressed as a multi-linear matrix factorization problem:

where the $N$ camera view matrices are contained in the block-diagonal matrix $\bar{M}=\operatorname{diag}(M_{1},\dots,M_{N}).$ Like SFM, this factorization has mild ambiguities, discussed in the supplementary material.

3 Monocular motion and structure estimation

Once the shape basis $S$ is learned, model (3) can be used to reconstruct viewpoint and pose given a single view $Y$ of the object, yielding monocular reconstruction. However, this still requires solving a matrix factorization problem.

For C3DPO, we propose to instead learn a mapping $\Phi$ that performs this factorization in a feed-forward manner, recovering the view matrix $M$ and the pose parameters $\alpha$ from the keypoints $Y$:

The benefit of using a learned mapping, besides speed, is the fact that it can embody prior information on the structure of the object which is not apparent in the linear model. The mapping itself is learned by minimizing the re-projection loss obtained by averaging the loss over visible keypoints:

where $(\alpha,\theta)=\Phi(Y,v)$ and $\|z\|_{\epsilon}=(\sqrt{1+(\|z\|/\epsilon})^{2}-1)\epsilon$ is the pseudo-huber loss with soft threshold $\epsilon$We set $\epsilon=0.01$ in all experiments.. Given a dataset $(Y,v)\in\mathcal{D}$ of views of an object category, the neural network $\Phi$ is trained by minimizing the empirical average of this loss. This setup is illustrated in the bottom half of fig. 2.

4 Consistent factorization via canonicalization

A challenge with NR-SFM is the ambiguity in decomposing variations in the 3D shape of an object into viewpoint changes (rigid motions) and internal object deformations [xiao2004dense]. In this section, we propose a novel approach to directly encourage the reconstruction network $\Phi$ to be consistent in the way reconstructions are performed. This means that it must not be possible for the network to produce two different 3D reconstructions that differ only by a rigid motion, because such a difference should have been instead explained as a viewpoint change.

Formally, let $\mathcal{X}_{0}$ be the set of all reconstructions $X(\alpha;S)$ obtained by the network, where the parameters $(\alpha,\theta)=\Phi(Y,v)$ are obtained by considering all possible views $(Y,v)$ of the object. If the network factorizes viewpoint and pose consistently, then there cannot be two different reconstructions $X,X^{\prime}\in\mathcal{X}_{0}$ related by a mere viewpoint change $X^{\prime}=RX$. This is formalized by the following definition:

For the purpose of learning, we propose to enforce transversality via the following characterizing property (proofs in the supplementary material):

Intuitively, this lemma states that, if $\mathcal{X}_{0}$ has the transversal property, then any rotation of its elements can be undone unambiguously. Otherwise stated, we can construct a canonicalization function with range in the set of reconstructions $\mathcal{X}_{0}$ if, and only if, this set contains only canonical elements, i.e. it has the transversal property (definition 1).

For C3DPO, the lemma is used to enforce a consistent decomposition in viewpoint and pose via the following loss:

where $R\in SO(3)$ is a randomly-sampled rotation, and $\Psi$ is a regressor canonicalization network trained in parallel with the factorization network $\Phi$.

5 In-plane rotation invariance

Rotation equivariance is another property of the factorization network that can be used to constrain learning. Let $Y=\Pi RX$ be a view of the 3D structure $X$. Rotating the camera around the optical axis has the effect of applying a rotation $r_{z}\in SO(2)$ to the keypoints. Hence, the two reconstructions $\Phi(Y,v)=(\alpha,\theta)$ and $\Phi(r_{z}Y,v)=(\alpha^{\prime},\theta^{\prime})$ must yield the same 3D structure $\alpha=\alpha^{\prime}$. This is captured via a modified reprojection loss that exchanges $\alpha$ for $\alpha^{\prime}$:

Experiments

In this section, we compare our method against several strong baselines. First, the employed benchmarks are described followed by quantitative and qualitative evaluations.

We consider three diverse benchmarks containing images of objects with 2D keypoints annotations. The datasets differ by keypoint density, object type, deformations, and intra-class variations.

Synthetic Up3D (S-Up3D) We first validate C3DPO in a noiseless setting using a large synthetic 2D/3D dataset of dense human keypoints based on the Unite the People 3D (Up3D) dataset [lassner2017unite]. For each Up3D image, the SMPL body shape and pose parameters are provided and are used to produce a mesh with 6890 vertices. Each of the 8515 meshes is randomly rotated into 30 different views and the orthographic projection of each vertex is recorded along with its visibility (computed using a ray tracer). The goal is then to recover the 3D shapes given the set of 2D keypoint renders. We maintain the same train/test split as in the Up3D dataset.

Similar to [lassner2017unite], performance is evaluated on the 79 representative vertices of the SMPL model. Although C3DPO can reconstruct the original set of 6890 SMPL model keypoints effortlessly, we evaluate on a subset of points due to a limited scalability of some of the baselines [torresani2008nonrigid, fragkiadaki2014grouping]. For the same reason, we further randomly sampled the generated test poses to 15k images. Performance is measured by averaging a 3D reconstruction error metric (see below) over all frames in the test set.

PASCAL3D+ [xiang2014beyond] Similar to [kanazawa2018learning, tulsiani2017learning], we evaluate our method on the the PASCAL3D+ dataset which consists of PASCAL VOC and ImageNet images for 12 rigid object categories with a set of sparse keypoints annotated on each image (deformations still arise due to intra-class shape variations). There are up to 10 CAD models available for each category, from which one is manually selected and aligned for each image, providing an estimate of the ground truth 3D keypoint locations. To maintain consistency between the 2D and 3D keypoints, we use the 2D orthographic projections of the aligned CAD model keypoints as opposed to the per-image 2D keypoint annotations, and update the visibility indicators based on the CAD model annotations.

Human3.6M [ionescu2014human36M] is perhaps the largest dataset of human poses annotated with 3D ground truth extracted using MoCap systems. As in [kudo2018unsupervised], two variants of the dataset are used: the first contains ground-truth 2D keypoints during both train and test time and in the second, 2D keypoint locations are obtained by the Stacked Hourglass network of [toshev2014deeppose]. We closely follow the evaluation protocol of [kudo2018unsupervised] and report absolute errors measured over 17 joints without any procrustes alignment. We maintain the same train and test split as [kudo2018unsupervised], and report an average over errors attained for each frame in a given MoCap sequence of an action type.

CUB-200-2011 [Wah2011cubbirds] consists of 11,788 images of 200 bird species. Each image is annotated with 2D locations of 15 semantic keypoints and corresponding visibility indicators. There are no ground truth 3D keypoints for this dataset so we only perform a qualitative evaluation. We use the 2D annotations from [kanazawa2018learning].

2 Evaluation metrics

In order to evaluate MPJPE properly, two types of projection ambiguities have to be handled. To deal with the absolute depth ambiguity, for Human3.6M we follow [kudo2018unsupervised] and normalize each pose by applying a translation that puts the skeleton root to the origin of the coordinate system. For PASCAL3D+ and S-Up3D, the mean depth of predicted and ground truth point clouds is zero centered before evaluation. The second, depth flip ambiguity, is resolved as in [torresani2008nonrigid] by evaluating MPJPE twice for the original and depth-flipped point cloud, retaining the better of the two.

3 Baselines

C3DPO is compared to several strong baselines. EM-SfM [torresani2008nonrigid] and GbNrSfM [fragkiadaki2014grouping] are NR-SFM methods with publicly available code. Because, when using [fragkiadaki2014grouping, torresani2008nonrigid], it is difficult to make predictions on previously unseen data, we run the two methods directly on the test set and report results after convergence. This gives the two baselines an advantage over our method. On Human3.6M, out of several available methods, we compare with [kudo2018unsupervised] (Pose-GAN) which is a current state-of-the-art approach for unsupervised 3D pose estimation that does not require any 3D, multiview or video annotations. Unlike other weakly supervised methods [drover20183dpose, kocabas2019self], Pose-GAN does not assume knowledge of the camera intrinsic parameters, hence it is the most comparable to our approach. To ensure fair comparison, we use their public evaluation code together with the provided keypoint detections of the Stacked Hourglass model. Pose-GAN was not tested on other datasets as the method cannot handle inputs with occluded keypoints. On PASCAL3D+, our method is compared with Category-Specific Mesh Reconstruction (CMR) from [kanazawa2018learning]. CMR provides results for 2 categories out of the 12 of PASCAL3D+, but we trained models for all 12 categories using the public code. Note that CMR additionally uses segmentation masks during training, hence has a higher level of supervision than our method.

4 Technical details

Networks $\Psi$ and $\Phi$ share the same core architecture and consist of 6 fully connected residual layers each with 1024/256/1024 neurons (please refer to the supplementary material for architecture details). Residual skip connections were found important since they prevented networks from converging to a rigid average shape.

For Human3.6M, we did not model the translation $T$ of the camera as the centroid of the input 2D keypoints coincides with the centroid of the 3D shape (due to the lack of occluded keypoints). For the other datasets, which contain occlusions, we estimate the camera translation as the difference vector between the mean of the input visible points and the re-projected visible 3D shape keypoints.

In order to adapt our method for the multiclass setting of PASCAL3D+, which has different sets of keypoints for each of the 12 object categories, we adjust the keypoint annotations as follows. For each object category $\mathcal{C}\in\{1,\dots,12\}$ with a set of $K_{\mathcal{C}}$ keypoints $Y_{n}^{\mathcal{C}}\in\mathcal{R}^{2\times K_{\mathcal{C}}}$ in an image $n$, we form a multiclass keypoint annotation $Y_{n}=\left[\mathbf{0},\dots,Y_{n}^{\mathcal{C}},\dots,\mathbf{0}\right]$ by assigning ${Y_{n}^{\mathcal{C}}}$ to the $\mathcal{C}$-th block of $Y_{n}$ and padding with zeros. The visibility indicators $v_{n}$ are expanded in a similar fashion. This avoids reconstructing each class separately, allowing our method to train only once for all classes. This also tests the ability of the model to capture non-rigid deformations not only within, but also across object categories. While this expanded version of keypoint annotations was also tested for GbNrSfM, for EM-SfM, we could not obtain satisfactory performance and reconstructed each class independently. Similarly for CMR, 12 class-specific models were trained separately.

5 Results

Table 1 reports the results on the S-Up3D dataset. Our method outperforms both EM-SfM and GbNrSfM, which validates our approach as a potential replacement for existing NR-SFM methods based on matrix factorization. The table also shows that C3DPO performs substantially better than C3DPO-base, highlighting the importance of the canonicalization network $\Psi$.

For PASCAL3D+ we consider two types of methods. Methods of the first type include GbNrSfM and EM-SfM and take as input 2D ground truth keypoint annotations on the PASCAL3D+ test set, reconstructing it directly. The second type is CMR which uses ground truth annotations for training, but does not use keypoint annotations for evaluation on the test data. In order to make our method comparable with CMR, we used as a detector the High Resolution Residual network (HRNet [sun2019deep]), training it on the 2D keypoint annotations from the PASCAL3D+ training set. The trained HRNet is applied to the test set to extract the 2D keypoints $Y$ and these are lifted to 3D by applying C3DPO (abbreviated as C3DPO+HRNet).

The results are reported in table 2. C3DPO performs better than EM-SfM and GbNrSfM when ground truth keypoints are available during testing. Our method also outperforms CMR by 16%. On several classes (motorbike, train), we obtain significantly better results due to the reliance of CMR on an initial off-the-shelf rigid SFM algorithm that fails to obtain satisfactory reconstructions. This result is especially interesting since, unlike CMR, C3DPO is trained for all classes at once without ground truth segmentation masks. Figure 4 contains qualitative evaluation.

Results on the Human3.6M dataset are summarized in table 3. C3DPO outperforms Pose-GAN for both ground truth and predicted keypoint annotations. Again, C3DPO improves over baseline C3DPO-base by a significant margin. Example reconstructions are in Figure 5.

Similar to PASCAL3D+, in order to make our method comparable with CMR, HRNet is trained on keypoints from the CUB-200-2011 train set and used to predict keypoints on unseen test images which are then input to C3DPO. Figure 6 compares qualitatively our reconstructions to CMR. Our method is capable of modelling more flexible poses than CMR. We hypothesize this is because of the reliance of CMR on an estimate of the camera matrices obtained using rigid SFM which limits the flexibility of the learned deformations. On the other hand, CMR does not use a keypoint detector.

Conclusions

We have proposed a new approach to learn a model of a 3D object category from unconstrained monocular views with 2D keypoints annotations. Compared to traditional solutions that cast this as NR-SFM and solve it via matrix factorization, our solution is based on learning a deep network that performs monocular 3D reconstruction and factorizes internal object deformations and viewpoint changes. While this factorization is an ambiguous task, we have shown a novel approach that constrains the solution recovered by the learning algorithm to be as consistent as possible by means of an auxiliary canonicalization network. We have shown that this leads to considerably better performance, enough to outperform strong baselines on benchmarks that contain large non-rigid deformations within a category (Human3.6M, Up3D) and across categories (PASCAL3D+).

References

Appendix A Theoretical analysis

This section contains additional information regarding various theoretical aspects of the NR-SFM task.

This section summarizes well known results on data centering in orthographic SFM and NR-SFM.

Equations $y_{nk}=\Pi R_{n}X_{k}+\Pi T_{n}$ hold true for all $n=1,\dots,N$ and $k=1,\dots,K$ if, and only if, equations $\bar{y}_{nk}=\Pi R_{n}\bar{X}_{k}$ hold true, where

Average and remove the LHS and RHS of each equation from both sides. ∎

Equation $y_{nk}=\Pi(R_{n}\sum_{d=1}^{D}\alpha_{nd}S_{dk}+T_{n})$ holds true for all $n=1,\dots,N$ and $k=1,\dots,K$ if, and only if, equation $\bar{y}_{nk}=\Pi R_{n}\left(\sum_{d=1}^{D}\alpha_{nd}\bar{S}_{dk}\right)$ holds true, where

Average and remove the LHS and RHS of each equation from both sides. ∎

A.2 Degrees of freedom and ambiguities

Seen as matrix factorization problems, SFM and NR-SFM have intrinsic ambiguities; namely, no matter how many points and views are observed, there is always a space of equivalent solutions that satisfy all the observations. Next, we discuss what are these ambiguities and under which conditions they are minimized.

The ambiguity can be further reduced by considering the fact that the view matrices $M$ are not arbitrary; they are instead the first two rows of rotation matrices. We can exploit this fact by setting $M_{1}=I_{2\times 3}$ (which also standardize the rotation of the first camera), fixing 6 of the 9 DoF.

A.2.2 Non-rigid structure from motion

The NR-SFM equation contains $2NK$ constraints and $6N+ND+3DK$ unknowns. The intrinsic ambiguity has at least 9 DoF as in the SFM case. Hence, for a unique solution (up to the intrinsic ambiguity) we must have $2NK\geq 6N+ND+3DK-9$. Compared to the SFM case, the number of unknowns grows with the number $N$ of views as $(6+D)N$ instead of just $6N$, where $D$ is the dimension of the shape basis. Since the number of constraints grows as $(2K)N$, we must have $K\geq 3+D/2$ keypoints.

Note that once the shape basis $S$ is learned, it is possible to perform 3D reconstruction from a single view by solving (3) for $N=1$; in this case there are $2K$ equations and $6+D$ unknowns, which is once more solvable when $K\geq 3+D/2$.

A.3 Proof of section 3.4

Assume first that $\mathcal{X}_{0}$ has the transversal property. Then the function $\Psi$ is obtained by sending each $RX$ for each $X\in\mathcal{X}_{0}$ back to $X$. This definition is well posed: if $RX=\bar{R}\bar{X}$ where both $X,\bar{X}\in\mathcal{X}_{0}$, then $\bar{X}=(\bar{R})^{-1}RX$ and, due to the transversal property, $X=\bar{X}$.

Assume now that the function $\Psi$ is given and let $X,X^{\prime}\in\mathcal{X}_{0}$ such that $X^{\prime}=RX$ and so $\Phi(X^{\prime})=\Phi(RX)$. However, by definition, $\Phi(RX)=X$ and $\Phi(X^{\prime})=\Phi(IX^{\prime})=X^{\prime}$, so that $X=X^{\prime}$. ∎

Appendix B Architecture of ΨΨ\Psi and ΦΦ\Phi

Figure 8 contains a schema of the architecture of $\Psi$ and $\Phi$ (both share the same core architecture). It consists of 5 fully connected residual blocks with a kernel size of 1. Empirically, we have observed that using residual blocks, instead of the simpler variant with fully connected layers directly followed by batch normalization and no skip connections, prevents the network from predicting flattened shapes.

Appendix C Analysis of robustness

In order to test the robustness of C3DPO to the noise present in the input 2D keypoints, we devised the following experiment.

We generated several noisy versions of the Synthetic Up3D dataset by adding 2D Gaussian noise (with variance $\sigma$) to the 2D input and randomly occluded each 2D input point with probability $p_{OCC}$. Experiments were ran for different number of input of keypoints (79, 100, 500, 1000) and the evaluation was always conducted on the representative 79 vertices (section 4.1) of S-Up3D-test.

The results of the experiment are depicted in fig. 10. We have observed improved robustness to noise with higher numbers of used keypoints. At the same time, the performance without noise ($\sigma=0$, $p_{OCC}=0$) is slightly worse for the setup higher number of keypoints ($\geq$ 500 keypoints). We hypothesize that, when more keypoints are used, the performance deteriorates because the optimizer focuses less on minimizing the reprojection losses of the 79 keypoints that are used for the evaluation.

Appendix D Additional qualitative results

In this section we present additional qualitative results. Figure 7 contains monocular reconstructions of C3DPO trained on the full set of 6890 SMPL vertices of the S-Up3D dataset. Note that we were unable to run [fragkiadaki2014grouping, torresani2004learning] on this dataset due to scalability issues of the two algorithms.

Appendix E C3DPO failure modes

The main sources of failures of our method are: (1) Failures of the 2D keypoint detector [sun2019deep]; (2) Reconstructing “outlier” test 2D poses not seen in training (mainly on Human3.6m); (3) Reconstructing strongly ambiguous 2D poses (in a frontal image of a sitting human, the knee angle cannot be recovered uniquely). The failure mode (1) is depicted in fig. 10.