Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild

Introduction

3D human body shape and pose estimation from an RGB image is a challenging computer vision problem, partly due to its under-constrained nature wherein multiple 3D human bodies may explain a given 2D image, especially when the subject is significantly occluded, as is common for in-the-wild images. Several recent works use deep neural networks to regress a single body shape and pose solution, which can result in impressive 3D body reconstructions given sufficient visual evidence in the input image. However, when visual evidence of the subject’s shape and pose is obscured, e.g. due to occluding objects or self-occlusions, a single solution does not fully describe the space of plausible 3D reconstructions. In contrast, we aim to estimate a structured probability distribution over 3D body shape and pose, conditioned on the input image, thereby allowing us to sample any number of plausible 3D reconstructions and quantify prediction uncertainty over the 3D body surface, as shown in Figure 1.

Moreover, to ensure that 3D bodies sampled from the predicted distributions match the 2D input image, we implement a reprojection loss between predicted samples and ground-truth visible 2D joint annotations. To allow for the backpropagation of gradients through the sampling operation, we present a differentiable rejection sampler for matrix-Fisher distributions over relative 3D joint rotations.

Finally, a key obstacle for SMPL body shape regression from in-the-wild images is the lack of training datasets with accurate and diverse body shape labels . To overcome this, we follow and utilise synthetic data, randomly generated on-the-fly during training. Inspired by , we use convolutional edge filters to close the large synthetic-to-real gap and show that using edge-based inputs yields better performance than commonly-used silhouette-based inputs , due to improved robustness and capacity to retain visual shape information.

In summary, our main contributions are as follows:

Given an input image, we predict a novel hierarchical matrix-Fisher distribution over relative 3D joint rotation matrices, whose structure is explicitly informed by the SMPL kinematic tree, alongside a Gaussian distribution over SMPL shape parameters.

We present a differentiable rejection sampler to sample any number of plausible 3D reconstructions and quantify prediction uncertainty over the body surface. This enables a reprojection loss between predicted samples and ground-truth coordinates of visible 2D joints, further ensuring that the predicted distributions are consistent with the input image.

We use simple convolutional edge filters to improve the random synthetic training framework used by . Edge filtering is a computationally-cheap and robust method for closing the domain gap between synthetic RGB training data and real RGB test data.

Related Work

This section reviews approaches to monocular 3D human body shape and pose estimation, as well as deep-learning-based methods for probabilistic rotation estimation.

Monocular 3D shape and pose estimation methods can be classified as optimisation-based or learning-based. Optimisation-based approaches fit a parametric 3D body model to 2D observations, such as 2D keypoints , silhouettes or body part segmentations , by optimising a suitable cost function. These methods do not require expensive 3D-labelled training data, but are sensitive to poor intialisations and noisy observations.

Learning-based approaches can be further split into model-free or model-based. Model-free methods use deep networks to directly output human body vertex meshes , voxel grids or implicit surfaces from an input image. In contrast, model-based methods regress 3D body model parameters , which give a low-dimensional representation of a 3D human body. To overcome the lack of in-the-wild 3D-labelled training data, several methods use diverse 2D-labelled data as a source of weak supervision. extends this approach by incorporating optimisation into their model training loop, lifting 2D labels to self-improving 3D labels. These approaches often result in impressive 3D pose predictions, but struggle to accurately predict a diverse range of body shapes, since 2D keypoint supervision only provides a sparse shape signal. Shape prediction accuracy may be improved using synthetic training data consisting of synthetic input proxy representations (PRs) paired with ground-truth body shape and pose. PRs commonly consist of silhouettes and 2D joint heatmaps , necessitating accurate silhouette segmentations at test-time, which is not guaranteed for challenging in-the-wild inputs. Other methods pre-train on synthetic RGB inputs and then fine-tune on the scarce and limited-shape-diversity real 3D training data available , to avoid over-fitting to artefacts in low-fidelity synthetic data. In contrast, we utilise edge-based PRs, hence dropping the reliance on accurate segmentation networks without requiring fine-tuning on real data or high-fidelity synthetic data.

3D human shape and pose distribution estimation. Early optimisation-based 3D pose estimators specified a cost function corresponding to the posterior probability of 3D pose given 2D observations and analysed its multi-modal structure due to ill-posedness. Strategies to sample multiple 3D poses with high posterior probability included cost-covariance-scaled and inverse-kinematics-based global search and local refinement, as well as cost-function-modifying MCMC . Recently, several learning-based methods predict multi-modal distributions over 3D joint locations conditioned on 2D inputs, using Bayesian mixture of experts , mixture density networks or normalising flows . Our method extends beyond 3D joints and predicts distributions over human pose and shape. This has been addressed by Biggs et al. , who predict a categorical distribution over a set of SMPL parameter hypotheses. Sengupta et al. estimate an independent Gaussian distribution over both SMPL shape and joint rotation vectors. In contrast, we note that 3D rotations lie in $SO(3)$, motivating our hierarchical matrix-Fisher distribution.

Rotation distribution estimation via deep learning. Prokudin et al. use biternion networks to predict a mixture-of-von-Mises distribution over object pose angle. Gilitschenski et al. use a Bingham distribution over unit quaternions to represent orientation uncertainty. However, these works have to enforce constraints on the parameters of their predicted distributions (e.g. positive semi-definiteness). To overcome this, Mohlin et al. train a deep network to regress a matrix-Fisher distribution over 3D rotation matrices. We adapt this approach to define our hierarchical matrix-Fisher distribution over relative 3D joint rotation matrices.

Method

This section provides an overview of SMPL and the matrix-Fisher distribution , presents our structured, hierarchical pose and shape distribution estimation architecture and discusses the loss functions used to train it.

2 Matrix-Fisher distribution over S​O​(3)𝑆𝑂3SO(3)

The properties of $\mathcal{M}(\bm{R};\bm{F})$ can be described in terms of the singular value decomposition (SVD) of $\bm{F}$, denoted by $\bm{F}=\bm{U}^{\prime}\bm{S}^{\prime}\bm{V}^{\prime T}$, with $\bm{S}^{\prime}=\text{diag}(s^{\prime}_{1},s^{\prime}_{2},s^{\prime}_{3})$. $\bm{U}^{\prime}$ and $\bm{V}^{\prime}$ are orthonormal matrices, but they may have a determinant of -1 and thus are not necessarily elements of $SO(3)$. Therefore, a proper SVD $\bm{F}=\bm{U}\bm{S}\bm{V}^{T}$ is used, where

which ensures that $\bm{U},\bm{V}\in SO(3)$. Then, the mode of the distribution is given by

The columns of $\bm{U}$ define the distribution’s principal axes of rotation (analogous to the principal axes of a multivariate Gaussian distribution), while the proper singular values in $\bm{S}$ give the concentration of the distribution for rotations about the principal axes . Specifically, the concentration along rotations of $\bm{R}_{\text{mode}}$ about the $i$-th principal axis ($i$-th column of $\bm{U}$) is given by $s_{j}+s_{k}$ for $(i,j,k)\in\{(1,2,3),(2,3,1),(3,1,2)\}$. The concentration of the distribution may be different about each principal axis, allowing for axis-dependent rotation uncertainty modelling.

3 Proxy representation computation

4 Body shape and pose distribution prediction

Our goal is to predict a probability distribution over relative 3D joint rotations $\{\bm{R}_{i}\}_{i=1}^{23}$ and SMPL shape parameters $\bm{\beta}$ conditioned upon a given input proxy representation $\bm{X}$. We also predict deterministic estimates of the global body rotation $\bm{\gamma}$ and weak-perspective camera parameters $\mathbf{c}=[s,t_{x},t_{y}]$, representing scale and $xy$ translation.

Since $\bm{\beta}$ represents the linear coefficients of a PCA shape-space, a Gaussian distribution with a diagonal covariance matrix is suitable ,

where the mean $\bm{\mu}_{\beta}$ and variances $\bm{\sigma}^{2}_{\beta}$ are functions of $\bm{X}$.

The matrix-Fisher distribution (Equation 1) may be naively used to define a distribution over 3D joint rotations

for $i\in\{1,2,...,23\}$. Here, each joint is modelled independently of all the other joints. Thus, the matrix parameter of the $i$-th joint, $\bm{F}_{i}$, is a function of the input $\bm{X}$ only.

To predict the parameters of this naive, independent distribution over 3D joint rotations, in addition to the shape distribution parameters, global body rotation and weak-perspective camera, we learn a function $f_{\text{indep}}$ mapping the input $\bm{X}$ to the set of desired outputs $Y=\{\{\bm{F}_{i}\}_{i=1}^{23},\bm{\mu}_{\beta},\bm{\sigma}_{\beta}^{2},\bm{\gamma},\mathbf{c}\}$, where $f_{\text{indep}}$ is represented by a deep neural network with weights $\mathbf{W}_{\text{indep}}$.

However, the independent matrix-Fisher distribution in Equation 5 does not model SMPL 3D joint rotations faithfully, since the rotation of each part/bone is defined relative to its parent joint in the SMPL kinematic tree. Hence, a distribution over the $i$-th rotation matrix $\bm{R}_{i}$ conditioned on the input $\bm{X}$ should be informed by the distributions over all its parent joints $P(i)$, as well as the global body rotation $\bm{\gamma}$, to enable the distribution to match the 2D visual pose evidence present in $\bm{X}$. Furthermore, 3D joints in the SMPL rest-pose skeleton are dependent upon the shape parameters $\bm{\beta}$, while the mapping from 3D to the 2D image plane is given by the camera model. Hence, a distribution over $\bm{R}_{i}$ given $\bm{X}$ should also consider the predicted shape mean $\bm{\mu}_{\beta}$ and variance $\bm{\sigma}_{\beta}^{2}$, as well as the predicted camera $\mathbf{c}$. This is similar to the rationale behind the deterministic iterative/hierarchical predictors in , except we model these relationships in a probabilistic sense, by defining

for $i\in\{1,2,...,23\}$. Now, the matrix parameter of the $i$-th joint is a function of all its parent distributions, represented by the principal axes $\bm{U}_{j}$, singular values $\bm{S}_{j}$ and modes $\bm{R}_{\text{mode}_{j}}=\bm{U}_{j}\bm{V}_{j}^{T}$ for $j\in P(i)$, as well as the shape distribution $\{\bm{\mu}_{\beta},\bm{\sigma}_{\beta}^{2}\}$, global rotation $\bm{\gamma}$, camera parameters $\mathbf{c}$ and the input $\bm{X}$. Note that the parent distributions are themselves functions of their respective parent joints, while $\bm{\gamma},\bm{\mu}_{\beta},\bm{\sigma}_{\beta}^{2}$ and $\mathbf{c}$ are all functions of $\bm{X}$.

To predict the parameters of the hierarchical matrix-Fisher distribution in Equation 6, we propose a hierarchical neural network architecture $f_{\text{hier}}$, with weights $\mathbf{W}_{\text{hier}}$ (Figure 2). When considered as a black-box, $f_{\text{hier}}$ yields the same set of outputs $Y$ as $f_{\text{indep}}$. However, $f_{\text{hier}}$ utilises the iterative hierarchical architecture presented in Figure 2, which amounts to multiple streams of fully-connected layers, each following one “limb” of the kinematic tree. In contrast, $f_{\text{indep}}$ predicts pose similarly to shape, camera and global rotation parameters, using a single stream of fully-connected layers. We compare the naive independent formulation with the hierarchical formulation in Section 5.1.

5 Loss functions

Distribution prediction networks are trained with a synthetic dataset $\{\bm{X}^{n},(\{\bm{R}_{i}^{n}\}_{i=1}^{23},\bm{\beta}^{n},\bm{\gamma}^{n})\}_{n=1}^{N}$ (Section 4).

Negative log-likelihood (NLL) loss on distribution parameters. The NLL corresponding to the Gaussian body shape distribution (Equation 4) is given by:

The NLL corresponding to the matrix-Fisher distribution over relative 3D joint rotations is defined as :

for $i\in\{1,2,...,23\}$, where $\bm{F}^{n}_{i}$ may be obtained via the independent or hierarchical matrix-Fisher models presented above. Intuitively, the trace term pushes the predicted distribution mode $\bm{R}^{n}_{\text{mode}_{i}}$ (Equation 3) towards the target $\bm{R}^{n}_{i}$, while the log normalising constant acts as a regulariser, preventing the singular values of $\bm{F}^{n}_{i}$ from getting too large . All predicted distribution parameters are dependent on the model weights, $\mathbf{W}_{\text{indep}}$ or $\mathbf{W}_{\text{hier}}$, which are learnt in a maximum likelihood framework aiming to minimise the joint shape and pose NLL: $\mathcal{L}_{\text{NLL}}=\mathcal{L}_{\beta\text{-NLL}}+\mathcal{L}_{R\text{-NLL}}$.

Loss on global body rotation. We predict deterministic estimates of the global body rotation vectors $\hat{\bm{\gamma}}^{n}$, which are supervised using ground-truth global rotations ${\bm{\gamma}}^{n}$, with loss $\mathcal{L}_{\text{global}}=\sum_{n=1}^{N}\|\bm{R}(\bm{\gamma}_{n})-\bm{R}(\bm{\hat{\gamma}}_{n})\|_{F}^{2}$. $\bm{R}(\bm{\gamma})\in SO(3)$ is the rotation matrix corresponding to $\bm{\gamma}$.

2D joints loss on samples. Applying $\mathcal{L}_{\text{NLL}}$ alone results in overly uncertain predicted 3D shape and pose distributions (see Section 5.1). To ensure that the predicted distributions match the visual evidence in the input $\bm{X}^{n}$, we impose a reprojection loss between ground-truth 2D joint coordinates (in the image plane) and predicted 2D joint samples, which are obtained by differentiably sampling 3D bodies from the predicted distributions and projecting to 2D using the predicted camera $\mathbf{c}^{n}=[s^{n},t_{x}^{n},t_{y}^{n}]$. Ground-truth 2D joints $\bm{J}^{n}_{\text{2D}}$ are computed from $\{\{\bm{R}^{n}_{i}\}_{i=1}^{23},\bm{\beta}^{n},\bm{\gamma}^{n}\}$ during synthetic training data generation (see Section 4).

We adapt the rejection sampler presented in to sample from a matrix-Fisher distribution $\mathcal{M}(\bm{R};\bm{F})$, modifying it to allow for backpropagation of gradients through the proposal sampling step (lines 5-7 in Algorithm 1). We refer the reader to for further details about the rejection sampler. In short, to simulate a matrix-Fisher distribution with parameter $\bm{F}=\bm{U}\bm{S}\bm{V}^{T},$ we sample unit quaternions from a Bingham distribution over the unit 3-sphere $S^{3}$, with Bingham parameter $\bm{A}$ computed from $\bm{S}$, and then convert the sampled quaternions into rotation matrices with the desired matrix-Fisher distribution. Rejection sampling is used to sample from the Bingham distribution, which has pdf $p_{\text{Bing}}(\bm{x})\propto\exp(-\bm{x}^{T}\bm{A}\bm{x})$ for $\bm{x}\in S^{3}$. The proposal distribution for the rejection sampler is an angular central Gaussian (ACG) distribution, with pdf $p_{\text{ACG}}(\bm{x})\propto(\bm{x}^{T}\bm{\Omega}\bm{x})^{-2}$. The ACG distribution is easily simulated by sampling from a zero-mean Gaussian distribution with covariance matrix $\bm{\Omega}^{-1}$ and normalising to unit-length (lines 5-7 in Algorithm 1). The re-parameterisation trick is used to differentiably sample from this zero-mean Gaussian, thus allowing for backpropagation of gradients through the rejection sampler.

Algorithm 1 samples $K$ sets of relative 3D joint rotation matrices $\{\{\hat{\bm{R}}^{n}_{i,k}\}_{i=1}^{23}\}_{k=1}^{K}$ from the corresponding distributions $\{\mathcal{M}(\bm{R}^{n}_{i};\bm{F}^{n}_{i})\}_{i=1}^{23}$. Furthermore, we differentiably sample $K$ SMPL shape vectors from the predicted Gaussian distribution $\{\hat{\bm{\beta}}^{n}_{k}\sim\mathcal{N}(\bm{\beta};\bm{\mu}_{\beta}(\bm{X}^{n}),\text{diag}(\bm{\sigma}^{2}_{\beta}(\bm{X}^{n})))\}_{k=1}^{K}$, again using the re-parameterisation trick .

The body shape and 3D joint rotation samples are converted into 2D joint samples using the SMPL model and weak-perspective camera parameters

where $\Pi()$ is an orthographic projection. The reprojection loss applied between the predicted 2D joint samples and the visible target 2D joint coordinates is given by

where the visibilities of the target joints are denoted by $\bm{\omega}^{n}\in\{0,1\}^{L}$ (1 if visible, 0 otherwise).

Implementation Details

Synthetic training data. To train our 3D body shape and pose distribution prediction networks, we require a training dataset $\{\bm{X}^{n},(\{\bm{R}_{i}^{n}\}_{i=1}^{23},\bm{\beta}^{n},\bm{\gamma}^{n})\}_{n=1}^{N}$. We extend the synthetic training frameworks presented in , which involve generating inputs and corresponding SMPL body shape and pose (i.e. 3D joint rotation) labels randomly and on-the-fly during training. In brief, for every training iteration, SMPL shapes $\bm{\beta}^{n}$ are randomly sampled from a prior Gaussian distribution while relative 3D joint rotations $\{\bm{R}_{i}^{n}\}_{i=1}^{23}$ and global rotation $\bm{\gamma}^{n}$ are chosen from the training sets of UP-3D , 3DPW or Human3.6M . These are converted into training inputs $\bm{X}^{n}$ and ground-truth 2D joint coordinates $\bm{J}^{n}$ using the SMPL model and a light-weight renderer . Cropping, occlusion and noise augmentations are then applied to the synthetic inputs.

Previous synthetic training frameworks often use silhouette-based training inputs. This necessitates accurate human silhouette segmentation at test-time, which may be challenging to do robustly. In contrast, our input representations consist of edge-images concatenated with 2D joint heatmaps. To generate edge-images, we first create synthetic RGB images by rendering textured SMPL meshes. For each training mesh, clothing textures are randomly chosen from . The textured SMPL mesh is rendered onto a background image (randomly chosen from LSUN ), using randomly-sampled lighting and camera parameters. Canny edge detection is used to compute edge-images from the synthetic RGB images. We show in Section 5.1 that, despite the lack of photorealism in the synthetic RGB images, edge-filtering bridges the synthetic-to-real domain gap at test-time - and performs better than either silhouette-based or synthetic-RGB-based training inputs in our experiments. Examples of synthetic training samples are given in the supplementary material.

Training details. We use Adam with a learning rate of 0.0001, batch size of 80 and train for 150 epochs. For stability, the 2D joints reprojection loss is only applied on the mode pose and shape (projected to 2D) in the first 50 epochs and not on the samples, which are supervised in the next 100 epochs. To boost 3D pose metrics, an MSE loss on the mode 3D joint locations is applied in the final 50 epochs.

Evaluation datasets. 3DPW is used to evaluate 3D pose prediction accuracy. We report mean-per-joint-position-error after scale correction (MPJPE-SC) and after Procrustes analysis (MPJPE-PA), both in mm. Both metrics are computed using the mode 3D joint coordinates of the predicted shape and pose distributions.

SSP-3D is primarily used to evaluate 3D body shape prediction accuracy, using per-vertex Euclidean error in a T-pose after scale-correction (PVE-T-SC) in mm, computed with the mode 3D body shape from the predicted shape distribution. We also evaluate 2D joint prediction error (2D Joint Err. Mode/Samples) in pixels, computed using both the mode 3D body and 10 3D bodies randomly sampled from the predicted shape and pose distributions, projected onto the image plane using the camera prediction. 2D joint error is evaluated on visible target 2D joints only.

Finally, we use a synthetic test dataset for our ablation studies investigating different input representations. It consists of 1000 synthetic input-label pairs, generated in the same way as the synthetic training data, with poses sampled from the test set of Human3.6M. .

Experimental Results

This section investigates different input representations and the benefits of the 2D joints samples loss, compares independent and hierarchical distribution predictors and benchmarks our method against the state-of-the-art.

Input proxy representation. Rows 1-3 in Table 1 compare different choices of input proxy representation: binary silhouettes, RGB images and edge-filtered images (each additionally concatenated with 2D joint heatmaps). The independent network architecture is used for all three input types. To investigate the synthetic-to-real domain gap, metrics are presented for synthetic test data, as well as real test images from SSP-3D and 3DPW. For the latter, silhouette segmentation is carried out with DensePose . Using RGB-based input representations (row 2) results in the best 3D shape and pose metrics on synthetic data, which is reasonable since RGB contains more information than both silhouettes and edge-filtered images. However, metrics are significantly worse on real datasets, suggesting that the network has over-fitted to unrealistic artefacts present in low-fidelity (i.e. computationally cheap) synthetic RGB images. Silhouette-based input representations (row 1) also demonstrate a deterioration of 3D metrics on real test data compared to synthetic data, since they are heavily reliant upon accurate silhouettes, which are difficult to robustly segment in test images containing challenging poses or severe occlusions. Inaccurate silhouette segmentations critically impair the network’s ability to predict 3D body pose and shape. In contrast, edge-filtering is a simpler and more robust operation than segmentation, but is still able to retain important shape information from the RGB image. Thus, edge-images (concatenated with 2D joint heatmaps) can better bridge the synthetic-to-real domain gap, resulting in improved metrics on real test inputs (row 3).

Hierarchical architecture and reprojection loss on 2D joints samples. Figure 3 and rows 3-6 in Table 1 compare the independent and hierarchical distribution prediction architectures ($f_{\text{indep}}$ and $f_{\text{hier}}$) presented in Section 3.4, both with and without the reprojection loss on sampled 2D joints ($\mathcal{L}_{\text{2D Samples}}$) from Section 3.5. When $\mathcal{L}_{\text{2D Samples}}$ is not applied, the shape and pose distributions predicted by both the independent and hierarchical network architectures do not consistently match the the input image, as evidenced by the significant gap between the visible 2D joint error computed using the distributions’ modes versus samples drawn from the distributions (in rows 3 and 5 of Table 1) on both synthetic test data and SSP-3D . This implies that the predicted distributions are overly uncertain about parts of the subject’s body that are visible and unambiguous in the input image. The visualisations corresponding to the hierarchical architecture trained without $\mathcal{L}_{\text{2D Samples}}$ in Figure 3 (centre) further demonstrate that the predicted samples often do not match the input image, particularly at the extreme ends of the body. This results in significant undesirable per-vertex uncertainty over unambiguous body parts.

Applying $\mathcal{L}_{\text{2D Samples}}$ to the independent network $f_{\text{indep}}$ partially alleviates the mismatch between inputs and predicted samples, as shown by Figure 3 (right) and row 4 in Table 1, where the mode versus sample 2D joint error gap has reduced. However, training with $\mathcal{L}_{\text{2D Samples}}$ deteriorates the independent architecture’s mode pose prediction metrics (MPJPE-SC and 2D Joint Err. Mode in row 3 vs 4 of Table 1) on both synthetic and real test data. This is because $f_{\text{indep}}$ naively models each joint’s relative rotation independently of its parents’ rotations (Equation 5); however, to predict realistic human pose samples that match the visible input, each joint’s rotation distribution must be informed by its parents. $\mathcal{L}_{\text{2D Samples}}$ attempts to force predicted samples to match the input despite this logical inconsistency, which causes a trade-off between mode and sample pose prediction metrics, particularly worsening MPJPE-SC.

In contrast, applying $\mathcal{L}_{\text{2D Samples}}$ to the hierarchical network $f_{\text{hier}}$ improves metrics corresponding to both mode and sample predictions, as shown by row 6 in Table 1. Now, each SMPL joint’s relative rotation distribution is conditioned on all its parents’ distributions (Equation 6). Thus, $\mathcal{L}_{\text{2D Samples}}$ and $\mathcal{L}_{\text{NLL}}$ work in conjunction in enabling predicted hierarchical distributions (and samples) to match the visible input, while yielding improved 3D metrics. Figure 3 (left) exhibits such visually-consistent samples and demonstrates greater prediction uncertainty for ambiguous parts. Note that uncertainty can arise even without occlusion in a monocular setting, e.g. due to depth ambiguities as shown by the left arm samples in the last row of Figure 3. Further visual results are in the supplementary material.

2 Comparison with the state-of-the-art

Shape prediction. Table 3 evaluates 3D body shape metrics on SSP-3D for single image inputs and multi-image input sets, which we evaluate using both mean and probabilistic combination methods from . Our network surpasses the state-of-the-art , mainly due to our use of an edge-based proxy representation, instead of the silhouette-based representations used in and . These methods rely on accurate human silhouettes, which may be difficult to compute at test-time, as discussed in Section 5.1, while our method does not have such dependencies. However, our method may result in erroneous shape predictions when the subject is wearing loose clothing which obscures body shape, in which case the shape prediction over-estimates the subject’s true proportions (see rows 1-2 in Figure 3).

Pose prediction. Table 2 evaluates 3D pose metrics on 3DPW . Our method is competitive with the state-of-the-art and surpasses other methods that do not require 3D-labelled training images . Figure 4(a) shows that our method performs well for most test examples in 3DPW, even matching pose-focused approaches that do not attempt to accurately predict diverse body shapes . However, some images in 3DPW contain significant occlusion, which can lead to noisy 2D joint heatmaps in the proxy representations, resulting in poor 3D pose metrics as shown by the right end of the curve in Figure 4(a).

Further quantitative comparison with other shape and pose distribution/multi-hypothesis prediction approaches is given in the supplementary material.

Conclusion

In this paper, we have proposed a probabilistic approach to the ill-posed problem of monocular 3D human shape and pose estimation, motivated by the fact that multiple 3D bodies may explain a given 2D image. Our method predicts a novel hierarchical matrix-Fisher distribution over relative 3D joint rotations and a Gaussian distribution over SMPL shape parameters, from which we can sample any number of plausible 3D reconstructions. To ensure that the predicted distributions match the input image, we have implemented a differentiable rejection sampler to impose a loss between predicted 2D joint samples and ground-truth 2D joint coordinates. Our method is competitive with the state-of-the-art in terms of pose metrics on 3DPW, while surpassing the state-of-the-art for shape accuracy on SSP-3D.

Acknowledgements. We thank Dr. Yu Chen (Metail), Mr. Jim Downing (Metail), Dr. David Bruner (SizeStream) and Dr. Delman Lee (TAL Apparel) for providing body shape evaluation data and supporting this research.

Supplementary Material: Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild

Section 7 in this supplementary material contains implementation details, particularly regarding synthetic training data generation and per-vertex uncertainty visualisation. Section 8 discusses qualitative results on the SSP-3D and 3DPW datasets, and compares distribution predictions on images with versus without artificial occlusions. Table 5 compares several recent multi-hypothesis 3D human shape and pose estimation approaches.

Implementation Details

Our shape and pose distribution prediction neural networks are trained using synthetic training data, consisting of edge-and-joint-heatmap inputs paired with ground truth SMPL shape and pose parameters. Inputs are rendered on-the-fly during model training using randomly sampled camera extrinsics, lighting, backgrounds and clothing textures. Examples of synthetic training and validation data are given in Figure 5. Note how each body pose may be paired with a different body shape, clothing, camera and background, as well as occlusion and noise augmentations. Thus, we are able to render highly diverse training data on-the-fly during training, enabling the network to see a new pose/shape/clothing/camera/background combination in each training iteration.

Our synthetic RGB images (Figure 5) are computationally cheap but clearly far from photorealistic, resulting in a large synthetic-to-real domain gap. However, simple edge detection is able to significantly reduce this gap , motivating the use of edge-filtered images as part of our input proxy representation. We found that noisy edge detections (as seen in Figure 5) retained sufficient visual shape and pose information, and efforts to produce clean edge-images (e.g. hysteresis-based edge tracking or further hyperparameter tuning) did not improve performance.

The required body shape, pose, clothing and backgrounds are obtained as follows. For training, ground-truth SMPL 3D joint rotation matrices are sampled from the training splits of 3DPW and UP-3D , as well as Human3.6M subjects 1, 5, 6, 7 and 8, giving a total of 91106 training poses. Validation poses are sampled from the 3DPW/UP-3D validation splits and Human3.6M subjects 9 and 11, resulting in 33347 validation poses. SMPL body shape parameters are randomly sampled from $\mathcal{N}(\beta_{i};0,1.25^{2})$ for $i=1,...,10$ . RGB clothing textures for the SMPL body mesh are selected from SURREAL and MultiGarmentNet , resulting in 917 training textures and 108 validation textures. Backgrounds are obtained from LSUN , which contains a collection of diverse indoor and outdoor scenes. We sample from 397582 different training backgrounds and 3000 different validation backgrounds. Note that background training images may contain other humans, which is intentional and essential for robustness against test images with multiple people. The network learns to focus on the person corresponding to the input joint heatmaps and ignore persons in the background.

Textured SMPL meshes are rendered with Pytorch3D , using a perspective camera model and Phong shading. Camera and lighting parameters are randomly sampled, with sampling hyperparameters given in Table 4. Generated images are cropped around the rendered body using a square bounding box, where the bounding box size is randomly scaled by a factor in range (0.8, 1.2).

To further bridge the gap synthetic-to-real gap, we implement random occlusion, body part removal, 2D joint removal and 2D joint noise augmentations during training. Hyperparameters associated with data augmentations are given in Table 6.

2 Visualisation of Per-Vertex Uncertainty

Figures 6, 7 and 8 in this supplementary material, as well as several figures in the main manuscript, visualise per-vertex 3D location uncertainties corresponding to the predicted shape and 3D joint rotation distributions. These are computed by i) sampling 100 shape parameter vectors and relative 3D joint rotations (for the entire kinematic tree) from the predicted distributions, ii) passing each of these samples through the SMPL function to get the corresponding vertex meshes, iii) computing the mean location of each vertex over all the samples and iv) determining the average Euclidean distance from the sample mean for each vertex over all the samples, which is ultimately visualised in the vertex scatter plots as a measure of per-vertex 3D location uncertainty.

Qualitative Results

Figure 7 presents results on artificially occluded images from SSP-3D . In particular, note that i) occluded/invisible body parts result in increased 3D location uncertainty for corresponding vertices and ii) 3D body samples from the predicted distributions match the visible body parts in the 2D image, while invisible body part samples are more diverse. However, occluded sample diversity is still somewhat limited and samples tend to be clustered around the mode predictions, which is a weakness of our method. This may be alleviated by predicting multi-modal distributions over 3D shape and pose in future work. Figure 7 also illustrates our method’s ability to predict a range of body shapes, owing to the synthetic training framework used.

Figure 6 presents results on the test split of 3DPW . Again, note the increased uncertainty and sample diversity for occluded and out-of-frame body parts, and the reprojection consistency between predicted samples and the visible bodies in the images. Results on 3DPW highlight another key challenge for future work: when faced with baggy/loose clothing, our method tends to over-estimate the subject’s body proportions. This is because our synthetic training data does not model the shape of clothing on the human body surface, but only its texture. Future work could focus on using synthetic clothed humans for training.

Figure 8 compares shape and pose distribution predictions on images from SSP-3D with versus without artificial occlusions, further corroborating that ambiguous parts result in greater uncertainty and more diverse 3D samples. However, it is again apparent that sample diversity for highly ambiguous parts is more limited than expected, as samples tend to be closely clustered around the mode prediction.

Note that uncertainty does not only arise from occlusion - depth ambiguities are prevalent when estimating 3D pose from a monocular 2D image . This is demonstrated in the non-occluded images in Figure 8 (left), by the left arm samples in rows 1 and 5 and the right arm in row 4.

References