Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network

Introduction

3D human pose estimation from a single RGB image is an extensively studied problem in computer vision because of many potential useful real world applications such as forensic science, sports analysis and surveillance etc. Significant progress in 3D human pose estimation has been made with deep learning in the recent years. One of the commonly used and effective deep learning based methods for 3D human pose estimation is the two-stage approach, where the 2D joints are first detected from the image input followed by the 3D joint estimations from the detected 2D joints . The advantage of the two-stage approach is that it decouples the harder problem of 3D depth estimation from the easier 2D pose estimation. In particular, variations in background scene, lighting, clothing shape, skin color etc. are removed before the 3D joint estimation stage. Furthermore, the model can be trained on different domains, e.g. indoor and outdoor, with 2D annotations that are readily available.

Despite the significant progress with deep learning, 3D human pose estimation remains as a very challenging task due to the ambiguity in recovering 3D information from a single RGB image. More specifically, recovering 3D information from a single RGB image or 2D joint locations is an inverse problem where multiple solutions may exist for the depth of a 3D joint along the light ray that reprojects onto the same 2D joint location, as illustrated in Figure 1. The problem is further aggravated by the non-rigidity of the human pose and joint occlusions on the 2D image. Consequently, there could be many solutions of the 3D pose that satisfy the same 2D pose on an image, even after eliminating the infeasible 3D pose solutions by enforcing various geometric constraints, e.g. joint limits and bone ratio etc. In view of the inherent ambiguity of the 3D human pose estimation problem, we argue that it is more reasonable to design a model that generates multiple hypotheses of geometrically feasible 3D human pose that are consistent with the detected 2D joints from a single RGB image. In contrast, the widely adopted single estimate for the inverse problem with inherent ambiguity could lead to overfitting the model to the training data, and might not generalize well. This idea of generating multiple 3D pose hypotheses was first suggested very recently by Jahangiri and Yuille in .

To this end, we introduce the mixture density networks (MDN) to the 3D joint estimation module of the two-stage approach. Contrary to most existing works that generate a single 3D pose by minimizing the negative log-likelihood of an unimodal Gaussian, i.e. a mean squared error, we propose to estimate multiple hypotheses of the 3D pose by minimizing the negative log-likelihood of a multimodal mixture-of-Gaussians. The outputs of our mixture model is a set of mixing coefficients and parameters of the Gaussian kernels, i.e. means and variances. The set of 3D pose hypotheses are given by the means of the Gaussian kernels, and the mixing coefficient and variances represent the uncertainties of each 3D pose hypothesis. Specifically our network consists of a feature extractor to lift the 2D joints into a feature space, and a hypotheses generator to generate multiple hypotheses. The whole network is a simple network made up of several linear layers with different non-linear activation units.

We show that our network achieves state-of-the-art results on the Human3.6M dataset in both best hypothesis and multi-view settings. We also report results of our network on the outdoor MPII dataset and the MPI-INF-3DHP dataset, where 3D pose labels are not used for training the network. Furthermore, we show the robustness of our network by applying it to scenarios where one or two limb joints are occluded/missing. Our main contributions are as follows:

We explore the idea of generating multiple 3D pose hypotheses to alleviate the ambiguity problem that has not received much attention in the literature.

To the best of our knowledge, we are the first to introduce the mixture density model into 3D human pose estimation, which is more powerful than the single Gaussian distribution.

Our network achieves state-of-the art results on Human3.6M dataset in both best hypothesis and multi-view settings, and in cases where one or two limb joints are occluded/missing.

Related Work

Existing human 3D pose estimation approaches fall into two categories according to their training techniques. The first category is to train deep convolutional neural networks (CNNs) end-to-end to estimate 3D human poses directly from the input images . Zhou et al. use a sparse representation for 3D poses and predict the 3D pose with an expectation-maximization (EM) algorithm. The 2D poses are regarded as a hidden variable in the EM algorithm to remove the need of synchronized 2D-3D data. Park et al. improve conventional CNNs by concatenating 2D pose estimation as well as information on relative positions with respect to multiple joints. Pavlakos et al. use volumetric representation to represent 3D poses and adopt the stacked hourglass network , which is originally designed for 2D pose estimation, to predict 3D volumetric heatmaps. Mehta et al. use transfer learning to transfer the knowledge learned for 2D pose estimation to the task of 3D pose estimation. Similarly, Zhou et al. propose a weakly-supervised transfer learning method that uses mixed 2D and 3D labels. The 2D pose estimation sub-network and 3D depth regression sub-network share the same features such that the 3D pose labels for indoor environments can be transferred to in-the-wild images. The direct approach benefits from the rich information contained in images, e.g. the front-back orientation of limbs. However, it will also be affected by a number of factors such as background, lighting, clothing etc. A network trained on one dataset can not be generalized well to the other datasets with different environment, for example from indoor and outdoor environment.

The second category decouples 3D pose estimation into the well-studied 2D joint detection and 3D pose estimation from the detected 2D joints. Akhter et al. propose a multi-stage approach to estimate the 3D pose from 2D joints using an over-complete dictionary of poses. Bogo et al. estimate 3D pose by first fitting a statistical body shape model to the 2D joints, and then minimizing the error between the reprojected 3D model and detected 2D joints. Chen and Yasin regard 3D pose estimation as a matching between the estimated 2D pose and the 3D pose from a large pose library. Martinez et al. design a simple fully connected residual network to regress 3D pose from 2D joint detections. The decoupled approach can make use of both indoor and in-the-wild images to train the 2D pose estimators. More importantly, this approach is domain invariant since the input of the second stage is the 2D joints. However, estimating 3D pose from 2D joints is more challenging because 2D pose data contains less information than images, thus there are more ambiguities.

To solve the ill-posed problem of estimating 3D pose from 2D joints, Jahangiri and Yuille first proposed to generate multiple diverse pose hypotheses. They first learned a 3D Gaussian mixture model (GMM) model from a uniformly sampled set of Human3.6M poses, and then use conditional sampling to get samples of the 3D poses with reprojected joints errors that are within a threshold. Inspired by their work, we solve the ambiguity problem by generating multiple hypotheses. Instead of using the traditional GMM approach, we introduce the MDN which was first proposed in . The MDN can represent arbitrary conditional distributions by combining a conventional neural network with a mixture density model. Ye et al. used a hierarchical MDN to solve the occlusion problem in hand pose estimation. Inspired by the work of Ye et al., we use the MDN to solve the depth ambiguity and occlusion problem in 3D human pose estimation.

Our Mixture Density Network

where $M$ is the number of Gaussian kernels, i.e. the number of hypotheses. $\alpha_{i}(\textbf{x})$ is the mixing coefficients, which can be regarded as a prior probability of a 3D pose data y being generated from the $i^{th}$ Gaussian kernel given the input 2D joints x. Here $\alpha_{i}(\textbf{x})$ must satisfy the constraint

$\phi_{i}(\textbf{y}\mid\textbf{x})$ is the conditional density of the 3D pose y for the $i^{th}$ kernel, which can be expressed as a Gaussian distribution

$\mu_{i}(\textbf{x})$ and $\sigma_{i}(\textbf{x})$ denote the mean and variance of the $i^{th}$ kernel, respectively. $d$ is the dimension of the output 3D pose y. All the parameters of the mixture model, including the mixing coefficients $\alpha_{i}(\textbf{x})$, the mean $\mu_{i}(\textbf{x})$ and the variance $\sigma_{i}(\textbf{x})$ are functions of the input 2D pose x.

Note that the mixture model degenerates to a single Gaussion distribution when the means and variances of all Gaussian kernels are similar, i.e. $\mu_{i}(\textbf{x})\approx\mu(\textbf{x})$, $\sigma_{i}(\textbf{x})\approx\sigma(\textbf{x})$ for $i=1,...,M$. Hence,

Specifically in our case, the 3D pose hypotheses generated by the MDN will collapse into approximately a single Gaussian when the given 2D pose is simple and less ambiguous, e.g. no occlusions and/or missing joints.

2 Network Architecture

From Eqn. (1), (2) and (3), we can see that all parameters $\Theta(\textbf{x})=\{\bm{\mu}(\textbf{x}),\bm{\sigma}(\textbf{x}),\bm{\alpha}(\textbf{x})\}$ of the Gaussian mixture distribution of y are functional form of x. Hence, we learn this function $f:\textbf{x}\mapsto\Theta$ using a deep network which can be expressed as

where w is the set of learnable weights in the deep network. The probability density in Eqn. (1) can be rewritten to include the learnable weights w of the deep network, i.e.,

The parameters $\Theta(\textbf{x},\textbf{w})=\{\bm{\mu}(\textbf{x},\textbf{w}),\bm{\sigma}(\textbf{x},\textbf{w}),\bm{\alpha}(\textbf{x},\textbf{w})\}$ are now dependent on the learnable weights w of the deep network $f(\textbf{x};\textbf{w})$.

for the variance $\bm{\sigma}(\textbf{x},\textbf{w})$ to keep it positive. Here, $\gamma$ is a scale for negative factor.

3 Optimization

Given a training dataset with $K$ pairs of ground truth labels for the corresponding 2D joints X and 3D poses Y, i.e. $\{\textbf{X},\textbf{Y}\}=\{\{\textbf{x}_{j},\textbf{y}_{j}\}\mid j=1,...,K\}$, the objective is to find the maximum a posterior of the set of learnable weights w. More formally, assuming that each training data is independent and identically distributed (i.i.d), the posterior distribution of w is given by

where $\Psi$ is the hyperparameter of the prior over the learnable weights w. Hence, the optimal weight $\textbf{w}^{*}$ can be obtained from the minimization of the negative log-posterior

where $\mathcal{L}$ is taken to be the loss function for training our deep network $f(\textbf{x};\textbf{w})$. More specifically,

The prior loss $\mathcal{L}_{\text{prior}}$ can be further evaluated into:

where the term $\ln{p(\textbf{X}\mid\Psi)}$ can be dropped in the loss function since it is independent of w, and we write the random variables $\{\textbf{w},\textbf{X}\}$ in its functional form $\Theta(\textbf{w},\textbf{X})$ given by the deep network. We further assume a uniform prior over $\bm{\mu}(\textbf{w},\textbf{X})$ and $\bm{\sigma}(\textbf{w},\textbf{X})$, and a Dirichlet conjugate prior over the mixing coefficients $\bm{\alpha}(\textbf{w},\textbf{X})$ that follows a Categorical distribution, we get

$\Gamma{[.]}$ is the Gamma function, and $\Lambda=\{\lambda_{1},...,\lambda_{M}\}$ are the hyperparameters of the Dirichlet distribution, where $\lambda_{i}>0$ for $1\leq i\leq M$. The total loss function to train our deep network is given by

Note that we drop $\frac{\Gamma{[\sum_{i=1}^{M}\lambda_{i}]}}{\prod_{i=1}^{M}\Gamma{[\lambda_{i}]}}$ in $\mathcal{L}_{\text{prior}}$ because it is independent of w.

The term $\mathcal{L}_{\text{prior}}$ regularizes the mixing coefficients of our mixture model. Setting $\lambda_{i}=1$ for $i=1,..,M$ implies that we have no prior knowledge over the mixing coefficients. In our experiments, we set $\lambda_{1}=...=\lambda_{M}=C$, where $C>1$ is a constant scalar value to prevent overfitting of a single Gaussian kernel in the MDN to the training data, i.e. a single mixing coefficient $\alpha_{i}\approx 1$ and $\alpha_{j\neq i}\approx 0$.

Experiments

We show numerical results for the Human3.6M dataset and compare with other state-of-the-art approaches. We also apply our approach to other datasets including MPII and MPI-INF-3DHP datasets to test the generalization capacity of our network.

This is currently the largest available video pose dataset, which provides accurate 3D body joint locations recorded by a Vicon motion capture system. There are 15 activity scenarios in total such as “Walking”, “Eating”, “Sitting” and “Discussion”, each action is performed by 7 professional actors. Accurate 2D joint locations , 3D pose annotations and camera parameters are provided. Following , we apply standard normalization to the 2D inputs and 3D outputs by subtracting the mean and dividing by the standard deviation of the training data. We also zero-center the 3D poses around the hip joint since we do not predict the global position of the 3D pose.

MPII dataset:

This is a standard dataset for 2D human pose estimation, which contains 25K unconstrained images collected from YouTube videos. This is the most challenging in-the-wild dataset and we use it to test the generalization of our approach. We report qualitative result for this datset because 3D pose information is not provided.

MPI-INF-3DHP dataset:

It is a newly released 3D human pose dataset which is captured by a Mocap system in both indoor and outdoor scenes. We only use the test split of this dataset that includes 2935 frames from six subjects performing seven actions.

D detections:

We use the state-of-the-art stacked hourglass network to get the 2D joint detections. The stacked hourglass network is pretrained on the MPII dataset and then fine-tuned on the Human3.6M dataset.

Evaluation protocols:

For the Human3.6M dataset, we follow the standard protocol of using S1, S5, S6, S7 and S8 for training, and S9 and S11 for testing. The evaluation metric is the Mean Per Joint Position Error (MPJPE) in millimeter between the ground truth and the estimated 3D pose. Since our network generating multiple hypotheses for each 2D detection, we follow to compute the MPJPE between the ground truth and the best 3D hypothesis generated by our network. The 3D Percentate of Correct Keypoints (3DPCK) is adopted as the metric for the MPI-INF-3DHP dataset .

2 Results on Human3.6M dataset

We first report our results on the Human3.6 dataset and compare with other state-of-the-art approaches. From the results shown in Table 1, we can see that our method outperforms the others in most cases. Our approach achieves an improvement of 5.5% compared to the previous best result 55.8 mm and 16.2 % compared to the baseline architecture . This indicates the efficiency of our approach by generating multiple hypotheses. Moreover, our network outperforms which also generates multiple hypotheses by 22.5%. Following previous work, We show our result under Protocol #2 where the estimated pose has been further aligned with the ground truth via a rigid transfermation. The MPJPE error in Table 1 shows that our approach consistently outperforms other approaches.

It is difficult to disambiguate the multiple 3D pose hypotheses generated by our model in a monocular view because most of them are feasible solutions to the inverse 2D-to-3D problem. Hence, we utilize the multi-view images from the set of calibrated cameras provided by the Human3.6M dataset to disambiguate and verify the correctness of the multiple 3D pose hypotheses generated by our network. Specifically, we transform the same pose under different cameras into the global world coordinates, and then we choose the pose which is most consistent with the poses from other camera coordinates. Finally, we get our estimated pose by averaging all poses from different camera coordinates. We list our result in Table 2 and compare with other state-of-the-art approaches based on multi-view (spatial constraint) or video (temporal constraint) information . Note that it is however not a fair comparison with other results listed in Table 1 because they did not use any multi-view or video information. The results show that our approach has the best performance among both spatial and temporal constraints based methods, indicating the advantage of our approach by generating multiple hypotheses.

In realistic scenarios, it is common that some joints are occluded and cannot be detected. In order to show that our model can handle with missing joints, we ran experiments with different number of missing joints selected randomly from the limb joints including l/r elbow, l/r wrist, l/r knee, l/r ankle. We show our results in Table 3 and compared with the baseline 2D-to-3D estimator and the GMM based methods which also focus on generating multiple hypotheses. The baseline outperforms GMM based methods by a large margin, which indicates the advantage of using deep networks. Moreover, our method improves the baseline for all actions with average error decreased by 10.4mm for both cases, further showing the robustness of our method.

3 Transfer to MPII and MPI-INF-3DHP datasets

We test our method on the MPII and MPI-INF-3DHP datasets to validate the generalization capacity. Note that we train the feature extractor and hypotheses generator on the Human3.6 dataset which contains data from only the indoor environment. The validation set of MPI-INF-3DHP dataset includes images recorded under three different scenes: 1143 images in studio with green screen background (Studio GS), 1064 images in studio without green screen background (Studio no GS) and 728 images in outdoor environment (Outdoor). We use the 2D joints provided by the dataset as input and compute the 3DPCK. The results in Table 4 show that the 3DPCK of our approach is slightly lower than even though we did not train on their dataset, indicating the generalization of our network. Moreover, our results for different scenes do not change too much compared to the results of . This further suggests the domain-invariant capability of the two-stage approach that we adopted. We only give qualitative results for the MPII dataset in Figure 3 since the ground truth 3D pose data is not provided. We can see that our network can be generalized well to outdoor unseen scenes.

4 Ablation Study

Our hypotheses generator is based on MDN where each of the $M$ Gaussian kernels in Eqn.(1) yields different result. We note that our network cannot fit the data completely if $M$ is too small, while larger $M$ requires more computation resource. We thus train three different models with $M$ setting to 3, 5, 8, respectively. We show the average MPJPE on the Human3.6M dataset in Table 5 and compare them with the baseline method, which is based on single Gaussian distribution. The results suggest that our MDN has better performance than single Gaussion based method. Moreover, the performance does not improve much when $M$ is larger than five. Consequently, we set $M$ to five in our experiments in view of the trade-off between accuracy and computational complexity.

Dirichlet prior

We add a Dirichlet conjugate prior to the distribution of the mixture coefficients $\bm{\alpha}(\textbf{x}$) to prevent overfitting of a single Gaussion kernel to the training data. In order to explore the role of the Dirichlet prior, we compare the performance of our model with and without $\mathcal{L}_{\text{prior}}$. The results are shown in Table 6, it can be seen that the performance improves by adding the Dirichlet conjugate prior, especially for the difficult poses in actions “Sitting” and “SittingDown”. The reason is that most of the poses in the Human3.6 dataset are in a standing position, resulting in a worse performance on the “Sitting” and “SittingDown” actions. This further indicates that the Dirichlet conjugate prior can prevent overffiting effectively.

What is generated by each kernel?

In order to explore the relation between different hypotheses, we reproject all five pose hypotheses into the image plane and compute the difference between projections and the 2D input joints. We adopt the PCKh@0.5 score which is the standard metric for 2D pose estimation to measure the difference. The high PCKh@0.5 score in Table 7 suggests that all the five hypotheses have almost the same 2D reprojections which are consistent with the 2D input. Note that we do not add any constraint as did to force all hypotheses to be consistent in the 2D reprojections.

We give several visualization results in Figure 4 to further illustrate the relations between all pose hypotheses. As described by Eqn. (3.1), each Gaussian kernel can be seen to generate the same hypotheses for simple pose with less ambiguity, e.g. standing (first row). This means that single Gaussian distribution is sufficient for simple poses. In comparison, our network can be seen to generate different hypotheses for challenging poses like “GettingDown” or “SittingDown” (second and third rows) due to two reasons. Firstly, our network receives lesser information on this type of poses since most of the poses in the Human3.6 dataset are the “standing” poses. Secondly, there are more ambiguities and occlusions for the “GettingDown” or “SittingDown” poses. As a result, our network generates multiple pose hypotheses to mitigate the increase of the uncertainty. We also visualize the 2D reprojections of all hypotheses in the last column. We indicate the corresponding 2D reprojection and 3D pose with the same color. The overlaps between the 2D reprojections further validate that our network generates hypotheses that are consistent in the 2D image coordinates.

Conclusion

In this work, we introduce the use of a mixture density network to generate multiple feasible hypotheses for the inverse problem of 3D human pose estimation from 2D inputs. Experimental results show that our network achieves state-of-the-art results in both best hypothesis and multi-view settings. Furthermore, the 3D pose hypotheses generated by our network are consistent in the 2D reprojections suggests that the hypotheses model the ambiguity along the depth of the joints. Results on the MPII and MPI-INF-3DHP datasets further show the generalization capacity of our network.

References