Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression

Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, Jingdong Wang

Introduction

Human pose estimation is a problem of predicting the keypoint positions of each person from an image, i.e., localize the keypoints as well as identify the keypoints belonging to the same person. There are broad applications, including action recognition, human-computer interaction, smart photo editing, pedestrian tracking, etc.

There are two main paradigms: top-down and bottom-up. The top-down paradigm first detects the person and then performs single-person pose estimation for each detected person. The bottom-up paradigm either directly regresses the keypoint positions belonging to the same person, or detects and groups the keypoints, such as affinity linking , associative embedding , HGG and HigherHRNet . The top-down paradigm is more accurate but more costly due to an extra person detection process, and the bottom-up paradigm, the interest of this paper, is more efficient.

The recently-developed pixel-wise keypoint regression approach, CenterNet , estimates the $K$ keypoint positions together for each pixel from the representation at the pixel. Direct regression to keypoint positions in CenterNet performs reasonably. But the regressed keypoints are spatially not accurate and the performance is worse than the keypoint detection and grouping scheme. Figure 1 (left) shows two examples in which the salient areas for keypoint regression spread broadly and the regression quality is not satisfactory.

We argue that regressing the keypoint positions accurately needs to learn representations that focus on the keypoint regions. Starting from this regression by focusing concept, we present a simple yet effective approach, named disentangled keypoint regression (DEKR). We adopt adaptive convolutions, through pixel-wise spatial transformer (a pixel-wise extension of spatial transformer network ), to activate the pixels lying in the keypoint regions, and then learn the representations from these activated pixels, so that the learned representations can focus on the keypoint regions.

We further decouple the representation learning for one keypoint from other keypoints. We adopt a separate regression scheme through a multi-branch structure: each branch learns a representation for one keypoint with adaptive convolutions dedicated for the keypoint and regresses the position for the corresponding keypoint. Figure 1 (right) illustrates that our approach is able to learn highly concentrative representations, each of which focuses on the corresponding keypoint region.

Experimental results demonstrate that the proposed DEKR approach improves the localization quality of the regressed keypoint positions. Our approach, that performs direct keypoint regression without matching the regression results to the closest keypoints detected from the keypoint heatmaps, outperforms keypoint detection and grouping methods and achieves superior performance over previous state-of-the-art bottom-up pose estimation methods on two benchmark datasets, COCO and CrowdPose.

Our contributions to bottom-up human pose estimation are summarized as follows.

We argue that the representations for regressing the positions of the keypoints accurately need to focus on the keypoint regions.

The proposed DEKR approach is able to learn disentangled representations through two simple schemes, adaptive convolutions and multi-branch structure, so that each representation focuses on one keypoint region and the prediction of the corresponding keypoint position from such representation is accurate.

The proposed direct regression approach outperforms keypoint detection and grouping schemes and achieves new state-of-the-art bottom-up pose estimation results on the benchmark datasets, COCO and CrowdPose.

Related Work

The convolutional neural network (CNN) solutions to human pose estimation have shown superior performance over the conventional methods, such as the probabilistic graphical model or the pictorial structure model . Early CNN approaches directly predict the keypoint positions for single-person pose estimation, which is later surpassed by the heatmap estimation based methods . The geometric constraints and structured relations among body keypoints are studied for performance improvement .

Top-down paradigm. The top-down methods perform single-person pose estimation by firstly detecting each person from the image. Representative works include: HRNet , PoseNet , RMPE , convolutional pose machine , Hourglass , Mask R-CNN , CFN , Integral pose regression , CPN , simple baseline , CSM-SCARB , Graph-PCNN , RSN , and so on. These methods exploit the advances in person detection as well as extra person bounding-box labeling information. The top-down paradigm, though achieving satisfactory performance, takes extra cost in person box detection.

Other developments include improving the keypoint localization from the heatmap , refining pose estimation , better data augmentation , developing a multi-task learning architecture combining detection, segmentation and pose estimation , and handling the occlusion issue .

Bottom-up paradigm. Most existing bottom-up methods mainly focus on how to associate the detected keypoints that belong to the same person together. The pioneering work, DeepCut , DeeperCut , and L-JPA formulate the keypoint association problem as an integer linear program, which however takes longer processing time (e.g., the order of hours).

Various grouping techniques are developed, such as part-affinity fields in OpenPose and its extension in PifPaf , associative embedding , greedy decoding with hough voting in PersonLab , and graph clustering in HGG .

Several recent works densely regress a set of pose candidates, where each candidate consists of the keypoint positions that might be from the same person. Unfortunately, the regression quality is not high, and the localization quality is weak. A post-processing scheme, matching the regressed keypoint positions to the closest keypoints (which is spatially more accurate) detected from the keypoint heatmaps, is usually adopted to improve the regression results.

Our approach aims to improve the direct regression results, by exploring our regression by focusing idea. We learn $K$ disentangled representations, each of which is dedicated for one keypoint and learns from the adaptively activated pixels, so that each representation focuses on the corresponding keypoint area. As a result, the position prediction for one keypoint from the corresponding disentangled representation is spatially accurate. Our approach is superior to and differs from that uses the mixture density network for handling uncertainty to improve direct regression results.

Disentangled representation learning. Disentangled representations have widely been studied in computer vision , e.g., disentangling the representations into content and pose , disentangling motion from content , disentangling pose and appearance .

Our proposed disentangled regression in some sense can be regarded as disentangled representation learning: learn the representation for each keypoint separately from the corresponding keypoint region. The idea of representation disentanglement for pose estimation is also explored in the top-down approach, part-based branching network (PBN) , which learns high-quality heatmaps by disentangling representations into each part group. They are clearly different: our approach learns representations focusing on each keypoint region for position regression, and PBN de-correlates the appearance representations among different part groups.

Approach

Given an image $\mathsf{I}$ , multi-person pose estimation aims to predict the human poses, where each pose consists of $K$ keypoints, such as shoulder, elbow, and so on. Figure 2 illustrates the multi-person pose estimation problem.

The pixel-wise keypoint regression framework estimates a candidate pose at each pixel $\mathbf{q}$ (called center pixel), by predicting an $2K$ -dimensional offset vector $\mathbf{o}_{q}$ from the center pixel $\mathbf{q}$ for the $K$ keypoints. The offset maps $\mathsf{O}$ , containing the offset vectors at all the pixels, are estimated through a keypoint regression head,

where $\mathsf{X}$ is the feature computed from a backbone, HRNet in this paper, and $\mathcal{F}(~{})$ is the keypoint position regression head predicting the offset maps $\mathsf{O}$ .

The structure of the proposed disentangled keypoint regression (DEKR) head is illustrated in Figure 3. DEKR adopts the multi-branch parallel adaptive convolutions to learn disentangled representations for the regression of the $K$ keypoints, so that each representation focuses on the corresponding keypoint region.

Adaptive activation. One normal convolution (e.g., $3\times 3$ convolution) only sees the pixels nearby the center pixel $\mathbf{q}$ . A sequence of several normal convolutions may see the pixels farther from the center pixel that might lie in the keypoint region, but might not focus on and highly activate these pixels.

We adopt the adaptive convolutions, to learn representations focusing on the keypoint region. The adaptive convolution is a modification of a normal convolution (e.g., $3\times 3$ convolution):

Here, $\mathbf{q}$ is the center ( $2$ D) position, and $\mathbf{g}^{q}_{si}$ is the offset, $\mathbf{g}^{q}_{si}+\mathbf{q}$ corresponds to the $i$ th activated pixel. $\{\mathbf{W}_{1},\mathbf{W}_{2},\dots,\mathbf{W}_{9}\}$ are the kernel weights.

Separate regression. The offset regressor $\mathcal{F}$ in is a single branch, and estimates all the $K$ $2D$ offsets together from a single feature for each position. We propose to use a $K$ -branch structure, where each branch performs the adaptive convolutions and then regresses the offset for the corresponding keypoint.

We divide the feature maps $\mathsf{X}$ output from the backbone into $K$ feature maps, $\mathsf{X}_{1},\mathsf{X}_{2},\dots,\mathsf{X}_{K}$ , and estimate the offset map $\mathsf{O}_{k}$ for each keypoint from the corresponding feature map:

where $\mathcal{F}_{k}(~{})$ is the $k$ th regressor on the $k$ th branch, and $\mathsf{O}_{k}$ is the offset map for the $k$ th keypoint. The $K$ regressors, $\{\mathcal{F}_{1}(~{}),\mathcal{F}_{2}(~{}),\dots,\mathcal{F}_{K}(~{})\}$ have the same structures, and their parameters are learned independently.

Each branch in separate regression is able to learn its own adaptive convolutions, and accordingly focuses on activating the pixels in the corresponding keypoint region (see Figure 4 (b - e)). In the single-branch case, the pixels around all the keypoints are activated, and the activation is not focused (see Figure 4 (a)).

The multi-branch structure explicitly decouples the representation learning for one keypoint from other keypoints, and thus improves the regression quality. In contrast, the single-branch structure has to decouple the feature learning implicitly which increases the optimization difficulty. Our results in Figure 5 show that the multi-branch structure reduces the regression loss.

2 Loss Function

Regression loss. We use the normalized smooth loss to form the pixel-wise keypoint regression loss:

Here, $Z_{i}=\sqrt{H_{i}^{2}+W_{i}^{2}}$ is the size of the corresponding person instance, and $H_{i}$ and $W_{i}$ are the height and the width of the instance box. $\mathcal{C}$ is the set of the positions that have groundtruth poses. $\mathbf{o}_{i}$ ( $\mathbf{o}_{i}^{*}$ ), a column of the offset maps $\mathsf{O}$ ( $\mathsf{O}^{*}$ ), is the $2K$ -dimensional estimated (groundtruth) offset vector for the position $i$ .

Keypoint and center heatmap estimation loss. We also estimate $K$ keypoint heatmaps each corresponding to a keypoint type and the center heatmap indicating the confidence that each pixel is the center of some person, using a separate heatmap estimation branch,

The heatmaps are used for scoring and ranking the regressed poses. The heatmap estimation loss function is formulated as the weighted distances between the predicted heat values and the groundtruth heat values:

Here, $\|\cdot\|_{2}$ is the entry-wise $2$ -norm. $\odot$ is the element-wise product operation. $\mathsf{M}^{h}$ has $K$ masks, and the size is $H\times W\times K$ . The $k$ th mask, $\mathsf{M}^{h}_{k}$ , is formed so that the mask weight of the positions not lying in the $k$ th keypoint region is $0.1$ , and others are $1$ . The same is done for the mask $\mathbf{M}^{c}$ for the center heatmap. $\mathsf{H}^{*}$ and $\mathbf{C}^{*}$ are the target keypoint and center heatmaps.

Whole loss. The whole loss function is the sum of the heatmap loss and the regression loss:

where $\lambda$ is a trade-off weight, and set as $0.03$ in our experiments.

3 Inference

A testing image is fed into the network, outputting the regressed pose at each position, and the keypoint and center heatmaps. We first perform the center NMS process on the center heatmap to remove non-locally maximum positions and the positions whose center heat value is not higher than $0.01$ . Then we perform the pose NMS process over the regressed poses at the positions remaining after center NMS, to remove some overlapped regressed poses, and maintain at most $30$ candidates. The score used in pose NMS is the average of the heat values at the regressed $K$ keypoints, which is helpful to keep candidate poses with highly accurately localized keypoints.

We rank the remaining candidate poses using the score that is estimated by jointly considering their corresponding center heat values, keypoint heat values and their shape scores. The shape feature includes the distance and the relative offset between a pair of neighboring keypoints A neighboring pair $(i,j)$ corresponds to a stick in the COCO dataset, and there are $19$ sticks (denoted by $\mathcal{E}$ ) in the COCO dataset.: $\{d_{ij}|(i,j)\in\mathcal{E}\}$ and $\{\mathbf{p}_{i}-\mathbf{p}_{j}|(i,j)\in\mathcal{E}\}$ , and keypoint heat values indicating the visibility of each keypoint. We input the three kinds of features to a scoring net, consisting of two fully-connected layers (each followed by a ReLU layer), and a linear prediction layer, which aims to learn the OKS score for the corresponding predicted pose with the real OKS as the target on the training set.

4 Discussions

Separate regression, group convolution and complexities. In the multi-branch structure, we divide the channel maps into $K$ non-overlapping partitions, and feed each partition into each branch, for learning disentangled representations. This process resembles group convolution. The difference lies in: group convolution usually increases the capacity of the whole representation through reducing the redundancy and increasing the width within the computation and parameter budget, while our approach does not change the width and aims to learn rich representations focusing on each keypoint.

Let’s look at an example with HRNet-W $32$ as the backbone. The standard process in HRNet-W $32$ concatenates the channels obtained from $4$ resolutions and feeds the concatenated channels to a $1\times 1$ convolution, outputting $256$ channels. When applied to our disentangled regressors, we modify the $1\times 1$ convolution to output $255$ ( $=17\times 15$ ) channels, so that each partition has $15$ channels. This modification does not increase the width. The parameter complexity and the computation complexity for the regression head are reduced, and in particular the overall computation complexity is significantly reduced. The detailed numbers are given in Table 1.

Separate group regression. It is noticed that the salient regions and the activated pixels for some keypoints might have some overlapping. For example, the salient regions of the five keypoints in the head are overlapped, and three keypoints in the arms have similar characteristics. We investigate the performance if grouping some keypoints into a single branch instead of letting each branch handle one single keypoint. We consider two grouping schemes. (1) Five keypoints in the head use a single branch, and there are totally $13$ branches. (2) Five keypoints in the head use a single branch, the three keypoints in left arm (right arm, left leg, right leg) use a single branch. There are totally $5$ branches. Empirical results show that separate group regression performs worse than separate regression, e.g., the AP score for five branches decreases by $0.4$ on COCO validation with the backbone HRNet-W $32$ .

Experiments

Dataset. We evaluate the performance on the COCO keypoint detection task . The train $2017$ set includes $57K$ images and $150K$ person instances annotated with $17$ keypoints, the val $2017$ set contains $5K$ images, and the test-dev $2017$ set consists of $20K$ images. We train the models on the train $2017$ set and report the results on the val $2017$ and test-dev $2017$ sets.

Training set construction. The training sets consist of keypoint and center heatmaps, and offset maps.

Groundtruth keypoint and center heatmaps: The groundtruth keypoint heatmaps $\mathsf{H}^{*}$ for each image contains $K$ maps, and each map corresponds to one keypoint type. We build them as done in : assigning a heat value using the Gaussian function centered at a point around each groundtruth keypoint. The center heatmap is similarly constructed and described in the following.

Groundtruth offset maps: The groundtruth offset maps $\mathsf{O}^{*}$ for each image are constructed from all the poses $\{\mathcal{P}_{1},\mathcal{P}_{2},\cdots,\mathcal{P}_{N}\}$ . We use the $n$ th pose $\mathcal{P}_{n}$ as an example and others are the same. We compute the center position $\bar{\mathbf{p}}_{n}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{p}_{nk}$ and the offsets $\mathcal{T}_{n}=\{\mathbf{p}_{n1}-\bar{\mathbf{p}}_{n},\mathbf{p}_{n2}-\bar{\mathbf{p}}_{n},\cdots,\mathbf{p}_{nK}-\bar{\mathbf{p}}_{n}\}$ as the groundtruth offsets for the pixel corresponding to the center position. We use an expansion scheme to augment the center point to the center region: $\{\mathbf{m}_{n}^{1},\mathbf{m}_{n}^{2},\cdots,\mathbf{m}_{n}^{M}\}$ , which are the central positions around the center point $\bar{\mathbf{p}}_{n}$ with the radius $4$ , and accordingly update the offsets. The positions not lying in the region have no offset values.

Each central position $\mathbf{m}_{n}^{m}$ has a confidence value $c_{n}^{m}$ indicating how confident it is the center and computed using the way forming the groundtruth center heatmap $\mathbf{C}^{*}$ In case that one position belongs to two or more central regions, we choose only one central region whose center is the closest to that position.. The positions not lying in the region have zero heat value.

Evaluation metric. We follow the standard evaluation metrichttp://cocodataset.org/#keypoints-eval and use OKS-based metrics for COCO pose estimation. We report average precision and average recall scores with different thresholds and different object sizes: $\operatorname{AP}$ , $\operatorname{AP}^{50}$ , $\operatorname{AP}^{75}$ , $\operatorname{AP}^{M}$ , $\operatorname{AP}^{L}$ , $\operatorname{AR}$ , $\operatorname{AR}^{M}$ , and $\operatorname{AR}^{L}$ .

Training. The data augmentation follows and includes random rotation ( $[<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">,</annotation></semantics></math>,]$ ), random scale ( $[0.75,1.5]$ ) and random translation ($ $). We conduct image cropping to$ 512\times 512 $for HRNet-W$ 32 $or$ 640\times 640 $for HRNet-W$ 48$ with random flipping as training samples.

Testing. We resize the short side of the images to $512/640$ and keep the aspect ratio between height and width, and compute the heatmap and pose positions by averaging the heatmaps and pixel-wise keypoint regressions of the original and flipped images. Following , we adopt three scales $0.5,1$ and $2$ in multi-scale testing. We average the three heatmaps over three scales and collect the regressed results from the three scales as the candidates.

2 Results

COCO Validation. Table 2 shows the comparisons of our method and other state-of-the-art methods. Table 3 presents the parameter and computation complexities for our approach and the representative top competitors, such as AE-Hourglass , PersonLab , and HrHRNet .

Our approach, using HRNet-W $32$ as the backbone, achieves $68.0$ AP score. Compared to the methods with similar GFLOPs, CenterNet-DLA and PersonLab (with the input size $601$ ), our approach achieves over $9.0$ improvement. In comparison to CenterNet-HG whose model size is far larger than HRNet-W $32$ , our gain is $4.0$ . Our baseline result $61.9$ (Table 5) is lower than CenterNet-HG that adopts post-processing to match the predictions to the closest keypoints identified from keypoint heatmaps. This implies that our gain comes from our methodology.

Our approach benefits from large input size and large model size. Our approach, with HRNet-W $48$ as the backbone and the input size $640$ , obtains the best performance $71.0$ and $3.0$ gain over HRNet-W $32$ . Compared with state-of-the-art methods, our approach gets $7.0$ gain over CenterNet-HG, $4.5$ gain over PersonLab (the input size $1401$ ), $3.6$ gain over PifPaf whose GFLOPs are more than twice as many as ours, and $1.1$ gain over HrHRNet-W $48$ that uses higher resolution representations.

Following , we report the results with multi-scale testing. This brings about $2.7$ gain for HRNet-W $32$ , $1.3$ gain for HRNet-W $48$ .

COCO test-dev. The results of our approach and other state-of-the-art methods on the test-dev dataset are presented in Table 4. Our approach with HRNet-W $32$ as the backbone achieves $67.3$ AP score, and significantly outperforms the methods with the similar model size. Our approach with HRNet-W $48$ as the backbone gets the best performance $70.0$ , leading to $3.5$ gain over PersonLab, $3.3$ gain over PifPaf , and $1.6$ gain over HrHRNet .

With multi-scale testing, our approach with HRNet-W $32$ achieves $69.8$ , even better than PersonLab with a larger model size. Our approach with HRNet-W $48$ achieves $71.0$ AP score, much better than associative embedding , $2.3$ gain over PersonLab, and $0.5$ gain over HrHRNet .

3 Empirical Analysis

Ablation study. We study the effects of the two components: adaptive activation (AA) and separate regression (SR). We use the backbone HRNet-W $32$ as an example. The observations are consistent for HRNet-W $48$ .

The ablation study results are presented in Table 5. We can observe: (1) adaptive activation (AA) achieves the gain $3.5$ over the regression baseline ( $61.9$ ). (2) separate regression (SR) further improves the AP score by $2.6$ . (3) separate regression w/o adaptive activation gets $1.7$ AP gain. The whole gain is $6.1$ .

We further analyze how each component contributes to the performance improvement by using the coco-analyze tool . Four error types are studied: (i) Jitter error: small localization error; (ii) Miss error: large localization error; (iii) Inversion error: confusion between keypoints within an instance. (iv) Swap error: confusion between keypoints of different instances. The detailed definitions are in .

Table 5 shows the errors of the four types for four schemes. The two components, adaptive activation (AA) and separate regression (SR), mainly influence the two localization errors, Jitter and Miss. Adaptive activation reduces the Jitter error and the Miss error by $2.9$ and $0.9$ , respectively. Separate regression further reduces the two errors by $1.7$ and $0.6$ . The other two errors are almost not changed. This indicates that the proposed two components indeed improve the localization quality.

Comparison with grouping detected keypoints. It is reported in HigherHRNet that associative embedding with HRNet-W $32$ achieves an AP score $64.4$ on COCO validation. The regression baseline using the same backbone HRNet-W $32$ gets an lower AP score $61.9$ (Table 5). The proposed two components lead to an AP score $68.0$ , higher than associative embedding $+$ HRNet-W $32$ .

Matching regression to the closest keypoint detection. The CenterNet approach performs a post-processing step to refine the regressed keypoint positions by absorbing the regressed keypoint to the closest keypoint among the keypoints identified from the keypoint heatmaps.

We tried this absorbing scheme. The results are presented in Table 6. We can see that the absorbing scheme does not improve the performance in the single-scale testing case. The reason might be that the keypoint localization quality of our approach is very close to that of keypoint identification from the heatmap. In the multi-scale testing case, the absorbing scheme improves the results. The reason is that keypoint position regression is conducted separately for each scale and the absorbing scheme makes the regression results benefit from the heatmap improved from multiple scales. Our current focus is not on multi-scale testing whose practical value is not as high as single-scale testing. We leave finding a better multi-scale testing scheme as our future work.

4 CrowdPose

Dataset. We evaluate our approach on the CrowdPose dataset that is more challenging and includes many crowded scenes. The train set contains $10K$ images, the val set includes $2K$ images and the test set consists of $20K$ images. We train our models on the CrowdPose train and val sets and report the results on the test set as done in .

Evaluation metric. The standard average precision based on OKS which is the same as COCO is adopted as the evaluation metrics. The CrowdPose dataset is split into three crowding levels: easy, medium, hard. We report the following metrics: $\operatorname{AP}$ , $\operatorname{AP}^{50}$ , $\operatorname{AP}^{75}$ , as well as $\operatorname{AP}^{E}$ , $\operatorname{AP}^{M}$ and $\operatorname{AP}^{H}$ for easy, medium and hard images.

Test set results. The results of our approach and other state-of-the-art methods on the test set are showed in Table 7. Our approach with HRNet-W $48$ as the backbone achieves $67.3$ AP and is better than HrHRNet-W $48$ ( $65.9$ ) that is a keypoint detection and grouping approach with a backbone that is designed for improving heatmaps. With multi-scale testing, our approach with HRNet-W $48$ achieves 68.0 AP score and by a further matching process (see Table 6) the performance is improved, leading to 0.7 gain over HrHRNet-W $48$ .

Conclusions

The proposed direct regression approach DEKR improves the keypoint localization quality and achieves state-of-the-art bottom-up pose estimation results. The success stems from that we disentangle the representations for regressing different keypoints so that each representation focuses on the corresponding keypoint region. We believe that the idea of regression by focusing and disentangled keypoint regression can benefit some other methods, such as CornetNet and CenterNet for object detection.