Accurate 3D Body Shape Regression using Metric and Semantic Attributes

Introduction

The field of 3D human pose and shape (HPS) estimation is progressing rapidly and methods now regress accurate 3D pose from a single image [bogo2016keep, Joo2018_adam, kanazawa_2019_cvpr, VIBE:CVPR:2020, Kolotouros2019_spin, Pavlakos2019_smplifyx, xu2020ghum, pare, spec, pymaf]. Unfortunately, less attention has been paid to body shape and many methods produce body shapes that clearly do not represent the person in the image (Fig. 1, top right). There are several reasons behind this. Current evaluation datasets focus on pose and not shape. Training datasets of images with 3D ground-truth shape are lacking. Additionally, humans appear in images wearing clothing that obscures the body, making the problem challenging. Finally, the fundamental scale ambiguity in 2D images, makes 3D shape difficult to estimate. For many applications, however, realistic body shape is critical. These include AR/VR, apparel design, virtual try-on, and fitness. To democratize avatars, it is important to represent and estimate all possible 3D body shapes; we make a step in that direction.

Note that commercial solutions to this problem require users to wear tight fitting clothing and capture multiple images or a video sequence using constrained poses. In contrast, we tackle the unconstrained problem of 3D body shape estimation in the wild from a single RGB image of a person in an arbitrary pose and standard clothing.

Most current approaches to HPS estimation learn to regress a parametric 3D body model like SMPL [SMPL:2015] from images using 2D joint locations as training data. Such joint locations are easy for human annotators to label in images. Supervising the training with joints, however, is not sufficient to learn shape since an infinite number of body shapes can share the same joints. For example, consider someone who puts on weight. Their body shape changes but their joints stay the same. Several recent methods employ additional 2D cues, such as the silhouette, to provide additional shape cues [sengupta2020straps, sengupta2021hierarchicalICCV]. Silhouettes, however, are influenced by clothing and do not provide explicit 3D supervision. Synthetic approaches [Liang_2019_ICCV], on the other hand, drape SMPL 3D bodies in virtual clothing and render them in images. While this provides ground-truth 3D shape, realistic synthesis of clothed humans is challenging, resulting in a domain gap.

To address these issues, we present SHAPY, a new deep neural network that accurately regresses 3D body shape and pose from a single RGB image. To train SHAPY, we first need to address the lack of paired training data with real images and ground-truth shape. Without access to such data, we need alternatives that are easier to acquire, analogous to 2D joints used in pose estimation. To do so, we introduce two novel datasets and corresponding training methods.

First, in lieu of full 3D body scans, we use images of people with diverse body shapes for which we have anthropometric measurements such as height as well as chest, waist, and hip circumference. While many 3D human shapes can share the same measurements, they do constrain the space of possible shapes. Additionally, these are important measurements for applications in clothing and health. Accurate anthropometric measurements like these are difficult for individuals to take themselves but they are often captured for different applications. Specifically, modeling agencies provide such information about their models; accuracy is a requirement for modeling clothing. Thus, we collect a diverse set of such model images (with varied ethnicity, clothing, and body shape) with associated measurements; see Fig. 2.

Since sparse anthropometric measurements do not fully constrain body shape, we exploit a novel approach and also use linguistic shape attributes. Prior work has shown that people can rate images of others according to shape attributes such as “short/tall”, “long legs” or “pear shaped” [Streuber:SIGGRAPH:2016]; see Fig. 3. Using the average scores from several raters, Streuber et al. [Streuber:SIGGRAPH:2016] (BodyTalk) regress metrically accurate 3D body shape. This approach gives us a way to easily label images of people and use these labels to constrain 3D shape. To our knowledge, this sort of linguistic shape attribute data has not previously been exploited to train a neural network to infer 3D body shape from images.

We exploit these new datasets to train SHAPY with three novel losses, which can be exploited by any 3D human body reconstruction method: (1) We define functions of the SMPL body mesh that return a sparse set of anthropometric measurements. When measurements are available for an image we use a loss that penalizes mesh measurements that differ from the ground-truth (GT). (2) We learn a “Shape to Attribute” (S2A) function that maps 3D bodies to linguistic attribute scores. During training, we map meshes to attribute scores and penalize differences from the GT scores. (3) We similarly learn a function that maps “Attributes to Shape” (A2S). We then penalize body shape parameters that deviate from the prediction.

We study each term in detail to arrive at the final method. Evaluation is challenging because existing benchmarks with GT shape either contain too few subjects [vonMarcard2018] or have limited clothing complexity and only pseudo-GT shape [sengupta2020straps]. We fill this gap with a new dataset, named “Human Bodies in the Wild” (HBW), that contains a ground-truth 3D body scan and several in-the-wild photos of 35 subjects, for a total of 2543 photos. Evaluation on this shows that SHAPY estimates much more accurate 3D shape.

Models, data and code are available at shapy.is.tue.mpg.de.

Related Work

3D human pose and shape (HPS): Methods that reconstruct 3D human bodies from one or more RGB images can be split into two broad categories: (1) parametric methods that predict parameters of a statistical 3D body model, such as SCAPE [anguelov2005scape], SMPL [SMPL:2015], SMPL-X [Pavlakos2019_smplifyx], Adam [Joo2018_adam], GHUM [xu2020ghum], and (2) non-parametric methods that predict a free-form representation of the human body [varol2018bodynet, saito2020pifuhd, Jafarian_2021_CVPR, xiu2022icon]. Parametric approaches lack details w.r.t. non-parametric ones, e.g., clothing or hair. However, parametric models disentangle the effects of identity and pose on the overall shape. Therefore, their parameters provide control for re-shaping and re-posing. Moreover, pose can be factored out to bring meshes in a canonical pose; this is important for evaluating estimates of an individual’s shape. Finally, since topology is fixed, meshes can be compared easily. For these reasons, we use a SMPL-X body model.

Parametric methods follow two main paradigms, and are based on optimization or regression. Optimization-based methods [balan2007detailed, bogo2016keep, guan_iccv_scape_2009, Pavlakos2019_smplifyx] search for model configurations that best explain image evidence, usually 2D landmarks [OpenPose_PAMI], subject to model priors that usually encourage parameters to be close to the mean of the model space. Numerous methods penalize the discrepancy between the projected and ground-truth silhouettes [MuVS_3DV_2017, lassner2017unite] to estimate shape. However, this needs special care to handle clothing [Balan:ECCV]; without this, erroneous solutions emerge that “inflate” body shape to explain the “clothed” silhouette. Regression-based methods [Choutas2020_expose, georgakis2020hierarchical, jiang2020multiperson, Kanazawa2018_hmr, Kolotouros2019_spin, Liang_2019_ICCV, VIBE:CVPR:2020, mueller2021tuch, zanfir2020weakly] are currently based on deep neural networks that directly regress model parameters from image pixels. Their training sets are a mixture of data captured in laboratory settings [ionescu2013human36m, sigal_ijcv_10b], with model parameters estimated from MoCap markers [AMASS:ICCV:2019], and in-the-wild image collections, such as COCO [lin2014coco], that contain 2D keypoint annotations. Optimization and regression can be combined, for example via in-the-network model fitting [Kolotouros2019_spin, mueller2021tuch].

Estimating 3D body shape: State-of-the-art methods are effective for estimating 3D pose, but struggle with estimating body shape under clothing. There are several reasons for this. First, 2D keypoints alone are not sufficient to fully constrain 3D body shape. Second, shape priors address the lack of constraints, but bias solutions towards “average” shapes [bogo2016keep, Pavlakos2019_smplifyx, Kolotouros2019_spin, mueller2021tuch]. Third, datasets with in-the-wild images have noisy 3D bodies, recovered by fitting a model to 2D keypoints [bogo2016keep, Pavlakos2019_smplifyx]. Fourth, datasets captured in laboratory settings have a small number of subjects, who do not represent the full spectrum of body shapes. Thus, there is a scarcity of images with known, accurate, 3D body shape. Existing methods deal with this in two ways.

First, rendering synthetic images is attractive since it gives automatic and precise ground-truth annotation. This involves shaping, posing, dressing and texturing a 3D body model [Hoffmann:GCPR:2019, sengupta2020straps, sengupta2021probabilisticCVPR, varol17_surreal, weitz2021infiniteform], then lighting it and rendering it in a scene. Doing this realistically and with natural clothing is expensive, hence, current datasets suffer from a domain gap. Alternative methods use artist-curated 3D scans [saito2019pifu, saito2020pifuhd, patel2020agora], which are realistic but limited in variety.

Second, 2D shape cues for in-the-wild images, (body-part segmentation masks [omran2018neural, Ruegg:AAAI:2020, sai2021dsr], silhouettes [agarwal_trigs_3d_poses, MuVS_3DV_2017, pavlakos2018learning]) are attractive, as these can be manually annotated or automatically detected [gong2019graphonomy, He2020maskRCNN]. However, fitting to such cues often gives unrealistic body shapes, by inflating the body to “explain” the clothing “baked” into silhouettes and masks.

Most related to our work is the work of Sengupta et al. [sengupta2020straps, sengupta2021probabilisticCVPR, sengupta2021hierarchicalICCV] who estimate body shape using a probabilistic learning approach, trained on edge-filtered synthetic images. They evaluate on the SSP-3D dataset of real images with pseudo-GT 3D bodies, estimated by fitting SMPL to multiple video frames. SSP-3D is biased to people with tight-fitting clothing. Their silhouette-based method works well on SSP-3D but does not generalize to people in normal clothing, tending to over-estimate body shape; see Fig. 1.

In contrast to previous work, SHAPY is trained with in-the-wild images paired with linguistic shape attributes, which are annotations that can be easily crowd-sourced for weak shape supervision. We also go beyond SSP-3D to provide HBW, a new dataset with in-the-wild images, varied clothing, and precise GT from 3D scans.

Shape, measurements and attributes: Body shapes can be generated from anthropometric measurements [allen2003space, seo2003synthesizing, seo2003automatic]. Tsoli et al. [tsoliWACV14] register a body model to multiple high-resolution body scans to extract body measurements. The “Virtual Caliper” [pujades2019virtual] allows users to build metrically accurate avatars of themselves using measurements or VR game controllers. ViBE [hsiao2020vibe] collects images, measurements (bust, waist, hip circumference, height) and the dress-size of models from clothing websites to train a clothing recommendation network. We draw inspiration from these approaches for data collection and supervision.

Streuber et al. [Streuber:SIGGRAPH:2016] learn BodyTalk, a model that generates 3D body shapes from linguistic attributes. For this, they select attributes that describe human shape and ask annotators to rate how much each attribute applies to a body. They fit a linear model that maps attribute ratings to SMPL shape parameters. Inspired by this, we collect attribute ratings for CAESAR meshes [CAESAR] and in-the-wild data as proxy shape supervision to train a HPS regressor. Unlike BodyTalk, SHAPY automatically infers shape from images.

Anthropometry from images: Single-View metrology [criminisi2000single] estimates the height of a person in an image, using horizontal and vertical vanishing points and the height of a reference object. Günel et al. [gunel2019face] introduce the IMDB-23K dataset by gathering publicly available celebrity images and their height information. Zhu et al. [SingleViewMetrology] use this dataset to learn to predict the height of people in images. Dey et al. [Ratan2014] estimate the height of users in a photo collection by computing height differences between people in an image, creating a graph that links people across photos, and solving a maximum likelihood estimation problem. Bieler et al. [Bieler_2019_ICCV] use gravity as a prior to convert pixel measurements extracted from a video to metric height. These methods do not address body shape.

Representations & Data for Body Shape

We use linguistic shape attributes and anthropometric measurements as a connecting component between in-the-wild images and ground-truth body shapes; see Fig. 4. To that end, we annotate linguistic shape attributes for 3D meshes and in-the-wild images, the latter from fashion-model agencies, labeled via Amazon Mechanical Turk.

2 Model-Agency Images

Model agencies typically provide multiple color images of each model, in various poses, outfits, hairstyles, scenes, and with a varying camera framing, together with anthropometric measurements and clothing size. We collect training data from multiple model-agency websites, focusing on under-represented body types, namely: curve-models.com, cocainemodels.com, nemesismodels.com, jayjay-models.de, kultmodels.com, modelwerk.de, models1.co.uk. showcast.de, the-models.de, and ullamodels.com. In addition to photos, we store gender and four anthropometric measurements, i.e. height, chest, waist and hip circumference, when available. To avoid having the same subject in both the training and test set, we match model identities across websites to identify models that work for several agencies. For details, see Sup. Mat.

After identity filtering, we have $94,620$ images of $4,419$ models along with their anthropometric measurements. However, the distributions of these measurements, shown in Fig. 5, reveal a bias for “fashion model” body shapes, while other body types are under-represented in comparison to CAESAR [CAESAR]. To enhance diversity in body-shapes and avoid strong biases and log tails, we compute the quantized 2D-distribution for height and weight and sample up to $3$ models per bin. This results in $N=1,185$ models ($714$ females, $471$ males) and $20,635$ images.

3 Linguistic Shape Attributes

Human body shape can be described by linguistic shape attributes [hill2015exploring]. We draw inspiration from Streuber et al. [Streuber:SIGGRAPH:2016] who collect scores for $30$ linguistic attributes for $256$ 3D body meshes, generated by sampling SMPL’s shape space, to train a linear “attribute to shape” regressor. In contrast, we train a model that takes as input an image, instead of attributes, and outputs an accurate 3D shape (and pose).

We crowd-source linguistic attribute scores for a variety of body shapes, using images from the following sources:

Rendered CAESAR images: We use CAESAR [CAESAR] bodies to learn mappings between linguistic shape attributes, anthropometric measurements, and SMPL-X shape parameters, $\bm{\beta}$. Specifically, we register a “gendered” SMPL-X model with $100$ shape components to $1,700$ male and $2,102$ female 3D scans, pose all meshes in an A-pose, and render synthetic images with the same virtual camera.

Model-agency photos: Each annotator is shown $3$ body images per subject, sampled from the image pool of Sec. 3.2.

Annotation: To keep annotation tractable, we use $A=15$ linguistic shape attributes per gender (subset of BodyTalk’s [Streuber:SIGGRAPH:2016] attributes); see Tab. 1. Each image is annotated by $K=15$ annotators on Amazon Mechanical Turk. Their task is to “indicate how strongly [they] agree or disagree that the [listed] words describe the shape of the [depicted] person’s body”; for an example, see Sup. Mat. Annotations range on a discrete 5-level Likert scale from 1 (strongly disagree) to 5 (strongly agree). We get a rating matrix $\mathbf{A}\in\{1,2,3,4,5\}^{N\times A\times K}$, where $N$ is the number of subjects. In the following, $a_{ijk}$ denotes an element of $\mathbf{A}$.

Mapping Shape Representations

In Sec. 3 we introduce three body-shape representations: (1) SMPL-X’s PCA shape space (Sec. 3.1), (2) anthropometric measurements (Sec. 3.2), and (3) linguistic shape attribute scores (Sec. 3.3). Here we learn mappings between these, so that in Sec. 5 we can define new losses for training body shape regressors using multiple data sources.

We obtain anthropometric measurements from a 3D body mesh in a T-pose, namely height, $0pt(\bm{\beta})$, weight, $W(\bm{\beta})$, and chest, waist and hip circumferences, $C_{\text{c}}(\bm{\beta})$, $C_{\text{w}}(\bm{\beta})$, and $C_{\text{h}}(\bm{\beta})$, respectively, by following Wuhrer et al. [wuhrer2013estimating] and the “Virtual Caliper” [pujades2019virtual]. For details on how we compute these measurements, see Sup. Mat.

2 Attributes and 3D Shape

Attributes to Shape (A2S): We predict SMPL-X shape coefficients from linguistic attribute scores with a second-degree polynomial regression model. For each shape $\bm{\beta}_{i}$, i=1\dots\color[rgb]{0,0,0}N, we create a feature vector, $\mathbf{x}_{i}^{\text{\mbox{A2S}}}$, by averaging for each of the $A$ attributes the corresponding $K$ scores:

where $i$ is the shape index (list of “fashion” or CAESAR bodies), $j$ is the attribute index, and $k$ the annotation index.

We then define the full feature matrix for all $N$ shapes as:

where $\phi(\mathbf{x}_{i}^{\text{\mbox{A2S}}})$ maps $\mathbf{x}_{i}$ to 2nd order polynomial features.

The target matrix $\mathbf{Y}=[\bm{\beta}_{1},\dots,\bm{\beta}_{N}]^{\top}$ contains the shape parameters $\bm{\bm{\beta}}_{i}=[\bm{\beta}_{i,1},\dots,\bm{\beta}_{i,B}]^{\top}$. We compute the polynomial model’s coefficients $\bm{W}$ via least-squares fitting:

Empirically, the polynomial model performs better than several models that we evaluated; for details, see Sup. Mat.

Shape to Attributes (S2A): We predict linguistic attribute scores, $A$, from SMPL-X shape parameters, $\bm{\beta}$. Again, we fit a second-degree polynomial regression model. S2A has “swapped” inputs and outputs w.r.t. A2S:

Attributes & Measurements to Shape (AHWC2S): Given a sparse set of anthropometric measurements, we predict SMPL-X shape parameters, $\bm{\beta}$. The input vector is:

where $c_{c},c_{w},c_{h}$ is the chest, waist, and hip circumference, respectively, $h$ and $w$ are the height and weight, and HWC2S means Height + Weight + Circumference to Shape. The regression target is the SMPL-X shape parameters, $\mathbf{y}_{i}$.

When both Attributes and measurements are available, we combine them for the AHWC2S model with input:

In practice, depending on which measurements are available, we train and use different regressors. Following the naming convention of AHWC2S, these models are: AH2S, AHW2S, AC2S, and AHC2S, as well as their equivalents without attribute input H2S, HW2S, C2S, and HC2S. For an evaluation of the contribution of linguistic shape attributes on top of each anthropometric measurement, see Sup. Mat.

Training Data: To train the A2S and S2A mappings we use CAESAR data, for which we have SMPL-X shape parameters, anthropometric measurements, and linguistic attribute scores. We train separate gender-specific models.

3D Shape Regression from an Image

We present SHAPY, a network that predicts SMPL-X parameters from an RGB image with more accurate body shape than existing methods. To improve the realism and accuracy of shape, we explore training losses based on all shape representations discussed above, i.e., SMPL-X meshes (Sec. 3.1), linguistic attribute scores (Sec. 3.3) and anthropometric measurements (Sec. 4.1). In the following, symbols with/-out a hat are regressed/ground-truth values.

We convert shape $\hat{\bm{\beta}}$ to height and circumferences values $\{\hat{0}pt,\hat{C}_{\text{c}},\hat{C}_{\text{w}},\hat{C}_{\text{h}}\}=$ $\{0pt(\hat{\bm{\beta}}),C_{\text{c}}(\hat{\bm{\beta}}),C_{\text{w}}(\hat{\bm{\beta}}),C_{\text{h}}(\hat{\bm{\beta}})\}$, by applying our virtual measurement tool (Sec. 4.1) to the mesh $M(\hat{\bm{\beta}})$ in the canonical T-pose. We also convert shape $\hat{\bm{\beta}}$ to linguistic attribute scores, with $\hat{A}=\text{\text{\mbox{S2A}}}(\hat{\bm{\beta}})$.

We train various SHAPY versions with the following “SHAPY losses”, using either linguistic shape attributes, or anthropometric measurements, or both:

These are optionally added to a base loss, $L_{\text{\mbox{base}}}$, defined below in “training details”. The architecture of SHAPY, with all optional components, is shown in Fig. 6. A suffix of color-coded letters describes which of the above losses are used when training a model. For example, SHAPY-AH denotes a model trained with the attribute and height losses, i.e.: L_{\text{{\color[rgb]{0,0,0}SHAPY}-\text{\mbox{{\color[rgb]{0,0.75,0.16}{A}}{\color[rgb]{0.76,0.01,0}{H}}2S}}}}=L_{\text{\mbox{base}}}+L_{\text{{\color[rgb]{0,0.75,0.16}{attr}}}}+L_{\text{{\color[rgb]{0.76,0.01,0}{height}}}}.

Training Details: We initialize SHAPY with the ExPose [Choutas2020_expose] network weights and use curated fits [Choutas2020_expose], H3.6M [ionescu2013human36m], the SPIN [Kolotouros2019_spin] training data, and our model-agency dataset (Sec. 3.2) for training. In each batch, 50% of the images are sampled from the model-agency images, for which we ensure a gender balance. The “SHAPY losses” of Eqs. 8, 9 and 10 are applied only on the model-agency images. We use these on top of a standard base loss:

where $L_{\text{joints}}^{\text{2D}}$ and $L_{\text{joints}}^{\text{3D}}$ are 2D and 3D joint losses:

$L_{\bm{\theta}}$ and $L_{\bm{\beta}}$ are losses on pose and shape parameters, and $L_{\bm{\beta}}^{\text{prior}}$ is PIXIE’s [feng2021pixie] “gendered” shape prior. All losses are L2, unless otherwise explicitly specified. Losses on SMPL-X parameters are applied only on the pose data [ionescu2013human36m, Choutas2020_expose, Kolotouros2019_spin]. For more implementation details, see Sup. Mat.

Experiments

3D Poses in the Wild (3DPW) [vonMarcard2018]: We use this to evaluate pose accuracy. This is widely used, but has only 5 test subjects, i.e., limited shape variation. For results, see Sup. Mat.

Sports Shape and Pose 3D (SSP-3D)[sengupta2020straps]: We use this to evaluate 3D body shape accuracy from images. It has $62$ tightly-clothed subjects in $311$ in-the-wild images from Sports-1M [KarpathyCVPR14], with pseudo ground-truth SMPL meshes that we convert to SMPL-X for evaluation.

Model Measurements Test Set (MMTS): We use this to evaluate anthropometric measurement accuracy, as a proxy for body shape accuracy. To create MMTS, we withhold 2699/1514 images of 143/95 female/male identities from our model-agency data, described in Sec. 3.2

CAESAR Meshes Test Set (CMTS): We use CAESAR to measure the accuracy of SMPL-X body shapes and linguistic shape attributes for the models of Sec. 4. Specifically, we compute: (1) errors for SMPL-X meshes estimated from linguistic shape attributes and/or anthropometric measurements by A2S and its variations, and (2) errors for linguistic shape attributes estimated from SMPL-X meshes by S2A. To create an unseen mesh test set, we withhold 339 male and 410 female CAESAR meshes from the crowd-sourced CAESAR linguistic shape attributes, described in Sec. 3.3.

Human Bodies in the Wild (HBW): The field is missing a dataset with varied bodies, varied clothing, in-the-wild images, and accurate 3D shape ground truth. We fill this gap by collecting a novel dataset, called “Human Bodies in the Wild” (HBW), with three steps: (1) We collect accurate 3D body scans for 35 subjects (20 female, 15 male), and register a “gendered” SMPL-X model to these to recover 3D SMPL-X ground-truth bodies [Dyna]. (2) We take photos of each subject in “photo-lab” settings, i.e., in front of a white background with controlled lighting, and in various everyday outfits and “fashion” poses. (3) Subjects upload full-body photos of themselves taken in the wild. For each subject we take up to 111 photos in lab settings, and collect up to 126 in-the-wild photos. In total, HBW has 2543 photos, 1,318 in the lab setting and 1,225 in the wild. We split the data into a validation and a test set (val/test) with 10/25 subjects (6/14 female 4/11 male) and 781/1,762 images (432/983 female 349/779 male), respectively. Figure 7 shows a few HBW subjects, photos and their SMPL-X ground-truth shapes. All subjects gave prior written informed consent to participate in this study and to release the data. The study was reviewed by the ethics board of the University of Tübingen, without objections.

2 Evaluation Metrics

We use standard accuracy metrics for 3D body pose, but also introduce metrics specific to 3D body shape.

Anthropometric Measurements: We report the mean absolute error in mm between ground-truth and estimated measurements, computed as described in Sec. 4.1. When weight is available, we report the mean absolute error in kg.

MPJPE and V2V metrics: We report in Sup. Mat. the mean per-joint point error (MPJPE) and mean vertex-to-vertex error (V2V), when SMPL-X meshes are available. The prefix “PA” denotes metrics after Procrustes alignment.

Mean point-to-point error ($\text{P2P}_{20\text{K}}$): SMPL-X has a highly non-uniform vertex distribution across the body, which negatively biases the mean vertex-to-vertex (V2V) error, when comparing estimated and ground-truth SMPL-X meshes. To account for this, we evenly sample $20$K points on SMPL-X’s surface, and report the mean point-to-point ($\text{P2P}_{20\text{K}}$) error. For details, see Sup. Mat.

3 Shape-Representation Mappings

We evaluate the models A2S and S2A, which map between the various body shape representations (Sec. 4).

A2S and its variations: How well can we infer 3D body shape from just linguistic shape attributes, anthropometric measurements, or both of these together? In Tab. 2, we report reconstruction and measurement errors using many combinations of attributes (A), height (H), weight (W), and circumferences (C). Evaluation on CMTS data shows that attributes improve the overall shape prediction across the board. For example, height+attributes (AH2S) has a lower point-to-point error than height alone. The best performing model, AHWC, uses everything, with $\text{P2P}_{20\text{K}}$-errors of $5.8\pm 2.0$ mm (males) and $6.2\pm 2.4$ mm (females).

S2A: How well can we infer linguistic shape attributes from 3D shape? S2A’s accuracy on inferring the attribute Likert score is $75\%/69\%$ for males/females; details in Sup. Mat.

4 3D Shape from an Image

We evaluate all of our model’s variations (see Sec. 5) on the HBW validation set and find, perhaps surprisingly, that SHAPY-A outperforms other variants. We refer to this below (and Fig. 1) simply as “SHAPY” and report its performance in Tab. 3 for HBW, Tab. 4 for MMTS, and Tab. 5 for SSP-3D. For images with natural and varied clothing (HBW, MMTS), SHAPY significantly outperforms all other methods (Tabs. 3 and 4) using only weak 3D shape supervision (Attributes). On these images, Sengupta et al.’s method [sengupta2021hierarchicalICCV] struggles with the natural clothing. In contrast, their method is more accurate than SHAPY on SSP-3D (Tab. 5), which has tight “sports” clothing, in terms of PVE-T-SC, a scale-normalized metric used on this dataset. These results show that silhouettes are good for tight/minimal clothing and that SHAPY struggles with high BMI shapes due to the lack of such shapes in our training data; see Fig. 5. Note that, as HBW has true ground-truth 3D shape, it does not need SSP-3D’s scaling for evaluation.

A key observation is that training with linguistic shape attributes alone is sufficient, i.e., without anthropometric measurements. Importantly, this opens up the possibility for significantly larger data collections. For a study of how different measurements or attributes impact accuracy, see Sup. Mat. Figure 8 shows SHAPY’s qualitative results.

Conclusion

SHAPY is trained to regress more accurate human body shape from images than previous methods, without explicit 3D shape supervision. To achieve this, we present two different ways to collect proxy annotations for 3D body shape for in-the-wild images. First, we collect sparse anthropometric measurements from online model-agency data. Second, we annotate images with linguistic shape attributes using crowd-sourcing. We learn mappings between body shape, measurements, and attributes, enabling us to supervise a regressor using any combination of these. To evaluate SHAPY, we introduce a new shape estimation benchmark, the “Human Bodies in the Wild” (HBW) dataset. HBW has images of people in natural clothing and natural settings together with ground-truth 3D shape from a body scanner. HBW is more challenging than existing shape benchmarks like SSP-3D, and SHAPY significantly outperforms existing methods on this benchmark. We believe this work will open new directions, since the idea of leveraging linguistic annotations to improve 3D shape has many applications.

Limitations: Our model-agency training dataset (Sec. 3.2) is not representative of the entire human population and this limits SHAPY’s ability to predict larger body shapes. To address this, we need to find images of more diverse bodies together with anthropometric measurements and linguistic shape attributes describing them.

Social impact: Knowing the 3D shape of a person has advantages, for example, in the clothing industry to avoid unnecessary returns. If used without consent, 3D shape estimation may invade individuals’ privacy. As with all other 3D pose and shape estimation methods, surveillance and deep-fake creation is another important risk. Consequently, SHAPY’s license prohibits such uses.

Acknowledgments: This work was supported by the Max Planck ETH Center for Learning Systems and the International Max Planck Research School for Intelligent Systems. We thank Tsvetelina Alexiadis, Galina Henz, Claudia Gallatz, and Taylor McConnell for the data collection, and Markus Höschle for the camera setup. We thank Muhammed Kocabas, Nikos Athanasiou and Maria Alejandra Quiros-Ramirez for the insightful discussions.

Disclosure: https://files.is.tue.mpg.de/black/CoI_CVPR_2022.txt

Appendix A Data Collection

Each pair with $\mathcal{S}$ and $\mathcal{S}_{T}$ that has no element larger than the similarity threshold $\tau=0.3$ is ignored, as it contains dissimilar models. Finally, we check if $\mathcal{S}_{TQ}$ is larger than $\tau$, and we keep a list of all pairs for which this holds true.

A.2 Crowd-Sourced Linguistic Shape-Attributes

To collect human ratings of how much a word describes a body shape, we conduct a human intelligence task (HIT) on Amazon Mechanical Turk (AMT). In this task, we show an image of a person along with $15$ different gender-specific attributes. We then ask participants to indicate how strongly they agree or disagree that the provided words describe the shape of this person’s body. We arrange the rating buttons from strong disagreement to strong agreement with equal distances to create a $5$-point Likert scale. The rating choices are “strongly disagree” (score $1$), “rather disagree” (score $2$), “average” (score $3$), “rather agree” (score $4$), “strongly agree” (score $5$).

We ask multiple persons to rate each body and image, to “average out” the subjectivity of individual ratings [Streuber:SIGGRAPH:2016]. Additionally, we compute the Pearson correlation between averaged attribute ratings and ground-truth measurements. Examples of highly correlated pairs are “Big / Weight”, and “Short / Height”.

The layout of our CAESAR annotation task is visualized in Fig. A.1. To ensure good rating quality, we have several qualification requirements per participant: submitting a minimum of $5000$ tasks on AMT and an AMT acceptance rate of $95\%$, as well as having a US residency and passing a language qualification test to ensure similar language skills and cultures across raters.

Appendix B Mapping Shape Representations

An important part of our project is the computation of body measurements. Following “Virtual Caliper” [pujades2019virtual], we present a method to compute anatomical measurements from a 3D mesh in the canonical T-pose, i.e. after “undoing” the effect of pose. Specifically, we measure the height, $0pt(\bm{\beta})$, weight, $W(\bm{\beta})$, and the chest, waist and hip circumferences, $C_{\text{c}}(\bm{\beta})$, $C_{\text{w}}(\bm{\beta})$, and $C_{\text{h}}(\bm{\beta})$, respectively. Let $v_{\text{head}}(\bm{\beta}),v_{\text{left heel}}(\bm{\beta}),v_{\text{chest}}(\bm{\beta}),v_{\text{waist}}(\bm{\beta}),v_{\text{hip}}(\bm{\beta})$ be the head, left heel, chest, waist and hip vertices. $0pt(\bm{\beta})$ is computed as the difference in the vertical-axis “Y” coordinates between the top of the head and the left heel: $0pt(\bm{\beta})=\lvert v_{\text{head}}^{y}(\bm{\beta})-v_{\text{left heel}}^{y}(\bm{\beta})\rvert$ . To obtain $W(\bm{\beta})$ we multiply the mesh volume by 985 kg/m3, which is the average human body density. We compute circumference measurements using the method of Wuhrer et al. [wuhrer2013estimating].

where $i,j$ are point indices for line segments of $\mathcal{E}$. The process is the same for the waist and hips, but the intersection plane is computed using $v_{\text{waist}},v_{\text{hip}}$. All of $0pt(\bm{\beta}),W(\bm{\beta}),C_{\text{c}}(\bm{\beta}),C_{\text{w}}(\bm{\beta}),C_{\text{h}}(\bm{\beta})$ are differentiable functions of body shape parameters, $\bm{\beta}$.

Note that SMPL-X knows the height distribution of humans and acts as a strong prior in shape estimation. Given the ground-truth height of a person (in meter), $0pt(\bm{\beta})$ can be used to directly supervise height and overcome scale ambiguity.

B.2 Mapping Attributes to Shape (A2S)

We introduce A2S, a model that maps the input attribute ratings to shape components $\bm{\beta}$ as output. We compare a $2^{\text{nd}}$ degree polynomial model with a linear regression model and a multi-layer perceptron (MLP), using the Vertex-to-Vertex (V2V) error metric between predicted and ground-truth SMPL-X meshes, and report results in Tab. A.1. When using only attributes as input (A2S), the polynomial model of degree $d=2$ achieves the best performance. Adding height and weight to the input vector requires a small modification, namely using the cubic root of the weight and converting the height from (m) to (cm). We. With these additions, the $2^{\text{nd}}$ degree polynomial achieves the best performance.

B.3 Images to Attributes (I2A)

We briefly experimented with models that learn to predict attribute scores from images (I2A). This attribute predictor is implemented using a ResNet50 for feature extraction from the input images, followed by one MLP per gender for attribute score prediction. To quantify the model’s performance, we use the attribute classification metric described in the main paper. I2A achieves $60.7$ / $69.3\%$ (fe-/male) of correctly predicted attributes, while our S2A achieves $68.8$ / $76\%$ on CAESAR. Our explanation for this result is that it is hard for the I2A model to learn to correctly predict attributes independent of subject pose. Our approach works better, because it decomposes 3D human estimation into predicting pose and shape. Networks are good at estimating pose even without GT shape [li2021hybrik]. “SHAPY ’s losses” affect only the shape branch. To minimize these losses, the network has to learn to correctly predict shape irrespective of pose variations.

Appendix C SHAPY- 3D Shape Regression from Images

Implementation details:: To train SHAPY, each batch of training images contains $50\%$ images collected from model agency websites and $50\%$ images from ExPose’s [Choutas2020_expose] training set. Note that the overall number of images of males and females in our collected model data differs significantly; images of female models are many more. Therefore, we randomly sample a subset of female images so that, eventually, we get an equal number of male and female images. We also use the BMI of each subject, when available, as a sampling weight for images. In this way, subjects with higher BMI are selected more often, due to their smaller number, to avoid biasing the model towards the average BMI of the dataset. Our pipeline is implemented in PyTorch [pytorch] and we use the Adam [adam] optimizer with a learning rate of $1e-4$. We tune the weights of each loss term with grid search on the MMTS and HBW validation sets. Using a batch size of $48$, SHAPY achieves the best performance on the HBW validation set after 80k steps.

Appendix D Experiments

Given two meshes $M_{1}$ and $M_{2}$ of topology $T_{1}$ and $T_{2}$ we obtain the mesh surface points $P_{1}=\mathbf{H}_{T_{1}}U_{1}$ and $P_{2}=\mathbf{H}_{T_{2}}U_{2}$, where $U_{1}$ and $U_{2}$ denote the vertices of the shaped zero posed (t-pose) meshes. To compute the $\text{P2P}_{20\text{K}}$ error we correct for translation $t=\bar{P}_{2}-\bar{P}_{1}$ and define

D.2 Shape Estimation

A2S and its variations: For completeness, Table A.5 shows the results of the female A2S models in addition to the male ones. The male results are also presented in the main manuscript. Note that attributes improve shape reconstruction across the board. For example, in terms of $\text{P2P}_{20\text{K}}$, AH2S is better than just H2S, AHW2S is better than just HW2S. It should be emphasized that even when many measurements are used as input features, i.e. height, weight, and chest/waist/hip circumference, adding attributes still improves the shape estimate, e.g. HWC2S vs. AHWC2S.

Attribute/Measurement ablation: To investigate the extent to which attributes can replace ground truth measurements in network training, we train SHAPY’s variations in a leave-one-out manner: SHAPY-H uses only height and SHAPY-C only hip/waist/chest circumference. We compare these models with SHAPY-AH and SHAPY-AC, which use attributes in addition to height and circumference measurements, respectively. For completeness, we also evaluate SHAPY-HC and SHAPY-AHC, which use all measurements; the latter also uses attributes. The results are reported in Tab. A.2 (MMTS) and Tab. A.3 (HBW). The tables show that attributes are an adequate replacement for measurements. For example, in Tab. A.2, the height (SHAPY-C vs. SHAPY-CA) and circumference errors (SHAPY-H vs. SHAPY-AH) are reduced significantly when attributes are taken into account. On HBW, the $\text{P2P}_{20\text{K}}$ errors are equal or lower, when attribute information is used, see Tab. A.3. Surprisingly, seeing attributes improves the height error in all three variations. This suggests that training on model images introduces a bias that A2S antagonizes.

S2A: Table A.4 shows the results of S2A in detail. All attributes are classified correctly with an accuracy of at least $58.05\%$ (females) and $68.14\%$ (males). The probability of randomly guessing the correct class is 20%.

AHWC and AHWC2S noise: To evaluate AHWC’s robustness to noise in the input, we fit AHWC using the per-rater scores instead of the average score. The $\text{P2P}_{20\text{K}}$ $\downarrow$ error only increases by $1.0$ mm to $6.8$ when using the per-rater scores.

D.3 Pose evaluation

3D Poses in the Wild (3DPW) [vonMarcard2018]: This dataset is mainly useful for evaluating body pose accuracy since it contains few subjects and limited body shape variation. The test set contains a limited set of $5$ subjects in indoor/outdoor videos with everyday clothing. All subjects were scanned to obtain their ground-truth body shape. The body poses are pseudo ground-truth SMPL fits, recovered from images and IMUs. We convert pose and shape to SMPL-X for evaluation.

We evaluate SHAPY on 3DPW to report pose estimation accuracy (Tab. A.6). SHAPY’s pose accuracy is slightly behind ExPose which also uses SMPL-X. SHAPY’s performance is better than HMR [Kanazawa2018_hmr] and STRAPS [sengupta2020straps]. However, SHAPY does not outperform recent pose estimation methods, e.g. HybrIK [li2021hybrik]. We assume that SHAPY’s pose estimation accuracy on 3DPW can be improved by (1) adding data from the 3DPW training set (similar to Sengupta et al. [sengupta2021hierarchicalICCV] who sample poses from 3DPW training set) and (2) creating pseudo ground-truth fits for the model data.

D.4 Qualitative Results

We show additional qualitative results in Fig. A.5 and Fig. A.7. Failure cases are shown in Fig. A.8. To deal with high-BMI bodies, we need to expand the set of training images and add additional shape attributes that are descriptive for high-BMI shapes. Muscle definition on highly muscular bodies is not well represented by SMPL-X, nor do our attributes capture this. The SHAPY approach, however, could be used to capture this with a suitable body model and more appropriate attributes.