GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB

Introduction

Estimating the 3D pose of the hand is a long-standing goal in computer vision with many applications such as in virtual/augmented reality (VR/AR) and human–computer interaction . While there is a large body of existing works that consider marker-free image-based hand tracking or pose estimation, many of them require depth cameras or multi-view setups . However, in many applications these requirements are unfavorable since such hardware is less ubiquitous, more expensive, and does not work in all scenes.

In contrast, we address these issues and propose a new algorithm for real-time skeletal 3D hand tracking with a single color camera that is robust under object occlusion and clutter. Recent developments that consider RGB-only markerless hand tracking problem come with clear limitations. For example, the approach by Simon et al. achieves the estimation of 3D joint locations within a multi-view setup; however in the monocular setting only 2D joint locations are estimated. Similarly, the method by Gomez-Donoso et al. is also limited to 2D. Recently, Zimmermann and Brox presented a 3D hand pose estimation method from monocular RGB which, however, only obtains relative 3D positions and struggles with occlusions.

Inspired by recent work on hand and body tracking , we combine CNN-based 2D and 3D hand joint predictions with a kinematic fitting step to track hands in global 3D from monocular RGB. The major issue of such (supervised) learning-based approaches is the requirement of suitable annotated training data. While it has been shown to be feasible to manually annotate 2D joint locations in single-view RGB images , it is impossible to accurately annotate in 3D due to the inherent depth ambiguities. One way to overcome this issue is to leverage existing multi-camera methods for tracking hand motion in 3D . However, the resulting annotations would lack precision due to inevitable tracking errors. Some works render synthetic hands for which the perfect ground truth is known . However, CNNs trained on synthetic data may not always generalize well to real-world images. Hence, we propose a method to generate suitable training data by performing image-to-image translation between synthetic and real images. We impose two strong requirements on this method. First, we want to be able to train on unpaired images so that we can easily collect a large-scale real hands dataset. Second, we need the algorithm to preserve the pose of the hand such that the annotations of the synthetic images are still valid for the translated images. To this end, we leverage the seminal work on CycleGANs , which successfully learns various image-to-image translation tasks with unpaired examples. We extend it with a geometric consistency loss which improves the results in scenarios where we only want to learn spatially localized (e.g. only the hand part) image-to-image conversions, producing pose-preserving results with less texture bleeding and sharper contours. Once this network is trained, we can use it to translate any synthetically generated image into a “real” image while preserving the perfect (and inexpensive) ground truth annotation. Throughout the rest of the paper we denote images as “real” (in quotes), or GANerated, when we refer to synthetic images after they have been processed by our translation network such that they follow the same statistical distribution as real-world images.

Finally, using annotated RGB-only images produced by the proposed GAN, we train a CNN that jointly regresses image-space 2D and root-relative 3D hand joint positions. While the skeletal hand model in combination with the 2D predictions are sufficient to estimate the global translation of the hand, the relative 3D positions resolve the inherent ambiguities in global rotation and articulation which occur in the 2D joint positions. In summary, our main contributions are:

The first real-time hand tracking system that tracks global 3D joint positions from unconstrained monocular RGB-only images.

A novel geometrically consistent GAN that performs image-to-image translation while preserving poses during translation.

Based on this network, we are able to enhance synthetic hand image datasets such that the statistical distribution resembles real-world hand images.

A new RGB dataset with annotated 3D hand joint positions. We overcome existing datasets in terms of size (${>}260$k frames), image fidelity, and annotation precision.

Related Work

Our goal is to track hand pose from unconstrained monocular RGB video streams at real-time framerates. This is a challenging problem due the large pose space, occlusions due to objects, depth ambiguity, appearance variation due to lighting and skin tone, and camera viewpoint variation. While glove-based solutions would address some of these challenges , they are cumbersome to wear. Thus, in the following we restrict our discussion to markerless camera-based methods that try to tackle these challenges.

Multi-view methods: The use of multiple RGB cameras considerably alleviates occlusions during hand motion and interaction. Wang et al. demonstrated hand tracking with two cameras using a discriminative approach to quickly find the closest pose in a database. Oikonomidis et al. showed tracking of both the hand and a manipulated object using 8 calibrated cameras in a studio setup. Ballan et al. also used 8 synchronized RGB cameras to estimate pose with added input from discriminatively detected points on the fingers. Sridhar et al. used 5 RGB cameras and an additional depth sensor to demonstrate real-time hand pose estimation. Panteleris and Argyros propose using a short-baseline stereo camera for hand pose estimation without the need for a disparity map. All of the above approaches utilize multiple calibrated cameras, making it hard to setup and operate on general hand motions in unconstrained scenes (e.g. community videos). More recently, Simon et al. proposed a method to generate large amounts of 2D and 3D hand pose data by using a panoptic camera setup which restricts natural motion and appearance variation. They also leverage their data (which is not yet publicly available) for 2D hand pose estimation but cannot estimate 3D pose in monocular RGB videos. Our contributions address both data variation for general scenes and the difficult 3D pose estimation problem.

Monocular methods: Monocular methods for 3D hand pose estimation are preferable because they can be used for many applications without a setup overhead. The availability of inexpensive consumer depth sensors has lead to extensive research in using them for hand pose estimation. Hamer et al. proposed one of the first generative methods to use monocular RGB-D data for hand tracking, even under partial occlusions. As such methods often suffer from issues due to local optima, a learning-based discriminative method was proposed by Keskin et al. . Numerous follow-up works have been proposed to improve the generative component , and the learning-based discriminator . Hybrid methods that combine the best of both generative and discriminative methods show the best performance on benchmark datasets .

Despite all the above-mentioned progress in monocular RGB-D or depth-based hand pose estimation, it is important to notice that these devices do not work in all scenes, e.g. outdoors due to interference with sunlight, and have higher power consumption. Furthermore, 3D hand pose estimation in unconstrained RGB videos would enable us to work community videos as shown in Figure 1. Some of the first methods for this problem did not produce metrically accurate 3D pose as they only fetched the nearest 3D neighbor for a given input or assume that the $z$-coordinate is fixed. Zimmermann and Brox proposed a learning-based method to address this problem. However, their 3D joint predictions are relative to a canonical frame, i.e. the absolute coordinates are unknown, and it is not robust to occlusions by objects. Furthermore, their method is not able to distinguish 3D poses with the same 2D joint position projection since their 3D predictions are merely based on the abstract 2D heatmaps and do not directly take the image into account. In contrast, our work addresses these limitations by jointly learning 2D and 3D joint positions from image evidence, so that we are able to correctly estimate poses with ambiguous 2D joint positions. In addition, our skeleton fitting framework combines a prior hand model with these predictions to obtain global 3D coordinates.

Training of learning-based methods: One of the challenges in using learning-based models for hand pose estimation is the difficulty of obtaining annotated data with sufficient real-world variations. For depth-based hand pose estimation, multiple training datasets have been proposed that leverage generative model fitting to obtain ground truth annotations or to sample pose space better . A multi-view bootstrapping approach was proposed by Simon et al. . However, such outside-in capture setups could still suffer from occlusions due to objects being manipulated by the hand. Synthetic data is promising for obtaining perfect ground truth but there exists a domain gap when models training on this data are applied to real input .

Techniques like domain adaptation aim to bridge the gap between real and synthetic data by learning features that are invariant to the underlying differences. Other techniques use real–synthetic image pairs to train networks that can generate images that contain many features of real images. Because it is hard to obtain real–synthetic image pairs for hands, we build upon the unpaired image-to-image translation work of Zhu et al. . Without the need for corresponding real–synthetic image pairs, we can generate images of hands that contain many of the features found in real datasets.

Hand Tracking System

The main goal of this paper is to present a real-time system for monocular RGB-only hand tracking in 3D. The overall system is outlined in Fig. 2. Given a live monocular RGB-only video stream we use a CNN hand joint regressor, the RegNet, to predict 2D joint heatmaps and 3D joint positions (Sec. 3.2). The RegNet is trained with images that are generated by a novel image-to-image translation network, the GeoConGAN, (Sec. 3) that enriches synthetic hand images. The output images of the GeoConGAN–the GANerated images–are better suited to train a CNN that will work on real imagery. After joint regression, we fit a kinematic skeleton to both the 2D and 3D predictions by minimizing our fitting energy (Sec. 3.3), which has several key advantages for achieving a robust 3D hand pose tracking: it enforces biomechanical plausibility; we can retrieve the absolute 3D positions; and furthermore we are able to impose temporal stability across multiple frames.

Since the annotation of 3D joint positions in hundreds of real hand images is infeasible, a common approach is to use synthetically generated images. While the main advantage of synthetic images is that the ground truth 3D joint positions are known, an important shortcoming is that they usually lack realism. Such discrepancy between real and synthetic images limits the generalization ability of a CNN trained only on the latter. In order to account for this disparity, we propose to use an image-to-image translation network, the GeoConGAN, with the objective to translate synthetic images to real images. Most importantly, to train this network we use unpaired real and synthetic images, as will be described in the following. Note that for both the real and the synthetic data we use only foreground-segmented images that contain a hand on white background, which facilitates background augmentation for learning the joint position regressor, as described in Sec. 3.2.

Real hand image acquisition: To acquire our dataset of real images we used a green-screen setup to capture hand images with varying poses and camera extrinsics from 7 different subjects with different skin tones and hand shapes. In total, we captured 28,903 real hand images using a desktop webcam with image resolution $640\times 480$.

Synthetic hand image generation: Our synthetic hand image dataset is a combination of the SynthHands dataset that contains hand images from an egocentric viewpoint, with our own renderings of hand images from various third-person viewpoints. In order to generate the latter, we used the standard strategy in state-of-the-art datasets , where the hand motion, obtained either via a hand tracker or a hand animation platform, is re-targeted to a kinematic 3D hand model.

Geometrically consistent CycleGAN (GeoConGAN): While the above procedure allows to generate a large amount of synthetic training images with diverse hand pose configurations, training a hand joint regression network based on synthetic images alone has the strong disadvantage that the so-trained network has limited generalization to real images, as we will demonstrate in Sec. 4.1

To tackle this problem, we train a network that translates synthetic images to “real” (or GANerated) images. Our translation network is based on CycleGAN , which uses adversarial discriminators to simultaneously learn cycle-consistent forward and backward mappings. Cycle-consistency means that the composition of both mappings (in either direction) is the identity mapping. In our case we learn mappings from synthetic to real images (synth2real), and from real to synthetic images (real2synth). In contrast to many existing image-to-image or style transfer networks , CycleGAN has the advantage that it does not require paired images, i.e. there must not exist a real image counterpart for a given synthetic image, which is crucial for our purpose due to the unavailability of such pairs.

The architecture of this GeoConGAN is illustrated in Fig. 3. The input to this network are (cropped) synthetic and real images of the hand on a white background in conjunction with their respective silhouettes, i.e. foreground segmentation masks. In its core, The GeoConGAN resembles CycleGAN with its discriminator and cycle-consistency loss, as well as the two trainable translators synth2real and real2synth. However, unlike CycleGAN, we incorporate an additional geometric consistency loss (based on cross-entropy) that ensures that the real2synth and synth2real components produce images that maintain the hand pose during image translation. Enforcing consistent hand poses is of utmost importance in order to ensure that the ground truth joint locations of the synthetic images are also valid for the “real” images produced by synth2real. Fig. 4 shows the benefits of adding this new loss term.

In order to extract the silhouettes of the images that are produced by both real2synth and synth2real (blue boxes in Fig. 3), we train a binary classification network based on a simple UNet that has three 2-strided convolutions and three deconvolutions, which we call SilNet. Note that this is a relatively easy task as the images show hands on a white background. Our SilNet is trained beforehand and it is kept fixed while training synth2real and real2synth. Training details can be found in the supplementary document.

Data augmentation: Once the GeoConGAN is trained, we feed all synthetically generated images into the synth2real component and obtain the set of “real” images that have associated ground truth 3D joint locations. By using the background masks from the original synthetic images, we perform background augmentation by compositing GANerated images (foreground) with random images (background) . Similarly, we also perform an object augmentation by leveraging the object masks produced when rendering the synthetic sequences . Fig. 5 shows some GANerated images.

2 Hand Joints Regression

In order to regress the hand pose from a (cropped) RGB image of the hand, we train a CNN, the RegNet, that predicts 2D and 3D positions of $21$ hand joints. The 2D joint positions are represented as heatmaps in image space, and the 3D positions are represented as 3D coordinates relative to the root joint. We have found that regressing both 2D and 3D joints are complementary to each other, as the 2D heatmaps are able to represent uncertainties, whereas the 3D positions resolve the depth ambiguities.

The RegNet, shown in Fig. 6, is based on a residual network consisting of 10 residual blocks that is derived from the ResNet50 architecture , as done in . Additionally, we incorporate a (differentiable) refinement module based on a projection layer (ProjLayer) to better coalesce the 2D and 3D predictions. The idea of the ProjLayer is to perform an orthographic projection of (preliminary) intermediate 3D predictions, from which 2D Gaussian heatmaps are created (within the layer). These heatmaps are then leveraged in the remaining part of the network (conv) to obtain the final 2D and 3D predictions. In Fig. 7(a) we show that this leads to improved results.

The training is based on a mixture of GANerated (Sec. 3) and synthetic images, in conjunction with corresponding (normalized) 3D ground truth joint positions. The training set contains $\approx 440,000$ examples in total from which 60% are GANerated. We train the RegNet with relative 3D joint positions, which we compute by normalizing the absolute 3D ground truth joint positions such that the middle finger metacarpophalangeal (MCP) joint is at the origin and the distance between the wrist joint and the middle MCP joint is 1. Further details can be found in the supplementary document.

During test time, i.e. for hand tracking, the input to the RegNet is a cropped RGB image, where the (square) bounding box is derived from the 2D detections of the previous frame. In the first frame, the square bounding box is located at the center of the image, with size equal to the input image height. Also, we filter the output of RegNet with the 1€ filter to obtain temporally smoother predictions.

3 Kinematic Skeleton Fitting

After obtaining the 2D joint position predictions in the form of heatmaps in image space, and the 3D joint coordinates relative to the root joint, we fit a kinematic skeleton model to this data. This ensures that the resulting hand pose is anatomically plausible, while at the same time it allows to retrieve the absolute hand pose, as we will describe below. Moreover, when processing a sequence of images, i.e. performing hand tracking, we can additionally impose temporal smoothness.

For fitting the kinematic model to data, we minimize the energy

where the individual energy terms are described below.

2D Fitting Term: The purpose of the term $E_{\text{2D}}$ is to minimize the distance between the hand joint position projected onto the image plane and the heatmap maxima. It is given by

3D Fitting Term: The term $E_{\text{3D}}$ has the purpose to obtain a good hand articulation by using the predicted relative 3D joint positions. Moreover, this term resolves depth ambiguities that are present when using 2D joint positions only. We define $E_{\text{3D}}$ as

Joint Angle Constraints: The term $E_{\text{limits}}$ penalizes anatomically implausible hand articulations by enforcing that joints do not bend too far. Mathematically, we define

Temporal Smoothness: The term $E_{\text{temp}}$ enforces temporal smoothness for hand pose tracking by making sure that the current velocity of the change in $\Theta$ is close to the velocity in the previous time step. We formulate

where the gradients of the pose parameters $\Theta$ are determined using finite (backward) differences.

Optimization: In order to minimize the energy in (1) we use a gradient-descent strategy. For the first frame, $\theta$ and $\mathbf{t}$ are initialized to represent a neutral hand pose (an open hand) that is centered in the image and 45cm away from the camera plane. For the remaining frames we use the translation and articulation parameters $\mathbf{t}$ and $\theta$ from the previous frame as initialization. In our experiments we have found that global hand rotations are problematic for the kinematic model fitting as they may lead to unwanted local minima due to the non-convexity of energy (1). In order to deal with this problem, for the global rotation $\mathbf{R}$ we do not rely on the previous value $\mathbf{R}^{\text{prev}}$, but instead initialize it based on the relative 3D joint predictions. Specifically, we make use of the observation that in the human hand the root joint and its four direct children joints of the non-thumb fingers (the respective MCP joints) are (approximately) rigid (cf. Fig. 2, Skeleton Fitting block). Thus, to find the global rotation $\mathbf{R}$ we solve the problem

Experiments

We quantitatively and qualitatively evaluate our method and compare our results with other state-of-the-art methods on a variety of publicly available datasets. For that, we use the Percentage of Correct Keypoints (PCK) score, a popular criterion to evaluate pose estimation accuracy. PCK defines a candidate keypoint to be correct if it falls within a circle (2D) or sphere (3D) of given radius around the ground truth.

Ablative study: In Fig. 7(a) we compare the detection accuracy when training our hand joint regression network RegNet with different configurations of training data. Specifically, we compare the cases of using synthetic images only, synthetic images plus color augmentation, and synthetic images in combination with GANerated images, where for the latter we also considered the case of additionally using the ProjLayer in RegNet. While we evaluated the RegNet on the entire Stereo dataset comprising 12 sequences, we did not train on any frame of the Stereo dataset for this test. We show that training on purely synthetic data leads to poor accuracy (3D PCK@50mm $\approx 0.55$). While color augmentation on synthetic images improves the results, our GANerated images significantly outperform standard augmentation techniques, achieving a 3D PCK@50mm $\approx 0.80$. This test validates the argument for using GANerated images for training.

Comparison to state-of-the-art: Fig. 7(b) evaluates our detection accuracy on the Stereo dataset, and compares it to existing methods . We followed the same evaluation protocol used in , i.e. we train on 10 sequences and test it on the other 2. Our method outperforms all existing methods. Additionally, we test our approach without training on any sequence of the Stereo dataset, and demonstrate that we still outperform some of the existing works (green line in Fig. 7(b)). This demonstrates the generalization of our approach.

Figure 7(c) shows the 2D PCK, in pixels, on the Dexter+Object and EgoDexter datasets. We significantly outperform Zimmerman and Brox (Z&B) , which fails under difficult occlusions. Note that here we cannot report 3D PCK because only outputs root-relative 3D, and these datasets do not have root joint annotations.

2 Qualitative Evaluation

We qualitatively evaluate our method on three different video sources: publicly available datasets, real-time capture, and community (or vintage) video (i.e. YouTube).

Fig. 8 presents qualitative results on three datasets, Stereo , Dexter+Object and EgoDexter , for both Z&B and our method. We show that our method provides robust tracking of the hand even in severely occluded situations, and significantly outperforms in these cases. While we already outperformed Z&B in our quantitative evaluation (Fig. 7(c)), we emphasize that this is not the full picture, since the datasets from only provide annotations for visible finger tips due to the manual annotation process. Thus, the error due to occluded joints is not at all reflected in the quantitative analysis. Since our method is explicitly trained to deal with occlusion–in contrast to –our qualitative analysis in the supplementary video and in columns 3–6 of Fig. 8 highlights the superiority of our method in such scenarios.

We show real-time tracking results in Fig. 9 as well as in the supplementary video. This sequence was tracked live with a regular desktop webcam in a standard office environment. Note how our method accurately recovers the full 3D articulated pose of the hand. In Fig. 1 we demonstrate that our method is also compatible with community or vintage RGB-only video. In particular, we show 3D hand tracking in YouTube videos, which demonstrates the generalization of our method.

Limitations & Discussion

One difficult scenario for our method is when the background has similar appearance as the hands, as our RegNet struggles to obtain good predictions and thus tracking becomes unstable. This can potentially be addressed by using an explicit segmentation stage, similar to Zimmermann and Brox . Moreover, when two or more hands are close in the input image, detections may also become unreliable. While our approach can handle sufficiently separate hands —due to our bounding box tracker— tracking of interacting hands, or hands of multiple persons, is an interesting direction for follow-up work.

The 3D tracking of hands in purely 2D images is an extremely challenging problem. While our real-time method for 3D hand tracking outperforms state-of-the-art RGB-only methods, there is still an accuracy gap between our results and existing RGB-D methods (mean error of ${\approx}5\text{cm}$ for our proposed RGB approach vs. ${\approx}2\text{cm}$ for the RGB-D method of on their dataset Dexter+Object). Nevertheless, we believe that our method is an important step towards democratizing RGB-only 3D hand tracking.

Conclusion

Most existing works consider either the problem of 2D hand tracking from monocular RGB images, or they use additional inputs, such as depth images or multi-view RGB, to track the hand motions in 3D. While the recent method by Zimmermann and Brox considers monocular 3D hand tracking from RGB images, our proposed approach tackles the same problem but goes one step ahead with regards to several dimensions: we obtain the absolute 3D hand pose due to our model fitting procedure, our approach is more robust to occlusions, and our method generalizes better due to enrichment of our synthetic dataset such that it resembles the distribution of real hand images. Our experimental evaluation demonstrates these benefits as our method significantly outperforms the method by Zimmermann and Brox, particularly in difficult occlusion scenarios. In order to further encourage future work on monocular 3D RGB hand tracking we make our dataset available to the research community.

Acknowledgements: This work was supported by the ERC Starting Grant CapReal (335545). Dan Casas was supported by a Marie Curie Individual Fellow, grant 707326.

In this appendix we provide details of the RegNet and GeoConGAN networks (Sec. A), additional quantitative evaluations (Sec. B), as well as detailed visualizations of our CNN RegNet output and final results (Sec. C)

Appendix A CNN and GAN Details

Network Design: The architecture of GeoConGAN is based on the CycleGAN , i.e. we train two conditional generator and two discriminator networks for synthetic and real images, respectively. Recently, also methods using only one generator and discriminator for enrichment of synthetic images from unpaired data have been proposed. Shrivastava et al. and Liu et al. both employ an L1 loss between the conditional synthetic input and the generated output (in addition to the common discriminator loss) due to the lack of image pairs. This loss forces the generated image to be similar to the synthetic image in all aspects, i.e. it might hinder the generator in producing realistic outputs if the synthetic data is not already close. Instead, we decided to use the combination of cycle-consistency and geometric consistency loss to enable the generator networks to move farther from the synthetic data thus approaching the distribution of real world data more closely while preserving the pose of the hand. Our GeoConGAN contains ResNet generator and Least Squares PatchGAN discriminator networks.

Training Details: We train GeoConGAN in Tensorflow for 20,000 iterations with a batch size of 8. We initialize the Adam optimizer with a learning rate of $0.0002$, $\beta_{1}=0.5$, and $\beta_{2}=0.999$.

A.2 RegNet network

Projection Layer ProjLayer: Recent work in 3D body pose estimation has integrated projection layers to leverage 2D-only annotated data for training 3D pose prediction . Since our training dataset provides perfect 3D ground truth, we employ our projection layer merely as refinement module to link the 2D and 3D predictions. We project the intermediate relative 3D joint position prediction using orthographic projection where the origin of the 3D predictions (the middle MCP joint) projects onto the center of the rendered heatmap. Hence, our rendered heatmaps are also relative and not necessarily in pixel-correspondence with the ground truth 2D heatmaps. Therefore, we apply further processing to the rendered heatmaps before feeding them back into the main network branch. Note that the rendered heatmaps are differentiable with respect to the 3D predictions which makes backpropagation of gradients through our ProjLayer possible.

Training Details: We train RegNet in the Caffe framework for 300,000 iterations with a batch size of 32. We use the AdaDelta solver with an initial learning rate of $0.1$ which is lowered to $0.01$ after 150,000 iterations. All layers which are shared between our network and ResNet50 are initialized with the weights obtained from ImageNet pretraining . Both, the 2D heatmap loss and the local 3D joint position loss, are formulated using the Euclidean loss with loss weights of 1 and 100, respectively.

Computational Time: A forward pass of RegNet in our real-time tracking system takes 13 ms on a GTX 1080 Ti.

Appendix B Comparison with RGB-D methods

The 3D tracking of hands in purely RGB images is an extremely challenging problem due to inherent depth ambiguities of monocular RGB images. While our method advances the state-of-the-art of RGB-only hand tracking methods, there is still a gap between RGB-only and RGB-D methods . A quantitative analysis of this accuracy gap is shown in Fig. 10, where we compare our results (dark blue) with the RGB-D method from Sridhar et al. (red).

In order to better understand the source of errors, we perform an additional experiment where we translated the global z-position of our RGB-only results to best match the depth of the ground truth. In Fig. 10 we compare these depth-normalized results (light blue) with our original results (blue). It can be seen that a significant portion of the gap between methods based on RGB and RGB-D is due to inaccuracies in the estimation of the hand root position. Reasons for an inaccurate hand root position include a skeleton that does not perfectly fit the user’s hand (in terms of bone lengths), as well as inaccuracies in the 2D predictions.

Appendix C Detailed Qualitative Evaluation

In Figs. 11 and 12 we qualitatively evaluate each of the intermediate stages along our tracking solution as well as the final result. In particular, Fig. 11 shows results on the EgoDexter dataset where a subject grabs different objects in an office environment, and Fig. 12 shows results on community videos downloaded from YouTube. In both figures, we provide visualizations of: heatmap maxima of the 2D joint detections (first row); root-relative 3D joint detections (second row); global 3D tracked hand projected into camera plane (third row); and global 3D tracked hand visualized in a virtual scenario with the original camera frustum (fourth and fifth rows). Please see the supplementary video for complete sequences.

References