SurfelWarp: Efficient Non-Volumetric Single View Dynamic Reconstruction

I Introduction

The wide availability of commodity depth cameras provides robots with powerful but low-cost 3D sensing capabilities. However, measurements from depth sensors are noisy and incomplete, and they often contain numerous outliers. To address this issue, KinectFusion and many related approaches estimate the camera pose and fuse depth frames online for a complete, smoothed 3D geometry scanning. In contrast to rigid scenes where the motion is encoded by a single 6 degree of freedom (DOF) camera pose, researchers also focus on the tracking and reconstruction of dynamic scenes , which is more challenging but critically important to enable robots to interact with non-static working environments.

The problem of non-rigid reconstruction has been widely studied in recent years. To deal with the extraordinarily large deformation space, researchers have exploited carefully designed capture environments , well-controlled lighting conditions , and the use of many cameras for multi-view observations . Some approaches also rely on motion priors such as templates or embedded skeletons . These approaches can achieve high quality and visually pleasing tracking and reconstruction results. However, special purpose capturing environments are not easy to setup and calibrate, and pre-scanned templates or skeletons require problem-specific initial alignments. These restrictions limit the applicability of these methods to typical robot tasks.

The work of DynamicFusion represents a substantial step forward, which uses only a depth camera to acquire a live stream of depth images and fuse multiple frames by online non-rigid registration techniques. Later works extend this pipeline for improved robustness and capabilities: Innmann et al. used sparse SIFT features and fine-scale volumetric deformation field to improve the non-rigid motion estimation; Guo et al. exploited RGB frames to estimate the surface albedo and low-frequency lighting; Slavcheva et al. proposed to directly align TSDF fields for motion estimation; Dou et al. and Dou et al. extended the framework to multi-view setups, used combined canonical and live TSDF volumes for better robustness, and achieved the performance capture of challenging scenes.

Despite diversities among these contributions, they all rely on volumetric data structures as the underlying representation of geometry (TSDF fields) and motion (nearest neighbor field for and deformation field for ). As pointed out by Keller et al. , there is a quality and efficiency trade-off between volumetric and surfel based reconstructions: volumetric methods generate a smooth triangle mesh, but they are performance and memory intensive; surfel based methods are more efficient, but they need post-processing if mesh model is required. Compared with rigid SLAM, the computational burden is much more prominent for non-rigid scenes, which indicates surfel based representation is a promising alternative for online dynamic reconstructions.

In this paper, we propose a novel surfel based approach for real-time reconstruction of dynamic scenes, which requires no prior models or templates. As we will present, standard graphics pipelines and GPGPU computing can be leveraged for efficient implementation of all central operations: i.e., nearest-neighbor maintenance, correspondence association, non-rigid deformation estimation and fusion of depth measurements. The elimination of volumetric data structures avoids expensive volumetric operations such as marching cubes, volumetric fusion and dense deformation field updates, which leads to significantly improved performance. Moreover, the explicit surfel representation enables direct recovery from tracking failures and topology changes, which further improves the robustness of our pipeline.

II Related Works

Dynamic scene reconstruction typically involves the estimation of both the geometry and deformation field. Compared with high-quality offline reconstruction algorithms such as Collet et al. , online reconstruction methods are more suitable for robotic applications. Zollhöfer et al. proposed to first perform a live static scanning, yielding a template for later online tracking. Newcombe et al. combined the motion and geometry estimation into a unified framework, achieved the online reconstruction of dynamic scenes from a single depth camera. Many later works improve the pipeline of DynamicFusion with additional capabilities, as reviewed in Sec. I.

Compared with these contributions, the key distinction of our work is a highly efficient, non-volumetric pipeline for the non-rigid fusion of depth observations and the maintenance of deformation field. As we will demonstrate in the following text, the elimination of volumetric operations leads to better performance, less memory usage, and improved robustness.

II-B Non-Rigid Tracking

Non-rigid tracking approaches typically used predefined geometry models, instead of estimating it online. For instance, Li et al. and Cagniart et al. exploited pre-scanned meshes as shape templates, and estimated deformation fields that align templates with incoming observations. It is also observed that many captured objects, for instance human bodies, hands and robots, contain articulate structures. Thus, incorporating skeletons into geometry templates can reduce the non-rigid deformation to joint space, results in significantly improved performance and robustness .

Although non-rigid tracking pipelines with geometry templates or prior motion models have demonstrated impressive performance, they usually require either controlled capturing environments or careful initial alignments. These restrictions limit their applicability.

II-C Surfel-Based 3D Reconstruction

In Pfister et al. , a surfel is formally defined as a zero-dimensional n-tuple with shape and shade attributes that locally approximate an object surface. In Keller et al. , a surfel is associated with a 3D position, a normal orientation and a radius. Whelan et al. also associates surfels with albedo information. The surfel-based online rigid SLAM system was first introduced in Keller et al. . Whelan et al. further improved this technique by surface albedo reconstruction, light-source estimation and online loop closure.

Conceptually, our contribution can be regarded as an extension of Keller et al. which enables the tracking and reconstruction of dynamic scenes. To achieve this goal, we redesign the pipeline of Keller et al. for data fusion and deformation estimation of dynamic objects.

III Preliminaries

In this section, we present symbols, definitions and other notations used in this paper. A consumer depth camera such as Kinect or Xtion is used to record a depth image sequence $\{\mathcal{D}^{t}\}$, where $t$ is the frame label. A vertex map $\mathcal{V}$ is an image where each pixel records a 3-dimensional (3D) position; similarly, a normal map $\mathcal{N}$ is an image where each pixel records a normal orientation.

We follow DynamicFusion to represent the deformation field $\mathcal{W}$ as a node graph $\mathcal{G}$ , which is sparse but efficient compared to the volumetric deformation field . More specifically, $\mathcal{W}=\{[p_{j}\in R^{3},\sigma_{j}\in R^{+},T_{j}\in SE(3)]\}$, where $j$ is the node index, $p_{j}$ is the position of the $j$th node, $\sigma_{j}$ is a radius parameter, and $T_{j}$ is the 6 DOF transformation defined on $j$th node. $T_{j}$ is represented by DualQuaternion $\hat{q}_{j}$ for better interpolation. For a 3D point $x$, the deformation by $\mathcal{W}$ at $x$ can be interpolated as

where $N(x)$ is the nearest neighbor set of $x$, and the weight $w_{k}(x)$ can be computed as $w_{k}(x)=\text{exp}(-||x-p_{k}||_{2}^{2}/(2\sigma_{k}^{2}))$.

A surfel $s$ is a tuple composed of position $v\in R^{3}$, normal $n\in R^{3}$, radius $r\in R^{+}$, confidence $c\in R$, the initialization time $t_{\text{init}}\in N$ and the most recent observed time $t_{\text{observed}}\in N$. Upper case $S$ is used to denote the array of surfels and ${S}[i]$ is the $i$th surfel in this array. A surfel can be deformed by the deformation field $\mathcal{W}$ by Equ. 1,

where $v_{\text{live}}$ and $n_{\text{live}}$ are the deformed vertex position and normal, $v_{\text{ref}}$ and $n_{\text{ref}}$ are the vertex position and normal before deformation. Other attributes, such as radius, time and confidence, remain the same after deformation.

IV Overview

As shown in Fig. 1, our system runs in a frame-by-frame manner to process an input depth stream. Upon receiving a new depth image, we first solve a deformation field that aligns the reference geometry with the current depth observation. This is performed by first initializing the deformation field from previous frame, followed by a iterative closest point (ICP) based optimization similar to Newcombe et al. . Once the deformation field has been updated, we perform data fusion to accumulate current depth observations into a global geometry model.

Unlike previous approaches, we adopt an efficient surfel-based representation of geometry and deformation field throughout our approach. Our system maintains two arrays of surfels, one in the reference frame $S_{\text{ref}}$ and the other in the live frame $S_{\text{live}}$. The warp field $\mathcal{W}$ is defined in the reference frame, i.e., nodes $p_{j}$ in the warp field $\mathcal{W}$ are sub-sampled from $S_{\text{ref}}$ and $\mathcal{W}$ warps $S_{\text{ref}}[i]$ to $S_{\text{live}}[i]$. The nearest neighbor set $N(s_{i})$ and associated weights $\{w(s_{i})\}$ are maintained and shared for $S_{\text{ref}}[i]$ and $S_{\text{live}}[i]$. An important observation is: given appropriately maintained nearest neighbor sets and weights, we can define inverse warping as

where the SE(3) deformation $\mathcal{W}(v_{\text{ref}})$ can be computed by querying $N(s_{\text{ref}})$ and $\{w(s_{\text{ref}})\}$. Thus, unlike previous volumetric approaches that perform the geometry update in the reference frame, our pipeline updates geometry models in the live frame and warps them back to the reference frame.

$S_{\text{ref}}[i]$ shares radius $r$, confidence $c$, time stamps $t_{\text{init}}$ and $t_{\text{observed}}$ with $S_{\text{live}}[i]$, as they are invariant during warping.

V Depth Image Processing

Let $\mathbf{u}=(x,y)^{T}\in R^{2}$ denote the pixel coordinate. At each pixel $\mathbf{u}$, we will compute a surfel $s_{\text{depth}}$ defined in Sec. III. We first transform the depth value $\mathcal{D}(\mathbf{u})$ to the position $\mathcal{V}(\mathbf{u})$ of the surfel $s_{\text{depth}}$ by $\mathcal{V}(\mathbf{u})=\mathcal{D}(\mathbf{u}){K^{-1}}(\mathbf{u}^{T},1)^{T}$, where ${K}$ is the intrinsic matrix of the depth camera. Then the normal at this pixel $\mathcal{N}(\mathbf{u})$ is estimated from the vertex map $\mathcal{V}$ using central difference. The confidence of this surfel $s_{\text{depth}}$ is computed as $c=\text{exp}(-\gamma^{2}/(2\phi^{2}))$, where $\gamma$ is the normalized radial distance of the current depth measurement from the camera center, and $\phi=0.6$ in accordance with Keller et al. . The time stamps $t_{\text{init}}$ and $t_{\text{observed}}$ are initialized to be current frame $t$. Similar to Whelan et al. , the radius $r$ of the surfel $s_{\text{depth}}$ is computed as

where $d=\mathcal{D}(\mathbf{u})$ is the depth value, $f$ is the focal length of the camera, $n_{z}$ is the $z$ component of the normal expressed in camera frame. To prevent arbitrarily large surfels from oblique views, we follow Keller et al. to clamp radii for grazing observations exceeding $75^{\text{o}}$.

VI Depth Map Fusion and Warp Field Update

Assuming an already solved deformation field (the solving method will be described in Sec. VII), our depth map fusion and warp field update pipeline first performs data fusion in the live frame, then warps the live surfels back to the reference frame, after which the warp field is updated base on newly observed reference surfels, as shown in Fig. 1.

After solving the deformation field, we first perform a forward warp on reference surfel array $S_{\text{ref}}$ to obtain the live surfel array $S_{\text{live}}$ that is aligned with the depth observation. Then, methods similar to Keller et al. are used to fuse the depth map with live surfels. Specifically, we render the live surfel array $S_{\text{live}}$ into a Index Map $\mathcal{I}$: given camera intrinsic ${K}$ and the estimated camera pose $T_{\text{camera}}$, each live surfel $S_{\text{live}}[k]$ is projected into the image plane of current camera view, where the respective point index $k$ is stored. Surfels are rendered as unsized points in index map $\mathcal{I}$, i.e., each surfel can be projected to at most one pixel. The index map is super-sampled over the depth map to avoid nearby surfels projecting to the same pixel. We use a $4\times 4$ super-sampled index map in our implementation.

For each pixel $\mathbf{u}$ in the depth map $\mathcal{D}$, we identify at most one live surfel $s_{\text{live}}$ that is in correspondence with the depth observation at $\mathbf{u}$ by a $4\times 4$ window search centered at $\mathbf{u}$ on index map $\mathcal{I}$ (assuming suitable coordinate transform from $\mathcal{D}$ to $\mathcal{I}$). The criterion to find the corresponded surfel are:

Discard surfels whose distance to the depth vertex are larger than $\delta_{\text{distance}}$.

Discard surfels that the normal is not aligned with the depth normal, i.e., $\text{dot}(n_{\text{depth}},n_{\text{surfel}})<\delta_{\text{normal}}$.

For remaining surfels, select the one with highest confidence.

If multiple such surfels exist, select the one which is closest to $s_{\text{depth}}$.

If a corresponded surfel $s_{\text{live}}$ is found, the depth surfel $s_{\text{depth}}$ at $\mathbf{u}$ is fused into $s_{\text{live}}$ by:

where $v$, $n$, $c$, $t$ and $r$ are vertex, normal, confidence, recent observed stamp and radius associated with surfels; the depth surfel $s_{\text{depth}}$ is computed as described in Sec. V.

VI-B Live Surfel Skinning

In Sec. VI-A, for any depth pixel $\mathbf{u}$, if no corresponding live surfel $s_{\text{live}}$ is found, then this depth surfel $s_{\text{depth}}$ would potentially be appended to the live surfel array $S_{\text{live}}$ to complete the geometry (The appending pipeline will be described in section. VI-D). As mentioned in Sec. IV, it is required to compute the nearest neighbor set $N(s_{\text{depth}})$ and weights for this surfel to warp $s_{\text{depth}}$ back to the reference frame. However, the node graph $\mathcal{G}$ is defined in the reference frame, computing the nearest neighbor set $N(s_{\text{depth}})$ in the reference frame before knowing the warp-back position of $s_{\text{depth}}$ is not feasible.

One way to resolve this paradox is to first warp the node graph $\mathcal{G}$ to the live frame $\hat{\mathcal{G}}$, then perform skinning at the live frame. However, at open-to-close topology changes, surfel $s_{\text{depth}}$ might be skinned to inconsistent nodes: in Fig. 6 the surfels on the hand might be skinned to nodes on the face when they come close. To solve this problem, we propose the following pipeline:

Compute an initial nearest neighbor set $N(s_{\text{depth}})$ for $s_{\text{depth}}$ using node graph $\hat{\mathcal{G}}$ at the live frame. Let $node_{\text{0\_live}}\in N(s_{\text{depth}})$ denote the node which is closest to $s_{\text{depth}}$ at the live frame.

For any $node_{\text{i\_live}}\in N(s_{\text{depth}}),i\neq 0$, if

then keep $node_{\text{i\_live}}$ in $N(s_{\text{depth}})$, else remove this node.

In Equ. 6, the threshold $\epsilon$ measures the extent of intrinsic deformation. The above criteria ensure the deformation of $s_{\text{depth}}$ is controlled by a consistent set of nodes, i.e., nodes in $N(s_{\text{depth}})$ are close in the reference frame and have smooth SE(3) deformation values.

VI-C Resolving Compressive Warp Field

Another problem that arises from open-to-close topology changes is the compressive warp field, or colliding voxels in volumetric approaches . A one dimensional illustration is presented in Fig. 2. Suppose a compressive warp field on segment AD maps points A, B, C and D at the reference frame to the same live frame position, it is impossible to infer the reference frame position from the live observation at D’. Thus, directly performing data fusion at open-to-close topology changes will generate erroneous surfaces, at shown in Fig. 6.

Our solution is to bound the sensitivity of inverse warping at the live frame. Using the one dimensional illustration in Fig. 2, we discard live surfels at which

where the threshold $\epsilon$ is defined in Equ. 6. In 3D case, the gradient in Equ. 7 becomes a $3\times 3$ strain tensor and is computed by finite difference. The reference frame position is computed by inverse warping defined in Sec. IV and skinning method in Sec. VI-B.

VI-D Surfel Appending and Warp Field Update

We use methods similar to Keller et al. for appending depth surfels $s_{\text{depth}}$ to $S_{\text{live}}$ that do not correspond with any model surfels, with two additional criteria:

Discard surfels that the sum of nearest neighbor weights is less than $\delta_{nn}$. These surfels are far away from nodes and are likely to be outliers.

Discard surfels believed to be compressively mapped according to Sec. VI-C.

Existing surfels $s_{\text{live}}$ will also be discarded if:

The surfel remains unstable for a long time, i.e., its confidence $c$ is less than a confidence threshold $\delta_{\text{stable}}$ for a period $t_{\text{low\_confid}}$ after its initialization time $t_{\text{init}}$.

It is very similar to its neighbors on the index map $\mathcal{I}$.

The formal criterion and detailed methods can be found in Keller et al. . After fully updating $S_{\text{live}}$, an inverse warping defined in Sec. IV is performed to update $S_{\text{ref}}$ from $S_{\text{live}}$.

The warp field extending pipeline introduced in DynamicFusion is performed on the appended reference frame surfels $S_{\text{ref\_append}}$, to update the node graph $\mathcal{G}$ and the warp field $\mathcal{W}$. Intuitively, the extent which the new geometry $S_{\text{ref\_append}}$ is covered by the current node graph $\mathcal{G}$ is computed. Then the set of uncovered vertices are sub-sampled and appended to the node graph $\mathcal{G}$. Finally, the edges in the node graph are recomputed. The formal description and detailed methods can be found in Sec. 3.4 of DynamicFusion .

The nearest neighbors and weights of surfels are also updated base on the new node graph $\mathcal{G}$. For each surfel $s_{\text{ref}}$ in $S_{\text{ref}}$, its nearest neighbor set $N(s_{\text{ref}})$ is compared with appended nodes and updated if required. The number of appended nodes is usually no more than 10, thus a brute-force search over appended nodes is sufficient.

VI-E Initialization

Upon receiving the first depth image $\mathcal{D}^{0}$, we first compute the depth surfels according to Sec. V. Then, surfels corresponded to valid depth pixels are collected into both $S_{\text{ref}}$ and $S_{\text{live}}$, using GPU selection. The nodes in the warp field $\mathcal{W}$ are initialized by sub-sampling $S_{\text{ref}}$. The edges in node graph $\mathcal{G}$ are computed according to Newcombe et al. , and the SE(3) deformation values of nodes are initialized to identity. After the initialization of the warp field $\mathcal{W}$, the skinning of surfels is performed: the nearest neighbors and weights of each surfel are computed base on their reference frame positions and nodes in $\mathcal{W}$.

VI-F Model Reinitialization

As mentioned in Dou et al. , it is impossible to interpret all motions from a single reference, and incorrect to assume the non-rigid tracker never fails. Thus, Dou et al. used two TSDF volumes, one for the reference frame and another for the live frame, and reset the reference volume by the live volume to handle large motions, tracking failures and topology changes. However, maintaining an additional TSDF volume and the associated nearest neighbor field as in inevitably increases computation cost and memory usage.

Our pipeline naturally supports similar ideas in a more compact, efficient way: inferring erroneous surfels from depth observations is more explicit and easier than incorrect TSDF values. We discard any live surfel $s_{\text{live}}$ that should be visible from current camera view but do not have corresponded depth observations. The correspondence is identified by projecting the live surfel $s_{\text{live}}$ into the depth image and performing a window search. After cleaning $S_{\text{live}}$, we reset $S_{\text{ref}}$ to $S_{\text{live}}$ and initialize the warp field on it.

In our implementation, the model reinitialization is invoked by the misalignment energy defined in Equ. 8 and the number of appended surfels. When large misalignment energy values and lots of appended surfels are observed for several continuous frames, it is likely that the non-rigid tracker is struggling with incoming depth observations.

VI-G Complexity

In this subsection, we will analyze the complexity of all the operations that are introduced in this section.

Let $|V|$ denote the size of an array $V$, $|\mathcal{W}|$ denote the number of nodes in the warp field, and $S_{\text{append}}$ denote the array of appended surfel candidates. The data fusion (Sec. VI-A), inverse warping and nearest neighbor array updating (Sec. VI-D) are in the complexity of $O(|S_{\text{live}}|)$. The live surfel skinning (Sec. VI-B) and compression resolving (Sec. VI-C) are in the complexity of $O(|S_{\text{append}}|)$. An $O(|S_{\text{live}}|+|S_{\text{append}}|)$ GPU selection is performed to compact the surfel array. It is also noted that all these operations can be trivially GPU parallelized.

In the initialization (Sec. VI-E) and the model reinitialization (Sec. VI-F), the number of nodes $|\mathcal{W}|$ in surfel skinning is large enough to benefit from structured search algorithms like Muja and Lowe . We choose to build the search index on CPU and perform approximate nearest neighbor query on GPU, where the complexity is approximately $O(|S_{\text{ref}}|\text{log}(|\mathcal{W}|))$. Although divergent executions in the GPU parallel querying may weaken the performance, the (re)initialization is typically efficient (i.e., 3-5 ms) in our pipeline because of the compact sized $S_{\text{ref}}$.

From our observation, $S_{\text{live}}$ contains about $3\times 10^{5}$ surfels. $S_{\text{append}}$ typically has no more than $5000$ surfels, i.e., less than $\frac{1}{20}$ of valid depth pixels. Thus, sophisticated operations such as live surfel skinning and compression resolving are performed only on a small fraction of pixels, which further ensures the efficiency of our pipeline.

As a comparison, in volumetric approaches such as , the data fusion, forward voxel warping, marching cubes, update volumetric nearest-neighbor field and collision resolving are all in the complexity of $O(\text{number of voxels})$, which is usually one or two orders larger than $|S_{\text{live}}|$. Moreover, in the initialization frame, there is typically 300-1000 nodes in the warp field $\mathcal{W}$, which incurs a substantial computational burden to estimate the skinning of each voxel. From the analysis, our method is much more efficient than volumetric approaches in terms of performance and memory usage.

VII Non-Rigid Warp Field Estimation

The SE(3) deformations $T_{j}\in\mathcal{W}$ are estimated given a new depth observation $\mathcal{D}$, the previous deformation field $\mathcal{W}^{\text{prev}}$, and geometry models $S_{\text{ref}}$ and $S_{\text{live}}$ ($S_{\text{live}}$ is deformed from $S_{\text{ref}}$ according to $\mathcal{W}^{\text{prev}}$). To perform this estimation, we first predict the visibility of live surfels $S_{\text{live}}$, then follow DynamicFusion to cast the deformation estimation into an optimization problem.

The visibility prediction is similar to the approach of Keller et al. : we render the live surfels $S_{\text{live}}$ as overlapping, disk-shaped surface splats that are spanned by the position $v_{\text{live}}$, normal $n_{\text{live}}$ and radius $r_{\text{live}}$ of the live surfel $s_{\text{live}}$. Although advanced rendering techniques such as Zwicker et al. are available, we simply render opaque splats for optimal performance.

Different from rigid SLAM pipelines such as , we also render an index map $\mathcal{I}$ at the same resolution of depth images for efficient querying of nearest neighbors and weights. Moreover, to improve the robustness of the warp field estimation to outliers in $S_{\text{live}}$, we only render stable surfels, i.e., surfels with confidence $c$ larger than $\delta_{\text{stable}}$ defined in Sec. VI-D. For the first few frames or frames just after model reinitialization, we also render surfels that are observed recently, i.e., surfels such that $t_{\text{now}}-t_{\text{observed}}\leq\delta_{\text{recent}}$, where $t_{\text{now}}$ is current time and $t_{\text{observed}}$ is the last observed time of this surfel.

Following the approach of DynamicFusion , we solve the following optimization problem to estimate SE(3) deformations $T_{j}\in\mathcal{W}$:

where $E_{\text{depth}}$ is the data term that constrains the deformation to be consistent with the depth input $\mathcal{D}$, $E_{\text{regulation}}$ regularizes the motion to be as rigid as possible, $\lambda$ is a constant value to balance between two energy terms.

$E_{\text{depth}}$ is a point to plane energy term :

where $P$ is a set of corresponded model and depth surfel pairs, $n$ and $v$ are the normal and the vertex associated with the surfel. The method to find $P$ given the depth observation $\mathcal{D}$ and the rendered live geometry can be found in Guo et al. .

$E_{\text{regulation}}$ is an as-rigid-as-possible regularizer:

where $N_{j}$ is the set of neighboring nodes of the $j$th node in the node graph. This term ensures the non-visible parts of the geometry model will move with visible regions as rigidly as possible. Also, this term avoids arbitrarily deformed geometry to fit the noisy depth inputs.

The optimization problem Equ. 8 is a nonlinear least squares problem and solved by Gaussian-Newton algorithm implemented on GPU. Before the non-rigid estimation, we first perform a rigid alignment using the method in KinectFusion .

VIII Implementation Details

Our pipeline consists of three major components: the depth image processor, the non-rigid solver and the geometry update module. The depth image processor fetches depth images and computes depth surfels as described in Sec. V. We follow Dou et al. to implement the non-rigid solver: first construct the matrix $J^{T}J$, then solve the linear system $J^{T}Jx=J^{T}e$, where $J$ and $e$ are the Jacobian and residual for least square terms in Equ. 8. We constrain the maximum number of Gauss-Newton iterations to be 10, while the solver typically converges in 3 or 4 iterations. For the geometry update module, the pipeline parallelizes operations in Sec. VI over either the array of surfels $S_{\text{live}}$ or depth surfels from the depth image processor.

On a single Titan Xp GPU, the time to process one depth image is usually about 2 ms, and the geometry update typically takes 2-3 ms per frame. Thus, the overall performance is primarily determined by the non-rigid solver, which varies a lot with different dynamic scenes. Currently, our implementation is single threaded, and the performance can be further improved by the thread-level parallelism used in Guo et al. or the stream mechanism used in Dou et al. .

IX Results

In this section, we present a variety of dynamic scenes reconstructed by our technique. To demonstrate the improved performance and robustness, we compare our approach with our implementation of DynamicFusion , which uses the same non-rigid tracker but volumetric representations of geometry and nearest neighbor field. Parameters used for all reconstructions presented in this section are summarized in Table. 1. The depth streams recorded by Innmann et al. and Slavcheva et al. are used in the presented results. The video demo and source code are available on our project page

Fig. 3 shows the reconstruction of “upperbody”, “minion” and “hoodie” datasets by our techniques. The live geometries are rendered as normal maps. Readers are recommended to watch the accompanying video for more reconstruction results.

Fig. 4 shows the number of surfels remains roughly constant after objects are fully observed. It is noted that each depth frame provides about $10^{5}$ valid surfels in these examples. This demonstrates new surfels are not continuously added and the global model is refined but kept compact.

Fig. 5 shows the performance comparison between our method and DynamicFusion (our implementation) on “upperbody” dataset. The hardware platform for this comparison is a Nvidia Titan Xp GPU. Note that both methods share the same implementation of the non-rigid tracker. The process time result is the average of over 1200 frames. Due to the non-volumetric pipeline, our method represents the geometry and motion more compactly and demonstrates a significant speedup.

Fig. 6 compares our method with DynamicFusion and VolumeDeform on the “boxing” dataset. This sequence contains a open-to-close topology change. The result of VolumeDeform is from the original author. From the figure, DynamicFusion cannot correctly resolve compressive deformation and yields erroneous surfaces. VolumeDeform almost disables the update of the geometry model and results in incomplete surfaces. Our method correctly resolves the compressive deformation field and generates a faithful reconstruction of this scene.

Fig. 7 and Fig. 8 demonstrate that the model reinitialization mechanism enables our technique to handle tracking failures and close-to-open topology changes. In Fig. 8, the sequence contains tangential motion that cannot be solved by the depth-only non-rigid tracker. In Fig. 7, motion is fast (performed in approximately 2 seconds) and contains a close-to-open topology change. Our approach successfully recoveries from these failures. In comparison, DynamicFusion cannot correctly reconstruct the motion and geometry. For sequence in Fig. 8, the model reinitialization is invoked periodically alongside the criteria in Sec. VI-F.

X Limitation and Future Work

Model reinitialization significantly improves the robustness of our pipeline. However, cleaning of erroneous surfels may lead to incomplete geometry, as shown in Fig. 9 (a). Currently, the non-rigid tracker in our pipeline uses the same energy terms as DynamicFusion , which struggles with tangential or fast motions, as shown in Fig. 9 (b). We plan to address these limitations by extending our non-rigid tracker with energy terms in Dou et al. , using adaptive node-graph refinement similar to Li et al. and reconstructing the surface albedo as in Guo et al. .

XI Conclusion

This paper presents a dense SLAM system that reconstructs both the motion and geometry from a single depth camera. Unlike previous approaches, our method is purely point-based, i.e., without resorting to volumetric data structures such as TSDF or nearest neighbor fields, which makes our system highly efficient. We demonstrate all central operations, such as the nearest neighbor maintenance, warp field estimation and data fusion, can be parallelized by leveraging standard graphics pipeline and GPGPU computing. The explicit, flexible surfel array representation of geometry also enables efficient recovery from topology changes and tracking failures, which further improves the robustness of our pipeline. Experimental comparison with existing methods demonstrates the effectiveness of our method.

Acknowledgments

The authors would like to acknowledge the support from NSF Award IIS1427050 and Amazon Research Award. The views expressed in this paper are those of the authors themselves and are not endorsed by the funding agencies.

References