POCO: Point Convolution for Surface Reconstruction

Introduction

Constructing a surface or volume representation from 3D points sampled at the surface of an object or scene has numerous applications, from digital twins processing to augmented and virtual reality. Cheaper sensors directly producing 3D points (depth cameras, low-cost lidars) and mature multi-view stereo techniques operating on images offer increasing opportunities for such reconstructions.

Traditional 3D reconstruction approaches generally express the target surface as the solution to an optimization problem under some prior constraints. Possibly leveraging visibility or normal information, they are generally scalable to large scenes and offer a substantial robustness to noise and outliers . Although some try to cope with density variation , a common limitation of these approaches is their inability to properly complete parts of the scene that are less densely sampled or that are missing (typically due to occlusions). A variety of hand-crafted priors try to address this completeness issue: local or global smoothness , decomposition into geometric primitives (in particular for piecewise-planar man-made environments ) and structural regularities . Data-driven priors have also been explored, based on shape retrieval , possibly with deformations . But it remains limited in applicability.

To use richer priors, learning-based methods have been proposed, using explicit shape representations. Voxel-based approaches leverage a regular grid structure, extending 2D image-based techniques to 3D, but suffer from resolution limitations due to large memory consumption . Directly generating a mesh with a neural network remains difficult and is limited in practice to template deformation . Some forms of implicit representations have been used for point cloud generation, but providing much weaker geometrical and topological information .

A few recent methods , however, obtain a form of translational equivariance via Convolutional Neural Networks (CNNs). At least in theory, they can thus scale to larger scenes, possibly benefiting both from local and non-local information. But they operate on a voxelized discretization whose vertices may be far from the input point cloud. They thus lose the direct connection with points sampled on the surface of objects. They are also suboptimal in that the features or latent vectors holding the occupancy or distance information are more or less uniformly distributed in space rather than focused where difficult decisions have to be made, i.e., near the surface.

Our approach, based on point convolution, overcomes these issues. It is illustrated on Fig. 2. Our contributions are:

We attach features representing the implicit function to input points. Not only does it preserve point positions until later processing stages, rather than abstract them away too soon, but it concentrates the information to learn where it matters the most: close to the surface.

We compute features using point convolution, which yields a natural coverage and scalability to scenes of arbitrary size. (Rather than tailor yet another specific network architecture, we rely on a general point convolution backbone, which offers prospects for improvement when better point convolutions are designed.)

Rather than relying on hand-designed forms of averaging, we extend prior learning to interpolation, which we apply to query-relative features rather than global features, as others do, as it leads to better results.

We propose an efficient test-time augmentation to treat inputs of high density or large size.

While simple, our approach outperforms other methods both on object and scene datasets, yielding finer details. It is robust to domain shift (training on objects, testing on scenes) and faster than methods that overfit to a scene or infer from scratch for each query.

Related work

Voxels have been a natural choice for learning to represent 3D volumes . However, they come with a cubic complexity in space, leading to coarse discretizations due to memory constraints. Multi-scale refinement and sparsity-based octrees only partly reduce the impact of conforming to a 3D grid.

Points clouds are also produced as a sparse 3D representation, with various density and sampling distribution . Point processing and generation do not suffer from the complexity and discretization induced by 3D grids; yet, the range of applications is limited regarding representing actual surfaces and volumes.

Meshes are a preferred representation for many uses, such as visualizations and simulations, but they are harder to directly produce from a neural network (vertex regression and face construction) . Most existing approaches thus prefer to operate by deforming geometric primitives , voxelized approximations or learned templates . Rather than actually inferring vertices, a mesh can also be extracted from labels inferred on a Delaunay tetrahedralization .

Implicit representations rely on a neural network to model a function expressing the occupancy of a given 3D point or its distance to the surface, either signed , unsigned or sign-agnostic . The signed or unsigned distance field (SDF, UDF) is often truncated (TSDF, TUDF) and estimated via a multi-layer perceptron (MLP). The isosurface can then be extracted from this occupancy or distance field with various methods such as Marching cubes . Whereas voxels, points and mesh vertices are intrinsically discrete representations, implicit representations offer a virtually infinite resolution. Moreover, while mesh-based approaches struggle to enforce watertightness, to limit self-intersections and to address complex topologies (non genus-0), meshes reconstructed from implicit representations are guaranteed to be watertight and have no self-intersections. Besides, they can easily model arbitrary complex topologies. These advantages may explain the recent success of this representation, including to model 3D shapes from images without 3D supervision , with texturing or specific rendering . Departing from occupancy or distance fields, ShapeGF models a shape by learning the gradient field of its log-density, then samples points on high likelihood regions of the shape and meshes them. Other work also study the decomposition of shapes and implicit surfaces into parts , possibly overfitting networks to generate or render a single object or scene .

Scalability, however, is an issue for all these methods. While they can encode reasonably well one object or a class of objects, they cannot cope with the variability and size of an arbitrary scene involving several objects. Even considering a single object and assuming a powerful decoder, the encoding of a single or a few latent vectors hardly can develop into detailed shape information. Using periodic activation functions or adding a 2D convolutional component on input images helps, but is not enough.

A solution is to split the input points on a regular 3D grid and to optimize one latent vector per voxel (DeepLS), possibly from overlapping input patches. Patch splitting can also be irregular and optimization-driven to favor self-similarities, with a global post-optimization to flip inconsistent local signs (SAIL-S3). But whether these methods optimize only the latent vectors or a whole network as well, for patch decoding, they make surface reconstruction significantly slower, leading to reduced test sets.

Besides, these methods rely on fully-connected architectures whereas, we believe, convolutions, and in particular point convolutions , are the key to scalability and increased details.

2 Convolutions for implicit representations

LIG divides the input point cloud along a regular 3D grid to create 3D patches and capture local geometric shapes shared by several objects at a medium scale. For each of these patches, a 3D CNN then computes a local feature vector, which goes through a reduced IM-NET for SDF decoding. However, later on, only the learned decoder is exploited; no local embedding is inferred. Given an input point cloud, latent vectors on the grid are optimized from scratch to minimize an objective function similar to the loss used for training. LIG additionally requires to be provided with oriented normals to make use of points known to be inside or outside the shape. This, however, may introduce artificial back-faces, which can partly be addressed in a postprocessing stage. In contrast, we can work without normals, we directly operate with convolutions on surface points rather than on a regular grid, and we directly use inferred embeddings without any heavy optimization.

IF-Net introduces a multi-scale pyramid of 3D convolutional encoders aligned on a discrete voxel grid and trained on voxels at different scales. The occupancy of a query point is decided by a decoder taking as input the interpolated features extracted at this point for each pyramid level. In contrast, we do not discretize into voxels; we use point cloud convolution. Also, we learn how to interpolate the latent vectors rather than use a basic trilinear interpolation. Last, we provide results on scenes, not just on objects.

NDF uses the same multi-scale encoding as IF-Net but relies on a UDF rather than occupancy for decoding. It allows the generation of very dense points clouds that can directly be meshed into possibly open surfaces.

SG-NN uses a sparse 3D convolution to learn a TSDF in a self-supervised setting, training for completion from partial scans. In contrast, we use point convolution and infer occupancy rather than SDF, which is easier to learn.

ConvONet also uses a grid-based convolution, training an autoencoder that predicts occupancy. (It generalizes ONet , which only uses a single encoding and full connection.) For input point clouds, the encoder is a shallow PointNet operating on points rather than on a voxelized discretization, and the decoder is a 3D U-Net . The occupancy of a 3D point is inferred from a trilinear interpolation of grid features. Besides 3D convolution, variants based on a combination of 2D convolutions in a few spatial directions are proposed. DP-ConvONet is a variant that considers a dynamic family of such directions. SA-ConvONet overfits a pre-trained ConvONet model on the input using a sign-agnostic optimization of the implicit field. It improves accuracy at the cost of computation time.

As inference applies to grids, whose vertices or centers may be far from input points, the above methods lose the direct connection with the input surface samples. They are also suboptimal in that the latent vectors holding the information are uniformly distributed in space rather than concentrated where it matters the most, i.e., near the surface. To address these issues, we use point convolution and compute latent vectors at each input point. We then interpolate occupancy decisions of nearest neighbors using learned weights.

AdaConv uses point convolution like us but aggregates multi-scale information on an adaptive voxel grid, while we attach features to points, closer to the surface. Besides, it requires oriented normals, contrary to us.

RetrievalFuse splits a scene along a regular grid and encodes each 3D chunk as a latent vector via convolutional layers. But rather than using them for decoding, it retrieves similar chunks from the training set and combines their distance field to create a surface, enhancing the completion capability. In contrast, we are fully convolutional and the implicit function is directly obtained by interpolating inferred features, without the need to maintain the dataset samples used for training and with more generalization capacity.

Points2Surf collects, for each query point, both a patch of neighbors (which gives a convolution flavor) and globally-sampled input points to help to provide a sign to the local distance field. The local patch and the global sub-sampling go through an MLP to create latent vectors that are concatenated and decoded into a signed distance. In contrast, we directly get non-local information as our receptive field is much larger. Besides, we are faster as we only compute a limited number of latent vectors (one per input point) that we later use for interpolation given a query point, while Points2Surf samples local+global points and goes through the whole encoder for each query point, i.e., a large number of times, that grows with the Marching-cubes resolution.

To infer occupancy or distance of a query point, methods that compute several latent vectors for a single object or scene either select the most appropriate latent vector to decode, typically in a multi-scale grid , or interpolate the latent vectors of query neighbors . We perform interpolation too, based on features computed on input points. However, given a query point, we do not interpolate the features themselves but the occupancy logits, as our experiments shows it leads to better results. Besides, we use a learned interpolation rather than the usual tri-linear interpolation or the inverse-distance distance weighting . Although different in nature, learning has also been used in to blend retrieved chunks.

Our method

Overview. Our method consist of the following steps:

We encode input points ${\mathbf{p}}\,{\in}\,{\mathcal{P}}$ into latent vectors ${\mathbf{z}}_{\mathbf{p}}$.

Given an arbitrary query point ${\mathbf{q}}$, we consider a neighborhood ${\mathcal{N}}_{\mathbf{q}}$ of input points in ${\mathcal{P}}$ to interpolate from.

For each neighbor ${\mathbf{p}}\,{\in}\,{\mathcal{N}}_{\mathbf{q}}$, we construct a relative latent vector ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$ from ${\mathbf{z}}_{\mathbf{p}}$ and local coordinates ${\mathbf{q}}\,{-}\,{\mathbf{p}}$.

We extract significance weights $s_{{\mathbf{p}},{\mathbf{q}}}$ to sum the relative latent vectors ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$: ${\mathbf{z}}_{\mathbf{q}}=\sum_{{\mathbf{p}}\in{\mathcal{N}}_{\mathbf{q}}}s_{{\mathbf{p}},{\mathbf{q}}}\,{\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$.

We decode the resulting feature vector ${\mathbf{z}}_{\mathbf{q}}$ as two full-empty logits $\mathbf{o}_{\mathbf{q}}$, and turn them into probabilities ${o}_{\mathbf{q}}$.

These steps, illustrated on Figure 5, are detailed below.

Absolute encoding. A point convolution first produces a latent vector ${\mathbf{z}}_{\mathbf{p}}\,{=}\,E({\mathbf{p}})$ for each input point ${\mathbf{p}}\,{\in}\,{\mathcal{P}}$. The encoder $E$ can be implemented by any point cloud segmentation backbone, only changing the last layer to yield a vector of some chosen dimension $n$ as the size of vectors ${\mathbf{z}}_{\mathbf{p}}$. (In our experiments, the convolution backbone is FKAConv and $n\,{=}\,32$.) To also use normals (optionally), the input points are just augmented with the 3 normal coordinates.

Query neighborhood. Given an arbitrary query point ${\mathbf{q}}$ (when training or to predict occupancy at test time), we construct a set of neighbors ${\mathcal{N}}_{\mathbf{q}}$ from input points ${\mathcal{P}}$. (In our experiments, ${\mathcal{N}}_{\mathbf{q}}$ is the $k$ nearest neighbors of ${\mathbf{q}}$, with $k\,{=}\,64$.)

Relative encoding. We augment the latent vector ${\mathbf{z}}_{\mathbf{p}}$ of each neighbor ${\mathbf{p}}\,{\in}\,{\mathcal{N}}_{\mathbf{q}}$ with the local coordinates ${\mathbf{q}}\,{-}\,{\mathbf{p}}$ of query point ${\mathbf{q}}$ relatively to ${\mathbf{p}}$. These augmented latent vectors are then processed by an MLP $R$ to produce relative latent vectors ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}=R({\mathbf{z}}_{\mathbf{p}}\,\|\,{\mathbf{q}}\,{-}\,{\mathbf{p}})$, where $\|$ is the concatenation. (In our experiments, ${\mathbf{z}}_{\mathbf{p}}$ and ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$ have size $n\,{=}\,32$.)

Feature weighting. As PRNet , we observe that the norm of embeddings ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$ tends to correlate with their significance, hinting how much an input point ${\mathbf{p}}$ matters for deciding the occupancy of query point ${\mathbf{q}}$, given ${\mathbf{p}}$’s neighbors and the position of ${\mathbf{q}}$ w.r.t. ${\mathbf{p}}$. We use it to infer significance weights for relative latents vectors ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$. Concretely, we use an attention mechanism (blue frame in Fig. 5): The relative embeddings ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$ go through a linear layer parameterized by a weight vector ${\mathbf{w}}$, also of size $n$, producing relative weights $w_{{\mathbf{p}},{\mathbf{q}}}\,{=}\,{\mathbf{w}}\,\odot\,{\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$, that are normalized by softmax over ${\mathcal{N}}_{\mathbf{q}}$ into positive interpolation weights $s_{{\mathbf{p}},{\mathbf{q}}}$ summing to $1$. We actually use a multi-head strategy to obtain a form of ensembling. We learn $h$ independent linear layers, parameterized by $h$ corresponding weight vectors $({\mathbf{w}}_{i})_{i=1..h}$, producing $h$ relative weights $w_{{\mathbf{p}},{\mathbf{q}},i}\,{=}\,{\mathbf{w}}_{i}\,\odot\,{\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$, that are finally softmaxed as $s_{{\mathbf{p}},{\mathbf{q}},i}$ and averaged as $s_{{\mathbf{p}},{\mathbf{q}}}\,{=}\,\frac{1}{n}\sum_{i}s_{{\mathbf{p}},{\mathbf{q}},i}$. (In our experiments, we use $h\,{=}\,64$.)

Interpolation. The feature vector ${\mathbf{z}}_{\mathbf{q}}$ at query point ${\mathbf{q}}$ is interpolated from the relative latent vectors ${\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$ of neighbors ${\mathbf{p}}$, as the weighted sum ${\mathbf{z}}_{\mathbf{q}}\,{=}\sum_{{\mathbf{p}}\in\smash{{\mathcal{N}}_{\mathbf{q}}}}\!s_{{\mathbf{p}},{\mathbf{q}}}\,{\mathbf{z}}_{{\mathbf{p}},{\mathbf{q}}}$.

Decoding. A linear layer $D$ decodes the feature vector ${\mathbf{z}}_{{\mathbf{q}}}$ into occupancy scores $\mathbf{o}_{{\mathbf{q}}}\,{=}\,D({\mathbf{z}}_{{\mathbf{q}}})$, which is a two-logit vector classifying position ${\mathbf{q}}$ as occupied or not, that is then turned via softmax into occupancy probabilities ${o}_{{\mathbf{q}}}$.

Loss function. To train the network, we use a cross-entropy loss that penalizes wrong occupancy predictions. Please note that using a binary cross-entropy, like in IF-Net or ConvONet , leads to identical results.

Refinements

Adapting to high density. We train our network with a fixed number $N_{\text{train}}$ of input points for easy mini-batching. (In our experiments, $N_{\text{train}}\,{=}\,$3k or 10k.) At test time, if the surface is more densely sampled, the receptive field of the backbone may lack enough global context to decide which side of the surface is full or empty, unless oriented normals are also provided with points. A way to broaden enough the receptive field is to downsample the input point cloud, but it then naturally leads to a loss of details.

To reduce this effect, we rely on test-time augmentation (TTA) , which can be seen as a form of ensembling: we average several runs on different subsamples. However, aggregating final results, as often done in TTA , would be very time consuming in our case as we would have to do it to answer the occupancy of each query, basically multiplying the inference running time by the number of subsamples.

Instead, we perform TTA at latent vector level, thus running several times only the first step of our approach (absolute encoding), before query decoding. It depends on the number of input points (to attach a latent vector on), rather than on the number of query points, which is much larger. Concretely, we randomly create enough subsamples so that each point ${\mathbf{p}}\,{\in}\,{\mathcal{P}}$ is seen at least $N_{\text{view}}$ times, and average each ${\mathbf{z}}_{\mathbf{p}}$ over all samples. (In experiments, $N_{\text{view}}\,{=}\,10$.) The subsamples are randomly generated by sequentially picking a point ${\mathbf{p}}\,{\in}\,{\mathcal{P}}$ with a priority that is the opposite of the number of times ${\mathbf{p}}$ appears in previous subsamples.

Adapting to large size. As our method is convolutional, it naturally adapts to input point clouds ${\mathcal{P}}$ of arbitrary size. Yet, while ${\mathcal{P}}$ may contain millions of points, GPU memory limits in practice the number of points $N_{\text{test}}$ that can be treated together by the backbone. (We use $N_{\text{test}}\,{=}\,$100k.)

As with semantic segmentation , we can use a sliding-window with overlapping chunks of ${\mathcal{P}}$ of maximum size $N_{\text{test}}$. Alternatively, as above, we can make subsamples of ${\mathcal{P}}$ by iteratively picking a low-priority point ${\mathbf{p}}\,{\in}\,{\mathcal{P}}$ and its $N_{\text{test}}{-}1$ nearest neighbors. (In our experiments, $N_{\text{view}}\,{=}\,3$.)

Scene scaling. At inference time, the scale of the input point cloud may differ from the scales in the training set. As point-based backbones can be sensitive to variations of scale and density, we rescale the input such that the average distance between a point and its nearest neighbor is the same both in the training set and in the test point cloud.

Experiments

We experiment both on objects and scenes, in different point density regimes, with or without normal information depending on the baseline methods we compare with.

Because existing methods often perform well in some setting but not in others, most published papers tend to evaluate on different datasets or in specific configurations: number of train/test points, added noise, normals, generalization, etc. Some methods are also too slow to be evaluated on full datasets and report results only on dataset fractions. To be fair with these methods, we evaluate in their setting (when enough information is provided to do so) rather than impose them specific settings. It also illustrates the ability of our method to adapt to various configurations.

ShapeNet , as pre-processed by , contains watertight meshes of shapes in 13 classes, with train/val splits and 8500 objects for testing. As , we sample 3000 points from each mesh (at each epoch) and apply a Gaussian noise with zero mean and standard deviation 0.05.

Synthetic Rooms has 5000 synthetic scenes with random walls and populated with ShapeNet objects. We use ’s protocol for sampling 10k pts on the meshes to create train/val/test data, with noise as for ShapeNet. Shapes are scenes in terms of complexity, objects in terms of size.

ABC is a set of CAD models, mainly mechanical parts. We use splits and point preprocessing from : 4950 shapes for training, 100 for validation and 100 for testing.

Famous contains 22 shapes of various origins, e.g., from the Stanford 3D Scanning Repository .

Thingi10k , as prepared by , has 100 shapes.

SceneNet is a synthetic dataset of indoor scenes. Data prepared in the same way as yield 34 scenes.

MatterPort3D has indoor scenes too. We use the same 2 scenes as prepared and used by : with 65k pts.

Baselines are drawn among the state-of-the-art methods presented in Section 2.2. We also compare to SPR , a popular, non-learning-based reconstruction method that requires oriented normals (which is a strong hypothesis) and, possibly, a trimming parameter tuning (factor 6 in Tab. 4).

Our method, unless otherwise stated, uses the FKAConv backbone , feature size $n\,{=}\,32$ as in ConvONet or LIG , $k\,{=}\,64$ neighbors, $h\,{=}\,64$ interpolation heads, and does not use normals nor TTA.

Mesh Generation, for implicit functions, is done with the Marching cubes with resolution $256^{3}$ for objects, 1 cm for SceneNet, 2 cm for MatterPort3D.

Metrics. We use the following common metrics: volumetric IoU, symmetric Chamfer L1-distance $\times 10^{2}$ (CD), normal consistency (NC), i.e., mean absolute cosine of normals in one mesh and normals at nearest neighbors in the other mesh, and F-Score with threshold value 1% (FS). Surface metrics are approximated by point sampling.

2 Alternative and ablation studies

To justify our algorithmic choices, we experiment on ShapeNet in generalization mode, training on chairs but evaluating on all the classes. We use the same train/test split as , evaluating on 130 shapes (10 per class).

As can be seen in Table 1(a), the convolutional backbone FKAConv is more efficient by a large margin than the PointNet-based segmentation network with residual connections , which loses small scale information .

Though interpolating from $k\,{=}\,64$ neighbors rather than $k\,{=}\,128$ has a slightly worse CD and NC (cf. Tab. 1(b)), it has a better IoU and it is faster; we use this setting in the following. We note we get better results with a multi-head attention (using $h\,{=}\,64$ rather than $h\,{=}\,1$) and when interpolating relative rather than global features (cf. Tab. 1(c)).

Last, Tab. 2 and Fig. 3 show the benefits of the TTA strategy with models trained with 3k and 10k points on ABC.

3 Reconstruction

Reconstruction without normals. Because of long running times, only a few published methods evaluate on the whole ShapeNet dataset. We outperform them on all metrics with a significant margin (Table 4). We reconstruct finer details (Figure 6) and we do not have the same tendency as ConvONet to fill volumes; we can instead generate more easily thin surfaces, which explain our superior IoU. We outperform other methods as well on Synthetic Rooms (Table 4), where also we capture much finer details.

Generalization. LIG is specifically designed for scalability and generality. It learns to reconstruct small shape patches from a given dataset, and then applies it to any new object or scene. Points2Surf is a patch-learning method too, although its requirement for a global view of the input and its running time make it less suited for scene reconstruction.

We compare to LIG, training both methods on ShapeNet objects (with normals as LIG requires them) and testing on SceneNet. We generalize better (Tab. 5) at all densities, capturing finer details and not erasing thin objects (Fig. 4).

We compare to Points2Surf, training on ABC in the same setting. We outperform Points2Surf on most of their settings (Tab. 2), both on ABC and when generalizing to Famous and Thingi10k. Points2Surf outperforms POCO only on very noisy or dense inputs, and only with a small margin.

Scene reconstruction without normals. We compare to SA-ConvONet on MatterPort3D scenes (Fig. 1) in their same actual setting (downsampling to 65536 pts). Our reconstruction is less smooth than SA-ConvONet but has finer details. As SA-ConvONet overfits many networks at inference time on top of ConvONet, it is notably slower too.

4 Discussion and limitations

Our approach is suited both for single-object and whole-scene reconstruction. However, although it can cope with a substantial variation of point density, it cannot complete shapes when large parts are missing. Apart from a few methods like , only object-targeted methods can presently do it, for classes known at training time, but they cannot reconstruct scenes at all.

Inferring surface orientation, when normals are not provided, requires wide context information. But a high density may reduce the receptive field, yielding orientation failures and artifacts. Our TTA only partly addresses the issue; handling it directly at backbone level would be better.

Nevertheless, POCO reaches the state of the art for both object and scene reconstruction, with or without oriented normals. It shows good generalization capabilities to shapes and scenes that are very different from the training set.

More details and visuals on the method and on the experiments are in the supplementary material.

Acknowledgments to Gilles Puy for fruitful discussions.

References

Supplementary material

Appendix A Implementation details

The code of our method is available at https://github.com/valeoai/POCO.

Framework and hardware.

Our code uses PyTorch as deep learning framework. All experiments were done with a single NVIDIA RTX 2080 Ti GPU with 11GB memory.

Backbone.

We used FKAConv as convolutional backbone, with default parameters (number of layers, number of layer channels). Only the latent vector size $n$, i.e., the output dimension of the backbone, was changed. It was set to 32, which is also the output dimension of all linear layers of the occupancy decoder (except the last one, which outputs the occupancy). As a comparison, methods like ConvONet and LIG also use latent vectors of size 32.

Architecture.

The network architecture is described in Figure 5 of the main paper. We phrase here some parts of it.

The input size of the relative encoder (green area in Figure 5) is the size of the latent vectors (i.e., the backbone output size) plus the size of point coordinates, i.e., $32\,{+}\,3\,{=}\,35$. All linear layers have an output size of 32, except the multi-head layer for the computation of significance weights, of output size $h\,{=}\,64$, and the final occupancy layer, of output size 2, corresponding to classes empty and full. The layer activations all are ReLUs. Batch norms are only used in the backbone, i.e., the absolute encoder $E$; there are none in the relative encoder $R$, nor in the decoder $D$.

Point sampling at training time

is not part of POCO. We reused existing dataset samplings (from ConvONet and Points2Surf ) to compare on the same training data. The other datasets are only used for inference.

Training settings.

We train using Adam with learning rate $10^{-3}$. The training batch size is 16 for 3k input points and 8 for 10k input points. We train for 600k iterations.

Appendix B Meshing for occupancy

Mesh generation, for implicit functions, generally relies on the Marching cubes (MC) algorithm , evaluating occupancy on a regular 3D grid.

Recently, the MC variant used in ONet has often been used due to its higher speed. It operates on a coarse grid but locally refines the resolution thanks to a heuristics: Unless all corners of a cube at a given resolution agree on being empty of full, i.e., as soon as two corners of a cube disagree on occupancy, the cube is subdivided into 8 subvoxels. The initial grid is typically of size $32^{3}$, and it is typically refined (subdivided) up to two times, leading to a local resolution equivalent to a $128^{3}$ grid. The resulting mesh, after MC, is furthermore simplified and refined using first and second order gradient information . While the heuristics may miss thin details, this MC with refinement (MC-refin) leads to a much faster running time than plain MC, with a factor up to $8^{2}$ when using up to two refinement steps.

Marching cubes based on region growing (MC-regro).

To ensure we have little chances of missing refinements, in particular for locally complex surfaces or thin volumes, we use a different strategy. We consider from the outset a fine-grained resolution but, to prevent many useless queries in large empty or full regions, we adopt a region-growing approach (MC-regro). The seeds are the input points, for which we compute the occupancy. We then compute the occupancy for query points that are both in the close neighborhood (voxels at distance at most 2 grid steps) of both a location in the empty volume and a location in the full volume, i.e., close to the surface. And we iterate.

Besides, with the Marching cubes algorithm, a vertex is placed on the edge of a cube by linearly interpolating the two scalar values at the edge’s endpoints. But contrary to distance fields, occupancy fields may have sharp transitions. Consequently, opposite-side endpoints frequently have values close to 0 and 1, and vertices tend to be placed in the middle of segments, creating discretization effects. To prevent it, we perform a dichotomic search along edges to better locate the occupancy transition. We operate 10 dichotomies, which is more than enough in most cases.

In general, reconstructions with MC-regro are qualitatively better than MC-refin on scenes, but similar on objects. In fact, quantitative results on ShapeNet show a similar reconstruction accuracy of POCO with either MC-refin or MC-regro. The reason probably is that thin details have little impact on the different metrics. This ability to capture thin details makes MC-regro generally slower than MC-refin (see Section C, Table 7).

Appendix C Running times

Some running times are given in Figures 1 and 4 in the paper, as well as here in Tables 6 and 7.

The time for the backbone to extract features is negligible (${<}\,1$%). The bottleneck is the decoding, as we have to respond to many occupancy queries depending on the resolution of the Marching cubes (MC). And to answer an MC query, the bottleneck is the computation of nearest neighbors, which currently is not optimized, requiring communications between the GPU and the CPU. (It could probably be optimized by pre-computing neighbors at low MC resolution to reduce the GPU-CPU communication overhead.)

In contrast, grid-based methods such as those based on ConvONet do not need such an optimization as they do not depend on nearest neighbors. However, while our approach requires extracting one feature per point for encoding (typically a few thousands points for an object), these other methods extract one feature per grid cell, typically $64^{3}\,{\approx}\,262$k. Besides, as we show in the paper, losing input points induces a loss of details.

Although in this case, because of the high point-cloud density (50k pts), we apply the test-time augmentation (TTA) strategy and run the latent vector inference on many different point cloud subsamples (such that each point is seen at least $N_{\text{view}}\,{=}\,10$ times), our method is still significantly faster than Points2Surf.

In fact, as our encoding time is negligible compared to the numerous decoding queries for meshing with MC, our TTA strategy at feature level brings little slowdown, e.g., +5% for $N_{\text{view}}\,{=}\,10$, compared to $N_{\text{view}}\,{=}\,1$.

Overall reconstruction time.

In Table 7, we report the average reconstruction time of different methods. To be fair, given that mesh generation via occupancy queries is a running time bottleneck, we compare the methods using the same MC algorithm, namely MC-refin with a coarse grid of size $32^{3}$ that can be refined up to twice, i.e., into a grid of size $128^{3}$. We also report the running time of POCO with our MC-regro variant on a grid of size $128^{3}$. As said in Section B, the quantitative results of POCO with either MC-refin or MC-regro are similar.

On ShapeNet with medium-density points clouds (3k points per shape), we rank second behind ConvONet for speed. Note however that LIG is faster on denser scenes (see Figure 4 of the main paper) as the computation time per patch is constant, while our kNN search based on a kd-tree gets slower. (It could be faster by precomputing neighbors in the data loader, to limit GPU-CPU exchanges.)

Appendix D Receptive field

A question that naturally arises to understand the power and benefits of different approaches is the size of the receptive field for inferring occupancy features.

Because it is based on nearest neighbors, the receptive field of the backbone varies based on the scene geometry. It naturally tends to augment with the number of layers but sometimes, as when a separate group of points are mutual neighbors, the local receptive field does not increase.

To evaluate the actual (in fact, maximum) receptive field of a given point, we apply the following procedure:

We use a variant of the network without ReLUs and where the convolutions are replaced with averaging.

We apply the loss on a single output location.

We identify input points receiving a non-zero gradient.

On a SceneNet living-room scene, with density 100 pts/m2, we obtain an average receptive field of 29k points when looking at non-zero gradient (see Figure 7). If we only look at points for which the back-propagated gradient has a norm greater than $10^{-7}$ (i.e., a significant gradient), then the receptive field encompasses 16k points.

Appendix E Experiments

Following the success of methods such as AtlasNet and DeepSDF , a dozen of new learning-based reconstruction methods have been published every year.

As said in the main paper, existing methods often perform well in some settings but not in others. Consequently, most published papers tend to evaluate on different datasets (see Table 9) or in specific configurations: low or high density of train/test points, with or without added noise and outliers, with or without oriented normals, training specifically for a class of shapes or generalizing to any shape, addressing single object or whole scene reconstruction, etc. Some methods are also too slow to be evaluated on full datasets and report results only on dataset fractions. Last, although most methods make code available, some do not offer pre-trained models, or scripts, or parameters (at the time of writing). This makes comparisons particularly difficult.

We chose to compare to some of the most cited or most recent methods. To be fair with these methods, we evaluate in their setting (when enough information is provided to do so) rather than impose them other specific settings. It also illustrates the ability of our method to adapt to various configurations. Method codes are referenced in Section F.3.

The datasets that we used in our experiments are listed in Table 8. Datasets references are in Section F.3.

On SceneNet, we chose points with normals, which allows comparing to LIG (which requires normals).

On MatterPort3D, we chose points without normals, allowing comparison to SA-ConvONet but not to LIG.

On ShapeNet, we chose in the main paper points without normals and with noise, allowing comparison to ConvONet; in this supplement, we use points with normals and without noise, allowing to compare to LIG.

E.2 Metrics

We use exactly the same evaluation metrics as ConvONet , as specified formally in the supplementary material. However, for our report to be more self-contained, we reformulate here explicitly the metrics that we use.

The surface metrics measure different forms of deviations between two surfaces, i.e., the deviation between the reconstructed surface and the ground-truth surface. In practice, the metrics are approximated by replacing the continuous distances by the distances between points sampled on both surfaces. In particular, the distance of a point $p$ to a surface $S$ is approximated by the distance of $p$ the nearest point $q$ sampled on surface $S$. In our experiments, we sample on each surface: 100k points for ShapeNet and Synthetic Rooms; 10k for ABC, Famous and Thingi10k; and 4M points for SceneNet. As can be seen by the performance of ‘Oracle’ in Table 5 of the paper, which compares the ground-truth against itself via two different samplings, this discretization is a reasonable approximation, although POCO gets close to the error margin when the point cloud is dense and the normals are provided.

The Chamfer distance between two point clouds $P_{1},P_{2}$ is defined as follows:

where $d(p_{1},p_{2})$ is the distance between points $p_{1},p_{2}$. In the paper, following ONet and ConvONet , we use the L1-norm. What we name ‘CD’ in tables is Chamfer$\,\times\,10^{2}$.

Normal consistency (NC).

The normal consistency between two point clouds $P_{1},P_{2}$ is defined as follow:

is the closest point to $p$ in point cloud $P$ and where $n_{p}$ is the normal at point $p$, given by the orientation of the mesh face on which the point is sampled.

F-Score (FS).

The F-Score between two point clouds $P_{1}$ and $P_{2}$ at a given threshold $t$ is given by:

In the paper, following ONet and ConvONet , we use $t=0.01$.

Intersection over Union (IoU).

Compared to the previous metrics, which evaluates the quality of the generated surface, the IoU is a volume metric.

Noting TP (resp. FP and FN) the number of true positive, i.e., the number of points correctly predicted as full (resp. the number of points wrongly predicted as full, and the number of points wrongly predicted as empty), the IoU is defined as follows:

E.3 More qualitative results

We provide on Figure 8 more visualizations of ShapeNet reconstructions, comparing LIG to POCO at various densities of input points (with normals). LIG reconstructions were done using the best parameter setting for the method, i.e., with part size 0.20 for 512 and 2048 points, and part size 0.10 for 8192 points. Nevertheless, POCO reconstructs surfaces with more robustness and much sharper details.

POCO vs SPR and LIG on SceneNet (various densities).

As a complement to Table 5 in the main paper, we provide here on Figure 9 the visualization of a reconstruction fragment of a SceneNet scene, also with varying input point densities, comparing SPR, LIG and POCO. As can be seen, POCO provides a better robustness at low point densities and more details at high point densities.

POCO vs SPR (generalization ability).

In fact, POCO out-of-the-box adapts well to new shape domains without retraining (Figures 1, 3, 4 and Table 2), especially when given normals (Table 5). SPR only works well on high-density point clouds (Figure 4, Tables 2, 4, 5).

POCO vs ConvOnet on Synthetic Rooms.

As a complement to Table 4 in the main paper, we provide here on Figure 10 the visualization of reconstructions on the SyntheticRooms dataset (2 first scenes of each data bunch), comparing ConvOnet and POCO. In general, we provide more and sharper details; we are also more robust to thin surfaces, e.g., selves of the bookcase in “Room 05 - scene 801” and coffee table in the foreground of “Room 08 - scene 801”.

E.4 More quantitative results

As a complement to Table 3 in the main paper, we provide here in Table 10 classwise quantitative results on ShapeNet, comparing POCO to PointConv, ONet and ConvONet (the $3\times 64^{2}$ variant, that performs best on ShapeNet).

PointConv is a baseline method which is defined in the ConvONet paper . It proceeds as follows: point-wise features are extracted using PointNet++ , interpolated using Gaussian kernel regression and feed into the same fully-connected network used in ConvONet . While this baseline uses local information, it does not exploit convolutions. ONet is not convolutional either; it operates on shapes as a whole.

As can be seen in the table, POCO largely outperforms the compared methods on all categories, especially on classes featuring complex details such as lamp, rifle, vessel and, to a lesser extent, airplane, car, chair and loudspeaker. Yet, the most difficult classes are more or less the same for all methods, including POCO: lamp and car.

Appendix F Use of existing assets

The implementation of our approach has several dependencies, that are all free to use for research purposes. The main dependencies of our code are as follows:

FKAConvhttps://github.com/valeoai/FKAConv , under Apache License v2.0.

PyTorchhttps://pytorch.org/, under the Apache CLA,

PyTorch-Geometrichttps://pytorch-geometric.readthedocs.io/, under the MIT License,

The code of POCOhttps://github.com/valeoai/POCO itself is freely available, under Apache License v2.0.

F.2 Datasets

For the experiments, we used several datasets that are freely available for research purpose:

ABChttps://deep-geometry.github.io/abc-dataset/ is under the Onshape Terms of Usehttps://www.onshape.com/en/legal/terms-of-use. We used the subset preprocessed and made available by the authors of Points2Surf${}^{\text{\ref{foot:points2surf}}}$ .

Famous is a set of shapes of various origins, among which the Stanford 3D Scanning Repositoryhttp://graphics.stanford.edu/data/3Dscanrep/ . This set of shapes is described, preprocessed and made available by the authors of Points2Surf${}^{\text{\ref{foot:points2surf}}}$ .

MatterPort3Dhttps://niessner.github.io/Matterport/ is under a user license agreement for academic use. We used scenes preprocessed by the authors of SA-ConvONet${}^{\text{\ref{foot:saconvonet}}}$ .

Real-World point clouds used in the paper are described, preprocessed and made available by the authors of Points2Surf${}^{\text{\ref{foot:points2surf}}}$ .

SceneNethttps://robotvault.bitbucket.io/ is under the CC BY-NC 4.0, for research purposes only. We made meshes watertight using Watertight Manifoldhttps://github.com/hjwdzh/Manifold , that enables code use under mild conditions.

ShapeNethttps://shapenet.org/ has a licence for non commercial research or educational purposes. We used the version of ShapeNet as preprocessed by the authors of ONethttps://github.com/autonomousvision/occupancy_networks , which itself reuses the preprocessing of the authors of 3D-R2N2https://github.com/chrischoy/3D-R2N2 .

Synthetic Rooms${}^{\text{\ref{foot:convonet}}}$ is a dataset created by the authors of ConvONet based on ShapeNet models.

Thingi10Khttps://ten-thousand-models.appspot.com is a freely available collection of shapes under various licences. We used the subset preprocessed and made available by the authors of Points2Surf${}^{\text{\ref{foot:points2surf}}}$ .

F.3 Methods

We compared to a number of reconstruction methods, reusing the code made available by their authors:

ConvONethttps://github.com/autonomousvision/convolutional_occupancy_networks under the MIT License.

LIGhttps://github.com/tensorflow/graphics/tree/master/tensorflow_graphics/projects/local_implicit_grid probably under Apache License v2,

Neural Splineshttps://github.com/fwilliams/neural-splines under the MIT License,

Points2Surfhttps://github.com/ErlerPhilipp/points2surf under the MIT License,

SA-ConvONethttps://github.com/tangjiapeng/SA-ConvONet under the MIT License,

SPRhttps://github.com/mkazhdan/PoissonRecon under the MIT License.

We also compared to AtlasNet , DeepSDF , DP-ConvONet , ONet , but only reusing the numbers mentioned in .

Here are some methods we would have liked to compare to, but could not in practice:

AdaConvhttps://github.com/isl-org/adaptive-surface-reconstruction : The repository provides raw code but no pre-trained model nor instructions or scripts to train or to test, which may lead to misuses and wrong comparisons.

NDFhttps://github.com/jchibane/ndf : The repository provides code but only a pre-trained model for ShapeNet cars. For scene reconstruction, it does not offer preprocessed data or any data preprocessing procedure to retrain a model, nor instructions to run NDF using a sliding window scheme, as alluded to in the supplementary material.

As indicated in Table 9, some authors also have not made their code or their model available to allow comparisons.

Appendix G Societal impact

We believe our 3D reconstruction approach has very little potential for malicious uses (including disinformation, surveillance, invasion of privacy, endangering security), not more, e.g., than image enhancement methods in the 2D data case, and not more than hundreds of previously published 3D reconstruction methods. Besides, we are not bound nor promoting any dataset that would lead to unfairness in any sense. The use of our method has a modest environmental impact as the training time (a few days on a single GPU for a large dataset) and the inference times (minutes, or hours for very large point clouds) are somewhat moderate, and favorably compare to many learning-based approaches.

On the contrary, applications of our method can be found in various domains, with positive societal impacts:

Heritage preservation. Digitizing cultural objects and monuments allows a form of heritage preservation and enables virtual museums to make works of art and culture more widely accessible.

Infrastructure and building maintenance. Reconstructing models of existing infrastructures and buildings is of high interest for the construction industry. These models are particularly useful to plan and organize maintenance. This is particularly useful in a context of aging infrastructures and building renovation for energy-saving insulation.

Augmented and virtual reality. Surface and volume reconstruction are useful assets for augmented and virtual reality, whether it is for professional use (e.g., on-site maintenance of equipment) or entertainment (video games, special effects for the film industry), which is however to be consumed in moderation.