Hierarchical Surface Prediction for 3D Object Reconstruction

I Introduction

We live in a world composed of 3D objects bounded by 2D surfaces. Fundamentally, this means that we can either represent geometry implicitly as 3D volume or explicitly as 2D mesh surface which lives within the 3D space. When working with 3D geometry in practice for many tasks a specific representation is more suited than the other.

Recently, 3D prediction approaches which directly learn a function to map an input image to the output geometry have emerged . The function is represented as Convolutional Neural Network (CNN) and the geometry is represented as voxel grid, which is a natural choice for CNNs. The advantage of such an approach is that it can represent arbitrary topology and allows for large shape variations. However, such approaches have a major limitation. Due to the cubic growth of the volume with increasing resolution only a coarse voxel grid is predicted. A common resolution is $32\times 32\times 32$. Using higher resolutions very quickly becomes computationally infeasible. Moreover, due to the 2D nature of the surface, the ratio between surface or close to the surface and non-surface voxels becomes smaller and smaller with increasing resolution. Voxels far from the surface are generally predicted correctly quite early in the training and hence do not induce any gradients which can be used to teach the system where the surface lies. Therefore, with increasing resolution also the work done on computing gradients which are uninformative increases. The underlying reason why this problem exists is that current systems do not exploit the fact that surfaces are only two dimensional. Can we build a system that does?

In this paper we introduce a general framework, hierarchical surface prediction (HSP), for high resolution 3D object reconstruction which is organized around the observation that only a few of the voxels are in the vicinity of the object’s surface i.e. the boundary between free and occupied space, while most of the voxels will be “boring”, either completely inside or outside the object. The basic principle therefore is to only predict voxels around the surface, which we can now afford to do at higher resolution. The key insight to enable our method to achieve this is to change the standard prediction of free and occupied space into a three label prediction with the labels free space, boundary and occupied space. Furthermore, we do not directly predict high resolution voxels. Instead, we hierarchically predict small blocks of voxels from coarse to fine resolution in an octree, which we call voxel block octree. Thereby we have at each resolution the signal of the boundary label which tells us that the descendants of the current voxel will contain both, free space and occupied space. Therefore, only nodes containing voxels with the boundary label assigned need higher resolution prediction. By hierarchically predicting only those voxels we build up our octree. This effectively reduces the computational cost and hence allows us to predict significantly higher resolution voxel grids than previous approaches. In our experiments we predict voxels at a resolution of $256\times 256\times 256$ (cf. Fig. 1).

One of the fundamental questions which arises once higher resolution predictions are feasible, is whether or not a higher resolution prediction leads to more accurate results. In order to analyze this we compare our high resolution predictions to upsampled low resolution predictions in our quantitative evaluation and determine that our method outperforms these baselines. Our results are not only quantitatively more accurate, they are also qualitatively more detailed and have a higher surface quality.

The remainder of the paper is organized as follows. In Sec. II we discuss the related work. We then introduce our framework in Sec. III. Quantitative and qualitative evaluations are done in Sec. IV and eventually we draw the conclusions in Sec. V.

II Related Work

Traditionally dense 3D reconstruction from images has been done using a large collection of images. Geometry is extracted by dense matching or direct minimization of reprojection errors. The common methods can be broadly grouped as implicit volumetric reconstruction or explicit mesh based approaches. All these approaches facilitate a reconstruction of arbitrary geometry. However, in general a lot of input data is required to constrain the geometry enough. In difficult cases it is not always possible to recover the geometry of an object with multi-view stereo matching. This can be due to challenging materials such as transparent and reflective surfaces or lack of images from all around an object. For such cases object shape priors have been proposed .

All the methods mentioned above in general use multiple images as input data. Approaches which are primarily designed to reconstruct 3D geometry from a single color image were also studied. Vetter et al. propose to build a morphable shape model for faces, which allows for the reconstruction of faces from just a single image. In order to build such a model a lot of manual interaction and high quality scanning of faces is required. Other works propose to learn a shape model from silhouettes or silhouettes and keypoints .

Recently, CNNs have also been used to predict geometry from single images. Choy et al. propose to use an encoder decoder architecture in a recurrent network to facilitate prediction of coarse resolution voxel grids from a single or multiple color images. An alternative approach proposes to first train an autoencoder on the coarse resolution voxel grids and then train a regressor which predicts the shape code from the input image. It has also been shown that CNNs can be leveraged to predict alternative primitive based representations . While these approaches rely on ground truth shapes, CNNs can also be trained to predict 3D shapes using weaker supervision for example image silhouettes. These types of supervisions utilize ray formulations which originally have been proposed for multi-view reconstruction .

Predicting geometry from color images has also been addressed in a 2.5D setting where the goal is, given a color image to predict a depth map . These approaches can generally produce high resolution output but do only capture the geometry from a single viewpoint. Tatarchenko et al. propose to predict a collection of RGBD (color and depth) images from a single color image and fuse them in a post-processing step.

The volumetric approach has the major limitation that using high resolution voxel grids is computationally demanding and memory intensive. Multi-view approaches alleviate this, by the use of data adaptive discretization of the volume with Delaunay tedrahedrization , voxel block hashing or octrees . Our work is inspired by these earlier works on octrees. A crucial difference is that in these works the structure of the octree can be determined from the input geometry but in our case we are looking at a more general problem where the structure of the tree needs to be predicted. Similarly, a very recent work which proposes to use octrees in a CNN, also assumes that the structure of the octree is given as input. We predict the structure of the tree together with its content.

Our hierarchical surface prediction framework also relates to coarse-to-fine approaches in optical flow computation. Similar to volumetric 3D reconstruction also dense optical flow is computationally demanding due the the large label space. By computing optical flow hierarchically on a spatial pyramid, e.g. , real-time computation of optical flow is feasible . The benefits of using a coarse-to-fine approach in learning based optical flow prediction has also recently been pointed out .

III Formulation

In this Section we describe our hierarchical surface prediction framework, which allows for high resolution voxel grid predictions. We first briefly discuss how the state-of-the-art, coarse resolution baseline works and then introduce the voxel block octree structure which we are predicting.

The basic voxel prediction framework is adapted from . We consider an encoder/decoder architecture which takes an input $\mathcal{I}$, which in our experiments is either a color image, depth image or a partial low resolution voxel grid. A convolutional encoder encodes the input to a feature vector or shape code $\mathcal{C}$ which, in all our experiments, is a 128 dimensional vector. An up-convolutional decoder then decodes $\mathcal{C}$ into a predicted voxel grid $\mathcal{V}$. A labeling of the voxel space into free and occupied space can be determined using a suitable threshold $\tau$ (cf. Sec. IV-D) or alternatively a mesh can be extracted using marching cubes directly on the predicted voxel occupancies at an iso value $\sigma$.

The main problem which prevents us from directly utilizing this formulation with high resolutions for $\mathcal{V}$ is that each voxel, even if it is far from the surface, needs to be represented and a prediction is made for it. Given that the surface area grows quadratically and the volume cubically with respect to edge division, the ratio of surface to non-surface voxels becomes smaller with increasing resolution.

III-B Voxel Block Octree

In our hierarchical surface prediction method, we propose to predict a data structure with an up-convolutional decoder architecture, which we call ‘voxel block octree’. It is inspired from octree formulations used in traditional multi-view reconstruction approaches. The key insight which allows us to use such a data structure in a prediction framework is to extend the standard two label formulation to a three label formulation with labels inside, boundary and outside. As we will see later our data structure allows us to generate a complete voxel grid at high resolution while only making predictions around the surface. This leads to an approach which facilitates efficient training of an encoder/decoder architecture end-to-end.

An octree is a tree data structure which is used to partition the 3D space. The root node describes the cube of interest in the 3D space. Each internal node has up to 8 child nodes which describe the subdivision of the current node’s cube into the eight octants. Note that we slightly deviate from the standard definition of the octree where either none or all the 8 child nodes are present.

The predictions stored at any given level of the tree might be sparse. In order to reconstruct a complete model in high resolution, we upsample all the voxels which have not been predicted on the highest resolution from the closest predicted resolution. As we will see later, the voxels which get upsampled are in the inside and outside of the object not directly next to the boundary. Therefore the resolution of the actual surface remains high. In order to be able to extract a smooth high quality surface using marching cubes it is crucial that all the voxels which are close to the surface are predicted at high resolution and the predicted values are smooth. Therefore we aim to always evaluate the highest resolution around the boundary. At this point we would like to note that we can also extract a model at intermediate resolution by considering the boundary label as occupied space. This choice is consistent with the choice of considering any voxel as occupied space which intersects the ground truth mesh during voxelization (cf. Sec. IV-A).

III-C Network Architecture

In the previous section we introduced our voxel block octree data structure. We will now describe the encoder / decoder architecture which we are using to predict voxel block octrees using CNNs. An overview of our architecture is depicted in Fig. 2.

In the first level of the tree, which predicts the root node from the shape code $\mathcal{C}$, a small decoding network directly predicts the first feature block without a cropping module in between. Likewise, in the deepest level of the tree, no explicit feature block is needed. Therefore the output is directly generated from the cropped features of the previous level. Also note that the output and upsampling modules have their individual filters at each level of the tree. Furthermore, thanks to this architecture all the convolutions and up-convolutions are standard layers and no special versions for the octree are required. Detailed layer configurations can be found in the appendix.

III-D Efficient Training with Subsampling

As we have seen above, only predicting voxels on the boundary at each level largely reduces the complexity of the prediction task. In the beginning of the training, this is not the case because the boundaries are not yet predicted correctly. Moreover, even once the training has advanced enough and the boundaries are mostly placed correctly, an evaluation of the complete tree is still slow for training (approx. $0.7s$ for only a forward pass).

Therefore, we propose to utilize a subsampling of the child nodes during training. This is possible thanks to our hierarchical structure of the prediction. The tree gets traversed in a depth first manner. Each time the boundary label is present in an octant according to Eq. 1 the child node is traversed with a certain probability $\rho$. Different schedules for $\rho$ can be used. We experimented with both, having a fixed probability $\rho$ or start with a very low $\rho$ and gradually increase it during training. The first version leads to a simpler training procedure and was hence used in our experiments. Thanks to the depth first traversal the memory consumption (weights and filter responses) grows linearly with the number of levels if the same upsampling and output architecture is used for each level.

In order to be able to train with mini-batches we sum up all the gradients during traversal of the tree and only do a gradient step as soon as we have done a forward and backward traversal of the subsampled tree for the training examples of the whole mini-batch. As loss function we use Cross-Entropy.

IV Experiments

For our evaluation we use the synthetic dataset ShapeNetCore , which is commonly used to evaluate gemetry prediction networks. It is composed of Computer Aided Design (CAD) models of objects which are organized in categories. We use three categories for our evaluation, aeroplanes, chairs and cars. Our system is implemented in Torchhttp://torch.ch/ and we use Adam for stochastic optimization, with a mini-batch size of 4.

In a prepossessing step we voxelize the ground truth data. A common approach is to consider all the voxels which intersect the ground truth mesh as occupied space and then fill in the interior by labeling all voxels which are not reachable through free space from the boundary as occupied space. While this works well for the commonly used $32^{3}$ resolution it does not directly generalize to high resolutions. The reason is that the CAD models are generally not watertight meshes and with increasing resolution the risk that a hole in the mesh prevents filling the interior increases. In our case we are voxelizing the models at a resolution of $256^{3}$. In order to have filled interiors of the objects at high resolution we utilize a multi-scale approach. We first fill the interior at $32^{3}$ resolution. Then erode the occupied space by one voxel. We then fix every high resolution voxel which falls within this eroded low resolution occupied space as occupied space and also fix every high resolution voxel which falls within the original low resolution free space as free space. Furthermore, we fix all the high resolution voxels which intersect the original mesh surface as occupied space. To determine the ground truth label for the remaining high resolution voxels we run a graph-cut based regularization with a small smoothness term and a preference to keep the unlabeled voxels as free space.

Given the high resolution voxels at $256^{3}$ resolution we can now build the ground truth for all the remaining levels of the pyramid. At each level of the pyramid we label the voxels which contain the boundary at highest resolution as boundary and the other labels either free or occupied space, our coarsest resolution is $16^{3}$.

IV-B Baselines

We consider two baselines. Both of the baselines use the state-of-the-art coarse resolution voxel prediction framework from Sec. III-A. For both baselines we use a uniform prediction resolution of $32^{3}$ (except for one evaluation where we use $64^{3}$ resolution), they differ in the way the ground truth is computed. The first baseline follows the standard approach of labeling all voxels which intersect the ground truth mesh surface as occupied space and then fill in the interior. This can be achieved by downsampling our high resolution ground truth and label all low resolution voxels which contain at least one high resolution occupied space voxel as occupied space and all the other ones as free space. We call this baseline “Low Resolution Hard” (LR Hard). The other baseline uses a soft assignment for the low resolution, the label is given by the ratio of high resolution free to occupied space voxels that the low resolution voxel contains. Therefore these labels can have fractional assignments. We call this baseline “Low Resolution Soft” (LR Soft). Note that the baseline LR Soft makes use of the high resolution voxelization but the baseline LR Hard is equivalent to voxelizing at low resolution. As our goal is prediction of high resolution geometry we trininearly upsample the raw classification output of the baselines from $32^{3}$ to $256^{3}$ and conduct the evaluation at high resolution.

IV-C Input Data

To obtain the RGB/Depth images which are used as input for training and testing our CNNs, we render the CAD models in the ShapeNet dataset using Blenderhttp://www.blender.org. For each CAD model, we render 10 images from random viewpoints obtained via uniformly sampling azimuth from $[0,360)$ degrees and elevation from $$ degrees. We also use random lighting variations to render the RGB images.

We also use partial voxel grids as input. In practice such an input can arise when multiple depth maps of an object are available but they are all from the same side of the object. To imitate this task we use the ground truth of the LR Soft baseline and randomly zero the data in half of the voxel grid. The network then learns to predict the complete high resolution geometry from the partial low resolution input.

IV-D Quantitative Evaluation

We conducted a quantitative evaluation on the task of predicting high resolution geometry from a single RGB image. We assign each of the 3D models from the dataset randomly to the train, validation or test split with probabilities 70%, 10% and 20%, respectively. We trained category specific networks for our method Hierarchical Surface Prediction (HSP) and both baselines for the three categories, aeroplanes, chairs and cars. To quantify the accuracy we use use the Intersection over Union (IoU) score and the symmetric Chamfer Distance (CD). We compute the measures for each model and average over the dataset. To compute the measures we binarize the non-binary predictions with a suitable threshold. In order to determine the best threshold for each measure and category we test an exhaustive set of thresholds using the validation set. The plots given in Fig. 7 show that the choice of a good threshold has a very drastic influence to the error measure. As a first experiment, we study predictions at a coarse resolution of $64^{3}$. For this experiment both the baseline and HSP are trained to predict $64^{3}$ resolution and the evaluation is conducted at the same resolution. We demonstrate that our hierarchical prediction performs similar to the LR hard baseline i.e. the performance of HSP does not degrade compared to uniform prediction at the same resolution. For the baseline LR Hard we get an IoU of 43.47% and 43.12% for HSP, respectively on the validation set for the class chair. However, unlike HSP, the uniform prediction baselines cannot scale to finer resolutions and we now empirically show the benefits of being able to predict at finer resolutions. The evaluation is conducted at $256^{3}$ resolution. We compute the IoU and Chamfer distance for each of the models in the test set using the per category and per error measure best threshold determined using the validation set. The results are given in Table I. We outperform the baselines for all categories and evaluation metrics.

IV-E Qualitative Evaluation

In Figs. 8,9 and 10 we show example results on the task of geometry prediction from a single color image for the three categories. The results are selected to demonstrate the variablility of shapes within a category. Random results are shown in the appendix. The results are predicted with the same networks which we used in the quantitative analysis above. The meshes are extracted with marching cubes using the same thresholds as for the quantitative analysis using the Chamfer Distance. For chair the LR Hard baseline produces visually pleasant results. However, this same baseline does not achieve detailed results on the categories aeroplane and car. Due to the fractional labels the LR Soft baseline often misses thin structures. The results generated with our proposed HSP have high quality surfaces for all the three categories.

We also downloaded two images of cars with a white background from the Internet using Google image search and processed them with our car prediction network, which was trained only on synthetic data. As can be seen in Fig. 11, the network is able to generalize to this type of input. Additionally, we trained a network on the category car which takes a partial low resolution voxel grid as input and predicts a high resolution model of the whole object (c.f. Fig. 11).

V Conclusion and Future Work

In this work, we presented a hierarcical surface prediction framework which facilitates high resolution 3D reconstruction of objects from very little input data. We presented results on the task of predicting geometry from a single color image, a single depth image and from a partial low resolution voxel grid. Our quantitative evaluation shows that the high resolution prediction leads to more accurate results than low resolution baselines. Qualitatively, our predicted surfaces are superior to the low resolution baselines.

In this work we have demonstrated that high resolution geometry prediction is feasible therefore future work will investigate the potential to use prediction based reconstruction methods in multi-view stereo. Furthermore, for almost symmetric objects methods which explicitly take the symmetry into account are an interesting further direction which might improve the quality of prediction based approaches.

Acknowledgments

C. Häne received funding from the “Early Postdoc.Mobility” fellowship No. 165245 from the Swiss National Science Foundation. The project received funding from the Intel/NSF VEC award IIS-153909 and NSF award IIS-1212798.

References

Appendix A Detailed CNN Architectures

In this section we give the detailed layer configurations of our networks, including the baselines. We use the following abbreviations:

For both our proposed hierarchical surface prediction and the baselines we use the same encoder. The layers of the encoder for color or depth images are given in Tab. II. The encoder for the partial low resolution grids has the same general structure. The decoder for both the baselines is given in Tab. III. Note that both baselines use the exact same network. In one case the ground truth are hard binary labels and in the other case the ground truth are soft labels which represent the ratio of free/occupied space of the high resolution voxels, cf. main text.

Our proposed hierarchical surface prediction (HSP) uses a set of modules in the decoder. The module which decodes the shape code from the bottle neck to the first feature block is given in Tab. IV. As described in the main text each transition from one level to the next is done using an upsampling module Tab. V and an output module Tab. VI. On the last level the output is generated by the module given in Tab. VII.

Appendix B Qualitative Results

In order to demonstrate the quality of our method we show the reconstruction on randomly selected elements. We show every twentieth element of the validation set of the respective category. Category aeroplanes is depicted in Figs 12 and 13, category cars is depicted in Figs 14-16 and category chairs is depiced in Figs 17 and 18.