CSGNet: Neural Shape Parser for Constructive Solid Geometry

Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji

Introduction

In recent years, there has been a growing interest in generative models of 2D or 3D shapes, especially through the use of deep neural networks as image or shape priors . However, current methods are limited to the generation of low-level shape representations consisting of pixels, voxels, or points. Human designers, on the other hand, rarely model shapes as a collection of these individual elements. For example, in vector graphics modeling packages (Inkscape, Illustrator, and so on), shapes are often created through higher-level primitives, such as parametric curves (e.g., Bezier curves) or basic shapes (e.g., circles, polygons), as well as operations acting on these primitives, such as boolean operations, deformations, extrusions, and so on. The reason for choosing higher-level primitives is not incidental. Describing shapes with as few as possible primitives and operations is highly desirable for designers since it is compact, makes subsequent editing easier, and is perhaps better at capturing aspects of human shape perception such as view invariance, compositionality, and symmetry .

The goal of our work is to develop an algorithm that parses shapes into their constituent modeling primitives and operations within the framework of Constructive Solid Geometry (CSG) modeling as seen in Figure 1. This poses a number of challenges. First, the number of primitives and operations is not the same for all shapes i.e., our output does not have constant dimensionality, as in the case of pixel arrays, voxel grids, or fixed point sets. Second, the order of these operations matters. Figure 1 demonstrates an example where a complex object is created through boolean operations that combine simpler objects. If one performs a small change e.g., swap two operations, the resulting object becomes entirely different. From this aspect, the shape modeling process could be thought of as a visual program i.e., an ordered set of modeling instructions. Finally, a challenge is that we would like to learn an efficient parser that generates a compact program (e.g., with the fewest instructions) without relying on a vast number of shapes annotated with their programs for a target domain.

To tackle these challenges we designed a memory-enabled network architecture, that given a target 2D image of a shape, or a target 3D shape, generates a CSG program to generate it. To train our network we created a large synthetic dataset of automatically generated 2D and 3D programs. Networks trained on this dataset however lead to poor generalization when applied to new domains. To adapt models to new domains without program annotations we employ policy gradient techniques from the reinforcement learning literature . Combining our parser with a CSG rendering engine allows the parser to receive feedback based on the visual difference between the target shape and generated shape. Thus the parser network can be trained to minimize this difference.

Our contributions are as follows. First we show that the proposed architecture is efficient and effective at inferring CSG programs for 2D and 3D shapes across a number of domains. Second we show that the parser can be learned using reinforcement learning techniques on novel datasets without program annotations. Third, we show that the parser is a better and faster shape detector than state-of-the art detection approaches that only rely on bottom-up cues. We conjecture that this is because the parser jointly reasons about presence and ordering during parsing unlike the detector.

Related Work

Our work is primarily related to neural program induction methods. Secondly, it is also related to “vision-as-inverse-graphics” approaches, as well as neural network-based methods that predict shape primitives or parameters of procedural graphics models. Below, we briefly overview these prior methods, and explain differences from our work.

Our method is inspired by recent progress in neural network-based methods that infer programs expressed in some high-level language to solve a task. These methods often employ variants of recurrent neural networks whose parameters are trained to predict desired program outputs given exemplar inputs, such as answers to questions involving complex arithmetic, logical, or semantic parsing operations .

In the context of visual reasoning, several authors proposed architectures that produce programs composed of functions that perform compositional reasoning on the input image. They also incorporate an execution engine that produces the result of the program through a neural module network . In contrast our method aims to produce a generative program consisting of shape modeling functions that match a target image.

A well-known approach to visual analysis is to generate and fit hypotheses of scenes or objects to input image data i.e., perform analysis-by-syntesis . Kulkani et al. proposed sampling-based probabilistic inference to estimate parameters of stochastic graphics models (e.g., human body parameters, or parameters of rotationally symmetric objects) representing the space of hypothesized scenes given an input image. Shape grammars (or so-called inverse procedural modeling techniques) have alternatively been used in analysis-by-synthesis image parsing frameworks , yet they have the disadvantage of not modeling long-range dependencies in the parsing task, and are often specific to a particular shape class (e.g., buildings). More recent approaches employ Convolutional Neural Network (CNN) to infer parameters of objects or whole scenes . A similar trend is observed in graphics applications where CNNs are used to map input images or partial shapes to procedural model parameters . Wu et al. detect objects in scenes by employing a network for producing object proposals and a network that predicts whether there is an object in a proposed segment, along with various object attributes. Eslami et al. uses a recurrent neural network to attend to one object at a time in a scene, and learn to use an appropriate number of inference steps to recover object counts, identities and poses.

In contrast, we do not aim at parsing images or scenes into a collection of objects and their parameters. We instead parse input images or 3D shapes into a sequence of modeling operations on primitives (i.e, a visual program) to match a target image. In our setting, the space of outputs is much larger and the order of operations in our visual programs matter. To deal with this complexity, we use a combination of supervised pretraining, reinforcement learning, reward design, and post-optimization of modeling parameters, described in the next Section.

Tulsiani et al. proposed a volumetric convolutional network architecture that predicts a fixed number of cuboidal primitives to describe an input 3D shape. To better handle a variable number of primitives, Zou et al. instead proposed an LSTM-based architecture that predicts boxes given input depth images. We also aim at deriving geometrically interpretable explanations of shapes in terms of primitives. However, our network is not limited to predicting a single type of primitives (e.g., cubes), but also outputs modeling operations acting on them, or in other words supports a significantly richer modeling paradigm. The program can be used not only to geometrically describe the input shape but can also be directly edited to manipulate it if desired. Finally, Ellis et al. proposed a neural network architecture to extract various hand-drawn primitives (lines, circles, rectangles) in images, which are then grouped into Latex programs. Their program synthesis is posed as a constraint satisfaction problem which is computationally expensive and can take hours to solve. Instead, our program is created by a neural network that takes a fraction of a second to evaluate at test time.

Our work is related to approaches for shape parsing using grammars . These have been applied to objects that can be represented using tree-structured grammars (e.g., human bodies, buildings). However such approaches often use shallow grammars or accurate bottom-up proposals (e.g., face and limb detection) to guide parsing. In the context of CSG, primitive detection is challenging as shapes change significantly when boolean operations are applied to them. Parse trees for CSG also tend to be deeper. As a result, bottom-up parsing becomes computationally expensive since the complexity scales exponentially with the program length.

Designing a Neural Shape Parser

In this section, we first present our neural shape parser that can induce programs for 2D/3D shapes. The goal of the parser $\pi$ is to produce a sequence of instructions given an input shape. The parser can be implemented as an encoder-decoder using neural network modules as shown in Figure 2. The encoder takes as input an image $I$ and produces an encoding $\Phi(I)$ using a CNN. The decoder $\Theta$ takes as input $\Phi(I)$ and produces a probability distribution over programs $P$ represented as a sequence of instructions. Decoders can be implemented using Recurrent Neural Networks (RNNs). We employ Gated Recurrent Units (GRUs) that have been widely used for sequential prediction tasks such as generating natural language and speech. The overall network can be written as $\pi(I)=\Theta\circ\Phi(I)$ . The space of programs can be efficiently described according to a context-free grammar . For example, in constructive solid geometry the instructions consist of drawing primitives (e.g., spheres, cubes, cylinders, etc.) and performing boolean operations described as a grammar with the following production rules:

Each rule indicates possible derivations of a non-terminal symbol separated by the $|$ symbol. Here $S$ is the start symbol, $\texttt{OP}_{i}$ is chosen from a set of defined modeling operations and the $\texttt{SHAPE}_{i}$ is a primitive chosen from a set of basic shapes at different positions, scales, orientations, etc. Instructions can be written in a standard post-fix notation, e.g. $\texttt{SHAPE}_{1}\texttt{SHAPE}_{2}\texttt{OP}_{1}\texttt{SHAPE}_{3}\texttt{OP}_{2}$ . Figure 2 also gives an example of a program predicted by the network, that follows the grammar described above.

Given an input $I$ the parser network $\pi$ generates a program that minimizes a reconstruction error between the shape produced by executing the program and a target shape. Note that not all programs are valid hence the network must also learn to generate grammatical programs.

When target programs are available the architecture can be trained with standard supervised learning techniques. Training data in this case consists of shape and program pairs $(I^{i},P^{i}),i=1,\ldots,N$ . In our implementation, the RNN produces a categorical distribution $\pi_{\theta}$ over instructions $a\in A$ at every time step. Similarly the ground-truth program $P^{i}$ can be written as sequence of instructions $g^{i}_{1}$ , $g^{i}_{2}$ .. $g^{i}_{T_{i}}$ , where $T_{i}$ is the length of the program $P^{i}$ . The parameters $\theta$ can be learned to maximize the log-likelihood of the ground truth instructions:

Without target programs one can minimize a reconstruction error between the shape obtained by executing the program and the target. However, directly minimizing this error using gradient-based techniques is not possible since the output space is discrete and execution engines are typically not differentiable. Policy gradient techniques from the reinforcement learning (RL) literature can instead be used in this case.

Concretely, the parser $\pi_{\theta}$ , that represents a policy network, can be used to sample a program $y$ = ( $a_{1}$ , $a_{2}$ .. $a_{T}$ ) conditioned on the input shape $I$ . Then a reward $R$ can be estimated by measuring the similarity between the generated image $\hat{I}$ obtained by executing the program and the target shape $I$ . With this setup, we want to learn the network parameters $\theta$ that maximize the expected rewards over programs sampled under the predicted distribution $\pi_{\theta}(I)$ across images $I$ sampled from a distribution ${\cal D}$ :

The outer expectation can be replaced by a sample estimate on the training data. The gradient of the inner expectation can be obtained by rearranging the equation as:

It is often intractable to compute the expectation $J_{\theta}(I)$ since the space of programs is very large. Hence the expectation must be approximated. The popular REINFORCE algorithm computes a Monte-Carlo estimate as:

by sampling $S$ programs from the policy $\pi_{\theta}$ . Each program $y^{s}$ is obtained by sampling instructions $\hat{a}^{s}_{t=1\mathrel{\mathop{\mathchar 58\relax}}T}$ from the distribution $\hat{a}_{t}^{s}\sim\pi_{\theta}(a_{t}|\hat{a}^{s}_{1\mathrel{\mathop{\mathchar 58\relax}}t-1};I)$ at every time step $t$ , till the stop symbol (EOS) is sampled. The reward $R^{s}$ is calculated by executing the program $y^{s}$ . Sampling-based estimates typically have high variance that can be reduced by subtracting a baseline without changing the bias as:

A good choice of the baseline is the expected value of returns starting from $t$ . We compute baseline as the running average of past rewards.

The rewards should be primarily designed to encourage visual similarity of the generated program with the target. Visual similarity between two shapes is measured using the Chamfer distance (CD) between points on the edges of each shape. The CD is between two point sets, $\mathbf{x}$ and $\mathbf{y}$ , is defined as follows:

The points are scaled by the image diagonal, thus $Ch(\mathbf{x},\mathbf{y})\in~{}\forall\mathbf{x},\mathbf{y}$ . The distance can be efficiently computed using distance transforms. In our implementation, we also set a maximum length $T$ for the induced programs to avoid having too long or redundant programs (e.g., repeating the same modeling instructions over and over again). We then define the reward as:

where $f$ is a shaping function and $Z$ is the CSG rendering engine. Since invalid programs get zero reward, the maximum length constraint on the programs encourages the network to produce shorter programs with high rewards. We use maximum length $T=13$ in all of our RL experiments. The function $f$ shapes the CD as $f(x)=(1-x)^{\gamma}$ with an exponent $\gamma>0$ . Higher values of $\gamma$ encourages CD close to zero. We found that $\gamma=20$ provides a good trade-off between program length and visual similarity.

2 Inference

Estimating the most likely program given an input is intractable using RNNs. Instead one usually employs a greedy decoder that picks the most likely instruction at each time step. An alternate is to use a beam search procedure that maintains the k-best likely sequences at each time step. In our experiments we report results with varying beam sizes.

Our parser produces a program with a discrete set of primitives. However, further refinement can be done by directly optimizing the position and size of the primitives to maximize the reward. The refinement step keeps the program structure of the program and primitive type fixed but uses a heuristic algorithm to optimize the parameters using feedback from the rendering engine. On our dataset where shapes have up to $7$ primitives, the search space is relatively small and the algorithm converges to a local minima in about $10$ iterations and consistently improves the results.

Experiments

We describe our experiments on different datasets exploring the generalization capabilities of our network (CSGNet). We first describe our datasets: (i) an automatically generated dataset of 2D and 3D shapes based on synthetic generation of CSG programs, (ii) 2D CAD shapes mined from the web where ground-truth programs are not available, and (iii) logo images mined also from the web where ground-truth programs are also not available. We discuss our qualitative and quantitative results on the above datasets.

To train our network in the supervised learning setting, we automatically created a large set of 2D and 3D CSG-based synthetic programs according to the grammars described below.

We sampled derivations of the following CSG grammar to create our synthetic dataset in the 2D case:

Primitives are specified by their type: square, circle, or triangle, locations $L$ and circumscribing circle of radius $R$ on a canvas of size $64\times 64$ . There are three boolean operations: $intersect$ , $union$ , and $subtract$ . L is discretized to lie on a square grid with spacing of $8$ units and R is discretized with spacing of $4$ units. The triangles are assumed to be upright and equilateral. The synthetic dataset is created by sampling random programs containing different number of primitives from the above grammar, constraining the distribution of various primitive types and operation types to be uniform. We also ensure that no duplicate programs exist in our dataset. The primitives are rendered as binary images and the programs are executed on a canvas of $64\times 64$ pixels. Samples from our dataset are shown in Figure 3. Table 1 provides details about the size and splits of our dataset.

We sampled derivations of the following grammar in the case of 3D CSG:

The same three binary operations are used as in the 2D case. Three basic solids are denoted by ‘ $sp$ ’: Sphere, ‘ $cu$ ’: Cube, ‘ $cy$ ’: Cylinder. $L$ represents the center of primitive in 3D voxel grid. $R$ specifies radius of sphere and cylinder, and also specifies size of cube. $H$ is the height of cylinder. The primitives are rendered as voxel grids and the programs are executed on a 3D volumetric grid of size $64$ $\times$ $64$ $\times$ $64$ . We used the same random sampling method as described for the synthetic 2D dataset, resulting in 3D CSG programs. 3D shape samples from this dataset are shown in Figure 3.

We collected $8K$ CAD shapes from the Trimble 3DWarehouse dataset in three categories: chair, desk and lamps. We rendered the CAD shapes into $64$ $\times$ $64$ binary masks from their front and side views. In Section 4, we show that the rendered shapes can be parsed effectively through our visual program induction method. We split this dataset into $5K$ shapes for training, $1.5K$ validation and $1.5K$ for testing.

We mined a collection of binary logos from the web that can be modeled using the primitives in our output shapes. We test our approach on these logos without further training or fine-tuning our net on this data.

2 Implementation details

The input 2D or 3D shape $I$ is represented as pixel and voxel occupancy grid respectively. Our encoder is based on an image-based convnet in the case of 2D inputs, and a volumetric convnet in the case of 3D inputs. The output of the encoder $\Phi(I)$ is passed as input to our GRU-based decoder at every program step. The hidden state of our GRU units is passed through two fully-connected layers, which are then converted into a probability distribution over program instructions through a classification layer. For the 2D CSG there are $400$ unique instructions corresponding to $396$ different primitive types, discrete locations and sizes, the $3$ boolean operations and the stop symbol. For the 3D CSG there are $6635$ unique instructions with $6631$ different types of primitives with different sizes and locations, plus $3$ boolean modeling operations and a stop symbol. During training, on synthetic dataset, we sample images rendered from programs of variable length (up to $13$ for 2D and up to $7$ for 3D dataset) from training dataset. More details about the architecture of our encoder and decoder (number and type of layers) are provided in the supplementary material.

For supervised learning, we use the Adam optimizer with learning rate $0.001$ and dropout of $0.2$ in non-recurrent network connections. For reinforcement learning, we use stochastic gradient descent with $0.9$ momentum, $0.01$ learning rate, and with the same dropout as above. Our implementation is based on PyTorch . Our source code and datasets are available on our project page: https://hippogriff.github.io/CSGNet.

3 Results

We evaluate our network, called CSGNet, in two different ways: (i) as a model for inferring the entire program, and (ii) as model for inferring primitives, i.e., as an object detector.

We perform supervised learning to train CSGNet on the training split of this synthetic dataset, and evaluate performance on its test split under different beam sizes. We compare with a baseline that retrieves a program in the training split using a Nearest Neighbor (NN) approach. In NN setting, the program for a test image is retrieved by taking the program of the train image that is most similar to the test image. Table 2 compares CSGNet to this NN baseline using the Chamfer distance between the test target and predicted shapes. Our parser is able to outperform the NN method. One would expect that NN would perform well here because the size of the training set is large. However, our results indicate that our compositional parser is better at capturing shape variability, which is still significant in this dataset. Results are also shown with increasing beam sizes (k) during decoding, which consistently improves performance. Figure 4 also shows the programs retrieved through NN and our generated program for a number of characteristic examples in our test split of our synthetic dataset.

For this dataset, we report results on its test split under two conditions: (i) when training our network only on synthetic data, and (ii) when training our network on synthetic data and also fine-tuning it on the training split of 2D CAD dataset using policy gradients.

Table 3 shows quantitative results on this dataset. We first compare with the NN baseline. For any shape in this dataset, where ground truth program is not available, NN retrieves a shape from synthetic dataset and we use the ground truth program of the retrieved synthetic shape for comparison. We then list the performance of CSGNet trained in supervised manner only on our synthetic dataset. With beam search, the performance of this variant improves compared to NN. Most importantly, further training with Reinforcement Learning (RL) on the training split of the 2D CAD dataset improves the results significantly and outperforms the NN approach by a considerable margin. This also shows the advantage of using RL, which trains the shape parser without ground-truth programs. We note that directly training the network using RL alone does not yield good results which suggests that the two-stage learning (supervised learning and RL) is important. Finally, optimizing the best beam search program with visually guided refinement yielded results with the smallest Chamfer Distance. Figure 5 shows a comparison of the rendered programs for various examples in the test split of the 2D CAD dataset for variants of our network. Visually guided refinement on top of beam search of our two stage-learned network qualitatively produces results that best match the input image.

Here, we experiment with the logo dataset described in Section 4.1 (none of these logos participate in training). Outputs of the induced programs parsing the input logos are shown in Figure 6. In general, our method is able to parse logos into primitives well, yet performance can degrade when long programs are required to generate them, or when they contain shapes that are very different from our used primitives.

Finally, we show that our approach can be extended to 3D shapes. In the 3D CSG setting, we train a 3D-CNN + GRU (3D-CSGNet) network on the 3D CSG synthetic dataset explained in Section 4.1. The input to our 3D-CSGNet are voxelized shapes in a $64<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>×</mo></mrow><annotation encoding="application/x-tex">\times</annotation></semantics></math>×64<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo>×</mo></mrow><annotation encoding="application/x-tex">\times</annotation></semantics></math>×64$ grid. Our output is a 3D CSG program, which can be rendered as a high-resolution polygon mesh (we emphasize that our output is not voxels, but CSG primitives and operations that can be computed and rendered accurately). Figure 7 show pairs of input voxel grids and our output shapes from the test split of the 3D dataset. The qualitative results are shown in the Table 4, where we compare our 3D-CSGNet at different beam search decodings with NN method. The results indicate that our method is promising in inducing correct programs, which also have the advantage of accurately reconstructing the voxelized surfaces into high-resolution surfaces.

3.2 Primitive detection

Successful program induction for a shape requires not only predicting correct primitives but also correct sequences of operations to combine these primitives. Here we evaluate the shape parser as a primitive detector (i.e., we evaluate the output primitives of our program, not the operations themselves). This allows us to directly compare our approach with bottom-up object detection techniques.

In particular we compare against a state-of-the-art object detector (Faster R-CNNs ). The Faster R-CNN is based on the VGG-M network and is trained using bounding-box and primitive annotations based on our 2D synthetic training dataset. At test time the detector produces a set of bounding boxes with associated class scores. The models are trained and evaluated on 640 $\times$ 640 pixel images. We also experimented with bottom-up approaches for primitive detection based on Hough transform and other rule-based approaches. However, our experiments indicated that the Faster R-CNN was considerably better.

For a fair comparison, we obtain primitive detections from CSGNet trained on the 2D synthetic dataset only (same as the Faster R-CNN). To obtain detection scores, we sample $k$ programs with beam-search decoding. The primitive score is the fraction of times it appears across all beam programs. This is a Monte Carlo estimate of our detection score.

The accuracy can be measured through standard evaluation protocols for object detection (similar to those in the PASCAL VOC benchmark). We report the Mean Average Precision (MAP) for each primitive type using an overlap threshold between the predicted and the true bounding box of $0.5$ intersection-over-union. Table 5 compares the parser network to the Faster R-CNN approach.

Our parser clearly outperforms the Faster R-CNN detector on the squares and triangles category. With larger beam search, we also produce slighly better results for circle detection. Interestingly, our parser is considerably faster than Faster R-CNN tested on the same GPU.

Conclusion

We believe that our work represents a first step towards the automatic generation of modeling programs given target visual content, which we believe is quite ambitious and hard problem. We demonstrated results of generated programs in various domains, including logos, 2D binary shapes, and 3D CAD shapes, as well as an analysis-by-synthesis application in the context of 2D shape primitive detection.

One might argue that the 2D images and 3D shapes our method parsed are relatively simple in structure or geometry. However, we would also like to point out that even in this ostensibly simple application scenario (i) our method demonstrates competitive or even better results than state-of-the-art object detectors, and most importantly (ii) the problem of generating programs was far from trivial to solve: based on our experiments, a combination of memory-enabled networks, supervised and RL strategies, along with beam and local exploration of the state space all seemed necessary to produce good results. As future work, a challenging research direction would be to generalize our approach to longer programs with much larger spaces of parameters in the modeling operations and more sophisticated reward functions balancing perceptual similarity to the input image and program length. Other promising directions would be to explore how to combine bottom-up proposals and top-down approaches for parsing shapes, in addition to exploring top-down program generation strategies.

Acknowledgments. We acknowledge support from NSF (CHS-1422441, CHS-1617333, IIS-1617917) and the MassTech Collaborative grant for funding the UMass GPU cluster.

References

Supplementary

In this supplementary material, we include the following topics in more detail: a) synthetic dataset creation in the 2D and the 3D case, b) neural network architecture used in our experiments, c) more qualitative results on our test dataset.

1 Dataset

We use the grammar described in the Section $4.1$ to create our 2D dataset. The dataset is created by randomly generating programs of lengths $3$ to $13$ following the grammar. While generating these programs we impose additional restrictions as follows: a) Primitives must lie completely inside the canvas, b) Each operation changes the number of ON pixels by at least a threshold set to $10\%$ of sum of pixels in two shapes. This avoids spurious operations such as subtraction between shapes with little overlap. c) The number of ON pixels in the final image is above a threshold. d) The previous rules promotes programs with the $union$ operation. To ensure a balanced dataset we boost the probabilities of generating programs with $subtract$ and $intersect$ operations. Finally we remove duplicates. We only use upright, equilateral triangles and upright squares. Note that locations (L) are discretized to lie on square grid with spacing of $8$ units and size (R) are discretized with spacing of $4$ units. Figure 8 shows examples from our dataset.

We use the grammar described in the Section $4.1$ to create our 3D dataset. While generating shapes we followed a strategy similar to the 2D case. For 3D case, we only use programs of up to length $7$ (up to $4$ shape primtives and upto $3$ boolean operations). Note that the cube and cylinder are upright. The dataset contains $64\times 64\times 64$ voxel-grid shapes and program pairs. Also note that locations (L) are discretized to lie on cubic grid with spacing of $8$ units, and size (R) and height (H) are discretized with spacing of $4$ units.

We implemented a CSG engine that reads the instructions one by one. If it encounters a primitive (e.g. c(32, 32, 16)) it draws it on an empty canvas and pushes it on to a stack. If it encounters an operation (e.g. union, intersect, or subtract) it pops the top two canvases on its stack, applies the operation to them, and pushes the output to the top of the stack. The execution stops when no instructions remain at which point the top canvas represents the result. The above can be seen as a set of shift and reduce operations in a LR-parser . Figure 9 describes execution procedure to induce programs for 3D shapes.

2 Network Architecture

Table 6 shows the CNN architecture used as the encoder. The input $I$ is an image of size $64\times 64$ and output $\Phi(I)$ is a vector of size $2048$ . Table 7 describes the architecture used in the decoder. The RNN decoder is based on a GRU unit that at every time step takes as input the encoded feature vector and previous instruction encoded as a $128$ dimensional vector obtained by a linear mapping of the $401$ dimensional one-hot vector representation. At first time step, the previous instruction vector represents the START symbol. Embedded vector of previous instruction is concantenated with $\Phi(I)$ and is input to the GRU. The hidden state of GRU is passed through two dense layer to give a vector of dimension $400$ , which after softmax layer gives a probability distribution over instructions. The output distribution is over $396$ different shape primitives, $3$ operations (intersect, union and subtract) and a STOP. We exclude the START symbol from the output probability distribution. Note that the circle, triangle or square at a particular position in the image and of a particular size represents an unique primitive. For example, $c(32,32,16)$ , $c(32,28,16)$ , $s(12,32,16)$ are different shape primitives.

Input to 3D shape encoder (3DCNN) is a voxel grid $I$ of size $64$ x $64$ x $64$ and outputs an encoded vector $\Phi(I)$ of size $2048$ , as shown in the Table 8. Similar to the 2D case, at every time step, GRU takes as input the encoded feature vector and previous ground truth instruction. The previous ground truth instruction is a $6636$ -dimensional (also includes the start symbol) one-hot vector, which gets converted to a fixed $128$ -dimensional vector using a learned embedding layer. At first time step the last instruction vector represents the START symbol. Embedded vector of previous instruction is concatenated with $\Phi(I)$ and is input to the GRU. The hidden state of GRU is passed through two dense layers to give a vector of dimension $6635$ , which after Softmax layer gives a probability distribution over instructions. The output distribution is over $6631$ different shape primitives, $3$ operations (intersect, union and subtract) and a STOP. We exclude the START symbol from the output probability distribution. Similar to 2D case, $cu(32,32,16,16)$ , $cu(32,28,16,12)$ , $sp(12,32,16,28)$ are different shape primitives. Table 9 shows details of decoder.

3 Qualitative Evaluation

In this section, we show more qualitative results on different dataset. We first show peformance of our CSGNet trained using only Supervised learning on 2D synthetic dataset, and we compare top-10 results from nearest neighbors and and top-10 results from beam search, refer to the Figure 10 and 11. Then we show performance of our full model (using RL + beam search + visually guided search) on CAD 2D shape dataset, refer to the Figure 12 and 13.