Generative Neurosymbolic Machines

Introduction

Two central abilities in human and machine intelligence are to learn abstract representations of the world and to generate imaginations in such a way to reflect the causal structure of the world. Deep latent variable models like variational autoencoders (VAEs) offer an elegant probabilistic framework to learn both these abilities in an unsupervised and end-to-end trainable fashion. However, the single distributed vector representations used in most VAEs provide in practice only a weak or implicit form of structure induced by the independence prior. Therefore, in representing complex, high-dimensional, and structured observations such as a scene image containing various objects, the representation is rather difficult to express useful structural properties such as modularity, compositionality, and interpretability. These properties, however, are believed to be crucial in resolving limitations of current deep learning in various System 2 related abilities such as reasoning , causal learning , accountability , and systematic out-of-distribution generalization .

There have been remarkable recent advances in resolving this challenge by learning to represent an observation as a composition of its entity representations, particularly in an object-centric fashion for scene images . Equipped with more explicit inductive biases such as spatial locality of objects, symbolic representations, and compositional scene modeling, these models provide a way to recognize and generate a given observation via the composition of interacting entity-based representations. However, most of these models do not support the other crucial ability of a generative model: generating imaginary observations by learning the density of the observed data. Although this ability to imagine according to the density of the possible worlds plays a crucial role, e.g., in world models required for planning and model-based reinforcement learning , most previous entity-based models can only synthesize artificial images by manually configuring the representation, but not according to the underlying observation density. Although this ability is supported in VAEs , lacking an explicitly compositional structure in its representation, it easily loses in practice the global structure consistency when generating complex images .

In this paper, we propose Generative Neurosymbolic Machines (GNM), a probabilistic generative model that combines the best of both worlds by supporting both symbolic entity-based representations and distributed representations. The model thus can represent an observation with symbolic compositionality and also generate observations according to the underlying density. We achieve these two crucial properties simultaneously in GNM via a two-layer latent hierarchy: the top layer generates the global distributed latent representation for flexible density modeling and the bottom layer yields from the global latent the latent structure map for entity-based and symbolic representations. Furthermore, we propose StructDRAW, an autoregressive prior supporting structured feature-drawing to improve the expressiveness of latent structure maps. In experiments, we show that for both the structure accuracy and image clarity, the proposed model significantly outperforms the previous structured representation models as well as highly-expressive non-structured generative models.

Symbolic and Distributed Representations in Latent Variable Models

Variable binding and value inference. What functions are a representation learning model based on an autoencoder (e.g., VAE) performing? To answer this, we provide a perspective that separates the function of the encoder into two: variable binding and value inference. Variable binding (or grounding) is to assign a specific role to a variable (or a group of variables) in the representation vector. For instance, in VAEs each variable in the latent vector is encouraged to have its own meaning through the independence prior. In an ideal case with perfect disentanglement, we would expect to find a specific variable in the latent vector that is in charge of controlling $x$-coordinate of an object in an image . That is, the variable is grounded on the object’s position. However, in practice, such perfect disentanglement is difficult to achieve , and thus the representation shows correlations among the values in it. Value inference is to assign a specific value to the binded variable, e.g., in our example, a coordinate value. In VAE, the variable binding is fixed after training—the same variable represents the same semantics for different inputs—but the inferred value can be changed per observation (e.g., if the object position changes). In VAEs, both variable binding and value inference are learned jointly.

Distributed vs. Symbolic Representations. We define a symbolic representation as a latent variable to which a semantic role is solely assigned independently to other variables. For example, in object-centric latent variable models , a univariate Gaussian distribution $p(z_{x}^{\text{where}})={\cal N}(\mu_{x},\sigma_{x})$ can be introduced to define a symbolic prior on the $x$-coordinate of an object in an image. (Then, the final $x$-coordinate can be computed by $I_{x}\times\text{sigmoid}(z_{x}^{\text{where}})$ with $I_{x}$ the image width.) On the contrary, in distributed representations, variable binding can be distributed. That is, a semantic variable can be represented in a distributed way across the whole latent vector with correlation among the vector elements. A single Gaussian latent vector of the standard VAE is a representative example. Although VAEs objective encourages the disentanglement of each variable, it is, in general, more difficult to achieve such complete disentanglement than symbolic representations.

Distributed latent variable models (D-LVM) in general provides more flexibility than symbolic latent variable models (S-LVM) as the variable binding can be distributed and, more importantly, learned from data. This learnable binding allows turning the prior latent distribution into the distribution of complex high-dimension observations. In S-LVMs, such flexibility can be significantly limited to representing the semantics of the fixed and interpretable binding. For instance, if we introduce a prior on a symbolic variable representing the number or positions of objects in an image but in a way that does not match the actual data distribution, S-LVMs cannot fix this to generate according to the observed data distribution. However, S-LVM brings various advantages that are, in general, more difficult to be achieved in D-LVMs. The completely disentangled symbols facilitate interpretability, reasoning, modularity, and compositionality. Also, since the encoder only needs to learn value inference, learning can be facilitated. See Fig. 1 (a)-(c) for an illustration.

Object-Centric Representation Learning. There are two main approaches to this. The bounding-box models infer object appearances along with their bounding boxes and reconstruct the image by placing objects according to their bounding boxes. Scene-mixture models try to partition the image into several layers of images, potentially one per object, and reconstruct the full image as a pixel-wise mixture of these layered images. The bounding-box models utilize many symbolic representations such as the number of objects, and positions and sizes of the bounding boxes. Thus, while entertaining various benefits of symbolic representation, it also inherits the above-mentioned limitations, and thus currently no bounding-box model can generate according to the density of the data. Scene-mixture models such as rely less on symbolic representations as each mixture component of a scene is generated from a distributed representation. However, these models also do not support the density-based generation as the mixture components are usually independent of each other. Although GENESIS has an autoregressive prior on the mixture components and thus can support the density-aware generation in principle, our experiment results indicate limitations of the approach.

Generative Neurosymbolic Machines

We formulate the generative process of the proposed model as a simple two-layer hierarchical latent variable model. In the top layer, we generate a distributed representation ${\bf z}^{g}$ from the global prior $p({\bf z}^{g})$ to capture the global structure with the flexibility of the distributed representation. From this, the structured latent representation ${\bf z}^{s}$ containing symbolic representations is generated in the next layer using the structuring prior $p({\bf z}^{s}|{\bf z}^{g})$. The observation is constructed from the structured representation using the rendering model $p({\bf x}|{\bf z}^{s})$. With ${\bf z}=\{{\bf z}^{g},{\bf z}^{s}\}$, we can write this as

Global Representation. The global representation ${\bf z}^{g}$ provides the flexibility of the distributed representation. That is because the meaning of a representation vector is distributed and not predefined but endowed later by learning from the data, it allows complex distributions (e.g., highly multimodal and correlated distribution on the number of objects and their positions in a scene) to be modeled with the representation. In this way, the global representation ${\bf z}^{g}$ contains an abstract and flexible summary necessary to generate the observation but lacks an explicit compositional and interpretable structure. Importantly, the role of the global representation in our model is different from that in VAE. Instead of directly generating the observation from this distributed representation by having $p({\bf x}|{\bf z}^{g})$, it acts as high-level abstraction serving for constructing a structured representation ${\bf z}^{s}$, called the latent structure map, via the structuring model $p_{\theta}({\bf z}^{s}|{\bf z}^{g})$. A simple choice for the global representation is a multivariate Gaussian distribution ${\cal N}({\bf 0},{\bf 1}_{d_{g}})$.

Structured Representation. In the latent structure map, variables are explicitly and completely disentangled into a set of components. To obtain this in the image domain, we first build from the global representation ${\bf z}^{g}$ a feature map ${\bf f}$ of $(H\times W\times d_{f})$-dimension with $H$ and $W$ being the spatial dimension and $d_{f}$ being the feature dimension in each spatial position. Thus, $H$ and $W$ are hyperparameters controlling the maximum number of components and usually a much smaller number (e.g., $4\times 4$) than the image resolution. Then, for each feature vector ${\bf f}_{hw}$, a component latent ${\bf z}_{hw}^{s}$ of the latent structure map is inferred. Depending on applications, ${\bf z}_{hw}^{s}$ can be a set of purely symbolic representations or a hybrid of symbolic and distributed representations.

For multi-object scene modeling, which is our main application, we use a hybrid representation $\smash{{\bf z}_{hw}^{s}=[{\bf z}_{hw}^{\text{pres}},{\bf z}_{hw}^{\text{where}},{\bf z}_{hw}^{0}pt,{\bf z}_{hw}^{\text{what}},]}$ to represent the presence, position, depth, and appearance of a component, respectively. Here, appearance ${\bf z}_{hw}^{\text{what}}$ is a distributed representation while the others are symbolic. We use Bernoulli distributions for presence and Gaussian distributions for the others. We also introduce the background component ${\bf z}^{b}$, which represents a part of the observation remained after the explanation by the other foreground components. We can consider the background as a special foreground component for which we only need to learn the appearance while fixing the other variables constant. Then, we can write the structuring model as follows:

where ${\bf z}^{s}={\bf z}^{b}\cup\{{\bf z}_{hw}^{s}\}$ and $p_{\theta}({\bf z}_{hw}^{s}|{\bf z}^{g})=p_{\theta}({\bf z}_{hw}^{\text{pres}}|{\bf z}^{g})p_{\theta}({\bf z}_{hw}^{\text{what}}|{\bf z}^{g})p_{\theta}({\bf z}_{hw}^{\text{where}}|{\bf z}^{g})p_{\theta}({\bf z}_{hw}^{0}pt|{\bf z}^{g})$.

The latent structure map might look similar to that in SPACE . However, in SPACE, independent symbolic priors are used to obtain scalability, and thus it cannot model the underlying density. Unlike SPACE, the proposed model generates the latent structure map from the global representation, which is distributed and groundable (binding learnable). This is crucial because by doing so, we achieve both flexible density modeling and benefits of symbolic representations. Unlike the other models , this approach is also efficient and stable for object-crowded scenes .

Renderer. Our model adopts the typical renderer module $p({\bf x}|{\bf z}^{s})$ used in bounding-box models, e.g., SPACE. We provide the implementation details of the renderer in Appendix.

2 StructDRAW

One limitation of the above model is that the simple Gaussian prior for $p({\bf z}^{g})$ may not have enough flexibility to express complex global structures of the observation, a well-known problem in VAE literature . One way to resolve this problem is to generate the image autoregressively at pixel-level , or by superimposing several autoregressively-generated sketches on a canvas . However, these approaches cannot be adopted in GNM as they generate images directly from the global latent without structured representation.

3 Inference

For inference, we approximate the intractable posterior by the following mean-field decomposition:

As shown, our model provides dual representations for an observation ${\bf x}$. That is, the global latent ${\bf z}^{g}$ represents the scene as a flexible distributed representation, and the structured latents $\smash{{\bf z}^{s}}$ provides a structured symbolic representation of the same observation.

Image Encoder. As all modules take ${\bf x}$ as input, we share an image encoder $f_{\text{enc}}$ across the modules. The encoder is a CNN yielding an intermediate feature map ${\bf f}^{x}=f_{\text{enc}}({\bf x})$.

Component Encoding. The component encoder $q_{\phi}({\bf z}_{hw}^{s}|{\bf x})$ takes the feature map ${\bf f}^{x}$ as input to generate the background and the component latents in a similar way as done in SPACE except that the background is not partitioned. We found that conditioning the foreground ${\bf z}^{s}$ on the background ${\bf z}^{b}$ (or vice versa) does not help much because if both modules are learned simultaneously from scratch, one module can dominantly explain ${\bf x}$ and weaken the training of the other module. To resolve this, we found curriculum training to be effective (described in a following section.)

4 Learning

We train the model by optimizing the following Evidence Lower Bound (ELBO): ${\cal L}_{\text{ELBO}}({\bf x};\theta,\phi)=$

where $D_{\text{KL}}(q\parallel p)$ is Kullback-Leibler Divergence. For the latent structure maps, as an auxiliary term, we also add standard KL terms between the posterior and unconditional prior such as $D_{\text{KL}}\left[q_{\phi}({\bf z}^{\text{pres}}\mid{\bf x})||\text{Ber}(\rho)\right]$ and $D_{\text{KL}}\left[q_{\phi}({\bf z}^{b}\mid{\bf x})\parallel{\cal N}({\bf 0},{\bf 1})\right]$. This allows us to impose prior knowledge to the learned posteriors . See the supplementary material for detailed equations for this auxiliary loss. We apply curriculum training to deal with the racing condition between the background and component modules, both trying to explain the full observation. For this, we suppress the learning of the background network in the early training steps and give a preference to the foreground modules to explain the scene. When we begin to fully train the background, it focuses on the background.

Experiments

Goals and Datasets. The goals of the experiments are (i) to evaluate the quality and properties of the generated images in terms of clarity and scene structure, (ii) to understand the factors of the datasets and hyperparameters that affect the performance, and (iii) to perform ablation studies to understand the key factors in the proposed architecture. We use the following three datasets:

MNIST-4. In this dataset, an image is partitioned into four areas (top-right, top-left, bottom-right, and bottom-left), and one MNIST digit is placed in each quadrant. To make structural dependency among these components, we generated the images as follows. First, a random digit of a class randomly sampled between 0 and 6 is generated in a random position in the top-left quadrant. Then, starting from the top-left, a random digit is placed to each of the other quadrants with the digit class increased by one in the clockwise direction. The positions of these digits are symmetric to each other on the $x$-axis and $y$-axis whose origin is the center of the image. See Fig. 2 for examples.

MNIST-10. To evaluate the effect of the number of components and complexity of the dependency structure, we also created a similar dataset containing ten MNIST-digits and a more complex dependency structure. The images are generated as follows. An image is also split into four quadrants. For each quadrant, four mutually exclusive sets of digit classes are assigned: $Q_{1}=\{0,1\}$, $Q_{2}=\{2,3,4\}$, $Q_{3}=\{8,9\}$, and $Q_{4}=\{5,6,7\}$ in the clock-wise order from top-left quadrant ($Q_{1}$), respectively. Then, the following structural conditions are applied. $Q_{1}$ and $Q_{3}$ are placed randomly and at the same within-quadrant position. Digits in $Q_{2}$ and $Q_{4}$ have no position dependency and are placed randomly within the quadrants. To impose a stochastic dependency, the quadrants are diagonally swapped at random.

Arrow Room. This dataset contains four 3D objects in a 3D space similar to CLEVR . The objects are combinatorially generated from 8 colors, 4 shapes, and 2 material types. Among the four objects, one always has the arrow shape, two other objects always have the same shape, and the last one, which the arrow always points to, has a unique shape. Object colors are randomly sampled, but the same material is applied to all objects within an image. The arrow is the closest to the camera.

Baselines. We compare GNM to the following baselines. (i) GENESIS is the main baseline which, like GNM, is supposed to support both structured representation and density-based generation. (ii) ConvDRAW is one of the most powerful VAE models that focuses on density-based generation without the burden of learning structured representation. Here we want to investigate whether GNM can match or outperform ConvDRAW even while simultaneously learning a structured representation. Finally, (iii) VAE is a model representing the no-structure and no-autoregressive-prior case. We set the default drawing steps of GNM and ConvDRAW to 4 but also tested with 8 steps.

Evaluation Metrics. We use three metrics to evaluate the performance of our model. For the (i) scene structure accuracy (S-Acc), we manually classified the 250 generated images per model into success or failure based on the correctness of the scene structure in the image without considering generation quality. When we cannot recognize the digit class, however, we also labeled those images as failures. For the (ii) discriminability score (D-Steps), we measure how difficult it is for a binary classifier to discriminate the generated images from the real images. This metric considers both the image clarity and dependency structure because a more realistic image, i.e., satisfying both of these criteria, should be more difficult to discriminate, i.e., it takes more time for the binary classifier to converge. For this metric, we measure the number of training steps required for the binary classifier to reach 90% classification accuracy. Finally, we estimated the (iii) log-likelihood (LL) using importance sampling with 100 posterior samples .

Qualitative Analysis of Samples. In Figure 2, we show the samples from the compared models. We first see that the GNM samples are almost impossible to distinguish from the real images. The image is not only clear but also has proper scene structure following the constraints in the dataset generation. GENESIS generates blurry and unrecognizable digits, and the structure is not correct in many scenes. For the ARROW dataset, we see that the generation is oversimplified and does not model the metal texture. The shape is also significantly distorted by lighting. For ConvDRAW, many digits look different from the real and sometimes unrecognizable, and many scenes with incorrect structures are also observed. For the ARROW dataset, object colors are sometimes not consistent, and the arrow directs the wrong object. We can also see a scene where all objects have different shapes not existing in the real dataset. Finally, the VAE samples are significantly worse than the other models. In Figure 3, we also compare the decomposition structure between GNM and GENESIS. It is interesting to see that GENESIS cannot decompose objects with the same color. Not surprisingly, VAE with neither the autoregressive drawing prior nor the structured representation performs the worst. See supplementary for more generation results and different effects on ${\bf z}^{\text{g}}$ and ${\bf z}^{\text{s}}$ sampling.

Scene Structure Accuracy. For quantitative analysis, we first see whether the models can learn to generate according to the required scene structure. As shown in Table 1, GNM provides almost perfect accuracy for ARROW and MNIST-4, while the baselines show significantly low performance. It is interesting to see that for MNIST-10 all baselines completely fail while the accuracy of GNM remains high. This indicates that the learnability of the scene structure is affected by the number of components and the dependency complexity, and GNM is more robust to this factor. ConvDRAW with 8 steps (ConvDRAW-8) performs better than ConvDRAW with 4 steps (ConvDRAW) but still much worse than the default GNM which has 4 drawing steps. This indicates that the hierarchical architecture and structured representation of GNM is a meaningful factor making the model efficient. Also, from Table 2, we can see that GNM with 8 drawing steps brings further improvement. Although GENESIS is designed to learn both structured representation and density-based generation, it performs poorly in all tasks. From this, it seems that GENESIS cannot model such scene dependency structures.

Discriminability. Although the dataset allows us to evaluate the correctness of the scene structure manually, it is difficult to evaluate the clarity of the generated images manually. Thus, we use discriminability as the second metric. Note that to be realistic (i.e., difficult for the discriminator to classify), the generated image should have both correct scenes structure and clarity. From the result in Table 1 and Figure 4 (right), we observe a consistent result as the scene structure accuracy: GNM samples are significantly more difficult for a discriminator to distinguish from the real images than those generated by the baselines. The poor performance of GENESIS for this metric indicates that its generation quality is poor even if it can learn structured representation. Interestingly, GNM is more difficult to discriminate than non-structured generative models (ConvDRAWs) even if it learns the structured representation together. This, in fact, can be considered as evidence showing that the GNM model utilizes the structured representation in such a way to generate more realistic images.

Log-Likelihood. While GNM provides a better log-likelihood for the ARROW dataset than ConvDRAWs, for the MNIST datasets, ConvDRAWs perform slightly better than GNM even if the previous two metrics and the qualitative investigation clearly indicate that the ConvDRAWs provide much less realistic images than GNM. In fact, this result is not surprising but reaffirms a well-known fact that log likelihood is not a good metric for evaluating generation quality; as studied in , for high-dimensional data like our images, a high log-likelihood value does not necessarily mean a better generation quality, and vice versa. However, the log-likelihood of GENESIS is significantly and consistently worse than the other models.

Ablation Study. In Table 2, we compare various architectures to figure out the key factors making GNM outperform others. See the table caption for the description of each model. First, from the comparison between GNM-Struct and GNM-Gaussian, it seems that the StructDRAW global prior is a key factor in GNM. Also, by comparing GNM-Struct and GNM-NoMLP, we can see that the interaction layer inside StructDRAW, implemented by an MLP, is also an important factor. However, from the comparison between GNM-Struct and ConvDRAW-MLP, it seems that the MLP interaction layer is not a sole factor providing the GNM performance because, for MNIST-10, ConvDRAW-MLP still provides poor performance. Also, adding MLP interaction to ConvDRAW tends to degrade its D-Steps and LL, but it helps improve GNM. This indicates that the hierarchical modeling and StructDRAW are the key factors realizing the performance of GNM.

Effects of $\beta$. As the baselines in Table 1 show a very low accuracy with the default value $\beta=1$ for the hyperparameter for KL term , we also tested different values of $\beta$. As shown in Figure 4, the scene structure accuracy of ConvDRAW and VAE improved as the beta value increases. However, even for the largest value $\beta=10$, the structure accuracy of ConvDRAW is still lower than (for ARROW room) or similar to (for MNIST-10) GNM with $\beta=1$ while their log-likelihoods are significantly degraded. For GNM, we only tested $\beta=$ for MNIST-10 as it provides good and robust performance for these low values. GNM also shows an improved structure accuracy, but a more graceful degradation of the log-likelihood is observed.

Novel Image Synthesis. The dual representation of GNM, (${\bf z}^{g}$ for distributed representation and ${\bf z}^{s}$ for symbolic structure), can provide an interesting way to synthesize novel scenes. As shown in Figure 5, we can generate a novel scene by controlling an object’s structured representation, such as the position, independently of other components. On the other hand, we can also traverse the global distributed representation and generate images. In this case, we can see the generation also reflects the correlation between components because the arrow changes not only its position but also its pointing direction so as to keep pointing the gold ball.

Conclusion

In this paper, we proposed the Generative Neurosymbolic Machines (GNM), which combine the benefits of distributed and symbolic representation in generative latent variable models. GNM not only provides structured symbolic representations which are interpretable, modular, and compositional but also can generate images according to the density of the observed data, a crucial ability for world modeling. In experiments, we showed that the proposed model significantly outperforms the baselines in learning to generate images clearly and with complex scene structures following the density of the observed structure. Applying this model for reasoning and causal learning will be interesting future challenges. We hope that our work contributes to encouraging further advances toward combining connectionism and symbolism in deep learning.

Broader Impact

The applicability of the proposed technology is broad and general. As a generative latent variable model that can infer a representation and also generate synthetic images, the proposed model generally shares similar effects of the VAE-based generative models. However, its ability to learn object-centric properties in an unsupervised way can help various applications requiring heavy object-centric human annotations such as various computer vision tasks. The model could also be used to synthesize a scene that can be seen as novel or fake depending on the purpose of the end-user. Although the presented model cannot generate images realistic enough to deceive humans, it may achieve this ability when combined with more powerful recent VAE models such as NVAE .

Acknowledgement

SA thanks Kakao Brain and Center for Super Intelligence (CSI) for their support. The authors also thank Zhixuan Lin and the reviewers for helpful discussion and comments.

References

Appendix A Additional qualitative Results

In Figure 6 - 9, we show additional generation results for GNM and the baseline models. For ConvDRAW and ConvDRAW-8, we show the results for $\beta$ value 1 and 10.

In Figure 10 and 11 we show the generation results with different ${\bf z}^{\text{s}}$ samples while the ${\bf z}^{\text{g}}$ are fixed. In the arrow room dataset, we see the image variation is small for different ${\bf z}^{\text{s}}$ samples, we can occasionally see object’s color changes. And in the two MNIST datasets, we see some variation on the digit styles in different ${\bf z}^{\text{s}}$ samples while the overall scene structure remains the same. Comparing the variation in Figure 10 and 11, we find the same level of certainty on the ${\bf z}^{\text{s}}$ on both the posterior ${\bf z}^{\text{g}}$ samples and the prior ${\bf z}^{\text{g}}$ samples. This implies that the global representation ${\bf z}^{\text{g}}$ is flexible enough to capture most of the information in the scene.

Appendix B Additional experiment

To evaluate GNM’s ability to model more complex data variations, we generate a new task by combining datasets MNIST-4 and MNIST-10. In this setting, the model is required to model the correlation between the number of objects and the corresponding scene structure. As we can see in Figure 12, GNM can generate new scenes that reflect the ground-truth design while all baseline models fail to achieve it. This also reflects on the quantitative result shown in Table 3. We see that the generation from GNM is more difficult to distinguish from the real images and has a higher scene structure accuracy. In this task, the default GNM model with draw steps 4 has relatively lower scene accuracy than those on MNIST-4 and MNIST-10. Increasing the number of draw steps to 8 (GNM-8) significantly improves the scene accuracy. This shows that, in this task, more interaction steps are needed to model the scene structure correctly. We also test GNM and ConvDRAW with different $\beta$. Similar to the result on MNIST-4 and MNIST-10, a larger $\beta$ term brings a higher scene accuracy and a lower likelihood value. Note that GNM and GNM-8 with $\beta$ 2 still outperform ConvDRAW and ConvDRAW-8 with $\beta$ 10 in terms of scene accuracy.

B.2 Representation Learning

The goal of this experiment is to measure the quality of the learned structured representation. Here we use SPACE as the baseline. The results are shown in Table 4. First, we test the model’s ability to infer the object position by measuring the inferred bounding boxes’ average precision with the ground-truth boxes on different IoU thresholds. Second, we measure the quality of the inferred ${\bf z}^{\text{what}}$ representation by an object-wise classification task. More specifically, we train a two-layer MLP to classify the inferred ${\bf z}^{\text{what}}$ representation into 10 digit classes. The digit label of the nearest object in the dataset is used as the ground-truth lable. Both metrics are computed using the test set. As we can see, GNM and SPACE have the similar performance on the two tasks, this showcase GNM’s ability to obtain good structured representations.

Appendix C Auxiliary Losses and Curriculum Learning

GNM is trained by maximizing the Evidence Lower Bound (ELBO) with additional KL terms. The ELBO is shown in the following

Here, the structure representation is split into the latent structure map ${\bf z}^{s}$ and background representation ${\bf z}^{b}$. The coefficient $\beta_{g}$ for the KL of global representation is used in the curriculum training period and will be 1 in the remaining training stage.

Unlike the prior distribution in SPACE that serve as the regularization on the posterior distributions, the structure prior of $p({\bf z}^{b}|{\bf z}^{g})$ and $p({\bf z}^{o}|{\bf z}^{g})$ in GNM are both conditional and learned from the posterior distributions. This causes the problem that by optimizing the ELBO, we cannot provide any prior knowledge to the posterior distribution to guide the inference process. To solve this problem, we introduce the following additional KL terms in the optimization objective.

The $\boldsymbol{\mu}$ and $\boldsymbol{\sigma}$ is further split into $\boldsymbol{\mu}_{\text{what}}$, $\boldsymbol{\sigma}_{\text{what}}$, and $\boldsymbol{\mu}_{\text{where}}$ and $\boldsymbol{\sigma}_{\text{where}}$. Here, the ${\cal N}(\boldsymbol{\mu}_{\text{what}},\boldsymbol{\sigma}_{\text{what}}^{2})$ is chosen to be a standard normal distribution. The ${\cal N}(\boldsymbol{\mu}_{\text{where}},\boldsymbol{\sigma}_{\text{where}}^{2})$ is chosen to encourage the bounding boxes to be tighter and closer to each grid center. The parameter $\rho$ for Bernoulli distribution is set to have a small value to encourage the model to explain the scene with as few objects as possible. With these additional KL terms, the objective function becomes the following

Here $\beta_{b}$ is used for curriculum training which will be described in detail in the following section.

For a neural network module, modeling an individual component is usually a much simpler task than modeling a full multi-object scene. Thus, when provided multiple modules, the model should be encouraged to utilize different modules to model the individual components, e.g., modeling the foreground objects with foreground bounding boxes and the background with the background module.

However, when training GNM, we observed a different learning pattern. The model tends to explain the full scene only using the background module. We found that this is the result of the learning behavior in the early training iterations. At the initial training stage, the background module is provided more signals to optimize because, by design, it is always an activated module (${\bf z}^{\text{pres}}=1$). This allows the background model to learn an accurate full scene reconstruction quickly. On the other hand, the foreground model is usually turned off (${\bf z}^{\text{pres}}=0$) at the early training stage since it is under-optimized and provides rather bad object reconstructions. This again encourages the model to bias more on the background module, and, as a result, the background module dominates.

To solve this problem, we employ a curriculum learning procedure to provide more learning signal for the foreground modules in the early training iteration. First, we set its object mask for each object bounding box to occupy the full box and assign a non-zero value, e.g., 0.9, for each pixel. This forces the foreground module to be responsible for 90% of the pixel value in the boxed area. Second, $\beta_{b}$ is set to be 50 at the beginning and gradually annealed to 1 in 50000 steps. This limits the background capacity and thus encourages the background module to learn a simpler and more static component.

Apart from the curriculum training on the foreground module, we also perform a warm-up on the KL term of global representation. This is done by gradually increasing the value of $\beta_{g}$ from 0 to 1 in the first 100k steps. It allows the model to first learn a meaningful structure representation for ${\bf z}^{s}$ before optimizing the global representation that generates them.

Appendix D Implementation Details

In this section, we describe the details of the model design. The detailed architecture is shown in Table 6 - 11. In these tables, Layer denotes the layer normalization and Subconv denotes the sub-pixel convolutional layers .

The inputs to GNM are images with $128\times 128$ resolutions. It is first provided to a convolutional neural network (CNN) to obtain a $4\times 4$ encoding ${\bf f}^{x}$. The architecture of the image encoder is shown in Table 6. The $4\times 4$ feature map is then used to infer the representations ${\bf z}^{g}$, ${\bf z}^{b}$, and ${\bf z}^{s}$. Here, ${\bf z}^{s}=\{{\bf z}^{s}_{hw}\}$ and $\smash{{\bf z}_{hw}^{s}=[{\bf z}_{hw}^{\text{pres}},{\bf z}_{hw}^{\text{what}},{\bf z}_{hw}^{\text{where}},{\bf z}_{hw}^{0}pt]}$. We first use an MLP layer on top of the image encoding to infer the background representation. For the structure representations $q({\bf z}^{s}\mid{\bf f}^{x})$, we apply additional CNN layers to infer each of the representations $\smash{{\bf z}_{hw}^{\text{pres}}}$, ${\bf z}_{hw}^{\text{what}}$, ${\bf z}_{hw}^{\text{where}}$, and $\smash{{\bf z}_{hw}^{0}pt}$.For the global representation, the encoding ${\bf f}^{x}$ is provided to the StructDRAW module.

The output feature map from StructDRAW is used to generation the symbolic representation map $p({\bf z}^{s}\mid{\bf f})$. For all of our generation samples, we directly take the mode of ${\bf z}^{s}$ instead of sampling from the prior distribution. Note that we share the parameter of the network $q({\bf z}^{s}\mid{\bf f}^{x})$ and $p({\bf z}^{s}\mid{\bf f})$. This encourages the model to generate the structure feature map that is consistent with the input feature during training.

Given the symbolic representation map ${\bf z}^{s}$, the rendering process is similar to that in SPACE and SCALOR . For each object in the foreground, we first obtain its RGB appearance ${\bf o}_{hw}$ and segmentation mask ${\bf m}_{hw}$ by decoding from the ${\bf z}^{\text{what}}$ representation using a CNN decoder, which is shown in Table 7. The full foreground mask ${\bf M}$ is then obtained by summing all object masks into a full image using the spatial transformer network (STN) using ${\bf z}^{\text{where}}$. Similarly, each object image ${\bf o}_{i}$ is mapped into the full-image size and gives ${\bf x}_{i}$. To determine which object should be drawn in a foreground pixel position (when multiple objects occupy the pixel), we first compute the responsibility $\boldsymbol{\gamma}_{i}$ using ${\bf z}^{\text{pres}}$, ${\bf z}^{0}pt$, and ${\bf z}^{\text{where}}$ and then the full foreground image ${\bf x}^{\text{fg}}$ is obtained by multiplying the object images with the normalized responsibilities. The background image ${\bf x}^{b}$ is generated by a background decoder shown in Table 8. The final image is then computed by ${\bf x}={\bf x}^{\text{fg}}+(1-{\bf M})\odot{\bf x}^{b}$. The full rendering process is illustrated in Algorithm 2.

Table 9 describes the rest of the network structures that are not specified by Table 6 - 8. Note that in Table 9, all convolutional and MLP layers except the output layers are followed by a CELU activation function and a layer normalization .

D.2 Baseline models

Our implementation of GENESIS is based on the official PyTorch implementation. We found that with the default setting in official code, we are unable to make GENESIS decompose the scene into components. Instead, the model tends to cluster the objects into components base on their colors or locations. Thus, to encourage a correct decomposition and generation, we make the following modifications on the official code: (a) we design a learning rate schedule where it starts with a higher value and reset to a lower one in a few thousand steps. (b) instead of optimizing the Constrained Optimisation objective (GECO) , we optimize the evidence lower bound with a $\beta$ value of 15 for arrow room dataset and 10 for MNIST dataset. (3) we reduce the number of layers for the spatial broadcast decoder from 4 to 3. We found that the modifications allow GENESIS to decompose the scene correctly while also improve its generation quality on three datasets.

For VAE and ConvDRAW, we use our own implementation. We first use an image encoder to obtain an image encoding for both models. Its architecture is designed to have the same structure as the image encoder in GNM, shown in Table 6. Then for VAE, we use a 2-layer CNN with filter sizes of , and kernel sizes of on top of the image encoding to compute the parameter for the latent representation. For ConvDRAW, we apply a similar architecture of StructDRAW on top of the image encoding shown in Table 9, while the interaction MLP is replaced with a 2-layer CNN with filter sizes of , and kernel sizes of 3. The architecture of the image decoders for the two models are shown in Table 10 and 11.

Table 5 shows the model size each model used for the three datasets