LEGO-Net: Learning Regular Rearrangements of Objects in Rooms

Introduction

What makes the arrangement of furniture and objects in a room appear regular? While exact preferences may vary, humans have by-and-large universally shared criteria of regular room arrangements: for instance, heavy cabinets are arranged to align with walls, chairs are positioned evenly around a table in linear or circular configurations, or night stands are placed symmetrically on the two sides of a bed. Humans also share a common dislike of physically performing the task of rearranging a messy room. To build automated robotic systems that can guide or actually rearrange objects in a room, we first need methods that understand the shared human criteria for regular room rearrangements and respect the physical constraints of rearrangements.

Human criteria for regular rearrangements can be subtle and complex, including geometric rules of reflexional, translational, or rotational symmetry, linear or circular alignments, and spacing uniformity. Functional and stylistic inter-object relationships are also important: for example, a TV tends to be in front of and facing a sofa, chairs are next to a table, etc. Many of these criteria interact and, at times, conflict with one another. As a result, in general, there is more than one desirable clean arrangement for any given messy arrangement. In our setting, we further desire that the clean rearrangement we create to be informed by the initial messy arrangement – and not be entirely different – for multiple reasons. First, there may have been a particular clean arrangement that gave rise to the messy one – and it may be desirable to recover a similar arrangement. Second, we want to minimize the motion of objects as much as possible to respect the physical constraints and effort involved – especially the motion of big and heavy furniture. Unfortunately, extant methods fail to capture these criteria: methods for scene synthesis from scratch ignore the initial state of objects in a room, and rearrangement methods often require scene-specific human input in the form of a goal state or language description .

In this paper, we present LEGO-Net, a method for LEarning reGular rearrangement of Objects in rooms directly from data. Different from work that focuses on arranging new objects from scratch or requires goal state specification, we focus on rearranging existing objects without any additional input at inference time. We take as input the position, orientation, class label, and extents of room objects in a specific arrangement, and output a room with the same objects but regularly re-arranged. LEGO-Net uses a transformer-based architecture that is, in part, motivated by recent denoising diffusion probabilistic models that learn a reverse diffusion process for generative modeling . We learn human criteria for regular rearrangements from a dataset of professionally designed clean (regular) scenes , and represent each scene as a collection of objects and a floor plan. Prior to training, we perturb the regular scenes to generate noisy configurations. During training, our transformer learns to predict the original, de-noised arrangement from the perturbed scene and its floor plan. During inference, instead of directly re-arranging scenes with our model, which would amount to naïve regression, we run a Langevin dynamics-like reverse process to iteratively denoise object positions and orientations. This iterative process retains the flavor of original room state, while limiting object movement during re-arrangement.

We conduct extensive experiments on public datasets to show that our approach realistically rearranges noisy scene arrangements, while respecting initial object positions. We also demonstrate that our method is able to generalize to previously unseen collection of objects in a wide variety of floor plans. Furthermore, we include extensive experimental results (e.g., Fig. 1 and Fig. 4), including a new metric to evaluate regularity of re-arrangements, aimed at measuring the presence of sparse linear integer relationships among object positions in the final state (using the PSLQ algorithm ). To sum up, we contribute:

A generalizable, data-driven method that learns to regularly re-arrange the position and orientation of objects in various kinds of messy rooms.

An iterative approach to re-arrangement at inference time that retains flavor of the original arrangement and minimizes object travel distance.

An in-depth analysis of the performance and characteristics of the denoising-based scene rearrangement.

A new metric to measure the regularity of object arrangements based on integer relation algorithms.

Related Work

In this section, we discuss literature in two related areas: (1) scene synthesis from scratch, (2) scene rearrangement where an end goal is specified, and (3) diffusion models.

Indoor 3D Scene Synthesis: Indoor room synthesis is the problem of synthesizing the layout of objects in a scene from scratch. Many classical methods in the computer graphics literature use heuristics and guidelines to constrain the location of pre-specified objects . identified a collection of functional, visual, and design constraints and formulated an optimization problem. Work has also focused exclusively on inter-object relationships . Other methods address the open world layout problem when objects are not pre-specified.

An alternative approach is to adopt procedural modeling using generative grammars . Some methods adopt the scenegraph representation and formulate it as a graph problem . Both procedural and graph-based methods often rely on curated data . Some methods learn directly from data using neural networks, for instance from images . Both SceneFormer and ATISS introduce autoregressive methods for scene generation. Different from all these methods, we focus on rearranging rooms given an initial messy state.

Scene Rearrangement: Scene rearrangement takes an initial state of the scene and aims to bring it to a goal state specified by the user. This task is deeply connected to planning in robotics . Some works consider robot pushing and manipulation for rearrangement . Many of these methods require datasets for training and often use datasets like AI2-THOR , Habitat or Gibson . Some of these methods operate on visual observations , while others assume fully-observed synthetic environments . To specify the goal state, some recent methods use language input driven by large language models . Related to these advances in robotics, there have also been attempts to apply these specifically for room rearrangements .

In this paper, we focus on the task of room rearrangements without the need to specify the goal state. We directly learn arrangements that satisfy human criteria from professionally arranged dataset provided by 3D-FRONT . Note that a concurrent work addresses the same problem but with a focus on physical simulation, incorporating reinforcement learning and path planning.

Denoising Diffusion Models: 2D Diffusion models have emerged as a powerful technique for unconditional image synthesis, outperforming existing 2D generative models . Diffusion models have also seen great success in conditional image generations, receiving conditions in the form of class labels , text , or input images . Various methods apply diffusion models for restoring corrupted or user-provided images to realistic images. Our method shares the same philosophy and adopts related techniques from the diffusion models, e.g., Langevin Dynamics, to project messy object configurations onto the manifold of “clean” scenes.

Method

Our method takes the position, orientation, class label, and extents of objects in a ‘messy’ room as input and outputs a rearranged version in a ‘regular’ state. Since objects in rooms primarily move on the floor, we only consider 2D object pose, but our method can be combined with existing instance segmentation and canonicalization methods to directly operate from a 3D mesh or point cloud. We represent each scene $X$ as an unordered set of $n$ objects and their attributes:

2 LEGO-Net: Learning Regular Room Rearrangements

3 Architecture

Transformer Architecture: We use our custom positional encodings and procedures to prepare the tokens but use the original Transformer encoder architecture without notable modifications. We use $8$ multi-headed attentions with $512$-dimensional hidden layers and $512$-dimensional key, query, and value vectors. The output of the transformer network is the estimated object transformations, a ${|X|\times 4}$ matrix.

4 Training and Inference

Data: We employ the 3D-FRONT dataset for the task of indoor scene rearrangement. For each valid clean scene in the dataset, we preprocess it into $X=\{o_{1},...,o_{n}\}$ and extract the contour of its floor plan.

Training: We use the denoising auto-encoder formulation of Eq. 2 to train our denoiser function $f_{\theta}.$ We uniformly randomly sample training examples and sample a noise level $\sigma$ from a normal distribution. The sampled examples are perturbed using an independent Gaussian kernel with standard deviation $\sigma$. For the perturbation, we do not consider objects going outside of the floor plans or colliding with one another. Each perturbed scene uses its original clean scene as the source of ground truth but re-establishes object correspondence through Earth Mover’s Distance assignment to enable invariance among identical objects and further promote distance minimization in movement prediction. We use Adam optimizer with a learning rate of $10^{-4}$ to train.

Experiments

We conduct a number of experiments to test LEGO-Net’s ability to automatically capture scene regularities from data. To this end, we prepare two testbeds for experiments: our custom-designed Table-Chair environment where we mathematically constructed the regularities among objects and 3D-Front , which contains tens of thousands of synthetic rooms designed by professionals. The Table-Chair dataset is useful because we can model one regularity at a time and quantify network performance. 3D-Front dataset exhibits complex and subtle rules that designers commonly perceive as ideal configurations, e.g., geometry, semantic relations, styles, and functionalities.

In the Table-Chair environment, we study four main regularities: symmetry, parallelism, uniform spacing, and grouping by shapes. For each of the proposed experiments, we generate clean scenes based on the designed rules. Then, for training, we perturb the scenes on the fly to generate clean-messy pairs and re-associate objects within each class through Earth Mover’s Distance assignment to train a network with the loss of Eq. 2. We measure each task with the success rate of the rearrangement, whose specific criteria we discuss in the supplementary.

Symmetry and Parallelism: One of the most important notions of regular arrangement is symmetry, which involves both object-object and room-level symmetries. We use a setup of 2 groups of rectangular tables and chairs. We vertically align the 2 tables and horizontally distribute them at a distance uniformly drawn from a fixed range. We arrange 3 chairs in a linear row on one side of the table and 3 chairs in another linear row on the opposite side.

Uniform Spacing: We prepare a highly-challenging setup to stress test LEGO-Net’s ability to capture the concept of uniform spacing. In this setup, we have 2 circular tables, each with 2-6 chairs randomly and uniformly rotated around them. The network has to deal with the unknown number of chairs and the pair-wise spacing.

Grouping by Shape: We test LEGO-Net’s ability to group objects based on their pose-invariant shapes. We augment our setup in ‘Symmetry and Parallelism’ to include 2 types of chairs with different shapes. We arrange the scenes such that chairs with the same shapes are on the same side of the table. The pose-invariant shape features $h_{i}\in o_{i}$ from Eq. 1 provide the necessary shape information.

Results: We visualize the rearrangement results of LEGO-Net for the above three cases in Fig. 4. Across the board, the denoising network successfully learns to capture these important regularities from data, without explicit supervision about the underlying rules. The success rate of each task is shown in Tab. 1. As expected, directly applying the regression-trained network $f_{\theta}$ results in the worst results.

2 3D-Front Experiments

We benchmark LEGO-Net’s ability to conduct regular scene rearrangements on the bedrooms and livingrooms of the 3D-FRONT dataset, which respectively contains 2338/587 and 5668/224 scenes for train/test splits.

We train our LEGO-Net as described in Sec. 3 with $\sigma\sim\mathcal{N}(0,0.1^{2})$. While we maintain a single denoising network $f_{\theta}$, we explore three variants of inference algorithms to provide greater insight of our approach: (1) LEGO-Net direct, (2) grad. with noise, and (3) grad. w/o noise respectively denote the inference strategy of predicting the clean outcome with one network pass, running Langevin Dynamics of Eq. 5 with noise term $\beta\neq 0$, and $\beta=0$.

We compare our rearrangement results against the current SOTA scene synthesis method, ATISS . While ATISS is designed to synthesize a scene from scratch rather than to rearrange one, it provides an auto-regressive generative model that can be flexibly applied to our task. Specifically, we use three variants of ATISS that share the same network weights. First, ATISS vanilla performs its original scene synthesis task given a floor plan. Second, ATISS with labels performs object placement using a predefined set of objects per scene. Third, ATISS failure-correction takes a noisy scene and cleans it up by iteratively finding an object with low probability and re-placing it within the current scene. This variant of ATISS is given the same perturbed scene as LEGO-Net and aims to clean the scene. Note that we omit to compare against prior works that have already been compared against ATISS, e.g., . We could not find a prior data-driven method that is designed to solve the same rearrangement problem as ours.

Metrics

To gauge how well LEGO-Net captures datasets’ regularities, we adopt the popular FID and KID scores. These metrics compare the closeness of statistics of two data distributions. We follow prior works to render ground truth and generated scene arrangements from top-down views and compute the metrics in the image space. Note that KID is more applicable to our setting because FID is known to present huge bias when the number of data is low. Another important criterion for our rearrangement task is how much distance the objects travel between the initial and final scene states. Similarly, when applicable, we measure the Earth Mover’s Distance (EMD) between the ground truth scene and our cleaned-up scene.

Finally, we introduce a new metric that measures scene regularities by finding integer relations among object positional coordinates $t_{i}$’s. To do this, we select two or three random objects within a scene and check if we can find integral $a_{i}$’s that satisfy:

where $\eta$ sets the maximum magnitude of the coefficients. Intuitively, these integer relations can capture regularities such as colinearities ($-t_{1}+t_{2}=0$) and symmetries ($t_{1}-2t_{2}+t_{3}=0$). See supplementary for detailed descriptions.

Results

We conduct the 3D-FRONT arrangement experiments with five algorithms (three ATISS variants and two of ours) and compute their metrics. The main numerical results, which can be found in Tab. 2, show that LEGO-Net outperforms all variants of ATISS, including the failure-correction variant that tackles the same object cleaning problem as demonstrated in the original paper.

In Fig. 6, we plot the chance of finding integer relations in scenes perturbed with different noise levels, which peaks for the original clean 3D-FRONT scenes and sharply decreases as noise is added. Also, note that the rearranged scenes of LEGO-Net demonstrate high regularities according to this measure, outperforming the results of ATISS variants. See supplementary for more experiment details.

Qualitatively, as shown in Fig. 5, LEGO-Net is able to robustly project messy scenes onto clean manifolds. While ATISS and ATISS with labels were able to synthesize realistic rooms, their object arrangements are entirely different from the original input scene. Importantly, we notice that ATISS failure correction leads to unexpectedly low-quality results. We hypothesize that this is due to their discrete, one-object-at-a-time strategy, which can easily fall into the local minimum of the likelihood space. In contrast, our score-based iterative denoising leads to robust success rates.

3 Analysis

To more deeply understand the behavior of our system, we analyze and discuss important aspects of LEGO-Net. We refer to supplementary for more analysis of our method.

As we discuss throughout Sec. 4, we explore three inference strategies, namely direct, gradient with noise, and gradient without noise. For the 3D-FRONT experiment, we report that the grad. without noise variant consistently outperforms the other variants. However, we believe that this is likely because we used relatively low noise to the scenes (std $0.1$) to more naturally simulate messy indoor rooms. Indeed, our experiment on the synthetic environments (Tab. 1) with larger noise (std $0.25$) shows that the grad. with noise variant outperforms. The results suggest that adding noise during Langevin dynamics allows a better success rate for highly noisy data, but at the cost of losing accuracy in recovering the originals (as shown in Tab. 2).

Cleaning Uncertainty

LEGO-Net is trained to handle input perturbations at various noise levels. In the high-noise regime, there is high uncertainty on the structure of the original information. As input noise increases, our denoising process converges into an unconditional generative model. On the other hand, LEGO-Net has the capacity to capture original regularities when the noise is low, leading to almost precise reconstruction of the original scenes. We visually show these insights in Fig. 7.

Out-of-Distribution Inputs

We showcase LEGO-Net’s ability to handle scenes perturbed with noise patterns significantly different from the one used in training, i.e., zero-mean Gaussian. In the first example, we only perturb chairs. Secondly, we perturb the scene only in the translation dimensions without rotations. Shown in Fig. 8, LEGO-Net can successfully handle out-of-distribution inputs, demonstrating the robustness and versatility of our algorithm.

Conclusion

In this paper, we presented LEGO-Net, a method for regular rearrangement of objects in a room. Different from previous methods, LEGO-Net learns human notions of regularity (including symmetry, alignments, uniform spacing, and stylistic and functional factors) directly from data without the need to explicitly specify a goal state. During training, we learn from a large dataset of professionally-designed room layouts that are randomly perturbed. During inference, we follow a Langevin Dynamics-like strategy to iteratively “denoise” the scene. Quantitative results including comparisons and ablations show that our method performs well, which qualitative results confirm.

Limitations & Future Work: Our method has important limitations that provide extensive opportunities for future work. First, our method is currently limited to 2D room rearrangement and cannot perform 3D rearrangement, for instance in kitchen shelves. However, we do incorporate 3D shape features which can be used to extend our method to 3D. We also currently do not handle interpenetration of objects during denoising, which future work should explore.

Acknowledgements

This work was supported by AFOSR grant FA9550-21-1-0214, NSF CloudBank, an AWS Cloud Credits award, ARL grant W911NF-21-2-0104, a Vannevar Bush Faculty Fellowship, and a gift from the Adobe Corporation. We thank Kai Wang, Daniel Ritchie, Rao Fu, and Selene Lee.

References

Supplementary Document

Appendix A Overview

This supplementary document contains extended technical details, along with qualitative and quantitative results that supplement the main document. After introducing our video results, we cover details of our network architectures and their applications (Sec C). We then provide detailed explanations of our two main experiment setups: Table-Chair (Sec. D) and 3D-FRONT (Sec. E). Next, we conduct additional analysis experiments in Sec. F. Finally, we discuss failure modes (Sec. G) and future directions (Sec. H).

Appendix B Video Results

In order to better visualize the 3D structures of the outputs and the denoising process, we provide videos of these processes in the format of an HTML website. We highly encourage the viewers to open the file “LEGO.html” and watch the videos for a direct view of the denoising process.

Appendix C Architecture Details

We embed the translation $t_{i}$, rotation $r_{i}=[\cos(\theta_{i}),\sin(\theta_{i})]^{T}$, and bounding box dimension $b_{i}$ with a sinusoidal positional encoding of 32 frequencies. The frequencies are a geometric sequence with initial term $1$ and common ratio $128^{\frac{1}{31}}$, which gives an ending term of $128$. The positional encoding is therefore

As mentioned, for object class $c_{i}$, we utilize a 2-layer MLP with leaky ReLU activation to process the one-hot encoding into a $128$-dimensional attribute. The above four features are concatenated to form a $512$-dimensional vector.

C.2 Floor Plan Encoder Architecture

C.3 Output Layers

C.4 Inference Langevin Dynamics Parameters

During inference, we use the Langevin Dynamics scheme to iteratively denoise a messy scene input. As mentioned, for time step $\tau$, we select $\alpha(\tau)=\alpha_{0}/(1+a_{1}*\tau)$ to regulate the step size and $\beta(\tau)=\beta_{0}*b_{1}^{\lfloor\tau/b_{2}\rfloor}$ to regulate the noise added at each iteration. We empirically select $a_{1}=0.005$ and $b_{1}=0.9$. For the living room, we adopt $\alpha_{0}=0.1$, $\beta_{0}=0.01$, and $b_{2}=10$. For bedroom, we adopt $\alpha_{0}=0.08$, $\beta_{0}=0.008$, and $b_{2}=8$.

We break from the iterative denoising process upon any one of two conditions: (1) if for $3$ consecutive iterations, both the predicted translation displacement vector has Frobenius norm less than $0.01$ and the predicted rotation angle displacement is less than $0.005$ radians, or (2) we have reached $1500$ iterations.

Appendix D Table-Chair Experiment

To analyze the regularities our model can capture, we propose three synthetic Table-Chair experiment settings, with a focus on Symmetry and Parallelism, Uniform Spacing, and Grouping by Shapes respectively.

For each of the proposed experiments, we generate clean scenes based on designed rules and take a bimodal approach at perturbations when generating clean-messy training data pair. More specifically, for half of the synthesized clean scenes, we employ a Gaussian noise kernel whose standard deviation is drawn from a zero-mean Gaussian distribution with a small standard deviation ($0.01$ for translation and $\pi/90$ for rotation angle). For the other half of the clean scenes, we employ a Gaussian noise kernel whose standard deviation is drawn from a zero-mean Gaussian distribution with a relatively larger standard deviation ($0.25$ for translation and $\pi/4$ for rotation angle). For the other training details, we follow the same paradigm as in the 3D-FRONT experiments.

D.2 Inference Parameters

As for the 3D-FRONT experiments, we employ the Langevin Dynamics scheme to rearrange a given perturbed Table-Chair arrangement. We empirically adjust the parameters to slightly increase the step size, accelerate the noise decay schedule, and loosen the termination condition. In particular, we select $\alpha_{0}=0.12$, $a_{1}=0.005$, $\beta_{0}=0.01$, $b_{1}=0.9$, $b_{2}=2$, and terminate once the predicted displacements are small enough in magnitude for $1$ iteration.

D.3 Success Rate

As mentioned, we measure LEGO-Net’s performance in each Table-Chair environment through the success rate of its rearrangement. We will now elaborate on its criteria.

Symmetry and Parallelism: For a rearrangement to be classified as a success, it must satisfy the following:

The mean euclidean distance of per-object movement averaged across scenes is less than $0.5$.

For each chair, the angular offset between its orientation and the table-facing orientation is less than $\pi/60$ radians.

Given the two rearranged table positions, we compute their respective chair positions and perform Earth Mover’s Distance assignment using these as target and the final predicted chair positions as source. The total distances summed across all $12$ chairs for the $2$ tables need to be less than $0.08$. Note that this metric integrates colinearity, parallelism, and symmetry, and penalizes collision.

Uniform Spacing: For a rearrangement in the Uniform Spacing experiment to be classified as a success, it must satisfy the first two criteria for the Symmetry and Parallelism experiment. For the third criteria, because the number of chairs arranged around each of the $2$ circular tables is variable, we cannot formulate the Earth Mover’s Distance assignment as in the Symmetry and Parallelism experiment. Instead, to measure how well an arrangement captures the object-object relationships and regular relative positioning, we compute two other metrics.

For each table, we compute the angular distances between each adjacent pair of its chairs and measure the variance of these distances. We designate that a successful arrangement must have an angular distance variance of less than $0.009$ radians. Additionally, given we utilize a fixed radius to generate clean chair arrangements around tables, we can measure the mean difference between the chair-to-closest-table distance and this radius. We designate that the magnitude of this difference needs to be less than $0.01$ for an arrangement to be considered successful.

Grouping by Shapes: Similarly, for a rearrangement in the Grouping by Shapes experiment to be classified as a success, it must satisfy the first two criteria for the Symmetry and Parallelism experiment. Additionally, we once again can compute the exact regular arrangement of chairs with respect to the table, enabling us to calculate the Earth Mover’s Distance from the final predicted chair positions to the clean configuration with respect to the predicted table position. We designate that the distance summed across the $6$ chairs needs to be less than $0.05$.

Furthermore, to measure success at grouping, we require that each row must be assigned exactly $3$ chairs and that all chairs assigned to the same row must have the same shape feature.

D.4 Additional Qualitative Results

In Fig. 9, we provide additional qualitative renderings for the three Table-Chair experiments.

Appendix E 3D-FRONT Experiment

To process the 3D-FRONT dataset , we closely follow the preprocessing protocol of ATISS . For each scene, we extract from the given meshes and parameters the translation, rotation, class, and bounding box size for every object, and we normalize all lengths to be in $$. We additionally extract accurate contours of the floor plans by running an iterative closest point algorithm , using the contour corner points of ATISS’s binary floor plan masks as source and the relevant vertices from 3D-FRONT floor meshes as target.

We compare against three variants of ATISS: vanilla, labels, and failure-correction. As described in the main text, vanilla is the original ATISS approach that generates a scene from scratch given the floor plan. ATISS labels is given the floor plan, as well as a set of furniture labels and the transformations and sizes of the labeled objects. ATISS failure-correction is proposed as an application to the probabilistic generative modeling of ATISS. It identifies which object is likely to be a failure and resamples that object given all the other objects. While the original paper only showed the technique to work when a single object is perturbed, we find it reasonable to extend the algorithm to multi-object perturbation cases. Specifically, we provide a scene with all objects perturbed (same input as LEGO-Net) and iteratively resample the lowest-probable object. We stop the iteration when it reaches 1,000 times or when the minimum probability is higher than a manually set threshold. We note that while failure-correction did not perform as expected when all of the objects are perturbed, as shown in Fig. 10, it is the closest baseline we could find in the literature that performs data-driven denoising of a scene.

For the comparisons, we use the official training and testing code provided by the authors of ATISS without modifications. For ATISS failure-correction, we add a for-loop and stopping criteria on top of their implementation, and maintain the scales of objects as fixed to be coherent with the rearrangement task.

E.2 Metric Description

For FID and KID computation, we first generate the same number of scenes as in the test dataset and randomly select 500 from the generated scenes. We then randomly select 500 real scenes from the 3D-FRONT dataset to compare against. For both FID and KID, we repeat the metric computation 5 times and report the average. Note that for computing the FID scores for ATISS, we used the officially provided code and followed their exact evaluation procedure, but we failed to reproduce their numbers. Hence, we use our own way of computing the metrics and report ours.

We note that the Frechet Inception Distance (FID) is known to present significant positive bias when the number of images is small (e.g., $\leq$ 2000). In our case, we’re dealing with an even smaller number of test images. Therefore, to compute a metric that is less biased in the small-data regime, we adopt the Kernel Inception Distance , which is known to address the bias problem and present small variance even at a few hundred samples.

As discussed in the main text, we aim at rearranging the messy scenes while retaining the flavor of the original scenes. Practically, cleaning a room should move objects as minimally as possible while realizing regularities. Therefore, we measure and report the mean of the average distance traveled for scenes. More specifically, for each scene, we compute the average Euclidean distance between the corresponding objects in the initial and final states, and take the mean across scenes.

Note that for ATISS vanilla, this metric is not applicable as the method randomly places objects into the scene. For ATISS failure-correction, we calculate this metric by computing the distance between the initial position of each object and its final position after applying the algorithm.

EMD to GT

In order to measure how accurately our method recovers the original scene configuration, we measure the Earth Mover’s Distance between the final and the ground truth scene states. Note that computing the difference between the denoising prediction and the ground truth is widely used in the image-denoising literature, using such metrics as PSNR or SSIM . ATISS vanilla and labels do not receive the messy scene as input, so this metric is not applicable to them. On the other hand, ATISS failure-correction directly fixes the input scene, thus we may measure how accurately it recovers the original clean scene. Finally, we note that the EMD to GT metric becomes highly noisy and irrelevant when the noise added to perturb the clean scenes becomes too high, as then, there is scarsely any locational information left in the messy inputs.

E.3 Additional Qualitative Results

We provide additional qualitative results on the 3D-FRONT dataset. We show more comparisons against the closest method on the scene rearrangement task, ATISS failure-correction, in Fig. 10, and more results of our method in Fig. 17.

Appendix F Analysis (Continued)

In this section, we analyze our choice of floor plan encoder. As described in the main text, we extract points on the boundary of the binary floor plan mask, and process them with a PointNet to obtain a unified feature vector describing the floor plan. We note that, in ATISS , a 2D convolutional network with residual connections (ResNet) was used to process the floor plans. Here, we conduct an experiment to justify our use of PointNet architecture. As a baseline, we use the ResNet architecture from ATISS, but augment the input floor plan with two additional channels corresponding to the xy coordinate for each pixel center, which is known to provide “spatial awareness” to the 2D CNN (in CoordConv ). We expect this variant of ResNet to work at least as well as the vanilla ResNet with binary mask input used in ATISS.

To compare the two methods of floor plan encoding, we train two variants of the model, using ResNet or PointNet floor plan encoding architecture. To test the effectiveness of the two methods, we measure how often furniture is moved outside the floor boundaries. Specifically, a scene rearrangement is considered successful when 90% of objects are placed inside the floor boundary within 4% of its length margin. While respecting the floor boundaries does not necessarily lead to high-quality, regular scenes, we empirically find this metric as a reasonable proxy. As can be seen from the numerical results of Tab. 3, the use of PointNet outperforms that of ResNet by a slight margin. However, we note that using PointNet is significantly faster, having almost no computational overhead for operating on the sampled 250 boundary points. We, therefore, choose to use the simpler but similar-performing PointNet to encode the scene floor plans.

F.2 Relative vs Absolute

2D diffusion models perform better at predicting the noise than the un-noised images . However, for our setting, we observed that the absolute prediction models generally outperform their relative counterparts in the Table-Chair environment. We trained $3$ variants of the same architecture network for the Uniform & Parallelism Table-Chair setting: (i) absolute translation and rotation prediction, (ii) relative translation and rotation prediction, and (iii) relative translation and absolute rotation prediction.

Following the criteria in D.3, both variants (ii) and (iii) surprisingly report success rates of $0\%$. Upon further investigation (shown in Tab. 4), variants (ii) and (iii) perform comparably, if not better, at limiting the distance of movement and orienting chairs to face the tables, but they significantly underperform (i) in terms of Earth Mover’s Distance to Ground Truth chair positions with respect to the predicted table position. With the same denoising parameters, the relative variants consistently fail to place the chairs as precisely as the absolute variant. The superior performance of the absolute variant may partly be explained by the fact that this setting has a relatively limited space of possible regular arrangements. The observed deficiency of relative predictions may diminish as the complexity of the scene increases.

We believe that relative transformation prediction is an important future direction to explore, as they offer translation invariance, which is particularly valuable for large-scale scenes. Currently, the position and orientation information in our input object attributes are global, and thus our system is not translationally invariant. An interesting direction for future investigation is to explore a sliding-window-style input processing to ensure translation invariance and to apply positional encoding to the output transformations.

F.3 Distance to Ground Truth and Distance Moved vs. Noise

Since the task of rearrangement values affinity to the starting configuration of objects, one question of interest is how closely we recover the original clean arrangement when given a perturbed version of it, versus another possibly equally valid, clean arrangement. Its correlation with the level of noise added is intuitive–we expect that when the perturbation is low, we more closely reconstruct the original scen with a low distance moved whereas when the perturbation is high, our model may choose a regular arrangement different from that of the original scene in an effort to minimize the distance moved (see Fig. 11). This is indeed what we have observed numerically, as shown in Fig 12.

Note that with a low degree of noise, our model performs the task of rearrangement, but as the degree of noise increases, our model gradually transitions to the task of arrangement. In the extreme case where we give LEGO-Net a scene with objects outside of the floor plan, LEGO-Net is able to perform scene arrangement from scratch (see Fig. 13).

The open-endedness in the definition of regularity is one of the reasons why this task is both challenging and interesting. The interpolation-like behavior of LEGO-Net conditioned on the degree of noise signifies the learnable relationship between rearrangement and synthesis, and it demonstrates that a diffusion-like approach holds promising potential at such open tasks.

F.4 Integer Relations

The task of evaluating regularity in an object arrangement is itself an interesting research problem because, in general, multiple regular solutions are possible. To evaluate and quantify the notion of “regularity” in object arrangements, we propose using number-theoretic machinery for detecting and evaluating sparse linear integer relations among object coordinates. That is, given coordinates $t_{i}$’s for $n$ objects, we seek to find integral coefficients $a_{i}$’s such that:

When the subset size $n$ and maximum coefficient magnitude constraint $\eta$ are small, the integer relations can be intuitively understood (e.g. representing co-linearity, symmetry, uniform spacing among few objects). To capture more complex and more general notions of regularity, we increase $n$ and $\eta$. Doing so preicipitates two challenges. First, the number of possible subsets of size $n$ for each scene increases rapidly as $n$ increases, compromising efficiency. Secondly, experimentally running the algorithm on pure noise shows that with looser constraints, we may find many trivial relations that do not appear to correspond to high-level ‘cleanness’. To counter these, we introduce two filtering mechanisms to increase the subset sampling efficiency and to filter out the insignificant relations.

We observe that regualrities of interest to us mostly occur among objects in close proximity to one another, such as tables and chairs. Therefore, for $n>2$, instead of sampling from all possible subsets of the objects in the scene, we iterate through each object and sample from the object’s positional neighborhood. For $n=3$, for each object, we sample $2$ from the closest $4$ neighboring objects to form $\{t_{1},t_{2},t_{3}\}$. This greatly improves sampling efficiency, and it also helps eliminate irrelevant candidates as integer relations satisfied by objects in vicinity to one another are more likely to be meaningful for the purpose of our evalutions.

Additionally, to filter out relations that may have been satisfied by numerical coincidence, we require all relations to be translation-invariant. Specifically, for each subset, we sample a noise $\mu$ from uniform distribution $U(-1,1)$ and apply the PSLQ algorithm to $\{t_{1}+\mu,...,t_{n}+\mu\}$. We repeat the process $10$ times and only deem a subset to have a valid relation if the algorithm succeeds for all $10$ times. This helps the metric to focus on regularities with respect to the relative positions instead of the absolute positions of objects.

In Fig 14, we demonstrate that with these two filtering mechanisms, our metric is still meaningful for $\eta=5$ and potentially larger parameters, which would be useful for measuring wider ranges of regularities. We also show that averaging all the integer relation metrics across the various settings suggest more general notions of regularity, for which LEGO-Net outperforms the ATISS variants.

Appendix G Failure Modes

While LEGO-Net generates unprecedented-quality indoor scenes through the iterative denoising process, we notice that it often suffers from objects going out of the floor boundaries and objects penetrating each other. These failure modes are illustrated in Fig. 15 and 16. In fact, in Tab 3, we measure that around half of living room realizations have at least one object outside of the boundaries.

We propose two possible remedies for these related issues. First, we could apply post-processing steps to physically resolve the two problems. That is, we optimize the locations of the objects within the scene such that penetrations and out-of-boundary issues are resolved with minimal required movement. We believe a possible formulation would involve a signed distance function (SDF), with which it is easy to compute the gradient to minimize the penetrations. When a point lies on the negative territory of another shape’s SDF, we can optimize the location of that point out towards the SDF’s gradient directions.

Secondly, one can consider richer encoding of the floor plan. One possibility is to encode each line segment of the floor plan separately as a token. This will essentially treat each line segment as an object in the scene and could enforce stricter constraints on the boundaries. At a glance, this strategy might increase the computational cost significantly, due to the quadratic nature of Transformer time complexity. However, one could consider limiting the communication between the line segment tokens to prevent quadratic scaling of the complexity.

We leave these two potential remedies for our failure modes as future work.

Appendix H Future Work

In this work, we introduced LEGO-Net, an iterative-denoising-based method for tackling scene rearrangement task, which is relatively understudied compared to the scene synthesis task. We show through extensive experiments that our method is able to capture regularities of complex scenes, generating high-quality object rearrangements that could not be achieved by existing approaches to date.

However, LEGO-Net in its current form only operates on a 2D plane for the rearrangement. Extending our work to operate on the $SE(3)$ transformation space would make it more applicable to real-world scenes. A significant barrier to achieving 3D rearrangement is the lack of data. We notice that most of the indoor scene arrangement datasets deal with objects laid on the floor plan, which could limit the progress of studying $SE(3)$ scene arrangements. Designing and collecting such a dataset, e.g., small objects on top of one another is an interesting future direction.

Moreover, the trajectories generated during the denoising process of LEGO-Net are not meant to respect physical constraints, e.g., penetrations and collisions. We find that enforcing the physical constraints during the denoising steps could significantly limit the space of possible scene rearrangement. Currently, if one wants to move objects in a scene according to the initial and final states of our algorithm, one needs to run a motion planning algorithm. Extending our work to output motion plans, along with the final states, is worth pursuing.

Finally, LEGO-Net has only been shown to work well on relatively small room-scale scenes. Extending our work to operate on larger-scale scenes such as warehouses might require changes to some of our architecture choices, including strengthening translational invariance. Exploring such strategies remains an understudied challenge, which we continue to explore.