TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors

Overview

Section 2 contains more details of the methods described in the main paper. Section 3 provides additional details on the experiments. Section 4 provides additional evaluation of the networks.

Implementation details

We use the following actions: move forward, rotate right, rotate left, look up, look down, pick up, put down. We rotate in the yaw direction by 90 degrees, and rotate in the pitch direction by 30 degrees. We do not constrain our agent to grid locations. The RGB and depth sensors are at a resolution of 480x480, a field of view of 90 degrees, and lie at a height of 0.9015 meters. The agent’s coordinates are parameterized by a single $(x,y,z)$ coordinate triplet with $x$ and $z$ corresponding to movement in the horizontal plane and $y$ reserved for the vertical direction. Picking up objects occurs by specifying an (x,y) coordinate in the agent’s egocentric frame. If by ray-tracing, the point intersects an object that is pickupable and within 1.5 meters of the agent, then the pickup action succeeds. Placing objects occurs by specifying an (x,y) coordinate in the agent’s egocentric frame to place the object. If by ray-tracing, the point intersects an object that is a receptacle class, has enough free space in the radius of the target location, and within 1.5 meters of the agent, then the place action succeeds if the agent is holding an object. Since some objects require their state to be open for placement to successfully occur (e.g. Fridge), the agent will also try to open the receptacle if placement initially fails.

2 Pseudo code for TIDEE

We present pseudo code for the TIDEE algorithm in Algorithm 1. We denote FMM to mean Fast Marching Method [sethian1996fast], g to denote the point goal in the 2D overhead map $\mathbf{M}^{\textrm{2D}}$, $r$ to denote a receptacle, and fps to denote farthest point sampling. If TIDEE does not find one of the predicted receptacles from the rGCN network, TIDEE will attempt to retrieve a general receptacle class from its memory of detected objects, navigate there, and attempt to place it. If after $m$ placement attempts the object is still not placed successfully (for example if TIDEE gets stuck while navigating), TIDEE will drop the object at its current location and resume the out-of-place search.

3 Semantic mapping and planning

TIDEE explores the scene using a classical mapping method. We take the initial position of the agent to be the center coordinate in the map. We rotate the agent in-place and use the observations to instantiate an initial map. Second, the agent incrementally completes the maps by randomly sampling an unexplored, traversible location based on the 2D occupancy map built so far, and then navigates to the sampled location, accumulating the new information into the maps at each time step. The number of observations collected at each point in the 2D occupancy map is thresholded to determine whether a given map location is explored or not. Unexplored positions are sampled until the environment has been fully explored, meaning that the number of unexplored points is fewer than a predefined threshold.

To navigate to a goal location, we compute the geodesic distance to the goal from all map locations using a fast-marching method [sethian1996fast] given the top-down occupancy map $\mathbf{M}^{\textrm{2D}}$ and the goal location in the map. We then simulate action sequences and greedily take the action sequence which results in the largest reduction in geodesic distance.

4 2D-to-3D unprojection

For the $i$-th view, a 2D pixel coordinate $(u,v)$ with depth $z$ is unprojected and transformed to its coordinate $(X,Y,Z)^{T}$ in the reference frame:

We voxelize the point cloud into a 128x64x128 occupancy $\in\{0,1\}$ centered at the initial position of the agent, and aggregate (take max) the occupancies across views to obtain $M_{t}^{o}\in\{0,1\}$.

5 Object tracking and semantic aggregation.

6 Out-of-place detector

As described in Section 3.2 of the main paper, our OOP detector makes use of visual and relational language as input to our OOP network. We generate training scenes with some objects out-of-place using the same algorithm described in Section 3.1. We first finetune deformable-DETR [zhu2020deformable] (pretrained on COCO [lin2014microsoft]) on the training houses (object seed randomized) to predict the bounding boxes, semantic segmentation masks, and semantic labels by generating random trajectories through the scene. We then train on the messup configurations and add an additional classification loss on the output decoder queries to predict whether the object is in- or out-of-place. We use the output decoder queries for the dDETR-OOP classifier.

For the language detector, we freeze the detector described above, and use it to update our object tracker $\mathcal{M}^{\textrm{O}}$ while the agent explores the scene. Then, the agent visits a location to search for an out-of-place object and for each object detected in view above a confidence threshold, we infer its relations described in Section 2.7 with all objects in memory, and systematically combine them into a paragraph of text. An example paragraph is shown below. The pillow is next to the key chain. The pillow is next to the laptop. The pillow is next to the side table. The pillow is next to the mug. The pillow is next to the teddy bear. The pillow is supported by the side table. The pillow is closest to the mug. We make use of the extensive pretraining of the BERT language model [devlin2018bert] as a starting point for our language classifier. We tokenize the paragraph text and give it as input to the BERT model. For the language-only detector (BERT-OOP), we give the pooled output {cls} token from BERT to a three-layer fully-connected classifier to predict in or out-of-place.

For the language and visual detector (dDETR+BERT-OOP), we concatenate the pooled output {cls} token from BERT with the output query embedding corresponding to the detected object from deformable-DETR, and give this concatenated embedding to a three-layer fully-connected classifier to predict in or out-of-place. We train the classifiers using known labels of in or out-of-place from our mess up algorithm.

For the BERT-only model, we give the pooled output {cls} token from BERT as input to our classifier. For the visual-only model, we give the output query embedding corresponding to the detected object from deformable-DETR to the classifier.

We use the same hyperparameters for training all classifiers. We use a batch size of 25, an AdamW optimizer with a learning rate of 2e-7 and weight decay of 0.01, and train for 20k iterations.

7 Object centroid relations

As described in Section 3.2 of the main paper, we define a set of three relations based on the estimated centroids of the detected objects within the scene. We use these relations for building our input to the BERT out-of-place detector. These relations are computed with the following metrics:

(i) Supported-by: A receptacle is defined as a type of object that can contain or support other objects. Sinks, refrigerators, cabinets, and tabletops are some examples of receptacles. For the floor receptacle class, we consider the point directly below the object at the height of the floor (lowest height in our map). For all centroids $C_{t}^{\textrm{rec}}$ corresponding to receptacle classes $L_{t}^{\textrm{rec}}\subseteq L_{t}$, we define the single object $L^{\textrm{supp}}\in L_{t}^{\textrm{rec}}$ that supports the detected $C^{det}$ object as:

Where $D(x,Y)$ is the euclidean distance between centroid $x$ and each centroid in $Y$, and $\textrm{ydiff}<0$ takes all tracked centroids which are below the height of the detected centroid.

(ii) next-to: We define the objects $L^{\textrm{next}}$ that are next to the detected $C^{\textrm{det}}$ object as:

Where $D(x,Y)$ is the euclidean distance between centroid $x$ and all centroids $Y$, and d is a distance threshold.

(ii) closest-to: We define the single object $L^{\textrm{closest}}$ that is closest to the detected $C^{det}$ object as:

Where $D(x,Y)$ is the euclidean distance between centroid $x$ and all centroids $Y$.

8 Relational graph convolutional network

As described in Section 3.3 of the main paper, we use a relational graph convolutional network to predict plausible receptacle classes for the out-of-place object. The memex graph nodes are the sum of a learned object category embedding and visual features obtained from cropping the deformable-DETR backbone with the object’s bounding box at the closest navigable location to the object. We connect nodes in the memory graph by computing their relations as described in Section 2.9. For the out-of-place object node, we similarly sum the learned embedding of the object’s category label and visual features obtained from cropping the deformable-DETR backbone with the detected bounding box. The scene graph nodes are deformable-DETR output query features in the initial mapping of the scene for all detections above a confidence threshold. We include a map type node which is initialized with a learned embedding for each of the four room types.

We use the rGCN to message pass 1) within the memory graph, and 2) to bridge the memory, scene, and out-of-place nodes. Let $n_{\textrm{OOP}}$ denote the node of the out-of-place object initialized with a learned category class embedding and visual features.

Following the rGCN formulation in [schlichtkrull2018modeling], we first update the nodes in the memory graph to distribute information within the memory:

Inspired by [zareian2020bridging], we then define a set of four bridging edges $\mathcal{R}^{bridge}$, one to connect $n_{\textrm{OOP}}$ to the updated memory nodes of the same object class, one to connect $n_{\textrm{OOP}}$ to all current scene nodes, one to connect $n_{\textrm{OOP}}$ to the room type node, and one to connect the the updated memory nodes to current scene nodes with the same category label. We then message pass via the bridging edges:

where $\mathcal{N}^{bridge}_{i,r}$ denotes the set of bridge neighbor indices of the target node under bridge relation $r\in\mathcal{R}^{bridge}$.

We use four relational graph convolutional layers for each stage of message passing. Finally, we run the updated out-of-place object node through a classifier layer to predict a probability distribution over proposed receptacle classes to search for placing the target object. We optimize with a cross entropy loss using the object’s ground truth receptacle label from the training scenes.

9 Memex graph

We use 20 of the 80 training rooms to construct the memex graph. As described in section 3.3 of the main paper, the memex graph is a large graph of object nodes and relational edges that provide the relational graph convolutional network with exemplar context of object-object and object-scene relations. We obtain the ground truth category labels for the objects and use ground truth information from the simulator to obtain the relations above, below, next to, supported by, aligned with, and facing. The memex remains a constant graph throughout all remaining training and testing scenes. We use simulator ground truth information for convenience, but note that we could instead obtain the neural memex graph from human annotations of real-world houses. We compute above, below, next to, and supported by similar to Section 2.7, but instead use a distance metric on the 3D bounding boxes. For aligned with, we check if the 3D bounding boxes have parallel faces. For facing, we note that the back of an object usually carries more of its mass (e.g. the back of a sofa). Thus, we look at the mass distribution of the object within its 3D bounding box, and take the box face with the most of the point mass in its direction to be the back of the object. An object is facing a second object if the frustum of its front 3D bounding box face intersects the second object. We only consider facing for the following classes: Toilet, Laptop, Chair, Desk, Television, ArmChair, Sofa, Microwave, CoffeeMachine, Fridge, Toaster.

10 Visual search network

Experimental details

Our tidying task begins with moving $N$ objects out of their natural locations in the scene. We use $N=5$ and generate five messy configurations per test room (total of 20 rooms $\times$ 5 configurations = 100 test configurations). For each object to be moved out-of-place, we randomly select a pickupable object, spawn an agent to a random navigable location in the scene at a random orientation in increments of 90 degrees, and with probability $p$, drop the object at the agent’s location, or with probability $1-p$, throw the object with a constant force and let AI2THOR’s physics engine resolve the final location (action ”ThrowObject” in AI2THOR). We use $p=0.5$. In AI2THOR, the throw distance of an object depends on its pre-defined mass, and thus the throw distance will change depending on the object. We keep the throw force constant at 150.0 newtons. We disable object breaking so that no objects are changed to their breaking state after dropping or throwing them. We show examples of out-of-place objects in Figure 1.

We define an episode as the time from the spawn of the agent in the messy environment to the time the agent executes the “done” action, or 1000 steps have been taken (whichever comes first). Once the tidying episode begins, the agent is spawned near the center of the map. At each time step, the agent is given an RGB and depth sensor, and its exact egomotion in terms of how far each action takes the agent and in what direction. During the out-of-place detection phase, TIDEE samples random locations within its 2D map to search.

2 Human placement evaluation

We report in Section 4.2 of the main paper a human evaluation of TIDEE placements compared to baselines. We use the Amazon Mechanical Turk interface to query human evaluators as to whether they prefer TIDEE placements compared to baseline placements. For all successful placements by the agents, we generate three images of each placement to show the object from three distinct viewing angles, as shown in Figure 2. We instruct the evaluators to choose between the placements of TIDEE and the baseline placement by looking at the images and picking which position of the object they would prefer. The full instructions given to the human evaluators for an example statue placement is displayed below. For this evaluation, we only consider objects which were picked up by both agents (TIDEE and the baseline).

Consider a scenario where you are putting the statue into its correct location in a room. Please choose which location you would prefer to place the statue within the room. The two options (A & B) represent two different possible locations of the statue in the same room (in the images the location of the statue is shown with a box). Each option (A & B) show the object from three distinct camera angles to help you make your decision. Important: Please judge only by the placement location of the object within the room, and NOT by the orientation of the object on the supporting surface.

3 Out-of-place detection evaluation

We evaluate the out-of-place detector performance in Section 4.3 on the same messy test scenes used for the tidying-up task. We generate 20 random views of each messy configuration where at least one out-of-place objects is in view. The total evaluation consists of 2000 images (20 scenes $\times$ 5 configurations $\times$ 20 views = 2000). We evaluate each detector by measuring average precision across all the images, where in and out-of-place are the two categories.

4 Exploration with visual search network evaluation

We evaluate the visual search network to assist in object goal navigation for objects in their default locations in the AI2THOR test scenes (20 scenes in total) in Section 4.4. For each test scene, the agent is tasked with finding each object category that exists at least once in the test scene. Each episode involves finding an instance of a given category. We consider all object categories across the AI2THOR simulator (116 categories). Tasking the agent under these specifications provides 591 total episodes in the evaluation. As mentioned in the main text, the agent is successful when the agent is within 1.5 meters of the target object and the object is visible to the agent. To declare success, the agent must execute the ”Stop” command. If ”Stop” is not executed within the maximum number of steps (200 max), the episode is automatically considered a failure and the next episode will begin. Both TIDEE and the baseline presented in Table 2 of the main text use the same object detector and navigation modules from Section 3.1 of the main paper. The only difference is how the model selects locations in the scene to search for the object-of-interest. For both TIDEE and the baseline, the agent executes the ”Stop” command after the object category has been detected above a threshold and the agent has navigated to the detected object using the estimated 3D centroid.

5 Updating placement priors by instruction

We show that we can alter the output of the language out-of-place detector by pairing specific language input with a desired label after additional training in Section 4.3. To do so, we first train the language detector (BERT-OOP) as described in Section 2.6 and Section 3.2 of the main paper. We then target a relation-label pairing. For example, we may want the relation ”alarm clock supported-by the desk” to output the label ”out-of-place” (which does not appear in the unaltered training set) whenever the relation occurs. Then, for an additional amount of (9k) iterations, whenever the relation ”alarm clock supported-by the desk” appears in the training batch, we pair the sample with the ”out-of-place” label as supervision.

Additional results

In section 4.5 of the main paper, we report the performance of TIDEE on the 2022 rearrangement benchmark. We additionally report performance on the 2021 rearrangement benchmark in Table 1.

2 Visualizations of the Visual Search Network

In Section 4.4 of the main paper, we displayed visualizations of the Visual Search Network predictions. We provide additional visualizations of the sigmoid output of our Visual Search Network conditioned on an object category in test rooms in Figure 3. We display an overhead view of the full scene on the left, and the network predictions corresponding to the overhead spatial locations on the right conditioned on four randomly-selected object categories. Darker red corresponds to higher probability. The blue dot indicators plotted in the prediction maps correspond to the search locations for the agent to visit after thresholding and farthest point sampling (for # location = 3). The output generally puts the highest probability at plausible areas for the category to exist. However, occasionally the network puts high probability where it should not. For example, the network puts high probability near a dresser for category ”Bed”, or near the armchair for category ”Coffee Table”. This may be in part due to our training procedure to prioritize high recall over precision of the true location in our cross entropy weighting.

3 Evaluation of altering priors with natural language

In Section 4.6 of the main paper, we showed for a single example that we can alter the learned priors of the out-of-place detector using external language input. We augment training with nine additional object relation pairs that are among the most commonly found in the AI2THOR houses and pair the relation with an out-of-place label. The relation pairs include ”alarm clock is supported by desk” (from main text), ”Soap bottle is supported by countertop”, ”Pen is supported by desk”, ”Laptop is supported by desk”, ”Pillow is supported by bed”, ”Toilet paper is support by toilet”, ”salt shaker is supported by countertop”, ”Spatula is supported by countertop”, ”Statue is supported by shelf”, and ”Vase is supported by shelf”. We follow the same training procedure as in Section 3.5. The average change in probability across test houses for examples where the relation appears is shown in Table 2. The significant change in probability indicates we are able to change the detector output with simple language instructions.