The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI

Introduction

Household robots able to sense and act in the physical world remain an important goal of the computer vision and robotics communities. Because directly training models with real robots is expensive and involves safety risks, there has been a trend toward incorporating simulators to train and evaluate AI algorithms. In recent years, the development of 3D virtual environments that can simulate photo-realistic scenes has served as the major driving force for the progress of vision-based robot navigation .

Most tasks defined thus far in these virtual environments primarily focus on visual navigation in high-quality synthetic scenes and real-world RGB-D scans , and thus pay little or no attention to physical interaction. Since the ultimate goal of Embodied AI is to develop systems that can perceive and act in physical environments, it has become increasingly apparent that physical interaction is a necessary component for the training of home assistant robots. Recent work has aimed to improve sim-to-real transfer in the physical world by developing 3D virtual environments with both photo-realistic rendering and high-fidelity physics simulations (e.g. iGibson , TDW and SAPIEN ). iGibson made initial progress in this area, introducing an interactive navigation task with partial observations in a 3D cluttered environment. An agent with a mobile manipulator was encouraged to take interactive action (e.g. pushing obstacles away and opening a door) to navigate towards the goal position. But the task itself was still a relatively simple navigation task, and did not explicitly require agents to change object states.

In this paper, we propose a new embodied AI challenge: an agent must take actions to move and change the state of a large number of objects to fulfill a complex goal in a photo- and physically-realistic virtual environment. In our challenge, an embodied agent equipped with two 9-DOF articulated arms is spawned randomly in a physically-realistic virtual house environment. The agent is required to explore the house, searching for a small set of objects scattered in different rooms, and transporting them to a desired final location, as shown in Fig.1. We also position various containers around the house; the agent can find these containers and place objects into them. Without using a container as a tool, the agent can only transport up to two objects at a time. However, using a container, the agent can collect several objects and transport them together.

We build our challenge using ThreeDWorld (TDW) , an interactive physical simulation platform. We first create a TDW-based house dataset of multi-room environments filled with objects that respond to physics. We further develop a fully physics-driven high-level navigation and interaction API that we can use to train AI agents to physically interact with the virtual world. The embodied agent needs to pick up, move and drop objects using physics to drive a set of joints; this is fundamentally different from the approach taken by AI2-THOR and VirtualHome , where objects close to the agent simply “attach” to it or are animated to move into position, with no physical interaction at all. We believe that such a long-horizon task in the physically realistic environment poses several challenges for embodied agents beyond semantic exploration of unknown environments, including:

Synergy between navigation and interaction. The agent cannot move to grasp an object if this object is not in the egocentric view, or if the direct path to it is obstructed (e.g. by a table).

Physics-aware Interaction. Grasping might fail if the agent’s arm cannot reach an object.

Physics-aware navigation. Collision with obstacles might cause objects to be dropped and significantly impede the transport efficiency.

Reasoning about tool usage. While the containers help the agent transport more than two items, it also takes some time to find them. The agent thus has to reason about a case-by-case optimal plan.

We evaluate several existing agents on this benchmark. Experimental results suggest that existing state-of-the-art models for embodied intelligence all struggle to complete this task efficiently. We believe models that perform well on our transport challenge will enable more intelligent robots that can function in the real physical world. Our contributions are summarized as follow:

We introduce a new embodied AI task that aims to measure AI agents’ ability to change the states of multiple objects to accomplish a complex task in a photo- and physically-realistic virtual environment.

We have developed a fully physics-driven high-level command API that enables agents to execute interactive actions in this simulated physical world.

We evaluate several agents and find that a pure RL model struggles to succeed at the challenge task. The hierarchical planning-based agent achieves better performance but still far from solving this task.

Related Work

Our challenge builds on prior work on 3D interactive environments, embodied intelligence and task-and-motion planning.

Recently, the development of 3D interactive environments, including AI2-THOR, HoME , VirtualHome , Habitat , Gibson , Deepmind Lab , RoboTHOR , SAPIEN , and TDW have facilitated novel algorithms in visual navigation , visual-language navigation , embodied question answering and other tasks.

There are several stand-alone physics engines widely used in the robotics community, including PyBullet , MuJuCo , and V-REP . Many robotic manipulation and physical reasoning challenges are also built on top of these engines ( e.g. RL-Benchmark , Meta-World and CLEVRER ). However, these platforms cannot render photo-realistic images, limiting the visual perception abilities of systems trained therein. Most recently, physics-based simulators with realistic images and physics have been introduced , aiming to reduce the sim-to-real gaps seen when deploying trained systems in the physical world. However, we still lack challenging embodied AI benchmarks with a clear task and evaluation metric that require an agent to move and change multiple object states for a long-horizon task. We hope to fill this gap with the ThreeDWorld Transport Challenge. Our challenge involves complex physical scene understanding and a long-horizon task and can serve as a benchmark for running and testing embodied agents’ task-and-motion planning abilities in 3D simulated physical home environments.

2 Embodied Intelligence

The training of embodied agents in 3D virtual environments has generated increasing interest within the computer vision and robotics communities. Popular tasks include point-goal navigation , audio-visual navigation , vision-language navigation , and semantic-goal navigation . However, a majority of these navigation tasks are collision-free and do not involve much physical interaction.

Inspired by early work on Navigation Among Movable Objects (NAMO) , iGibson introduced an interactive navigation task in a cluttered 3D environment. In , an agent is encouraged to push movable objects with non-prehensile manipulators to clear a path to aid navigation. In , more complicated interactive navigation tasks (e.g. open a door) and a few mobile manipulation tasks (e.g. kitchen rearrangement) are developed. However, in both cases, the tasks have a shorter horizon and involve limited physical constraints compared with ours. We aimed to use a more advanced physical virtual environment simulator to define a new embodied AI task requiring an agent to change the states of multiple objects under realistic physical constraints. Perhaps the closest challenge task to that proposed here is a recently introduced Animal AI Olympic challenge . This challenge aims to test AI reasoning and planning abilities in a Unity game engine. However, the embodied agent is restricted to navigation actions, and does not contain interactive actions that are important for building household robots. Our new benchmark goes beyond these existing challenge datasets by requiring the agents to change multiple object states in a rich and realistic physical world. Hierarchical reasoning and planning abilities in the face of realistic physics constraints are necessary to succeed at our task.

Current state-of-the-art methods on embodied tasks mainly fall into two broad categories: end-to-end training of neural policies using RL or hierarchical RL and map building for path planning . We believe that the proposed ThreeDWorld transport challenge could provide new opportunities to develop novel algorithms that combine visual perception, reasoning and hierarchical planning to solve more challenging tasks in the physical world.

3 Task and Motion Planning

Our work falls in the domain of task and motion planning (TAMP) , recently reviewed in . The goal of TAMP is to operate a robot in environments containing a large number of objects, taking actions to move and change the state of the objects in order to fulfill a goal. A task planner is responsible for reasoning over the large set of states of the environment, and a motion planner computes a path to accomplish the task. TAMP has demonstrated its power in many mobile manipulation tasks . In our work, we hope to create a visually guided TAMP-like challenge that requires robots to carry out a complex and long-horizon task involving physical interaction in a cluttered 3D environment. Rearrangement of objects in an environment has recently been proposed as a potential test bed for embodied AI . Our work provides an example rearrangement challenge with a visually guided task-and-motion planning benchmark in a photo- and physically-realistic virtual environment.

ThreeDWorld Transport Challenge

The ThreeDWorld Transport challenge aims to assess an AI agent’s reasoning and planning abilities in a physical realistic environment. We build this challenge on top of the TDW platform , which is a general-purpose virtual world simulation platform supporting both near-photo-realistic image rendering, physically-based sound rendering , and realistic physical interactions between objects and agents.

TDW enables the training of embodied agents to perform tasks in a simulated 3D physical world. To support such training, we created 15 physically-simulated houses filled with objects that respond to physics. We also developed a high-level fully physics-driven navigation and interaction API. An agent equipped with an RGB-D camera and an oracle visual perception model can navigate through the virtual physical world to transport objects. Since the simulated actions and environment are fully physics-based, they pose additional challenges compared to previous non-physics or partial-physics virtual environments. For instance, interactive action only succeeds if the target is physically reachable (i.e. close by and not obstructed). The agent can not successfully go to grasp an object if this object is not in the egocentric view, or if the direct path to it is obstructed (e.g. by a table). And physical collisions with objects in the house might also significantly impede transport progress. Therefore, the agent must learn to leverage visual signals to synchronize navigation and manipulation under these physical constraints.

As shown in Figure 2, an embodied agent is required to transport a set of predefined objects to a target location. There are three types of objects that the agent needs to pay attention to:

Target objects: objects the agent needs to grasp and transport, placed at various locations in the house.

Containers: objects that can be used as tools to assist with the transport of target objects, also located in various rooms.

Goal position: a goal area is defined around one unique object (e.g, a bed) in the house.

The objective of this task is to transport as many objects as possible, given a limited interaction budget. The robot agent can only transport two objects with their arms, but it can transport more with the container. In each task instance, the target object positions and the object list for transport are varied, so the agent must learn how to explore the home environment to find target objects, containers, and goal positions. They can then plan a sequence of actions to transport the target objects to their desired location. Since both the objects in the room and high-level actions are responsive to physics, the robot agent needs to learn to use vision to decide when to go to grasp objects that can be successfully reached while avoiding collisions (e.g. objects and/or the container might fall if the agent collides with obstacles) under the realistic physic constraints.

2 Scenes

We have created a TDW-House dataset to support this challenge. As shown in Figure 2 (c), each house environment contains between 6 and 8 interconnected rooms such as bedrooms, living rooms, and kitchens. These rooms are fully populated with furniture and other items (see “Objects” below). The agent’s view might be blocked by obstacles such as walls, so it needs to explore the house in order to predict a full occupancy map of the scenes.

The dataset is modular in its design, comprising several physical floor plan geometries with wall and floor material variations (e.g. parquet flooring, ceramic tile, stucco, carpet etc.) and various furniture and prop layouts (tables, chairs, cabinets etc.), for a total of 15 separate environments. Furniture arrangements were interactively laid out by a 3D artist using 3D assets from the TDW model library . We will make this dataset publicly available.

3 Objects

To build physical house environments for the agent to navigate, we also populated the various floor plans with around 50 categories of objects. These include typical household items like lamps, toys, vases, pillows, printers, and laptops. Except for non-movable objects like walls, all objects respond to physics, so the embodied agent can interact with them.

4 Embodiment

We use the Magnebot as an embodied agent. The Magnebot is equipped with an RGB-D camera and also capable of returning image masks for object ID segmentation, semantic segmentation, normals and pixel flow. For interactions with scene objects, the avatar has two articulated arms with 9-DOF “magnet” end-effectors for picking up objects (see supplementary material for details). All motion and arm articulation actions are fully physics-driven.

The observation space in this challenge contains a first-person view RGB image, a depth map, and a semantic segmentation mask. The agent could use these observations to estimate the occupancy map (see below). And it can also obtain an object list based on the semantic segmentation mask.

5 Action space

We developed a high-level API with action commands designed to fit our task definitions, enabling an agent to move and manipulate objects. Our design choices were intended to abstract away the details of low-level control, which could otherwise form a barrier to exploring novel solutions for this kind of TAMP-style embodied AI challenge.

There are two types of actions we consider in this transport challenge: navigation and interactive actions. API functions for navigation include Move Forward By ($x$ m), Rotate By ($\theta$ degrees), Rotate To (an object or target position). In our experiments, we make the actions discrete by setting $x=0.5meter$, $\theta=\pm 15^{\circ}$. Rotate to $+15^{\circ}$ and Rotate to $-15^{\circ}$ means to rotate left or right by 15 degrees, respectively. Interactive action functions include Go to Grasp, Drop and Put In Container. Taking as an example a simple task that involves the agent selecting an object to pick up based on the segmentation mask returned by sensors, going to grasping it, then moving to a container and dropping the object into the container, the action sequence we applied is Go to Grasp (object), Go To Grasp (container), and Put In Container. We use Inverse Kinematics to implement these interactive actions. The details can be found in the supplementary materials.

Since our high-level action API is fully physics-driven, these features also pose new challenges that have not appeared in previous navigation environments, but indeed exist in the real physical world. For example, the agent might fail to grasp one target object when we apply a Go to Grasp action. The reason might be that its arm cannot reach the target object. In this case, the agent needs to move close to the object and make an attempt again. We also observe that the objects/container held by the avatar can fall off if they hit a heavy obstacle during the navigation. The agent would then have to find and pick up these items again. 3433

6 Action Status

When performing tasks, the simulation will also provide feedback on its status and the status of its current action, in order to decide what action to take next. Each API function returns an ActionStatus value indicating the status of the agent – whether it is performing an action, succeeded at an action, or failed to complete the action. There are 18 types of action status, including: whether a target object is too close or too far to reach, or is behind the agent; whether the agent succeeded or failed to pick up an object, or overshot the target; whether the agent collided with something heavy or is obstructed by an obstacle, etc. We can use these meta data to define a reward function for policy learning or priors for planning.

7 Dataset Generation

Target Objects The objects that the agent needs to transport as part of the challenge task are known as “target objects”. These objects are located at “target positions” within the environment, guaranteed to be reachable by the agent. The set of target objects to be transported are randomly placed across different rooms. In addition, there is a 25% chance of spawning a container at a traversable position in each room. When a scene is initialized, the simulation loads a pre-calculated “occupancy map”, i.e. a data file representing grid spaces occupied by objects.

Goal Positions The locations within the environment where target objects must be deposited to complete the task are known as “goal positions”. The surfaces that target objects are to be deposited on are known as “goal position surfaces”. Goal position surfaces can be on any of the following furniture objects: Sofa, Bench, Table, Coffee table, and Bed. For each task, we set one unique piece of furniture in the scene as the goal position (e.g. only one coffee table, or one sofa in the house) in the TDW-House dataset.

Robot Agent Initialization After the scene configuration is initialized, the robot agent is spawned at an open location that is free of other objects.

Task Definition We define the goal of each task with two components: 1) a set of objects and their counts, 2) the Goal Position. For instance, “vase:2, bowl:2, jug:1; bed” means that the agent must transport 2 vases, 2 toys, and 1 jug to the bed. There are five types of objects used as potential target objects to be transported. Each task requires transporting 6 to 8 objects.

Experiments

In this section, we discuss our experimental setup, baselines, implementation details and evaluation results.

Setup. We use 15 physically distinct houses for the experiments. Example house layouts can be found in Figure 3. Since this challenge aims to test the agent’s generalization abilities, we split them into 10 seen houses (training) and 5 unseen houses (test). We generated 100 tasks for each seen house and 20 tasks for each unseen house by randomly placing target objects and containers into different rooms as described above (for a total of 1000 training and 100 test tasks). We report the models’ performance on the test set tasks.

Evaluation Metrics. The objective of this challenge is to transport the maximum number of objects as efficiently as possible. We use the transport rate as an evaluation metric. The transport rate is the fraction of the objects successfully transported to the desired position within a given interaction budget (defined as a maximum episode length in steps). For testing we set this maximum episode length to 1000 steps.

2 Baseline Models

We implemented several baseline agents using both learning- and planning-based algorithms. We found empirically that it was difficult for an end-to-end RL learning model (Pure RL baseline) to take observation data as input and directly search for an optimal policy over the agent’s action space, presumably because our task is very challenging (e.g. long-horizon, partial observability, involving complex physical interaction, sparse rewards, etc.). Therefore, we implemented additional baselines using a hierarchical planning framework with different exploration strategies.

As shown in Figure 4, we define four types of sub-goals for this task: exploration, pick up a container, pick up an object, and place. We use a rule-based high-level planner to decide when to transition between sub-goals. For the exploration sub-goal, the agent will either directly use the policy to navigate around the environment (RL Exploration baseline) or plan a shortest path from the current location to the waypoints (Active, Frontier and Semantic Exploration baselines). The agent can accumulate RGB images, depth maps, and segmentation masks to construct a 2D occupancy map and a semantic map during its exploration. When setting the sub-goal as pick up a container or pick up an object, the agent will plan a sequence of actions to move towards a container or a target object and pick it up. For the place sub-goal, the agent uses a planner to generate a deterministic local policy to go to the goal position and drop the target objects. We summarize the baseline models as follows:

Pure RL: We train an end-to-end RL policy using PPO by maximizing the reward of finding objects (either target objects, containers, or the goal position), grasping objects, putting objects into a container, and dropping them onto the goal location. This model takes the inputs of observation and estimated occupancy map and directly outputs actions that the agent should execute in the environment.

RL Exploration: We train a policy network that maximizes occupancy map coverage. The network yields an action to execute.

Frontier Exploration: Similar to , the agent uses depth images and semantic segmentation masks to construct an occupancy map and a semantic map of the scene. The agent randomly samples a waypoint from an unexplored area as a sub-goal for exploration. It marks the location of target objects, containers, and the goal position on its semantic map.

Active Exploration: We adapted a baseline from . Instead of randomly choosing an unexplored position in the occupancy map as a goal location to explore (as in the Frontier baseline), the Active baseline learns a goal-agnostic policy by maximizing map coverage (using neural SLAM). The output is a waypoint position. The local exploration planner uses this waypoint as a sub-goal.

Semantic Exploration: We adapted a baseline from . The Semantic baseline uses neural SLAM to learn an object/goal-oriented “semantic” exploration policy. The reward function encourages the agent to find more target objects, containers, and the goal position.

3 Implementation Details

For all the baselines, the input observation data are 256 $\times$ 256 size RGB images, depth maps, and semantic segmentation masks. The action spaces include 3 navigation actions (i.e. move forward, rotate left, and rotate right) and 3 interactive actions (i.e. go to grasp, put in container, and drop). Below we provide the details of the occupancy map, semantic map, policy training, high-level and low-level planners.

Occupancy Map. We represent the global top-down occupancy map as $O_{g}\in R^{2\times N\times N}$, where N$\times$N denotes the map size and each element in this occupancy map corresponds to a region of size 0.25m$\times$0.25m in the physical world. We can recover a local occupancy map $O_{l}\in R^{2\times K\times K}$ from egocentric depth map, which comprises a local area in front of the camera. The two channels in each cell represent the probability scores of the cell being occupied and explored, respectively. We considered a cell to be occupied if there is an obstacle and a cell to be explored if the agent already knows whether it is occupied or not. The occupancy map is initialized with all zeros and the agent always starts at the center of the map at the beginning of each episode.

Semantic Map. Similar to the occupancy map, we can also construct a global top-down semantic map $S\in R^{3\times N\times N}$ using both depth images and semantic segmentation masks. The value in each channel indicates if a cell contains target objects, containers or goal position. Similar to the occupancy map, we also initialized the semantic map with all zeros and set the agent starting position as the map center. The agent can gradually update the semantic map when they find objects during the navigation exploration.

Policy learning. For all policy learning, we use a four-layer CNN network and train for 10 million frames using Proximal Policy Optimization . In the Pure RL, RL Exploration, and Semantic Exploration baselines, we take the depth image, estimated occupancy map, and semantic map as input. In the Active Exploration baseline, we take the depth image and estimated occupancy map as inputs. In the Pure RL and RL Exploration baselines, the model directly outputs an action. For Active and Semantic Exploration baselines, the models predict waypoints as sub-goals. We use the same reward function for the Pure RL and Active Exploration baselines by encouraging the agent to find new objects, grasp target objects, put objects into a container, and drop objects onto the goal position. In the RL Exploration and Active Exploration baselines, we use a goal-agnostic reward function by encouraging the agent to improve the map coverage.

High-level planner. We define a rule-based high-level planner to decide the sub-goal. The agent takes exploration as a sub-goal by default. At the beginning, the sub-goal can switch from exploration to picking up a container once a container has been found. After a container is found or 20% of the interaction budget has been used up, the agent can switch the sub-goal from exploration to picking up an object if they find a target object. After 90% of the interaction steps, the agents will switch their sub-goal to place. If there is no container found, the agent will switch their sub-goal from picking up an object to place if they hold two objects.

Low-level planner. We use a A-Star based planner to find the shortest path from the current location to the sub-goal location and execute the high-level interactive actions for achieving the sub-goal. Every interaction steps, we update the map and re-plan the path to the sub-goal.

4 Result Analysis

In this section, we first present the overall benchmark results, and then perform in-depth analysis of the model design. We hope that our experimental findings will provide useful benchmarks for physically realistic Embodied AI.

Overall Results. We first plot the histogram of transport rate for all five unseen houses in Figure 5. We calculate the results by averaging over all 20 testing tasks in each house. We also summarize the overall quantitative evaluation results of all baseline models measured by average transport rates of 5 unseen rooms in Table 1. The best model achieves an average of 0.52 transport rates under 1000 episodes. We have two key observations: 1) There are no agents that can successfully transport all the target objects to the goal locations. This means our proposed task is very challenging and could be used as a benchmark to track the progress of embodied AI in physically realistic scenes. 2) The Pure RL baseline performs poorly. We believe this reflects the complexity of physical interaction and the large exploration search space of our benchmark. Compared to the previous point-goal navigation and semantic navigation tasks, where the agent only needs to navigate to specific coordinates or objects in the scene, the ThreeDWorld Transport challenge requires agents to move and change the objects’ physical state in the environment(i.e. task-and-motion planning), which the end-to-end models might fall short on.

Exploration policy. The results also indicate that combining RL exploration policy into a hierarchical planning framework could significantly improve results over a purely RL-based approach (better performance of all Exploration baselines over Pure RL). However, predicting a way-point (Frontier, Active, and Semantic Exploration) produced better results than predicting an action (RL Exploration). We speculate that RL Exploration can find different positions, but that it is hard for this model to learn complex physical interactions. In practice, we observed that this agent always failed to grasp objects, and frequently collided with the obstacles, resulting in a low transport rate. These results support the idea that our benchmark can enable novel algorithms to combine learning-based and planning-based methods to solve complex tasks in physically realistic environments.

Neural-based Exploration. We also found that learning to generate a waypoint for exploration (in the Active and Semantic Exploration baselines) did not show clear advantages over the model without learning (Frontier Exploration). The reasons might be two-fold: 1) the layout of our test scenes are quite different from those of the training scenes. 2) The occupancy and semantic maps used for learning method do not convey much physics information. We believe future work that explicitly incorporates physics information might help improve the generalization abilities of learning-based methods.

Qualitative Results. We further visualize the trajectory of different agents in Figure 6. We can observe that the planning-based agent using waypoints as sub-goals can always find the shortest path to grasp objects and transport them to the goal locations. The RL agent is good at exploration, but might frequently hit obstacles, thus fail to transport objects efficiently. We will also provide demo videos in the supplementary materials.

Conclusions and Future Work

We introduce a visually-guided task-and-motion planning benchmark that we call the ThreeDWorld Transport Challenge. Our preliminary experimental results indicate that the proposed challenge can assess AI agents’ abilities to rearrange multiple objects in a physically realistic environment. We believe our benchmark will also remove the barrier to entering the field of Task and Motion Planning (TAMP), allowing more people to study such a challenging TAMP-style embodied AI task in the face of realistic physical constraints. We plan to include deformable or soft body objects in future versions of this challenge.

References

Appendix A Magnebot Details

At a low level, the Magnebot is driven by robotics commands such as set_ revolute_target, which will turn a revolute drive. The high-level API combines the low-level commands into ”actions”, such as grasp(target object) or move_by(distance).

To facilitate reaching and grasping we utilize an inverse kinematics (IK) solver to reach the desired target object or position.

The Magnebot end-effectors achieve 9 degrees of freedom from a 3-DOF spherical joint at its “shoulder”, a 1-DOF revolute joint at its “elbow”, and a 3-DOF spherical joint at its ”wrist”; the two additional DOF come from the ability to move the entire torso structure up and down the central column, as well as rotate the torso left/right about the column axis.

The Magnebot moves by turning its wheels, each of which is a revolute drive. When turning, the left wheels will turn one way and the right wheels in the opposite way, allowing the Magnebot to turn in place.

Appendix B Implementation Details of High-Level API

The ThreeDWorld Transport Challenge is built on top of the ThreeDWorld (TDW), a high-fidelity platform for interactive physical simulation.

A TDW simulation consists of two basic components: (i) the Build or runtime simulation engine; and (ii) the Controller, an external Python interface to communicate with the build. Running a simulation follows a cycle in which: 1) The controller sends commands to the build; 2) The build executes those commands and sends simulation output data back to the controller.

B.2 High-level API.

TDW’s Python API contains over 200 commands covering a broad range of use case scenarios. We can use this low-level “atomic” API as building-blocks to create controller programs to run simulations. For the ThreeDWorld Transport Challenge we created a new set of high-level commands built on top of TDW’s existing API, significantly reducing the number of commands required to perform complex actions such as grasping objects and placing them into containers. The Transport Challenge follows the OpenGym API convention. Each executed action will return new observation and scene state data as well as a status value indicating whether the action succeeded or failed. The Transport Challenge action space includes:

move_forward() Move forward by a given distance.

rotate_left() Rotate left by a given angle.

rotate_right() Rotate right by a given angle.

go_to_grasp(): Move to a target object or container and bend the arm using an inverse kinematics (IK) solver to grasp the object.

put_in_container() Bend the agent’s arms using an IK solver in order to put an object held by one magnet into a container held by another magnet.

Appendix C Full List of Action Status Values

The high-level API functions in this simulation platform return ActionStatus values indicating whether the action succeeded or failed (and if so, why). Some of these values are not used in the Transport Challenge API. We can use this meta data to define reward functions. The full set of ActionStatus values includes:

failed_to_move: Tried to move to a target position or object but failed.

failed_to_turn: Tried to turn but failed to align with the target angle, position, or object.

cannot_reach: Didn’t try to reach for the target position because it can’t.

failed_to_reach: Tried to reach for the target but failed; the magnet isn’t close to the target.

failed_to_grasp: Tried to grasp the object and failed.

not_holding: Didn’t try to drop the object(s) because it isn’t holding them.

clamped_camera_rotation: Rotated the camera but at least one angle of rotation was clamped.

failed_to_bend: Tried to bend its arm but failed to bend it all the way.

collision: Tried to move or turn but failed because it collided with the environment (such as a wall) or a large object.

tipping: Tried to move or turn but failed because it started to tip over.

not_in: Tried and failed to put the object in the container.

still_in: Tried and failed to pour all objects out of the container.

Appendix D SceneState data

In addition to the ActionStatus values, this simulation platform also provides per-action SceneState data per-frame data that an agent can use to decide what action to take next. This data includes:

Visual data – rendered image, object color segmentation and depth values as numpy arrays

The projection matrix of the agent’s camera as a numpy array

The camera matrix of the agent’s camera as a numpy array

The positions and rotations of each object in the scene

The position and rotation of the Magnebot

The position and angle of each of the Magnebot’s joints

Which objects are currently held by each of the Magnebot’s magnets