Inner Monologue: Embodied Reasoning through Planning with Language Models

Introduction

Intelligent and flexible embodied interaction requires robots to be able to deploy large repertoires of basic behaviors in appropriate ways, sequence these behaviors as needed for long horizon tasks, and also recognize when to switch to a different approach if a particular behavior or plan is unsuccessful. High-level planning, perceptual feedback, and low-level control are just a few of the sub-tasks that would need to be seamlessly combined together to perform the sort of reasoning required for an embodied agent, such as a robot, to intelligently act in the world. While conventionally these challenges have been approached from the perspective of planning (e.g., TAMP ) or hierarchical learning (e.g., HRL ), effective high-level reasoning about complex tasks also requires semantic knowledge and understanding of the world.

One of the remarkable observations in recent machine learning research is that large language models (LLMs) can not only generate fluent textual descriptions, but also appear to have rich internalized knowledge about the world . When appropriately conditioned (e.g., prompted), they can even carry out some degree of deduction and respond to questions that appear to require reasoning and inference . This raises an intriguing possibility: beyond their ability to interpret natural language instructions, can language models further serve as reasoning models that combine multiple sources of feedback and become interactive problem solvers for embodied tasks, such as robotic manipulation?

Prior studies show that language helps humans internalize our knowledge and perform complex relational reasoning through thinking in language . Imagine the “inner monologue” that happens when a person tries to solve some task: “I have to unlock the door; let me try to pick up the key and put it in the lock… no, wait, it doesn’t fit, I’ll try another one… that one worked, now I can turn the key.” The thought process in this case involves choices about the best immediate action to solve the high-level task (“pick up the key”), observations about the outcomes of attempted actions (“it doesn’t fit”), and corrective actions that are taken in response to these observations (“I’ll try another one”). Inspired by the human thought process, we propose that such an inner monologue is a natural framework for incorporating feedback for LLMs.

Our work studies these questions by combining LLMs with various sources of textual feedback, only utilizing few-shot prompting without any additional training. We observe that similarly to recent work , natural language provides a universal and interpretable interface for such grounding of model communication and allows them to incorporate their conclusions in an overarching inner monologue driven by a language model. While prior work has investigated using language models as planners or incorporating multimodal-informed perception through language , to the best of our knowledge no work has studied the critical link of not only planning with language, but also informing embodied feedback with language, which we investigate in this work.

Specifically, we study methods and sources of feedback for closing the agent-environment loop via an inner monologue and their impact on downstream execution success and new capabilities arising from such interaction. In particular, we combine multiple perception models that perform various tasks such as language-conditioned semantic classification or language-based scene description, together with feedback provided by a human user that the robot is cooperating with. To execute the commands given by a user, the actions are chosen from a set of pre-trained robotic manipulation skills together with their textual descriptions that can be invoked by a language model. Our proposed system Inner Monologue chains together these various components (perception models, robotic skills, and human feedback) in a shared language prompt, enabling it to successfully perform user instructions.

Finally, we show that Inner Monologue, without requiring additional training beyond a frozen language model and pre-trained robotic skills, can accomplish complex, long-horizon, and unseen tasks in simulation as well as on two real-world robotic platforms. Notably, we show that it can efficiently retry under observed stochastic failure, replan under systematic infeasibility, or request human feedback for ambiguous queries, resulting in significantly improved performance in dynamical environments. As a demonstration of the versatility of LLMs and grounded closed-loop feedback, we additionally show several surprising capabilities emerging from the inner monologue formulation, including continued adaptation to new instructions, self-proposed goals, interactive scene understanding, multilingual interactions, and more.

Related Work

Task and motion planning requires simultaneously solving a high-level, discrete task planning problem , and a low-level, continuous motion planning problem . Traditionally, this problem has been solved through optimization or symbolic reasoning , but more recently machine learning has been applied to aspects of the problem via learned representations, learned task-primitives, and more . Some works utilize language for planning and grounding . Others have approached the problem through hierarchical learning . In this work, we leverage pre-trained LLMs and their semantic knowledge, along with trained low-level skills, to find feasible plans.

Various prior works have explored using language as a space for planning . Similar to ours are recent task planning approaches that leverage pre-trained autoregressive LLMs to decompose abstract, high-level instructions into a sequence of low-level steps executable by an agent in a zero-shot manner. Specifically, Huang et al. prompt GPT-3 and Codex to generate action plans for embodied agents, where each action step is semantically translated to an admissible action with a Sentence-RoBERTa model . SayCan instead grounds the actions by multiplying each candidate action’s probability under FLAN with the action’s value function, which serves as a proxy for affordance . However, both approaches effectively produce the plan while assuming that each proposed step is executed successfully by the agent. As a result, these approaches may not be robust in handling intermediate failures in dynamic environments or with poor lower level policies. We explore in Inner Monologue ways to incorporate grounded feedback from the environment into the LLM as we produce each step in the plan.

Various works have investigated strategies for the challenging problem of fusing vision, language, and control . While pretrained LLMs typically train only on text data, pretrained visual-language models (e.g., CLIP ) are trained on joint image(s) and corresponding text captions via variants of masked language modeling (MLM) objective , contrastive loss or other supervised objectives. CLIP has been employed in several robotics and embodied settings in zero-shot manner , or combined with Transporter networks as in CLIPort . Socratic Models combines several foundation models (e.g., GPT-3 , ViLD ) and language-conditioned policies, using language as the common interface, and demonstrated manipulating objects in a simulated vision-based robotic manipulation environment.

Leveraging Embodied Language Feedback with Inner Monologue

In this section, we introduce how LLMs can act as interactive problem solvers and incorporate embodied environment observations into grounded planning through a process we refer to as Inner Monologue.

2 Inner Monologue

We formulate an “inner monologue” by continually injecting information from the various sources of feedback into the LLM planning language prompts as the robot interacts with the environment. While LLMs have demonstrated exceptional planning capabilities for embodied control tasks , prior works have found it crucial to ground LLM predictions with external components such as affordance functions in order to produce useful plans that are executable by robots. However, LLMs used in this context have thus far remained one-directional – providing a list of skills, without making corrections or leveraging opportunities to replan accordingly. In contrast, Inner Monologue studies settings where grounded environment feedback is provided directly to the LLM in a closed-loop fashion. This promotes improved LLM reasoning in complex long-horizon settings, even before any external affordance-based grounding methods are applied.

Our analysis assumes textual feedback is provided to the planner, but does not assume a single specific method of fusing LLM planning with low-level robotic control or a specific method of extracting environment feedback into language. Rather than focusing on a particular algorithmic implementation, our aim is to provide a case study on the value of incorporating different types of feedback into closed-loop LLM-based planning. Thus, Inner Monologue in Sec 4 utilizes language feedback within separate systems that incorporate different LLMs, different methods of fusing planning with control, different environments and tasks, and different methods of acquiring control policies. We note that in our specific implementations of Inner Monologue, we use pre-trained LLMs for planning that are not finetuned, but rather evaluated solely with few-shot prompting; the full prompts can be found in the Appendix.

3 Sources of Feedback

In theory any type of environment feedback can inform the LLM planner, as long as it can be expressed through language. We focus on the specific forms of feedback shown in Fig 2, which can be broken down into task-specific feedback, such as success detection, and scene-specific feedback (either “passive” or “active”), which describes the scene. Specific instantiations and implementation details of each type of feedback can be found in Sec 4.1, Sec 4.2, and Sec 4.3 respectively for each domain.

Semantic success detection is a binary classification problem of whether the low-level skill $\pi_{k}$ has succeeded. Engineered success detectors can operate on ground-truth state in simulation, while learned success detectors can be trained on real examples of successes and failures in the real world . We use the output of success detectors in language form, which we refer to as Success feedback.

While there are many ways to describe the semantics contained within a scene, we use the term Passive Scene Description to broadly describe sources of scene feedback that are consistently provided and follow some structure. Passive Scene Description covers all sources of environment grounding feedback that are automatically provided and injected into the LLM prompt without any active prompting or querying by the LLM planner. One common type of such feedback is object recognition – we refer to the textual outputs of such object recognizers as Object feedback. We also demonstrate the use of a task-progress scene description in the simulated tabletop rearrangement environment, to which we refer as Scene feedback.

As the proactive counterpart to Passive Scene Description, Active Scene Description encompasses sources of feedback that are provided directly in response to active queries by the LLM planner. In this case, the LLM can directly ask a question about the scene, and this question can be answered either by a person, or by another pretrained model, such as a Visual Question Answering (VQA) model . While the previous types of feedback are strictly structured and narrow in their scope, in the Active Scene Description setting the LLM can receive unstructured answers to open-ended questions, allowing it to actively gather information relevant to the scene, the task, or even preferences of the user (in the case of human-provided response). The combined output we send to the LLM planner includes both the LLM-generated question along with the response. As we aim to investigate whether and how a LLM planner can incorporate such feedback and wish to study both structured VQA-style human feedback as well as unstructured human preferences feedback, we only consider human-provided response in this work, which we refer to as Human feedback.

Experimental Results

In order to study how different sources of environment feedback can support a rich inner monologue that enables complex robotic control, we analyze diverse long-horizon manipulation and navigation tasks in simulation and in the real world. As Inner Monologue is not dependent on a specific LLM or a type of grounding feedback, we study different Inner Monologue implementations in three environments with different LLM planning methods and different sources of feedback from the environment. Below, we show results for a tabletop manipulation environment in sim (Sec 4.1) and real (Sec 4.2) as well as a mobile manipulation environment in real (Sec 4.3). For more details about the experiment setup and results, please refer to the Appendix.

We experiment with vision-based block manipulation tasks in a Ravens-based simulation environment to evaluate our method against several baselines and ablate across varying amounts of textual feedback. Given a number of blocks and bowls on a table, a robotic arm containing a gripper is tasked with rearranging these objects in some desired configuration, specified by natural language (e.g., “putting the blocks in the bowls with matching colors”). We evaluate each method on four seen tasks and four unseen tasks, where seen tasks may be used for training (in the case of supervised baseline) or used as few-shot prompting for LLM planner.

This instantiation of Inner Monologue uses (i) InstructGPT as the LLM for multi-step planning , (ii) scripted modules to provide language feedback in the form of object recognition (Object), success detection (Success), and task-orogress scene description (Scene), and (iii) a pre-trained language-conditioned pick-and-place primitive (similar to CLIPort and Transporter Nets ). Object feedback informs the LLM planner about the objects present in the scene, and the variant using only Object feedback is similar to the demonstrated example in in this environment. Success feedback informs the planner about success/failure of the most recent action. However, in the presence of many objects and test-time disturbances, the complex combinatorial state space requires the planner to additionally reason about the overall task progress (e.g., if the goal is to stack multiple blocks, the unfinished tower of blocks may be knocked over by the robot). Therefore, task-progress scene description (Scene) describes the semantic sub-goals inferred by the LLM towards completing the high-level instruction that are achieved by the agent so far. For the variant that uses Object + Scene feedback, due to the additional reasoning complexity, we find that adding chain-of-thought can improve the consistency between inferred goals and achieved goals.

We additionally compare to a multi-task CLIPort policy directly trained on long-horizon task instructions (i.e., without using LLM for planning). Because CLIPort is a single-step policy and does not terminate spontaneously during policy rollout, we report CLIPort evaluations with oracle termination (i.e., repeat until oracle indicates task completion) and fixed-step termination (i.e., repeat for $k$ steps). While Inner Monologue terminates when the LLM stops generating new steps, we similarly set the maximum number of steps to be $k$ for practical considerations. We use $k=15$. To simulate real-world disturbances and evaluate the system’s robustness to disturbances, we add Gaussian noise to multiple levels of the system at test time: $\mathcal{N}(0,3)$ for pixel observation, $\mathcal{N}(0,2.5)$ for policy primitive (i.e., pick-place pixel heatmaps), $\mathcal{N}(0,0.02m)$ for place locations.

Results are shown in Table 1, and example prompt can be found in Fig 3.

As shown in Table 1, all variants of Inner Monologue perform well on seen tasks, and by leveraging rich semantic knowledge in the pre-trained LLM, the performance can be directly translated to unseen tasks without further training. Furthermore, aligned with our hypothesis, Inner Monologue with Object + Scene performs the best because of its ability to keep track of all goal conditions and currently achieved goals. Finally, we observe that non-hierarchical and solitary systems such as CLIPort (i) struggle at generalizing to unseen long-horizon tasks under test-time disturbances, and (ii) on training tasks, an oracle is also often required to indicate task completion for good performance.

2 Real-World Tabletop Rearrangement

We evaluate Inner Monologue on a real-world robot platform designed to resemble the simulation experiments presented in Sec 4.1 using motion primitives for tabletop pick and place. The setup consists of a UR5e robot arm equipped with a wrist-mounted Intel RealSense RGB-D camera overlooking a workspace of diverse objects – from toy blocks to food items to condiments (shown in Fig 3). We use an instantiation of Inner Monologue using (i) InstructGPT as the LLM for multi-step planning, (ii) a pretrained open-vocabulary object recognition with MDETR to generate a list of currently visible objects and list of previously visible objects that are no longer visible (Object), (iii) heuristics on the object bounding box predictions from MDETR for Success Detection (Success), and (iv) a zero-shot pick and place policy that uses an LLM to parse target objects from a language command (e.g., given by the planner) and then executes scripted suction-based picking and placing primitives at the center of the target objects’ bounding boxes. Aside from the pretraining of the LLM and MDETR (which are available out-of-the-box), the system does not require any model finetuning to perform pick and place tasks with new objects.

We investigate two tasks: (i) a simple 3-block stacking task where 2 blocks are already pre-stacked, and (ii) a more complex long-horizon sorting task to place food in one plate and condiments in another (where categorizing food versus condiments is autonomously done by the LLM planner). Since default pick and place performance is generally quite high on the system, we artificially inject Gaussian noise into the policy actions (i.e., add standard deviation $\sigma{=}4\text{mm}$ clipped at $2\sigma$) to stress test recovery from failures via replanning with grounded closed-loop feedback. Results are presented in Table 2. Note that the system is also subject to noisy object and success detections due to the additional challenge of real-world perception and clutter.

We compare different variants of Inner Monologue with different LLM-informed closed-loop feedback, as well as an open-loop variant that only runs object recognition once at the beginning of the task (similar to the system demonstrated in ). The partial 3-block stacking task highlights an immediate failure mode of this baseline, where the initial scene description struggles to capture a complete representation of the scene (due to clutter and occlusion) to provide as input to the multi-step planner. As a result, the system only executes one pick and place action – and cannot recover from mistakes. To address these shortcomings, Inner Monologue (Object + Success) leverages closed-loop scene description and success detection after each step, which allows it to successfully replan and recover from policy mistakes.

Additional ablations with Inner Monologue also show that (i) common failures induced by lack of closed-loop scene description are largely due to initially occluded objects not being part of the LLM generated plans, and (ii) failures induced by lack of success detection come from not retrying pick and place actions that have failed due to policy noise. Overall, we observe that both components are complementary and important in maintaining robust recovery modes for real rearrangement tasks.

3 Real-World Mobile Manipulator in a Kitchen Setting

We implement Inner Monologue in a robotic system using the kitchen environment and task definitions described in SayCan . The Everyday Robots robot, a mobile manipulator with RGB observations, is placed in an office kitchen to interact with common objects using concurrent continuous closed-loop control.

The baseline, SayCan , is a method that plans and acts in diverse real world scenarios by combining an LLM with value functions of underlying control policies. While SayCan creates plans that are grounded by the affordances of value functions, the LLM predictions in isolation are never given any closed-loop feedback.

We use an instantiation of Inner Monologue that uses (i) PALM as the LLM for multi-step planning, (ii) value functions from pre-trained control policies for affordance grounding , (iii) a learned visual classification model for success detection feedback (Success), (iv) human-provided object recognition feedback (Object), and (v) pre-trained control policies for relevant skills in the scene. We also perform a case study where we allow the agent to ask questions and source Human feedback directly; results are shown in Fig 5(a) and the Appendix.

We evaluate methods on 120 evaluations over three task families: 1) four manipulation tasks, 2) two dexterous manipulation tasks utilizing drawers, and 3) two long-horizon combined manipulation and navigation tasks. In order to better study how Inner Monologue improves reasoning in especially challenging scenarios, we consider an experiment variant where we add adversarial disturbances during control policy executions that cause skill policy rollouts to fail. These disturbances may be fairly simple and just require the policy to try again, or it may complex enough that the policy needs to replan and select a completely new skill. While these failures occur naturally even without perturbances, the adversarial disturbances creates a consistent comparison between methods that requires retrying or replanning to accomplish the original instruction.

The results of real robot experiments are shown in Table 3. Without adversarial disturbances, the baseline method SayCan performs reasonably on all three task families, yet incorporating LLM-informed feedback on skill success/failure and presence of objects allows the Inner Monologue to effectively retry or replan under natural failures, providing further improvement to the baseline. The most notable difference is in the cases with adversarial disturbances when a policy failure is forced to occur. Without any LLM-informed feedback SayCan has success rate close to 0% since it is does not have explicitly high-level retry behavior. Inner Monologue significantly outperforms SayCan because of its ability to invoke appropriate recovery modes depending on the environment feedback. In-depth analysis on the failure causes indicates that Success and Object feedback can effectively reduce LLM planning failures and thus overall failure rate, albeit at the cost of introducing new failure modes to the system.

4 Emergent Capabilities

Although LLMs can generate fluent continuation from the prompted examples, we surprisingly find that, when informed with environment feedback, Inner Monologue demonstrates many impressive reasoning and replanning behaviors beyond the examples given in the prompt. Using a pre-trained LLM as the backbone, the method also inherits many of the appealing properties from its versatility and general-purpose language understanding. In this section, we demonstrate a few of these emergent capabilities.

Although not explicitly prompted, the LLM planner can react to human interaction that changes the high-level goal mid-task. Fig 5(a) demonstrates a challenging case, where Human feedback changes the goal during the plan execution, and then changes the goal yet again by saying “finish the previous task”. We can see that the planner incorporates the feedback correctly by switching tasks twice. In another instance, despite not being explicitly prompted to terminate after a human says “please stop”, the LLM planner generalizes to this scenario and predicts a “done” action.

Instead of mindlessly following human-given instructions, Inner Monologue can also act as an interactive problem solver by proposing alternative goals to achieve when the previous goal becomes infeasible. In Fig 5(b), to solve the task “put any two blocks inside the purple bowl”, Inner Monologue first attempts an action of picking up the purple block – the action fails as the purple block is intentionally made to be too heavy for the robot. After a hint “the purple block is too heavy”, it proposes to “find a lighter block” and successfully solves the task in the end.

Pre-trained LLMs are known to be able to translate from one language to another, without any finetuning. We observe that such multilingual understanding also transfers to the embodied settings considered in this work. Specifically, in Fig 5(c), the human-provided new instruction is written in Chinese, but the LLM can correctly interpret it, re-narrate it as a concrete goal to execute in English, and accordingly replan its future actions. Occasionally, we find that this capability even extends to symbols and emojis.

We also observe that Inner Monologue demonstrates interactive understanding of the scene using the past actions and environment feedback as context. In Fig 5(d), after a task instruction has been executed, we turn to ask questions about the scene, again a structure that has not appeared in the prompt. Surprisingly, we find that it can often correctly answer these questions that require temporal and embodied reasoning.

In the main experiments of the paper, we prompted the language model following certain conventions. For instance, in the simulated tabletop domain, the convention is [Robot action, Scene, and Robot thought]. In practice, we find that the LLM planner is robust to occasionally swapping the order of feedback. In Appendix Fig 9(a), a new human instruction is injected in the middle of the plan execution, but this structure has not been seen in the example prompts. Yet the planner recognizes the change and generates a new “Robot thought: Goal state is…” statement allowing it to solve the new task.

Inherited from the LLM backbone, our approach is robust to typos in human instruction, as seen in Appendix Fig 9(b).

Despite the appealing findings about these emergent capabilities, we observe that they are of varying levels of consistency when no similar examples have been provided in the prompt, likely limited by the current capabilities of the language models. However, we believe that further investigations into these behaviors and addressing their limitations would each lead to exciting future directions.

Limitations

In Sec 4.1 and Sec 4.3, we assume access to oracle scene descriptors in the form of human observers or scripted systems to provide textual description back to the LLM planner. We study the viability of learned systems scene desription and object recognition in Appendix Table 5. As for failure modes, Inner Monologue may fail due to several sources of errors: (1) success detections, (2) LLM planning errors, and (3) control errors. False negative predictions from the success detector lead to additional retry attempts, while false positive predictions add adversarial partial observability to the environment. In some instances, we found that the LLM planners ignored the environment feedback and still proposed policy skills involving objects not present in the scene.

The performance of low-level control policies limits not only overall high-level instruction completion performance, but also limits the scope of tasks that the LLM is able to reason over: no matter how much the LLM reasoning improves, it can still be bottlenecked by what low-level control policies are able to achieve. Future work remains open on several fronts. First, with advances in image/video captioning and visual-question answering, a fully automated system of Inner Monologue can be implemented without a human in the loop as an oracle. Second, improvements can be made on how to aggregate potentially inaccurate sources of information, such as using text to describe the uncertainty of the feedback modules, or including additional feedback modules for safety and ethics for the proposed plans.

Conclusion

In this work, we investigated the role that environment feedback plays for LLMs reasoning in tasks involving embodied robotic planning and interaction. We presented a general formulation Inner Monologue that combines different sources of environment feedback with methods fusing LLM planning with robotic control policies. Then, we studied how these methods scaled to three different robotic manipulation settings in simulation and the real world. We found that environment feedback significantly improves high-level instruction completion, especially in challenging scenarios with adversarial disturbances. Finally, we analyze emergent capabilities of Inner Monologue that highlight how closed-loop language feedback enables replanning even in complex unseen settings.

The authors would like to thank Kanishka Rao and Vincent Vanhoucke for valuable feedback and discussions. In addition, the authors would like to acknowledge the large team who built , upon which we construct our Kitchen Mobile Manipulation experiments.

References

Appendix A Inner Monologue Implementation Details

We study three different implementations of Inner Monologue for each of the experimental settings. While each version incorporates textual environment feedback into planning, there are differences in the internal components of each system, as seen in Table 4.

We use InstructGPT , a 1.3B parameter language model fine-tuned from GPT-3 with human feedback, accessed through OpenAI API.

We use a CLIP-based Transporter Net (inspired by CLIPort ) as the pick-and-place primitive that can be invoked by the LLM planner. The policy is trained on $20000$ pre-collected demonstrations, where each demonstration contains 1) language instruction of the format “pick up [x] and place it on [y]”, 2) top-down view of RGB-D observation of the current environment, 3) expert pick coordinates, and 4) expert place coordinates. The expert pick and place coordinates are obtained by accessing ground-truth object pose in the simulator. Unlike the evaluated settings in , the demonstrations cover all objects because the focus of this work is not on generalization across object instances but on novel long-horizon behaviors. The policy outputs a 3D pick location and a 3D place location, and a scripted pick-and-place motion is executed following the predicted locations.

We provide the list of objects present in the scene at the start of each episode for the language model (without bounding boxes or coordinates because spatial manipulation is handled by low-level policies). Although it has been shown in that this can be obtained by off-the-shelf pre-trained open-vocabulary object detection models such as ViLD and MDETR , we obtain the list of objects by accessing simulator state for simplicity.

For Object + Scene method, we provide task-progress scene description as a list of achieved sub-goals after each pick-and-place execution. At the start of the action plan, the language model first generates a list of desired sub-goals given the high-level instruction. The sub-goals are of the format “[x] is on top of [y]”. After each pick-and-place execution, we iterate through all desired sub-goals inferred by the language model, and check which are satisfied using the success detector described above. The full list of currently satisfied sub-goals are appended to the language model prompt. Additionally, we allow the language model to generate chain-of-thought summarization following the achieved sub-goals (i.e., “Robot thought: …”), which we find to be useful empirically.

A.2 Inner Monologue for Real-World Tabletop Rearrangement

We use InstructGPT , a 1.3B parameter language model fine-tuned from GPT-3 with human feedback, accessed through OpenAI API.

We use a single low-level policy for the real tabletop rearrangement environment that is responsible for performing object-centric pick and place actions as instructed by the language model. The policy takes as input 1) the bounding boxes of all the objects in the scene, 2) the names of the object to pick and the target object for place, 3) the depth image corresponding to the RGB image from which object detection was done, 4) camera intrinsics, and 5) camera pose. It then outputs a 3D pick position and a 3D place location for the robot to follow.

The pick position is the 3D coordinate of the center of the object bounding box in the robot base frame. This is obtained by first deprojecting the depth image pixel corresponding to the center of the pick object bounding box with the given camera intrinsics, then transforming this point from the camera frame to the robot frame with the given camera pose. Camera pose is known because it is a wrist camera, and we register the delta pose from the mounted camera to the robot end-effector.

For the block stacking task, the place location is obtained the same way as the pick position. For the object sorting task, the place location is chosen to be a point in the target plate’s bounding box that is farthest way from the bounding boxes of other objects. This is done to prevent undesired stacking behavior when placing objects into the plate, which may cause the object to roll off and fall off the table.

Object detection is done by MDETR , an open-vocabulary object detection model. It takes as input an RGB image with an image caption, and the model tries to identify the objects described in the caption. To form the caption for our tasks, we join the list of all possible object names together, separated by periods. Note that not all provided objects will be detected due to occlusions. MDETR detections are not always robust or consistent. For example, the detected object bounding box may some times be bigger or smaller than the object, or its center may be far away from the object center. Its performance also drops when objects are in close clutter. These issues with perception further motivate the need for success detection and multi-step environment feedback, so the high-level language model planner can be robust to these inaccuracies.

For the block stacking task, the scene description contains a list of currently visible objects and a list of previously visible objects that are no longer visible. We may add new objects to the currently visible objects list if, after a robot action, MDETR detects an object that we have not seen before. This happens for the block stacking task as there is an initial partially constructed tower of two blocks, the bottom of which is initially occluded.

For the object sorting task, the scene description contains a list of currently visible objects and a list of objects that the robot has successfully moved into a plate. Like the block stacking task, some objects are initially occluded, so later actions may reveal them and they will be added to the visible objects list. The successfully moved object list is needed so that the planner does not stop before task completion and that it stops on task completion.

A.3 Inner Monologue for Real-World Mobile Manipulation in a Kitchen Setting

We use PALM , a 540B parameter language model trained on a large datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code.

Following the implementation described in SayCan , we use a combination of learned and scripted control policies for navigation and manipulation. The learned manipulation policies responsible for counter picking, drawer opening and closing, drawer picking, and countertop object manipulation are Behavior Cloning (BC) policies trained on $68000$ teleoperated demonstrations and $12000$ autonomous successes that were collected over the course of 11 months using a fleet of 10 robots. The teleoperators use VR headset controllers to track the motion of their hand, which is then mapped onto the robot’s end-effector pose. The scripted navigation policies utilize a ground-truth map along with a learned perception module to navigate between different points in the environment. The scripted manipulation policy is solely responsible for countertop placing when preceded by a navigation policy, and follows pre-computed motions. The Value Functions used by SayCan for affordance grounding are provided by the $Q$-networks of trained RL agents; we follow the RL training setup described in .

We use human-provided object recognition to provide feedback about the presence of objects visible to the robot camera. For example, if there were only a coke can and an apple on top of the kitchen counter, then the human-provided object recognition feedback would appear as “[scene: coke can, apple]”.

While we utilize human-provided object recognition to gauge the role of maximally accurate Object feedback, we also study the feasibility of using learned object recognition models to provide fully automated Object feedback. We find that two such models, ViLD and MDETR , perform worse than humans but still quite resonably at providing Object feedback, even without domain-specific data for fine-tuning. As advances in computer vision improve object detection models that can transfer zero-shot or few-shot to novel environments like our kitchen environment, we expect that fully automated Object feedback will increasingly approach human-provided accuracy.

We trained a “foresight” success detector on offline data collected from a combination of teleoperated demonstrations and autonomous roll-outs performing the low-level tasks. The input to the model consists of: (1) $o_{0}$, the initial image observation, (2) $o_{f}$, the final image observation after the policy chose to terminate its chosen skill execution, and (3) $l_{k}$, the low-level skill which the agent was trying to execute (e.g., “Pick coke can”). The model uses image encoders from CLIP to embed $o_{0}$ and $o_{f}$, concatenate them, and fuse these representations with a fusion MLP. This image embedding is concatenated with the text embedding obtained by the CLIP text encoder, then passed through another MLP. The output of the model is a scalar denoting the probability of the agent succeeding at the specified task (Fig 7(a)). The model is trained with the binary cross entropy loss with respect to the ground truth binary label. We fine tune the CLIP model as well during the training. At inference time within Inner Monologue, we output the text “[success: no]” when the probability is below a certain threshold.

To reduce false positive predictions, we trained a separate “hindsight” success predictor model. Given the first and last observation, the model outputs a probability distribution over all the possible skills. We used this model to disambiguate between possible tasks that the agent may have achieved. To train this model, we use the symmetric contrastive loss as used in CLIP (Fig 7(b)). At inference time, similar to the CLIP model, we take a dot product between the image-fusion embedding and the text embeddings and apply softmax with a learned temperature.

We combine the foresight and hindsight model by first thresholding the probability from the foresight model by some value $\tau$. If the probability is above, then we run the hindsight model and only predict success if the argmax across all skills is indeed the foresight task.

We perform a case study where we allow the LLM agent to ask questions and source Human feedback directly. After each LLM planning step, the LLM chooses between and continue or and ask: for the and continue option the LLM proceeds to the planning the next action to execute, and for the and ask option the LLM proceeds to propose a question to ask the human. The questions proposed by the LLM do not follow any pre-determined format or scope, so human-provided answers are similarly open-ended. These answers can range from confirming factual observations about the scene to providing refinements or changes to the high-level plan. Some examples can be found in Sec D.

Appendix B Experiment Details

We additionally train a multi-task CLIPort policy on $4$ of the total $8$ tasks as an additional baseline (see Table 1 for train/test split). The goal is to evaluate whether solitary language-conditioned policies can perform well on long-horizon tasks and generalize to new task instructions. Note that this is a different policy as the low-level primitive used in Inner Monologue, though sharing the same architecture. The policy also shares the same training details as the low-level policy used in Inner Monologue but is trained on $20000$ demonstrations across $4$ training tasks.

B.2 Real Tabletop Rearrangement

There are two tasks in the real tabletop rearrangement domain – a block stacking task and an object sorting task.

For the object sorting task, the initial state consists of three fruits, three bottles, and three plates. The fruits and bottles are placed in a clutter in one of the three plates, and the robot is asked to sort them by putting fruits on one of the remaining plates, and bottles on the other. Due to occlusions and reduced object detection performance in clutter, some objects may not be detected in the initial clutter, so scene description feedback is still needed. The three bottles used are a ketchup bottle, a mustard bottle, and a soft drink bottle. The three fruits used are an apple, a strawberry, and a plum. Note these are toy objects made out of plastic, so they are smaller and lighter than their real-world counterparts, and they are also rigid, allowing for robust suction grasps. They have realistic visual appearances, which allow MDETR detections.

B.3 Real Kitchen Mobile Manipulation

We follow the environment and open-ended task setup described in SayCan . An Everyday Robots robot, a mobile manipulator robot with RGB observations, is placed in a kitchen with multiple named locations that contain a set of 15 household items. Given an unstructured human instruction, the robot must complete the instruction successfully by executing a set of navigation and manipulation skills, which are vision-based concurrent continuous closed-loop policies. Fig 8 presents a high-level overview of the environment. The full list of tasks are shown in Table 6.

Appendix C Additional Results

We show additional results of emergent capability in Fig 9. Please refer to Sec 4.4 for context.

Appendix D Prompts