Interactive Language: Talking to Robots in Real Time

I Introduction

The goal of building a robot that can follow a diverse array of natural language instructions has been a longstanding goal of AI research, since at least the SHRDLU experiments starting in the late 1960s. While recent research on this topic has been abundant , few efforts have actually produced a robot that (i) exists in the real world, and (ii) can capably respond to a large number of rich, diverse language commands. We expect that future research will continue to produce larger and more diverse sets of behaviors, either by sequencing raw skills together or growing the number of raw skills themselves . However, we are also interested in (iii), the capacity to follow interactive language commands, by which we mean that the robot reacts capably and in-the-moment to new natural language instructions provided during ongoing task execution.

Although we might expect such a robot to be possible given current methods, natural language-interactable robots are frequently slow in practice, and often use blocking parameterized skills or simplifying self-resetting behaviors that prohibit this kind of live, real-time interaction.

In this paper, we demonstrate a framework for producing real-world, real-time-interactable, natural-language-instructable robots (Fig. 1, a) that by certain metrics operate at an order of magnitude larger scale than prior works. To accelerate further research in this setting, we accordingly provide our associated recipe, dataset, models, hardware environment description, simulated analogue environment, and a research benchmark for language conditioned manipulation (Fig. 1, c). In terms of scale, the produced robot policies can address 87,000 unique commands at an estimated 93.5% success rate (Fig. 1, b), with continuous 5Hz visuolinguomotor control, and are capable of chaining raw skills to reach hundreds of thousands of long horizon goals in its environment. This robot exists in an environment which we designed to provide a tractable yet difficult level of challenge (perception from pixels, feedback-rich control, multiple objects, ambiguous natural language instructions). We cast real time language guidance as a large scale imitation learning problem (Figure 2). The learning algorithm recipe itself is intentionally simple, and instead the complexity of this effort was primarily in the data effort itself, for which we detail insights and techniques. We hope the dataset and benchmark may catalyze further work which may improve on our demonstrated sample complexity and performance.

Beyond demonstrating diverse short-horizon skills, we also use these capabilities to study the nonobvious benefits of a real-time language robot. For one, we show that through occasional human natural-language feedback, the robot can accomplish complex long-horizon rearrangements such as “put the blocks into a smiley face with green eyes” that require multiple minutes of precise coordinated control (Figure 5, left). We also find that real-time language competency unlocks new capabilities like simultaneous, multi-robot instruction – in which a single human can guide multiple real-time robots through long-horizon tasks (Figure 5, right).

Contributions. Our primary contributions include (i) Interactive Language, a framework for producing real world robots that can capably receive interactive open vocabulary language conditioning in real-timeFor the scope of this paper, by real-time we mean new language conditioning can occur in the “blink of an eye”, i.e. approximately 3 Hz or greater. while performing continuous-control visuomotor manipulation. Interactive Language combines existing techniques, together with novel components like event-selectable hindsight relabeling, to define a simple and scalable recipe for learning large repertoires of natural-language-conditionable skills. (ii) We use this system to present and study the setting of interactive language guidance, showing that the combination of real-time language feedback and a low-level language-conditionable policy can address long-horizon manipulation goal states in a tabletop rearrangement setting. (iii) To facilitate future research in this domain, we release Language-Table, a dataset and simulated multitask imitation learning benchmark. With nearly 600,000 diverse demonstrations across simulation and the real world, Language-Table is, to our knowledge, the largest natural language conditioned imitation learning dataset of its kind by an order of magnitude (Table III).

II Related work

From single-task imitation to multi-task and language conditioning. Imitation learning (see review ), the perspective we adopt in this work, provides a simple and stable way for robots to acquire behaviors from human expert demonstrations. While historically imitation learning has been applied to individual tasks from instrumented state , the desire for more general purpose robots has motivated study into policies capable of learning multiple skills at once from more generic on-board sensory observations like RGB pixels . To condition multiple learned behaviors, prior setups have relied on discrete one-hot task identifiers , which can be difficult to scale to many tasks, or goal images , which can be impractical to provide in real world scenarios. Alternatively, a long history of prior work in broader AI research has sought a more convenient form of specification in the form of natural language conditioning (survey ), with some results on physical robots . This focus has yielded many varied and impressive approaches to tackling the grounding problem—learning to relate language to one’s embodied observations and actions. However, in both simulation and the real world, instruction-following robots rarely leverage the full capabilities of continuous control, instead employing simplified, parameterized action spaces . Furthermore, once provided, language conditioning is typically presumed fixed over robot execution , with little opportunity for subsequent interaction by the instructor. Our work, in contrast, studies the first combination, to our knowledge, of real-time natural language guidance of a physical robot engaged in continuous visuomotor manipulation.

Interactively guiding robot behavior with language. Our work exists in a larger setting of humans modifying or correcting the behavior of autonomous agents , historically addressed in forms like teleoperation , kinesthetic teaching , or sparse human preference feedback . Certain works have studied language as a means of correction, but typically do so under simplifying assumptions that we relax in the current work. For example, , , , and study language corrections, but under the respective simplifying assumptions of hand-defined optimization for grounding, undivided operator attention, paired iterative corrections at training time, and presumed access to motion planners and task cost functions. Additionally, to the best of our knowledge, none of these works support multiple-Hz iterative specification over the course of execution. Closest to our approach is and , which study language-interactive agents learned via imitation, but entirely in simulation and under varying degrees of actuation realism. In contrast to these prior studies, our work learns real-time natural language policies end-to-end from RGB pixels to continuous control outputs with a simple behavioral cloning objective , and applies them to contact-rich real-world manipulation tasks.

Scaling real world imitation learning. One of the largest bottlenecks in robot imitation is often simply the amount of diverse robot data made available to learning . Many multi-task imitation learning frameworks determine the set of tasks to be learned upfront . While this may simplify collection conceptually, it also often requires that reset protocols and success criteria be designed manually for each behavior. Another challenge particular to large scale multi-operator collections is that typically not all data can be considered optimal , often requiring manual post-hoc success filtering . These per-task manual efforts have historically been difficult to scale to a large and diverse task setting, like the one studied in this work. We sidestep both these scaling concerns by instead having operators continuously teleoperate long-horizon behaviors, with no requirements on low level task segmentation or resets and then leverage after-the-fact crowdsourced language annotation . In contrast to the “random window” relabeling explored in , we give annotators precise control over the start and end of behaviors they are annotating, which we find in practice better aligns relabeled training data to the actual commands given at test time.

III Problem Setup

IV Interactive Language: Methods and Analysis

First we introduce Interactive Language, summarized in Figure 2, a simple and generically applicable imitation learning framework for training real-time natural-language-interactable robots. Interactive Language combines a scalable method for collecting varied, real world language-conditioned demonstration datasets, with straightforward language conditioned behavioral cloning (LCBC).

High throughput raw data collection. Interactive Language adopts purposefully minimal collection assumptions to maximize the flow of human demonstrated behavior to learning. Operators teleoperate a variety of long-horizon behaviors constantly, without low-level task definition, segmentation, or episodic resets. This strategy shares assumptions with “play” collection , but additionally guides collect towards temporally extended low-entropy states like lines, shapes, and complex arrangements. Each collect episode lasts $\sim$10 minutes before a break, and is guided by multiple randomly chosen long-horizon prompts $p\in\mathcal{P}$ (e.g. “make a square shape out of the blocks”), drawn from the set of target long-horizon goals, which teleoperators are free to follow or ignore. We do not assume all of the data collected for each prompt $p$ is optimal (each $p$ is discarded after collecting). In practice, our collection includes many inevitable edge cases that might otherwise require data cleaning, e.g. solving for the wrong $p$ or knocking blocks off table. We log all of these cases and incorporate them later on as training data. Concretely, this collect procedure yields a semi-structured, optimality-agnostic collection $\mathcal{D}_{\operatorname{collect}}=\{\tau_{i}\}^{\mathcal{D}_{\operatorname{collect}}}_{i=0}$. The purpose of $\mathcal{D}_{\operatorname{collect}}$ is to provide a sufficiently diverse basis for crowdsourced hindsight language relabeling , described next.

Event-selectable hindsight relabeling. We convert $\mathcal{D}_{\operatorname{collect}}$ into natural language conditioned demonstrations $\mathcal{D}_{\operatorname{training}}=\{(\tau,l)_{i}\}^{\mathcal{D}_{\operatorname{training}}}_{i=0}$, using a new variant of hindsight language relabeling we call “Event-Selectable Hindsight Relabeling” (Fig.2, left). Previous “random window” relabeling systems have at least two drawbacks: each random window is not guaranteed to contain “usefully describable” actions, and random window lengths must be determined upfront as a sensitive hyperparameter. We instead ask annotators to watch the full collect video, then find $K$ coherent behaviors ($K=24$ in our case). Annotators have the ability to mark the start and end frame of each behavior, and are asked to phrase their text descriptions as natural language commands. In Table I, we compare event-selectable relabeling to prior “random window” relabeling on a subset of our training data. We find that while both strategies tend to describe contact-rich behaviors, our analysis suggests event-selectable relabeling yields more well-matched data: fewer complex compound instructions, and more compositionally directed instructions.

Throughput and bottleneck analysis. Here, we share some insights gained from scaling our robot collect and hindsight relabeling operation. See statistics on our collected data in Table II. We find, perhaps surprisingly, that the main bottleneck in our data operation is not robot teleoperation but rather the crowdsourced language annotation that follows, with 18.06% of the raw data having undergone annotation prior to model training (5.5x as much unlabeled collected data as annotated data). This is true even though there are 16x as many hindsight annotators as robots. Bottlenecks like this may be addressed by exploiting language-free co-training , or by simply continuing to horizontally scale crowdsourced annotators.

IV-B Policy Learning

Training. We train our policies with a standard supervised language conditioned behavioral cloning (LCBC) objective. While we expect that more complex loss functions or policy classes may acquire even better results, all the policies we present were trained as deterministic policies with a simple mean squared error loss: $\underset{\theta}{\min}\sum_{(s,a,l)\sim\mathcal{D}_{\operatorname{training}}}||a-\pi_{\theta}(s,\;\;l)||_{2}^{2}$, e.g. as in .

V Language-Table: Datasets and Environment

To facilitate further research in language-conditioned visuomotor learning, we release Language-Table, which consists of (i) a suite of datasets and (ii) a simulated multi-task language conditioned control environment and benchmark.

Dataset. Language-Table provides our human-relabeled $\mathcal{D}_{\operatorname{training}}$ and the underlying human-teleoperated $\mathcal{D}_{\operatorname{collect}}$, both in simulation and the real world. The $\mathcal{D}_{\operatorname{training}}$ real and sim datasets are highlighted in Table III – an order of magnitude larger than comparable, previously-available datasets.

Environment and Benchmark. Language-Table’s simulated environment resembles our real-world tabletop manipulation scenario, which consists of an xArm6 robot, constrained to move in a 2D plane with a cylindrical end-effector as in , in front of a smooth wooden board with a fixed set of 8 plastic blocks, comprising 4 colors and 6 shapes (Fig. 5). In both simulation and real collection, we use high-rate human teleoperation with a 3rd person view (line-of-sight in real). Actions are 2D delta Cartesian setpoints, from the previous setpoint to the new one. We batch collected training and inference data to 5hz observations and actions.

The Language-Table benchmark computes automated metrics for 5 task families, with 696 unique task variations. In addition to thresholded task success, a metric we find that better correlates with human-preferred performance is Success weighted by Path Length (SPL) , which trades off success rate against the efficiency of the path it took to succeed. We note that policy hyperparameters ordered by SPL in Language-Table have thus far been ordered similarly in real-world performance. This provides a degree of validation for the simulated benchmark’s relevancy to real world robotics.

VI Policy Results and Discussion

We present experiments aimed at answering the following questions: (1) How capably can the system follow a wide variety of short-horizon natural language conditioned commands? (2) How capably can these skills be composed through interactive language guiding to accomplish a wide variety of multi-step long-horizon compositional rearrangements? (3) What is the benefit of being able to provide interactive language feedback, compared to open-loop language plans? (4) Can one operator simultaneously guide several robots equipped with our policy? (5) Ablations: How does our transformer-based policy architecture compare to an existing visuo-linguo-motor baseline? How does our presented approach scale with varying amounts of data?

Ideally, we would be able to evaluate an Interactive Language policy on any short-horizon command a real human might give it, which is intractable in general. As a surrogate, we estimate a $95\%$ confidence interval on average success over the 87,588 unique language instructions collected via crowdsourcing (20 randomly selected instructions, 10 trials each) available at time of analysis (Table II). To succeed, policies must ground object properties and compositional spatial concepts (e.g. “…top right side of the yellow hexagon” vs “top right side of the board”), and resolve difficult ambiguities (e.g. “nudge the cube left a bit”). We report results in Table IV, with examples in Figure 4. We see that Interactive Language obtains a $93.5\%$ expected average success rate over all 87,588 instructions, $95\%$ CI $[90.08\%,96.92\%]$. To our knowledge, this is the largest set of language conditioned behaviors a real-world policy has been shown to capably address, demonstrating a solid base capacity for language conditioned visuomotor control.

VI-B Real world: long-horizon real-time language guidance

Long horizon goal reaching. Next we aim to see whether humans can guide Interactive Language policies through a wide range of multi-step compositional rearrangements. We define over 100,000 language-distinct compositional goal states on our tabletop from 11 high level families (e.g. make high-level shapes, sort by color, place all blocks in specific locations, arrange into lines, etc.), then sample 20 uniformly from all 11. See Figure 5 for examples of different goal states. We evaluate each long horizon goal 3 times from randomly reset board states, yielding 60 total evaluations of a single policy. We report success rates in Table V. We see that our policy obtains an $85.0\%$ expected average success rate on this diverse set of goals, $95\%\;CI\;[69.35\%,100.00\%]$. These results are best appreciated by watching the supplementary videos. We note that reaching precise long horizon goals in the real world for even a single goal is a notoriously difficult problem for learning robots . Even though our policies do not do so fully autonomously, we believe the fact that a real robot can address such a large and varied set of goals with real-time language feedback suggests a synergistic mode of future operation (at least until large improvements are made in the fully autonomous setting): robots learn a set of general-purpose low-level skills, and humans put them together in a familiar way using natural language, interrupting at any time to offer situation-specific corrections.

Open-loop vs real-time language feedback. Next, we attempt to quantify the benefit of being able to provide real-time language feedback, over the more common “open-loop” evaluation setting where the sequence of subgoals is decided up front . We hypothesize that many of the tasks in our environment might require several rounds of iterative and interactive specification, due to the stochastic nature of single-point-of-contact pushing. We perform the same evaluation as in the previous section, but the human operator commits up front to the set and order of commands they will provide. We present results for this ablation in Table V, finding that performance deteriorates from 85% to 25% when real-time language is removed. This indicates that for contact-rich tasks like the ones studied in this work, success depends heavily on sufficient real-time feedback—not only for the low-level policy, but also for the agent providing it instructions.

Multi-robot control via spoken language. Finally, we investigate a new competency afforded by Interactive Language: simultaneous multi-robot control. In Figure 5, see video as well, we see that four robots equipped with Interactive Language policies can be guided at the same time by one operator. This language guided multi-robot control is, as far as we know, a capability not yet demonstrated in the literature. Importantly, due to short-horizon skill competency, this shows that language can relax the assumption of undivided operator attention, which is common for prior ways of correcting online robot behavior .

VI-C Simulation: Architecture and data ablation

In Figure 6, we present results in simulation ablating (i) our transformer-based policy architecture LAVA against the FiLM-conditioned ResNet architecture in and (ii) the amount of data provided to policy training. We report average success and SPL over the multi-task benchmark in Language-Table (see “Environment and Benchmark” in Section V), and all numbers are reported with confidence intervals over three seeded training runs. We see the presented architecture is indeed responsible for significant gains over prior work in SPL, a path-length-aware success metric we find correlates best with real world quality in our setup. When sweeping the amount of training data, we find that policy performance is seeing diminishing returns, but not yet plateauing across each doubling of data. While perhaps surprising given the scale of our collect, we believe that this result highlights the environment’s complexity as well as the difficulty of open vocabulary visuomotor learning.

VII Conclusion, Limitations, and Future Work

We have presented and analyzed the Interactive Language framework and we provide a number of associated assets, notably the Language-Table dataset and environment. We believe the scale of the dataset assets, the recipe used to produce them, the scale of the demonstrated policy diversity, and the exploration of new capabilities, each offer benefit to the research community in further advancing capable, realtime-conditionable visuo-linguo-motor robots. While simple and scalable, our approach does have a number of limitations. The open problems in broader human-robot collaboration are numerous , including intention detection, non-verbal communication, physically collaborative task completion, etc. Our approach addresses only the setting of real-time language-guided manipulation. Future work may investigate applying Interactive Language to important domains like real-time assistive robots, which may benefit from more capable natural language interfaces . We hope that our work can be useful as a basis for future research in capable, helpful robots with visuo-linguo-motor control.

References

Acknowledgements

We would like to thank everyone who supported this research. This includes robot teleoperators: Alex Luong, Armando Reyes, Elio Prado, Eric Tran, Gavin Gonzalez, Jodexty Therlonge, Joel Magpantay, Rochelle Dela Cruz, Samuel Wan, Sarah Nguyen, Scott Lehrer, Norine Rosales, Tran Pham, Kyle Gajadhar, Reece Mungal, and Nikauleene Andrews; robot hardware support and teleoperation coordination: Sean Snyder, Spencer Goodrich, Cameron Burns, Jorge Aldaco, Jonathan Vela; data operations and infrastructure: Muqthar Mohammad, Mitta Kumar, Arnab Bose, Wayne Gramlich; and the many who helped provide language labeling of the datasets. We would also like to thank Pierre Sermanet, Debidatta Dwibedi, Michael Ryoo, Brian Ichter and Vincent Vanhoucke for their invaluable advice and support.

Our real-world experiments use UFACTORY xArm6 robot arms with all state logged at 100 Hz. Observations are recorded from an Intel RealSense D415 camera, using RGB-only images at 640x360 resolution, logged at 30 Hz, which we resize to 320x180 before handing to robot policies. Policies use 320x180 single-camera RGB-only images, with no other observations besides language. The asynchronous observations and actions are batched to psuedo-synchronous 5 Hz pairs for training the policy, with camera latency (characterized at roughly 80 ms) accounted for when forming psuedo-synchronous training pairs. The cylindrical end-effector is made from a 6 inch long plastic PVC pipe sourced from McMaster-Carr (9173K515). The work surface is 24 x 18 inch smooth wood cutting board. The manipulated objects are from the Play22 Baby Blocks Shape Sorter toy kit (Play22). The 6DOF robot is constrained to move in a 2D plane above the table.

-B Language-Table: Datasets

Here we outline the various datasets available in Language-Table, across simulation and real.

This dataset consists of 6 teleoperators teleoperating a robot in simulation, following long horizon prompts. See representative prompts in Table VIII. 8318 episodes were collected with an average length of 36.8 $\pm$15 seconds, yielding a total of 85.5 hours of raw data.

-B2 Simulation-Relabeled

The Simulation-Raw-Collect data was sent to 64 crowdsourced annotators, who used the interface described in Appendix -E to generate 181,020 hindsight relabeled trajectories, with 78,623 unique instructions. See representative instructions in Table VII.

-B3 Real-World-Raw-Collect

This dataset consists of 11 teleoperators alternating over four robots, following long horizon prompts. See representative prompts in Table VIII. 23498 total episodes were collected with an average length of 9.9 minutes $\pm$5.6 seconds, yielding a total of 3865 hours of raw data. Note that 16417 episodes totaling 2701 hours went into the actual training of policies, and the remaining was collected after training the demonstrated policy, but before releasing the dataset.

-B4 Real-World-Relabeled

The Real-World-Raw-Collect data was sent to 64 crowdsourced annotators, who used the interface described in Appendix -E to generate 414,798 total hindsight relabeled trajectories, with 119,959 unique instructions. See representative instructions in Table VII. Note that 298,782 relabeled trajectories went into training, with 87,140 unique instructions, and the remaining was collected post-training, but pre-release.

-C Language-Table: Environment

Our simulated environment is intended to roughly match our real world setup, and consists of a simulated 6DoF robot xArm6 implemented in PyBullet equipped with a small cylindrical end effector. Third person perspective 320x180 RGB-only images from a simulated camera are used as visual input. On a board in front of the robot are 8 blocks: red crescent, red pentagon, blue crescent, blue cube, green cube, green star, yellow star, and yellow pentagon. Like in the real world, the arm is constrained to the 2D plane and the action space is the delta 2D cartesian setpoint of the end effector. We run all experiments from RGB and language input only, but the environment additionally exposes 26-dimensional state observations (2D position and 1D rotation angle for each block, 2D position of end effector). While our real world policies perform asynchronous inference and control at 5hz, the policies in Language-Table perform blocking control at 10hz. Despite this difference, and others like differences between real and simulated images, we found policy performance in Language-Table was highly correlated with policy performance in the real world.

-D Language-Table: Evaluation

We define five simulated evaluation task families (spanning 696 unique task conditions) in Language-Table, each with a hand-defined success criterion:

block2block: Push a block to another block. Success is thresholded distance between source and target block. There are 56 unique task conditions (8 source blocks x 7 target blocks).

block2abs: Push a block to an absolute location on the board: top left, top center, top right, center left, center, center right, bottom left, bottom center, bottom right. Success is thresholded distance between block and target location. There are 72 unique task conditions (8 blocks x 9 locations).

block2rel: Push a block to a relative offset location: left, right, up, down, up and left, up and right, down and left, down and right. Success is the thresholded distance between the block and the invisible target offset location. There are 64 unique task conditions (8 blocks x 8 offset directions).

block2blockrel: Push a block to a relative offset location of another block: left side, right side, top side, bottom side, top left side, top right side, bottom left side, bottom right side. Success is the thresholded distance between the source block and the invisible target offset location of the target block. There are 448 unique task conditions (8 source blocks x 7 target blocks x 8 offset directions).

separate: Separate two blocks. Success is the thresholded distance between the two blocks. There are 56 unique task conditions (8 source blocks x 7 target blocks).

These task families were used to benchmark models in simulation, allowing us to find hyperparameters that transferred well to fully-real-world training (we note that there was no sim-to-real component in the training employed by this work). The language conditioning for the automated evaluation tasks are generated synthetically from predefined synonym sets for each task condition.

-E Event selectable hindsight relabeling details

Figure 7 depicts a mockup of the interface our crowdsourced workers used to do event selectable hindsight relabeling. We asked data labelers to first watch an entire long horizon video, then produce 12 medium horizon and 12 short horizon actions, where the definition is left to rater discretion. Labelers have control of temporal segmentation tools, allowing them to mark the beginning and end of each action, and they describe each action as an open vocabulary natural language instruction.

-F Model architecture details

Here we describe LAVA (“Language Attends to Vision for Actions”), the transformer-based visuo-linguo-motor neural network architecture we use in this work. Internally, our architecture consists of a perception module, language module, vision-language fusion module, temporal fusion module, and policy output. We describe each below.

Language module. We use a pretrained CLIP text encoder , which is finetuned on our in-domain data, but remains fixed during policy training. We use a simple contrastive method for finetuning models pretrained on (image, language) pairs to domains where the observations are (video, language) pairs: generate (start frame $s_{0}$, goal frame $s_{g}$, language $l$) from all videos, and then during finetuning, pass concatenated image encodings $\operatorname{concat}([z_{0},z_{g}])$ through an MLP to get a single encoding $z^{\operatorname{im}}$ with the same dimensionality as encoded language $z^{\operatorname{lang}}$. We preprocess text by stripping punctuation and extra spaces, but apply no additional preprocessing or augmentation. Cleaned text is passed through the CLIP embedder to get a sentence embedding with dimensionality 512.

Vision-Language Fusion Module. We fuse visual and lingual information using a “Language-Attends-to-Vision” transformer block. For a single image position, this block takes as input (i) multi-scale pixel features (in our case the CNN features at zero-indexed layers 2, 3, 4 with $H,W$ sizes , , ) and (ii) a sentence embedding (in our case the 512-dimensional CLIP-encoded $l$). First, we map each layer $n$ to $[H_{n},W_{n},\operatorname{dmodel}]$ using a layer-specific MLP, then 2D position encode each feature map with 2D sinusoidal positional embeddings. We then flatten all the multi-scale features into one long visual token list (in our case with shape [1029, $\operatorname{dmodel}$]). We project language to $\operatorname{dmodel}$ using an MLP, and apply dropout to both projected image and language features. We then iteratively fuse vision and language features, handing the sentence token as query and visual tokens as keys and values to a standard pre-norm decoder-only transformer performing cross-attention, with only language on the residual path. Our vision-language transformer had 4 layers, with $\operatorname{dmodel}$=128, 2 heads, feed forward width of 128, and dropout of 0.1.

Temporal Fusion Module. The output from applying our vision-language fusion module to each image in the $seqlen=$4 context history is a $[\operatorname{seqlen},\operatorname{dmodel}]$ sequence of vision-language embeddings. We apply 1D sinusoidal positional encoding to each element of the sequence, then feed the sequence to a standard pre-norm transformer performing self-attention, also outputting $[\operatorname{seqlen},\operatorname{dmodel}]$, which we average pool over the time dimension. Our temporal transformer had 2 layers, with $\operatorname{dmodel}$=128, 2 heads, feed forward width of 128, and dropout of 0.1.

Policy Output. We hand the average-pooled $\operatorname{dmodel}$ embedding to a deep residual MLP with 2 blocks of residual width 1024. Each block has 3 MLP layers, the first two with width 256 and the final with width 1024. All MLPs have ReLU activation with normal initialization on kernel and bias. Finally we use a linear projection to the 2D action space.

-G Training details

We train our policies on a TPUv3 8x8 pod (64 TPUv3 chips) for approximately 500,000 steps or until training loss plateaus. At roughly 7.6 steps/second, policies finish training in 18 hours. All models are trained with Adam with default TensorFlow momentum parameters, learning rate 1e-3, and a batch size of 4096. Action labels are normalized using statistics collected from training to $\mathcal{N}(0,1)$.

-H Ablations

We ablate the following training details. The results are reported in Table VI.

CLIP Finetuning. We evaluate the importance of finetuning the CLIP language module on in-domain data. We use the pretrained weights for the ViT-B/32 model from to encode language instructions without finetuning the text encoder on any Language-Table data. The text encoder still remains fixed during training. Although this results in only a few percent drop in the Language-Table sim, we observe a much stronger qualitative difference in model behavior on the real robot. This difference between sim and real could be explained by the Language-Table sim evaluation using a fixed set of templated instructions, while the real world evaluation uses more diverse language from human operators.

Temporal Fusion Module. We evaluate the importance of using our transformer-based temporal fusion module. We encode each image in the $seqlen=4$ context history by stacking the images channel-wise and encoding the images using a randomly initialized ConvNet. The multi-scale visual feature descriptors from this ConvNet are still fed into the ”Language-Attends-to-Vision” transformer block, with no additional self-attention temporal fusion transformer.

Batch Size. We evaluate the effect of batch size on our model by reducing our batch size in half to 2048. We see that this results in a performance drop after 350k steps.

-I Extended Related Work

Recent work has leveraged large language models (LLMs) to generate sequences of subgoals for language conditioned policies. These can be “open-loop” , which lack an mechanism for replanning, or “closed-loop” , which generate up-to-date plans by prepending textual descriptions of the current scene to the prompting of the LLM planner. A limitation of both formulations is that when tasks that call for fine-grained spatial detail (like those examined in this work), it is difficult for LLMs to generate accurate subgoals from purely textual scene descriptions. Although visual language models (VLMs) like suggest a promising direction, matching human levels of perception and cognition to effectively guide policies towards arbitrary goals remains a difficult open challenge. Our work is complementary in that our focus is instead on obtaining a large diverse set of short-horizon behaviors, and ones that can be interactively conditioned in real time. Combining autonomous long-horizon planning together with our demonstrated recipe for short-horizon behaviors is a strong candidate for future work.