Visual Language Maps for Robot Navigation

I INTRODUCTION

People are excellent navigators of the physical world – due in part to their remarkable ability to build cognitive maps that form the basis of spatial memory to (i) localize landmarks at varying ontological levels, such as a book; on the shelf; in the living room, or to (ii) determine whether the layout permits navigation between two points. Classic methods for robot navigation build geometric maps for path planning and can parse goals from natural language commands , but struggle to generalize to unseen instructions. Learning methods directly optimize for navigation policies grounded in language end-to-end (commands to actions) , but require copious amounts of data.

Meanwhile, recent works show that visual-language models (VLMs) pretrained on Internet-scale data (e.g., image captions) can be used out-of-the-box to ground language to the visual observations of a navigating agent, without additional data collection or model fine-tuning. These models enable mobile robots to handle new instructions that specify unseen object goals and can be combined with exploration algorithms to search for the first instance of any object (CoW) or traverse object-centric landmarks in graphs (LM-Nav) . While promising, these methods predominantly use VLMs as critics to match image observations to object goal descriptions, but do so in ways that remain disjoint from the mapping of the environment. Without grounding language onto a spatial representation, these systems may struggle to (i) recognize correspondences that associate independent observations of the same object, to (ii) localize spatial goals e.g., “in between the sofa and the TV”, or to (iii) build persistent representations that can be shared across different embodiments, e.g., mobile robots, drones. Existing VLM-based solutions generalize to new object goals, but lose the spatial precision of classic geometric maps – is it possible to get the best of both?

In this work, we investigate the utility of a spatial visual-language map representation VLMaps, which fuses pretrained visual-language features from image observations directly with a 3D reconstruction of the physical world. VLMaps can be effectively built from video feed on robots using standard exploration algorithms. When paired with large language models (LLMs) in Socratic fashion , VLMaps can translate natural language instructions into a sequence of open-vocabulary goals, directly localized in the map. A key aspect of VLMaps is that they are spatial, which enables them to:

Localize spatial goals beyond object-centric ones, e.g., “in between the TV and sofa” or “to the right of the chair” or “kitchen area” using code-writing LLMs, expanding beyond capabilities of CoW or LM-Nav.

Generate new obstacle maps for new embodiments given natural language descriptions of landmark categories that they can or cannot traverse, e.g., “tables” are obstacles for a large mobile robot, but traversable for a drone.

Extensive experiments show that using VLMaps enables more effective long-horizon multi-object goal navigation than baseline alternatives, e.g., CoW and LM-Nav , and, in particular, excels at enabling spatial open-vocabulary navigation tasks. We also provide ablations on different ways of constructing VLMaps with different language models as well as a discussion on limitations, which point to areas for future work. Code and videos are available at https://vlmaps.github.io.

II Related Work

Semantic Mapping. The maturity of traditional SLAM techniques together with the advancements in semantic understanding capabilities of convolutional neural networks has recently spurred considerable interest around augmenting 3D maps with semantic information . The literature has focused on either densely annotating 3D volumetric maps with 2D semantic segmentation CNNs or object-oriented approaches , which build 3D maps around detected objects to enable object-level pose-graph optimization. Although progress has been made at generating more abstract maps, such as scene graphs , current approaches are limited to a predefined set of semantic classes. In contrast to this, VLMaps are open-vocabulary semantic maps that, unlike prior work, enable natural language indexing in the map.

Vision and Language Navigation. Recently, also Vision-and-Language Navigation (VLN) has received increased attention . Further work has focused on learning end-to-end policies that can follow route-based instructions on topological graphs of simulated environments . However, agents trained in this setting do not have low-level planning capabilities and rely heavily on the topological graph, limiting their real-world applicability . Moreover, despite extensions to continuous state spaces , most of these learning-based methods are data-intensive.

Zero-shot Models. The recent success of large pretrained vision and language models has spurred a flurry of interest in applying their zero-sot capabilities to different domains including object detection and segmentation , robot manipulation , and navigation . Most related to our work is the approach denoted LM-Nav , which combines three pre-trained models to navigate via a topological graph in the real world. CoW performs zero-shot language-based object navigation by combining CLIP-based saliency maps and traditional exploration methods. However, both LM-Nav and CoW are limited to navigating to object landmarks and are less capable to understand finer-grained queries, such as “to the left of the chair” and “in between the TV and the sofa”. In contrast, our method enables spatial language indexing beyond object-centric goals and can generate open-vocabulary obstacle maps. A concurrent work is NLMap , which demonstrates that VLMs can be used to build queryable scene representations to allow LLM robot planning with new objects and locations.

III Method

Our goal is to build a spatial visual-language map representation, in which landmarks (“the sofa”) or spatial references (“between the sofa and the TV”) can be directly localized using natural language. We propose VLMaps as one such representation, which can be constructed using off-the-shelf visual-language models (VLMs) and standard 3D reconstruction libraries. In the following subsections, we describe (i) how to build a VLMap (Sec. III-A), (ii) how to use these maps to localize open-vocabulary landmarks (Sec. III-B), (iii) how to build open-vocabulary obstacle maps from a list of obstacle categories for different robot embodiments (Sec. III-C), and (iv) how VLMaps can be used together with large language models (LLMs) for zero-shot spatial goal navigation on real robots from natural language commands (Sec. III-D), without additional data collection or model fine-tuning. Our pipeline is visualized in Fig. 3.

The key idea behind VLMaps is to fuse pretrained visual-language features with a 3D reconstruction. We achieve this by computing dense pixel-level embeddings from an existing visual-language model (over the video feed of the robot) and by back-projecting them onto the 3D surface of the environment (captured from depth data used for reconstruction with visual odometry).

In our work, we utilize LSeg as the visual-language model, a language-driven semantic segmentation model that segments the RGB images based on a set of free-form language categories. The LSeg visual encoder maps an image such that the embedding of each pixel lies in the CLIP feature space. In our approach, wie fuse the LSeg pixel embeddings with their corresponding 3D map locations. In this way, without explicit manual segmentation labels, we incorporate a powerful language-driven semantic prior that inherits the generalization capabilities of VLMs. The only assumption we make is access to odometry, which is readily available from RGB-D SLAM systems and enables us to build a map from sequences of RGB-D images,

where $p^{x}_{map}$ and $p^{y}_{map}$ represent the coordinates of the projected point in the map $\mathcal{M}$.

III-B Localizing Open-Vocabulary Landmarks

III-C Generating Open-Vocabulary Obstacle Maps

Building a VLMap enables us to generate obstacle maps that inherit the open-vocabulary nature of the VLMs used (LSeg and CLIP). Specifically, given a list of obstacle categories described with natural language, we can localize those obstacles at runtime to generate a binary map for collision avoidance and/or shortest path planning. A prominent use case for this is sharing a VLMap of the same environment between different robots with different embodiments (i.e., cross-embodiment problem ), which may be useful for multi-agent coordination . For example, a large mobile robot may need to navigate around a table (or other large furniture), while a drone can directly fly over it. By simply providing two different lists of obstacle categories – one for the large mobile robot (that contains “table”), and another for the drone (that does not), we can generate two distinct obstacles maps for the two robots to use respectively, sourced on-the-fly from the same VLMap.

To do so, we first extract an obstacle map $\mathcal{O}\in\{0,1\}^{\bar{H}\times\bar{W}}$ where each projected position of the depth point cloud in the top-down map is assigned 1, and otherwise 0. To avoid points from the floor or the ceiling, points $P_{W}$ are filtered out depending on their height,

III-D Zero-Shot Spatial Goal Navigation from Language

In this section, we describe our approach to long-horizon (spatial) goal navigation, given a set of landmark descriptions specified by natural language instructions such as

move first to the left side of the counter, then move between the sink and the oven, then move back and forth to the sofa and the table twice Notably different from prior work , VLMaps allow us to reference precise spatial goals such as: “in between the sofa at the TV” or “three meters to the east of the chair.” Specifically, we use a large language model (LLM) to interpret the input natural language commands and break them down into subgoals . In contrast to prior work, which may reference these subgoals with language and map to low-level policies with semantic translation or affordances , we leverage the code-writing capabilities of LLMs to generate executable Python robot code that can (i) make precise calls to parameterized navigation primitives, and (ii) perform arithmetic when needed. The generated code can directly be executed on the robot with the built-in Python exec function.

Note that recent works have shown that code-writing language models (e.g., Codex ) trained on billions of lines of code from Github can be used to synthesize new simple Python programs from docstrings. In this work, we re-purpose these models for mobile robot planning, by priming them with several input examples of natural language commands (formatted as comments) paired with corresponding robot code (via few-shot prompting). The robot code can express functions or logic structures (if-then-else statements or for/while loops) and parameterize API calls (e.g., robot.move_to(target_name) or robot.turn(degrees). The full list is available in the Appendix, Sec. -A) that map to spatial behaviors specified by the language commands. At test time, the models can subsequently take in new commands and autonomously re-compose API calls to generate new robot code respectively (prompt in gray, input task commands in green, and generated outputs are highlighted):

# move a bit to the right of the fridge robot.move_to_right(‘refrigerator’) # face the toilet robot.face(‘toilet’) # move to the west of the chair robot.move_west(‘chair’) # turn right 20 degrees robot.turn(20) # move back and forth to the chair and table 3 times pos1 = robot.get_pos(‘chair’) ... # move forward for 3 meters robot.move_forward(3) # move first to the left side of the counter, then move between the sink and the oven, then move back and forth to the sofa and the table twice robot.move_to_left(‘counter’) robot.move_in_between(‘sink’, ‘oven’) pos1 = robot.get_pos(‘sofa’) pos2 = robot.get_pos(‘table’) for i in range(2): robot.move_to(pos1) robot.move_to(pos2) # move 2 meters north of the laptop, then move 3 meters rightward robot.move_north(‘laptop’) robot.face(‘laptop’) robot.turn(180) robot.move_forward(2) robot.turn(90) robot.move_forward(3) The code-writing LLM generates code that not only references the new landmarks mentioned in the language commands (as comments), but also can chain together new sequences of API calls to follow unseen instructions accordingly. The prompt has been truncated for brevity here. Please see the full prompt in the Appendix (Sec. -B).

The navigation primitive functions being called by the language model (e.g., robot.move_to_left(‘counter’)) use a pre-generated VLMap to localize the coordinates of the open-vocabulary landmarks (“counter”) in the maps (described in Sec. III-B) modified with predefined scripted offsets (to define “left”). We then navigate to these coordinates using an off-the-shelf navigation stack that takes as input the embodiment-specific obstacle map (generated using the same VLMap, with the process described in Sec. III-C).

IV Experiments

The goals of our experiments are four-fold: (i) to quantitatively evaluate our VLMaps approach against recent open-vocabulary navigation baselines on the standard task of multi-object goal navigation (Sec. IV-B), (ii) to investigate whether our method can better navigate to spatial goals specified by language commands versus alternative approaches (Sec. IV-C), (iii) to study whether VLMaps with their capacity to specify open-vocabulary obstacle maps can provide utility in improving the navigation efficiency of different robots with different embodiments (Sec. IV-D), and (iv) to demonstrate on real robots that VLMaps can enable zero-shot spatial goal navigation given unseen language instructions (Sec. IV-E).

Experimental setup. We use the Habitat simulator with the Matterport3D dataset for the evaluation of multi-object and spatial goal navigation tasks. The dataset contains a large set of realistic indoor scenes that help evaluate the generalization capabilities of navigating agents. To evaluate the creation of open-vocabulary multi-embodiment obstacle maps, we adopt the AI2THOR simulator due to its support of multiple agent types, such as LoCoBot and drone. In these two environments, the robot is required to navigate in a continuous environment with actions: move forward 0.05 meters, turn left 1 degree, turn right 1 degree and stop. For map creation in Habitat, we collect 12,096 RGB-D frames across ten different scenes and record the camera pose of each frame. Similarly, we collect 1,826 RGB-D frames across ten rooms in AI2THOR.

Baselines. We evaluate VLMaps against three baseline methods, all of which utilize visual-language models and are capable of zero-shot language-based navigation:

LM-Nav creates a graph where image observations of an environment are stored as nodes while the proximity between images are represented as edges. By combining GPT-3 and CLIP, it parses language instructions into a list of landmarks and plans on the graph towards corresponding nodes.

CLIP on Wheels (CoW) achieves language-based object navigation by building a saliency map for the target category with CLIP and GradCAM . By thresholding the saliency values, it retrieves a segmentation mask for the target object category and then plans the path on the map.

CLIP-features-based map (CLIP Map) is an ablative baseline that generates a feature map for the environment in a similar way as ours. Instead of using LSeg visual features, it projects the CLIP visual features onto the map averaged across views. Object category masks are generated by thresholding the similarity between map features and the object category features.

For additional context and analysis, we also report results from a system that has access to a ground truth semantic map for navigation, to provide a systems-level upper bound on performance.

IV-B Multi-Object Navigation

We collect 91 sequences of tasks for the evaluation of object navigation. In each sequence, we randomly specify a starting position of the robot in one scene and then pick four among 30 object categories as subgoal object types. The robot is required to navigate to these four subgoals sequentially. In each sequence of subgoals, when the robot reaches one subgoal category, it should call the stop action to indicate its progress. We consider the navigation to one subgoal as success when the distance of stop position from the correct object is within one meter. To evaluate the long-horizon navigation capabilities of the agents, we compute the success rate (SR) of continuously reaching one to four subgoals in a sequence, shown in Tab. I. We also report the independent subgoal success rate, which indicates the total successful subgoals number divided by the total subgoals number (364 subgoals).

We observe that VLMaps performs consistently better compared to all baselines. LM-Nav has a weak performance as it is only able to navigate to locations represented by images stored in graph nodes. To obtain more insights into the map-based methods, we visualize the object masks generated by VLMaps, CoW, and CLIP Map, in comparison to GT, in Fig. 4. The masks generated by CoW (Fig. 4(d)) and CLIP (Fig. 4(c)) both contain considerable false positive predictions. Since the planning generates the path to the nearest masked target area, these predictions lead to planning towards wrong goals. In contrast, the predictions obtained with VLMaps shown in Fig. 4(e) are less noisy, which leads to higher success rates in object navigation.

IV-C Zero-Shot Spatial Goal Navigation from Language

In these experiments, we investigate the performance of VLMaps versus other baselines for zero-shot spatial goal navigation from language. Our benchmark consists of 21 trajectories in seven scenes, with manually specified corresponding language instructions for evaluation. Each trajectory contains four different spatial locations as subgoals. Examples of subgoals are “east of the table”, “in between the chair and the sofa”, or “move forward 3 meters”. There are also instructions for the robot to realign itself in reference to nearby objects such as “with the counter on your right”. We only consider a subgoal as having been achieved, when the robot reaches the subgoal location within a range of one meter. We compute the in-a-row success rate in the same way as in Sec. IV-B. For all map-based methods, including CoW, CLIP Map, ground truth semantic map and our method, we apply the code generation techniques introduced in Sec. III-D. For LM-Nav, we simply use the same parsing method in the original paper to break down the language instruction into subgoals.

Tab. II summarizes the zero-shot spatial goal navigation success rates. Our method outperforms other baselines in this task. Different from object navigation tasks where agents only need to approach a certain object type within a range disregarding the relative spatial shift to the object, the language-based spatial goal navigation tasks require the robot to accurately arrive at the described location in reference to the object. This poses a bigger challenge to the landmark localization ability of the method. The low localization ability of CoW and CLIP Map analyzed in the previous section (Sec. IV-B) leads to their high failure rates in this task.

IV-D Cross-Embodiment Navigation

We study the ability of VLMaps to improve navigation efficiency by retrieving different obstacle maps for navigation with different embodiments (given the same VLMap). We evaluate more than 100 sequences of subgoals as in Sec. IV-B in the AI2THOR simulator. We evaluate VLMaps on both a LoCoBot and a drone to test its capability of generating obstacle maps at runtime for multi-embodiment navigation. We apply the open-vocabulary obstacle map generation method in Sec. III-C to create an obstacle map for the drone (drone map) and one for the LoCoBot (ground map) by defining obstacles for them differently (see the prompts in Appendix Sec. -E). We test the navigation ability of these embodiments with three setups: a LoCoBot with a ground map, a drone with a ground map, and a drone with a drone map.

We evaluate the Success Rate (SR) and the Success rate weighted by the (normalized inverse) Path Length (SPL) defined as: $SPL=\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{\textit{l}_{i}}{max(\textit{p}_{i},\textit{l}_{i})}$ where $N$ is the total number of evaluated tasks, $S_{i}\in\{0,1\}$ is the binary indicator of success, $\textit{l}_{i}$ denotes the ground truth shortest path length, and $\textit{p}_{i}$ denotes the actual path length of the agent in navigation. This metric indicates how efficient the actual path is compared to the ground truth shortest path when the navigation task is achieved. In our three setups, the ground truth trajectories for the LoCoBot and the drone are planned on floor-level and on height level of 1.7 meters respectively.

The results provided in Tab. III show that the average navigation success rates of the ground-map version of the LoCoBot and the drone are similar because the same obstacles map is used for planning. However, there is an obvious gap between their SPL values. This is because when the drone does not have access to a customized obstacle map, it fails to benefit from flying over ground objects to improve the navigation efficiency. In contrast, while achieving similar success rate compared to the drone with a ground map, the drone with a drone map manages to navigate with higher path efficiency, reflected by the increased SPL values. The comparable SPL values for the drone with the drone map and the LoCoBot with the ground map shows that VLMaps help to generalize the navigation efficiency among different embodiments. An example of the multi-embodiment object navigation task is shown in Fig. 5, where by defining a more efficient obstacles map, the drone flies over the sofa and reaches the laptop target directly, while the LoCoBot has to move aside first to avoid colliding with the sofa.

IV-E Real Robot Experiments

We also perform real-world experiments using the HSR mobile robot for indoor navigation given natural language commands. For map creation, we record 374 frames for the evaluated scene and use an off-the-shelf RGB-D SLAM solution, RTAB-Map to estimate the camera poses. During inference, we also use the global localization module of RTAB-Map to initialize the robot pose. We test our VLMaps in a semantically rich indoor scene with more than ten different classes of objects. We define 20 different language-based spatial goals for testing purposes. Across different test runs, we initialize the robot at different locations.

The robot finishes ten navigation goals out of the 20. Among the successful trials, six of them are spatial goals like “move between the chair and the wooden box” or “move to the south of the table”. three of them are goals relative to the current position of the robot like “move 3 meters right and then move 2 meters left”. Another one is an instruction with repetition: “move between the keyboard and the laptop twice”. We observe that failure cases are caused by: 1) inaccurate depth, which introduces noise during the map creation and decreases the landmark indexing accuracy and 2) action noise, which can negatively influence the navigation performance at test time. Overall, these results demonstrate the ability of VLMaps to index landmarks with natural language in the real world and, more importantly, its applicability to achieve a wide variety of open-vocabulary language-based spatial navigation goals.

V Discussion and Limitations

In this work, we propose VLMaps, a spatial map representation enriched with pretrained visual-language features, which enables natural language indexing in the map. When combined with large language models, VLMaps can be applied in zero-shot spatial goal navigation and can be shared among multiple robots with different embodiments to generate new obstacles map in runtime. VLMaps are not without limitations. Notably, they remain sensitive to 3D reconstruction noise and odometry drift during navigation. They also cannot resolve object ambiguities during landmark indexing when the scene is cluttered with similar objects. In future work, we plan to improve VLMaps with better visual language models and to extend it to scenes with dynamic objects and moving humans.

References

-A Full List of Navigation Primitives

Our full list of navigation primitives are listed in Table IV.

-B Full Prompts

Our full prompts used for getting the navigation results are listed below.

-C Prompt engineering.

For all methods in this work (including baselines), when using CLIP text encoding, instead of simply prompting the label of the object categories, we use the ensemble of prompt templates like “A photo of label”, “A picture of label” mentioned in to improve the retrieval performance.

-D Top-Down Map Semantic Segmentation

For ablation purposes, we compute the semantic segmentation masks for the top-down maps in the Habitat simulator with the Matterport3D dataset. We use the collected RGB-D frames mentioned in Sec. IV-A to create the VLMaps and the CLIP on Wheels saliency maps. We evaluate all the semantic categories (the full list can be found in the linkhttps://github.com/niessner/Matterport/blob/master/metadata/mpcat40.tsv) supported in the Matterport3D dataset except “void”, “floor”, “ceiling”, “objects”, “misc”. To get the ground truth semantic masks, we use the RGB-D frames and the ground truth image semantic masks to create a semantic top-down map. We back-project the depth pixels to the 3D space and project them to the top-down map. We assign the associated semantic values to the top-down map pixels. If multiple points are projected to the same location, we overwrite the old value if the new point’s height is larger than the previous points. To compute semantic masks for VLMaps, we apply the open-vocabulary landmark indexing technique described in Sec. III-B to the whole list of categories. To compute semantic masks for the CLIP on Wheels, we compute the saliency values and apply the same thresholding process as in to get a binary mask for each category. We evaluate the semantic segmentation metrics used in . The segmentation results is shown in Table V.

We also show the IOU values of the top-10 frequent categories in Table VI. The table shows that VLMaps performs better than CoW Map in most of the top-10 frequent categories. This is mainly because the GradCam used in CoW introduces a lot of noise in the saliency map, causing over-segmentation in the results. We also note that in the class “seating”, VLMaps gets 0 IOU score. Since the LSeg model we used is pre-trained on segmentation datasets where some query classes might not be in the pre-defined training categories, LSeg’s visual encoder will encode visually unseen objects (“seating”) to a similar seen object’s embedding space (like “chair” or “sofa” here). As a result, visual-text misalignment could happen.

We visualize qualitative segmentation results in Figure 7. We observe that for categories “wall”, “chair”, “counter”, “table”, and “bed”, the segmentation results are mostly correct. Sometimes, when the “sofa” and “chair” are in similar material and shape (in Figure 7(a) and 7(b)), VLMaps might fail to differentiate them, leading to wrong planning behaviors. We also observe from Figure 7(c) and Figure 7(e) that the segmentation of some objects are noisy. This could be caused by the features fusion strategy we adopt. For example, in the top left corner of Figure 7(e), there are some chairs and tables predictions with noise compared to the ground truth in Figure 7(f). When we generate VLMaps for the scenes, we average the visual embeddings of points projecting to the same location on the top-down map. The averaging operation might introduce noise in the fused features, leading to noisy segmentation predictions (predicting “sink” on the table). In the future, more advanced fusion techniques can be explored to improve the segmentation results.

-E Prompts for Obstacle Maps Generation

In Sec. IV-D, we generate open-vocabulary obstacle maps for a drone and a LoCoBot with the method introduced in Sec. III-C. For the LoCoBot (ground robot), we first define a potential obstacle list as [“chair”, “wall”, “wall above the door”, “table”, “window”, “floor”, “stairs”, “other”] and perform open-vocabulary landmark indexing. Later, we only select the union of the masks for the objects “wall”, “chair”, “table”, “window”, “stairs”, “other” as the obstacle map. For the drone (flying robot), we perform landmark indexing with the potential obstacle list: [“chair”, “sofa”, “wall”, “table”, “counter”, “window”, “floor”, “stairs”, “ceiling lights”, “cabinet”, “counter support”, “other”]. Afterwards, we take union of the masks for [“wall”, “window”, “stairs”, “ceiling lights”, “cabinet”, “other”] to generate the obstacle map.