Legged Locomotion in Challenging Terrains using Egocentric Vision

Introduction

Of what use is vision during locomotion? Clearly, there is a role of vision in navigation – using maps or landmarks to find a trajectory in the 2D plane to a distant goal while avoiding obstacles. But given a local direction in which to move, it turns out that both humans and robots can do remarkably well at blind walking. Where vision becomes necessary is for locomotion in challenging terrains. In an urban environment, staircases are the most obvious example. In the outdoors, we can deal with rugged terrain such as scrambling over rocks, or stepping from stone to stone to cross a stream of water. There is a fair amount of scientific work studying this human capability and showing tight coupling of motor control with vision . In this paper, we will develop this capability for a quadrupedal walking robot equipped with egocentric depth vision. We use a reinforcement learning approach trained in simulation, which we are directly able to transfer to the real world. Figure LABEL:fig:teaser and the accompanying videos shows some examples of our robot walking guided by vision.

Humans receive an egocentric stream of vision which is used to control feet placement, typically without conscious planning. As children we acquire it through trial and error but for adults it is an automatized skill. Its unconscious execution should not take away from its remarkable sophistication. The footsteps being placed now are based on information collected some time ago. Typically, we don’t look at the ground underneath our feet, rather at the upcoming piece of ground in front of us a few steps away. A short term memory is being created which persists long enough to guide foot placement when we are actually over that piece of ground. Finally, note that we learn to walk through bouts of steps, not by executing pre-programmed gaits .

We take these observations about human walking as design principles for the visually-based walking controller for an A1 robot. The walking policy is trained by reinforcement learning with a recurrent neural network being used as a short term memory of recent egocentric views, proprioceptive states, and action history. Such a policy can maintain memory of recent visual information to retrieve characteristics of the terrain under the robot or below the rear feet, which might no longer be directly visible in the egocentric view.

In contrast, prior locomotion techniques rely on the metric elevation map of the terrain around and under the robot to plan foot steps and joint angles. The elevation map is constructed by fusing information from multiple depth images (collected over time). This fusion of depth images into a single elevation map requires the relative pose between cameras at different times. Hence, tracking is required in the real world to obtain this relative pose using visual or inertial odometry. This is challenging because of noise introduced in sensing and odometry, and hence, previous methods add different kinds of structured noise at training time to account for the noise due to pose estimation drift . The large amount of noise hinders the ability of such systems to perform reliably on gaps and stepping stones. We use vision as a first class citizen and show all the uneven terrain capabilities along with a high success rate on crossing gaps and stepping stones.

The design principle of not having pre-programmed gait priors turns out to be quite advantageous for our relatively small robot A1 standing height is $40\textrm{cm}$ as measured by us. Spot, ANYmalC both are $70\textrm{cm}$ tall reported here and here. (fig. 1). Predefined gait priors or reference motions fail to generalize to obstacles of even a reasonable height because of the relatively small size of the quadruped. The emergent behaviors for traversing complex terrains without any priors enable our robot with a hip joint height of 28cm to traverse the stairs of height upto 25cm, 89% relative to its height, which is significantly higher than any existing methods which typically rely on gait priors.

Since our robot is small and inexpensive, it has limited onboard compute and sensing. It uses a single front-facing D435 camera for exteroception. In contrast, AnymalC has four such cameras in addition to two dome lidars. Similarly, Spot has 5 depth cameras around its body. Our policy computes actions with a single feedforward pass and requires no tracking. This frees us from running optimization for MPC or localization which requires expensive hardware to run in real-time.

Overall, this use of learning “all the way” and the tight coupling of egocentric vision with motor control are the distinguishing aspects of our approach.

Method: Legged Locomotion from Egocentric Vision

Our goal is to learn a walking policy that maps proprioception and depth input to target joint angles at 50Hz. Since depth rendering slows down the simulation by an order of magnitude, directly training this system using reinforcement learning (RL) would require billions of samples to converge making this intractable with current simulations. We therefore employ a two-phase training scheme. In phase 1, we use low resolution scandots located under the robot as a proxy for depth images. Scandots refer to a set of $(x,y)$ coordinates in the robot’s frame of reference at which the height of the terrain is queried and passed as observation at each time step (fig. 2). These capture terrain geometry and are cheap to compute. In phase 2, we use depth and proprioception as input to an RNN to implicitly track the terrain under the robot and directly predict the target joint angles at 50Hz. This is supervised with actions from the phase 1 policy. Since supervised learning is orders of magnitude more sample efficient than RL, our proposed pipeline enables training the whole system on a single GPU in a few days. Once trained, our deployment policy does not construct metric elevation maps, which typically rely on metric localization, and instead directly predicts joint angles from depth and proprioception.

One potential failure mode of this two-phase training is that the scandots might contain more information than what depth can infer. To get around this, we choose scandots and camera field-of-view such that phase 2 loss is low. We formally show that this guarantees that the phase 2 policy will have close to optimal performance in Thm 2.1 below.

We instantiate our training scheme using two different architectures. The monolithic architecture is an RNN that maps from raw proprioception and vision data directly to joint angles. The RMA architecture follows , and contains an MLP base policy that takes $\bm{\gamma}_{t}$ (which encodes the local terrain geometry) along with the extrinsics vector $\mathbf{z}_{t}$ (which encodes environment parameters ), and proprioception $\mathbf{x}_{t}$ to predict the target joint angles. An estimate of $\bm{\gamma}_{t}$ is generated by an RNN that takes proprioception and vision as inputs. While the monolithic architecture is conceptually simpler, it implicitly tracks $\bm{\gamma}_{t}$ and $\mathbf{z}_{t}$ in its weights and is hard to disentagle. In contrast, the RMA architecture allows direct access to each input ($\bm{\gamma}_{t}$ or $\mathbf{z}_{t}$) through latent vectors. This allows the possibility of swapping sensors (like replacing depth by RGB) or using one stream to supervise the other while keeping the base motor policy fixed.

Given the scandots $\mathbf{m}_{t}$, proprioception $\mathbf{x}_{t}$, commanded linear and angular velocity $\mathbf{u}_{t}^{\textrm{cmd}}=\left(v_{x}^{\textrm{cmd}},\omega_{z}^{\textrm{cmd}}\right)$ we learn a policy using PPO without gait priors and with reward functions that minimize energetics to walk on a variety of terrains. Proprioception consists of joint angles, joint velocities, angular velocity, roll and pitch measured by onboard sensors in addition to the last policy actions $\mathbf{a}_{t-1}$. Let $\mathbf{o}_{t}=(\mathbf{m}_{t},\mathbf{x}_{t},\mathbf{u}_{t}^{\textrm{cmd}})$ denote the observations. The RMA policy also takes privileged information $\mathbf{e}_{t}$ as input which includes center-of-mass of robot, ground friction, and motor strength.

The scandots $\mathbf{m}_{t}$ are first compressed to $\bm{\gamma}_{t}$ and then passed with the rest of the observations to a GRU that predicts the joint angles.

Instead of using a monolithic memory based architecture for the controller, we use an MLP as the controller, pushing the burden of maintaining memory and state on the various inputs to the MLP. Concretely, we process the environment parameters ($\mathbf{e}_{t}$) with an MLP and the scandots ($\mathbf{m}_{t}$) with a GRU to get $\mathbf{z}_{t}$ and $\bm{\gamma}_{t}$ respectively which are given as input to the base feedforward policy.

Both the phase 1 architectures are trained using PPO with backpropagation through time truncated at $24$ timesteps.

We extend the reward functions proposed in to simply penalizing the energy consumption along with additional penalties to prevent damage to hardware on complex terrain (sec. B). Importantly, we do not impose any gait priors or predefined foot trajectories and let optimal gaits that are stable and natural to emerge for the task.

Similar to we generate different sets of terrain (fig. 5) of varying difficulty level. Following , we generate fractal variations over each of the terrains to get robust walking behaviour. At training time, the environments are arranged in a $6\times 10$ matrix with each row having terrain of the same type and difficulty increasing from left to right. We train with a curriculum over terrain where robots are first initialized on easy terrain and promoted to harder terrain if they traverse more than half its length. They are demoted to easier terrain if they fail to travel at least half the commanded distance $v_{x}^{\textrm{cmd}}T$ where $T$ is maximum episode length. We randomize parameters of the simulation (tab. 5) and add small i.i.d. gaussian noise to observations for robustness (tab. 2).

2 Phase 2: Supervised Learning

In phase 2, we use supervised learning to distil the phase 1 policy into an architecture that only has access to sensing available onboard: proprioception ($\mathbf{x}_{t}$) and depth $\mathbf{d}_{t}$.

We create a copy of the recurrent base policy 2. We preprocess the depth map through a convnet before passing it to the base policy.

We train with DAgger with truncated backpropagation through time (BPTT) to minimize mean squared error between predicted and ground truth actions $\|\mathbf{\hat{a}}_{t}-\mathbf{a}_{t}\|^{2}$. In particular, we unroll the student inside the simulator for $N=24$ timesteps and then label each of the states encountered with the ground truth action $\mathbf{a}_{t}$ from phase 1.

Instead of retraining the whole controller, we only train estimators of $\bm{\gamma}_{t}$ and $\mathbf{z}_{t}$, and use the same base policy trained in phase 1 (eqn. 5). The latent $\hat{\bm{\gamma}}$, which encodes terrain geometry, is estimated from history of depth and proprioception using a GRU. Since the camera looks in front of the robot, proprioception combined with depth enables the GRU to implicitly track and estimate the terrain under the robot. Similar to , history of proprioception is used to estimate extrinsics $\mathbf{\hat{z}}$.

As before, this is trained using DAgger with BPTT. The vision GRU 9 and convnet 8 are jointly trained to minimize $\|\bm{\hat{\gamma}}_{t}-\bm{\gamma}_{t}\|^{2}$ while the proprioception GRU 10 minimizes $\|\mathbf{\hat{z}}_{t}-\mathbf{z}_{t}\|^{2}$.

The student can be deployed as-is on the hardware using only the available onboard compute. It is able to handle camera failures and the asynchronous nature of depth due to the randomizations we apply during phase 1. It is robust to pushes, slippery surfaces and large rocky surfaces and can climb stairs, curbs, and cross gaps and stepping stones.

Experimental Setup

We use the IsaacGym (IG) simulator with the legged_gym library to train our walking policies. We construct a large terrain map with 100 sub-terrains arranged in a $20\times 10$ grid. Each row has the same type of terrain arranged in increasing difficulty while different rows have different terrain.

We compare against two baselines, each of which uses the same number of learning samples for both RL phase and supervised learning phase.

Blind policy trained with the scandots observations $\mathbf{m}_{t}$ masked with zeros. This baseline must rely on proprioception to traverse terrain and helps quantify the benefit of vision for walking.

Noisy Methods which rely on elevation maps need to fuse multiple depth images captured over time to obtain a complete picture of terrain under and around the robot. This requires camera pose relative to the first depth input, which is typically estimated using vision or inertial odometry . However, these pose estimates are typically noisy resulting in noisy elevation maps . To handle this, downstream controllers trained on this typically add a large noise in the elevation maps during training. Similar to , we train a teacher with ground truth, noiseless elevation maps in phase 1 and distill it to a student with large noise, with noise model from , added to the elevation map. We simulate a latency of 40ms in both the phases of training to match the hardware. This baseline helps in understanding the effect on performance when relying on pose estimates which introduce additional noise in the pipeline.

Results and Analysis

We report mean time to fall and mean distance travelled before crashing for different terrain and baselines in Table 1. For each method, we train a single policy for all terrains and use that for evaluation. Although the blind policy makes non trivial progress on stairs, discrete obstacles and slopes, it is significantly less efficient at traversing these terrains. On slopes our methods travel upto 27% farther implying that the blind baseline crashes early. Similarly, on stairs and discrete obstacles the distance travelled by our methods is much greater (upto 90%). On slopes and stepping stones the noisy and blind baselines get similar average distances and mean time to fall and both are worse than our policy. This trend is even more significant on the stepping stones terrain where all baselines barely make any progress while our methods travel upto 20m. The blind policy has no way of estimating the position of the stone and crashes as soon as it steps into the gap. For the noisy policy, the large amount of added noise makes it impossible for the student to reliably ascertain the location of the stones since it cannot rely on proprioception any more. We note that the blind baseline is better than the noisy one on stairs. This is because the blind baseline has learnt to use proprioception to figure out location of stairs. On the other hand, the noisy policy cannot learn to use proprioception since it is trained via supervised learning. However, the blind baseline bumps into stairs often is not very practical to run on the real robot. The noisy baseline works well in possibly because of predefined foot motions which make the phase 2 learning easier. However, as noted in sec. 1, predefined motions will not work for our small robot.

We compare the performance of our methods to the blind baseline in the real world. In particular we have 4 testing setups as shows in fig. 3: Upstairs, Downstairs, Gaps and Stepping stones. While we train a single phase 1 policy for all terrain, for running baselines, we obtain different phase 2 policies for stairs vs. stepping stones and gaps. Different phase 2 policies are obtained by changing the location of the camera. We use the in-built camera inside the robot for stairs and a mounted external camera for stepping stones and gaps. The in-built camera is less prone to damage but the stepping stones are gaps are not clearly visible since it is horizontal. This is done for convenience, but we also have a policy that traverses all terrain using the same mounted camera.

We see that the blind baseline is incapable of walking upstairs beyond a few steps and fails to complete the staircase even once. Although existing methods have shown stairs for blind robots, we note that our robot is relatively smaller making it a more challenging task for a blind robot. On downstairs, we observe that the blind baseline achieves 100% success, although it learns to fall on every step and stabilize leading to a very high impact gait which led to the detaching of the rear right hip of the robot during our experiments. We additionally show results in stepping stones and gaps, where the blind robot fails completely establishing the hardness of these setups and the necessity of vision to solve them. We show a 100% success on all tasks except for stepping stone on which we achieve 94% success, which is very high given the challenging setup.

We experiment on stairs, ramps and curbs (fig. LABEL:fig:teaser). The robot was successfully able to go upstairs as well as downstairs for stairs of height upto 24cm in height and 28cm as the lowest width. Since the robot has to remember terrain under its body from visual history, it sometimes misses a step, but shows impressive recovery behaviour and continues climbing or descending. The robot is able to climb curbs and obstacles as high as $26\textrm{cm}$ which is almost as high as the robot 1. This requires an emergent hip abduction movement because the small size of the robot doesn’t leave any space between the body and stair for the leg to step up. This behavior emerges because of our tabula rasa approach to learning gaits without reliance on priors or datasets of natural motion.

We construct an obstacle course consisting of gaps and stepping stones out of tables and stools (fig. 3). For this set of experiments we use a policy trained on stepping stones on gaps, and distilled onto the top camera instead of the front camera. The robot achieves a 100% success rate on gaps of upto 26cm from egocentric depth and 94% on difficult stepping stones. The stepping stones experiment shows that our visual policy can learn safe foothold placement behavior even without an explicit elevation map or foothold optimization objectives. The blind baseline achieves zero success rate on both tasks and falls as soon as any gap is encountered.

We also deploy our policy on outdoor hikes and rocky terrains next to river beds (fig. LABEL:fig:teaser). We see that the robot is able to successfully traverse rugged stairs covered with dirt, small pebbles and some large rocks. It also avoids stumbling over large tree roots on the hiking trail. On the beach, we see that the robot is able to successfully navigate the terrain despite several slips and unstable footholds given the nature of the terrain. We see that the robot sometimes gets stuck in the crevices and in some cases shows impressive recovery behavior as well.

Related Work

Legged locomotion an important problem which has been studied for decades. Several classical works use model based techniques, or define heuristic reactive controllers to achieve the task of walking .This method has led to several promising results in the real world, although they still lack the generality needed to deploy them in the real world. This has motivated work in using RL for learning to walk in simulation , and then successfully deploy them in a diverse set of real world scenarios . Alternatively, a policy learned in simulation can be adapted at test-time to work well in real environments . However, most of these methods are blind, and only use proprioceptive signal to walk.

To achieve visual control of walking, classical methods decouple the perception and control aspects, assuming a perfect output from perception, such as an elevation map, and then using it for planning and control . The control part can be further decoupled into searching for feasible footholds on the elevation map and then execute it with a low-level policy Chestnutt . The foothold feasibility scores can either be estimated heuristically or learned . Other methods forgo explicit foothold optimization and learn traversibility maps instead . Recent methods skip foothold planning and directly train a deep RL policy that takes the elevation map as input and outputs either low-level motor primitives or raw joint angles . Elevation maps can be noisy or incorrect and dealing with imperfect maps is a major challenge to building robust locomotion systems. Solutions to this include incorporating uncertainty in the elevation map and simulating errors at training time to make the walking policy robust to them .

Closest to ours is the line of work that doesn’t construct explicit elevation maps and predicts actions directly from depth. learn a policy for obstacle avoidance from depth on flat terrain, train a hierarchical policy which uses depth to traverse curved cliffs and mazes in simulation, use lidar scans to show zero-shot generalization to difficult terrains. Yu et al. train a policy to step over gaps by predicting high-level actions using depth from the head and below the torso. Relatedly, Margolis et al. train a high-level policy to jump over gaps from egocentric depth using a whole body impulse controller. In contrast, we directly predict target joint angles from egocentric depth without constructing metric elevation maps.

Discussion and Limitations

In this work, we show an end-to-end approach to walking with egocentric depth that can traverse a large variety of terrains including stairs, gaps and stepping stones. However, there can be certain instances where the robot fails because of a visual or terrain mismatch between the simulation and the real world. The only solution to this problem under the current paradigm is to engineer the situation back into simulation and retrain. This poses a fundamental limitation to this approach and in future, we would like to leverage the data collected in the real world to continue improving both the visual and the motor performance.

We would like to thank Kenny Shaw and Xuxin Cheng for help with hardware. Shivam Duggal, Kenny Shaw, Xuxin Cheng, Shikhar Bahl, Zipeng Fu, Ellis Brown helped with recording videos. We also thank Alex Li for proofreading. The project was supported in part by the DARPA Machine Commonsense Program and ONR N00014-22-1-2096.

References

Appendix A Proof of Theorem 3.1

where $c\propto\sum_{s\in\mathcal{S}}V^{\ast}(s)$ is a large but bounded constant.

Since the function $f$ maps the state spaces $\mathcal{S},\mathcal{S}^{\prime}$, we can assume for convenience that both policies $\pi^{1},\pi^{2}$ operate in the same state space $\mathcal{S}$. To obtain an action for $s^{\prime}\in\mathcal{S}^{\prime}$, we can simply query $\pi^{2}(f^{-1}(s^{\prime}))$. Assume that the reward function $R$ and transition function $P$ are Lipschitz

for all states $s_{t},s_{t+1}$ and actions $a_{t},a^{\prime}_{t}$. We generalize the structure of proof for the upper bound on the distance in approximate optimal-value functions to the teacher-student setting. Let $s_{0}$ be the point where the distance between $V^{\ast}$ and $V^{\pi^{S}}$ is maximal

Let $Q^{T}$ be the Q-function corresponding to $V^{T}$. $\pi^{T}$ is obtained by maximizing $Q^{T}(s,a)$ over actions $\pi^{T}(s)=\mathop{\textrm{argmax}}_{a}Q^{T}(s,a)$. Note that in general the value function $V^{\pi^{T}}$ may be different from $V^{T}$. Let $a^{\ast}$ be the action taken by the optimal policy at state $s_{0}$, while $a^{T}=\pi^{T}(s_{0})$ be the action taken by the teacher. Then, the return of the teacher’s greedy action $a^{T}$ must be highest under teacher’s value function $Q^{T}$,

We can expand each side of (15) above to get

Notice that $\left|V^{\ast}(s)-V^{T}(s)\right|<\epsilon$ implies that $V^{T}(s)\in\left[V^{\ast}(s)-\epsilon,V^{\ast}(s)+\epsilon\right]$ which we can plug into the inequality above to get

We can now write a bound for $V^{\ast}(s_{0})-V^{\pi^{S}}(s_{0})$. Let $a^{S}$ be the action taken by the student policy at state $s_{0}$. Then,

Since $s_{0}$ is the state at which the difference between $V^{\ast}$ and $V^{\pi^{S}}$ is maximal, we can claim

for all states $s\in\mathcal{S}$, where $c=\left(L_{R}+\gamma L_{P}\sum_{s\in\mathcal{S}}V^{\ast}(s)\right)$ ∎

Appendix B Rewards

Absolute work penalty $-|\bm{\tau}\cdot\mathbf{q}|$ where $\bm{\tau}$ are the joint torques. We use the absolute value so that the policy does not learn to get positive reward by exploiting inaccuracies in contact simulation.

Command tracking $v_{x}^{\textrm{cmd}}-\left|v_{x}^{\textrm{cmd}}-v_{x}\right|-|\omega_{z}^{\textrm{cmd}}-\omega_{z}|$ where $v_{x}$ is velocity of robot in forward direction and $\omega_{z}$ is yaw angular velocity ($x,z$ are coordinate axes fixed to the robot).

Foot jerk penalty $\sum_{i\in\mathcal{F}}\|\mathbf{f}^{i}_{t}-\mathbf{f}^{i}_{t-1}\|$ where $\mathbf{f}_{t}^{i}$ is the force at time $t$ on the $i^{\textrm{th}}$ rigid body and $\mathcal{F}$ is the set of feet indices. This prevents large motor backlash.

Survival bonus constant value $1$ at each time step to prioritize survival over following commands in challenging situations.

Appendix C Experimental Setup and Implementation Details

Phase 1 is simply reinforcement learning using policy gradients. We describe the pseudo-code for the phase 2 training in Algorithm 1.

C.2 Hardware

We use the Unitree A1 robot pictured in Figure 2 of the main paper. The robot has 12 actuated joints, 3 per leg at hip, thigh and calf joints. The robot has a front-facing Intel RealSense depth camera in its head. TThe compute consists of a small GPU (Jetson NX) capable of $\approx$0.8 TFLOPS and an UPboard with Intel Quad Core Atom X5-8350 containing 4GB ram and 1.92GHz clock speed. The UPboard and Jetson are on the same local network. Since depth processing is an expensive operation we run the convolutional backbone on the Jetson’s GPU and send the depth latent over a UDP socket to the UPboard which runs the base policy. The policy operates at $50\textrm{Hz}$ and sends joint position commands which are converted to torques by a low-level PD controller running at $400\textrm{Hz}$ with stiffness $K_{p}=40$ and damping $K_{d}=0.5$.

We use the IsaacGym (IG) simulator with the legged_gym library to develop walking policies. IG can run physics simulation on the GPU and has a throughput of around $2\textrm{e}5$ time-steps per second on a Nvidia RTX 3090 during phase 1 training with $4096$ robots running in parallel. For phase 2, we can render depth using simulated cameras calibrated to be in the same position as the real camera on the robot. Since depth rendering is expensive and memory intensive, we get a throughput of 500 time-steps per second with $256$ parallel environments. We run phase 1 for $15$ billion samples ($13$ hours) and phase 2 for $6$ million samples ($6$ hours).

We construct a large elevation map with 100 sub-terrains arranged in a $20\times 10$ grid. Each row has the same type of terrain arranged in increasing difficulty while different rows have different terrain. Each terrain has a length and width of $8\textrm{m}$. We add high fractals (upto $10\textrm{cm}$) on flat terrain while medium fractals ($4\textrm{cm}$) on others. Terrains are shown in Figure 5 with randomization ranges described in Table 5.