Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs

I Introduction

With the great progress on both hardware and algorithm, quadrupedal robots recently have shown significant performance in locomotion tasks, such as high-speed running , robust falling recovery , walking on challenging terrain . Compared to wheeled or biped robots, thanks to the flexible limbs and larger support region, quadrupedal robots are more robust to cope with complex environments, such as stairs and uneven terrain. Consequently, quadrupedal robots are expected to achieve various real-world missions. These vehicles have proven to be extremely robust and can cope with complex environments and terrains, which opened the potential for application in real world missions .

However, most quadrupedal robots’ applications focus on navigation and inspection, while active interaction and manipulation of the environment are still lacking. To overcome these limitations and extend the advanced locomotion capability with manipulation skills, several groups equipped their quadrupedal robots with a robotic arm and gripper. This enables basic manipulation tasks such as pick-and-place, door opening, and cooperative carrying, whereby the manipulation problem is largely decoupled from locomotion. On the downside, payload limitations of the existing quadrupedal robots mostly only allow carrying a single arm, which limits the manipulation skills.

In contrast to these developments in robotics, where locomotion and manipulation are tackled with separate hardware components, we can observe truly amazing manipulation skills in nature. Quadrupedal animals can utilize their legs to dexterously manipulate the environment (Fig. 1a). Inspired by this ability, a few approaches have suggested utilizing the legs as manipulators to achieve dexterity with minimal hardware modification. In , two limbs are used to pick up a box and the other two limbs for stance balancing. For multi-legs robots such as hexapod robots, four limbs are used balancing while the other two limbs are manipulating objects . Another solution is to add additional support legs to the quadrupedal robot to maximize the robustness of loco-manipulation . Compared to dual-limbs manipulation, the most extreme idea is to synthesize all the limbs for manipulation. Inchworm-like motion with four legs is achieved in to manipulate two boards. In , a quadrupedal robot in the simulation balances on a ball and simultaneously manipulates it.

Locomotion and manipulation are regarded as the dual problem in some aspects , whereby legs could be viewed as the fingers of in-hand manipulation. Dexterous in-hand manipulation has a long history and is still a challenging problem . A high-DoF hand’s controller must handle complex contact dynamics that are difficult to model accurately and challenging to compute online. Recently, deep reinforcement learning shows great progress on in-hand dexterous manipulation . Robots can learn robust policies in simulation or real environment to operate valve , rotate block and even solve the Rubik’s Cube in the real world.

On the other hand, for quadrupedal locomotion, deep reinforcement learning also plays a vital role in recent progress. Walking policy is learned on a small-size quadrupedal robot . Actuator network is developed to reduce the sim-to-real gap and achieve robust dynamic locomotion skills on ANYmal robot . Gait selection is learned to cope with rough terrain with less energy usage . Animal behavior is imitated and further improved by RL to deploy on the real robot .

Motivated by the recent progress in learning quadrupedal locomotion and in-hand manipulation, we revisit the leg-manipulation task to attain four-limbs manipulation skills. Our main contribution is to develop a framework that enables a quadrupedal robot to achieve a first step towards dexterous full-limbs manipulation. We demonstrate our framework’s effectiveness by showcasing it on a series of challenging circus tasks such as rotating a ball in roll, pitch, and yaw direction with different velocity. To the best of our knowledge, it is the first work to achieve the quadrupedal dexterous dynamic object manipulation on a real robot.

II Method

In this paper, we propose a method for in-limb manipulation with ANYmal , a quadruped robot actuated with $12$ series elastic actuators (SEAs), as illustrated in Fig. 2. Our method allows dynamically rotating a circus ball based on user-defined target velocities. To verify the $SO(3)$ rotation ability, we train the policy to rotate the ball in roll, pitch, and yaw directions. Moreover, we limit the manipulation contact points only on feet, where the robot’s base should not touch the ball.

The proprioceptive measurements are limited only to the joints’ position and velocity without any contact measurement such as contact state or contact force. To further simplify the problem, position and orientation of the ball are obtained from an external motion capture system, and mass and radius of the ball are known to the controller. We train the policy with model-free RL in the Raisim simulator , and achieve the zero-shot deployment on the real robot.

In the rest of this section, the whole learning process would be introduced, including the environment’s Markov Decision Process(MDP) formulation, detailed training settings, network architecture, and policy gradient algorithm.

The observation includes joint states, previous action, ball states, and task-related inputs. Compared to the locomotion task , the position and velocity of robot base are eliminated since the robot lies on its back during the manipulation task. The joint state includes positions and velocities measured by joint encoders on the robot. The action in the previous time step (i.e., joint position command) is recorded and included as well. The ball’s position and orientation (relative to the robot base) is observed by the external motion capture system. The original command is the ball’s target angular velocity. We translate it into two parts: (1) target orientation represented by quaternion and (2) time to reach the goal orientation. Both parts are updated periodically.

We stack the observations of the last three time steps , which is the same history length as the input to the actuator model. This enables the policy to handle latencies and partially observable states of the hardware

The action (output of the policy) is sent to the robot as the joint target position. A low-level joint position PD controller translates the policy output to motor torque command.

II-B Reward Function

The goal of our reward design is to encourage the robot to manipulate the ball to track the commanded speed while being intuitive and simple with a minimum number of terms. In contrast to locomotion tasks , here, the speed is implicitly represented by the periodically updated target orientation. The reasons are twofold: first, assuming sphere manipulation as the complex-terrain locomotion, it is difficult to keep constant speed ; and second, for most manipulation tasks, it is not necessary to maintain constant speed, but accurate tracking for target orientation is more critical . Thus the reward is represented as the angle difference $\delta q$ between the current and target orientation

where $\delta q$ could be calculated from quaternion difference in the observation state: $\delta q=2\hskip 2.84526ptcos^{-1}(\bm{q}_{t}(0))$.

Aggressive leg motions can result in base motion, which is dangerous to the hardware and can cause task failure. To avoid this behavior and keep the base still, a negative reward is added to punish any base velocity.

here $\bm{v}_{base}(t)$ denotes the linear velocity of robot base, which could be obtained in the simulation. This reward is dominant in the early training stage and reduces to zero in the middle stage.

Joint torque is punished for keeping the feasible and energy-efficient torque distribution and preventing high contact force during manipulation. High contact forces often denote that the legs are forcefully squeezing the ball, which can deform the object and leads to task failure.

Although less slippage at contact points is desirable, completely avoiding it is not realistic. It would cause a significant reality gap between simulation and real robot, which is also the limitation in the previous model-based method . Thus, we only moderately punish the slipping velocity instead

where $\bm{v}_{\tan}(t)$ denotes the relative tangent velocity between the ball and the end-effector of each leg.

Compared to walking on the stiff ground, most of the manipulated daily object is more deformable under large contact force. Since our simulator could not simulate deformation, it would lead to huge difference between the sim and real especially in the normal direction of contact forces. In our real experiment, a yoga ball is used, which is even more deformable. To tackle this problem, we punish the normal contact velocity.

where $\bm{v}_{norm}(t)$ is the relative contact velocity in the normal direction between ball and the end-effector of each leg.

Compared to the reward design in quadrupedal locomotion task , the joint speed reward, and foot clearance reward are eliminated. The former one is implicitly contained in the robot base and contact velocity reward. The latter one, whose essential part is the foot clearance threshold, is not intuitive to select in our setup with shaped objects compared to flat terrain.

II-C Early Termination

Early termination is one of the essential components in the training procedure , which eliminates the local minimum or corner case with unnatural performance. In our manipulation task, the early termination would be triggered when:

The ball contacts with other links except for the end-effector on feet.

The ball’s position is out of a feasible region.

The period in which the ball is in a no-contact state exceeds a threshold.

The threshold region is set as a horizontal plane being $\pm{1.5}$ radius of the ball and vertical axis being $\pm{1}$ radius. The limit on ball-no-contact time is to avoid solutions in which the robot throws the rotating ball into the air.

II-D Domain Randomization

Domain randomization is a simple technique to improve a policy’s robustness against modeling errors and sensor measurement noise as Fig. 3. In our circus task, the randomization takes place in the followings:

Ball contact model. The friction and restitution coefficients are sampled from uniform distributions $U(0.5,1.1)$ and $U(0.9,1.0)$

Initialization state, including initial robot position and configuration, ball position and orientation

where the policy is encouraged to learn strategies under these varying dynamics and initialization to better deal with real world conditions.

II-E External Disturbance

Injecting random disturbance force has shown to be effective in achieving sim-to-real transfer . During training, the ball is applied with 50 N external force from a random direction. The disturbance force lasts for 0.4 s and appears at the random time point with 20 % probability.

II-F Policy Training

The policy and value network are multi-layer perceptron (MLP) with $2$ hidden layers with 256 and 128 units for each and tanh activation function as Fig. 4.

The Proximal Policy Optimization (PPO) algorithm is used for training. The hyperparameters are: discount factor $\gamma=0.998$ , clipping range $\epsilon=0.2$ , learning rate $lr=0.001$ . The parameterized policy $\pi_{\theta}(a_{t}|o_{t})$ is used to maximize the expected reward return:

II-G Curriculum Learning

III Experiment

The policy is trained in simulation and achieves zero-shot transfer to the real robot. We demonstrate our framework on ANYmal-Chttps://www.anybotics.com with a total mass of about $40$ kg and $12$ SEAs with a maximum torque of $80$ Nm. The ball in the experiment is a commercial yoga ball with 3 kg mass and 0.8 m diameter. A VICON systemhttps://www.vicon.com measures Center-of-gravity (CoG) position and orientation of the ball.

We have identified the following mismatches between simulation and the real robot experiments: the joints position tracking errors, the softness and deformation in yoga ball, the unexpected slip between ball surface and feet, the non-standard ellipse shape after inflation and CoG position error. However, our learned control policy has proven to be robust to these uncertainties.

The policy is trained in the Raisim simulator , which is used to simulate the rigid-body contact dynamics. During training, an actuator network is used to reduce the modeling mismatch between simulation and real robots due to unmodeled actuator dynamics.

The quality of the trained policy is strongly related to both target speeds and updating periods of the target orientation. Larger target speed leads to more aggressive motion, while a shorter period leads to more frequent contact changes. The policy is trained with different updating periods, as described in Sec. II-G. The simulation results show that the policy with an extended period generates smoother motion with less contact switch but easier to fail when the target speed and domain randomization noises are increased. In contrast, the policy with a short period leads to more contact switch, but enables larger target speed with more robustness. Due to the joints’ physical limitations in velocity and acceleration, the updating period could not be infinitely small. Analogous to the gait scheduling settings in locomotion, the final period is selected to be 0.33 s by updating the target orientation at 3 Hz.

We also notice that policy becomes more conservative with increasing external disturbance and growing domain randomization. One example is foot clearance decreases during training to quickly respond to the unexpected disturbance.

III-B Real Robot

We run the same policy in the real robot without any modification. The policy runs 100 Hz on the real robot and sends joints position command to the robot’s joint position PD controllers.

The tracking performance is shown in Fig. 5, in which the results of accurate tracking and recovery from the external force are presented. We randomly poke the ball and robot limbs from different directions, as the snapshots in Fig. 6 and Fig. 7. Our robot quickly recovers and continues the task, though we do not have a high-level task planner, and our non-prehensile style is less stable. Moreover, we notice in Fig. 5(b) that the angular velocity fluctuates from the target value, even if the orientation is tracked pretty well. As Fig. 5(d), the peak and averages of joints’ torque and velocities during manipulation are about $\frac{1}{4}$ of locomotion tasks using a similar ANYmal robot .

IV Discussion

Multi-legged locomotion shows the duality with multi-fingered manipulation . This section compares the similarity and difference between locomotion and our manipulation task under the reinforcement learning settings.

In locomotion, the reward mostly directly depends on velocity . In contrast for our manipulation task, we discretize the time at a fixed frequency and update the target quaternion value based on the velocity command. Compared to direct velocity-based reward, we found that our reward design converges better during training. We speculate that the main reason is that the terrain is more complicated in our manipulation case than during flat ground locomotion, making it more difficult to track the constant speed. A similar observation was made for rough-terrain locomotion . As it is shown in the experimental results Fig. 5(b), the average velocity is pretty accurate, while the speed can be quite noisy. We also study humans conducting a similar task with a soccer ball and using their fingers. While establishing a direct parallel is impossible, we observe resembling undulant motions during the task execution.

Foot Clearance

Foot clearance denotes the distance between legs end-effector and the environment. It is a popular reward term in shaping gait behavior during locomotion to avoid foot scuffing . The reward is often calculated by variations to a heuristic clearance value. In our manipulation task, foot clearance term is removed because it is not intuitive to decide on a heuristic value.

Gait Pattern

In a quadrupedal locomotion task, the gait pattern is often preset with specific contact sequences provided as input to the learned policy . In our work, we do not define any gait pattern. In contrast to legged locomotion, where the heuristics can be directly inspired by well-studied animal gaits, we cannot rely on comparable gait information for in-limb manipulation.

Model-based and Model-free

In quadrupedal locomotion tasks, model-based RL combines learned gait with a model-based whole-body controller . However, due to the whole-body controller’s sensitivity to model-mismatches, it was not suitable for our task as the ball deformation and slippage impede any accurate modeling.

IV-B In-Limb vs Dexterous Hand Manipulation

V Conclusion

This paper has proposed a novel and robust approach based on deep reinforcement learning for the quadrupedal robot to achieve a first step towards dexterous full-limbs manipulation. The policy is trained in the simulation with model-free RL, and achieve zero-shot deployment on the real robot. This is, to our best knowledge, the first work to achieve dexterous dynamic manipulation on the real quadrupedal robot.

The proposed controller exhibits dynamic manipulation performance and achieves a maximum $15deg/s$ ball rotation speed on hardware experiments. The controller’s robustness is verified on hardware by showing first, a continuous manipulation for more than $2$ minutes, and second, a robust recovery under external disturbances during manipulation.

In the end, we revisit the classical argument of the duality between manipulation and locomotion. By comparing the similarities and differences in reward design under RL settings, for the challenging terrain, we hope to inspire a more closed connection on both sides.

Acknowledgments

This research was supported by the Swiss National Science Foundation through the National Center of Competence in Research (NCCR) Robotics. We appreciate the insightful discussion with Vassilios Tsounis, Bowen Yang, Jan Carius, Lorenz Wellhausen from ETH Zürich, Prof. Davide Scaramuzza from the University of Zürich, Jemin Hwangbo from KAIST, Hironori Yoshida from the University of Tokyo, Yifan Hou from CMU, Prof. Weiwei Wan from Osaka University.

References