Deep Imitative Models for Flexible Inference, Planning, and Control

Introduction

Imitation learning (IL) is a framework for learning a model to mimic behavior. At test-time, the model pursues its best-guess of desirable behavior. By letting the model choose its own behavior, we cannot direct it to achieve different goals. While work has augmented IL with goal conditioning (Dosovitskiy & Koltun, 2016; Codevilla et al., 2018), it requires goals to be specified during training, explicit goal labels, and are simple (e.g., turning). In contrast, we seek flexibility to achieve general goals for which we have no demonstrations.

In contrast to IL, planning-based algorithms like model-based reinforcement learning (MBRL) methods do not require expert demonstrations. MBRL can adapt to new tasks specified through reward functions (Kuvayev & Sutton, 1996; Deisenroth & Rasmussen, 2011). The “model” is a dynamics model, used to plan under the user-supplied reward function. Planning enables these approaches to perform new tasks at test-time. The key drawback is that these models learn dynamics of possible behavior rather than dynamics of desirable behavior. This means that the responsibility of evoking desirable behavior is entirely deferred to engineering the input reward function. Designing reward functions that cause MBRL to evoke complex, desirable behavior is difficult when the space of possible undesirable behaviors is large. In order to succeed, the rewards cannot lead the model astray towards observations significantly different than those with which the model was trained.

Our goal is to devise an algorithm that combines the advantages of MBRL and IL by offering MBRL’s flexibility to achieve new tasks at test-time and IL’s potential to learn desirable behavior entirely from offline data. To accomplish this, we first train a model to forecast expert trajectories with a density function, which can score trajectories and plans by how likely they are to come from the expert. A probabilistic model is necessary because expert behavior is stochastic: e.g. at an intersection, the expert could choose to turn left or right. Next, we derive a principled probabilistic inference objective to create plans that incorporate both (1) the model and (2) arbitrary new tasks. Finally, we derive families of tasks that we can provide to the inference framework. Our method can accomplish new tasks specified as complex goals without having seen an expert complete these tasks before.

We investigate properties of our method on a dynamic simulated autonomous driving task (see Fig. 1). Videos are available at https://sites.google.com/view/imitative-models. Our contributions are as follows:

Interpretable expert-like plans without reward engineering. Our method outputs multi-step expert-like plans, offering superior interpretability to one-step imitation learning models. In contrast to MBRL, our method generates expert-like behaviors without reward function crafting.

Flexibility to new tasks: In contrast to IL, our method flexibly incorporates and achieves goals not seen during training, and performs complex tasks that were never demonstrated, such as navigating to goal regions and avoiding test-time only potholes, as depicted in Fig. 1.

Robustness to goal specification noise: We show that our method is robust to noise in the goal specification. In our application, we show that our agent can receive goals on the wrong side of the road, yet still navigate towards them while staying on the correct side of the road.

State-of-the-art CARLA performance: Our method substantially outperforms MBRL, a custom IL method, and all five prior CARLA IL methods known to us. It learned near-perfect driving through dynamic and static CARLA environments from expert observations alone.

Deep Imitative Models

To learn agent dynamics that are possible and preferred, we construct a model of expert behavior. We fit an “Imitative Model” $q(\mathbf{S}_{1:T}|\phi)=\prod_{t=1}^{T}q(\mathbf{S}_{t}|\mathbf{S}_{1:t-1},\phi)$ to a dataset of expert trajectories $\mathcal{D}=\{(s^{i},\phi^{i})\}_{i=1}^{N}$ drawn from a (unknown) distribution of expert behavior $s^{i}\sim p(\mathbf{S}|\phi^{i})$. By training $q(\mathbf{S}|\phi)$ to forecast expert trajectories with high likelihood, we model the scene-conditioned expert dynamics, which can score trajectories by how likely they are to come from the expert.

After training, $q(\mathbf{S}|\phi)$ can generate trajectories that resemble those that the expert might generate – e.g. trajectories that navigate roads with expert-like maneuvers. However, these maneuvers will not have a specific goal. Beyond generating human-like behaviors, we wish to direct our agent to goals and have the agent automatically reason about the necessary mid-level details. We define general tasks by a set of goal variables $\mathcal{G}$. The probability of a plan $\mathbf{s}$ conditioned on the goal $\mathcal{G}$ is modelled by a posterior $p(\mathbf{s}|\mathcal{G},\phi)$. This posterior is implemented with $q(\mathbf{s}|\phi)$ as a learned imitation prior and $p(\mathcal{G}|\mathbf{s},\phi)$ as a test-time goal likelihood. We give examples of $p(\mathcal{G}|\mathbf{s},\phi)$ after deriving a maximum a posteriori inference procedure to generate expert-like plans that achieve abstract goals:

We perform gradient-based optimization of Eq. 1, and defer this discussion to Appendix A. Next, we discuss several goal likelihoods, which direct the planning in different ways. They communicate goals they desire the agent to achieve, but not how to achieve them. The planning procedure determines how to achieve them by producing paths similar to those an expert would have taken to reach the given goal. In contrast to black-box one-step IL that predicts controls, our method produces interpretable multi-step plans accompanied by two scores. One estimates the plan’s “expertness”, the second estimates its probability to achieve the goal. Their sum communicates the plan’s overall quality.

2 Constructing Goal Likelihoods

Costed planning: Our model has the additional flexibility to accept arbitrary user-specified costs $c$ at test-time. For example, we may have updated knowledge of new hazards at test-time, such as a given map of potholes or a predicted cost map. Cost-based knowledge $c(\mathbf{s}_{i}|\phi)$ can be incorporated as an (G) Energy-based likelihood: $p(\mathcal{G}|\mathbf{s},\phi)\propto\prod_{t=1}^{T}e^{-c(\mathbf{s}_{t}|\phi)}$ (Todorov, 2007; Levine, 2018). This can be combined with other goal-seeking objectives by simply multiplying the likelihoods together. Examples of combining G (energy-based) with F (Gaussian mixture) were shown in Fig. 1 and are shown in Fig. 4. Next, we describe instantiating $q(\mathbf{S}|\phi)$ in CARLA (Dosovitskiy et al., 2017).

3 Applying Deep Imitative Models to Autonomous Driving

4 Imitative Driving

We now instantiate a complete autonomous driving framework based on imitative models to study in our experiments, seen in Fig. 6. We use three layers of spatial abstraction to plan to a faraway destination, common to autonomous vehicle setups: coarse route planning over a road map, path planning within the observable space, and feedback control to follow the planned path (Paden et al., 2016; Schwarting et al., 2018). For instance, a route planner based on a conventional GPS-based navigation system might output waypoints roughly in the lanes of the desired direction of travel, but not accounting for environmental factors such as the positions of other vehicles. This roughly communicates possibilities of where the vehicle could go, but not when or how it could get to them, or any environmental factors like other vehicles. A goal likelihood from Sec. 2.2 is formed from the route and passed to the planner, which generates a state-space plan according to the optimization in Eq. 1. The resulting plan is fed to a simple PID controller on steering, throttle, and braking. In Pseudocode of the driving, inference, and PID algorithms are given in Appendix A.

Related Work

A body of previous work has explored offline IL (Behavior Cloning – BC) in the CARLA simulator (Li et al., 2018; Liang et al., 2018; Sauer et al., 2018; Codevilla et al., 2018; 2019). These BC approaches condition on goals drawn from a small discrete set of directives. Despite BC’s theoretical drift shortcomings (Ross et al., 2011), these methods still perform empirically well. These approaches and ours share the same high-level routing algorithm: an A∗ planner on route nodes that generates waypoints. In contrast to our approach, these approaches use the waypoints in a Waypoint Classifier, which reasons about the map and the geometry of the route to classify the waypoints into one of several directives: {Turn left, Turn right, Follow Lane, Go Straight}. One of the original motivations for these type of controls was to enable a human to direct the robot (Codevilla et al., 2018). However, in scenarios where there is no human in the loop (i.e. autonomous driving), we advocate for approaches to make use of the detailed spatial information inherent in these waypoints. Our approach and several others we designed make use of this spatial information. One of these is CIL-States (CILS): whereas the approach in Codevilla et al. (2018) uses images to directly generate controls, CILS uses identical inputs and PID controllers as our method. With respect to prior conditional IL methods, our main approach has more flexibility to handle more complex directives post-training, the ability to learn without goal labels, and the ability to generate interpretable planned and unplanned trajectories. These contrasting capabilities are illustrated in Table 2.

Our approach is also related to MBRL. MBRL can also plan, but with a one-step predictive model of possible dynamics. The task of evoking expert-like behavior is offloaded to the reward function, which can be difficult and time-consuming to craft properly. We know of no MBRL approach previously applied to CARLA, so we devised one for comparison. This MBRL approach also uses identical inputs to our method, instead to plan a reachability tree (LaValle, 2006) over an dynamic obstacle-based reward function. See Appendix D for further details of the MBRL and CILS methods, which we emphasize use the same inputs as our method.

Several prior works (Tamar et al., 2016; Amos et al., 2018; Srinivas et al., 2018) used imitation learning to train policies that contain planning-like modules as part of the model architecture. While our work also combines planning and imitation learning, ours captures a distribution over possible trajectories, and then plan trajectories at test-time that accomplish a variety of given goals with high probability under this distribution. Our approach is suited to offline-learning settings where interactively collecting data is costly (time-consuming or dangerous). However, there exists online IL approaches that seek to be safe (Menda et al., 2017; Sun et al., 2018; Zhang & Cho, 2017).

Experiments

We evaluate our method using the CARLA driving simulator (Dosovitskiy et al., 2017). We seek to answer four primary questions: (1) Can we generate interpretable, expert-like plans with offline learning and no reward engineering? Neither IL nor MBRL can do so. It is straightforward to interpret the trajectories by visualizing them on the ground plane; we thus seek to validate whether these plans are expert-like by equating expert-like behavior with high performance on the CARLA benchmark. (2) Can we achieve state-of-the-art CARLA performance using resources commonly available in real autonomous vehicle settings? There are several differences between the approaches, as discussed in Sec 3 and shown in Tables 2 and 2. Our approach uses the CARLA toolkit’s resources that are commonly available in real autonomous vehicle settings: waypoint-based routes (all prior approaches use these) and LIDAR (CARLA-provided, but only the approaches we implemented use it). Furthermore, the two additional methods of comparison we implemented (CILS and MBRL) use the exact same inputs as our algorithm. These reasons justify an overall performance comparison to answer (2): whether we can achieve state-of-the-art performance using commonly available resources. We advocate that other approaches also make use of such resources. (3) How flexible is our approach to new tasks? We investigate (3) by applying each of the goal likelihoods we derived and observing the resulting performance. (4) How robust is our approach to error in the provided goals? We do so by injecting two different types of error into the waypoints and observing the resulting performance.

Results: Towards questions (1) and (3) (expert-like plans and flexibility), we apply our approach with a variety of goal likelihoods to the CARLA simulator. Towards question (2), we compare our methods against CILS, MBRL, and prior work. These results are shown in Table 3. The metrics for the methods we did not implement are from the aggregation reported in Codevilla et al. (2019). We observe our method to outperform all other approaches in all settings: static world, dynamic world, training conditions, and test conditions. We observe the Goal Indicator methods are able to perform well, despite having no hyperparameters to tune. We found that we could further improve our approach’s performance if we use the light state to define different goal sets, which defines a “smart” waypointer. The settings where we use this are suffixed with “S.” in the Tables. We observed the planner prefers closer goals when obstructed, when the vehicle was already stopped, and when a red light was detected; we observed the planner prefers farther goals when unobstructed and when green lights or no lights were observed. Examples of these and other interesting behaviors are best seen in the videos on the website (https://sites.google.com/view/imitative-models). These behaviors follow from the method leveraging $q(\mathbf{S}|\phi)$’s internalization of aspects of expert behavior in order to reproduce them in new situations. Altogether, these results provide affirmative answers to questions (1) and (2). Towards question (3), these results show that our approach is flexible to different directions defined by these goal likelihoods.

Towards questions (3) (flexibility) and (4) (noise-robustness), we analyze the performance of our method when the path planner is heavily degraded, to understand its stability and reliability. We use the Gaussian Final-State Mixture goal likelihood.

Navigating with high-variance waypoints. As a test of our model’s capability to stay in the distribution of demonstrated behavior, we designed a “decoy waypoints” experiment, in which half of the waypoints are highly perturbed versions of the other half, serving as distractions for our Gaussian Final-State Mixture imitative planner. We observed surprising robustness to decoy waypoints. Examples of this robustness are shown in Fig. 8. In Table 4, we report the success rate and the mean number of planning rounds for failed episodes in the “$\nicefrac{{1}}{{2}}$ distractors” row. These numbers indicate our method can execute dozens of planning rounds without decoy waypoints causing a catastrophic failure, and often it can execute the hundreds necessary to achieve the goal. See Appendix E for details.

Navigating with waypoints on the wrong side of the road. We also designed an experiment to test our method under systemic bias in the route planner. Our method is provided waypoints on the wrong side of the road (in CARLA, the left side), and tasked with following the directions of these waypoints while staying on the correct side of the road (the right side). In order for the value of $q(\mathbf{s}|\phi)$ to outweigh the influence of these waypoints, we increased the $\epsilon$ hyperparameter. We found our method to still be very effective at navigating, and report results in Table 4. We also investigated providing very coarse 8-meter wide regions to the Region Final-State likelihood; these always include space in the wrong lane and off-road (Fig. 12 in Appendix B.4 provides visualization). Nonetheless, on Town01 Dynamic, this approach still achieved an overall success rate of $48\%$. Taken together towards question (4), our results indicate that our method is fairly robust to errors in goal-specification.

2 Producing Unobserved Behaviors to Avoid Novel Obstacles

To further investigate our model’s flexibility to test-time objectives (question 3), we designed a pothole avoidance experiment. We simulated potholes in the environment by randomly inserting them in the cost map near waypoints. We ran our method with a test-time-only cost map of the simulated potholes by combining goal likelihoods (F) and (G), and compared to our method that did not incorporate the cost map (using (F) only, and thus had no incentive to avoid potholes). We recorded the number of collisions with potholes. In Table 4, our method with cost incorporated avoided most potholes while avoiding collisions with the environment. To do so, it drove closer to the centerline, and occasionally entered the opposite lane. Our model internalized obstacle avoidance by staying on the road and demonstrated its flexibility to obstacles not observed during training. Fig. 8 shows an example of this behavior. See Appendix F for details of the pothole generation.

Discussion

We proposed “Imitative Models” to combine the benefits of IL and MBRL. Imitative Models are probabilistic predictive models able to plan interpretable expert-like trajectories to achieve new goals. Inference with an Imitative Model resembles trajectory optimization in MBRL, enabling it to both incorporate new goals and plan to them at test-time, which IL cannot. Learning an Imitative Model resembles offline IL, enabling it to circumvent the difficult reward-engineering and costly online data collection necessities of MBRL. We derived families of flexible goal objectives and showed our model can successfully incorporate them without additional training. Our method substantially outperformed six IL approaches and an MBRL approach in a dynamic simulated autonomous driving task. We showed our approach is robust to poorly specified goals, such as goals on the wrong side of the road. We believe our method is broadly applicable in settings where expert demonstrations are available, flexibility to new situations is demanded, and safety is paramount.

References

Appendix A Algorithms

In Algorithm 1, we provide pseudocode for receding-horizon control via our imitative model. In Algorithm 2 we provide pesudocode that describes how we plan in the latent space of the trajectory. Since $\mathbf{s}_{1:T}=f(\mathbf{z}_{1:T})$ in our implementation, and $f$ is differentiable, we can perform gradient descent of the same objective in terms of $\mathbf{z}_{1:T}$. Since $q$ is trained with $\mathbf{z}_{1:T}\sim\mathcal{N}(0,I)$, the latent space is likelier to be better numerically conditioned than the space of $\mathbf{s}_{1:T}$, although we did not compare the two approaches formally.

In Algorithm 3, we detail the speed-based throttle and position-based steering PID controllers.

Appendix B Goal Details

We now derive an approach to optimize our main objective with set constraints. Although we could apply a constrained optimizer, we find that we are able to exploit properties of the model and constraints to derive differentiable objectives that enable approximate optimization of the corresponding closed-form optimization problems. These enable us to use the same straightforward gradient-descent-based optimization approach described in Algorithm 2.

In this section we omit dependencies on $\phi$ for brevity, and use short hand $\mu_{t}\doteq\mu_{\theta}(\mathbf{s}_{1:t-1})$ and $\Sigma_{t}\doteq\Sigma_{\theta}(\mathbf{s}_{1:t-1})$. For example, $q(\mathbf{s}_{t}|\mathbf{s}_{1:t-1})=\mathcal{N}\left(\mathbf{s}_{t};\mu_{t},\Sigma_{t}\right)$.

Let us begin by defining a useful delta function:

By exploiting the fact that $q(\mathbf{s}_{T}|\mathbf{s}_{1:T-1})=\mathcal{N}\left(\mathbf{s}_{T};\mu_{T},\Sigma_{T}\right)$, we can derive closed-form solutions for

Note that equation 8 only helps solve equation 5 if equation 6 has a closed-form solution. We detail example of goal-sets with such closed-form solutions in the following subsections.

B.1.1 Point goal-set

More generally, multiple point goals help define optional end points for planning: where the agent only need reach one valid end point (see Fig. 9 for examples), formulated as:

B.1.2 Line-segment goal-set

The solution to equation 6 in the case of line-segment goals is:

To solve equation 15 is to find which point along the line $g_{\text{line}}(u)$ maximizes $\mathcal{N}\left(\cdot;\mu_{T},\Sigma_{T}\right)$ subject to the constraint $0\leq u\leq 1$:

Since $\mathcal{L}_{u}$ is convex, the optimal value $u^{*}$ is value closest to the unconstrained $\operatorname*{arg\,max}$ of $\mathcal{L}_{u}(u)$, subject to $0\leq u\leq 1$:

B.1.3 Multiple-line-segment goal-set:

B.1.4 Polygon goal-set

Solving equation 6 with a polygon has two cases: depending whether $\mu_{T}$ is inside the polygon, or outside. If $\mu_{T}$ lies inside the polygon, then the optimal value for $\mathbf{s}_{T}^{*}$ that maximizes $\mathcal{N}(\mathbf{s}_{T}^{*};\mu_{T},\Sigma_{T})$ is simply $\mu_{T}$: the mode of the Gaussian distribution. Otherwise, if $\mu_{T}$ lies outside the polygon, then the optimal value $\mathbf{s}_{T}^{*}$ will lie on one of the polygon’s edges, solved using B.1.3.

B.2 Waypointer Details

The waypointer uses the CARLA planner’s provided route to generate waypoints. In the constrained-based planning goal likelihoods, we use this route to generate waypoints without interpolating between them. In the relaxed goal likelihoods, we interpolate this route to every $2$ meters, and use the first $20$ waypoints. As mentioned in the main text, one variant of our approach uses a “smart” waypointer. This waypointer simply removes nearby waypoints closer than $5$ meters from the vehicle when a green light is observed in the measurements provided by CARLA, to encourage the agent to move forward, and removes far waypoints beyond $5$ meters from the vehicle when a red light is observed in the measurements provided by CARLA. Note that the performance differences between our method without the smart waypointer and our method with the smart waypointer are small: the only signal in the metrics is that the smart waypointer improves the vehicle’s ability to stop for red lights, however, it is quite adept at doing so without the smart waypointer.

B.3 Constructing Goal Sets

For the methods we implemented, the task is to drive the furthest road location from the vehicle’s initial position. Note that this protocol more difficult than the one used in prior work Codevilla et al. (2018); Liang et al. (2018); Sauer et al. (2018); Li et al. (2018); Codevilla et al. (2019), which has no distance guarantees between start positions and goals, and often results in shorter paths.

B.4 Planning visualizations

Visualizations of examples of our method deployed with different goal likelihoods are shown in Fig. 9, Fig. 10, Fig. 11, and Fig. 12.

Appendix C Architecture and Training Details

The architecture of $q(\mathbf{S}|\phi)$ is shown in Table 5.

We show examples of the priors multimodality in Fig. 13

Following are the values of the planning criterion on $N\approx 8\cdot 10^{3}$ rounds from applying the “Gaussian Final-State Mixture” to Town01 Dynamic. Mean of $\log q(\mathbf{s}^{*}|\phi)\approx 104$. Mean of $\log p(\mathcal{G}|\mathbf{s}^{*},\phi)=-4$. This illustrates that while the prior’s value mostly dominates the values of the final plans, the Gaussian Final-State Goal Mixture likelihood has a moderate amount of influence on the value of the final plan.

C.2 Dataset

Before training $q(\mathbf{S}|\phi)$, we ran CARLA’s expert in the dynamic world setting of Town01 to collect a dataset of examples. We have prepared the dataset of collected data for public release upon publication. We ran the autopilot in Town01 for over 900 episodes of 100 seconds each in the presence of 100 other vehicles, and recorded the trajectory of every vehicle and the autopilot’s LIDAR observation. We randomized episodes to either train, validation, or test sets. We created sets of 60,701 train, 7586 validation, and 7567 test scenes, each with 2 seconds of past and 4 seconds of future position information at 10Hz. The dataset also includes 100 episodes obtained by following the same procedure in Town02.

Appendix D Baseline Details

We designed a conditional imitation learning baselines that predicts the setpoint for the PID-controller. Each receives the same scene observations (LIDAR) and is trained with the same set of trajectories as our main method. It uses nearly the same architecture as that of the original CIL, except it outputs setpoints instead of controls, and also observes the traffic light information. We found it very effective for stable control on straightaways. When the model encounters corners, however, prediction is more difficult, as in order to successfully avoid the curbs, the model must implicitly plan a safe path. We found that using the traffic light information allowed it to stop more frequently.

D.2 Model-Based Reinforcement Learning:

We plan forwards over 20 time steps using a breadth-first search search over CARLA steering angle $\{-0.3,-0.1,0.,0.1,0.3\}$, noting valid steering angles are normalized to $$, with constant throttle at 0.5, noting the valid throttle range is$$. Our search expands each state node by the available actions and retains the 50 closest nodes to the waypoint. The planned trajectory efficiently reaches the waypoint, and can successfully plan around perceived obstacles to avoid getting stuck. To convert the LIDAR images into obstacle maps, we expanded all obstacles by the approximate radius of the car, 1.5 meters.

Appendix E Robustness Experiments details

In the decoy waypoints experiment, the perturbation distribution is $\mathcal{N}(0,\sigma=8m)$: each waypoint is perturbed with a standard deviation of $8$ meters. One failure mode of this approach is when decoy waypoints lie on a valid off-route path at intersections, which temporarily confuses the planner about the best route. Additional visualizations are shown in Fig. 14.

E.2 Plan Reliability Estimation

Besides using our model to make a best-effort attempt to reach a user-specified goal, the fact that our model produces explicit likelihoods can also be leveraged to test the reliability of a plan by evaluating whether reaching particular waypoints will result in human-like behavior or not. This capability can be quite important for real-world safety-critical applications, such as autonomous driving, and can be used to build a degree of fault tolerance into the system. We designed a classification experiment to evaluate how well our model can recognize safe and unsafe plans. We planned our model to known good waypoints (where the expert actually went) and known bad waypoints (off-road) on 1650 held-out test scenes. We used the planning criterion to classify these as good and bad plans and found that we can detect these bad plans with $97.5\%$ recall and $90.2\%$ precision. This result indicates imitative models could be effective in estimating the reliability of plans.

Appendix F Pothole Experiment Details

Appendix G Baseline Visualizations

See Fig. 15 for a visualization of our baseline methods.

Appendix H Hyperparameters

In order to tune the $\epsilon$ hyperparameter of the unconstrained likelihoods, we undertook the following binary-search procedure. When the prior frequently overwhelmed the posterior, we set $\epsilon\leftarrow 0.2\epsilon$, to yield tighter covariances, and thus more penalty for failing to satisfy the goals. When the posterior frequently overwhelmed the prior, we set $\epsilon\leftarrow 5\epsilon$, to yield looser covariances, and thus less penalty for failing to satisfy the goals. We executed this process three times: once for the “Gaussian Final-State Mixture” experiments (Section 4), once for the “Noise Robustness” Experiments (Section 4.1), and once for the pothole-planning experiments (Section 4.2). Note that for the Constrained-Goal Likelihoods introduced no hyperparameters to tune.