PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Imitative Models for Flexible Inference, Planning, and Control

Nicholas Rhinehart, Rowan McAllister, Sergey Levine

arXiv:1810.06544v4cs.LGcs.AIcs.CVcs.ROstat.ML

TL;DR

Imitation learning struggles to direct behavior toward arbitrary goals, while planning methods require difficult reward engineering to evoke desirable behavior. The paper proposes probabilistic Imitative Models that plan interpretable expert-like trajectories toward flexible goals. In simulated autonomous driving, the method substantially outperforms six imitation-learning approaches and an MBRL approach, while remaining robust to poorly specified goals.

  • Problem

    Imitation learning cannot readily pursue different or general goals, whereas planning-based methods depend on difficult reward engineering to produce desirable behavior.

  • Method

    Imitative Models learn probabilistic predictive models of expert behavior and combine them with test-time goal likelihoods to plan interpretable trajectories.

  • Results

    The method substantially outperformed six imitation-learning approaches and an MBRL approach in a dynamic simulated autonomous-driving task.

  • Takeaways & Limitations

    The approach can incorporate unseen goals, including goal regions and test-time pothole avoidance, without additional training.

  • Takeaways & Limitations

    The autonomous-driving application assumes access to ground-truth past positions and traffic-light signals alongside scene observations.

Abstract

from arXiv · show

Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

1 INTRODUCTION

The paper introduces Imitative Models to combine imitation learning’s desirable behavior with planning-based flexibility for new, complex goals. The method learns expert behavior from demonstrations, plans interpretable trajectories, and handles unseen goals and noisy specifications in autonomous driving.

  • 1 INTRODUCTION: Imitative Models learn a probability density of future expert behavior from demonstrations and use it to plan trajectories for specified tasks.The probabilistic model scores candidate plans by how likely they are to come from the expert.
  • 1 INTRODUCTION: The method produces interpretable multi-step expert-like plans without requiring reward-function crafting.This combines offline imitation learning with planning behavior that can be inspected as trajectories.
  • 1 INTRODUCTION: It incorporates goals not seen during training, including navigating to goal regions and avoiding potholes that appear only at test time.These tasks are illustrated in the method overview and are not demonstrated during training.
  • 1 INTRODUCTION: The method remains robust when goals are specified on the wrong side of the road, navigating toward them while staying on the correct side.This addresses noisy or poorly specified goals rather than requiring exact goal specifications.
  • 1 INTRODUCTION: In dynamic and static CARLA environments, the method substantially outperforms MBRL, a custom imitation-learning method, and five prior CARLA imitation-learning methods.The reported driving performance is learned from expert observations alone.

2 DEEP IMITATIVE MODELS

Deep Imitative Models learn probabilistic expert-trajectory forecasts and repurpose them for goal-directed planning. Test-time likelihoods and costs provide flexible objectives while preserving expert-like, interpretable behavior.

  • Imitative model: The model learns a scene-conditioned probability distribution over expert trajectories, scoring candidate plans by how likely they are to come from the expert.It is trained on expert trajectory data and factorizes the trajectory distribution autoregressively.
  • Imitative planning: Goal-directed planning combines the learned imitation prior q(s|φ) with a test-time goal likelihood p(G|s, φ) through posterior inference.Gradient-based optimization produces plans that satisfy goals while remaining similar to expert behavior.
  • Goal likelihoods: Constraint-based objectives use indicator likelihoods to force final states into points, line segments, or polygonal regions without introducing hyperparameters.The prior strongly penalizes final positions that are not expert-like.
  • Goal likelihoods: Unconstrained objectives use Gaussian final-state, state-sequence, or mixture likelihoods to encourage desired endpoints or sequences while tolerating poorly specified goals.These objectives introduce ϵ but remain useful when no goal state exactly matches observed expert behavior.
  • Costed planning: Test-time cost functions can encode newly observed hazards and combine multiplicatively with goal-seeking likelihoods.The framework can therefore incorporate cost maps such as pothole locations after training.
  • Autonomous driving instantiation: In autonomous driving, the model conditions on past positions, LIDAR-based scene observations, and traffic-light state to forecast future trajectories.The implementation uses 40 trajectory steps at 10 Hz and conditions on three past positions.
  • Autonomous driving instantiation: The complete driving system combines route planning, imitative path planning, and feedback control to follow selected expert-like trajectories.A route planner supplies waypoints, while the imitative planner selects paths and a low-level controller produces actions.

3 RELATED WORK

Prior work contrasts offline imitation learning, which can perform well but typically handles only a small set of directives, with model-based reinforcement learning, which plans for new tasks but requires carefully crafted rewards. Imitative Models combine trajectory-distribution modeling with test-time planning for varied goals.

  • Imitation learning: CARLA behavior-cloning methods condition on goals from a small discrete directive set and share route-generation inputs with this approach.These methods use an A* planner on route nodes to generate waypoints, while differing in how those waypoints are used.
  • Model-based reinforcement learning: MBRL plans using a one-step model of possible dynamics and delegates expert-like behavior to a potentially difficult reward function.The paper devised a CARLA MBRL baseline using the same inputs as its method.
  • Comparison tables: Table 1 compares desirable attributes across approaches, marking desirable properties with checks, undesirable properties with crosses, and implemented methods with daggers.The table provides a qualitative comparison rather than reporting performance values.
  • Comparison tables: Table 2 compares the algorithmic components used by the approaches and marks implemented methods with daggers.Its caption identifies component-level comparison as distinct from the desirable-attribute comparison.
  • Imitative Models: Imitation-learning architectures with planning-like modules differ by modeling distributions over trajectories and planning test-time trajectories that satisfy varied goals.The approach is positioned for offline-learning settings where expert demonstrations are available.

4 EXPERIMENTS

Experiments evaluate Imitative Models in CARLA for interpretable expert-like planning, performance, flexibility, and robustness to degraded or incorrect goals. The method outperformed comparison approaches across reported settings and adapted to test-time objectives such as pothole avoidance.

  • Evaluation design: The CARLA evaluation asks whether offline learning can produce interpretable expert-like plans without reward engineering and achieve state-of-the-art performance.It also tests flexibility to new tasks and robustness to goal-specification noise.
  • Evaluation design: The model was trained on 25 hours of driving in Town01 and evaluated in static and dynamic settings across Town01 and Town02.Test episodes randomly position the vehicle on a road, while goals are constructed from CARLA route outputs.
  • Metrics: The evaluation reports destination success, red-light violations, and the proportion of time spent in the wrong lane or off road.Success is maximized; the other two metrics are minimized.
  • Main results: The method outperformed all other approaches in static and dynamic worlds, under both training and test conditions.Goal Indicator methods also performed well without hyperparameters to tune, while a light-aware waypointer further improved performance.
  • Main results: The planner preferred closer goals when obstructed, stopped, or facing a red light, and farther goals when unobstructed or facing green or no lights.These behaviors were attributed to leveraging the learned imitation distribution in new situations.
  • Robustness: The experiments include a Gaussian Final-State Mixture evaluation under degraded route planning to test flexibility and noise robustness.The study specifically examines decoy waypoints and waypoints placed on the wrong side of the road.
  • Robustness: 48% overall success was achieved on Town01 Dynamic with coarse 8-meter-wide regions that included wrong-lane and off-road space.The method also remained effective when waypoints were systematically placed on the wrong side of the road.
  • Test-time flexibility: Test-time pothole avoidance was evaluated by adding a simulated pothole cost map and comparing plans with and without that cost.The experiment tests behavior not demonstrated during training.

5 DISCUSSION

The discussion presents Imitative Models as probabilistic predictive models that combine offline imitation learning with test-time, goal-directed planning. It develops flexible goal-set objectives and optimization procedures that preserve expert-like behavior while supporting new objectives.

  • Contribution: Imitative Models plan interpretable expert-like trajectories toward new goals, combining imitation-model learning with MBRL-like inference.The approach is intended to avoid reward engineering and costly online data collection.
  • Discussion: The method incorporates flexible goal objectives without additional training and substantially outperformed six imitation-learning approaches plus an MBRL approach in dynamic simulated driving.The discussion also reports robustness to goals on the wrong side of the road and identifies offline expert demonstrations, flexibility, and safety as the intended application setting.
  • Control: Receding-horizon control repeatedly plans with the imitative model, executes a finite control horizon, and replans until reaching the destination.The implementation uses a route planner, imitative planner, PID controller, and environment interaction loop.
  • Optimization: Gradient-based optimization can be performed in latent trajectory space because the trajectory decoder is differentiable.The latent space may be better numerically conditioned because it is trained from standard-normal latent samples.
  • Optimization: Set-constrained objectives exploit closed-form terminal optimization for suitable goal sets, reducing planning to unconstrained optimization over preceding states.The same gradient-descent approach can then optimize the resulting objective.
  • Planning formulation: The model represents goal-directed planning as a posterior combining an imitation prior over trajectories with a goal likelihood.The imitation prior captures expert-like behavior, while the goal likelihood communicates what the agent should achieve without specifying how.
  • Goal objectives: Point goals, finite goal sets, line segments, piecewise-linear paths, and polygon regions provide progressively flexible terminal objectives.For polygon goals, the optimal terminal state is the Gaussian mode when the mode lies inside the polygon; otherwise it lies on an edge.

B.3 CONSTRUCTING GOAL SETS

The driving implementation constructs point, line-segment, and polygon goal sets from CARLA waypoints and uses corresponding final-state indicators. The resulting plans terminate at selected points, segments, or regions while retaining lane-following behavior under broad regions.

  • Goal-set construction: CARLA waypoints define point, line-segment, and polygon goal sets corresponding to three final-state indicators.The point-goal formulation constrains the terminal state to one of the waypoint locations and includes the vehicle’s current position to permit stopping.
  • Evaluation protocol: The evaluation protocol asks the vehicle to drive to the furthest road location from its initial position.The paper characterizes this as more difficult than prior protocols without start-to-goal distance guarantees.
  • Point goals: Final-State Indicator plans end at one of the provided goal locations.Orange diamonds mark goal locations and red circles mark the selected plan.
  • Line-segment goals: Line Segment Final State Indicator plans end along one of the supplied line segments.Orange diamonds indicate segment endpoints and red circles indicate the chosen plan.
  • Region goals: Region Final State Indicator plans end inside the supplied polygonal region.The orange polygon marks the region and red circles mark the chosen plan.
  • Region goals: Even with a wider goal region, the vehicle remains in its lane because of the imitation prior.The paper reports that coarse wide regions still provide useful guidance.

C ARCHITECTURE AND TRAINING DETAILS

The architecture implements a probabilistic trajectory model whose multimodal prior supports alternative driving behaviors and whose planning criterion balances expert-likelihood with goal likelihood.

  • Architecture: The model architecture combines map interpolation, feature concatenation, recurrent and multilayer components, a matrix exponential, and a Verlet step.The listed modules include MapFeat, JointFeat, FutureRNN, FutureMLP, MatrixExp, and VerletStep.
  • Prior behavior: The prior q(S|φ) is multimodal, producing samples that choose different driving directions such as left or right, or forward or right.Figure 13 visualizes these alternative prior samples.
  • Planning criterion: Across approximately 8·10^3 Town01 Dynamic rounds, mean log q(s∗|φ) was approximately 104 while mean log p(G|s∗, φ) was −4.The prior value mostly dominates final-plan values, while the Gaussian Final-State Goal Mixture likelihood has moderate influence.

C.2 DATASET

The dataset was collected by running CARLA’s expert autopilot in dynamic Town01 environments with randomized train, validation, and test splits. The same trajectories and LIDAR observations were used for the imitation-learning baseline and the main method.

  • Expert data collection: The researchers ran the Town01 autopilot for over 900 episodes of 100 seconds with 100 other vehicles, recording every vehicle’s trajectory and the autopilot’s LIDAR observation.Episodes were randomized into training, validation, and test sets, and the collected dataset was prepared for public release upon publication.
  • Baseline training: The conditional imitation-learning baseline predicts PID-controller setpoints from the same LIDAR observations and trajectories used by the main method.It also observes traffic-light information and uses nearly the original CIL architecture.
  • Baseline behavior: The baseline was effective for stable control on straightaways but encountered problems at corners.

D.2 MODEL-BASED REINFORCEMENT LEARNING:

The model-based reinforcement learning baseline learns forward dynamics from expert data and plans low-cost trajectories toward waypoints. Static planning handles obstacles, while dynamic planning adds temporal state transitions but is limited by constant-velocity prediction accuracy.

  • Static-world: The baseline learns a forward dynamics model from observed expert vehicle actions using an MLP with two 100-unit hidden layers.The model predicts s_t+1 from recent states and the current action.
  • Static-world: The planner selects the lowest-cost trajectory that ends near the specified goal.Its cost combines terminal distance to the goal with position-dependent obstacle penalties.
  • Static-world: 20-step breadth-first search uses five steering angles, constant throttle, and retains the 50 nodes closest to the waypoint.The search can plan around perceived obstacles and avoid getting stuck.
  • Dynamic-world: Dynamic-world planning adds a discrete temporal dimension and updates moving-object observations with a constant-velocity model.Static obstacles remain unchanged, while colliding LIDAR points are replaced at each time step.
  • Dynamic-world: The main reported dynamic-world failure mode involves inaccuracies in constant-velocity prediction.

E ROBUSTNESS EXPERIMENTS DETAILS

The robustness experiments perturb waypoint specifications and test decoy or entirely misplaced waypoints. The planner generally favors waypoints consistent with expert driving distributions, while valid off-route decoys can temporarily confuse route selection.

  • Decoy waypoints: The decoy-waypoint experiment perturbs each waypoint with Gaussian noise N(0, σ = 8m).The perturbation standard deviation is 8 meters.
  • Decoy waypoints: A failure mode occurs when decoy waypoints lie on a valid off-route path at an intersection, temporarily confusing the planner about the best route.
  • Wrong-side goals: Figure 14 compares planning with one or two decoy waypoints against planning when all waypoints are on the wrong side of the road.The planner prefers waypoints within the expert-behavior distribution, such as road locations at reasonable distances.

E.2 PLAN RELIABILITY ESTIMATION

The imitative model’s likelihood-based planning criterion can distinguish reliable from unreliable waypoint plans, including off-road plans. Across 1,650 test scenes, it achieved high recall and precision under a simple validation-based threshold.

  • Plan reliability classification: 97.5% recall and 90.2% precision were achieved when classifying waypoint plans as reliable or unreliable across 1,650 test scenes.The classifier used the planning criterion to distinguish known good and bad waypoints.
  • Waypoint reliability: Expert-arrival waypoints were 89.4% reliable, compared with 73.8% for route-based waypoints and 2.5% for waypoints shifted 2.5m off-road.These reliability values show a strong preference for valid waypoints.
  • Threshold calibration: The reliability threshold was calibrated as the planning criterion’s validation mean minus one standard deviation.The authors report that this simple calibration produced reasonably good performance.

F POTHOLE EXPERIMENT DETAILS

The pothole experiment inserted localized hazards into the driving cost map near waypoints, while Figure 15 visualizes the baseline methods used for comparison. An additional binary-search procedure tuned the unconstrained-likelihoods hyperparameter.

  • Pothole simulation: Potholes were simulated by inserting randomly sampled cost-map hazards near each waypoint, then dilating and blurring the resulting cost map.The hazard offsets were centered 15m before each waypoint on the right side of the lane.
  • Baseline comparison: The paper directs readers to Figure 15 for a visualization of the baseline methods.The figure provides the visual reference for the baseline comparison.
  • Baseline comparison: Figure 15 compares an imitation-learning baseline with a model-based reinforcement-learning baseline.The figure shows past agent positions and baseline-specific goal, setpoint, reachability, LIDAR, and obstacle-map elements.
  • Hyperparameter tuning: The unconstrained-likelihoods ϵ hyperparameter was tuned by repeatedly shrinking or expanding it according to whether the prior or posterior dominated.The procedure used ϵ ← 0.2ϵ for tighter covariances and ϵ ← 5ϵ for looser covariances, and was executed three times.
Loading 1810.06544v4…