PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Felipe Codevilla, Eder Santana, Antonio M. López, Adrien Gaidon

arXiv:1904.08980v1cs.CVcs.AI

TL;DR

Behavior cloning has not yet scaled reliably to the full range of complex driving behaviors, motivating a systematic benchmark of its capabilities and limitations. The paper evaluates large-scale conditional imitation learning in simulated driving and finds strong generalization, including to unseen environments, alongside failures involving dynamic objects, causal confusion, dataset bias, and training variance. These limitations constrain its progression toward real-world driving.

  • Problem

    Behavior cloning has not been shown to scale reliably from simple end-to-end visuomotor policies to the full spectrum of driving behaviors, especially interactions with multiple dynamic objects.

  • Method

    The paper introduces the NoCrash benchmark and evaluates conditional imitation learning with large-scale offline demonstrations in the CARLA simulator.

  • Results

    Behavior cloning substantially improves generalization performance over prior approaches, including in unseen environments, while handling complex driving maneuvers without explicit programming.

  • Takeaways & Limitations

    Behavior cloning can achieve strong state-of-the-art driving performance from large-scale off-policy demonstrations, but its scalability depends on addressing observed generalization and training issues.

  • Takeaways & Limitations

    Behavior cloning remains limited by dataset bias, causal confusion, dynamic-object generalization failures, and sensitivity to initialization and sampling order.

Abstract

from arXiv · show

Driving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, including in unseen environments, executing complex lateral and longitudinal maneuvers without these reactions being explicitly programmed. However, we confirm well-known limitations (due to dataset bias and overfitting), new generalization issues (due to dynamic objects and the lack of a causal model), and training instability requiring further research before behavior cloning can graduate to real-world driving. The code of the studied behavior cloning approaches can be found at https://github.com/felipecode/coiltraine .

1. Introduction

Behavior cloning offers a simple end-to-end alternative for learning driving perception and control from demonstrations, but its scalability remains limited. The paper introduces a benchmark and large-scale analysis showing strong generalization alongside failures in complex conditions and unstable training.

  • Behavior cloning jointly learns perception and control from human demonstrations without explicitly programming driving sub-tasks.End-to-end networks may implicitly learn task-relevant sub-skills from data.
  • Scaling end-to-end driving to the full spectrum of behaviors, including reactions to multiple dynamic objects, remains unresolved.
  • The NoCrash benchmark enables reproducible large-scale evaluation of behavior cloning in complex driving conditions using CARLA.The evaluation covers over 80 hours of driving under several conditions.
  • A Conditional Imitation Learning baseline improves upon modular, affordance-based, and reinforcement-learning approaches in training and unseen environments.
  • Behavior cloning shows strong performance but still faces dataset bias, dynamic-object generalization failures, and training variability that hinder real-world deployment.The reported limitations include diminishing returns, degradation on unseen environments, and sensitivity to initialization and sample order.

2. Related Work

Related work frames behavior cloning as a scalable supervised approach, while emphasizing dataset bias and high variance as major obstacles to generalization. The paper evaluates these issues in realistic, safety-critical driving benchmarks while using simulation for reproducibility.

  • Behavior cloning learns sensorimotor policies from offline sensory-observation and expert-action pairs, with prior work extending beyond lane following and driving.
  • Dataset bias can dominate imitation-learning objectives because common simple behaviors outweigh rare complex reactions in naturalistic driving data.Insufficient growth in dataset diversity can cause performance to degrade as more demonstrations are collected.
  • High variance links behavior-cloning performance to initialization and sampling order, raising reproducibility concerns similar to those in reinforcement learning.
  • Driving research uses online, photorealistic simulation benchmarks because strong offline dataset performance does not necessarily translate into effective driving policies.CARLA is presented as an emerging standard platform, while simulation-to-real transfer remains outside this paper’s scope.

3. A Strong Baseline for Behavior Cloning

The paper develops a robustified, strictly off-policy Conditional Imitation Learning baseline and examines behavior cloning’s limitations in complex driving. CILRS combines architectural and auxiliary-task changes, while the analysis identifies dataset bias, causal confusion, and training variance as persistent challenges.

  • Conditional Imitation Learning: Behavior cloning learns sensorimotor policies from offline observation-action demonstrations, conditioned on high-level navigation commands.The policy is trained to imitate expert low-level controls from sensor observations and commands.
  • Limitations: Naturalistic driving datasets can become dominated by simple behaviors, so additional demonstrations may increase data volume without sufficiently increasing behavioral diversity.The paper reports that this bias can degrade performance as more data is collected and confirms the phenomenon with NoCrash.
  • Limitations: Causal confusion links spurious training correlations to failures such as excessive stopping and difficult restarting when the ego vehicle is stopped.The paper attributes this inertia problem to the correlation between low speed and no acceleration in demonstrations.
  • Limitations: Training exhibits high variance because sequential demonstration snapshots violate the i.i.d. assumption underlying stochastic gradient descent.The reported sensitivity arises from initialization and sample order, and is measured through online benchmark scores.
  • CILRS baseline: CILRS uses a deeper ResNet34 perception backbone and remains strictly off-policy while targeting behavior-cloning limitations.The model is presented as a robustified CIL baseline designed to improve upon prior work.
  • CILRS baseline: A shared perception representation jointly predicts vehicle controls and ego speed, using speed prediction as regularization against the inertia problem.The auxiliary speed branch encourages speed-related features in the learned representation, although instances of inertia remain.

4. Evaluation

The evaluation introduces NoCrash, a larger-scale CARLA benchmark designed to test reactions to traffic changes and dynamic agents. It expands task difficulty from empty to dense traffic and uses collision-triggered episode failure to make success more safety-representative.

  • CARLA environment: The benchmark uses CARLA 0.8.4, whose two suburban towns provide 2.9 km and approximately 1.4 km of drivable roads.The environment includes photo-realistic roads, pedestrians, and cars with different cruising speeds.
  • Evaluation metric: Unlike the original CARLA benchmark’s separate success and infraction measures, NoCrash fails an episode after any collision exceeding a fixed magnitude.This provides a lower bound and guarantee of acceptable behavior based on measured success percentage, while returning the agent to a safe start in a new episode.
  • NoCrash Benchmark: NoCrash extends CARLA evaluation toward complex events caused by changing traffic conditions and dynamic agents.Its tasks and metrics are designed to measure reaction patterns expected of good urban drivers.
  • NoCrash Benchmark: NoCrash defines three 25-episode tasks with increasing difficulty: Empty Town, Regular Traffic, and Dense Traffic.The tasks progress from no dynamic objects to moderate traffic and then heavy traffic with many pedestrians.
  • NoCrash Benchmark: The benchmark tests six weather conditions and two towns, reserving two weather conditions and one town for testing.This setup evaluates generalization to both unseen weather and an unseen environment.

5. Experiments

The experiments introduce a large-scale evaluation protocol for behavior cloning and show strong benchmark performance alongside substantial failures under challenging generalization conditions. Results expose limitations from traffic complexity, dataset bias, causal confusion, and training randomness.

  • Protocol: The study trains behavior-cloning controllers on CARLA100, a 100-hour dataset collected from more than 400 hours of simulated driving.The data were collected in a single CARLA town using an expert driving agent and substantial computational resources.
  • Models and comparisons: CILRS adds a ResNet architecture and speed prediction to conditional imitation learning, while CILR removes the speed-prediction branch.The comparison includes original CIL and three state-of-the-art approaches, with evaluation on CARLA and NoCrash benchmarks.
  • Benchmark results: All methods perform significantly worse on NoCrash than on the original CARLA benchmark, especially when generalizing to new conditions.This supports NoCrash as a more challenging benchmark for studying driving-policy limitations.
  • Benchmark results: +9% and +26% are CILRS improvements over the state of the art on CARLA “Nav. Dynamic” in training and new conditions, respectively.On NoCrash Regular traffic, the corresponding improvements are +10% and +24%.
  • Dynamic-object generalization: −55% and −66% are the NoCrash training and new-condition drops when changing from Empty to Dense traffic.Empty-town performance drops only −7% when changing to a new environment and weather, highlighting the difficulty of many dynamic agents.
  • Dataset bias: The best results in most scenarios use 10 hours rather than the full 100 hours of demonstrations, particularly for Dense Traffic and novel conditions.The authors attribute this pattern to overfitting caused by insufficient diversity, limited spatial extent, and limited visual variety.
  • Causal confusion: The inertia problem increases with training-data quantity, while speed prediction substantially improves success rates but does not eliminate the failure mode.Inertia failures involve remaining still without intending to use the throttle for at least 8 seconds before timeout.
  • Training variance: Training randomness can change success rates by up to 42% on tasks with dynamic objects, with initialization and sampling order affecting the learned policy.ImageNet initialization tends to reduce variability, while different seeds can produce collision-prone or overly stopping policies.

6. Conclusion

The paper finds that large-scale behavior cloning can substantially improve generalization, but important challenges remain before deployment in real-world driving. Dynamic multi-agent scenes, dataset bias, causal confusion, and training variance limit reliability.

  • CILRS combines a deeper residual architecture, speed prediction, and regularization to improve generalization over state-of-the-art approaches.The comparison includes mediated perception approaches with additional supervision.
  • Dynamic objects directly hurt policy learning because multi-agent dynamics are not directly captured.
  • Driving-specific dataset biases cause diminishing returns or worse performance as demonstrations increase, particularly through causal confusion such as the inertia problem.
  • Initialization and sampling order create large performance variance, making multiple training runs important for identifying strong policies.
  • The released CARLA100 dataset, NoCrash benchmark, and CILRS code are intended to stimulate further research.
  • Training uses RGB images, ego-vehicle speed, high-level turn intentions, and ego-vehicle controls as model inputs or targets.

A.2. Content

CARLA100 consists of goal-directed simulated driving episodes with extensive ego-vehicle, sensor, world, and non-player-agent records. Data collection uses a privileged automated expert, controlled environments, and injected steering perturbations.

  • The dataset samples 50–100 pedestrians and 30–70 vehicles, with spawned-agent seeds and weather conditions recorded per episode.
  • Each 100 ms simulation step stores sensor data as PNG images and measurement data as JSON files.
  • Recorded sensors include RGB, depth, and semantic-segmentation cameras positioned centrally and 30 degrees to either side.
  • Measurements include ego state, intentions, high-level commands, waypoints, controls, noise signals, and non-player-agent attributes.
  • Steering noise is injected in 20% of collection time as a triangular impulse, with p_perturb = 0.1 per second, 0.5–2 second duration, and intensity 0.15.

A.4. NoCrash Benchmark

NoCrash evaluates goal-directed driving across Empty, Regular, and Cluttered tasks under training and multiple environmental generalization conditions. Success requires reaching the goal within a time limit without collisions.

  • NoCrash contains three tasks—Empty, Regular, and Cluttered—with 25 goal-directed episodes per task.
  • An episode succeeds only when the agent reaches its goal within the time limit without colliding with static or dynamic objects.
  • The benchmark evaluates four conditions: Training, New weather, New Town, and New Town & Weather.
  • New weather keeps the training city but introduces two unseen weather conditions, while New Town tests in Town02.

B.1. Architecture

The experiments use a standard architecture whose perception module processes cropped and resized raw images. Alternative perception backbones were also evaluated.

  • The experiments compare the standard architecture with ResNet 18, ResNet 50, and the architecture proposed in [10] for perception.
  • Raw 800 × 600 images are cropped by 125 pixels at the top and 90 pixels at the bottom, then resized to 200 × 88 pixels.

C. Additional Results

The proposed methods outperform prior approaches on CARLA benchmark comparisons, while architecture and initialization choices materially affect performance. Deeper ResNet models improve over the smaller convolutional architecture, but excessive depth and removing ImageNet pre-training can reduce results.

  • Comparison: The proposed methods outperform previously proposed methods in CARLA benchmark comparisons, although New Town is harder for some models than New Weather & Town.The paper notes that this task-difficulty discrepancy had also appeared in prior literature.
  • ImageNet Initialization: Removing ImageNet initialization changes success rates substantially, and even 100 hours of demonstrations remains below less-data models with ImageNet pre-training on dense-traffic tasks.Without ImageNet pre-training, the highest success rates occur with 100 hours of demonstrations, but they do not match the best pretrained results.

C.1. Reacting to Traffic Lights

Behavior cloning models learn traffic-light responses, including stopping at red lights, but their reliability varies across architectures, training conditions, and generalization settings. The benchmark also exposes inertia-related episode failures and substantial sensitivity to data, initialization, and model capacity.

  • Traffic-light behavior: The original CIL model provides a lower bound because it was trained without demonstrations of stopping at red traffic lights.The new 8-convolution model reacts to traffic lights more often but performs poorly, while some trained models develop more effective policies.
  • Traffic-light behavior: The best model records 47% traffic-light violations, while its 100-hour version records 27% violations in the Empty Town task.Lower percentages indicate fewer red-light crossings; the 100-hour result is described as more expressive but generalization still has room for improvement.
  • Failure analysis: The inertia problem becomes more prominent with more training data, based on episode percentages averaged with standard deviations over four training runs.The comparison includes models trained with different data amounts and models without ImageNet pre-training.
  • Training sensitivity: Increasing data generally improves overall results but not significantly, and changing training hours from 50 to 100 can worsen performance under New Weather & Town with Dense Traffic.This result illustrates sensitivity to training conditions rather than a uniformly improving scaling trend.
  • Failure analysis: Table 8 reports episode termination causes across methods, tasks, and weather conditions, averaged over three benchmark runs.The columns for each method, task, and condition sum to 1, with worse causes and best success highlighted separately.
Loading 1904.08980v1…