PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Aware Active Learning Enables Continual Improvement in Autonomous Driving

Dong Hu, Chao Huang, Carman K. M. Lee, Dimitrios Kanoulas

arXiv:2608.29772v1cs.RO

TL;DR

Learning-based driving policies often lack mechanisms to recognize when their competence is insufficient and to adapt safely after deployment. SAGE uses world-model signals to trigger expert intervention and targeted learning, improving robustness and reducing safety violations across diverse shifted and hazardous settings while maintaining task performance comparable to strong baselines.

  • Problem

    Deployed driving policies may fail under rare distribution shifts because they lack explicit awareness of competence limits and are commonly updated through separated offline retraining cycles.

    Source: §I · p. 1

  • Method

    SAGE combines predictive world-model self-monitoring, curiosity-calibrated fear gating, selective expert takeover, and safety-regularized learning from intervention trajectories.

    Source: §I · pp. 1–2, F. Active Expert Guidance Mechanism · p. 6

  • Results

    Across route shifts, adversarial Waymo scenarios, occlusion hazards, and real-world navigation, SAGE improves robustness and safety while maintaining comparable task performance; it reduces adversarial collision rates versus SAC by 76.8% in training and 69.9% in testing.

    Source: 3) Results · pp. 8, 10, 12, §IV · p. 14

  • Takeaways & Limitations

    Estimating competence limits and requesting guidance enables selective learning from rare high-value events during post-training adaptation.

    Source: §IV · p. 14

    Fear may be miscalibrated under severe perception degradation, sensor corruption, or highly atypical scenarios, causing delayed or unnecessary interventions.

    Source: §IV · p. 14

Abstract

from arXiv · show

Learning-based autonomous driving (AD) systems can perform reliably in familiar conditions, yet rare distribution shifts and long-tail events remain a major source of abrupt failure. A central limitation is that most agents learn primarily from passive experience and lack mechanisms to estimate when their competence is insufficient, seek timely assistance, and convert safety-critical encounters into targeted improvement. Here we present self-aware guided exploration (SAGE), an active learning framework for post-training adaptation in AD. SAGE learns a predictive world model that generates two online intrinsic signals: fear, which estimates short-horizon predictive risk and model uncertainty, and curiosity, which measures novelty through prediction error. Curiosity adaptively calibrates the intervention threshold for fear, allowing the agent to regulate risk in a context-dependent manner. When predicted fear exceeds this adaptive threshold, the agent transfers control to an expert or fallback policy and uses the resulting takeover trajectories for focused imitation learning. In parallel, fear is integrated into policy optimization and evaluation as a safety-oriented constraint to reduce performance regressions during adaptation. We evaluate SAGE in simulated route-transfer tasks, Waymo-based logged driving scenarios, CARLA occlusion hazards, and real-world mobile robot navigation tests. Across these settings, SAGE improves robustness in novel and safety-critical scenarios, reduces safety violations, and maintains task performance comparable to strong baseline policies. These results suggest that agents can improve after initial training by estimating the limits of their competence, requesting guidance when needed, and learning selectively from rare high-value events.

I. INTRODUCTION

SAGE addresses the limited ability of deployed driving policies to recognize competence gaps, request targeted supervision, and adapt safely after training. It combines world-model self-monitoring, selective expert intervention, and safety-preserving continual policy improvement for rare distribution shifts.

  • Rare, safety-critical distribution shifts remain difficult because most driving pipelines rely on passive data collection, offline failure analysis, and periodic retraining.Source: §I · p. 1
  • SAGE uses a learned world model to estimate novelty, predictive risk, and model uncertainty through curiosity and fear signals.Curiosity captures novelty through prediction error, while fear aggregates short-horizon predicted safety cost and epistemic uncertainty.Source: §I · pp. 1–2
  • Curiosity adaptively calibrates the fear threshold, determining when autonomous execution should continue and when control should transfer to an expert policy.The resulting takeover trajectories provide targeted demonstrations for subsequent policy improvement.Source: §I · p. 2, A. Overview · p. 3
  • SAGE couples selective expert intervention with fear-regularized reinforcement learning and value-weighted imitation from takeover trajectories.The framework treats interventions as both protective control actions and learning data.Source: §I · p. 2
  • The framework is organized around world-model self-awareness, selective expert intervention, and continual constrained adaptation after deployment.Source: §I · p. 3

C. Prospective Fear from World Model

SAGE derives prospective fear from an ensemble world model that predicts near-future safety-relevant outcomes and uncertainty. The resulting signal supports short-horizon reasoning about danger beyond immediate perception.

  • Fear combines predicted safety cost and epistemic uncertainty from an ensemble-based world model.The model predicts safety-relevant quantities from the current state-action pair.Source: C. Prospective Fear from World Model · p. 3
  • The world model predicts next states and safety costs such as collisions or traffic-rule violations for candidate actions.Source: C. Prospective Fear from World Model · p. 3
  • Ensemble predictions are aggregated into mean next-state and cost estimates together with uncertainty, where larger uncertainty indicates lower predictive confidence.Source: C. Prospective Fear from World Model · pp. 3–4
  • Fear is computed as a weighted combination of predicted cost and uncertainty, with β controlling their relative contributions.The predicted cost represents the probability of a safety violation, and uncertainty is normalized to a comparable scale.Source: C. Prospective Fear from World Model · p. 4
  • Short-horizon rollouts extend fear estimation across m predicted steps, allowing the agent to reason about near-future danger beyond immediate perception.Model-generated transitions are stored in a virtual replay buffer.Source: C. Prospective Fear from World Model · p. 4

D. Curiosity-modulated fear tolerance

SAGE uses curiosity to adjust fear tolerance, making intervention more likely in poorly modeled situations. Fear is also incorporated into policy evaluation and constrained optimization to discourage trajectories leading to safety violations.

2) Fear-regularized policy improvement:

Fear-regularized policy improvement maximizes task value while penalizing excessive predicted fear. A dual update adapts the penalty strength to keep average fear near a target during training.

F. Active Expert Guidance Mechanism

The active expert guidance mechanism transfers control when predicted fear exceeds a curiosity-regulated threshold and stores interventions as targeted supervision. Expert demonstrations are weighted by estimated utility rather than imitated uniformly.

A. Baselines

The evaluation compares SAGE-RL with standard RL, FNI-RL, interactive imitation, safe model-based, offline diffusion, human-guided, and classical planning baselines across continual-adaptation tasks.

Baseline coverage

  • The broader baseline suite includes model-free, value-based, interactive imitation, safe model-based, offline diffusion, human-guided, and classical navigation methods.The real-world UGV comparisons additionally include Diffusion-BC, Diffusion-QL, Multi-Hug RL, and TEB.Source: A. Baselines · pp. 6–7

RL baselines

  • SAC, TD3, and D3QN provide general RL comparisons, with SAC used by default and D3QN reserved for discrete-action long-tail experiments.D3QN combines double Q-learning with a dueling value architecture.Source: A. Baselines · pp. 6–7

Continual adaptation

  • SAGE-RL is compared with standard RL and FNI-RL in route-level distribution-shift experiments.Route 1 trains in episodes 1–1000, followed by Route 2 in episodes 1001–1500.Source: 1) Task Description · pp. 7–8
  • The route-level setup observes local traffic, traffic-light state, and navigation cues while outputting continuous longitudinal acceleration.The final observation is a 29-dimensional ego-centric state, and lane changes are handled by simulator routing.Source: 2) Learning Setup · p. 8

3) Results:

Across route transfer and Waymo adversarial training, SAGE-RL reduces degradation under distribution shift, uses interventions when risk rises, and improves robustness after learning from those interventions.

Route transfer

  • SAGE-RL shows lower degradation and faster recovery after switching from Route 1 to the unfamiliar Route 2.It ultimately reaches higher success while maintaining lower collision and red-light violation rates than the comparison methods.Source: 3) Results · p. 8
  • Fear and takeover frequency rise immediately after the route switch, then decline as the policy adapts from intervention data.Curiosity remains active as an additional novelty signal for adaptive intervention.Source: 3) Results · p. 8

Waymo adversarial training

  • Waymo training begins from 500 logged replay scenarios containing the ego vehicle and surrounding traffic agents.Surrounding vehicles initially follow their recorded behaviors.Source: 1) Task Description · p. 8

Intervention mechanism

  • Representative cases show fear exceeding a curiosity-regulated threshold, triggering expert takeover during safety-critical interactions.The cited passage describes this intervention mechanism in training-time Waymo cases.Source: Figure 5 · p. 9

Waymo adversarial training

  • Risk-conditioned resampling creates adversarial encounters with smaller intervehicle distances and lower time-to-collision than replay scenarios.The induced interactions include small gaps and low time-to-collision requiring braking or yielding.Source: 1) Task Description · p. 9

2) Learning Setup:

The Waymo learning setup combines onboard-style ego-centric observations, continuous vehicle controls, safety-aware rewards, and intervention-triggered adaptation evaluated under normal and adversarial scenarios.

State and action

  • The policy observes ego kinematics, route guidance, and a 72-beam LiDAR-style encoding with 50 m range.The resulting ego-centric vector is fully observable from onboard sensing.Source: 2) Learning Setup · pp. 9–10

Evaluation setup

  • The Waymo evaluation table reports autonomous-driving results under both normal and adversarial scenarios.The table caption identifies the comparison as an evaluation of autonomous driving tasks across these two conditions.Source: Table I · p. 10
  • Results are averaged over five random seeds using 400 training scenarios and 100 testing scenarios.The notation defines success rate, collision rate, average speed, and average acceleration.Source: 2) Learning Setup · p. 10

State and action

  • Actions are normalized two-dimensional controls mapped to physical steering, acceleration, and braking commands.The control vector is bounded in [-1, 1]^2 before simulator-limit mapping.Source: 2) Learning Setup · p. 10

Adaptation mechanism

  • Fear-triggered interventions are followed by autonomous emergency handling after updating, including timely braking and geometry-consistent lane changes.The examples are presented as evidence that intervention episodes were internalized rather than required as persistent assistance.Source: 3) Results · p. 10

Results

  • SAGE-RL reduces Waymo collision rate relative to SAC by 76.8% on adversarial training data and 69.9% on adversarial testing data.It also improves success rate over SAC across normal and adversarial settings while producing lower acceleration.Source: 3) Results · p. 10

D. Long-Tail Safety-Critical Experiments

CARLA experiments test SAGE-RL on an occluded pedestrian hazard, while the validation setup also includes simulated and real-world UGV navigation environments.

Task design

  • The CARLA task hides a crossing pedestrian behind a parked truck until the pedestrian enters the ego vehicle’s potential path.The resulting blindzone hazard requires timely braking or evasive action under partial observability.Source: 1) Task Description · p. 10

Training behavior

  • SAGE-RL achieves steadier reward improvement, whereas D3QN plateaus and fluctuates during training.The baseline continues to show collisions and recurrent safety violations after extended training.Source: 3) Results · p. 11

Intervention and adaptation

  • Fear-triggered expert intervention supplies corrective guidance during early unreliable encounters and supports later internalization of safer behavior.The policy gradually refines the expert-guided behavior into a more efficient and calibrated strategy.Source: 3) Results · pp. 11–12

Safety and efficiency

  • SAGE-RL’s time-to-collision increases in later training, while D3QN remains consistently lower after declining and partially recovering.SAGE-RL also transitions from conservative behavior toward stable operation with reduced acceleration variability.Source: 3) Results · p. 12

UGV validation

  • The experimental program includes Gazebo simulation, a physical UGV, static-obstacle tasks, and dynamic-obstacle interactions.The setup covers narrow passages, detours, repeated lateral crossings, and frontal approach with diagonal cut-ins.Source: Figure 7 · p. 12

E. Field Tests

The field tests evaluate navigation under static and dynamic obstacle configurations using a UGV platform and onboard LiDAR-based perception, planning, and control.

Platform and sensing

  • The UGV uses 270° LiDAR point clouds as its primary perception input and obtains robot pose from the LiDAR’s built-in IMU.The simulation replica matches the physical map scale, obstacle layout, and moving-obstacle pattern.Source: 1) Task Description · p. 12

Validation scenarios

  • Four real-world scenarios test narrow-passage traversal, static-obstacle detouring, lateral dynamic obstacles, and frontal dynamic obstacles.The scenarios cover constrained passages, blocked routes, and moving-obstacle interactions.Source: 1) Task Description · p. 12

State representation

  • The observation combines 54-sector LiDAR ranges with normalized goal distance, heading error, and previous control commands, yielding a 58-dimensional state.LiDAR distances are clipped and normalized by 10 m.Source: 2) Learning Setup · p. 12

Action space

Reward and cost

  • The objective combines progress toward the goal, motion smoothness, task completion, collision penalties, and a dense safety cost based on the minimum front-laser range.The reward structure jointly represents task progress, control smoothness, and safety-related events.Source: 2) Learning Setup · p. 12

3) Results:

The results show robust navigation and continual behavioral refinement, with successful real-world performance and progressively safer responses after expert-supervised adaptation.

Quantitative results

  • 100% success rate is achieved in all four real-world indoor scenarios, outperforming learning-based baselines in navigation reliability.The comparison includes SAC, TD3, Diffusion-BC, Diffusion-QL, and Multi-Hug RL.Source: 3) Results · p. 12
  • The proposed method maintains competitive motion quality, with relatively low path curvature and lateral acceleration while keeping speed and path length competitive.Source: 3) Results · p. 13

Continual adaptation

  • In dynamic scenario 3, fear exceeding the intervention threshold triggers expert takeover, after which selective imitation gradually produces safer and smoother autonomous behavior.Further fine-tuning can yield shorter detours than the conservative expert response.Source: 3) Results · p. 13
  • Across nine deployment-fine-tuning cycles, trajectories become progressively refined as the adapted policy improves beyond abrupt expert stopping responses.Source: 3) Results · p. 13

Cross-setting pattern

  • Across route shifts, adversarial traffic, occluded pedestrian hazards, and UGV navigation, elevated fear triggers intervention while stabilization reduces intervention frequency over adaptation.Source: §IV · p. 14

Mechanism and scope

  • The framework links self-monitoring, active expert intervention, and targeted policy improvement, concentrating supervision on safety-critical and information-rich moments.This design is intended to improve safety during learning and increase the relevance of collected experience for policy updates.Source: §IV · p. 14

Limitations

  • Fear may be miscalibrated under severe perception degradation, sensor corruption, or highly atypical scenarios, causing delayed or unnecessary interventions.Expert takeover also adds supervision cost and may make early training more conservative.Source: §IV · p. 14

Related papers

Loading 2608.29772v1…