PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Aware Active Learning Enables Continual Improvement in Autonomous Driving

Dong Hu, Chao Huang, Carman K. M. Lee, Dimitrios Kanoulas

arXiv:2608.29772v1cs.RO

TL;DR

Learning-based driving policies often lack mechanisms to recognize when their competence is insufficient and to adapt safely after deployment. SAGE uses world-model signals to trigger expert intervention and targeted learning, improving robustness and reducing safety violations across diverse shifted and hazardous settings while maintaining task performance comparable to strong baselines.

  • Problem

    Deployed driving policies may fail under rare distribution shifts because they lack explicit awareness of competence limits and are commonly updated through separated offline retraining cycles.

  • Method

    SAGE combines predictive world-model self-monitoring, curiosity-calibrated fear gating, selective expert takeover, and safety-regularized learning from intervention trajectories.

  • Results

    Across route shifts, adversarial Waymo scenarios, occlusion hazards, and real-world navigation, SAGE improves robustness and safety while maintaining comparable task performance; it reduces adversarial collision rates versus SAC by 76.8% in training and 69.9% in testing.

  • Takeaways & Limitations

    Estimating competence limits and requesting guidance enables selective learning from rare high-value events during post-training adaptation.

  • Takeaways & Limitations

    Fear may be miscalibrated under severe perception degradation, sensor corruption, or highly atypical scenarios, causing delayed or unnecessary interventions.

Abstract

from arXiv · show

Learning-based autonomous driving (AD) systems can perform reliably in familiar conditions, yet rare distribution shifts and long-tail events remain a major source of abrupt failure. A central limitation is that most agents learn primarily from passive experience and lack mechanisms to estimate when their competence is insufficient, seek timely assistance, and convert safety-critical encounters into targeted improvement. Here we present self-aware guided exploration (SAGE), an active learning framework for post-training adaptation in AD. SAGE learns a predictive world model that generates two online intrinsic signals: fear, which estimates short-horizon predictive risk and model uncertainty, and curiosity, which measures novelty through prediction error. Curiosity adaptively calibrates the intervention threshold for fear, allowing the agent to regulate risk in a context-dependent manner. When predicted fear exceeds this adaptive threshold, the agent transfers control to an expert or fallback policy and uses the resulting takeover trajectories for focused imitation learning. In parallel, fear is integrated into policy optimization and evaluation as a safety-oriented constraint to reduce performance regressions during adaptation. We evaluate SAGE in simulated route-transfer tasks, Waymo-based logged driving scenarios, CARLA occlusion hazards, and real-world mobile robot navigation tests. Across these settings, SAGE improves robustness in novel and safety-critical scenarios, reduces safety violations, and maintains task performance comparable to strong baseline policies. These results suggest that agents can improve after initial training by estimating the limits of their competence, requesting guidance when needed, and learning selectively from rare high-value events.

I. INTRODUCTION

SAGE addresses the limited ability of deployed driving policies to recognize competence gaps, request targeted supervision, and adapt safely after training. It combines world-model self-monitoring, selective expert intervention, and safety-preserving continual policy improvement for rare distribution shifts.

  • Rare, safety-critical distribution shifts remain difficult because most driving pipelines rely on passive data collection, offline failure analysis, and periodic retraining.
  • SAGE uses a learned world model to estimate novelty, predictive risk, and model uncertainty through curiosity and fear signals.Curiosity captures novelty through prediction error, while fear aggregates short-horizon predicted safety cost and epistemic uncertainty.
  • Curiosity adaptively calibrates the fear threshold, determining when autonomous execution should continue and when control should transfer to an expert policy.The resulting takeover trajectories provide targeted demonstrations for subsequent policy improvement.
  • SAGE couples selective expert intervention with fear-regularized reinforcement learning and value-weighted imitation from takeover trajectories.The framework treats interventions as both protective control actions and learning data.
  • The framework is organized around world-model self-awareness, selective expert intervention, and continual constrained adaptation after deployment.

C. Prospective Fear from World Model

SAGE derives prospective fear from an ensemble world model that predicts near-future safety-relevant outcomes and uncertainty. The resulting signal supports short-horizon reasoning about danger beyond immediate perception.

  • Fear combines predicted safety cost and epistemic uncertainty from an ensemble-based world model.The model predicts safety-relevant quantities from the current state-action pair.
  • The world model predicts next states and safety costs such as collisions or traffic-rule violations for candidate actions.
  • Ensemble predictions are aggregated into mean next-state and cost estimates together with uncertainty, where larger uncertainty indicates lower predictive confidence.
  • Fear is computed as a weighted combination of predicted cost and uncertainty, with β controlling their relative contributions.The predicted cost represents the probability of a safety violation, and uncertainty is normalized to a comparable scale.
  • Short-horizon rollouts extend fear estimation across m predicted steps, allowing the agent to reason about near-future danger beyond immediate perception.Model-generated transitions are stored in a virtual replay buffer.

D. Curiosity-modulated fear tolerance

SAGE uses curiosity to adjust fear tolerance, making intervention more likely in poorly modeled situations. Fear is also incorporated into policy evaluation and constrained optimization to discourage trajectories leading to safety violations.

  • Curiosity modulates fear tolerance through a time-varying safety threshold rather than serving as an intrinsic reward.
  • Larger prediction errors identify transitions that the world model explains poorly and therefore indicate greater unfamiliarity.
  • Higher curiosity lowers the intervention threshold, while lower curiosity raises it as the environment becomes more familiar.
  • Fear is treated as a soft regulator in constrained learning, limiting average anticipated danger without requiring fear to vanish.
  • Fear-regularized evaluation subtracts anticipated fear from action utility, so high-reward actions can receive low value when they lead toward dangerous regions.
  • Under a finite-horizon safety assumption, the selected penalty makes trajectories leading to violations less valuable than trajectories that remain safe.

2) Fear-regularized policy improvement:

Fear-regularized policy improvement maximizes task value while penalizing excessive predicted fear. A dual update adapts the penalty strength to keep average fear near a target during training.

  • Policy improvement maximizes a fear-regularized action value under a target upper bound on fear.
  • The actor objective prefers high-value actions while avoiding actions with excessive predicted fear, enabling proactive risk avoidance.
  • The dual variable increases when average fear exceeds the target and decreases when fear remains below it.

F. Active Expert Guidance Mechanism

The active expert guidance mechanism transfers control when predicted fear exceeds a curiosity-regulated threshold and stores interventions as targeted supervision. Expert demonstrations are weighted by estimated utility rather than imitated uniformly.

  • When fear exceeds the adaptive threshold, control transfers temporarily from the learned policy to an expert controller.
  • Expert control is maintained briefly after activation to prevent oscillatory frame-by-frame switching caused by transient fear spikes.
  • Intervention transitions are stored in the replay buffer and used as targeted supervision alongside reinforcement learning.
  • Imitation strength is continuously adjusted from the estimated value difference between expert and agent actions.
  • Expert actions receive stronger imitation weight when estimated to be superior and weaker weight when estimated to be inferior.
  • The dual-use intervention design protects control-time behavior while collecting demonstrations where autonomous decisions are unreliable.

A. Baselines

The evaluation compares SAGE-RL with standard RL, FNI-RL, interactive imitation, safe model-based, offline diffusion, human-guided, and classical planning baselines across continual-adaptation tasks.

  • Baseline coverage: The broader baseline suite includes model-free, value-based, interactive imitation, safe model-based, offline diffusion, human-guided, and classical navigation methods.The real-world UGV comparisons additionally include Diffusion-BC, Diffusion-QL, Multi-Hug RL, and TEB.
  • RL baselines: SAC, TD3, and D3QN provide general RL comparisons, with SAC used by default and D3QN reserved for discrete-action long-tail experiments.D3QN combines double Q-learning with a dueling value architecture.
  • Continual adaptation: SAGE-RL is compared with standard RL and FNI-RL in route-level distribution-shift experiments.Route 1 trains in episodes 1–1000, followed by Route 2 in episodes 1001–1500.
  • Continual adaptation: The route-level setup observes local traffic, traffic-light state, and navigation cues while outputting continuous longitudinal acceleration.The final observation is a 29-dimensional ego-centric state, and lane changes are handled by simulator routing.

3) Results:

Across route transfer and Waymo adversarial training, SAGE-RL reduces degradation under distribution shift, uses interventions when risk rises, and improves robustness after learning from those interventions.

  • Route transfer: SAGE-RL shows lower degradation and faster recovery after switching from Route 1 to the unfamiliar Route 2.It ultimately reaches higher success while maintaining lower collision and red-light violation rates than the comparison methods.
  • Route transfer: Fear and takeover frequency rise immediately after the route switch, then decline as the policy adapts from intervention data.Curiosity remains active as an additional novelty signal for adaptive intervention.
  • Waymo adversarial training: Waymo training begins from 500 logged replay scenarios containing the ego vehicle and surrounding traffic agents.Surrounding vehicles initially follow their recorded behaviors.
  • Intervention mechanism: Representative cases show fear exceeding a curiosity-regulated threshold, triggering expert takeover during safety-critical interactions.The cited passage describes this intervention mechanism in training-time Waymo cases.
  • Waymo adversarial training: Risk-conditioned resampling creates adversarial encounters with smaller intervehicle distances and lower time-to-collision than replay scenarios.The induced interactions include small gaps and low time-to-collision requiring braking or yielding.

2) Learning Setup:

The Waymo learning setup combines onboard-style ego-centric observations, continuous vehicle controls, safety-aware rewards, and intervention-triggered adaptation evaluated under normal and adversarial scenarios.

  • State and action: The policy observes ego kinematics, route guidance, and a 72-beam LiDAR-style encoding with 50 m range.The resulting ego-centric vector is fully observable from onboard sensing.
  • Evaluation setup: The Waymo evaluation table reports autonomous-driving results under both normal and adversarial scenarios.The table caption identifies the comparison as an evaluation of autonomous driving tasks across these two conditions.
  • Evaluation setup: Results are averaged over five random seeds using 400 training scenarios and 100 testing scenarios.The notation defines success rate, collision rate, average speed, and average acceleration.
  • State and action: Actions are normalized two-dimensional controls mapped to physical steering, acceleration, and braking commands.The control vector is bounded in [-1, 1]^2 before simulator-limit mapping.
  • Adaptation mechanism: Fear-triggered interventions are followed by autonomous emergency handling after updating, including timely braking and geometry-consistent lane changes.The examples are presented as evidence that intervention episodes were internalized rather than required as persistent assistance.
  • Results: SAGE-RL reduces Waymo collision rate relative to SAC by 76.8% on adversarial training data and 69.9% on adversarial testing data.It also improves success rate over SAC across normal and adversarial settings while producing lower acceleration.

D. Long-Tail Safety-Critical Experiments

CARLA experiments test SAGE-RL on an occluded pedestrian hazard, while the validation setup also includes simulated and real-world UGV navigation environments.

  • Task design: The CARLA task hides a crossing pedestrian behind a parked truck until the pedestrian enters the ego vehicle’s potential path.The resulting blindzone hazard requires timely braking or evasive action under partial observability.
  • Training behavior: SAGE-RL achieves steadier reward improvement, whereas D3QN plateaus and fluctuates during training.The baseline continues to show collisions and recurrent safety violations after extended training.
  • Intervention and adaptation: Fear-triggered expert intervention supplies corrective guidance during early unreliable encounters and supports later internalization of safer behavior.The policy gradually refines the expert-guided behavior into a more efficient and calibrated strategy.
  • Safety and efficiency: SAGE-RL’s time-to-collision increases in later training, while D3QN remains consistently lower after declining and partially recovering.SAGE-RL also transitions from conservative behavior toward stable operation with reduced acceleration variability.
  • UGV validation: The experimental program includes Gazebo simulation, a physical UGV, static-obstacle tasks, and dynamic-obstacle interactions.The setup covers narrow passages, detours, repeated lateral crossings, and frontal approach with diagonal cut-ins.

E. Field Tests

The field tests evaluate navigation under static and dynamic obstacle configurations using a UGV platform and onboard LiDAR-based perception, planning, and control.

  • Platform and sensing: The UGV uses 270° LiDAR point clouds as its primary perception input and obtains robot pose from the LiDAR’s built-in IMU.The simulation replica matches the physical map scale, obstacle layout, and moving-obstacle pattern.
  • Validation scenarios: Four real-world scenarios test narrow-passage traversal, static-obstacle detouring, lateral dynamic obstacles, and frontal dynamic obstacles.The scenarios cover constrained passages, blocked routes, and moving-obstacle interactions.
  • State representation: The observation combines 54-sector LiDAR ranges with normalized goal distance, heading error, and previous control commands, yielding a 58-dimensional state.LiDAR distances are clipped and normalized by 10 m.
  • Action space: Continuous actions consist of linear velocity and yaw-rate commands applied at 10 Hz.
  • Reward and cost: The objective combines progress toward the goal, motion smoothness, task completion, collision penalties, and a dense safety cost based on the minimum front-laser range.The reward structure jointly represents task progress, control smoothness, and safety-related events.

3) Results:

The results show robust navigation and continual behavioral refinement, with successful real-world performance and progressively safer responses after expert-supervised adaptation.

  • Quantitative results: 100% success rate is achieved in all four real-world indoor scenarios, outperforming learning-based baselines in navigation reliability.The comparison includes SAC, TD3, Diffusion-BC, Diffusion-QL, and Multi-Hug RL.
  • Quantitative results: The proposed method maintains competitive motion quality, with relatively low path curvature and lateral acceleration while keeping speed and path length competitive.
  • Continual adaptation: In dynamic scenario 3, fear exceeding the intervention threshold triggers expert takeover, after which selective imitation gradually produces safer and smoother autonomous behavior.Further fine-tuning can yield shorter detours than the conservative expert response.
  • Continual adaptation: Across nine deployment-fine-tuning cycles, trajectories become progressively refined as the adapted policy improves beyond abrupt expert stopping responses.
  • Cross-setting pattern: Across route shifts, adversarial traffic, occluded pedestrian hazards, and UGV navigation, elevated fear triggers intervention while stabilization reduces intervention frequency over adaptation.
  • Mechanism and scope: The framework links self-monitoring, active expert intervention, and targeted policy improvement, concentrating supervision on safety-critical and information-rich moments.This design is intended to improve safety during learning and increase the relevance of collected experience for policy updates.
  • Limitations: Fear may be miscalibrated under severe perception degradation, sensor corruption, or highly atypical scenarios, causing delayed or unnecessary interventions.Expert takeover also adds supervision cost and may make early training more conservative.
Loading 2608.29772v1…