PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo

arXiv:2610.17799v1cs.RO

TL;DR

Online RL could help driving policies adapt, but safe exploration and stable improvement remain challenging when filters alone leave repeated corrections and policy degradation unresolved. SOAR-Flow combines CBF–MPC protection with calibrated value learning and adaptive corrective guidance. In reported driving evaluations, it achieves high success with reduced correction demand.

  • Problem

    Online RL can adapt driving policies, but safe exploration remains necessary because UGV trial-and-error can cause collisions and filtering alone does not ensure stable improvement or fewer corrections.

    Source: §I · p. 1

  • Method

    SOAR-Flow protects offline-initialized Flow policies with CBF–MPC and combines Gaussian-return-aware value calibration with paired corrective learning and intervention-to-go adaptive guidance.

    Source: §I · p. 1

  • Results

    SOAR-Flow achieves 100% SS in blocked-side adaptation and 100% 8-shaped SS, with a 0.68 ± 0.11% correction rate in blocked-side adaptation.

    Source: B. Results and Analysis · p. 7

  • Takeaways & Limitations

    The analysis establishes conditional sample-time safety for protected execution and vanishing auxiliary gradients, linking protected data collection with continued task optimization under prescribed execution constraints.

    Source: §VI · p. 7

Abstract

from arXiv · show

Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)--model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration video and project overview are available at https://lzqw.github.io/SOAR-Flow/.

I. INTRODUCTION

Safe online learning must avoid unsafe exploration while improving beyond protected execution, which can still require repeated corrections and risk degrading learned behavior. SOAR-Flow combines protected interaction with calibrated value learning and adaptive corrective guidance.

I. INTRODUCTION

  • Online RL can improve driving policies from interaction, but trial-and-error exploration on UGVs risks collisions, making safety essential throughout learning.Offline demonstrations can provide useful initial skills.Source: §I · p. 1
  • Safety filters can correct actions before execution, yet repeated corrections and online policy degradation remain concerns.The paper frames the goal as learning from protected experience to improve task performance and reduce correction demand.Source: §I · p. 1
  • SOAR-Flow combines CBF–MPC protection with Gaussian-return-aware value calibration, paired corrective learning, and intervention-to-go prediction for online refinement.Calibration uses recorded behavior returns and predicted return variability; predicted intervention dependence also regulates guidance strength.Source: §I · p. 1
  • Recorded behavior returns and predicted return variability are used to smooth conservative critic updates and support stable learning from protected execution.This extends Cal-QL's return calibration to account for variability during continued online learning.Source: §I · p. 1
  • Paired corrective learning and intervention-to-go prediction guide feasible action adjustments and reduce reliance on future filter corrections while retaining task optimization.Guidance weakens as predicted dependence decreases.Source: §I · p. 1
  • The evaluation uses safety-filtered and policy-only tests in MetaDrive and on a UGV, retaining nominal MPC tracking.The introduction also states that conditions are established for sample-time safety, though the supplied contribution passage is truncated.Source: §I · pp. 1–2

II. RELATED WORK

Prior work addresses safe execution, value learning, policy refinement, and generative policies through complementary methods. SOAR-Flow builds on these strands and models driving as a continuous-action MDP, initializing a Flow policy through offline behavior cloning.

II. RELATED WORK

  • SOAR-Flow combines paired corrective supervision with prediction of future filter use, while CBF–MPC certification remains active.This connects intervention-based learning with execution-time protection.Source: §II · p. 2
  • SOAR-Flow builds on Cal-QL and DSAC’s Gaussian return model to smooth behavior-referenced value calibration using predicted return variability.This complements adaptive intervention guidance during protected online refinement.Source: §II · p. 2

III. METHODOLOGY

  • Autonomous driving is modeled as a continuous-action MDP whose state and action spaces represent driving states and references, with transitions, task rewards, and discounting.The policy maximizes an entropy-augmented SAC objective.Source: §III · p. 2
  • A conditional Flow policy transforms Gaussian noise into driving references through a state-conditioned vector field.The flow evolves from noise to an endpoint that induces the policy distribution.Source: §III · p. 2
  • Flow-based behavior cloning initializes the policy by minimizing a loss using offline state–reference pairs and interpolated flow inputs.The supplied method passage specifies the offline sampling setup and initialization role.Source: §III · p. 2

A. Calibrated Value Learning under Safety Filtering

SOAR-Flow certifies references before execution and learns from the resulting protected transitions. Its Gaussian-return-aware calibration uses recorded returns and predicted variability to smooth conservative critic updates.

Maintaining safe exploration

  • CBF–MPC certifies references against barrier, state, input, and terminal constraints, returning feasible corrections when needed before applying the MPC tracking action.Certification is repeated at each control step; the terminal set admits a feasible backup controller.Source: A. Calibrated Value Learning under… · pp. 2–3

Protected online data

  • Protected transitions store the raw reference and correction flag, while rewards and next states from executed certified actions supervise task critics.Recorded corrections also provide action targets for references that pass certification directly.Source: A. Calibrated Value Learning under… · p. 3

Avoiding policy degradation

  • Cal-QL calibrates candidate values against discounted returns from completed offline and online episodes, including rewards from corrected execution.The Gaussian return model extends this calibration using predicted variability to smooth conservative updates.Source: A. Calibrated Value Learning under… · p. 3

Gaussian-return-aware calibration

  • Predicted return variability sets the smoothing width around the recorded-return reference, making conservative pressure change continuously rather than abruptly.The mean-only truncation returns in the zero-variance limit, and gradients through the predicted standard deviation are disabled in the calibration term.Source: A. Calibrated Value Learning under… · p. 3

Critic objective

  • The critic objective combines calibration on current-policy actions with Gaussian return fitting on replay transitions, using recorded execution outcomes as behavior references.Its Bellman target follows protected execution, and the same objective is used for offline initialization and online refinement.Source: A. Calibrated Value Learning under… · pp. 3–4

B. Adaptive Guidance for Autonomous Policy Improvement

SOAR-Flow combines local corrective supervision with task-value learning and predictions of future filter use. Those predictions also regulate corrective guidance so its influence can diminish as assistance demand falls.

Motivation

  • Local corrective guidance addresses repeated interventions, while the policy can still seek higher-return behavior within prescribed safety constraints.A finite-horizon MPC cost need not maximize long-horizon task return.Source: B. Adaptive Guidance for Autonomous… · p. 4

Learning from task values and corrections

  • Task values guide policy improvement on uncorrected records, while corrected records provide feasible action targets through paired updates.The same generator input is reused to associate each correction with the raw action that triggered it.Source: B. Adaptive Guidance for Autonomous… · p. 4

Anticipating future interventions

  • The intervention-to-go predictor estimates discounted future filter use, with segmented targets propagating observed interventions and a continuation estimate.Longer segments propagate observed intervention signals farther in each update.Source: B. Adaptive Guidance for Autonomous… · p. 4

Actor objective

  • The actor combines task values, feasible corrections, and intervention-to-go predictions, which discourage actions associated with later assistance.Critics, correction targets, and coefficients are fixed during actor optimization.Source: B. Adaptive Guidance for Autonomous… · p. 4

Relaxing corrective guidance

  • Predicted intervention demand regulates corrective guidance, and vanishing feedback drives its weight toward zero while calibration regularization also decays.This reduces auxiliary learning influences while task improvement continues under unchanged execution constraints.Source: B. Adaptive Guidance for Autonomous… · p. 4

C. Safety and Asymptotic Guidance Analysis

The analysis separates sampling-time execution safety from the asymptotic influence of auxiliary learning terms. Under stated feasibility, residual, convergence, and bounded-gradient conditions, safety is preserved in the zero-residual case and auxiliary contributions vanish.

Certified safety

  • With feasible CBF–MPC and bounded residuals, realized barrier values satisfy a recursive lower bound; zero residual bounds preserve nonnegative barriers at every sampling instant.The safety statement assumes an initially feasible state and the stated transition inequality.Source: C. Safety and Asymptotic Guidance… · p. 4

Vanishing auxiliary guidance

  • If predicted intervention demand and calibration weight tend to zero, with the specified gradients uniformly bounded, the guidance weight and auxiliary gradient contributions also vanish.The guidance recursion has contraction factor 1 − ηλκλ in (0, 1).Source: C. Safety and Asymptotic Guidance… · pp. 4–5

Implication

  • As auxiliary influences vanish, the task and entropy objective remains active without a persistent recovery-choice penalty, while certified execution constraints remain enforced.This leaves scope for higher-return behavior within the certified region.Source: C. Safety and Asymptotic Guidance… · pp. 4–5

IV. SIMULATION EXPERIMENTS

The simulation experiments assess whether SOAR-Flow improves an initialized driving policy while limiting training violations and reducing reliance on safety corrections.

Evaluation aims

  • The experiments compare baselines and ablations to examine value calibration and adaptive guidance alongside policy improvement, training violations, and reliance on safety corrections.Source: §IV · p. 5

A. Experimental Setup

Simulation experiments use MetaDrive scenarios for offline initialization, protected online fine-tuning, and evaluation. The protocol compares policy performance and safety under both protected and unprotected execution modes.

Experimental environment and data

  • Experiments use SafeMetaDriveEnv, with ten fixed scenario seeds for offline initialization and 20 additional seeds held out for online fine-tuning and evaluation.The actor receives 259 observation dimensions; four additional safety-context variables are provided only to task critics.Source: A. Experimental Setup · p. 5

Control pipeline

  • The actor’s normalized output defines a local target, which CBF–MPC certifies or corrects before an NMPC tracker produces vehicle commands.Task reward and safety cost are recorded separately; maximum commanded speed is 10.75 m/s.Source: A. Experimental Setup · p. 5

Training protocol

Comparisons and metrics

  • Baselines include fixed CBF–MPC, Flow-BC, Cal-QL, PROTO-SAC, Gaussian-DSAC, and Flow-Q-only; PO disables CBF correction while SF retains certification.PO reports return, safe success, and evaluation safety cost; protected training reports cumulative safety cost, and results are averaged across seeds with sample SD.Source: A. Experimental Setup · p. 5

B. Results and Analysis

SOAR-Flow improves simulation return over offline Flow-BC while achieving low training cost and correction demand. Ablations indicate that the full method combines stronger return with fewer corrections than the tested variants.

Online improvement

Safety and dependence

  • SOAR-Flow’s training cost is 0.18 ± 0.10 and correction rate is 0.72 ± 0.12%, versus 0.85 ± 0.32 and 2.68 ± 0.28% for Gaussian-DSAC.The table also lists 100% safe success and zero evaluation cost under PO evaluation.Source: B. Results and Analysis · p. 6, Table I · p. 6

Ablations

  • Without corrective learning or P, correction rates are 4.98% and 1.98%, respectively, versus 0.72% for the full method.Removing calibration and using mean-only calibration yield return entries of 338.0 and 348.0; mean-only calibration has a 1.20% correction rate.Source: B. Results and Analysis · p. 6, Table I · p. 6

V. REAL-WORLD EXPERIMENTS

The paper evaluates SOAR-Flow on an uncrewed ground vehicle and tests a frozen policy on a held-out cornering task. The platform is shown in Fig. 4.

Real-world evaluation

  • A frozen SOAR-Flow policy is tested zero-shot on held-out cornering with obstacle avoidance, beyond the training layouts.Source: §V · p. 6
  • The evaluation examines protected online adaptation and the development of driving behavior requiring fewer safety corrections.Source: §V · p. 6

Platform

A. Experimental Setup

The UGV study pretrains a Flow policy from human demonstrations, then refines one policy across two obstacle-maneuver tasks under CBF–MPC protection. Checkpoints and a final frozen policy are evaluated across training and held-out layouts.

Experimental platform

  • The UGV carries LiDAR, cameras, radar, and an integrated navigation system; policy inference and NMPC run onboard at 10 Hz and 20 Hz, respectively.The actor specifies a local reference that is certified before NMPC generates steering and longitudinal commands.Source: A. Experimental Setup · p. 6

Training tasks

  • Twenty-four human-driver trajectories initialize the policy; online refinement alternates between blocked-side bypass and an 8-shaped maneuver using a shared replay buffer.The blocked-side task removes one demonstrated route, while the 8-shaped task requires successive turns around two obstacles.Source: A. Experimental Setup · p. 6

Experimental overview

  • Fig. 5 shows multimodal initialization, blocked-side and 8-shaped trajectories, held-out execution, and blocked-side learning curves.Red trajectory segments indicate corrections; the late 8-shaped run uses PO execution with MPC tracking retained.Source: Figure 5 · p. 7

Evaluation protocol

  • Checkpoints are evaluated without parameter updates on both training tasks, and the final frozen policy is tested zero-shot on a held-out cornering layout.The cornering layout is excluded from demonstrations, online training, tuning, and checkpoint selection; the protocol uses three training seeds.Source: A. Experimental Setup · p. 7

Comparisons and metrics

  • UGV comparisons include CBF–MPC, Flow-BC, and Gaussian-DSAC; each method’s safe success is computed from 20 evaluation trials per task.Learned-policy safe success and return use PO, while correction rates use SF; continuous metrics are reported as mean ± SD.Source: Table II · p. 6, A. Experimental Setup · p. 7

B. Results and Analysis

SOAR-Flow improved driving performance while reducing correction dependence during online refinement. Results span blocked-route adaptation, 8-shaped execution, and held-out cornering; the conclusion reports conditional sample-time safety and vanishing auxiliary gradients.

B. Results and Analysis

  • On blocked-side adaptation, SOAR-Flow achieved 100% SS and return 350.8 ± 1.8, versus 343.9 ± 2.6 for Gaussian-DSAC; correction rate was 0.68 ± 0.11%.Correction rates were 2.46 ± 0.32% for Gaussian-DSAC and 4.31 ± 0.41% for Flow-BC.Source: B. Results and Analysis · p. 7
  • SOAR-Flow completed the 8-shaped task in 15.36 ± 0.24 s versus 16.72 ± 0.31 s for CBF–MPC, a 1.36 s reduction, with minimum clearances of 0.801±0.019 m and 0.779±0.018 m, respectively.The shorter completion time accompanied slightly larger mean clearance.Source: B. Results and Analysis · p. 7
  • On held-out cornering, SOAR-Flow attained 95% SS, compared with 85% for fixed CBF–MPC, 55% for Flow-BC, and 90% for Gaussian-DSAC.The frozen policy transferred without further updates, although not every evaluation succeeded.Source: B. Results and Analysis · p. 7

VI. CONCLUSION

  • The analysis establishes conditional sample-time safety of protected execution and vanishing auxiliary gradients under the stated conditions.SOAR-Flow aims to reduce reliance on particular recovery behaviors while retaining prescribed execution constraints.Source: §VI · p. 7

B. Results and Analysis

  • The 8-shaped refinement progressed from corrected early and intermediate runs to late execution without CBF correction; SOAR-Flow reached 100% SS, versus 95% for CBF–MPC and 90% for Gaussian-DSAC.Red segments mark correction intervals in the early and intermediate runs.Source: B. Results and Analysis · p. 7, Figure 6 · p. 8

Related papers

Loading 2610.17799v1…