PaperlayerView exact revision on arXiv

Source-linked AI summary

CORAL: Curriculum-Optimized Reward Adaptation for LiDAR-Based Goal-Directed Urban Driving

Anisa Saleem, Duksu Kim

arXiv:2608.14332v1cs.ROcs.LG

TL;DR

Long-horizon urban driving requires learning competing navigation, safety, smoothness, and rule-following behaviors without a fixed learning order. CORAL couples a route-distance curriculum with stage-aware rewards and reaches the goal in all twenty hardest-route episodes, versus 5% and 10% for two PPO baselines.

  • Problem

    Existing reinforcement-learning driving systems often use fixed rewards even as agents progress from basic control to complex navigation, despite multiple competing objectives.

  • Method

    CORAL jointly advances a five-stage route curriculum and stage-aware reward on a compact route-aware LiDAR state using a multi-stream PPO actor–critic.

  • Results

    All twenty hardest-stage evaluation episodes reached the goal, compared with 5% and 10% for the two PPO baselines.

  • Takeaways & Limitations

    The factorial ablation indicates that coupling both schedules is important: neither schedule alone matches the full configuration, and disabling both reduces success to 55%.

  • Takeaways & Limitations

    Static, fixed-weather evaluation leaves obstacle safety around other vehicles and robustness to illumination or weather untested, while the LiDAR histogram’s contribution remains unestablished.

Abstract

from arXiv · show

Reinforcement learning is promising for autonomous urban driving, but long-horizon goal-directed navigation asks a policy to acquire several competing behaviors at once--reaching a distant goal, tracking a route, avoiding obstacles, obeying signals--and a fixed objective gives no order in which to learn them. This paper presents CORAL, which advances two schedules together: a five-stage curriculum that progressively lengthens routes and tightens behavioral constraints, and a stage-aware reward whose component weights shift emphasis from mission progress toward route following, safety, smoothness, and rule compliance as the task hardens. The policy is a multi-stream actor-critic network trained with Proximal Policy Optimization (PPO) in CARLA on a compact 99-dimensional state pairing a polar LiDAR histogram with vehicle telemetry, ego-frame route geometry, and traffic-rule indicators--no point-cloud encoder, no bird's-eye-view rasterization. Against two PPO baselines under an identical protocol, CORAL reaches the goal in all twenty evaluation episodes on the longest routes under the full set of behavioral constraints, where the baselines reach 5% and 10%; a factorial ablation shows that neither schedule alone matches their combination: removing either lowers both success and route completion, and disabling both drops success to 55%. Trained in one town, the policy transfers zero-shot to seven unseen towns, succeeding in 68-98% of episodes on routes of the same 100-150 m length, with mean lateral deviation below 0.35 m.

I. INTRODUCTION · II. RELATED WORK

The paper motivates CORAL as a reinforcement-learning framework for long-horizon urban driving, addressing distribution shift, sparse rewards, training instability, and sensory-representation trade-offs. It combines a compact route-aware LiDAR state with a five-stage curriculum and stage-aware reward scheduling, distinguishing the approach from prior PPO, curriculum, and reward-design methods.

  • I. INTRODUCTION: Urban driving requires robust perception, long-horizon planning, and smooth continuous control across diverse conditions, while imitation learning struggles with distribution shift, rare corner cases, and unseen road geometries.These limitations arise when expert demonstrations are unavailable for novel scenarios or complex intersections.
  • I. INTRODUCTION: Reinforcement learning avoids expert labels and can optimize long-horizon objectives such as route completion and smooth control, but realistic model-free driving faces high-dimensional observations, sparse rewards, and training instability.The framework is evaluated in simulators such as CARLA.
  • I. INTRODUCTION: LiDAR offers geometry largely invariant to illumination, but raw point clouds are computationally heavy and BEV rasterizations are lossy, motivating a compact geometry-based representation.The introduction contrasts these trade-offs with the vulnerability of RGB inputs to illumination changes, occlusions, and adverse weather.
  • I. INTRODUCTION: CORAL uses a compact route-aware state built from a 64-bin polar LiDAR histogram, vehicle telemetry, and upcoming ego-frame route information instead of raw point clouds or dense BEV maps.The related-work description further characterizes the route features as lateral deviation and heading alignment relative to the planned route.
  • I. INTRODUCTION: A five-stage curriculum lengthens routes and tightens behavioral constraints, while stage-aware reward weights shift the relative emphasis among mission progress, route following, safety, smoothness, and traffic-rule compliance.The two schedules advance together to stabilize learning over long horizons.
  • I. INTRODUCTION: On the static benchmark, removing the LiDAR histogram causes no measurable loss, which the paper reports as a limitation of the evaluation environment rather than of the representation itself.This finding comes from a leave-one-out observation test reported alongside the main results.
  • II. RELATED WORK: Prior autonomous-driving reinforcement-learning work spans value-based and actor–critic methods, while PPO is widely adopted for continuous control because of its stable clipped objective and effectiveness in simulated driving.Related studies also include end-to-end formulations that map observations directly to continuous control commands.
  • II. RELATED WORK: Curriculum learning progressively increases task difficulty, and reward design balances navigation accuracy, safety, comfort, and traffic-rule compliance; unlike CuRLA, CORAL schedules reward component weights across route-distance curriculum stages.CuRLA combines a vision-based latent state, traffic-density curriculum, and a mostly fixed reward with a collision penalty added partway through training.

III. METHODOLOGY · A. System Overview

CORAL formulates goal-directed urban driving as continuous-control reinforcement learning and combines a compact LiDAR-based observation pipeline with a multi-stream actor–critic policy. Its training process jointly uses stage-aware rewards and a curriculum that increases route difficulty while shifting behavioral priorities.

  • III. METHODOLOGY: CORAL presents a curriculum-guided reinforcement learning framework for goal-directed autonomous driving.
  • A. System Overview: Goal-directed urban driving is modeled as continuous control, with the agent producing steering, throttle, and brake commands from scene observations.Training occurs in the CARLA simulator using PPO.
  • A. System Overview: The pipeline first reduces raw LiDAR point clouds into a polar histogram encoding obstacle proximity in the ego-vehicle frame.
  • A. System Overview: This histogram is concatenated with vehicle telemetry, route-aware navigation features, and traffic-rule signals to create a low-dimensional observation vector.
  • A. System Overview: A multi-stream actor–critic network maps the combined observation to continuous driving commands.
  • A. System Overview: Training couples a stage-aware reward with a curriculum that lengthens routes and tightens behavioral constraints across stages.Reward priorities shift with curriculum difficulty, allowing basic vehicle control to be acquired before harder navigation behaviors.

B. Observation Space

The policy uses a compact, route-aware observation space combining vehicle telemetry, route geometry, traffic-rule signals, and a 64-bin polar LiDAR histogram. Continuous steering, acceleration, and braking actions support PPO-based control.

  • Observation components: Vehicle telemetry provides normalized speed, steering, throttle, and brake commands as four complementary driving-state quantities.The telemetry component is defined in R4.
  • Observation components: The route-aware features include the next ten waypoint offsets, lateral deviation, goal-relative quantities, heading alignment, and local route curvature.These features provide short-term trajectory guidance and long-horizon navigation context.
  • Observation components: Traffic-rule signals comprise three binary traffic-light indicators, trigger-region distance, and a red-light stop-zone indicator.Together, these five signals encode traffic-light state and stopping requirements.
  • Observation components: 64-bin polar LiDAR histograms encode minimum obstacle distance across uniformly divided −90° to +90° sectors, normalized to a 30 m sensing range.Smaller values indicate nearby obstacles, while larger values indicate free space, avoiding raw point-cloud processing.
  • Action space: Continuous actions control steering, acceleration, and braking, a formulation suited to PPO and smoother than discrete action sets.The action components are bounded in [0, 1] for acceleration and braking.

C. Multi-Stream Actor–Critic Policy Network

CORAL uses a multi-stream actor–critic network that independently encodes heterogeneous observation modalities before fusion. Shared fused features feed separate actor and critic heads for continuous control and PPO value estimation.

  • Architecture: Independent modality encoders process vehicle dynamics, route geometry, traffic-rule signals, and obstacle observations before feature fusion.This modality-specific processing reduces interference between heterogeneous inputs.
  • Architecture: 32-, 64-, 32-, and 64-dimensional embeddings encode telemetry, route, rule, and LiDAR inputs, respectively.The embeddings concatenate into a 192-dimensional feature before the shared fusion MLP.
  • Architecture: 192 →1024 →512 →256 are the layer widths of the shared fusion MLP.The fused representation is formed after concatenating the four modality embeddings.
  • Task-specific heads: 256 → 128 →64 are the hidden-layer widths of each task-specific head, which branches from the fused representation.The actor parameterizes a diagonal Gaussian policy over steering, throttle, and brake, while the critic estimates state value for PPO advantage estimation.

D. Stage-Aware Reward Design

CORAL uses an interpretable multi-objective reward combining five driving behaviors, while scheduling their weights across five curriculum stages instead of using fixed weighting. The design also clips step rewards, shapes progress and stopping behavior, and handles terminal outcomes separately.

  • Multi-objective reward: The reward combines mission progress, route following, safety, smoothness, and traffic-rule objectives as interpretable components.This replaces reliance on a single sparse reward signal.
  • Stage-aware weighting: The component weights depend on curriculum stage k ∈ {0, . . . , 4}, and the per-step reward is clipped to [−10, 10].Stage-aware weighting lets learning objectives evolve during training while clipping limits destabilizing updates.
  • Mission and idling: Mission progress is normalized and clipped, suppressed below 0.2 m/s, and paired with a single idle penalty to discourage prolonged unnecessary stopping.The idle penalty is disabled during legitimate red-light stops within the 20 m stop zone.
  • Driving behavior terms: Route following rewards route, waypoint, lateral-path, and heading alignment, while safety regulates speed using local curvature and penalizes overspeeding and excessive control combinations.Collisions are handled separately as terminal safety failures.
  • Rules and terminal events: Rule compliance activates from Stage 2 and ramps in, while terminal rewards separately scale goal completion by initial route distance and curriculum stage.The goal reward uses sd = max(0.75, d0/25) and sk = 1+0.4k, so longer routes and later stages receive larger completion bonuses.

E. Curriculum-Guided Training Strategy

CORAL trains with a five-stage curriculum that lengthens routes and tightens behavioral constraints, while advancing a stage-aware reward schedule alongside it. Stages advance on fixed interaction budgets, shifting emphasis toward route following and rule compliance as difficulty increases.

  • Rationale: The five-stage curriculum progressively increases task difficulty to address unstable optimization and inefficient exploration on complex routes with strict constraints.Early stages allow recovery from mistakes, whereas later stages enforce tighter constraints.
  • Curriculum progression: Five stages increase route distance from 10–20 m at Stage 0 to 100–150 m at Stage 4.Early training targets basic vehicle control and goal-directed motion before longer routes are introduced.
  • Constraint tightening: Three of five termination criteria are inactive at Stage 0, and every tolerance shrinks thereafter.A sidewalk incursion tolerated for 45 steps (2.25 s) at Stage 1 ends the episode within 14 steps (0.7 s) at Stage 4.
  • Stage advancement: 60,000, 150,000, 250,000, and 350,000 global steps trigger entry into Stages 1–4, respectively.The deterministic schedule allocates 60,000, 90,000, 100,000, 100,000, and 150,000 steps to Stages 0–4.
  • Coupled reward scheduling: 0.20 →0.50 is the route-following weight increase at Stage 1, while rule-compliance weights begin ramping in at Stage 2.The curriculum and stage-aware reward advance together, motivating the factorial ablation.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate CORAL in CARLA against two PPO baselines using shared training and evaluation protocols, then define stage-wise, cross-town, and ablation analyses. Metrics capture success, route completion, safety, route adherence, lateral deviation, lane invasions, and red-light compliance.

  • Experimental setup: CORAL is trained in CARLA v0.9.15 on static, goal-directed navigation in Town05, isolating route following and traffic-signal compliance without other vehicles or pedestrians.Collisions arise only from fixed scene geometry; the policy is evaluated in Town05 and across unseen towns.
  • Evaluation protocol: The per-stage comparison sweeps five curriculum stages in Town05, varying route length and behavioral difficulty while aggregating 20 training episodes per stage.This evaluation describes how each system behaves while acquiring the task, with the map held fixed.
  • Evaluation protocol: Cross-town evaluation samples 100 independent episodes per town, whereas each ablation replays the same 100 Stage-4 Town05 start–goal pairs across compared configurations.The separate route lists ensure within-table ablation differences reflect policies rather than route sampling.
  • Evaluation protocol: Evaluation episodes end upon destination arrival, collision, stuck-vehicle termination, or timeout, with stuck termination requiring 600 steps below 0.15 m/s and timeout occurring after 5000 steps.These correspond to 30 s and 250 s respectively at the stated simulation rate.
  • Metrics: Reported metrics include Psucc, Pcoll, Poff, lateral deviation, lane-invasion rate, red-light encounters and violations, and ablation-only route completion Pcomp.Higher is better for Psucc and Pcomp; lower is better for the remaining metrics, while red-light violations are interpreted relative to encounters.
  • Baselines: The framework compares CORAL with E2E PPO and CuRLA-Inspired PPO under shared CARLA, PPO, route-planning, budget, protocol, curriculum, and route-observation conditions.The comparison varies the remaining observation streams and reward design: CORAL uses polar LiDAR and traffic-rule signals with stage-aware weights, while baselines use VAE-encoded camera inputs and their respective rewards.

A. Comparative Performance Evaluation

CORAL maintains nearly flat success across the five-stage difficulty ladder while the PPO baselines deteriorate as horizons grow. It also improves route adherence and collision performance, but red-light compliance remains unreliable despite stage-aware reward shaping.

  • Success across curriculum stages: 100% success across four of five curriculum stages, with 95% in Stage 3, while both PPO baselines lose most success as route horizons grow.The key result is the proposed policy’s stability across the difficulty ladder rather than performance at one isolated stage.
  • Controlled comparison: All three policies receive identical waypoint-based route guidance, so their performance gap reflects objective optimization rather than different route information.Each policy receives route block, alignment, lateral-error, and curvature information from the same planner.
  • Route following and safety: Below 0.6% off-route at every stage, CORAL contrasts with baseline off-route rates of 8%-31% from Stage 1 onward.CORAL collides in one of twenty Stage-3 episodes and in none of the other reported episodes.
  • Traffic-light compliance: 35% overall red-light violations, including 71 of 176 in the training town, show that traffic-light compliance remains unreliable across the eight evaluated towns.The policy encountered 986 red lights and violated 347; compliance improves after Stage 1 but is not consistently applied.

B. Generalization Performance

The policy transfers across seven unseen CARLA towns, maintaining high success and route-following accuracy at fixed 100–150 m route lengths. Generalization is uneven, with collisions the dominant failure mode in difficult towns, while the design source of transfer remains unresolved.

  • Transfer results: 68.0%–99.0% success spans the eight towns, with at least 93.0% in Town01, Town03, Town06, and Town10.Town05, the training environment, reaches 99.0% success.
  • Transfer results: Below 0.35 m mean lateral deviation and below 1.5% lane-invasion rates are maintained in every town.All missions use the stage-4 setting with 100–150 m routes and 100 episodes per town.
  • Failure modes: 25.0% collision rates in Town04 and 22.0% in Town07 identify collisions as the dominant failure mode across the evaluated towns.Collision rates range from 1.0% in Town05 to 25.0% in Town04; in five towns, unsuccessful episodes terminate through collision.
  • Town difficulty: 68.0% success, 25.0% collisions, and 2.47% off-route behavior make Town04 the most challenging environment.Town07 is next, with 78.0% success, 22.0% collisions, and 2.11% off-route behavior.
  • Interpretation and limitation: Transfer mechanisms remain unresolved because ablations vary factors only within Town05 and baselines were not evaluated across towns.The observed success in all seven unseen towns supports transferable behavior rather than single-layout memorization, but does not identify its cause.

C. Ablation: Reward and Distance Schedules

A complete 2×2 factorial ablation shows that jointly advancing reward weights and sampled route distance outperforms either schedule alone or neither, on both success and route completion. The schedules contribute differently: distance progression supports graduated route-length learning, while reward progression tightens behavioral requirements.

  • Experimental design: The 2×2 factorial compares the full schedule, each single-factor variant, and a no-curriculum control while holding the terminal task constant.Disabled schedules remain at their Stage-4 settings throughout training, isolating how the policies are led to the same task.
  • Results: 99.0% success and 96.40% completion are achieved by the full schedule, the best of all four configurations on both scores.Route completion gives partial credit for route coverage even when the goal is not reached.
  • Results: 27.0 points of success are lost by removing reward scheduling, versus 9.0 for removing distance scheduling; completion losses are 6.58 and 6.21 points, respectively.The better single-factor variant reaches 90.0% success, below the full schedule’s 99.0%.
  • Factorial effects: 35.0 versus 27.0 points quantify reward scheduling’s success benefit when distance scheduling is off versus on, while distance scheduling contributes 17.0 versus 9.0 points under the reverse conditions.These results show that the schedules are not interchangeable within either level of the other factor.
  • Interpretation: Distance scheduling expands sampled route distance in graduated steps, whereas reward scheduling progressively tightens the behavior required to complete the route.The coordinated roles motivate advancing both mechanisms together.

D. Ablation: The LiDAR Observation

The LiDAR ablation finds navigation success insensitive to histogram resolution and unable to establish that the histogram contributes under the static benchmark. The 128-bin configuration increases mean lateral deviation, plausibly reflecting added capacity rather than useful obstacle information.

  • Resolution: 100, 99, and 96 successes out of 100 episodes occur with 32, 64, and 128 bins, respectively, with no statistically distinguishable pair.Fisher’s exact tests give p = 1.00, 0.37, and 0.12 for the three pairwise comparisons.
  • Interpretation: The observed insensitivity is attributed to the static environment, where relevant roadway obstacles are wide and their bearing changes only with ego motion.The passage expects the picture to change under dynamic traffic.
  • Stream contribution: 98.0% success without the polar histogram versus 99.0% with it leaves no resolvable difference, so the study does not claim the stream contributes.The comparison records two collisions without the histogram versus one with it, plus marginally lower lateral deviation without it.
  • Resolution: 0.19 m mean lateral deviation at 32 and 64 bins rises to 0.35 m at 128 bins.The 128-bin observation adds 64 dimensions while using the same 500,000-step training budget; capacity rather than information is offered as a hypothesis.

V. CONCLUSION

CORAL combines a five-stage route-distance curriculum with stage-aware rewards and a compact 99-dimensional observation, achieving perfect longest-route evaluation success and strong zero-shot transfer. The study also identifies static evaluation and limited evidence for LiDAR usefulness as key limitations.

  • Method: CORAL jointly advances a five-stage route-distance curriculum and stage-aware reward on a compact 99-dimensional route-aware observation without high-dimensional visual input or expert demonstrations.The policy is trained in CARLA.
  • Results: 20/20 episodes reached the goal at the hardest curriculum stage, versus 5% and 10% for the two PPO baselines.This comparison uses the same per-stage evaluation protocol described in the conclusion.
  • Transfer: 68–98% zero-shot success transferred to seven unseen towns on 100–150 m routes, with at least 93% in four towns and mean lateral deviation below 0.35 m.The training-town success rate was 99.0%.
  • Limitations: The static benchmark does not establish LiDAR-histogram usefulness: removing it yielded 98.0% success versus 99.0% with it, while confirming that the 99-dimensional state suffices.The study used no learned perception front-end.
  • Future work: Future work targets routes with obstacles and dynamic traffic, reliable red-light compliance, runtime measurement, multiple seeds, and richer semantic or language-model-based assistance.These directions address the stated evaluation limitations and extend the driving system.
Loading 2608.14332v1…