PaperlayerView exact revision on arXiv

Source-linked AI summary

Found-RL: foundation model-enhanced reinforcement learning for autonomous driving

Yansong Qu, Zihao Sheng, Zilin Huang, Jiancong Chen, Yuhao Luo, Tianyi Wang, Yiheng Feng, Samuel Labi, Sikai Chen

arXiv:2602.10458v1cs.AIcs.LG

TL;DR

Integrating computationally heavy foundation models into autonomous-driving RL remains challenging. Found-RL addresses this with asynchronous inference and supervision mechanisms, achieving performance comparable to VLMs with a compact 3.82M-parameter model.

  • Problem

    Integrating computationally heavy foundation models into multi-environment reinforcement learning remains a challenge for autonomous driving.

  • Method

    Found-RL integrates foundation models into autonomous-driving RL through asynchronous batch inference, action guidance, and context-specific CLIP-based reward shaping.

  • Results

    Found-RL accelerates learning over standard RL baselines and achieves performance comparable to computationally intensive VLMs with a compact model size (3.82M parameters).

  • Takeaways & Limitations

    Found-RL provides a foundation-model-enhanced RL framework that combines dense reward shaping and explicit action guidance for autonomous driving.

  • Takeaways & Limitations

    Future work targets inference acceleration to prevent stale feedback and broader robustness validation when scaling to larger models.

Abstract

from arXiv · show

Reinforcement Learning (RL) has emerged as a dominant paradigm for end-to-end autonomous driving (AD). However, RL suffers from sample inefficiency and a lack of semantic interpretability in complex scenarios. Foundation Models, particularly Vision-Language Models (VLMs), can mitigate this by offering rich, context-aware knowledge, yet their high inference latency hinders deployment in high-frequency RL training loops. To bridge this gap, we present Found-RL, a platform tailored to efficiently enhance RL for AD using foundation models. A core innovation is the asynchronous batch inference framework, which decouples heavy VLM reasoning from the simulation loop, effectively resolving latency bottlenecks to support real-time learning. We introduce diverse supervision mechanisms: Value-Margin Regularization (VMR) and Advantage-Weighted Action Guidance (AWAG) to effectively distill expert-like VLM action suggestions into the RL policy. Additionally, we adopt high-throughput CLIP for dense reward shaping. We address CLIP's dynamic blindness via Conditional Contrastive Action Alignment, which conditions prompts on discretized speed/command and yields a normalized, margin-based bonus from context-specific action-anchor scoring. Found-RL provides an end-to-end pipeline for fine-tuned VLM integration and shows that a lightweight RL model can achieve near-VLM performance compared with billion-parameter VLMs while sustaining real-time inference (approx. 500 FPS). Code, data, and models will be publicly available at https://github.com/ys-qu/found-rl.

Introduction

Found-RL addresses the latency and scalability barriers of foundation model-enhanced reinforcement learning for autonomous driving through an asynchronous, unified platform. It further distills VLM action knowledge and uses context-conditioned CLIP reward shaping to provide efficient semantic supervision.

  • Motivation: Foundation models offer semantic reasoning and scalable feedback that can combine imitation learning’s data efficiency with RL’s self-exploration, but their integration creates substantial engineering challenges.VLMs can interpret complex driving scenes and generate meaningful supervision signals, while inference latency and missing unified frameworks hinder closed-loop training.
  • Platform and asynchronous inference: Asynchronous batch inference resolves VLM latency bottlenecks by enabling online VLM supervision during real-time or near-real-time learning.The framework separates heavy reasoning from the real-time simulation loop.
  • Platform and asynchronous inference: Found-RL is a unified autonomous-driving platform that decouples computationally intensive VLM reasoning from the RL simulation loop for real-time or near-real-time closed-loop training.Its streamlined pipeline spans CARLA-based simulation and specialized training modules.
  • VLM action guidance: Value-Margin Regularization and Advantage-Weighted Action Guidance distill expert-like VLM action advice into the RL policy.The mechanisms are intended to improve exploration efficiency and decision-making quality.
  • CLIP-based reward shaping: Conditional Contrastive Action Alignment conditions CLIP prompts on discretized ego speed and route command to produce normalized, margin-based reward bonuses from action-anchor scores.This provides dense supervision while addressing CLIP’s dynamic blindness and probability dilution.

Related work

Related work spans RL-based autonomous driving, foundation models, and simulation platforms, identifying gaps in reproducibility, semantic guidance, and foundation-model integration. Found-RL addresses these gaps through a flexible platform connecting foundation models with RL training, evaluation, and benchmarking.

  • RL-based autonomous driving: RL supports complex sequential decision-making in autonomous driving but existing studies often use isolated customized environments, limiting reproducibility and comparability.These limitations motivate platforms that support standardized training, evaluation, and benchmarking.
  • RL-based autonomous driving: Mainstream RL still relies heavily on self-exploration and sparse rewards, often overlooking expert-guided learning and semantically rich reward shaping.The paper positions foundation models as sources of exploration guidance and reward signals for addressing these limitations.
  • Foundation models in autonomous driving: Foundation models offer broad generalization, while VLMs and LLMs support semantic understanding, language-guided decisions, explanations, and trajectory assessment in autonomous driving.Prior applications highlight potential improvements in transparency, flexibility, and safety.
  • Foundation models in autonomous driving: Unlike work using RL to improve foundation-model reasoning, Found-RL leverages foundation models to assist RL agents in autonomous driving.Its plug-and-play modules guide exploration and provide reward signals within RL pipelines.
  • Simulation platforms for autonomous driving: Existing RL libraries and driving simulators provide modular algorithms or realistic evaluation, but generally lack autonomous-driving-specific foundation-model integration.Found-RL adds asynchronous batch inference so VLM processing runs in parallel with simulation, mitigating inference-induced delays during real-time training.

Preliminaries

The preliminaries formulate autonomous driving as a discounted Markov decision process and extend RL with VLM-derived feedback. They also establish off-policy actor-critic learning as the base optimization framework.

  • Problem formulation: Autonomous driving is modeled as a discounted Markov decision process with observations, actions, transition dynamics, rewards, and discount factor γ.A policy induces trajectory distributions through the transition dynamics.
  • Problem formulation: The learning objective is to find a policy that maximizes expected discounted return.
  • VLM-enhanced RL: VLMs provide auxiliary feedback from observations and context, including action suggestions, priors, critiques, or constraint indicators.The RL agent conditions on the augmented input (o_t, g_t).
  • VLM-enhanced RL: VLM guidance can enter optimization through reward shaping or KL policy regularization toward a VLM action prior.The weighting parameters λ and β control the reward-shaping and KL-regularization terms, respectively.
  • Off-policy actor-critic learning: Off-policy actor-critic methods learn an action-value function from replay-buffer transitions and improve the policy using the critic.The framework uses temporal-difference bootstrapping with a slowly updated target network and may include clipped double-Q learning, target-policy smoothing, and entropy regularization.

Method

Found-RL is a unified autonomous-driving platform coupling CARLA simulation, classical RL algorithms, and foundation-model applications. Its method combines asynchronous VLM feedback, guided policy learning, and context-conditioned CLIP reward shaping to make foundation-model-enhanced RL practical.

  • Overall framework: Found-RL couples CARLA simulation, classical RL learners, and foundation-model applications into an end-to-end platform for autonomous driving.The platform provides standardized benchmarks, multimodal observations, simulator context, rewards, and systematic evaluation workflows.
  • Asynchronous batch inference: Asynchronous client-server inference micro-batches VLM requests across environments and time, decoupling rollout from inference so training remains robust to variable latency.Feedback is matched asynchronously to environments and stored with replay transitions for downstream masked learning objectives.
  • VLM action guidance: VLM action guidance constrains exploration through step-wise action proposals, supporting Value-Margin Regularization (VMR) and Advantage-weighting Action Guidance (AWAG).VMR raises the critic’s value for available VLM actions, whereas AWAG imitates them only when the estimated advantage is positive.
  • CLIP reward shaping: Conditional Contrastive Action-Alignment Reward discretizes speed into 4 categories and navigation goals into 6 commands to condition CLIP prompts on dynamic context.The method combines context tuples with longitudinal and lateral action descriptions, addressing CLIP’s limited perception of continuous ego-state values.
  • CLIP reward shaping: The method constructs 720 context-aware prompts and retrieves only 30 action anchors aligned with the current context instead of classifying over the global action space.Conditional slicing preserves relevant state information that the image encoder cannot directly observe while avoiding probability dilution.

Results … Metrics

Found-RL is evaluated with distinct VLM and RL observation/action interfaces, obstacle-aware reward shaping, benchmark-specific protocols, simulation adjustments, demonstration data, and four metric categories. Its stricter evaluation termination mirrors training more closely and tends to lower overall success rates.

  • 5.1. Experiment Setting Observation space and action space: VLM agents use 192×192×3 BEV images with text prompts, whereas RL agents use 96×96×15 BEV masks and compact state vectors.VLMs output continuous 3D throttle, steer, and brake actions; RL agents output 2D throttle/brake and steer actions.
  • Reward functions and terminal criteria: The reward combines speed tracking, route keeping, heading alignment, steering smoothness, and terminal reward or penalty, with obstacle-aware desired speed reduction.Desired speed equals the maximum without detected obstacles and decreases linearly to zero as obstacle distance decreases.
  • Reward functions and terminal criteria: Evaluation termination mirrors training more closely than ROACH through tighter route adherence checks and additional failure triggers, so overall success rates tend to remain lower.Episodes terminate for blocking, route deviation, red lights, stop signs, collisions, and, during evaluation, route completion.
  • Benchmarks: Leaderboard training uses Town01, Town03, Town04, and Town06, with evaluation across Town01–Town06; NoCrash trains on Town01 and evaluates on Town01 and Town02.Representative scenarios from both benchmarks are visualized alongside their map layouts.
  • Metrics: Evaluation metrics span Comprehensive, Route, Energy, and Safety categories, covering return, driving score, infractions, completion, speed, energy use, fuel rate, and collisions.Safety tracks pedestrian collisions, vehicle collisions, and red-light violations.
  • Metrics: Energy efficiency uses Icell from a single-cell equivalent circuit model and Fuel Rate from a longitudinal road-load model.These metrics complement route and safety measures in the evaluation framework.

5.2. Baselines · 5.3. Compare with RLs & ablation analysis

Found-RL is evaluated against fine-tuned VLMs, online RL algorithms, and additional AD, imitation-learning, and offline-RL baselines. Across CARLA benchmarks, VLM supervision improves driving performance, safety, energy efficiency, learning speed, and optimization stability, with method-specific guidance schedules supporting exploration without destabilization.

  • 5.2. Baselines: Three fine-tuned VLM architectures trained on 1.374M state-action transitions provide expert-action guidance and standalone baselines.The evaluated architectures are InternVL3, Qwen2.5-VL, and Visual RWKV, using variants from 0.1B to 7B parameters.
  • 5.2. Baselines: Online RL baselines include DrQ-v2, SAC, and TD3, while additional comparisons cover AD-domain methods, imitation learning, and offline RL.Online baselines use stable-baselines3; imitation-learning and offline-RL methods use d3rlpy.
  • 5.3. Compare with RLs & ablation analysis: Across four metric groups on CARLA benchmarks, VLM-enhanced agents outperform standard RL baselines while maintaining safety and energy efficiency.The groups are Comprehensive, Route, Safety, and Energy; experiments use three independent random-seed runs.
  • 5.3. Compare with RLs & ablation analysis: 0.77 Driving Score and 57% Success Rate make DrQv2-CLIP a state-of-the-art Leaderboard result, substantially surpassing vanilla DrQv2.VLM feedback improves Comprehensive and Route capabilities without compromising Safety; SAC-VMR reduces vehicle collisions from 0.50 to 0.20 per km and reaches Icell =0.07.
  • 5.3. Compare with RLs & ablation analysis: VLM feedback accelerates early learning, and VLM-enhanced methods—especially DrQv2-CLIP—achieve superior asymptotic performance with reduced variance.AWAG and VMR show steep initial increases in reward and route progression, while training curves plateau at stable, high-performing policies.
  • 5.3. Compare with RLs & ablation analysis: AWAG actor losses remain exceptionally flat and stable, whereas VMR produces a margin-driven dip-and-recover trajectory before losses converge.Both actor and critic losses converge to a steady state, indicating asynchronous semantic feedback supplies a robust signal without divergence.
  • 5.3. Compare with RLs & ablation analysis: Decaying guidance coefficients shift agents from VLM imitation toward self-exploration, but SAC-AWAG requires constant regularization to avoid instability and policy collapse.For most configurations, auxiliary losses rise desirably after decay; SAC-AWAG’s loss stays near zero with constant guidance to preserve semantic safety constraints.

5.4. Compare with VLMs

Across Leaderboard and NoCrash benchmarks, Found-RL’s lightweight RL agents achieve competitive driving performance and stronger rule compliance than large VLM baselines under substantially tighter resource constraints. DrQv2-CLIP matches or exceeds several VLM results while using 3.82 M parameters and 96×96 inputs.

  • Leaderboard benchmark: 0.01–0.02 red light violations per km for RL agents are below the VLM baselines’ 0.04–0.08 range.These results use a 3.82 M-parameter model with 96×96 visual inputs, versus VLMs’ 192×192 inputs.
  • NoCrash benchmarks: 0.65 and 0.71 Success Rates for DrQv2-CLIP in Town01 and Town02 exceed Visual RWKV’s 0.61 and 0.63, respectively.Large-scale VLMs achieve higher upper-bound Driving Scores, peaking at 0.92 in Town01 and 0.94 in Town02.
  • NoCrash benchmarks: 0.07 red light violations per km in Town02 are lower for the RL agents than Qwen2.5-vl-7b’s 0.29, despite higher vehicle collision rates.The reduced 96×96 resolution is identified as a likely limitation for handling complex dynamic agents.

5.4. Compare with other baselines · 5.5. Analysis on CLIP Scoring · 5.6. Efficiency analysis

Found-RL achieves strong, safety-oriented performance under stricter termination criteria, while CLIP scoring separates driving maneuvers and provides dense, semantically aligned rewards. Its 3.82M-parameter policy reaches 500 FPS, far exceeding the approximately 1 FPS of much larger VLMs.

  • 5.4. Compare with other baselines: Found-RL achieves the highest Driving Score (0.77) and a superior Infraction Penalty (0.90) despite stricter safety-critical termination criteria and compact BEV inputs.The protocol terminates episodes after unsafe deviations, prioritizing rule compliance over aggressive trajectory tracking.
  • 5.5. Analysis on CLIP Scoring: CLIP evaluation uses 20,000 held-out samples and 30 candidate prompts conditioned on navigation command and speed to classify fine-grained driving actions.Predictions select the text anchor with maximum image-text cosine similarity.
  • 5.5. Analysis on CLIP Scoring: The confusion matrix’s pronounced diagonal shows that CLIP effectively disentangles distinct maneuvers despite high visual similarity between adjacent bird’s-eye-view frames.Off-diagonal errors cluster locally, mainly between intensity variants such as “braking hard” and “braking.”
  • 5.5. Analysis on CLIP Scoring: CLIP shows negligible confusion between opposing lateral actions, indicating a latent space that preserves driving-behavior topology and supports semantic-neighbor masking.Prediction deviations are logical near-misses rather than fundamental perception failures, validating the reward formulation.
  • 5.5. Analysis on CLIP Scoring: Reward availability remains near 100%, supporting dense reward shaping, while the average semantic alignment margin rises before plateauing after approximately 300k steps.The plateau reflects convergence on coarse safety behaviors and elimination of gross semantic errors.
  • 5.6. Efficiency analysis: 500 FPS is achieved by Found-RL with only 3.82M parameters, compared with approximately 1 FPS for 1B to 7B-parameter VLMs.FPS is measured offline as the average over 1,000 consecutive inference runs.
  • 5.6. Efficiency analysis: VLM inference speed does not scale strictly with model size: Qwen2.5-vl-3b is slower than its 7B counterpart, while Visual RWKV 0.1B has the lowest FPS.The discrepancy is attributed to architecture, layer configuration, or visual-encoder computational overhead.

Conclusions

Found-RL integrates Foundation Models/VLMs into autonomous-driving RL through dense reward shaping, explicit action guidance, and asynchronous background reasoning. It accelerates learning, matches computationally intensive VLM performance with a 3.82M-parameter model, and identifies synchronization, scalability, and robustness as future priorities.

  • Contributions: Found-RL integrates Foundation Models/VLMs into the RL loop with dense reward shaping and explicit action guidance for sample inefficiency and safety challenges.The framework is designed specifically for autonomous driving.
  • Results: 3.82M parameters enable performance comparable to computationally intensive VLMs despite significantly smaller spatial inputs, less training data, and a compact model size.This contrasts with massive 0.1B-7B parameter models.
  • Limitations and future work: Asynchronous batch inference enables efficient background reasoning, but synchronized VLM feedback remains crucial during rapid RL training.Future work will explore inference acceleration and quantization to prevent stale feedback and ensure broad sample coverage when scaling to larger models.
  • Limitations and future work: Future evaluation will test robustness on more challenging benchmarks, including CARLA Leaderboard 2.0.The paper identifies broader robustness validation as an open direction.
  • Future directions: A multi-VLM ensemble could provide specialized feedback on different driving aspects, while lightweight VLMs could serve directly as policy backbones for end-to-end planning.These directions aim to collaboratively enhance learning efficiency or support end-to-end planning.

Appendix · Appendix A. Fine-tuning CLIP and discretion scheme

Appendix A defines a discretized representation for speed, controls, and navigation commands, then fine-tunes OpenCLIP to align visual observations with synthesized state-action captions. The scheme uses semantic thresholds and six standard high-level navigation commands.

  • Appendix A. Fine-tuning CLIP and discretion scheme: Speed and control actions are discretized using semantic thresholds defined in Tables 9–11.The referenced tables cover speed, throttle/brake, and steering discretion.
  • Appendix A. Fine-tuning CLIP and discretion scheme: Navigation goals use six high-level commands: turn left, turn right, go straight, follow the current lane, change left, and change right.The commands describe intersection turns, lane following, and lane changes.
  • Appendix A. Fine-tuning CLIP and discretion scheme: 10 epochs were used to fine-tune an OpenCLIP ViT-B-16 initialized from laion2b_s34b_b88k.The model is fine-tuned using the discretized schema and collected data.
  • Appendix A. Fine-tuning CLIP and discretion scheme: The fine-tuning aligns visual observations with synthesized state-action captions.This alignment uses the collected data and the discretized state-action representation.
  • Appendix A. Fine-tuning CLIP and discretion scheme: Throttle and braking are mapped to textual categories including braking hard, braking, accelerating fast, accelerating, accelerating gently, and idling.The categories are assigned by explicit throttle/brake ranges and an otherwise condition.
  • Appendix A. Fine-tuning CLIP and discretion scheme: Steering is represented with five textual prompts: going straight, turning right sharply, turning right, turning left sharply, and turning left.These prompts provide the steering discretion vocabulary.

Declaration of competing interest

The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

  • The authors declare no known competing financial interests or personal relationships that could have appeared to influence the reported work.
Loading 2602.10458v1…