Source-linked AI summary
Reinforcement Learning with Human Feedback for Realistic Traffic Simulation
Yulong Cao, Boris Ivanovic, Chaowei Xiao, Marco Pavone
TL;DR
Traffic simulation needs realistic, human-aligned traffic models, but capturing subjective preferences and supporting diverse models remain challenges. TrafficRLHF uses human feedback, a reward model, and fine-tuning to adapt existing traffic models; evaluations on nuScenes report trajectories closely aligned with human preferences.
Problem
Method
TrafficRLHF uses human feedback to train a reward model and fine-tunes traffic models with that reward while retaining their original training objective.
Source: §1 · p. 1, §3.4 · p. 4Results
On nuScenes, TrafficRLHF generated trajectories closely aligned with human preferences.
Source: §5 · p. 8Takeaways & Limitations
The paper contributes a realism-alignment dataset, a human-preference reward model, and a model-agnostic framework for enhancing existing traffic models.
Source: §1 · p. 2Data-collection demands have kept the authors from exploring iterative improvement using updated human-preference datasets and reward models.
Source: §5 · p. 9
Abstract
from arXiv · showhide
In light of the challenges and costs of real-world testing, autonomous vehicle developers often rely on testing in simulation for the creation of reliable systems. A key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge, an aspect that has proven challenging due to the need to balance realism and diversity. This works aims to address this by developing a framework that employs reinforcement learning with human preference (RLHF) to enhance the realism of existing traffic models. This study also identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models. To tackle these issues, we propose using human feedback for alignment and employ RLHF due to its sample efficiency. We also introduce the first dataset for realism alignment in traffic modeling to support such research. Our framework, named TrafficRLHF, demonstrates its proficiency in generating realistic traffic scenarios that are well-aligned with human preferences, as corroborated by comprehensive evaluations on the nuScenes dataset.
1 Introduction
TrafficRLHF addresses the difficulty of capturing human judgments of realism and incorporating them across diverse traffic models. It combines human feedback and RLHF to refine existing models, with a dataset and reward model designed for realism alignment.
1 Introduction
- Realistic traffic models are important for simulation testing, but balancing realism and diversity and aligning models with human knowledge remain challenging.Large-scale real-world testing carries significant expenses and risks.Source: §1 · p. 1
- TrafficRLHF is a three-stage framework that uses human feedback and an autoregressive backbone to enhance existing traffic models.The approach is intended to address preference capture and unify diverse simulation models.Source: §1 · p. 1
- TrafficRLHF reduces unrealistic collisions or off-road driving by up to 80% and uses a three-stage process of feedback collection, reward-based ranking, and model fine-tuning.Experiments on nuScenes confirm its ability to generate realistic trajectories aligned with human preferences.Source: Figure 1 · p. 2
- The work introduces a realism-alignment dataset, a reward model for human-preferred realism, and a model-agnostic RLHF framework for existing traffic models.Source: §1 · p. 2
2 Related Work
Prior traffic simulation methods include rule-based, learning-based, and safety-critical scenario-generation approaches, but producing scenarios aligned with human perceptions of realism remains a gap. The paper positions human feedback and RLHF as a way to address that gap.
2 Related Work
- Rule-based methods constrain behavioral variety, while learning-based models may not support customized behaviors or capture the full range of realistic scenarios.Safety-critical scenario generators also often struggle to align with human perceptions of realism.Source: §2 · pp. 2–3
- Traffic simulation still lacks a way to produce scenarios closely aligned with human perceptions of realism and to integrate preferences seamlessly.Source: §2 · p. 3
- RLHF is part of human-in-the-loop learning, and human feedback can be valuable when engineering a reward is challenging or costly.Prior LLM work, including InstructGPT, illustrates using demonstrations and preferences to tune large models.Source: §2 · p. 3
- The proposed approach uses video preferences to reduce data-collection and feedback time while producing large volumes of labeled data.The paper presents this as useful for realism, which lacks explicit formulations.Source: §2 · p. 3
3 TrafficRLHF
TrafficRLHF collects human judgments of traffic realism, trains a reward model on preferred scenarios, and uses that model to fine-tune traffic models. Its components are designed to work across model architectures while retaining the original training objective.
3 TrafficRLHF
- TrafficRLHF has three components: collecting human preferences, training a realism reward model, and fine-tuning traffic models to align with that model.The framework is intended to support a wide range of traffic models.Source: §3 · p. 3
3.1 Traffic Simulation Formulation
- A traffic scenario represents vehicles’ states and controls over time, with the next state computed from the current state and control under unicycle dynamics.The decision-relevant context includes local semantic maps and current and preceding vehicle states.Source: §3.1 · p. 3
3.2 Human Feedback Collection
- Annotators compare videos of scenarios generated from the same context; they may select the most realistic from a set or indicate that none is realistic.Selecting the best of N generated scenarios yields N − 1 preference pairs; if none appears realistic, ground truth is preferred.Source: §3.2 · p. 4
3.3 Reward Modeling
- The reward model is trained from human-labeled scenario preferences and uses an iterative architecture that accepts arbitrary sequence lengths.Its reward is averaged over a scenario sequence.Source: §3.3 · p. 4
3.4 Traffic Model Fine-tuning
- Fine-tuning combines the traffic model’s original loss with a scaled reward-model term to avoid reward-score overfitting and task-performance degradation.The authors note that fine-tuning the full motion-prediction model can yield higher performance improvements.Source: §3.4 · p. 4
4 Experiments
The experiments assess TrafficRLHF’s realism gains, reward-model validation, and transfer across traffic models. CTG fine-tuning reduces failures, while component and reward-model ablations compare alternative training choices.
4 Experiments
- The experiments test whether TrafficRLHF improves traffic realism and whether its reward model generalizes to diverse traffic models.Source: §4 · p. 4
4.1 Experimental Setup
- Evaluation uses nuScenes, reports failure rate, realism deviation, and reward cost, and tests CTG, BITS, and TrafficGen.Traffic-model evaluation uses 100 randomly sampled validation scenes.Source: §4.1 · p. 5
4.2 Experimental Details
- The preference dataset uses five CTG-generated scenarios per scene from 500 training scenes, with human selections converted into preferred-sample pairs.The reward model uses a CTG encoder with fully connected output layers.Source: §4.2 · p. 5
4.3 Reward Model Validation
- Reward-model validation uses 400 training and 100 validation preference units; accuracy converges with sample size, but variance broadens, so the 200-sample model is used for fine-tuning.The paper notes possible overfitting given the relatively small nuScenes dataset.Source: §4.3 · p. 5, Figure 2 · p. 6
4.4 Evaluation of Traffic Model
- CTG fine-tuning with the reward model decreases collision and off-road failures.Table 1 reports quantitative CTG results across guidance settings.Source: §4.4 · p. 6, Table 1 · p. 6
- Both BITS and TrafficGen improve on all three metrics after reward-model fine-tuning, though gains are smaller than for CTG.Fine-tuning was restricted to the models’ motion-forecasting components.Source: §4.4 · p. 7, Table 2 · p. 7
4.5 Ablation Study
- Joint encoder-decoder fine-tuning performs best, while decoder-only tuning outperforms encoder-only tuning.The authors suggest the decoder may be more responsive because it generates CTG scenarios.Source: §4.5 · pp. 7–8, Table 3 · p. 7
- Feedback collected from TrafficGen and BITS yields a slight realism improvement, while a TrafficGen-encoder reward model yields larger improvements than cross-model feedback.BITS was excluded from the reward-model architecture experiment because its complexity impeded bi-level optimization.Source: §4.5 · p. 8, Table 4 · p. 8
5 Conclusion
TrafficRLHF, tested on nuScenes, generates trajectories closely aligned with human preferences and addresses challenges in capturing those preferences and unifying traffic models. The authors identify unimplemented fine-tuning and data-collection work as opportunities for further improvement.
5 Conclusion
- TrafficRLHF generates nuScenes trajectories closely aligned with human preferences and offers an RLHF-based framework to enhance a wide range of existing traffic models.The work also introduces the first dataset for realism alignment in traffic modeling.Source: §5 · p. 8
- TrafficRLHF does not incorporate recently proposed advanced diffusion-model fine-tuning, and generated scenarios for BITS and TrafficGen have not yet been fully collected and labeled.The authors suggest model-specific reward-model training and bespoke fine-tuning strategies could enhance results.Source: §5 · p. 8
- Iterative improvement with updated human-preference datasets and reward models remains unexplored because it requires intensive data collection [27].The authors state that this approach might further elevate performance.Source: §5 · p. 9
A Algorithm of Training in Details
Training and sampling follow each benchmark model's established procedures, with model-specific fine-tuning choices. All fine-tuning uses amended PPO for 70 epochs, and closed-loop scenes run for 10 seconds with replanning at 2 Hz.
A Algorithm of Training in Details
- CTG follows for training and sampling; its reward model uses a CTG encoder and a fully connected network with hidden dimensions 512-512-512-128-32 and Tanh activations, with α = 0.1.The same sampling methods are used across guidance applications.Source: A Algorithm of Training in Details · p. 11
- BITS follows for training and sampling, fine-tunes only its spatial goal network, keeps the goal-conditional policy fixed, and uses α = 0.5.The authors avoid policy-network fine-tuning because it would require a different strategy and could make comparison uneven.Source: A Algorithm of Training in Details · p. 11
- TrafficGen follows [16] for training and sampling, maintains fixed initialization during fine-tuning, and uses α = 0.3.Fixed initialization is used for a fair comparison.Source: A Algorithm of Training in Details · p. 11
- All models use amended PPO from ColossalAI for fine-tuning over 70 epochs.Source: A Algorithm of Training in Details · p. 11
- Closed-loop scenes start from ground-truth driving logs, roll out for 10 seconds, and replan at 2 Hz after executing the initial few actions of each guided trajectory.The model is applied to all agents in a standard control loop.Source: A Algorithm of Training in Details · p. 11