Source-linked AI summary
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Mahmoud Selim, Cristina Cipriani, Karl Henrik Johansson
TL;DR
In interactive offline driving, ego trajectories can be supported marginally yet poorly supported when paired with logged surrounding-agent behavior. ICDP controls this interaction-level shift through contrastive estimation, and evaluations report consistent nuPlan improvements, particularly under reactive evaluation, alongside real-world truck feasibility.
Problem
Ego trajectories can remain well supported marginally while losing support when paired with the surrounding-agent evolution in logged interactions.
Source: §1 · p. 2, §4.2 · p. 5Method
Results
ICDP consistently improves over behavior cloning and unconstrained offline RL on nuPlan, particularly under reactive evaluation, reaching 72.87 on Test14-Hard reactive evaluation versus the strongest prior result of 70.42.
Source: §5.1 · p. 8, §6 · p. 9Takeaways & Limitations
Real-world deployment on a full-scale autonomous truck demonstrates feasibility across interactive and nominal driving conditions.
Source: §6 · p. 9, §5.3 · p. 9
Abstract
from arXiv · showhide
Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/
1 INTRODUCTION
Learning-based and generative planners can reproduce realistic, diverse behavior, but imitation alone does not optimize long-horizon driving objectives. ICDP addresses offline RL’s interaction-support gap by constraining policy improvement against logged multi-agent interactions, with evaluations reporting gains in challenging scenarios.
1 INTRODUCTION
- A candidate ego trajectory can be well supported marginally yet poorly supported jointly with logged surrounding-agent behavior, a gap existing ego-action-space constraints do not address.The paper names this loss of interaction support interaction distribution shift (IDS).Source: §1 · p. 2
- ICDP decomposes joint-support degradation into ego support and residual interaction support, estimating the residual contrastively without explicit joint-density modeling or prediction of agent futures.Policy optimization also avoids reactive-simulator or learned-world-model rollouts.Source: §1 · p. 2
- ICDP combines a diffusion trajectory policy with chunk-level value learning and interaction-aware constraints to improve driving policies from fixed logs.The framework is designed to preserve support at the level of multi-agent interactions.Source: §1 · p. 2
- Closed-loop nuPlan and Interplan evaluations and real-world truck experiments report suppression of high-value, interaction-unsupported trajectories and improved performance in challenging scenarios.The supplied introduction does not provide quantitative results for these evaluations.Source: §1 · p. 2
2 BACKGROUND & RELATED WORKS
Prior work advances learning-based motion planning and reward-driven reinforcement learning, while offline RL addresses value errors from unsupported actions. ICDP focuses on interaction support when only the ego policy is optimized against logged traffic.
Learning-based and generative motion planning
- Imitation and generative planning progressed from rasterized representations to structured vector and transformer models, with diffusion planners adding multimodal trajectory generation.These approaches remain largely driven by demonstrated behavior, whereas ICDP uses reward-driven improvement from offline interaction data.Source: §2 · p. 2
Reinforcement learning for autonomous driving
- Driving RL methods use closed-loop interaction or learned dynamics for feedback, while ICDP improves policies from fixed logs without simulator interaction or model-generated traffic evolution during optimization.The passage also cites intent inference and multi-agent policy learning as interaction-aware RL approaches.Source: §2 · p. 3
Offline RL and support constraints
- Offline RL mitigates distribution shift and value overestimation with conservative values, behavior or support constraints, and in-sample policy extraction.These approaches have also been extended to diffusion policies, temporally extended actions, and autonomous-driving settings.Source: §2 · p. 3
- Unlike offline multi-agent methods that optimize multiple learning agents, ICDP optimizes only the ego policy and separates ego-trajectory support from conditional interaction support.Surrounding traffic is treated as logged environment evolution.Source: §2 · p. 3
3 PROBLEM FORMULATION
The formulation treats driving as a discounted MDP with temporally extended ego actions and chunk-level value estimates. It motivates controlling joint interaction support because ego-space regularization alone does not ensure compatibility with logged surrounding-agent evolution.
3 PROBLEM FORMULATION
- The driving problem is formulated as a discounted MDP, with a scene as state, primitive ego control as action, and reward, initial-state distribution, dynamics, and discount factor specified.The policy and critic operate on ego-action chunks rather than isolated controls.Source: §3 · p. 3
- A chunk-level critic estimates return for executing the complete action sequence, and its backup uses a delayed target critic after an H-step return.Conditioning on the complete sequence avoids applying an H-step return to a single-action critic; surrounding-agent trajectories remain observed environment evolution.Source: §3 · p. 3
- Behavior regularization limits shift in ego-action space but does not ensure that a candidate trajectory remains compatible with surrounding-agent evolution in the same interaction.Thus high marginal ego support does not by itself establish high joint support.Source: §3 · pp. 3–4
- The overview pairs a diffusion actor and pessimistic chunk-level critic with an interaction-support module that compares joint ego–agent and ego-only support.The residual score provides the IDS estimate used to constrain policy improvement.Source: Figure 1 · p. 4
4 METHODOLOGY
ICDP improves a diffusion-based driving policy from fixed interaction logs while constraining policy updates to preserve support for logged multi-agent interactions. It decomposes joint-support degradation into marginal ego-support shift and an interaction-specific component, then estimates the latter contrastively for policy optimization.
4.1 ICDP OVERVIEW
- ICDP combines a diffusion actor, chunk-level critics, and an interaction-support module to evaluate candidate ego trajectories for value and compatibility with logged agent evolution.The framework constrains policy improvement to remain supported by multi-agent interactions represented in the dataset.Source: §4.1 · p. 4, §4 · p. 4
4.2 INTERACTION DISTRIBUTION SHIFT
- Joint-support degradation separates into departure from the logged ego distribution and loss of compatibility with the surrounding-agent evolution in the same interaction.The interaction-specific component, IDS, can be substantial even when the candidate trajectory remains marginally well supported.Source: §4.2 · p. 5
4.3 CONTRASTIVE RECOVERY OF INTERACTION DISTRIBUTION SHIFT
- Subtracting an ego-only contrastive score from a joint-interaction score isolates IDS without explicitly modeling the conditional surrounding-agent density.The joint score captures marginal ego shift and ego–agent dependence, while the ego score captures marginal shift; their expectation also quantifies distributional change in surrounding-agent behavior.Source: §4.3 · pp. 5–6
4.4 INTERACTION-CONSTRAINED POLICY OPTIMIZATION (ICDP)
- The interaction constraint regularizes diffusion-policy optimization separately from ego-support shift and behavior cloning, with NSPG used to optimize the policy.The behavior-cloning term anchors the policy to demonstrated behavior, while the two support terms control marginal ego and interaction-level distribution shift.Source: §4.4 · pp. 6–7
5 EXPERIMENTAL RESULTS
The experiments compare ICDP with prior planners and controlled offline-RL baselines, analyze regularization and agent-context choices, and assess deployment on a full-scale truck. Across these evaluations, ICDP performs strongly in closed-loop benchmarks, while truck trials provide qualitative evidence across interactive and standalone driving.
5.1 MAIN RESULTS
- 78.34 versus 76.47 on Test14-Hard non-reactive evaluation, and 72.87 versus 70.42 reactively, were ICDP’s scores and the strongest prior results, respectively.On Test14-Random, ICDP scored 93.33 non-reactively and 86.34 reactively; its gains were preserved when surrounding agents responded to the ego vehicle.Source: §5.1 · p. 8
- ICDP yielded the strongest performance in five of six controlled settings and raised reactive scores over unconstrained Offline RL from 70.07 to 72.87 on Test14-Hard and 83.02 to 86.34 on Test14-Random.Table 2 reports comparisons among BC, unconstrained Offline RL, and ICDP averaged over ten checkpoints.Source: §5.1 · p. 8, Table 2 · p. 8
5.2 ANALYSIS OF INTERACTION-AWARE REGULARIZATION
- Moderate regularization performed best among tested values: ego-support performance improved from λE = 0.1 to 0.3 before declining at higher weights, and λI = 1 outperformed weaker and stronger settings.The reported ICDP configuration uses λE = 0.3 and λI = 1.Source: §5.2 · p. 8
- Average performance improved as the estimator’s surrounding-agent context increased from 16 to 48 agents, then decreased slightly from 82.40 to 81.97 at 64 agents.Scores average Test14-Hard and Test14-Random under both non-reactive and reactive evaluation.Source: §5.2 · p. 8
5.3 REAL-WORLD DEPLOYMENT
- Real-world multi-vehicle trials involving overtaking and lead-vehicle following provide qualitative evidence that ICDP executes interaction-aware maneuvers under real sensing and vehicle dynamics.The reported trials use varying traffic configurations.Source: §5.3 · p. 9
- Curved-road trials without nearby interacting vehicles illustrate nominal trajectory generation and execution, and the truck experiments provide qualitative evidence across interactive and nominal driving conditions.These tests use real road geometry, sensing, and vehicle dynamics.Source: §5.3 · p. 9
6 CONCLUSION AND FUTURE WORK
The conclusion presents ICDP as an offline RL framework that controls support shift in multi-agent interactions by separating ego-trajectory support from interaction-specific support. The appendix describes contrastive objectives that distinguish logged interactions from policy-generated pairings and isolate marginal ego support.
6 CONCLUSION AND FUTURE WORK
- ICDP combines interaction-aware support control, chunk-level value learning, and diffusion-policy optimization to improve long-horizon driving performance while preserving support under logged interactions.The conclusion frames online reinforcement learning as a natural next step for refining behavior through closed-loop interaction while retaining interaction-aware support control.Source: §6 · p. 9
A.1 CONTRASTIVE OBJECTIVES
- The joint classifier contrasts logged ego–agent interactions with policy-generated ego trajectories paired with surrounding-agent trajectories from the same scene.The negative distribution preserves scene-conditioned ego and agent marginals while removing their dependence under the logged interaction distribution.Source: §A.1 · p. 13
- A second classifier operates only on ego trajectories to separate marginal ego support from interaction compatibility; its negative ego example is the candidate used in the negative joint interaction.The classifiers use binary cross-entropy objectives with balanced positive and negative sampling.Source: §A.1 · p. 13
A.2 PROOF OF THEOREM 1
The proof shows that matched candidate distributions let the difference between joint and ego contrastive scores isolate interaction-specific shift. Its expected value measures divergence between the conditional surrounding-agent distributions for logged and candidate ego trajectories.
A.2 PROOF OF THEOREM 1
- Using the same policy-generated candidate distribution in both classifiers cancels the proposal term, so their residual score difference recovers IDS at the population optimum.The proof assumes balanced class sampling and common support.Source: §A.2 · p. 14
A.3 KL INTERPRETATION OF INTERACTION DISTRIBUTION SHIFT
- The expectation of pointwise IDS under the logged ego’s conditional surrounding-agent distribution equals the divergence between agent-future distributions for logged and candidate ego trajectories.Pointwise IDS measures support change for a particular surrounding-agent realization.Source: §A.3 · pp. 14–15
B IMPLEMENTATION DETAILS
- The appendix identifies data construction, network architectures, and optimization as the implementation details provided next.Source: B IMPLEMENTATION DETAILS · p. 16
B.1 DATA AND TRAJECTORY REPRESENTATION
The dataset uses chunk-level transitions from nuPlan logs, with ego trajectories spanning 3.2 seconds. The reward combines route progress with driving-quality factors and invalidates collision or severe drivable-area-violation chunks.
B.1 DATA AND TRAJECTORY REPRESENTATION
- Ego trajectories contain 32 future steps at 10 Hz, forming a 3.2-second action chunk, while scene history covers 16 frames over the preceding 1.5 seconds.Trajectories use the current ego rear-axle frame; the next scene is expressed in the ego frame at the chunk end.Source: §B.1 · p. 16
- The processed dataset contains 744,832 transitions across 62 scenario types, including 12,108 recovery transitions, with 741,172 training and 3,660 validation transitions.Source: §B.1 · p. 16
B.2 REWARD FUNCTION
- The chunk reward combines speed-normalized route progress with route, lane, direction, speed, TTC, and comfort factors; poor quality reduces but does not reverse progress reward.Progress is clipped at zero when backward and capped above the local speed-limit distance; factors are bounded below.Source: §B.2 · p. 16
- A collision or drivable-area violation exceeding 0.5 m invalidates the chunk, assigns a terminal penalty, and disables bootstrapping; support penalties are applied separately from reward.Source: §B.2 · p. 16
B.3 NETWORK ARCHITECTURE
ICDP uses a diffusion actor, chunk-level critic, and separate ego and joint support classifiers. Policy optimization combines value guidance with interaction-support regularization while retaining a denoising behavioral anchor.
B.3 NETWORK ARCHITECTURE
- The actor encodes scene history and map information, then generates only the ego trajectory through a diffusion Transformer conditioned on the preferred route and diffusion time.Surrounding-agent trajectories are observed context, not prediction targets.Source: §B.3 · pp. 16–17
- A trajectory-attending critic uses five independent value heads to estimate chunk-level action value, with a pessimistic estimate formed from the ensemble mean and standard deviation.Source: §B.3 · p. 17
- Two classifiers estimate ego and joint interaction support; subtracting their logits yields the residual interaction score without a third network.The joint classifier also encodes surrounding-agent futures and their motion relative to the candidate ego trajectory.Source: §B.3 · p. 17
B.4 LEARNING AND OPTIMIZATION
- The critic learns chunk rewards with a delayed target, averaging lower-confidence-bound estimates from four target-policy trajectories for valid transitions.The reward includes route progress and driving-quality terms; invalid chunks disable bootstrapping.Source: §B.4 · pp. 17–18
- Actor optimization evaluates four decoded candidates using value and support estimates, while the diffusion denoising objective remains active as a behavioral anchor.Value guidance and interaction-support regularization begin after a 5,000-update warm-up.Source: §B.4 · p. 18
- Logged ego trajectories are positive examples and shared policy-generated candidates are negatives for both balanced classifiers; perturbed recovery examples are excluded from value guidance and support updates.The final configuration uses λE = 0.3 and λI = 1.Source: §B.4 · p. 18
B.5 QUALITATIVE SAMPLES AND DATASET COMPOSITION
The learned diffusion policy produces diverse trajectory samples across road geometries and traffic configurations. The offline dataset covers 62 nuPlan scenario categories with a long-tailed frequency distribution.
B.5 QUALITATIVE SAMPLES AND DATASET COMPOSITION
- Figure 4 shows ten independently sampled EMA-policy trajectories per scene without critic ranking, rejection, smoothing, or post-processing.The logged ego trajectory and surrounding-agent futures are included for reference.Source: §B.5 · p. 19
- The illustrated scenes span Singapore, Boston, and Las Vegas, including turning, lane-following, and curved-road scenarios.The examples illustrate trajectory-distribution diversity and geometric consistency, not closed-loop performance.Source: §B.5 · p. 19
- The offline dataset covers 62 recorded scenario categories whose frequencies vary substantially in a long-tailed distribution.Figure 5 separates more- and less-frequent categories for readability.Source: §B.5 · p. 19
C ADDITIONAL INTERPLAN EVALUATION
The additional InterPlan evaluation compares ICDP with representative planning methods where results are available, reporting overall and interaction-focused scenario scores.
C ADDITIONAL INTERPLAN EVALUATION
- InterPlan evaluation compares representative planning methods where results are available and reports an overall score plus three interaction-focused scenario categories.The supplied passage identifies Table 5 but does not provide its values or comparisons.Source: C ADDITIONAL INTERPLAN EVALUATION · p. 19
- Figure 4 shows ten independently generated ego trajectories per scene alongside the logged trajectory, preferred route, and surrounding traffic, without critic-based selection or post-processing.Source: Figure 4 · p. 20
- Figure 5 groups the processed offline dataset’s 62 recorded scenario categories into more- and less-frequent groups to make its long-tailed distribution legible.Source: Figure 5 · p. 21