PaperlayerView exact revision on arXiv

Source-linked AI summary

BranchDrive: A Branch-Structured Dataset for Action-Conditioned Driving Prediction

Feeza Khan Khanzada, Sudarshan Sridhar, Jaerock Kwon

arXiv:2609.27275v1cs.RO

TL;DR

Most driving datasets observe only the behavior policy’s action and resulting future, limiting direct comparison of alternative ego decisions. BranchDrive pairs canonical histories with nominal and physically executed intervention futures, then evaluates action-conditioned predictions and offline selection. The structured model outperformed restricted baselines, and its selector improved offline ranking, while conservative execution remains unresolved.

  • Problem

    Most datasets record only the behavior-policy action and resulting future, limiting direct supervision for comparing alternative ego decisions.

    Source: §I · p. 1

  • Method

    BranchDrive pairs each canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures, audited for collection integrity.

    Source: §I · pp. 1–2, §VII · p. 14

  • Results

    M3 achieved scalar macro NMAE of 0.5036 and trajectory ADE of 2.2042 m; O-M3 raised balanced policy value from 0.5364 to 0.5704 and reduced normalized regret from 0.2674 to 0.1495 versus the frozen ActionPrior.

    Source: §VII · p. 14

  • Takeaways & Limitations

    BranchDrive supports action-conditioned short-horizon prediction and fixed-bank offline ranking, but conservative execution remains unresolved and closed-loop or real-world safety improvement is not established.

    Source: F. Experimental Conclusions · p. 11

    The ranking utility excludes TTC and binary safety events, and its normalization and weights encode particular trade-offs rather than a universal definition of driving safety.

    Source: C. Threats to Validity · p. 13

Abstract

from arXiv · show

Most autonomous-driving datasets record only the action executed by a behavior policy and the single future that followed, providing limited supervision for comparing alternative ego decisions. We introduce BranchDrive, a branch-structured CARLA dataset and benchmark that pairs one canonical pre-decision history with one nominal expert future and twelve physically executed intervention futures spanning acceleration, braking, and left- and right-steering policies at three magnitudes. Each intervention lasts 2.5 s and is followed by expert recovery. Following control-compliance, modality-completeness, replay-fidelity, and action-leakage audits, the frozen benchmark contains 606 independent branch groups and 7,878 associated trajectories. We evaluate prediction of six continuous short-horizon outcomes and a ten-step ego trajectory using action-only, history-only, structured, visual, multimodal, and privileged bird's-eye-view models. On the held-out test split, the structured history-and-action model achieves a macro normalized mean absolute error of 0.5036 and an average displacement error of 2.2042 m, significantly outperforming both restricted baselines. In full-information offline evaluation, its outcome-derived selector increases balanced policy value from 0.5364 to 0.5704 and reduces normalized regret from 0.2674 to 0.1495 relative to the frozen action prior. However, a validation-calibrated minimum-separation guard rejects every intervention, showing that conservative execution remains unresolved. BranchDrive therefore supports action-conditioned short-horizon prediction and fixed-bank offline decision evaluation, but does not establish exact causal effects, binary safety prediction, or closed-loop safety improvement.

I. INTRODUCTION

Most driving datasets record only the behavior policy’s action and resulting future, limiting direct comparison of alternative ego decisions. BranchDrive pairs a canonical history with physically executed alternatives and evaluates short-horizon prediction and fixed-bank offline selection.

  • Because observational datasets leave unchosen-action outcomes unobserved, models queried about alternative maneuvers must extrapolate, particularly in rare interactive situations.BranchDrive supports short-horizon prediction and offline comparison among thirteen predefined policies, but does not establish exact causal effects or closed-loop safety improvement.Source: §I · pp. 1, 3
  • Integrity and replay audits reduced 24,116 raw rollout attempts to a core benchmark of 606 independent branch groups and 7,878 associated trajectories.All thirteen trajectories from a branch group remain in the same train, validation, or test partition, and the branch group is the statistical unit.Source: §I · p. 2
  • The benchmark predicts six continuous intervention-horizon outcomes and a ten-step ego trajectory, with undefined quantities masked rather than assigned artificially safe values.Complete executed outcomes for each candidate also enable direct fixed-bank value and regret measurement without behavior propensities or a separately estimated action-value function.Source: §I · p. 2
  • M3 reduced macro normalized mean absolute error to 0.5036 and average displacement error to 2.2042 m, significantly outperforming action-only and history-only models.M5 had the lowest numerical scalar error, 0.5008, but its advantage over M3 was not statistically significant and its trajectory error was higher.Source: §I · p. 2
  • The outcome-derived selector raised balanced policy value from 0.5364 to 0.5704 and reduced normalized regret from 0.2674 to 0.1495, but its separation guard rejected every intervention.The unconstrained selector failed the predeclared minimum-separation non-worsening gate.Source: §I · p. 2
  • BranchDrive associates one canonical pre-decision history with a nominal expert future and twelve physically executed intervention futures, followed by expert recovery.The interventions use acceleration, braking, and left- and right-steering policies at three magnitudes; each lasts 2.5 seconds.Source: §I · pp. 1–2

II. RELATED WORK

Prior datasets and benchmarks support forecasting, planning, and closed-loop evaluation, but generally do not release persistent, decision-centered banks of executed alternative-action outcomes. BranchDrive complements these resources with aligned, same-context intervention futures.

A. Autonomous-Driving Datasets and Planning Benchmarks

  • Real-world driving datasets provide broad multimodal observations and annotations, but their supervision is primarily observational, recording the future under the logging policy.nuScenes, Waymo Open Dataset, and Argoverse 2 are cited as examples,,.Source: A. Autonomous-Driving Datasets and… · p. 3
  • nuPlan supports common-scenario closed-loop planner execution, but does not organize its dataset as persistent decision-centered branch groups with outcomes for every fixed intervention.BranchDrive instead maps one pre-decision context and specified candidate policy to a short-horizon trajectory and continuous outcome.Source: A. Autonomous-Driving Datasets and… · p. 3

B. Safety-Critical Closed-Loop Simulation

  • Closed-loop simulation platforms evaluate policies across scenarios and routes, while ChauffeurNet uses perturbed trajectories for robustness; neither provides a reusable within-context outcome table for physically executed ego policies.BranchDrive holds route, scenario, seed, and branch location fixed while varying the ego intervention and recording applied controls.Source: B. Safety-Critical Closed-Loop… · p. 3

C. Reasoning and Alternative-Action Supervision

  • Language-oriented driving datasets supervise explanations and high-level reasoning, but generally lack realized closed-loop trajectories and continuous outcomes for every candidate action in the same context.BranchDrive is complementary: its primary reference signal is the trajectory produced by executing each intervention.Source: C. Reasoning and Alternative-Action… · p. 3

D. Action-Conditioned World Models and Counterfactual Rollouts

BranchDrive separates action-conditioned prediction from learned future generation by storing simulator-executed futures for a shared decision context. Its branch group links a canonical history to a fixed bank of policies, outcomes, and quality metadata.

D. Action-Conditioned World Models and Counterfactual Rollouts

  • BranchDrive stores directly executed reference futures separately from learned predictors, allowing predictions to be compared against the same future bank.Unlike learned-simulator or method-specific adaptation approaches, it uses simulator execution as the reference.Source: D. Action-Conditioned World Models and… · p. 4
  • BranchDrive is a same-context finite-action decision benchmark, not an unrestricted offline reinforcement-learning dataset or evidence of exact real-world causal effects.Its distinguishing combination includes a shared canonical history, fixed temporally extended policies, direct simulator execution, and branch-group evaluation.Source: D. Action-Conditioned World Models and… · p. 4

III. BRANCHDRIVE DATASET

  • Each branch group pairs one canonical pre-decision history with physically executed futures under a fixed bank of candidate ego policies.The organization supports direct prediction of how the same scene evolves under different actions.Source: §III · p. 4

A. Branch-Group Representation

  • For each candidate policy, the branch-group unit pairs its structured encoding with the simulator-executed future trajectory, continuous outcome vector, target-availability mask, and metadata covering intervention-compliance, replay-fidelity, modality-completeness, and provenance.The common history contains the preceding 80 control commands at 20 Hz, excludes the control applied at the branch point, and receives the branch action separately to prevent intervention leakage.Source: A. Branch-Group Representation · p. 4
  • Each complete group contains one nominal expert future and twelve intervention futures, and all thirteen remain together in one data split.The shared group is the statistical unit.Source: A. Branch-Group Representation · p. 4

B. Two-Pass Collection

BranchDrive collects a nominal expert rollout, then replays the matched route under interventions before returning control to the expert. The fixed intervention bank uses temporally extended policies, whose control combinations make them complete maneuvers rather than isolated actuator tests.

B. Two-Pass Collection

  • The nominal expert rollout supplies the canonical history, nominal future, branch-point evidence, and original control trace needed by selected interventions.The expert also resumes control after the intervention, but is not treated as an onboard sensor-only policy.Source: B. Two-Pass Collection · p. 4
  • The intervention future measures a finite-duration action followed by expert recovery, not an indefinitely applied command or an exact counterfactual state match.Routes are replayed from the beginning rather than restored from a serialized simulator snapshot at the branch point.Source: B. Two-Pass Collection · p. 5

C. Intervention Policy Bank

  • The intervention bank comprises acceleration, braking, left-steering, and right-steering policies at three magnitudes, each active for 50 ticks before expert recovery.The policy families combine forced, original-nominal, and online-expert controls differently.Source: C. Intervention Policy Bank · p. 5, Table I · p. 5
  • Accelerate-straight changes lateral and longitudinal behavior, while braking retains online-expert steering and suppresses throttle.Steering policies reuse throttle and brake commands from the original nominal rollout, which become open loop after trajectories diverge.Source: C. Intervention Policy Bank · p. 5

D. Recorded Modalities

BranchDrive records synchronized visual, structured, semantic, control, and outcome streams, while separating privileged information from sensor-oriented inputs. Model inputs pair canonical histories with semantic action encodings, and target masks exclude unavailable outcomes from evaluation.

D. Recorded Modalities

  • Recorded streams include synchronized visual observations, structured vehicle and route measurements, privileged spatial representations, exact controls, and frame-level outcome evidence.The dataset also preserves intervention timing and control-source records for auditing.Source: D. Recorded Modalities · pp. 5–6
  • The privileged semantic BEV is a 256×256 ego-centered raster at 2 pixels per meter, covering approximately 128 × 128 meters.Its raw representation distinguishes road, sidewalk, lane markings, signs, traffic-light states, vehicles, and pedestrians.Source: D. Recorded Modalities · p. 6

E. Model Inputs and Targets

  • The action encoding represents policy family, signed magnitude, duration, control semantics, and recovery policy without a free candidate-index identifier.All candidate queries for a group use the same canonical nominal history.Source: E. Model Inputs and Targets · pp. 6–7
  • The model inputs include 16 time steps from six surround cameras or, separately, a 16-frame privileged semantic-BEV history; future and branch-specific data are excluded.Semantic BEV is privileged information rather than an onboard sensor input.Source: D. Recorded Modalities · p. 6, E. Model Inputs and Targets · p. 7
  • Each scalar target has an availability mask, and unavailable values are excluded from loss and metrics, particularly TTC when no relevant closing interaction exists.The future ego trajectory contains ten observations spanning the 2.5-second intervention interval.Source: E. Model Inputs and Targets · p. 7

F. Release Structure

The release separates source attempts, canonical records, quality-control results, model-ready data, and reproducibility assets. The frozen benchmark contains 606 branch groups, with groups—not their dependent futures or frames—as the effective sample units.

F. Release Structure

  • The release distinguishes raw physical attempts from filtered benchmark-valid records and stores compliance, replay, modality, eligibility, and exclusion diagnostics.Canonical manifests link each intervention to its nominal expert rollout.Source: F. Release Structure · p. 7
  • The model-ready package provides frozen splits, canonical histories, action matrices, masked targets, future trajectories, modality indices, schemas, and training-only normalization statistics.Joins use branch-group identity and canonical action key rather than directory order.Source: F. Release Structure · p. 7
  • Reproducibility assets include target-generation code, baselines, evaluation scripts, configurations, predictions, and checksums, with separate sensor-oriented and privileged input contracts.This separation guards against presenting semantic BEV, simulator actor state, or retrospective trigger information as onboard observations.Source: F. Release Structure · p. 7
  • 606 independent branch groups and 7,878 associated trajectories remain in the core benchmark after collection-integrity and replay-fidelity filtering.The partition contains 431 training, 88 validation, and 87 test groups; the group is the effective sample unit.Source: F. Release Structure · p. 7
  • The benchmark does not use binary collision, lane-departure, or unsafe-action labels to support its principal claims.Its headline supervision is restricted to six validated continuous targets and the future ego trajectory.Source: F. Release Structure · p. 7

IV. BENCHMARK PROTOCOL

The benchmark evaluates candidate actions against a shared pre-decision context, using a frozen set of branch groups and predicting both short-horizon outcomes and ego motion.

A. Evaluation population.

  • 606 branch groups and 7,878 trajectories form the frozen population, split by context into 431/88/87 train/validation/test groups.All thirteen candidates from each context stay in one partition, and branch groups are the independent unit for training splits, uncertainty estimates, and tests.Source: A. Evaluation population · p. 7

B. Model input.

  • Each candidate query pairs a shared canonical history with a separate 29-dimensional action encoding, excluding the applied branch action from history.The history contains 16 structured observations at 4 Hz and 80 past control commands at 20 Hz.Source: B. Model input · p. 8

C. Prediction tasks.

  • Models predict six 2.5-s scalar outcomes and a ten-step ego trajectory; undefined TTC targets are masked as unavailable rather than treated as safe.The scalar outcomes include minimum TTC and separation, displacement, jerk, acceleration, and yaw rate.Source: C. Prediction tasks · p. 8

D. Compared models.

The protocol compares action, history, structured, visual, multimodal, and privileged models under frozen training and test procedures. It evaluates prediction and fixed-bank ranking, while leaving binary safety selection and realized closed-loop improvement outside scope.

D. Compared models.

E. Training and test locking.

G. Offline candidate ranking.

  • Under the balanced profile, selection utility weights minimum separation, signed displacement, and aggregate comfort equally, with selected outcomes measured by direct lookup.The fixed-bank oracle is limited to the best of thirteen tested policies, not a global driving oracle.Source: G. Offline candidate ranking · p. 8

H. Scope.

  • The validated scope covers continuous outcome and trajectory prediction plus strict numerical fixed-bank ranking, not binary unsafe-action selection, causal identification, or closed-loop improvement.Replay-noise-equivalent action classes are also not evaluated.Source: H. Scope · p. 8, A. Research Questions and Experimental… · p. 9

A. Research Questions and Experimental Protocol

  • The experiments ask whether branches pass audits, joint history-and-action models improve prediction, predictions support better fixed-bank decisions, and a separation guard remains useful.The evaluation stops at offline comparison rather than claiming binary unsafe-action labels, replay-calibrated outcome equivalence, or realized closed-loop improvement.Source: A. Research Questions and Experimental… · pp. 8–9
  • Each query uses canonical nominal history and a separately encoded candidate action to predict six scalar outcomes and a ten-step ego trajectory.TTC is masked when no relevant closing actor makes it unavailable.Source: A. Research Questions and Experimental… · p. 9

F. Metrics and statistics.

B. Collection Integrity, Replay Fidelity, and Leakage

The benchmark’s strict audits define a 606-group replay-valid population, and the results show that history-plus-action prediction supports both outcome forecasting and fixed-bank offline comparison. The offline gains do not establish safe-action selection.

B. Collection Integrity, Replay Fidelity, and Leakage

  • Strict filtering yielded 606 core-valid groups; exact-control noncompliance drove major attrition, replay pair retention missed its 98% target, and leakage classifiers remained at chance.The core group-retention criterion was met; positive controls reached at least 96.2% accuracy, while tested real-data classifiers showed no detectable leakage.Source: B. Collection Integrity, Replay… · p. 9

C. Action-Conditioned Outcome and Trajectory Prediction

  • M3 reduces branch-level macro NMAE by 0.1049 versus M1 and 0.0791 versus M2, while M5 has the lowest numerical scalar NMAE and M3 leads trajectory metrics.M5’s scalar advantage over M3 is small and nonsignificant; TTC prediction is weaker than the other outcomes, and shuffling history or intervention inputs worsens M3 performance.Source: C. Action-Conditioned Outcome and… · pp. 9–10, Table V · p. 11

D. Complete-Bank Ranking and Offline Policy Evaluation

  • O-M3 raises balanced policy value from 0.5364 to 0.5704 and reduces normalized regret from 0.2674 to 0.1495 against ActionPrior.These results support strict numerical ranking within the fixed bank, not safe-action selection; O-M5’s policy-value improvement does not survive correction.Source: D. Complete-Bank Ranking and Offline… · pp. 10–11, Table VI · p. 11

E. Conservative Calibration and Deployment Boundary

Across the fixed action bank, BranchDrive supports stronger action-conditioned prediction and offline ranking, but conservative intervention selection remains unresolved. The experiments also bound these findings to audited branch groups and the evaluated diagnostics.

E. Conservative Calibration and Deployment Boundary

  • The unconstrained O-M3 selector failed the predeclared non-worsening gate after selected mean minimum separation fell 0.033 m below ActionPrior.Without repeated nominal outcome replays, the practical materiality of the difference cannot be established.Source: E. Conservative Calibration and… · p. 11
  • The 7.544 m separation guard achieved 99.62% candidatewise coverage but rejected every intervention, collapsing selection to nominal in all 88 validation groups.Its policy value matched ActionPrior at 0.55226; no safety or formal coverage claim is made.Source: E. Conservative Calibration and… · p. 11, A. Main Findings · p. 12

F. Experimental Conclusions

A. Main Findings

  • On the test split, 1,078/1,140 replay pairs passed core criteria, below the 98% target, while 87/95 complete banks passed, exceeding the 90% criterion.The frozen benchmark retains complete banks under its group-level rule.Source: A. Main Findings · p. 11
  • Leakage diagnostics detected no future-action identity, but this does not establish that every possible leakage channel was eliminated.The smallest Holm-adjusted permutation-test value was 0.288; positive controls reached approximately 96–100% accuracy.Source: A. Main Findings · p. 12
  • M3 reduced scalar macro NMAE from 0.6085 for action-only M1 and 0.5826 for history-only M2 to 0.5036, and trajectory ADE to 2.2042 m.All four principal comparisons against M1 and M2 remained significant after Holm correction; M5's scalar macro NMAE of 0.5008 was not significantly different from M3.Source: A. Main Findings · p. 12
  • O-M3 increased policy value from 0.5364 for the nominal ActionPrior to 0.5704 and reduced normalized regret from 0.2674 to 0.1495.Both differences remained significant after Holm correction with adjusted p = 0.0024.Source: A. Main Findings · p. 12

B. Modality Interpretation

Structured history and candidate-action information performed strongly on the evaluated prediction targets, while the visual and privileged comparators did not establish a consistent advantage. The findings are bounded by the replay, policy, horizon, and utility choices of this benchmark.

B. Modality Interpretation

  • M3 substantially outperformed both restricted baselines and achieved the lowest ADE, FDE, heading error, and speed error among evaluated models.Structured ego, route, and control histories represented recent vehicle dynamics, route-relative state, navigation, and past controls.Source: B. Modality Interpretation · p. 12
  • M5's aggregate scalar NMAE improvement over M3 was not significant, and its trajectory errors were higher; this does not establish that visual information is unnecessary.The comparison used frozen ResNet-18 features and temporal fusion.Source: B. Modality Interpretation · p. 12
  • Privileged semantic-BEV model M6 did not outperform M3 or M5, so it is one comparator rather than a guaranteed upper bound.Its scalar macro NMAE was 0.5283 and ADE was 3.1053 m.Source: B. Modality Interpretation · p. 12
  • Predicted candidate-outcome variance was compressed: median predicted-to-observed variance ratios were 0.647 for M3 and 0.558 for M5.This compression can reduce margins between competing actions despite reasonable average regression error.Source: B. Modality Interpretation · p. 12
  • The visual path took approximately 1.37 s per branch query, versus approximately 13 ms on CPU for the structured three-seed O-M3 selector.The visual implementation is suited to offline benchmarking, not high-frequency closed-loop use without efficiency changes.Source: B. Modality Interpretation · p. 13

C. Threats to Validity

  • Matched replay filters discrepancies but cannot restore exact latent simulator state, so comparisons are quality-controlled rollouts rather than exact causal treatment effects.CARLA and actor-controller state, traffic phases, and sensor timing are not restored from a serialized branch snapshot.Source: C. Threats to Validity · p. 13
  • The benchmark is conditioned on recognizable expert maneuvers, and its 606 contexts do not establish broad generalization or real-road safety.The retained contexts may underrepresent unrecognized strategies, aborted maneuvers, and nominal expert failures; all outcomes are produced in CARLA.Source: C. Threats to Validity · p. 13
  • Action families are complete maneuver policies, not isolated actuator perturbations, and conclusions cover only the fixed 2.5-s intervention followed by expert recovery.Longer-horizon consequences depend on the recovery policy, and the fixed duration may disadvantage actions needing longer execution.Source: C. Threats to Validity · p. 13

D. Intended Use

BranchDrive is intended for research on short-horizon prediction and finite-bank comparison, with complete branch groups and their quality metadata preserved. Its reported prediction and ranking gains do not establish safe or deployment-ready action selection.

D. Intended Use

  • BranchDrive supports research on action-conditioned outcome and ego-trajectory prediction, finite-bank comparison, uncertainty estimation, and decision-model analysis.The complete bank also supports controlled ablations and may serve as pretraining or diagnostic data before independent closed-loop validation.Source: D. Intended Use · pp. 13–14
  • Use complete branch groups and preserve group-disjoint splits, policy semantics, availability masks, provenance, and quality flags.The thirteen candidates and route-geometry variants should remain together in splits.Source: D. Intended Use · p. 14
  • The dataset is not intended to certify vehicles, establish exact causal effects, train unrestricted offline RL, or justify real-road deployment.Privileged simulator information must not be presented as onboard sensor input, and the fixed-bank oracle is not globally optimal.Source: D. Intended Use · p. 14
  • The strict 606-group release is the appropriate population for benchmark results; raw collection variants should be reported separately.The raw release retains retries, partial attempts, protocol variants, and excluded groups.Source: D. Intended Use · p. 14

VII. Conclusion

  • BranchDrive pairs one canonical history with one nominal future and twelve executed intervention futures, yielding 606 replay-valid groups and 7,878 associated trajectories.The branch group, not each trajectory, is the independent statistical unit.Source: §VII · p. 14
  • M3 achieved scalar macro NMAE of 0.5036 and trajectory ADE of 2.2042 m; O-M3 raised policy value from 0.5364 to 0.5704 and reduced normalized regret from 0.2674 to 0.1495.M5's numerical scalar error of 0.5008 was not significantly better than M3.Source: §VII · p. 14
  • The unconstrained selector failed the minimum-separation non-worsening criterion, while the calibrated guard rejected every intervention.Conservative uncertainty calibration for execution remains unresolved.Source: §VII · p. 14
Loading 2609.27275v1…