PaperlayerView exact revision on arXiv

Source-linked AI summary

CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction

Zhengxu Tang, Guofeng Cui, Ziyu Gong, Xiaozhou Zhang, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao Wang

arXiv:2609.00242v1cs.CVcs.AIcs.CLcs.RO

TL;DR

Long-tail driving evaluation must test whether rare objects change feasible ego-vehicle actions, not only whether models recognize them. The paper introduces CoLT-Drive for this decision-level problem and KPA for knowledge-preserving small-VLM adaptation, with KPA reaching 60.8% pair accuracy on CoLT-Drive versus 50.3% for pretrained Qwen3-VL-2B and 32.4% for LoRA SFT.

  • Problem

    Long-tail driving benchmarks do not isolate whether rare objects change the feasible high-level action space for the ego vehicle.

  • Method

    The paper formulates decision-level affordance prediction and introduces CoLT-Drive plus KPA, combining controlled interventions with structured prompting, SLERP merging, and regime-aware LoRA experts.

  • Results

    60.8% pair accuracy on CoLT-Drive was achieved by KPA, versus 50.3% for the pretrained Qwen3-VL-2B baseline and 32.4% for LoRA SFT.

  • Takeaways & Limitations

    Robust long-tail driving requires grounding rare-object semantics in structured action affordances.

  • Takeaways & Limitations

    CoLT-Drive is a controlled image-level diagnostic and does not evaluate physical dynamics, closed-loop feedback, continuous trajectories, or real-world safety.

Abstract

from arXiv · show

Long-tail autonomous driving failures are often framed as rare-object recognition errors. We argue that this view is incomplete: the decision-critical question is not only whether a model recognizes an unusual object, but whether it infers how that object changes the ego vehicle's feasible high-level actions. We formalize this problem as decision-level driving affordance prediction, where a model maps a front-view image, ego-motion history, and navigation command to a structured longitudinal--lateral meta-action. To evaluate this capability, we introduce CoLT-Drive, a 3,536-sample counterfactual long-tail benchmark that inserts rare objects into otherwise fixed driving scenes and measures whether models predict acceptable action pairs. To improve deployable small VLMs, we propose KPA, a knowledge-preserving adaptation framework that combines structured perception-to-decision prompting, SLERP-based expert merging, and RegMoE, a regime-aware LoRA mixture-of-experts module. KPA preserves the pretrained model's open-world knowledge while allocating lightweight adaptation capacity to different driving decision regimes. Experiments on an in-domain driving split and CoLT-Drive show that KPA achieves 60.8\% pair accuracy on CoLT-Drive, outperforming the pretrained Qwen3-VL-2B baseline (50.3\%) and LoRA SFT (32.4\%) while maintaining competitive in-domain accuracy. Our benchmark and code are available at https://huggingface.co/datasets/tangzx2024/CoLT-Drive and https://github.com/tangzhengxu/CoLT-Drive.

1 Introduction

Long-tail driving failures involve more than recognizing rare objects: models must infer how those objects alter feasible ego-vehicle actions. CoLT-Drive evaluates this decision-level affordance capability and KPA adapts small VLMs while preserving pretrained knowledge.

  • Motivation: Rare-object recognition is insufficient because different unusual objects can imply different feasible driving responses.Examples include cautious passing, slowing with lateral adjustment, stopping, or rerouting.
  • Problem formulation: Decision-level driving affordance prediction maps rare-object semantics to structured longitudinal–lateral meta-actions.The representation targets high-level decisions rather than steering, throttle, or braking values.
  • Benchmark motivation: Existing benchmarks do not isolate whether rare objects change the feasible high-level action space under controlled interventions.Nominal driving logs can be dominated by frequent behaviors, leaving rare-object action reasoning untested.
  • Contributions: 3,536 reviewed samples in CoLT-Drive vary rare-object identity or position while preserving the surrounding driving context.The benchmark is constructed from fixed scenes and provides acceptable high-level action pairs.
  • Scope: CoLT-Drive is a controlled diagnostic benchmark rather than a simulation of alternative trajectories, causal effects in real traffic, or closed-loop safety.Its meta-action accuracy should not be interpreted as a measure of closed-loop driving safety.
  • Contributions: KPA combines structured prompting, SLERP-based merging, and regime-aware RegMoE adaptation to address specialization–retention tension in small VLMs.The framework is designed to preserve open-world knowledge while adapting to driving decision regimes.

2 Related Work

Related work spans driving VLMs and VLAs, long-tail evaluation, affordance grounding, and parameter-efficient adaptation. CoLT-Drive differs by targeting structured high-level driving affordances while addressing knowledge retention during adaptation.

  • Driving VLMs and VLAs: Driving VLMs and VLAs connect visual observations, language instructions, reasoning, planning, and driving decisions.Prior systems include language-supervised scene understanding, action-conditioned policies, and end-to-end multimodal driving approaches.
  • Long-tail driving evaluation: Existing long-tail resources evaluate corner-case detection, VLM understanding, driving generalization, and VLM reliability.These benchmarks address important driving evaluation dimensions but are presented as distinct from CoLT-Drive’s affordance focus.
  • Affordance grounding: Affordance-grounding research links observations to feasible actions, while driving VLA systems connect language-level reasoning with trajectories or action decisions.CoLT-Drive instead uses structured high-level meta-actions rather than low-level motor commands or continuous trajectories.
  • Knowledge-preserving adaptation: Parameter-efficient fine-tuning reduces adaptation cost but does not by itself prevent knowledge loss.The related work identifies forgetting as a concern in model adaptation, including driving-specific VLM adaptation.

3 Task: Affordance-Grounded Meta-Action Prediction

The task predicts an immediate structured meta-action from visual context, ego-motion, and navigation information. It evaluates whether models translate object semantics, spatial layout, and context into acceptable high-level longitudinal–lateral decisions.

  • Inputs and outputs: The model takes a front-view image, recent ego-motion history, and navigation command as inputs.These inputs are represented as I, E, and n in the task formulation.
  • Inputs and outputs: The output is one longitudinal action paired with one lateral action.Longitudinal choices include slowing, yielding, creeping, or stopping; lateral choices include keeping lane, nudging, or changing lanes.
  • Decision interface: Meta-actions capture immediate high-level responses rather than steering angles, control values, or terminal maneuvers.A distant blockage may initially prompt deceleration, with later stopping determined by subsequent replanning outside the single-frame decision.
  • Purpose: The task tests whether models transform object semantics, spatial layout, and context into a decision-level action pair.This positions the task between object recognition and low-level control.
  • Decision interface: Each scene can accept multiple safe longitudinal–lateral pairs, while unsafe directional choices are excluded.The reference is an acceptable set P(x), reflecting semantically equivalent safe actions.

4 The CoLT-Drive Benchmark

CoLT-Drive is a controlled benchmark that inserts rare objects into fixed driving scenes and evaluates acceptable action-pair revisions. It organizes interventions by affordance and uses reviewed labels across complementary contextual variants.

  • Benchmark design: 3,536 reviewed samples from 29 base driving scenes evaluate rare-object semantics through acceptable longitudinal–lateral action pairs.The benchmark measures affordance prediction rather than merely recognizing the inserted object.
  • Benchmark design: Counterfactual interventions preserve road geometry, viewpoint, ego-motion, navigation, and lead-vehicle context while varying inserted rare objects.The controlled design isolates whether changing the object changes the feasible high-level action space.
  • Affordance-oriented construction: Five affordance categories test distinct action constraints, including responses to living entities, hazards, full blockages, and false positives.The construction emphasizes action revision, including avoiding overreaction to low-risk distractors.
  • Counterfactual versions: Each intervention has vfull and vclean versions that provide complementary naturalistic and reduced-shortcut evaluation regimes.vclean removes moving vehicles so the inserted object and road geometry become more central to the decision.
  • Labels: Three reviewers independently annotate 1,768 vfull images, adjudicating disagreements into consensus acceptable-action sets.vclean labels extend adjudicated vfull labels with actions made safe by vehicle removal.
  • Validity checks: Human ratings support synthetic-image realism and plausibility, while model rankings correlate strongly between CoLT-Drive and a real control set.The reported Spearman’s ρ is 0.93 for the cross-model ranking correlation.
  • Evaluation: Pair accuracy evaluates whether a parsed prediction belongs to the acceptable action-pair set.Overall and per-category pair accuracy are reported.

5 Knowledge-Preserving Adaptation

KPA addresses long-tail driving as a semantic-action grounding problem by combining structured action supervision, knowledge-preserving expert merging, and regime-aware lightweight adaptation. Its RegMoE module routes low-rank corrections across coarse driving decision regimes while retaining a frozen merged backbone.

  • KPA targets semantic-action grounding, requiring driving-specific meta-action behavior without discarding open-world knowledge for rare-object affordance reasoning.
  • Structured Perception-to-Decision Interface: The structured interface prompts models to identify decision-relevant objects, infer constraints, and output a canonical longitudinal-lateral action pair.Only final action tokens are supervised because CoLT-Drive evaluates the decision interface rather than explanation fluency.
  • SLERP-Based Expert Merging: SLERP interpolates the pretrained model and a driving expert to balance driving specialization with retention of broader visual and commonsense knowledge.The interpolation follows the angular direction between matched tensors, with α controlling proximity to the driving expert or pretrained model; unstable cases fall back to linear interpolation.
  • Regime-Aware Adaptation: RegMoE adds multiple low-rank LoRA experts over the frozen merged backbone, using driving decision regimes to route lightweight adaptation capacity.The module keeps the SLERP projection as the main computation path while activating different residual directions for different regimes.
  • Regime-Aware Adaptation: The three coarse regimes are Routine, Maneuver, and Reactive, derived from the in-domain 72-label action ontology for adaptation-time routing.At inference, regime labels and forced expert assignments are unavailable, so routing depends exclusively on input hidden states.
  • Training Procedure: KPA trains a driving LoRA expert, merges it with the pretrained model using SLERP, then freezes the merged backbone while training RegMoE adapters without CoLT-Drive samples.The adaptation objective combines weighted action-token supervision with routing regularization for balanced usage and distinct expert mixtures.

6 Experiments

Experiments evaluate small VLM adaptation on nominal driving and counterfactual long-tail affordance prediction, using parsed action-pair accuracy and robustness analyses. KPA improves counterfactual performance while exposing category-specific errors, context sensitivity, and component contributions.

  • Main Results: KPA reaches 60.8% CoLT-Drive accuracy, versus 50.3% for pretrained Qwen3-VL-2B, 32.4% for LoRA SFT, and 53.0% for SLERP.LoRA SFT raises in-domain accuracy from 12.4% to 58.3% but degrades counterfactual accuracy, whereas KPA maintains competitive nominal performance.
  • Affordance Category Analysis: 82.5% living-entity and 69.7% full-block accuracy show KPA’s strongest gains in safety-critical categories, while nonliving entities and false positives remain weaker.Compared with SLERP, these increases are from 44.0% to 82.5% and from 41.0% to 69.7%, respectively.
  • Affordance Category Analysis: KPA is not globally conservative: it keeps speed on 48.3% of nominal samples, matches the 13.6% ground-truth stop rate at 13.8%, and scores 52.8% versus 24.7% for a constant slow-down, keep-lane policy.On CoLT-Drive, predictions shift primarily toward slow down at 84.7%, while stop and yield each account for 0.7%.
  • Ablation Study: The structured prompt is the largest ablation contributor, with direct action queries reducing accuracy by 16.29 points; removing SLERP, RegMoE, or routing bias reduces it by 4.98, 3.00, and 4.75 points.These results indicate that both the merged initialization and regime-conditioned routing contribute to counterfactual performance.
  • Ablation Study: RegMoE reaches 59.72 ± 0.88 vfull accuracy versus 55.52±0.99 for a parameter-matched rank-48 single LoRA across four seeds.The paired improvement is +4.20±0.58 in every run.
  • Context-Cue Analysis: KPA’s vclean−vfull gap is +1.36, while LoRA SFT drops from 34.6% to 30.2%, indicating more stable grounding when background vehicles are removed.The paired design tests reliance on surrounding context versus the inserted object’s affordance.

7 Conclusion

The paper reframes long-tail autonomous driving as decision-level affordance prediction and introduces CoLT-Drive to isolate rare-object-induced action changes. KPA combines structured prompting, SLERP merging, and regime-aware LoRA experts to improve long-tail prediction while preserving open-world knowledge.

  • Long-tail driving requires inferring how rare objects change feasible longitudinal and lateral actions, not merely recognizing them.
  • CoLT-Drive evaluates affordance-induced action revision through controlled rare-object interventions in fixed driving scenes.
  • LoRA SFT improves in-domain accuracy yet degrades CoLT-Drive performance, exposing a tension between nominal adaptation and long-tail robustness.
  • KPA combines structured perception-to-decision prompting, SLERP-based conservative merging, and regime-aware LoRA experts.
  • Robust long-tail driving requires grounding rare-object semantics in structured action affordances.

8 Limitations

The benchmark is an image-level diagnostic with abstract meta-actions, generated counterfactuals, and limited adaptation scope. Its results do not establish closed-loop feasibility, real-world safety, or uniform generalization.

  • CoLT-Drive does not simulate object dynamics, ego-action feedback, sensor noise, or downstream planner-control interactions.
  • Abstract meta-actions omit trajectory feasibility, vehicle kinematics, comfort, and multi-agent interaction.
  • Generated images and acceptable action-pair labels may contain artifacts, imperfect placement, or residual subjectivity despite review.
  • KPA is evaluated in a small-VLM setting, with non-uniform category gains and coarse regime routing.
  • CoLT-Drive is not a substitute for trajectory-level simulation, closed-loop evaluation, and platform-specific validation.

A Benchmark Construction Details

CoLT-Drive is built by varying rare obstacles and positions within fixed scenes, then labeling acceptable high-level action pairs with review. Its construction includes contrastive affordance tests, paired context conditions, and realism and reliability checks.

  • Construction overview: The five-stage pipeline selects base scenes, designs an obstacle taxonomy, generates counterfactuals, constructs vfull/vclean pairs, and labels acceptable actions with quality assurance.
  • Base scene selection: The benchmark uses 29 scenes: 9 from Alpamayo-R1 and 20 selected from nuPlan mini.
  • Obstacle taxonomy: Its 50 obstacle types are grouped by the action constraints they impose rather than by visual appearance.
  • Diagnostic contrast design: Diagnostic contrast pairs place visually similar objects in the same scene while assigning different actions when their physical affordances differ.
  • Position-sensitive action variation: Obstacle position changes acceptable lateral actions, with five representative types tested at left-biased, center, and right-biased locations.
  • vfull and vclean paired construction: vfull preserves surrounding traffic, whereas vclean removes background vehicles while retaining the target obstacle, road geometry, and static elements.
  • Action-pair labeling and review: Acceptable action-pair labels represent sets of safe or semantically equivalent high-level actions rather than single deterministic labels.
  • Annotation reliability: Exact agreement occurs on 78.51% of vfull samples, while 21.49% require adjudication.

B.3 Source-Stratified and Leave-Source-Out Evaluation

The evaluation controls for scene-source overlap while separating nominal driving performance from counterfactual long-tail affordance prediction. It also defines the structured action interface and scoring process used across models.

  • Source-Stratified Evaluation: The benchmark evaluates all 3,536 images, including 2,440 nuPlan-only and 1,096 Alpamayo-only images, with both vfull and vclean subsets.Source-stratified evaluation directly tests sensitivity to scene source when model pretraining inventories are unavailable.
  • Leave-Source-Out Evaluation: KPA retains gains of 11.6, 31.0, and 8.0 points over the pretrained backbone, LoRA-SFT, and SLERP on nuPlan-only scenes.KPA is nearly invariant to scene source, scoring 60.6 versus 61.1 across the reported source-specific comparisons.
  • In-Domain Evaluation: The in-domain corpus contains 10,000 clips split into 9,002 training clips and 998 held-out test clips, while nominal evaluation uses 3,613 held-out samples.The nominal subset covers all 998 test clips, 26 ODD categories, and 72 action labels.
  • Evaluation Roles: CoLT-Drive provides controlled rare-object interventions, whereas the in-domain split measures ordinary driving meta-action prediction after adaptation.The separation tests whether nominally trained models generalize to counterfactual affordance prediction.
  • Action Interface: Predictions consist of one longitudinal and one lateral action, scored against an acceptable action-pair set that can contain multiple safe or semantically equivalent pairs.The interface excludes unsafe directional choices and uses immediate-response semantics rather than terminal maneuvers.
  • Scoring: The shared scoring pipeline extracts a committed decision, normalizes it with a text-only LLM, and reports invalid outputs separately from pair accuracy.Curious-VLA-3B has 1,607 invalid outputs containing multiple uncommitted candidate actions, so its score reflects interface compatibility as well as driving reasoning.

F Language-Side Knowledge Retention Analysis

The retention analysis tests whether adaptation preserves language-side situational driving reasoning while improving long-tail affordance prediction. KPA largely maintains retention, whereas direct LoRA SFT degrades it.

  • Retention Evaluation: The retention probe uses 1,292 DriveQA-T situational driving-reasoning questions with rotation-averaged 4-choice accuracy.The visual encoder is frozen across variants, so identical linear-probe visual accuracy is treated only as a sanity check.
  • Main Result: 79.1% retention leaves KPA only 0.5 percentage points below the pretrained backbone while CoLT-Drive accuracy rises from 50.3% to 60.8%.The reported result supports adaptation toward driving affordance prediction without substantially erasing pretrained situational reasoning.
  • Baselines: Direct LoRA SFT reduces retention from 79.6% to 74.1%, while SLERP remains 4.4 percentage points below the pretrained model.These comparisons establish the retention degradation associated with direct adaptation and its partial mitigation by SLERP.
  • Fisher-Guided Adapters: Fisher-guided adapter placement avoids knowledge-sensitive layers but does not improve over SLERP on CoLT-Drive.The result motivates assigning different adaptation directions to different driving decision regimes.
  • Action-Grounded GRPO: Action-grounded GRPO slightly improves in-domain accuracy but leaves CoLT-Drive accuracy nearly unchanged, so it remains exploratory.The method optimizes parsed action correctness but introduces additional optimization sensitivity.
  • RegMoE: RegMoE uses driving decision regimes as an explicit routing signal rather than a general-purpose adapter-mixture rule.The final pipeline trains a driving expert, merges it with the pretrained model using SLERP, then trains RegMoE adapters on the frozen merged backbone.

I Seed Variance and Parameter-Matched Comparison

The parameter-matched comparison isolates RegMoE’s routing mechanism from capacity and run variance. Efficiency profiling shows substantial accuracy recovery with lightweight overhead, but not established automotive deployment performance.

  • Seed Variance: RegMoE outperforms the parameter-matched E1a adapter under every seed, with a mean paired gap of 4.20 points and a 95% confidence interval of [3.27, 5.13].Seed-to-seed variation is approximately 1 point per method, smaller than the paired gap.
  • Parameter-Matched Comparison: The comparison supports the complete RegMoE mechanism—action-label-guided routing with multiple low-rank experts—relative to an equally sized nonrouted adapter.The authors do not attribute the improvement to expert multiplicity independently of routing supervision.
  • Efficiency Profiling: RegMoE adds 19.8M parameters, or 0.93% of the 2.13B-parameter backbone, with approximately 0.23 GB peak-memory overhead over the merged backbone.Its fused implementation reaches 22.4 tokens/s under the reported H100 batch-size-one setup.
  • Accuracy–Memory Trade-off: KPA improves CoLT-Drive accuracy from 50.3% to 60.8% with an approximately 4.5 GB footprint, while Qwen3-VL-8B reaches 65.5% with 16.8 GB.KPA recovers approximately 69% of the accuracy gap between the zero-shot 2B and 8B models.
  • Deployment Boundary: H100 profiling quantifies relative overhead but does not establish direct in-vehicle deployability on an automotive accelerator.The reported measurements use the paper’s H100 profiling setup rather than an automotive platform.
  • Action Distributions: KPA’s longitudinal predictions shift mainly from keep speed to slow down on CoLT-Drive, while its nominal stop rate is 13.8% versus the 13.6% ground-truth rate.The distribution is interpreted as intervention-conditioned rather than a fixed global conservative bias.

K.2 Error Taxonomy

The error taxonomy separates visual recognition, affordance mapping, spatial direction, and interface failures. KPA reduces affordance-mapping errors but leaves lateral-direction errors prominent.

  • Taxonomy: The taxonomy distinguishes ordinary perception failures from the semantic-action grounding failures targeted by CoLT-Drive.Representative errors are grouped into four categories for analysis.
  • Error Meaning: Object-recognition errors reflect insufficient visual grounding, whereas affordance-mapping errors occur when recognized objects receive incorrect feasible-action implications.Direction errors concern spatial grounding, especially whether the model chooses the correct nudge direction.
  • Interface Failures: Invalid-format errors are interface failures directly targeted by the structured perception-to-decision prompt.The prompt standardizes the final longitudinal–lateral action output rather than evaluating explanation fluency.
  • KPA Failure Profile: KPA reduces affordance-mapping errors from 50.3% to 23.0% of failures, while direction errors rise to 59.1%.The profile is consistent with correct identification of the need to act but misjudgment of lateral direction.
Loading 2609.00242v1…