PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Action Autonomous Driving Agent with Language-based Memory

Kai Yan, Xiangyu Chen, Yulong Cao, Alex Naumann, Peter Karkus, Yan Wang, Jef Packer, Alex Schwing, Yuxiong Wang, Boris Ivanovic, Wenjie Luo, Marco Pavone

arXiv:2609.38641v1cs.CVcs.RO

TL;DR

Limited visual history leaves memory-dependent driving tasks, such as tracking all-way-stop arrival order, difficult to address with image inputs alone. AD-Memo adds language-based memory to future inputs and trains it with SFT and Da Capo; the authors report gains in driving, scene-question answering, and portable memory.

  • Problem

    High image-token costs limit VLA visual history, despite the need for past context in memory-dependent driving tasks.

    Source: §1 · p. 1

  • Method

    AD-Memo writes language memory alongside Chain-of-Thought for future inputs and trains with SFT and semi-closed-loop Da Capo.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    The authors report improved driving performance, better scene-understanding VQA, and memory portable to other models.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors present AD-Memo as a solution for memory-dependent autonomous-driving tasks.

    Source: §6 · p. 10

    The work focuses on around 10 seconds of memory, while longer-term needs such as remembering speed limits remain a potential direction.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Vision-Language-Action (VLA) foundation models have recently emerged as one of the prevailing solutions for autonomous driving, as they can utilize knowledge acquired during vision-language pretraining for accurate and interpretable driving. However, VLAs can take only a limited number of frames as visual input due to the high token cost of an image, which is problematic for memory-dependent tasks such as determining the arrival order at all-way stops and long-horizon driving scene understanding. Existing solutions use latent vector memories accessed through cross-attention, which are neither interpretable nor portable. In this paper, we propose AD-Memo, a general-purpose VLA driving agent with language-based memory. The agent outputs memory as an extension of its Chain-of-Thought (CoT) to record surrounding objects critical to driving; this memory becomes part of the agent's future input. We curate memory-based datasets and train VLAs with a two-stage recipe: Supervised Fine-Tuning (SFT) and \textit{Da Capo}, a novel semi-closed-loop Reinforcement Learning (RL) algorithm which uses trajectory-level advantage for memory and step-level advantage for driving, leading to better credit assignment. Across scenarios such as all-way stops and general driving, AD-Memo improves driving quality, enables better question answering on driving scenes, and produces plug-and-play memory for other models.

1 INTRODUCTION

Vision-language-action agents use vision-language pretraining to produce accurate, interpretable driving trajectories, but image costs limit their visual history. AD-Memo adds language-based in-episode memory and trains it for driving, scene understanding, and memory portability.

1 INTRODUCTION

  • High image-token and attention costs restrict VLA history, although memory-dependent tasks include all-way-stop arrival order, occluded objects, and long-horizon scene understanding.For a four-camera VLA, extending history from 0.4 to 4 seconds raises input length from 3,200 to 32K tokens and quadratic attention cost by 100×.Source: §1 · p. 1
  • Prior latent-vector memories lack interpretability and portability, while language in-context examples do not retain critical events within an episode.These shortcomings motivate brief, explainable, portable in-episode language memory.Source: §1 · pp. 1–2
  • AD-Memo extends Chain-of-Thought with language memory that enters future inputs, trained through SFT and Da Capo semi-closed-loop RL.The paper evaluates it across driving scenarios, scene understanding, and portable memory for other models.Source: §1 · p. 2
  • Da Capo replays ground-truth driving states while using self-generated memory and assigns distinct reward handling to driving and memory tokens.This combines open-loop driving with closed-loop memory to address exposure bias without costly environment simulation.Source: §1 · p. 2
  • The authors report that AD-Memo improves driving performance, aids scene understanding, and provides plug-and-play memory for driving-related reasoning tasks.The introduction describes datasets built with rule-based and agentic frameworks for task-specific and general driving.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define the agent’s inputs and tasks, distinguish open-loop from closed-loop reinforcement learning, and introduce GRPO as the policy-optimization framework.

Driving Vision-Language-Action (VLA) Models

  • A driving VLA receives past trajectory, chronologically ordered camera images, and language, then produces Chain-of-Thought and a future trajectory.The primary driving objective is to reduce the difference between the expert and predicted future trajectories.Source: §2 · p. 3
  • The paper also considers memory writing, which generates portable language memory from an image and past memory, and VQA about driving history.VQA uses visual input, optional language, and a natural-language question.Source: §2 · p. 3

Open-Loop and Closed-Loop RL for VLAs

  • Open-loop RL replays expert inputs so current actions do not affect later states; closed-loop RL lets actions affect future states, while semi-closed-loop combines replayed observations with self-generated memory.The paper notes closed-loop RL can address exposure bias and support exploration, but real-environment simulation is difficult and costly.Source: §2 · p. 3

Group Relative Policy Optimization (GRPO)

  • GRPO optimizes a policy over grouped responses using advantages calculated relative to the group’s mean reward.The preliminaries also define a policy-ratio term and distinguish the current sampling policy from a fixed reference policy.Source: §2 · p. 3

3 METHODOLOGY

AD-Memo records objects relevant to future driving as language memory, then trains driving, memory writing, and memory-dependent reasoning through supervised fine-tuning and semi-closed-loop reinforcement learning.

3.1 LANGUAGE-BASED MEMORY

  • AD-Memo focuses on roughly 10 seconds of memory, a horizon described as sufficient for many scenarios but costly to cover with visual input alone.The paper notes that longer-term memory, such as remembering speed limits, may be needed.Source: §3.1 · p. 4
  • At driving keyframes, the agent writes memory within its Chain-of-Thought for future inputs; at clip end, it answers questions using recent memories and current visual input.During deployment it remains in driving mode while continuing to record important road objects.Source: §3.1 · p. 4
  • The authors curate task-specific all-way-stop and general-driving datasets from Physical-AV, each split into SFT, RL, and test sets.The datasets contain video clips, questions, and answers; the all-way-stop dataset targets a setting where memory is beneficial, while general driving tests generalizability.Source: §3.1 · p. 4

3.2 TWO-STAGE TRAINING RECIPE

  • SFT jointly trains driving while writing memory, memory-based VQA, and standalone memory writing.The training-data illustration distinguishes per-step driving and memory-writing entries from per-clip question-answering entries.Source: §3.2 · p. 4, Figure 2 · p. 5
  • Semi-closed-loop RL uses the agent’s own prior memory in each rollout while replaying ground-truth visual observations and past trajectories across rollouts.This addresses the mismatch between SFT inputs and inference-time self-generated memory without requiring fully simulated environmental changes.Source: §3.2 · p. 5
  • Da Capo computes driving advantages across rollouts at the same step, while memory advantages account for subsequent driving and question-answering rewards.Terminal VQA answers are evaluated using the final multiple-choice reward.Source: §3.2 · p. 5
  • Da Capo’s causal advantage excludes conditionally independent reward terms, so an early open-loop driving action does not contribute to a later memory advantage.The paper states that this improves credit assignment while retaining the expected on-policy gradient, without standard-deviation normalization.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments assess AD-Memo on all-way-stop and general-driving tasks, including scene-question answering and memory portability. They compare it with baseline models and examine Da Capo and memory construction.

4 EXPERIMENTS

  • The evaluation compares a base model, no-memory and CoT-as-memory variants, and Alpamayo2 Super 34B across 2,479 test clips and 50,533 all-way-stop keyframes.The experiments also evaluate 9,112 general-driving clips over 145,462 keyframes.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

4.1 ALL-WAY STOP

  • On all-way-stop scenes, AD-Memo after SFT and RL outperforms the other baselines, including Alpamayo2 Super 34B, on driving and question answering.In the illustrated case, remembering that the left car arrived first helps the agent stop and better match the ground-truth trajectory.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • Reference memory outperforms SFT self-generated memory, while RL improves all three tested memory types; CoT-as-memory helps question answering but has less potential than the other memory approaches.CoT-as-memory performs similarly to or slightly better than no memory on driving; the authors attribute its weaker memory potential to CoT often missing road objects.Source: §4.1 · p. 7

4.2 GENERAL DRIVING SCENES

  • On general-driving scenes, AD-Memo outperforms the other baselines; RL improves average and most-likely discrepancy but not minimum rollout discrepancy.The authors relate this difference to RL concentrating the policy on an inaccurate intention, unlike all-way-stop scenes where the passing intention is clear.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 MEMORY PORTABILITY

  • Generated AD-Memo memory improves GPT-5.6 Luna’s zero-shot VQA performance on LingoQA and WaymoQA when it receives only the final frame and memory.The reported portability evaluation uses an SFT-trained checkpoint to generate memory sequentially from prior memories and current frames.Source: §4.3 · p. 8, Table 4 · p. 9
  • Unmerged memory performs significantly worse than merged memory, with repeated object descriptions averaging 51.62 versus 16.57 words and 97–98% fewer temporal words.The authors also report that repeated statements about the stopped ego vehicle can mislead the answering model about traffic-light color.Source: §4.3 · pp. 8–9

5 RELATED WORKS

Related work frames VLA driving as combining VLM reasoning with end-to-end trajectory output. AD-Memo is positioned against driving-memory approaches and robotics agents with in-episode language memory.

VLA driving agents

  • VLA driving agents use VLM-backbone reasoning while producing end-to-end trajectories, unlike modular perception-decision-action systems, vision-action models, or driving VLMs that predict only high-level meta-actions.AD-Memo is built on Alpamayo.Source: §5 · p. 9

Driving agents with memory

  • Prior driving-memory approaches mainly use latent vectors accessed by cross-attention or structured prior-experience records added through retrieval-augmented prompts; AD-Memo is described as the first VLA driving agent with in-episode language memory.The paper characterizes latent-vector memory as not interpretable or portable in its motivation.Source: §5 · pp. 9–10

VLA agents with in-episode language memory

  • Robotics agents such as Notes-to-Self and MEM use in-episode language memory, but the cited works do not study autonomous driving or use RL for training.The paper presents its driving focus and semi-closed-loop RL algorithm as distinctions from those works.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents AD-Memo as a VLA driving agent with brief, explainable, portable in-episode language memory, trained with curated datasets and Da Capo. It reports improvements in driving, scene-question answering, and memory portability.

6 CONCLUSION

  • AD-Memo outputs language memory alongside CoT and appends it to future inputs; its training combines curated datasets with Da Capo, which replays ground-truth driving states while using self-generated memory.The conclusion characterizes the memory as brief, explainable, and portable.Source: §6 · p. 10
  • The paper reports that AD-Memo improves driving performance and VQA on driving scenes, and produces memory portable to other models.The conclusion describes it as a solution for memory-dependent autonomous-driving tasks.Source: §6 · p. 10

APPENDIX: VISION-LANGUAGE-ACTION AUTONOMOUS DRIVING AGENT WITH LANGUAGE-BASED MEMORY

A MATHEMATICAL PROOFS

The proof establishes that, without standard-deviation normalization, Da Capo and trajectory-level group centering have the same expected on-policy score-function gradient, though they assign credit differently. It also explains why deviation adjustment breaks that equivalence and why the authors retain it empirically.

A MATHEMATICAL PROOFS

  • Conditional independence between the first-step driving reward and the second-step continuation lets the proof remove that reward from the later memory score weighting without changing the expected gradient.The later continuation uses the shared prefix and mem1, but not the first predicted trajectory.Source: A MATHEMATICAL PROOFS · p. 16
  • Without standard-deviation normalization, Da Capo and trajectory-level group centering yield the same expected on-policy score-function gradient, but assign credit differently.The claim applies to every output block.Source: A MATHEMATICAL PROOFS · p. 17
  • Deviation adjustment generally breaks this equivalence through stepwise reward scaling and a denominator that can prevent independent reward terms from canceling.The denominator is estimated from the same rollout group and can depend on the sampled output.Source: A MATHEMATICAL PROOFS · p. 17
  • The authors report that standard-deviation normalization performs better than no standard deviation or a fixed constant in their driving setting, where reward differences vary substantially across scenes and components.They motivate adaptive normalization with driving-reward differences ranging from about 0.02 for a 0.1 m ADE difference to 1 for an ADE difference exceeding 5 m.Source: A MATHEMATICAL PROOFS · p. 18

B.1 EVALUATION METRICS

The appendix specifies trajectory, stopping, and scene-question metrics, benchmark evaluation setups, and dataset construction details. It also reports the co-waiting horizon distribution, a departure-order bias, and examples of memory-based arrival-order reasoning.

B.1 EVALUATION METRICS

  • The evaluation metrics cover trajectory displacement, stopping and departure success, roll-throughs, stop position and duration errors, and end-of-clip multiple-choice scene understanding.The benchmark descriptions define MCQ accuracy as an indicator of general scene understanding.Source: §B.1 · pp. 18–19

B.2 LINGOQA AND WAYMOQA

  • LingoQA evaluation uses 500 questions from 100 held-out scenarios, while WaymoQA evaluation uses 896 video questions verified by rule-based scripts.LingoQA correctness is averaged using Lingo-Judge against two reference answers per question.Source: §B.2 · p. 20

B.3 HYPERPARAMETERS

  • For the general-driving no-memory and CoT-as-memory baselines, the authors also train on the same memory-writing data as their method to align training dynamics.Source: §B.3 · p. 20

C.1.1 DATASET CURATION

  • The all-way stop clips require ego and at least one other car to stop within one second, with at most one car already stopped before the clip; the splits contain 7,436 SFT, 2,479 RL, and 2,479 evaluation clips.The total dataset has 12,394 clips and no overlap between splits.Source: §C.1.1 · p. 20, §C.1.2 · p. 21
  • Memory entries encode six sequential vehicle states, and the VQA task asks for chronological stopping order because departure-order questions are strongly biased.For scenes with three or more cars, the question asks which arrival permutation occurred.Source: §C.1.1 · p. 20

C.1.2 STATISTICS

  • Only 2.72% of clips have at least five seconds of co-waiting and none exceed ten seconds, so the co-wait proxy suggests five seconds covers over 97% of memory requirements.The figure tracks keyframes per clip and maximum co-wait duration.Source: §C.1.2 · p. 22, Figure 7 · p. 22
  • Ego stops earlier in 31.7% of paired stop events and later in 42.6%, but starts moving later in 66.5% of paired go events, motivating exclusion of departure-order questions.Source: §C.1.2 · p. 22

C.1.3 EXAMPLES

  • In the example, AD-Memo uses arrival times to conclude that the left car and ego arrived within 0.5 seconds of each other, whereas the CoT-as-memory rationale relies on the final visual sequence.AD-Memo gives the order as 3 = 2 > 1; the CoT-as-memory example answers A based on the left car having arrived first.Source: §C.1.3 · p. 23

C.2.1 DATASET CURATION

This section describes the curation of a challenging-scene driving dataset, including language memories and VQA, and the prompts used to generate temporally grounded memories and verify rationales against fixed ground truth.

C.2.1 DATASET CURATION

  • A decision-graph pipeline selects driving-relevant evidence, links it to the calibrated ego action, and uses graph-connected, cross-frame object tracking to curate memory.Grounded detections and VLM-derived semantics become candidate evidence; the filtering, graph construction, and linting steps help ensure memory focuses on evidence relevant to driving decisions. Memories are merged across frames to avoid repetitive or misleading descriptions and generated independently of VQA questions.Source: §C.2.1 · pp. 23–25

C.2.2 STATISTICS

  • The general-driving dataset contains 27,246 SFT-training clips, 9,088 RL-training clips, and 9,112 test clips; memory entries average 29.054 tokens.Figure 10 shows keyframe-count distributions, with most clips around 21 frames, corresponding to about 8 seconds of video.Source: §C.2.2 · p. 25, Figure 10 · p. 25

C.2.3 EXAMPLES

  • The examples pair an easy ego-action question with hard temporal questions, and show CoT memory describing scene changes relevant to the answers.The supplied examples include a green-lit rightward-curving intersection route, a red vehicle crossing right-to-left after constrained passage, and an ego vehicle proceeding straight after being stopped amid close clearances.Source: §C.2.3 · pp. 26–27

C.3 PROMPTS FOR DATA GENERATION

  • Easy VQA questions target temporal behavior or ego-object interaction, while hard questions require evidence across supplied times and cannot be answerable from the final image alone.Hard-question prompts use synchronized images and decision-graph snapshots to track actors and state changes over time, while treating images as authoritative when they conflict with graph snapshots.Source: §C.3 · pp. 28–29
  • The memory-merging prompt requires one output per 0.4-second timestamp, preserving supported anchor facts and locating changes at the earliest visibly supported frame.The first output establishes the scene; later outputs prioritize concise temporal changes while retaining useful continuity for visible dynamic subjects.Source: §C.3 · p. 30
  • The prompt directs stable tracking of physical subjects, qualitative ego-motion descriptions when reliable, and recovery of failed “N/A” anchors from images and temporal context.Outputs must be concise present-tense sentences, omit unsupported behavior and identifiers, and follow the requested JSON shape.Source: §C.3 · p. 30
  • The VQA prompt asks for a concise rationale supporting a fixed correct answer using only the current scene and earlier memory, with unsupported cases marked as unsupported.It prohibits changing the fixed answer or adding details absent from the supplied evidence.Source: §C.3 · p. 31

D.1 MORE QUALITATIVE RESULTS

A general-driving example shows language memory preserving a pedestrian’s presence and intention when the pedestrian is hard to see in the current input, with better predicted trajectories than the no-memory and CoT-as-memory baselines. On the all-way-stop dataset, the finetuned 8B model performs best among the 8B models and far outperforms Alpamayo 2 Super 34B.

D.1 MORE QUALITATIVE RESULTS

  • The example’s visual sequence tracks the ego vehicle and pedestrians approaching and crossing the crosswalk, with construction barriers narrowing the right edge.The described memory records the pedestrian’s presence and intention.Source: §D.1 · p. 31
  • In a crosswalk example, memory records a pedestrian’s existence and intention, and predicted trajectories outperform both no memory and CoT-as-memory.The example describes the pedestrian as occluded by a crosswalk sign in the current input.Source: §D.1 · pp. 31–32, Figure 14 · p. 32

D.2 SCALING UP MODEL SIZE

  • The finetuned 8B model performs best among the 8B models and far outperforms Alpamayo 2 Super 34B on the all-way-stop dataset.Table 6 reports results for 8B models on All-Way-Stop v4.Source: §D.2 · p. 32, Table 6 · p. 32

D.3 OVERFITTING CHECK ON VQA OF THE GENERAL DRIVING DATASET

The VQA check finds that training and testing on questions from the same distribution can overfit, while the designed memory performs best on a harder, differently distributed test set. Other checks report gains on common challenging scenes, show that decision graphs retain driving-relevant objects, and report high decode success rates; the RL figures show training dynamics for both datasets.

D.3 OVERFITTING CHECK ON VQA OF THE GENERAL DRIVING DATASET

  • Training and testing on VQA questions from the same distribution can overfit, while designed memory performs best on the harder test set with different distributions.The report notes that accuracy on same-distribution test questions can exceed 96%, even when training on the hard split.Source: §D.3 · p. 32, Table 7 · p. 33

D.4 WHICH SCENE DOES AD-MEMO BENEFIT THE MOST?

  • AD-Memo improves performance over the no-memory model on common scenes, especially traffic control compliance, lead vehicle following, and stopping for vehicles.Table 8 reports per-scenario open-loop trajectory error gains in meters; positive values indicate improvement.Source: Table 8 · p. 34

D.5 THE NECESSITY OF DECISION GRAPH

  • Compared with direct memory generation from the whole video, decision-graph memory retains driving-relevant details such as a stroller, crosswalk, yield sign, and left-side obstruction.The illustrated graph represents objects that affect driving decisions as nodes.Source: §D.5 · pp. 33–34, Figure 16 · p. 34

D.6 DECODE SUCCESS RATE

  • The model maintains a high decode success rate, with the model and baselines all reasonably close to 100%.When memory cannot be decoded at a step, the corresponding slot is set to null moving forward.Source: §D.6 · p. 34, Table 9 · p. 35

D.7 SEMI-CLOSED LOOP RL TRAINING CURVE

  • The reported RL training dynamics include original and Exponential Moving Average curves for the all-way-stop and general-driving datasets.The all-way-stop curves are shown in Figure 17 and the general-driving curves in Figure 18.Source: §D.7 · p. 34, Figure 17 · p. 35

D.8 ABLATION ON MEMORY HORIZON T

Across both datasets and tested memory horizons, designed memory is more effective than CoT-as-memory; the reported MCQ accuracy varies with horizon, with 8 seconds sufficient for all-way stops and 4 seconds generally fine for general driving. The additional ablations report robustness to the VQA reward weight and identify two SFT epochs as optimal in the tested setup.

D.8 ABLATION ON MEMORY HORIZON T

  • Designed memory is consistently more effective than CoT-as-memory across tested memory horizons on both datasets.Figures 19 and 20 show performance against inference-time memory horizon for all-way stops and general driving, respectively.Source: §D.8 · p. 35, Figure 19 · p. 37, Figure 20 · p. 37
  • For MCQ accuracy, an 8-second memory is sufficient on all-way stops and 4 seconds is generally fine on general driving; accuracy is highly correlated with horizon.AD-Memo’s performance scales largely with horizon, whereas CoT-as-memory does not.Source: §D.8 · p. 36
  • Longer horizons benefit minADE, minFDE, and corner distance most, while average and most-likely metrics do not scale with horizon.The authors hypothesize that more diverse memory induces more diverse driving behavior and better coverage.Source: §D.8 · p. 36
  • All-way-stop success-stop, success-go, and roll-through metrics do not necessarily track ADE, supporting the use of task-specific metrics alongside ADE.Source: §D.8 · p. 36

D.9 ABLATION ON λVQA

  • Performance is generally robust to λVQA, with the optimal value varying by metric; larger values do slightly better, which the authors relate to noisy cumulative future driving rewards.The authors identify this reward noisiness as motivation for Da Capo’s improved credit assignment and variance reduction.Source: §D.9 · p. 36

D.10 ABLATION ON SFT TRAINING LENGTH

  • Two SFT epochs are optimal in the reported test of training lengths up to four epochs on a separate 2,000-clip validation set.Source: §D.10 · p. 36

E COMPUTATIONAL RESOURCES

The appendix reports compute costs for data curation, SFT, RL, and evaluation, alongside training-length and λVQA checks.

E COMPUTATIONAL RESOURCES

  • Table 10 reports generally consistent performance across λVQA choices, with no single choice prevailing across all metrics.Source: Table 10 · p. 38
  • After 2 SFT epochs, trajectories move farther from ground truth and format-following degrades, with more invalid sampled rollouts.The figure attributes the degradation to exposure bias.Source: Figure 21 · p. 38
  • Generating a decision graph takes an average of 10.43 model calls, about 15K input tokens, and 5K output tokens.Source: E COMPUTATIONAL RESOURCES · p. 38
  • A 8900-step SFT run on the general driving dataset takes approximately 16 hours on one node with 8 NVIDIA H100 GPUs.The node also has 96 AMD EPYC 9R14 CPUs and 512GB memory.Source: E COMPUTATIONAL RESOURCES · p. 38
  • RL experiments use four compute nodes—two for rollout and two for training—and typically take 2–3 days.Each node is identical to an SFT node.Source: E COMPUTATIONAL RESOURCES · p. 38
  • Evaluation uses 16 single-GPU nodes and usually takes 4–6 hours.Each node is otherwise identical to an SFT node.Source: E COMPUTATIONAL RESOURCES · p. 38

F LIMITATION AND FUTURE WORKS

The paper studies memory horizons of about 10 seconds and leaves longer-term memory and action-expert integration for future work. Comparisons with other methods are constrained by training-resource, architecture, and dataset differences.

F LIMITATION AND FUTURE WORKS

  • The study focuses on memories lasting around 10 seconds; speed signs can affect driving for minutes to hours, and long-term memory remains an area for further investigation.The authors say a heuristic long-term-memory buffer could address this, but do not investigate it in depth.Source: F LIMITATION AND FUTURE WORKS · p. 38
  • Experiments use a VLA backbone without Alpamayo’s action expert because the expert requires extensive training after backbone finalization; testing whether the memory benefit persists remains future work.Source: F LIMITATION AND FUTURE WORKS · p. 38
  • Direct comparison with vector-based baselines is difficult, so the authors use training-matched baselines without memory and with CoT-as-memory.Most related papers lack released training code or checkpoints.Source: F LIMITATION AND FUTURE WORKS · pp. 38–39
  • COMPACT-VA assumes at least 20 frames and fixed-length input without padding, unlike the four-frame-pretrained Alpamayo base model; many test clips are too short for fair evaluation.Source: F LIMITATION AND FUTURE WORKS · p. 39
  • Other baselines are difficult to compare: MoMAD uses a ResNet backbone, ST-Occ is not a driving model, and Orion uses CARLA rather than the paper’s real-life clips.The dataset difference prevents reliable calculation of collision loss and boundary loss.Source: F LIMITATION AND FUTURE WORKS · p. 39

Related papers

Loading 2609.38641v1…