Driving on Memory

Christian Löwens; Thorben Funke; Alexandru Paul Condurache

Source-linked AI summary

Driving on Memory

Christian Löwens, Thorben Funke, Alexandru Paul Condurache

arXiv:2608.31029v1Aug 31, 2026cs.CVcs.LGcs.RO

TL;DR

The paper asks how much end-to-end driving benchmark performance depends on reacting to the current traffic scene rather than using static information. It replaces evaluated-scene camera input with pose-retrieved memories from prior drives and finds near-camera-based performance on NAVSIM but much larger drops on Bench2Drive and RealEngine.

Problem
The paper examines whether high benchmark scores demonstrate understanding of the current traffic scene or can be achieved from static information and ego status.
Method
MemoryDrivoR replaces evaluated-scene camera input with pose-indexed memories from previous traversals, providing static and quasi-static context without current traffic dynamics.
Results
Across benchmarks, memory nearly matches leading camera-based performance on NAVSIM but is much less effective on Bench2Drive and RealEngine.
Takeaways & Limitations
NAVSIM scores should be interpreted cautiously because high performance can occur without observing the evaluated traffic scene.
Takeaways & Limitations
The protocol assumes repeated traversals of the same locations and accurate relative poses, excluding benchmarks without provided poses such as WOD-E2E.

Abstract

from arXiv · show

End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .

1 Introduction

The paper questions whether high end-to-end driving benchmark scores require understanding the current traffic scene. MemoryDrivoR tests this by replacing evaluated-scene camera input with memories containing static information, finding strong NAVSIM performance and benchmark-dependent robustness.

Motivation: High benchmark scores do not by themselves identify whether a model used current-scene information or simpler correlated cues.This matters because scores guide model design and leaderboard claims about scene understanding.
Motivation: The paper separates static information, such as map structure and location priors, from dynamic information, such as traffic participants, signal phase, and weather.
Contribution: MemoryDrivoR replaces evaluated-scene camera input with latent memories from previous camera-based traversals, restricting the planner to static and quasi-static information.Each memory stores scene tokens with the global pose where they were observed.
Results: MemoryDrivoR nearly reaches state-of-the-art camera-based performance on NAVSIMv1 and NAVSIMv2 while using static information and initial ego-status cues.
Results: The intervention is substantially less effective on Bench2Drive and RealEngine, while the study distinguishes its findings from location overfitting using geographical splits.

2 Related Work

Related work shows that driving benchmarks can reward non-perceptual shortcuts, while newer benchmarks add simulation and interaction to address these weaknesses. MemoryDrivoR repurposes historical context as an analytic probe of static-information dependence rather than as a perception aid.

Shortcut learning: Earlier work found competitive scores from ego status, navigation commands, or past trajectories without camera perception.
Shortcut learning: PDM-Open showed that a high nuPlan open-loop score can result from following a selected centerline while largely ignoring dynamic agents.
Modern benchmark responses: NAVSIMv1 addresses open-loop weaknesses with filtered scenes and short non-reactive simulation scoring for compliance, progress, and comfort.
Modern benchmark responses: NAVSIMv2 adds synthetic viewpoints and reactive background traffic, while HUGSIM and RealEngine evaluate fully closed-loop behavior.
Memory and map priors: MemoryDrivoR uses previous drives as a richer source of persistent context to test benchmark performance without observing current traffic dynamics.

3 Method

MemoryDrivoR builds a pose-indexed memory bank from previous traversals and injects retrieved, pose-conditioned memories into a DrivoR planner. The method removes online perception of the evaluated scene while preserving the planner’s downstream trajectory-scoring pipeline.

Memory construction: Previous camera-based traversals are encoded into latent scene tokens and stored with their ego poses in a memory bank.The stored tokens are called memory tokens.
Training: The method builds on DrivoR’s vision-transformer scene-token interface and fine-tunes its planner using memory tokens instead of camera tokens.
Retrieval: At query time, nearest-neighbor retrieval selects up to k memories within radius r, ranked by distance and limited to one memory per traversal.
Memory injection: Each retrieved memory is represented from its original viewpoint and receives an embedding of its pose relative to the current ego pose.
Memory injection: A transformer resampler compresses pose-conditioned memories, and the resulting tokens replace the keys and values of both planner decoder cross-attention layers.If no memories are available, the corresponding cross-attention layers are skipped.

4 Experiments

Experiments show that replacing camera observations with retrieved memories preserves surprisingly high NAVSIM performance, but the advantage weakens on longer, interactive closed-loop benchmarks. Additional analyses attribute the gap to evaluation horizon, dynamic interactions, and the limited role of location memorization.

NAVSIMv1: MemoryDrivoR reaches a PDMS of 91.1 on NAVSIMv1, approaching camera-based state of the art using ego status and static information.The result suggests most NAVSIMv1 performance can be achieved without direct access to the current traffic scene.
NAVSIMv1: NAVSIMv1 preserves road geometry and collision-related performance under memory, with the main substantial reduction occurring in ego progress.MemoryDrivoR drives more conservatively at locations where dense traffic is likely, reducing interactions over the 4 s evaluation horizon.
NAVSIMv2: NAVSIMv2 shows the same qualitative pattern: memory approaches camera-based methods despite reactive traffic and shifted Stage 2 starting points.Metrics tied to dynamic information do not substantially drop, while the main tradeoff again appears in ego progress.
Bench2Drive: MemoryDrivoR remains far below camera-based performance on Bench2Drive, where fully closed-loop routes average about 110 m and 79 s.The longer horizon, increasing gap with travel distance, and uninformative standing-start ego status make driving from memory more challenging.
Bench2Drive: MemoryDrivoR completes 21 of 220 Bench2Drive routes without infractions, including all five less-complex HighwayCutIn routes.These successes can be handled by slowing down before the merge when few other agents are present.
RealEngine: RealEngine exposes larger memory deficits in true closed loop, especially after inserting additional actors, with collision-related metrics accounting for most of the loss.After the first planning step, updated ego status creates uncharted state spaces, and the gap between DrivoR and no-camera baselines grows substantially.
Additional benchmarks: HUGSIM-nuScenes reports a 32.2% HD-Score for MemoryDrivoR versus 36.4% for camera-based DrivoR, without the larger gaps seen on Bench2Drive or RealEngine.The authors relate this to 87 scenarios derived from 12 mostly straight routes that are generally simple to solve.
Ablations: Removing test locations from training causes no degradation and slightly increases overall scores, making geography memorization an unlikely primary explanation.Geographical splits control for location memorization but do not remove the static-information behavior exposed by the audit.

5 Discussion

The audit shows that NAVSIM can score highly using static scene information and ego status, while removing dynamic information substantially hurts Bench2Drive. The protocol also highlights benchmark-design limitations and assumptions about localization and repeated traversals.

5.1 Interpretation: 91.1 PDMS on NAVSIMv1 leaves little score headroom for rewarding additional interaction capabilities.Recorded ego status can reveal whether braking has already begun, while memory supplies road context for conservative continuation.
5.1 Interpretation: Removing dynamic information substantially lowers Bench2Drive performance with the same underlying planner.Longer closed-loop routes create more collision opportunities and reduce the dominant influence of ego status.
5.1 Interpretation: Future benchmarks should combine long horizons with counterfactual actors to require necessary interactions.The paper identifies RealEngine-style counterfactual actors as complementary to longer evaluation horizons.
5.2 Limitations: The protocol assumes repeated traversals of the same locations and high relative pose accuracy across traversals.This excludes benchmarks without provided poses, although sensitivity tests find weaker diagnostic signals with consumer-grade accuracy or fewer traversals.

6 Conclusion

MemoryDrivoR audits benchmarks by replacing evaluated-scene camera input with memories from previous drives. It performs surprisingly well on NAVSIM but is much less successful on longer, interaction-focused Bench2Drive routes.

6 Conclusion: MemoryDrivoR replaces camera input from the evaluated scene with memory from previous drives.The memories provide static environmental information and are combined with ego status.
6 Conclusion: MemoryDrivoR achieves high NAVSIM performance competitive with leading camera-based methods without current-scene camera observations.Its inputs are limited to ego status and static environmental information.
6 Conclusion: MemoryDrivoR is much less successful on Bench2Drive’s longer closed-loop routes designed around traffic interactions.This qualifies the NAVSIM finding by showing a benchmark-dependent effect.
6 Conclusion: MemoryDrivoR does not establish the limit of what ego status and static information can support.Better retrieval, richer memories, and more capable planners could make the audit sharper.

A.1 Failure Analysis

Failure analysis shows that memory-based driving is weaker across Bench2Drive ability and interaction categories, while NAVSIM lacks a clear interaction-type separation favoring camera-based DrivoR.

A.1 Failure Analysis: Removing camera input lowers Bench2Drive performance across all five ability families.The families are Merging, Overtaking, Emergency Brake, Give Way, and Traffic Sign.
A.1 Failure Analysis: MemoryDrivoR scores lower than camera-based DrivoR for every Bench2Drive interaction type.The largest losses occur in interactions requiring observation of the current scenario, including pedestrians, cyclists, cut-ins, and sudden obstacles.
A.1 Failure Analysis: NAVSIM interaction types show no clear separation in favor of camera-based DrivoR.The largest gaps involve lead vehicles on NAVSIMv1 and vehicle merges on NAVSIMv2, while the smallest include vehicle merges and red lights respectively.

A.2 Controlled Evaluation Horizon

On Bench2Drive, the relative collision-free advantage of camera-based DrivoR grows as routes become longer. This indicates that extended route distance exposes more failures involving dynamic traffic participants.

A.2 Controlled Evaluation Horizon: The collision-free gap widens with cumulative route distance on Bench2Drive.Longer routes make MemoryDrivoR more likely than DrivoR to collide with other traffic participants.

A.3 Detailed HUGSIM Results

HUGSIM is a photorealistic closed-loop simulator with controllable traffic actors, and the evaluation covers all 87 nuScenes scenarios under the DrivoR protocol.

Figure 5 reports the ratio of collision-free routes by traveled distance on Bench2Drive.
HUGSIM reconstructs photorealistic real-world scenes with controllable traffic actors.
The evaluation uses all 87 nuScenes HUGSIM scenarios and follows DrivoR’s test protocol.A full breakdown by difficulty is reported in Table 11.

A.4 Sensitivity Tests

Sensitivity tests show that MemoryDrivoR remains diagnostically useful with imperfect localization and few retrieved memories, although pose noise reduces performance.

Inaccurate poses: 86.9 PDMS is achieved by the nominal-noise model, while noise-trained variants remain near 88–89 PDMS.The nominal noise level represents consumer-grade localization accuracy.
Inaccurate poses: 84.2 PDMS versus 78.5 is obtained under the largest perturbation by the heavy-noise and clean-trained models, respectively.Noise training trades clean-pose performance for robustness.
Fewer traversals: k = 1 reaches 89.8 PDMS, while performance saturates near 91.1 PDMS for k ≥6.The k = 0 point is the DrivoR control without cameras, and the retrieval radius remains fixed at r = 20 m.

A.5 Component Ablation

Component and representation ablations identify relative-pose conditioning as the dominant method component, while the resampler largely preserves map information and reduces agent information.

Component ablation: Removing relative-pose conditioning keeps PDMS at 82.6 even when the resampler is enabled.The ablation is conducted on NAVSIMv1 navval.
Representation probing: After resampling, map AP decreases by 0.54 points and IoU by 0.17 points, while agent AP decreases by 2.94 points and IoU by 2.30 points.The experiment probes whether the resampler filters dynamic information.
Representation probing: PCA removes less agent information than the resampler but loses more map information.
Camera-memory fusion: Fusion provides only a small effect across benchmarks compared with the memory gains from the main audits.Fusion retains current camera input and concatenates it with retrieved memory tokens.
Qualitative examples: MemoryDrivoR often slows down where dense traffic is likely but can progress more when retrieved scenes suggest less congestion.

B.4 Compute Resources

The study adds modest storage for memory construction across benchmarks, while compute accounting covers route jobs, full-suite simulator runs, and evaluation exclusions.

Memory-bank storage: The NAVSIM memory bank adds 6.3 GB across four cities, while the filtered Bench2Drive bank adds 21.2 GB across 12 towns.These figures exclude the original benchmark datasets.
Memory-bank storage: The HUGSIM-nuScenes memory bank adds 2.2 GB for 34,149 nuScenes trainval keyframes.
Evaluation accounting: Bench2Drive timing sums completed evaluation over 220 route jobs plus aggregation, while each HUGSIM-nuScenes row covers all 87 scenarios plus aggregation.Preliminary, failed, video-rendering, and exploratory runs are excluded.
Scope: The work is a benchmark-auditing method rather than a deployed driving system, and its supported claims are benchmark-level and benchmark-dependent.The Bench2Drive contrast limits how NAVSIM findings should be interpreted.