STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases

• Prior edge-case benchmarks emphasize recognition, while temporal benchmarks use simpler scenarios, leaving joint spatial-temporal reasoning in complex interactions insufficiently tested (Chen et al., 2025; Chi et al., 2026; Fruhwirth-Reisinger et al., 2026).Source: §1 · p. 2
• STRIDE uses 39 templates across six task families, pairing MCQs on intermediate judgments with open-ended questions on integrated driving explanations and decisions.It prioritizes unusual object behaviors and complex traffic interactions, testing object location, past-motion reconstruction, and future effects on the ego vehicle.Source: §1 · p. 2
• Across 12 models, Qwen3.6-27B and GPT-6-Astra achieve substantially lower STRIDE MCQ accuracy than on Fruhwirth-Reisinger et al. (2026)’s benchmark.The supplied passages do not report the exact comparison values.Source: §1 · p. 2, Figure 1 · p. 2
• GPT-5.5 achieves 72% accuracy for previous relative position but 18% for recent motion patterns.Source: §1 · p. 2
• High scenario-explanation VQA scores coexist with weak judgments about drivable space and specific risks, indicating that aggregate scores can conceal safety-relevant gaps.The passage suggests broad scene-caption tasks may partly rely on generic driving knowledge and notes substantial performance with black images (Xie et al., 2025).Source: §1 · p. 2

Driving VQA Benchmarks

• Driving VQA benchmarks progress from scene questions to reasoning and spatial grounding, while reliability studies warn that favorable scores may reflect textual shortcuts or ego-state reliance (Caesar et al., 2020; Marcu et al., 2024; Sima et al., 2024; Nie et al., 2024; Zhao et al., 2026; Xie et al., 2025).Efforts include graph-structured QA, annotated reasoning chains, counterfactual supervision, multi-view understanding, and explicit 3D localization.Source: §2 · p. 3

Spatiotemporal and Long-Tail Evaluation

• Recent benchmarks separately assess temporal understanding across multi-frame interactions and event tasks, and long-tail difficulty through unusual objects, unstructured roads, and challenging agent interactions (Fruhwirth-Reisinger et al., 2026; Chen et al., 2025; Chi et al., 2026; Xu et al., 2026).These efforts provide complementary foundations for evaluating temporal capabilities and scenario difficulty.Source: §2 · p. 3

3.1 TASK DEFINITION

• STRIDE assesses spatial perception and understanding, temporal memory and extrapolation, ego-trajectory prediction, and scene-context awareness.The tasks range from object positions and relations to driving risks, prior scene states, likely future events, ego motion, and environmental context.Source: §3.1 · p. 4

3.2 DATA CONSTRUCTION

• STRIDE combines nuScenes (Caesar et al., 2020) with long-tail-focused WOD-E2E (Xu et al., 2026), sampling challenging eligible scenes using template-based curation.Templates are used only where annotations support reliable ground truth; selection prioritizes dense interactions, hazardous conditions, unusual behavior, and scene complexity.Source: §3.2 · pp. 4–5
• Linked multiple-choice judgments and free-form VQA evaluate shared scene attributes independently, while human review and VLM-assisted error analysis refine open-ended references.Question types without sufficient source annotations are removed, and weak references are replaced after human scoring.Source: §3.2 · p. 5

3.3 DATASET STATISTICS

• STRIDE comprises 2,350 questions across 1,452 clips, with challenging scenario tags spanning all six task families and varied future-motion speed profiles.The questions include 1,700 multiple-choice and 650 open-ended items; 1,150 use nuScenes and 1,200 use Waymo E2E. Pairing maneuvers with multiple speed profiles prevents inferring speed from motion direction alone.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation spans proprietary, open-weight, and driving-specialized VLMs under shared inputs and prompting where interfaces permit; MCQs use accuracy, OEQs a 0–10 judge score, and trajectories ADE/FDE.Open-weight models were evaluated using vLLM on NVIDIA A6000 GPUs, proprietary models through official APIs, and decoding was deterministic or near-deterministic unless otherwise stated.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• 45.5% and 41.2%: GPT-6-Astra led MCQ accuracy on nuScenes and Waymo E2E, versus random baselines of 19.8% and 20.3%, leaving over half the questions incorrect.The results highlight that top MCQ performance does not establish reliable spatiotemporal judgment or superior trajectory prediction.Source: §4.2 · p. 7
• 6.68 and 7.01: GPT-5.5 led nuScenes OEQ, Claude-Sonnet-5 led Waymo E2E OEQ, and UniAD 2.0 led nuScenes trajectory prediction at ADE/FDE 0.56/0.79.On Waymo E2E, GPT-5.5 scored 7.00; on nuScenes, GPT-6-Astra and GPT-5.5 had trajectory ADE/FDE of 0.60/0.98 and 0.65/0.97, respectively.Source: §4.2 · p. 7

5.1 SPATIAL PERCEPTION

• On nuScenes, best relative-position and distance accuracies are 62% and 64%, versus 38% for trajectory direction and 44% for occlusion; integrated spatial descriptions remain difficult.GPT-5.5 leads integrated descriptions at 3.66/10, while Gemini-3.1-Pro scores 2.10/10 despite leading distance MCQs; on Waymo, Gemini-3.6-Flash leads distance MCQ at 62%, whereas Claude-Sonnet-5 leads distance OEQ at 6.28/10. Qwen3.6-27B gives valid-format answers on 34% of nuScenes SP-3 responses, with 22% accuracy overall and 64.7% among 17 parsed responses.Source: §5.1 · pp. 7–8

5.2 SPATIAL UNDERSTANDING

• On nuScenes, GPT-5.5 scores 9.22/10 for scene description but only 22% for both drivable-region identification and lane-change clearance.On Waymo E2E, GPT-5.5 scores 8.08/10 for scene/risk/action explanations, alongside 44%, 28%, and 48% accuracy for scene classification, risk identification, and preferred action; STRIDE’s task-specific prompts and scoring distinguish these capabilities.Source: §5.2 · p. 8

5.3 TEMPORAL MEMORY

• On nuScenes, GPT-5.5 and GPT-5.6-Sol each achieve 72% for previous object position but only 18% and 16% for recent motion, near its 16.7% chance baseline.Qwen3.6-35B-A3B leads memory of currently occluded objects at 36% against a 25% baseline; on Waymo, Qwen3-VL-30B-A3B-Thinking gets 32% for recent motion but 10% for previous position, while Claude-Sonnet-5 reaches 74% for past ego-maneuver classification.Source: §5.3 · p. 8

5.4 TEMPORAL EXTRAPOLATION

• Best future-object-motion accuracy is 40% on nuScenes; on Waymo E2E, best accuracies for future object motion, ego response, and next traffic event are 36%, 34%, and 34%.Qwen3-VL-30B-A3B-Thinking has the lowest Waymo ADE/FDE at 11.83/6.18 despite weaker aggregate VQA performance. Parsing affects interpretation: Qwen3.6-35B-A3B parses 16% of nuScenes TRJ-6 examples, with ADE 0.32 on valid outputs versus 8.19 overall; GPT-5.5 parses 100% on Waymo TRJ-5 but has ADE 23.44. Dolphins’ finite Waymo FDE of 8.60 reflects the stationary-at-origin fallback, not successful waypoint generation.Source: §5.4 · p. 8

5.5 A LOOK INTO FAILURE CASES OF GPT-6-ASTRA

• GPT-6-Astra’s favorable explanation scores coexist with weak safety-critical judgments: Waymo scene/risk/action explanations score 7.76/10, while risk-source and preferred-action accuracies are 36% and 44%.On nuScenes, spatial-situation explanations score 7.44/10; trajectory justifications score 7.24/10, while best-candidate selection and alternative-trajectory consequence judgments reach only 38% and 28%.Source: Figure 4 · p. 9, §5.5 · p. 9
• On nuScenes, GPT-6-Astra achieves 70% for previous object position but 20% for recent motion, a 50-point gap; all seven shown models have gaps of 10–56 points.Recent-motion accuracy remains 14–22%, near the 16.7% chance baseline, suggesting difficulty integrating observations across frames and tracking object histories.Source: §5.5 · pp. 8–9, Figure 4 · p. 9
• GPT-6-Astra’s positional judgments exceed its safety-relevant reasoning: nuScenes relative-position accuracy is 60% versus 26% for physical constraints and 24% for drivable regions.Future relative-position accuracy is 90% on nuScenes and 76% on Waymo E2E, while future object-motion accuracy is 34% and 26%; Waymo next-traffic-event accuracy is 22%.Source: Figure 4 · p. 9, §5.5 · p. 9

6 CONCLUSION

• STRIDE combines object-centric temporal prompting, six complementary task families, and linked MCQ–OEQ evaluation grounded in structured driving annotations.This design enables fine-grained diagnosis beyond broad scene-description quality.Source: §6 · pp. 9–10
• Across 12 proprietary, open-weight, and driving-specialized models, even the strongest answers fewer than half of multiple-choice questions correctly.The analysis also finds gaps between explanation quality and safety-critical judgments, and between recalling past object positions and reconstructing motion history.Source: §6 · p. 10
• The findings highlight the need to evaluate driving VLMs with precise spatial and temporal judgments in addition to fluent scene-level explanations.The authors hope STRIDE will support systematic model comparison and diagnosis of spatiotemporal limitations in challenging driving scenarios.Source: §6 · p. 10

ETHICS STATEMENT

• Although reference answers are grounded in existing driving-dataset annotations and human-verified, annotation errors and source-dataset biases may remain.The benchmark is intended for research evaluation, and results should be used responsibly with attention to real-world autonomous-driving safety requirements.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Reproducibility information covers data sources, preprocessing, annotation-grounded question-and-answer construction, evaluated models, inference settings, evaluation metrics, and question templates.These are documented in Sections 3 and 4 and Appendices C and E.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 WOD-E2E PREPROCESSING AND LABEL EXTRACTION

• SAM3 detections are filtered and heuristically tracked across frames to derive object motion, range rate, time-to-collision, position, lane, and occlusion signals.The pipeline retains up to ten high-quality objects per frame, selected by confidence, proximity, and driving relevance.Source: §A.1 · p. 13
• Object distance is estimated by back-projecting a detected box’s bottom-center pixel and intersecting the camera ray with a flat ground plane.This assumes the box bottom-center approximates the object’s contact point with the road.Source: §A.1 · pp. 13–14
• WOD-E2E annotations supply calibration, timestamps, ego motion and intent, preference trajectories, and scenario-cluster labels, which are combined with derived object annotations for rule-based answers.Source: §A.1 · p. 14

A.2 QUESTION TEMPLATE DESIGN

• Manually designed templates align with task families and instantiate questions only when the necessary visual evidence and structured annotations are available.A target can support complementary questions, while free-response templates combine capabilities such as relative position, motion direction, and distance.Source: §A.2 · p. 14

A.3 OBJECT-CENTRIC TEMPORAL PROMPTING

• Each instance presents five synchronized 360° observations sampled at 2 Hz, using four context frames and a query frame that highlights the target object.The temporal context supports motion and interaction reasoning, while object-centric prompting reduces ambiguity and makes responses easier to evaluate consistently.Source: §A.3 · pp. 14–15

A.4 TRUSTWORTHY GROUND TRUTH GENERATION

• Ground truth maps annotated longitudinal and lateral offsets to distance and bearing labels, then combines position, distance, motion, speed, and ego-trajectory interaction to assign higher-level risk.An object ahead and approaching at high relative speed in a collision-relevant region receives higher risk than one moving away.Source: §A.4 · p. 15
• For free-form questions, a language model verbalizes source annotations, derived labels, and multiple-choice answers into natural-language references, preserving traceability to driving data.The pipeline is designed to scale while minimizing unsupported or hallucinated supervision.Source: §A.4 · p. 15

A.5 EVALUATION WITH LVLM JUDGES

• The model-based judge receives visual observations, the question, reference answer, and model response to assess correctness in scene context.This approach is used instead of BLEU and METEOR, which measure lexical or surface similarity and may penalize semantically equivalent answers with different wording (Nie et al., 2024).Source: §A.5 · p. 15
• On a held-out STRIDE subset, the study compares NLP metrics, text-only LLM judges, and LVLM judges against human annotations.The authors note that Chen et al. (2025) found a text-only LLM judge aligned with humans better than the vision-language judge evaluated in CODA-LM.Source: §A.5 · p. 15
• Contemporary LVLM judges show strong agreement with human evaluation while retaining access to visual evidence needed for spatial and temporal questions.The passage reports this comparison in Table 4 but supplies no numerical agreement values.Source: §A.5 · p. 15
• Qwen3-VL-32B is adopted as the default judge for its competitive human alignment, reproducibility, and accessibility.The study evaluates both proprietary and open-weight judges.Source: §A.5 · p. 15

A.6.1 MULTIPLE-CHOICE QUESTIONS

• Reviewers sampled 50 items per multiple-choice template and retained a template only if its questions, options, and frame-supported ground truth were usable.The pre-review set covered 31 templates and 1,550 questions.Source: §A.6.1 · p. 15
• 800/1,550 (51.6%) multiple-choice questions were kept and 750/1,550 (48.4%) removed after 16/31 templates passed review.Failed templates were removed in full rather than edited item by item.Source: §A.6.1 · pp. 15–16

A.6.2 OPEN-ENDED QUESTIONS

• The review removed open-ended template TM-6 (50 questions) for inaccurate recent-motion descriptions, leaving five templates and 250 questions.The remaining templates were SC-6, SP-7, SU-7, TE-6, and TRJ-7.Source: §A.6.2 · p. 16
• Original open-ended labels, derived from rules or multiple-choice annotations, were often wrong because nuScenes lacks weather and lane labels and frame checks exposed errors.The remaining open-ended set contained 250 questions.Source: §A.6.2 · p. 16
• On a stratified 50-question sample, GPT-5.5 answers scored a mean 7.74/10; 19/50 (38%) were kept, 17/50 (34%) corrected, and 14/50 (28%) revised from original labels.Humans scored answers against the images; Qwen3-VL-32B-Instruct made corrections or revisions using human notes and frames.Source: §A.6.2 · p. 16
• For the other 200 questions, the vision-language model flagged 189/200 (94.5%) as inconsistent with images and proposed replacements, while 11/200 (5.5%) were accepted as-is.The proposed replacements for inconsistent labels were accepted.Source: §A.6.2 · p. 16
• Overall, 239/250 open-ended labels (95.6%) were rewritten; frequent audit errors involved object identity, unsafe-driving claims, distance, risk, weather, and lighting.The remaining 11 labels were marked acceptable.Source: §A.6.2 · p. 16

B MORE DATASET STATISTICS

• Table 6 reports nuScenes question and scene counts by task family; 310 scenes appear in more than one family, so family scene counts do not sum to the split total.Source: Table 6 · p. 17
• Table 7 reports Waymo question and scene counts by task family, including 691 unique bundles; 183 scenes appear in multiple families, so family scene counts do not sum to the split total.Source: Table 7 · p. 17

C ADDITIONAL EXPERIMENTAL DETAILS

• The appendix records evaluated models, decoding settings, and model-specific exceptions; the shared question template is described separately in Appendix E.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 17

C.1 EVALUATED MODELS

• Proprietary models use official APIs, while open-weight models are served from Hugging Face with vLLM on NVIDIA A6000 GPUs, except Dolphins and UniAD.Dolphins uses official OpenFlamingo inference, and UniAD is an end-to-end planner rather than a VLM.Source: §C.1 · p. 17
• Evaluated-model coverage includes every model reported in the main results, with GPT-5.5 accessed through OpenAI Chat Completions and Gemini and Claude through Vertex AI.Source: §C.1 · p. 17
• Decoding is deterministic or near-deterministic unless noted; models with reasoning controls use a low setting to limit latency and visible-token use while allowing reasoning.Source: §C.1 · p. 17
• Claude uses medium reasoning effort because low effort produced little or no reasoning in the authors’ setup.Source: §C.1 · p. 17

C.2 SHARED EVALUATION PROTOCOL

• Non-SC-6 items use five consecutive 360° multi-camera grids with the query target highlighted when specified, while SC-6 uses up to 40 chronological grids at reduced resolution.Images are resized to at most 640×640 pixels, or 360×360 for SC-6.Source: §C.2 · p. 17

C.3 MODEL-SPECIFIC SETTINGS

• GPT models receive 4096 completion tokens; Gemini and Claude use JPEG images capped at 768 pixels on the long side, with low and medium reasoning settings, respectively.Proprietary SC-6 inputs use all 40 frames; GPT uses image detail=low, Gemini thinking level=low, and Claude effort=medium.Source: §C.3 · p. 18
• Qwen3.6 models run with thinking disabled, and Qwen3.6 and Qwen2.5-VL-3B subsample SC-6 to eight frames for the 8k context window.Source: §C.3 · p. 18
• Qwen3-VL Thinking models retain chain-of-thought, use 4096-token budgets, and are scored only on the response after the final </think> tag.That span is also used before OEQ judging and trajectory parsing.Source: §C.3 · p. 18
• Dolphins use the official video-tensor inference format and beam search with three beams, with SC-6 capped at 16 frames.The output caps are 64 new tokens for MCQ and 512 for OEQ.Source: §C.3 · p. 18
• UniAD is evaluated only on TRJ-5, using ego-coordinate waypoints from the first five of six predicted 0.5-second steps to match the ground truth.The conversion flips x as in UniAD’s planning metric, then applies lidar2ego.Source: §C.3 · p. 18

C.4 SCORING NOTES

• MCQ accuracy is a micro-average over items, with denominators of 800 for nuScenes v6 and 900 for Waymo v1; unparseable answers count as incorrect.For GPT-5.5, pooled accuracy is 0.3925 (314/800) on nuScenes and 0.3867 (348/900) on Waymo; equal family-accuracy averages are 0.375 and 0.378. Random-guess baselines are 0.197991 and 0.203241, respectively, and are not subtracted from accuracy.Source: §C.4 · pp. 18–19
• Open-ended answers are judged by a vision LLM, while waypoint outputs are parsed as JSON coordinate lists and scored with ADE-style L2 metrics; unparseable trajectories count as failed.The judge is Qwen/Qwen3-VL-32B-Instruct at temperature 0 and sees the same frames as the candidate, with SC-6 subsampled to eight frames.Source: §C.4 · p. 19

D ADDITIONAL PER-TASK PERFORMANCE TABLES

• Tables 9–10 provide fine-grained per-task model results for the nuScenes and Waymo E2E splits.The tables report MCQ accuracy and LVLM-judged OEQ responses; ADE is lower-is-better, while all other metrics are higher-is-better.Source: D ADDITIONAL PER-TASK PERFORMANCE TABLES · p. 19, Table 9 · p. 19, Table 10 · p. 20
• MCQ accuracy is reported as a percentage, OEQ responses are evaluated by an LVLM judge, and ADE is the sole lower-is-better metric.All other reported metrics are higher-is-better.Source: Table 9 · p. 19, Table 10 · p. 20

E QUESTION TEMPLATES

• The taxonomy contains 39 question templates across six families; N and W mark use in the nuScenes and Waymo evaluation splits, and blanks indicate non-use.Tables 11–16 list the templates in the catalog.Source: E QUESTION TEMPLATES · p. 21
• Waymo reuses SC-1–4 for a scenario-corner-case family rather than scene-context and rewrites TRJ-5/6 for front-view input.The Waymo TRJ-5/6 variants drop the 360° preamble and pasted history and ask about a front-view 2.0-second horizon.Source: E QUESTION TEMPLATES · p. 21, Table 17 · p. 23
• Tables 11–13 cover scene-context, space-perception, and space-understanding templates; the latter family is nuScenes-only.Scene-context templates SC-1–5 were dropped from the reviewed nuScenes split.Source: Table 11 · p. 21, Table 12 · p. 22, Table 13 · p. 22
• Tables 14–16 list time-extrapolation, time-memory, and trajectory-prediction templates, with Waymo retaining TRJ-5/6 IDs but changing their visual preamble.The Waymo versions use front-view input and an approximately 2.0-second horizon.Source: Table 14 · p. 22, Table 15 · p. 23, Table 16 · p. 23