PaperlayerView exact revision on arXiv

Source-linked AI summary

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Shaozu Ding, Linan Song, Dajiang Suo

arXiv:2608.28762v1cs.CV

TL;DR

Existing traffic VQA benchmarks provide limited support for 3D-grounded reasoning over distances, trajectories, topology, and safety-critical interactions. Inter-3D VQA addresses this gap with a large-scale multimodal intersection benchmark, Inter-Geo, and Inter-Metrics; Inter-Geo outperforms image-based baselines, especially on grounded reasoning tasks.

  • Problem

    Existing traffic VQA benchmarks lack explicit 3D grounding for real-world distances, trajectories, lane-level positions, infrastructure relations, and safety-critical interactions.

  • Method

    The paper builds Inter-3D VQA from synchronized global point clouds and multi-view images, and proposes Inter-Geo with aligned object- and scene-level LiDAR representations plus Inter-Metrics.

  • Results

    Inter-Geo outperforms image-based VLMs, especially on grounded spatial and temporal reasoning tasks.

  • Takeaways & Limitations

    Inter-3D VQA provides a benchmark and baseline for evaluating 3D-aware multimodal reasoning in roadside intersection scenes.

  • Takeaways & Limitations

    The benchmark uses data from a single intersection with fixed geometry and layout, requiring cross-intersection validation in future work.

Abstract

from arXiv · show

Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic scenes. However, existing benchmarks are largely built from ego-vehicle views or 2D roadside videos, limiting their ability to evaluate 3D-grounded reasoning over real-world distances, trajectories, infrastructure topology, and safety-critical interactions. We introduce Inter-3D VQA, a large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA at intersections. Built from synchronized point clouds and multi-view images, Inter-3D VQA contains 407K QA pairs covering lane-level positions, object relationships, motion patterns, and near-miss-oriented interaction reasoning. We further propose Inter-Geo, an MLLM baseline that integrates object- and scene-level aligned LiDAR representations, and Inter-Metrics, a unified evaluation framework for textual consistency, numerical accuracy, and semantic correctness. Experiments show that Inter-Geo outperforms image-based VLMs, especially on grounded spatial and temporal reasoning tasks. Our benchmark and codes are available at https://github.com/ASU-Suo-Lab/Inter-3D-VQA .

1 Introduction

Inter-3D VQA addresses the lack of intersection-scale benchmarks for 3D-grounded roadside reasoning by combining synchronized global point clouds, multi-view images, and infrastructure topology. It introduces a 407K-pair dataset, the Inter-Geo MLLM, and Inter-Metrics for evaluating textual, numerical, and semantic correctness.

  • Research Gap: Existing traffic VQA benchmarks lack explicit 3D grounding for real-world distances, trajectories, lane-level positions, and relations among road users and infrastructure.Ego-centric benchmarks suffer from occlusion and local viewpoints, while roadside benchmarks mainly use single-view images or 2D videos.
  • Research Gap: 3D spatiotemporal grounding is needed to localize pedestrians, assess crosswalk occupancy, predict vehicle trajectories, and identify potential near-miss interactions.These safety-relevant questions require reasoning in physical 3D space beyond image-level semantics.
  • Contributions: Inter-Metrics jointly evaluates textual consistency, numerical accuracy, and semantic correctness for 3D roadside VQA.The evaluation framework complements the benchmark’s multimodal and spatially grounded task design.
  • Contributions: 407K QA pairs form a large-scale intersection-understanding dataset built from multi-view images, global point clouds, and infrastructure topology.The dataset enables 3D-aware reasoning in complex roadside scenes.
  • Contributions: Inter-Geo incorporates object- and scene-level aligned LiDAR representations and outperforms image-based VLMs, especially on 3D spatiotemporal reasoning.The model is designed to address the benchmark’s explicit geometric and temporal grounding needs.

2 Related Works

Related VQA benchmarks span autonomous driving, urban, and roadside settings, but differ in viewpoint, task design, construction, QA type, modality, and scale. Existing systems remain limited for global intersection reasoning, infrastructure-side understanding, and explicit 3D spatial structure.

  • Benchmark Landscape: Representative VQA benchmarks cover autonomous driving, urban scenes, and roadside understanding with differing task designs, construction methods, QA types, modalities, and scales.Table 1 organizes the comparison across these dimensions.
  • Autonomous Driving VQA: Autonomous-driving benchmarks predominantly use ego-centric, vehicle-mounted views, limiting evaluation of global intersection reasoning and infrastructure-side traffic understanding.These benchmarks include multimodal QA, video QA, and graph-structured reasoning over perception, prediction, and planning.
  • Roadside and Urban VQA: Roadside VQA enables fixed-view modeling of global traffic dynamics and multi-agent interactions, while urban benchmarks extend QA to drone-based perspectives.Examples include roadside video QA and urban 3D QA benchmarks.
  • Remaining Gap: Existing benchmarks still lack sufficient multi-directional views and explicit 3D spatial structure for intersection-scale reasoning.This limitation affects evaluation of infrastructure relations and real-world spatial understanding.
  • Model Modalities: Camera-based roadside models remain limited in capturing global scene structure, while LiDAR-based and multimodal methods are largely designed for ego-centric settings.The literature therefore leaves a gap between geometric sensing and roadside intersection reasoning.

3 Inter-3D VQA Dataset

Inter-3D VQA is constructed from synchronized multimodal roadside data and uses complementary object-reference schemes for four categories of 3D-aware reasoning. Its 407K QA pairs are generated through templating and LLM-assisted expansion, then filtered through multi-stage quality control.

  • 3.1 Dataset Construction: The dataset construction pipeline curates metadata, generates template-based QA with LLM-assisted expansion, and performs multi-stage sanity checks.The stages use synchronized images, LiDAR point clouds, HD maps, expert questions, and validation procedures.
  • 3.1 Dataset Construction: Metadata combines synchronized multi-view video and LiDAR streams sampled at 2 Hz with image semantics, annotated 3D boxes, temporal associations, and aligned HD maps.The resulting records support multimodal object and scene reasoning.
  • 3.2 Object Referring: Grounded QA uses spatiotemporal 3D-aware tuples for cross-view association, localization, tracking, and topology-aware disambiguation, while free-form QA uses infrastructure-enriched descriptions.The two formats provide complementary mechanisms for referring to traffic participants and infrastructure.
  • 3.3 Task Definition: The benchmark covers Basic Perception, Spatial Reasoning, Temporal Reasoning, and Scene Understanding using multi-view images and global point clouds.These categories span low-level perception through high-level reasoning in complex roadside scenes.
  • 3.3 Task Definition: 407K QA pairs are split 8:2 into training and test sets without scene overlap, with 16.8% covering queue dynamics, multi-agent conflicts, or abnormal events.Over 30% involve speeding or lane-changing behaviors rather than regular passing.
  • 3.4 Inter-Metrics: Inter-Metrics combines textual similarity metrics with absolute error for scalar quantities and Euclidean error for vector quantities such as 3D coordinates and trajectories.The framework evaluates answers across textual consistency, numerical accuracy, and semantic correctness.

C MLP

Inter-Metrics evaluates numerical and semantic correctness with normalized, field-aware scoring and exact matching of structured traffic semantics.

  • Numerical Metrics: Numerical errors are normalized by field-specific thresholds τ_i and mapped to scores in [0, 1].The thresholds accommodate heterogeneous quantities with different units and scales.
  • Numerical Metrics: The same field-specific thresholds are applied across all models to ensure fair comparison.
  • Semantic Metrics: Semantic correctness uses exact matching across structured fields including object categories, lane membership, movement direction, traffic states, and conflict labels.This evaluation goes beyond surface-level linguistic similarity.
  • Semantic Metrics: The semantic score compares predicted and ground-truth values field by field using an indicator function.N denotes the number of evaluated fields, while I(·) indicates whether a predicted field equals its ground truth.

4 Methods

Inter-Geo preserves a camera–text pipeline while adding object- and scene-level LiDAR features through DQNet and LDA for geometry-aware reasoning.

  • Overall Architecture: Inter-Geo integrates multi-view images and text with LiDAR features through decoder-level cross-attention.The design preserves pretrained multimodal alignment while enabling geometry-aware reasoning.
  • Overall Architecture: The model encodes multi-view images into visual embeddings and combines them with tokenized text as the multimodal input sequence.
  • 3D Encoder: A 3D encoder extracts object-level and scene-level features from point clouds for subsequent LiDAR representation learning.The scene features capture global context, while object features describe individual instances.
  • Dual Querying Net: DQNet converts variable-length LiDAR features into fixed-size language-aligned representations using complementary object and scene branches.The object branch captures instance details, and the scene branch captures global context through latent-query aggregation.
  • Dual Querying Net: A confidence gate suppresses unreliable or redundant object representations before they are used by the model.The gate uses a sigmoid function and a learnable projection.
  • LiDAR Decoder Adapter: LDA retrieves scene- and object-level geometric information through cross-attention and injects it selectively into decoder states.Token- and channel-wise gating regulates LiDAR contributions, followed by gated fusion through a residual module.

5 Experiments

Experiments show that Inter-Geo’s aligned LiDAR representations and decoder-level integration improve grounded 3D reasoning, while performance depends on explicit 3D perception, dual-branch context, and efficient deployment.

  • Quantitative Results: Inter-Geo achieves the best grounded QA performance and remains highly competitive on free-form QA.It outperforms existing VLMs across grounded metrics while preserving strong language understanding on free-form questions.
  • Quantitative Results: 20.7%: Inter-Geo improves grounded numerical accuracy over Qwen3-VL-8B.It also improves Snum and Ssem over Senna-VLM by 46.5% and 28.6%, respectively.
  • Ablation Studies: 19.0%: replacing the 3D encoder with an MLP reduces Snum, confirming the importance of explicit 3D perception.Ssem also decreases by 7.6%.
  • Ablation Studies: 9.5%: removing the scene branch causes a larger drop than removing the object branch, whose removal reduces Snum by 4.0%.The scene branch supplies global BEV context for lane topology, intersection layout, and long-range spatial reasoning.
  • Ablation Studies: 26.7%: removing LDA decreases Snum, showing that LiDAR features must reach decoder layers for language reasoning.Ssem also drops by 11.1%; encoding and aligning LiDAR features alone is insufficient.
  • Ablation Studies: 4B-to-8B scaling improves Stext, Snum, and Ssem by 1.2%, 4.7%, and 2.4%, respectively.Even the 2B variant outperforms existing VLM baselines, indicating complementary geometric cues from aligned LiDAR representations.
  • Runtime and Deployment Analysis: Inter-Geo incurs additional computational cost, while sensing, preprocessing, and transmission further contribute to end-to-end latency.The results motivate future work on more efficient deployment.

6 Conclusion

The paper introduces Inter-3D VQA for roadside 3D spatiotemporal reasoning and Inter-Geo as a LiDAR-enhanced baseline. Experiments show that explicit LiDAR geometry particularly benefits grounded spatial and numerical reasoning.

  • 6 Conclusion: Inter-3D VQA integrates multi-view images, global point clouds, and infrastructure topology for spatiotemporal roadside reasoning.The benchmark is designed for 3D-aware understanding of complex traffic scenes.
  • 6 Conclusion: Inter-Geo is a LiDAR-enhanced multimodal language model serving as a baseline for multimodal roadside reasoning.Its fusion design is evaluated through controlled ablations.
  • 6 Conclusion: Explicit LiDAR geometry particularly benefits grounded spatial and numerical reasoning.The conclusion identifies this as a central experimental finding.

Limitations

The benchmark and model have scope and evaluation limitations involving single-intersection data, structured semantic parsing, and limited temporal modeling.

  • Limitations: The benchmark data comes from a single intersection with fixed geometry and layout, requiring cross-intersection validation.Future work is needed to expand scenario diversity.
  • Limitations: Inter-Metrics depends on structured output parsing, so format deviations can affect Ssem even when answers are partly reasonable.The limitation concerns robustness of semantic evaluation.
  • Limitations: Inter-Geo mainly uses single-frame multimodal inputs and does not fully exploit multi-frame temporal cues.This limits modeling of long-term motion patterns and dynamic interactions.

Ethical Considerations

Inter-3D VQA targets fine-grained spatial, temporal, and infrastructure-aware reasoning in roadside intersection scenes. Its scope includes perception, geometry, motion, interactions, and global traffic-state understanding.

  • Ethical Considerations: Existing image-based roadside benchmarks struggle with precise geometric measurements, topology-aware relations, and dynamic interaction reasoning.Distance-to-crosswalk and lane-queue questions require geometry and infrastructure semantics, while conflict questions require modeling relationships over time.
  • Ethical Considerations: The benchmark combines synchronized multi-view images and global point clouds to extend image-based reasoning into multimodal 3D reasoning.
  • Ethical Considerations: Inter-3D VQA covers four complementary task types: Basic Perception, Spatial Reasoning, Temporal Reasoning, and Scene Understanding.These categories span object information, three-dimensional geometry, dynamic traffic interactions, and overall intersection conditions.

A.2 Dataset Statistics

Inter-3D VQA is broadly balanced across task categories and provides concise, semantically rich answers with substantial coverage of traffic behavior, objects, numerical targets, and 3D locations. Evaluation results and ablations show that explicit LiDAR integration and carefully sized decoder/query modules affect numerical and semantic performance.

  • Dataset Statistics: Most questions contain approximately 14–22 words, while task composition remains relatively balanced across the four categories.The vocabulary covers traffic participants, infrastructure topology, motion states, and interaction patterns.
  • Dataset Statistics: Most answers remain concise, prioritizing precise reasoning over verbose generation.Answer vocabulary emphasizes traffic participants, road structures, and motion-related concepts.
  • Dataset Statistics: The dataset covers diverse driving behaviors and traffic participants, including turning, lane changes, acceleration, speeding, vulnerable road users, and less frequent objects.
  • Dataset Statistics: Counts, distances, velocities, and 3D coordinates provide quantitative supervision across a wide range of intersection areas.
  • Dataset Statistics: All Inter-Geo variants outperform image-based VLM baselines, with LION achieving the best overall performance among evaluated LiDAR encoders.TransFusion-L attains Snum = 0.580, while CenterPoint attains Stext = 0.862 but lower Snum = 0.540.
  • Dataset Statistics: The four-layer LDA configuration performs best; compared with two layers, it improves Snum by 8.3% and Ssem by 0.9%.Eight layers instead reduce Snum by 6.1% and Ssem by 1.4%.
  • Dataset Statistics: Using 16 object queries and 64 scene queries provides the strongest balance between geometric evidence, global context, and representation compactness.Reducing object queries to 8 lowers Snum by 3.4%, while 24 lowers Snum by 7.5% and Ssem by 1.2%; 32 scene queries lower Snum by 3.7% and Ssem by 0.5%, while 128 lower them by 7.5% and 1.0%.

C.7 Numerical Error Analysis

Inter-Geo achieves strong grounded numerical performance by integrating aligned LiDAR evidence, while remaining less dominant for 2D localization and acceleration. The evaluation uses controlled multimodal prompting, structured references, and template-based QA construction to support consistent 3D reasoning.

  • C.7 Numerical Error Analysis: Inter-Geo achieves the lowest errors for distance, 3D coordinates, and waypoint prediction: 0.15, 0.48, and 0.31, respectively.These results are reported against image-based open-source, driving-specific, and roadside-specific VLMs.
  • C.7 Numerical Error Analysis: Inter-Geo is not clearly superior for 2D coordinate error, whose performance depends more on image-plane localization than 3D grounding.
  • C.7 Numerical Error Analysis: Acceleration error remains relatively high, indicating that fine-grained dynamic attributes are challenging and motivating tighter image–LiDAR alignment and stronger temporal modeling.
  • C.7 Numerical Error Analysis: The evaluation prompt prioritizes LiDAR for 3D geometry, global position, spatial layout, occlusion, and distance, while using images for appearance, signal states, and cross-view information.
  • C.7 Numerical Error Analysis: Image-only models receive prompts with LiDAR instructions removed, preventing unavailable-modality information from interfering with their evaluation.
  • C.7 Numerical Error Analysis: Grounded QA uses object references containing stable IDs, temporal offsets, coordinates, and multi-view projections, while templates are instantiated from synchronized multimodal metadata and HD maps.

D.2 Qualitative Evaluations

Qualitative examples show Inter-Geo using global point clouds to improve cross-view distance estimation, trajectory reasoning, nighttime counting, and safety-event interpretation. The benchmark presents both free-form and grounded templates across four reasoning categories.

  • D.2 Qualitative Evaluations: Inter-Geo estimates a daytime cross-view distance of 58.6 m against the 56.9 m ground truth, while image-based baselines show larger errors.
  • D.2 Qualitative Evaluations: Inter-Geo preserves trajectory direction and displacement trends with only minor deviation, whereas other models exhibit larger longitudinal or lateral drift.
  • D.2 Qualitative Evaluations: In a nighttime scene with blur and occlusion, Inter-Geo correctly identifies one truck while other models overestimate the count.
  • D.2 Qualitative Evaluations: Inter-Geo detects a speeding-risk event and supplies the relevant object category and semantic information in the nighttime scene-understanding example.
Loading 2608.28762v1…