Source-linked AI summary
NeuroSymbEAD: A Large Scale Neuro-Symbolic Caption Dataset for Omni-Directional Embodied Autonomous Driving
Muhammad Ahmed Ullah Khan, Mohammed Elamine, Sheikh Talha Uddin, Didier Stricker, Sk Aziz Ali, Muhammad Zeshan Afzal
TL;DR
Autonomous-driving models struggle with situational awareness because they cannot filter out unnecessary background knowledge. The paper constructs KITTI-360 object- and group-level captions from ego-relative semantic, spatial, orientation, heading, and motion attributes, and evaluates a knowledge-graph-focused captioning and grounding model. NeuroSymbEAD contains 39,723 scenes and 692,081 captions, while its ground-truth-box benchmark achieves BLEU 54.67 and ROUGE-L 61.45 for 3D captioning.
Problem
Autonomous-driving models struggle with situational awareness because they cannot filter out unnecessary background knowledge.
Method
The paper constructs KITTI-360 object- and group-level captions from ego-relative semantic, spatial, orientation, heading, and motion attributes, and evaluates a knowledge-graph-focused captioning and grounding model.
Results
NeuroSymbEAD contains 39,723 scenes and 692,081 captions, while its ground-truth-box benchmark achieves BLEU 54.67 and ROUGE-L 61.45 for 3D captioning.
Takeaways & Limitations
NeuroSymbEAD provides a benchmark for structured ego-centric language in traffic-scene explanation, 3D reasoning, and interpretable autonomous-driving perception.
Takeaways & Limitations
The dataset covers traffic-relevant human and vehicle categories with a sparse ontology, template-generated captions, ground-truth boxes, and one captioning/grounding baseline.
Abstract
from arXiv · showhide
This paper introduces NeuroSymbEAD, a large-scale neuro-symbolic caption dataset featuring an ego-centric knowledge graph (KG) of static and dynamic objects annotated with classes, categories, heading directions, orientations, and distances from the ego-vehicle. These annotations are used on the KITTI-360 dataset to generate multilevel textual captions representing a lightweight version of an ego-centric scene map. Outdoor scene-map reconstruction, visual recognition, and object grounding establish baselines for driving common sense and traffic/scene understanding. For these purposes, natural language-based grounded captioning of objects and their complex relationships is a widely adopted contextual representation for indoor scene tasks. Neuro-symbolic representations have proven effective in handling structured information for various computer vision and language applications. Our data annotation pipeline allows the generation of varied map segments, populating simulated or real objects within the bounding boxes predicted by any 3D object detection network, and building hierarchical text captions. We benchmark our neuro-symbolic and ontological caption generation using pre-trained grounding and learned auto-regressive captioning networks. By converting 3D driving scenes into structured ego-centric language, NeuroSymbEAD provides a benchmark for vision-language and foundation models for traffic-scene explanation, 3D reasoning, and interpretable autonomous-driving perception.
1 Introduction
NeuroSymbEAD addresses the need for explicit, efficient ego-centric representations that combine neural perception with symbolic and ontological scene knowledge for autonomous driving. It introduces a large-scale KITTI-360 caption dataset and a model centered on ego-centric knowledge graphs.
- 1 Introduction: NeuroSymbEAD combines neural networks with ego-centric knowledge graphs to represent traffic scenes through context-aware descriptions.The framework targets cars and pedestrians in LiDAR-recorded KITTI-360 driving sequences.
- 1 Introduction: Existing LiDAR scene-understanding models often omit symbolic and ontological structure, limiting their handling of unknown scene changes and unnecessary background knowledge.The paper motivates explicit background knowledge for more predictable driving and navigation.
- 1 Introduction: Dense multimodal sensor data creates storage and description challenges, motivating efficient ego-centric representations that support planning and decision making.Visual-language models offer a route from visual signals to detailed natural-language descriptions and may integrate with autonomous-driving SLAM systems.
- 1 Introduction: The dataset contains 39,723 ego-centric scenes across 9 KITTI-360 sequences, with 692,081 captions covering 4,450 unique object identities.It includes 486,844 object-level and 205,237 group-level captions, providing hierarchical descriptions of static and dynamic traffic-relevant objects.
- 1 Introduction: The proposed model focuses grounding and captioning on the explicit ego-centric knowledge graph, aligning its relational neural module with the annotations' symbolic structure.This design benchmarks structured scene descriptions for autonomous-driving perception and reasoning.
2 Related Work
Prior 3D vision-language and driving-language datasets support captioning, grounding, and related tasks but generally lack hierarchical ego-centric neuro-symbolic captions for outdoor LiDAR scenes. NeuroSymbEAD targets this gap with structured object- and group-level descriptions grounded in multiple ego-relative attributes.
- 2 Related Work: Prior vision-language work is dominated by 2D image, video, and indoor RGB-D settings, including captioning, grounding, retrieval, question answering, ScanRefer, and ReferIt3D.The paper positions outdoor autonomous-driving language understanding as a less explored 3D domain.
- 2 Related Work: Existing driving-language datasets do not explicitly provide hierarchical ego-centric neuro-symbolic captions grounded in object-level 3D attributes and ontological relations.This gap concerns the combination of ego-centric spatial structure, object dynamics, and ontology in LiDAR-based driving scenes.
- 2 Related Work: NeuroSymbEAD derives structured object- and group-level captions from semantic, spatial, orientation, heading, and motion attributes relative to the ego-vehicle.Table 1 compares these characteristics with related 3D vision-language and driving-language datasets.
3 Dataset Preparation
NeuroSymbEAD prepares KITTI-360 scenes as hierarchical ego-centric captions by combining 3D object geometry, semantic attributes, spatial relations, and ontological symbols. The resulting dataset covers object- and group-level descriptions of static and dynamic entities at substantial scale.
- Annotation Pipeline: The pipeline extracts 3D bounding boxes and parameters, computes ego-relative distance, orientation, and heading, then combines these attributes with ontological symbols into object- and group-level captions.
- Dataset Background: KITTI-360 supplies over 100,000 laser scans across 73.7 km, with static and dynamic 3D elements, dense semantic and instance annotations, and 11 driving sequences.
- Dataset Statistics: 692,081 captions across 39,723 scenes and 9 KITTI-360 sequences form the NeuroSymbEAD dataset, averaging 12.26 objects and 17.42 captions per scene.
- Dataset Statistics: Object-level captions cover 10 semantic classes, while group-level captions use human and vehicle categories; the dataset contains 4,450 unique objects.
- Dataset Statistics: The annotations include 552,066 captions for static entities or groups and 140,015 for dynamic entities or groups.
- Dataset Statistics: Cars dominate object-level coverage with 375,250 captions, or 77.08%, while person and rider classes account for 47,114 captions, or 9.68%.
4 Evaluation
The evaluation benchmarks NeuroSymbEAD on 3D captioning and visual grounding using a modified 3DJCG model with ground-truth bounding boxes. The validation results assess caption quality, grounding accuracy, and qualitative dense-caption predictions.
- 4.2 Experimental Setup: The benchmark uses 3DJCG without VoteNet’s detection head, directly supplying ground-truth boxes to isolate visual understanding from detection errors.The modified setup forwards ground-truth box parameters through fully connected layers and incorporates object–ego distances in the Relation Encoding Module.
- 4.2 Experimental Setup: Caption quality is evaluated with BLEU-4 for linguistic accuracy and ROUGE for reference-content recall.Together, these metrics evaluate coherence, contextual suitability, and content completeness of generated captions.
- 4.4 Experimental Results: 96.42% reference accuracy, with BLEU 54.67 and ROUGE-L 61.45, is achieved on captioning and visual grounding using ground-truth bounding boxes.The evaluation uses sequences 9 and 10, and the ground-truth boxes eliminate the need for IoU calculations.
- 4.4 Experimental Results: Figure 5 visualizes predicted group- and object-level captions overlaid on one validation scene, with the ego vehicle shown in green.The qualitative visualization illustrates the model’s dense caption predictions for multiple scene objects.
5 Conclusion
NeuroSymbEAD provides a large-scale neuro-symbolic caption dataset that encodes spatial, dynamic, and semantic attributes as an ego-centric ontology. Its structured-template pipeline supports adaptable multilevel captions and benchmarking, while the current scope remains limited in categories, relations, linguistic diversity, and evaluation settings.
- NeuroSymbEAD introduces a large-scale dataset encoding spatial, dynamic, and semantic object attributes from KITTI-360 as an ego-centric ontology.
- Its caption-generation methodology adapts to real-world autonomous-driving datasets and produces multilevel textual captions representing a lightweight ego-centric scene map.
- The dataset is benchmarked with a joint visual captioning and grounding network to assess its performance limits.
- The current system covers traffic-relevant human and vehicle categories with a sparse ontology, omits many object, map, and intent relations, and uses structured templates that limit linguistic diversity.
- The benchmark currently relies on ground-truth boxes and one captioning/grounding baseline, with future evaluation planned for predicted detections and modern driving VLMs.