Source-linked AI summary
Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel Kochenderfer, Chiho Choi, Behzad Dariush
TL;DR
Autonomous systems need interpretable scene understanding for trust, while no comprehensive real-world driving dataset combines the required traffic features and importance reasoning. The proposed ego-centric multimodal dataset combines 2D image and 3D LiDAR features, dense annotations, natural-language explanations, and a model jointly predicting importance and generating captions. The joint model outperforms unimodal baselines on several captioning metrics, while 88.23% and 98.61% of selected majority-label samples exceed 60% annotator consistency for High and Not-Important cases.
Problem
Autonomous systems need interpretable scene understanding for trust, while no comprehensive real-world driving dataset combines the required traffic features and importance reasoning.
Source: §1 · p. 1Method
The proposed ego-centric multimodal dataset combines 2D image and 3D LiDAR features, dense annotations, natural-language explanations, and a model jointly predicting importance and generating captions.
Source: §1 · pp. 1–2Results
The joint model outperforms unimodal baselines on several captioning metrics, while 88.23% and 98.61% of selected majority-label samples exceed 60% annotator consistency for High and Not-Important cases.
Source: §3.3.2 · p. 6, §5.4.1 · p. 8Takeaways & Limitations
Rank2Tell enables joint importance prediction and natural-language reasoning with frame-level traffic annotations for visual scene comprehension and autonomous-system interpretability.
Source: §6 · p. 8
Abstract
from arXiv · showhide
The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Furthermore, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations.
1. Introduction
Rank2Tell addresses limited trust in autonomous systems by introducing a multimodal dataset and joint model for ranking important traffic agents and explaining their importance. The approach targets interpretable scene understanding through dense annotations and natural-language reasoning.
Motivation
- Identifying critical traffic agents and explaining them can support risk-focused computation and human-facing transparency in autonomous systems.The paper also describes potential uses in conveying decisions, improving situational awareness, and warning passengers about hazards.Source: §1 · pp. 1–2
- The work responds to limited public trust and the absence of a comprehensive real-world dataset combining scene features, importance levels, and human-interpretable reasoning.A 2020 PAVE survey reported that 60% of respondents would be more likely to trust autonomous vehicles if they better understood model rationale.Source: §1 · p. 1
Contribution
- The dataset combines 2D images and 3D LiDAR with dense semantic, temporal, relational, and natural-language annotations for agents influencing ego-vehicle decisions.These annotations are intended to support scene understanding and explanations of why an agent matters.Source: §1 · p. 1
- A joint model uses multimodal 2D+3D features to predict agent importance levels and generate captions.The paper establishes benchmark tasks for joint importance classification and natural-language captioning.Source: §1 · p. 2
- Rank2Tell introduces the first multimodal dataset for importance-level ranking and natural-language explanation in urban traffic scenarios.Its annotations use visual question answering that combines video and object-level attributes.Source: §1 · p. 2
2. Related Work
Prior work studies object importance, visual captioning, and driving datasets, but Rank2Tell focuses on explaining the importance of multiple traffic agents using spatio-temporal observations.
2.1. Object Importance Classification
- Existing importance methods include pixel-level saliency, single-object selection, binary importance classification, and multi-level agent classification, but generally lack object-specific reasoning explanations.Only one cited study couples language with importance estimation, and it describes a single important object.Source: §2.1 · p. 2
2.2. Dense Captioning
- Dense captioning research generates language for multiple image regions, while Rank2Tell targets explanations for important traffic objects from spatio-temporal observations.The comparison connects Rank2Tell to dense image captioning while distinguishing its traffic-scene setting.Source: §2.2 · p. 2
2.3. Datasets
- Driving datasets cover object localization, anomaly detection, ego-action reasons, and important-object captions with spatio-temporal relationships.DRAMA is identified as the most relevant prior dataset, with major differences presented in Table 1.Source: §2.3 · p. 2
3. Rank2Tell Dataset
Rank2Tell is a multi-modal ego-centric dataset for ranking important traffic agents and explaining their importance in complex urban scenes. It combines diverse annotations, joint-task applications, dataset statistics, and inter-annotator consistency analysis.
3. Rank2Tell Dataset
- 116 intersection clips were collected with synchronized three-camera video, LiDAR, GPS, and vehicle CAN data, capturing ego-vehicle entry and exit from varied traffic environments.Annotators viewed stitched front, left, and right camera images with ego speed and intention, using only 40 historical frames while labeling every fourth frame.Source: §3 · p. 2, §3.1 · p. 2
3.1. Annotation Methodology
- The study uses five annotators to identify and localize agents important to the ego vehicle, rank them as High, Medium, or Low importance, and explain significance with free-form captions.Its annotation schema combines What, Which, Where, and How questions requiring predefined single-choice answers with an open-ended Why caption incorporating those answers in a 3W+1H format.Source: §3.1 · pp. 2, 4, 5
3.2. Features and Applications of Rank2Tell
- Multiple captions capture diversity in how people perceive importance, while the dataset’s semantic, spatial, temporal, and relational attributes support scene graphs, situational awareness, and interpretable models.The benchmark jointly addresses important-agent ranking and caption generation using multimodal features.Source: §3.1 · p. 5, §3.2 · p. 5
3.3.1 Dataset Statistics
- High-importance infrastructure such as traffic lights is common, and importance depends on ego intention: agents on the corresponding turning side are often high importance, whereas straight driving distributes agents across lanes.This dependence makes importance ranking nontrivial to estimate from ego intention alone; captions commonly encode agents’ intention, motion direction, and location.Source: §3.3.1 · pp. 5–6
3.3.2 Consistency Analysis
- The annotations show strong inter-rater agreement, with ICC 0.92 and more than 60% consistency for 88.23% of High-majority and 98.61% of Not-Important-majority instances.Ground-truth importance is formed from the mode of annotator labels, using the highest importance in ties and restricting the mode to annotators who marked an agent important to avoid underestimating importance.Source: §3.3.2 · p. 6
4. Methodology
The joint model combines multimodal object features and relational reasoning to classify importance and generate captions. Its training objective jointly optimizes importance classification and caption generation while penalizing underestimation.
4. Methodology
- The model jointly predicts object importance and generates context-aware captions using 2D features, 3D point-cloud features, relational graphs, and ego information.Its five components are 2D extraction, 3D extraction, relational graph modeling, importance classification, and caption decoding.Source: §4 · p. 6, Figure 5 · p. 7
4.1. Feature Extractor
- The feature extractor combines RGB, depth, semantic, inertial, bounding-box, and point-cloud inputs, then models nearby-object relations with a K-nearest-neighbor graph.Graph-enhanced object features are combined with relational, global, ego, and intention features before classification and captioning.Source: §4.1 · pp. 6–7
4.2. Training Objective
- The joint loss is a weighted sum of importance-classification and caption-generation losses, expressed as L = αL_imp + βL_cap.Importance classification uses cross-entropy, while the objective additionally penalizes predictions that underestimate an agent’s ground-truth importance.Source: §4.2 · p. 7
5. Experiments
The experiments benchmark importance classification and caption generation against task-specific baselines using standard metrics. The proposed joint model benefits from multimodal features and joint training, outperforming unimodal baselines on several captioning metrics.
5. Experiments
- Importance classification is evaluated with two-class and four-class settings, while captioning is evaluated with BLEU-4, METEOR, ROUGE, and CIDEr.F1 score and accuracy measure importance estimation, and the baselines use matched backbone modules and hyperparameters for fair comparison.Source: §5 · p. 7, §5.3 · p. 8, Table 4 · p. 8
5.1. Importance Level Classification: Baselines
- The importance baselines progress from tracklet-based OIE to interaction-graph INTERACT and ego-aware IOI, with IOI benefiting from speed, acceleration, and intention features.Because no importance-ranking baseline uses 3D features, the comparison benchmarks those baselines with 2D features only.Source: §5.1 · pp. 7–8, §5.4.1 · p. 8
5.4.1 Quantitative Comparison
- The joint model outperforms unimodal captioning baselines on several metrics by combining 2D and 3D features and jointly training importance classification with caption generation.The paper attributes the captioning advantage to complementary information from both tasks and to 3D cues such as agent distance and height.Source: §5.4.1 · p. 8, Table 5 · p. 8
5.2. Captioning: Baselines
- For captioning, Scan2Cap outperforms Show, and Tell because 3D point clouds and bounding boxes encode spatial cues that are more informative than 2D features.These cues connect agent location, including left- or right-side placement, to generated words.Source: §5.4.1 · p. 8, §5.2 · p. 8
5.4.1 Quantitative Comparison
- An ablation integrates action attributes with object features across baselines to assess the usefulness of Rank2Tell annotations.Detailed ablation results are reported in the supplementary materials.Source: §5.4.1 · p. 8
6. Conclusion
Rank2Tell is a multimodal urban-traffic dataset for joint importance prediction and reasoning with frame-level annotations. It integrates spatial, temporal, relational, and semantic information to support scene comprehension and autonomous-system interpretability.
6. Conclusion
- Rank2Tell enables joint importance-level prediction and reasoning in urban traffic scenes through frame-level, object-level annotations and language interpretations from the ego driver’s perspective.Its annotations integrate spatial, temporal, and relational features for important objects.Source: §6 · p. 8