Source-linked AI summary
Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language
Julian Truetsch, Felix Hauser, Christoph Stiller, Frank Bieder
TL;DR
Understanding dataset composition matters for reliable autonomous driving, but subset differences can be subtle and exceed what existing labels capture. This paper adapts set-difference captioning to object-centered driving data and introduces an in-domain benchmark. Across methods, two-stage proposer-to-ranker pipelines are most robust under sparsity and noise.
Problem
Dataset introspection needs ways to reveal meaningful subset differences, including subtle changes that may not be readily apparent to people or captured by existing labels.
Source: §I · p. 1Method
The paper compares object-centered image patches in a two-stage proposer-to-ranker setting and introduces AD-Diff Bench with controlled purity and concentration.
Source: B. Ranker · p. 5, §VII · p. 8Results
Across methods, two-stage proposer-to-ranker pipelines are most robust under sparsity and noise; image- and caption-based proposers achieve similar accuracy and significantly outperform feature-based proposers.
Source: §VII · p. 8Takeaways & Limitations
Pairwise natural-language comparisons can support scalable dataset introspection by revealing shifts, gaps, trends, and biases that fixed labels may not capture.
Source: §I · p. 1
Abstract
from arXiv · showhide
Understanding the composition of large-scale autonomous driving datasets is essential for safety, robustness, and reliable operation across domains. For example, domain shift between locations could lead to the operating environment being misaligned with the training data, resulting in potentially dangerous performance degradation. Yet, existing data analysis pipelines largely rely on metadata, predefined labels, or manual inspection, which provide limited semantic insight or do not scale. This paper studies set difference captioning: given two subsets of images, the goal is to produce a natural-language hypothesis describing differences between the target and reference set. Building on a two-stage formulation, we adapt the method to autonomous driving by focusing on object-centric patches derived from object detection, which simplifies aggregation and enables attribution of differences to specific object instances or categories. To evaluate this setting in-domain, we introduce a new benchmark, AD-Diff Bench. Low-concentration experiments assess the suitability of set-difference-captioning approaches to sparse, real-world differences. We restrict our experiments to open-weight models to support reproducibility and ease of deployment. The proposed benchmark and analysis provide a step towards practical, human-interpretable dataset introspection for autonomous driving datasets. Our implementation and benchmark dataset are available at https://github.com/KIT-MRT/AD-Diff
I. INTRODUCTION
Reliable, robust data-driven driving depends on data quantity and quality, while metadata and available annotations provide only limited semantic insight. The paper adapts set-difference captioning to driving data and studies its use for sparse, safety-relevant differences.
I. INTRODUCTION
- Data quantity and quality are primary factors in achieving reliability and robustness for data-driven autonomous driving.The introduction frames dataset composition as consequential for systems built from learned components.Source: §I · p. 1
- Metadata and partial semantic annotations constrain dataset analysis to predefined labels, providing only limited insight into image-set differences.The paper frames natural-language difference descriptions as a way to compare subsets beyond those fixed labels.Source: §I · p. 1
- Object-centric set difference captioning adapts natural-language comparisons to safety-relevant objects and long-tail visual concepts in driving data.The paper presents this as a domain-tailored formulation building on prior set-difference captioning.Source: §I · p. 2
- AD-Diff Bench provides a benchmark dataset and evaluation protocol, while the paper studies differences in the high-sparsity regime.The stated aims include reproducible comparisons and assessing real-world applicability to distribution shifts and safety-relevant dataset diagnostics.Source: §I · p. 2
II. RELATED WORK
Prior dataset-introspection approaches often depend on predefined categories or annotations, while image-difference methods and set-level captioning offer alternatives with distinct scaling and vocabulary constraints. The paper builds on set-difference captioning for autonomous driving.
II. RELATED WORK
- Dataset-introspection work often relies on predefined categories or annotations, or targets narrow tasks such as finding challenging subsets.This paper instead focuses on open-ended, human-interpretable descriptions of differences between subsets.Source: §II · p. 2
1) Vision Language Models:
- Vision-language models combine vision encoders with language models, and open-weight models such as Qwen3-VL and InternVL-3.5 enable deployment without proprietary APIs.The passage identifies data privacy and offline operation as relevant considerations for autonomous driving.Source: 1) Vision Language Models · p. 2
2) Difference and Change Captioning:
- Prior difference- and change-captioning methods compare image pairs, but scaling them to thousands of images entails quadratic comparisons and fragmented descriptions.These limitations matter for dataset-level analysis.Source: 2) Difference and Change Captioning · p. 2
3) Set Difference Captioning:
- VisDiff introduced a two-stage proposer-ranker for image-set differences, while GS-CLIP retrieves from a fixed text bank; this paper adapts VisDiff to driving with open-weight models and object-centric inputs.VisDiff combines a caption-based proposer with a CLIP-based ranker.Source: 3) Set Difference Captioning · p. 2
4) Domain Shift Datasets:
Existing set-difference benchmarks include few road-environment images, motivating AD-Diff Bench as a benchmark of domain-relevant autonomous-driving images. Set difference captioning describes how a target image set differs from a reference set.
4) Domain Shift Datasets:
- MetaShift and VisDiffBench provide few road-environment images, so AD-Diff Bench fills a domain-specific benchmark gap with autonomous-driving images.VisDiffBench contains web-scraped image-set pairs and additional pairs from ImageNetR and ImageNet*.Source: 4) Domain Shift Datasets · p. 2
III. SET DIFFERENCE CAPTIONING FOR AUTONOMOUS DRIVING
- The comparison question determines how data are split into target and reference subsets before applying set difference captioning.Examples include splitting by location or accident label.Source: §III · p. 2
A. Set Difference Captioning
- A correct difference description characterizes a concept that is more true of target set A than reference set B.The benchmark's ground-truth descriptions describe the target set.Source: A. Set Difference Captioning · pp. 2–3
B. Object-centric Set Difference Captioning
Whole-image comparisons in driving scenes can miss relevant differences and make aggregation difficult, motivating object-centric comparisons. AD-Diff Bench uses challenging, diverse set pairs and includes controls for sparse or noisy differences.
B. Object-centric Set Difference Captioning
- Complete-image comparisons can miss relevant differences among many traffic participants and make descriptions difficult to aggregate across sets.Descriptions may combine information about different objects across images.Source: B. Object-centric Set Difference… · p. 3
- Object-centric set difference captioning compares sets of objects, generally from the same class, to ease aggregation and permit attribution to instances.The method extracts an object-centered patch from a selected image, using detections or annotations and enlarging the patch by 50% for context.Source: B. Object-centric Set Difference… · p. 3
A. Set Differences Dataset
- AD-Diff Bench targets domain-relevant set pairs with subtle differences between traffic-relevant objects of the same category.The benchmark includes web-scraped, annotation-filtered, and CLIP-filtered datasets, collected using three approaches.Source: §IV · p. 3, A. Set Differences Dataset · pp. 3–4
1) Web-scraped:
AD-Diff Bench includes web-scraped, annotation-filtered, and CLIP-filtered image-set pairs, covering both web imagery and patches from autonomous-driving datasets.
1) Web-scraped:
- 180 web-scraped pairs use the top 100 Bing results for each target caption, with pairs manually filtered and grouped into three difficulty levels and six object categories.The categories include pedestrians, cars and light trucks, micromobility, large vehicles, traffic control devices, and miscellaneous objects.Source: 1) Web-scraped · p. 4
2) Annotation-filtered:
- 60 annotation-filtered pairs use same-class object patches from KITTI, nuImages, and Waymo, with set membership based on fine-grained annotations and descriptions manually defined from filtered attributes.The split ranges from 10 to 8,000 images per pair and includes annotation-filtered-60 and annotation-filtered-39 sub-splits.Source: 2) Annotation-filtered · p. 4
3) CLIP-filtered:
- 80 CLIP-filtered pairs use OpenCLIP to select 100 target patches per set from autonomous-driving data, with a same-category random sample forming each reference set.Pairs with too many incorrect matches are discarded by manual inspection.Source: 3) CLIP-filtered · p. 4
B. Set Difference Captioning of Diluted Sets
The benchmark tests how set-difference captioning handles noisy sets and sparse differences, then evaluates generated captions with an open-weight language-model judge whose scores were checked against human labels.
B. Set Difference Captioning of Diluted Sets
- Purity tests noisy sets by swapping images between two sets; lower purity means more swapped images, and zero purity means half the images are swapped.This benchmark concept was introduced by VisDiffBench.Source: B. Set Difference Captioning of… · p. 4
- Dilution tests sparse differences by adding images from a third set to both sets; lower concentration means greater dilution, and the dilution set is chosen to make high-dilution tests challenging.Sparse differences may be important even when the relevant images are rare, as illustrated by the ambulance example: 42 of 784,444 vehicle annotations in nuImages are ambulances.Source: B. Set Difference Captioning of… · pp. 4–5
C. Evaluation
- The open-weight gpt-oss-120b judge scores semantic equivalence to ground truth as 0, 0.5, or 1, and Acc@N averages the highest score among the top-N hypotheses across pairs.More than 1,000 hypotheses on VisDiffBench were manually labeled; gpt-oss matched human scores on 80.3% with a 0.104 mean absolute error.Source: C. Evaluation · p. 5
V. APPROACH
The approach generates candidate descriptions from samples of target and reference image sets, then ranks hypotheses against the complete sets; it compares three proposer variants.
V. APPROACH
- A proposer generates difference hypotheses from sampled subsets of target set A and reference set B, and a ranker scores them against the full sets before selecting the top-ranked explanation.The method uses publicly available open-weight models.Source: §V · p. 5
A. Proposer
- The proposer comparison uses Qwen3-VL-30B-A3B-Instruct across three rounds, sampling 20 images per set and generating ten hypotheses in each round.Source: A. Proposer · p. 5
1) Image-based:
- The image-based proposer gives sampled image subsets directly to the model to generate hypotheses.Source: 1) Image-based · p. 5
2) Caption-based:
- The caption-based proposer first captions sampled images, then gives the compiled captions to an LLM to generate a hypothesis.Source: 2) Caption-based · p. 5
3) Feature-based:
- The feature-based proposer subtracts mean image embeddings between sampled sets, then combines that direction with prompt-text embeddings to generate a hypothesis.Images are scaled to 224 × 224 so the embeddings have matching shapes for averaging.Source: 3) Feature-based · p. 5
B. Ranker
The ranker scores hypotheses on the full datasets, while a single-stage variant generates and ranks descriptions in one pass. Across the benchmark splits, autonomous-driving data is generally harder than web-scraped imagery.
B. Ranker
- The feature-based ranker computes image–hypothesis cosine similarities over the full datasets and uses AUROC as each hypothesis’s score.It is chosen because it substantially outperformed alternative ranking methods and produces continuous values.Source: B. Ranker · p. 5
C. Single-stage
- The single-stage variant gives the VLM 100 sampled images per dataset and asks it to generate and rank difference hypotheses in one inference pass.Source: C. Single-stage · p. 6
A. Comparison between Dataset Splits
- Accuracy is lowest for most approaches on annotation-filtered and highest on web-scraped data; autonomous-driving patches are often lower-resolution or degraded, and may also be more shifted from model training data.The authors say the observed difficulty supports a dedicated domain-specific benchmark, without resolving which factor contributes more.Source: A. Comparison between Dataset Splits · p. 6, Table II · p. 6
- The annotation-filtered split is harder than CLIP-filtered because its described differences can be difficult for humans to recognize, although its sets are cleaner.Examples include distinguishing a parked car from a car stopped with a driver inside.Source: A. Comparison between Dataset Splits · p. 6
- For the web-based split, image-based two-stage Acc@1 decreases across easy, medium, and hard levels: 0.84, 0.75, and 0.59.Source: A. Comparison between Dataset Splits · p. 6
B. Two-stage or Single-stage Approach?
The two-stage approach is more robust on sparse sets, while image-based and caption-based proposers perform similarly and outperform feature-based proposers. Accuracy falls at very low concentration and purity, limiting reliable detection of sparse differences.
B. Two-stage or Single-stage Approach?
- The two-stage approach is more accurate on web-scraped and CLIP-filtered splits, but performs similarly to the single-stage approach on the annotation-filtered split.The authors attribute its advantage on the sparser splits to the ranker aggregating signals across the full dataset; the single-stage approach can be much faster on large datasets.Source: B. Two-stage or Single-stage Approach? · p. 7
C. Comparison between different Proposers
- Image-based and caption-based proposers achieve about the same accuracy across all three splits, and both significantly outperform the feature-based proposer.The authors favor the image-based proposer as a simpler, marginally more efficient choice, citing modern VLMs' ability to accept multiple images natively.Source: C. Comparison between different… · p. 7
D. Noisy and Diluted Sets
- Accuracy remains stable until concentration reaches about 0.5, then drops rapidly; none of the tested approaches reliably describes differences at very low concentrations.Accuracy drops more rapidly for the single-stage approach than for two-stage approaches, consistent with the latter's ability to aggregate sparse differences across the dataset.Source: D. Noisy and Diluted Sets · p. 7
- Accuracy is fairly stable until purity reaches 0.5, then approaches zero at low purity; the single-stage proposer nevertheless predicts the original difference in 35% of cases at purity 0.At purity 0 the sets are statistically identical; retrieving the original hypothesis is therefore a false positive, and low-concentration accuracy is a more informative real-world metric than low-purity accuracy.Source: D. Noisy and Diluted Sets · p. 7
E. Application Example
The application compares nuImages pedestrian patches from Singapore Queenstown and Boston Seaport using a two-stage approach. Without ground-truth difference labels, the authors validate all top-eight hypotheses using dataset annotations and external sources.
E. Application Example
- The target comprises pedestrian patches from Singapore Queenstown and the reference comprises patches from Boston Seaport, extracted from nuImages bounding boxes and assigned by location metadata.The comparison is framed as a possible vehicle-fleet operational design domain expansion from Boston to Singapore.Source: E. Application Example · p. 7, Figure 5 · p. 8
- All top-eight generated hypotheses are validated using nuImages annotations and external sources, despite the absence of ground-truth labels for this city comparison.The approach uses an image-based proposer and ranker; the example is presented as a real-world case with sparse, diluted, initially unknown differences.Source: E. Application Example · p. 7
- Hypotheses 1, 3, 5, 6, and 8 concern greater construction activity in Singapore and construction workers near roads.In the proposed expansion scenario, these observations could prompt checks of worker detection robustness and potential hazards from denser construction sites.Source: E. Application Example · pp. 7–8
VII. SUMMARY AND CONCLUSIONS
The paper adapts set-difference captioning to object-centric autonomous-driving data and introduces AD-Diff Bench for in-domain evaluation. Experiments favor two-stage robustness under sparsity and noise, but very low concentration and purity remain difficult.
VII. SUMMARY AND CONCLUSIONS
- Object-centric patches from detections support aggregation across large collections while avoiding scene-level variation and enabling attribution to object instances or categories.AD-Diff Bench includes web-scraped, annotation-filtered, and CLIP-filtered splits, with controlled purity and concentration to model noisy and sparse differences.Source: §VII · p. 8
- Across methods, the two-stage proposer-to-ranker pipeline is most robust under sparsity and noise, while image-based and caption-based proposers perform similarly and outperform feature-based proposers.The ranker can aggregate weak signals over the full datasets.Source: §VII · p. 8
- Accuracy drops sharply at very low concentration and low purity, leaving reliable identification of rare, safety-critical differences an open challenge.The authors identify improved robustness to extreme sparsity as a focus for future work.Source: §VII · pp. 8–9