PaperlayerView exact revision on arXiv

Source-linked AI summary

Front-to-Back: Benchmarking Vision-Language Models for Asymmetric Cross-View Vehicle Re-Identification

Moseli Mots'oehli, Thulani Babeli

arXiv:2609.39492v2cs.CV

TL;DR

Front-to-rear vehicle matching is difficult because views differ and the vehicle may reappear after an observation gap. Front2Back-ReID benchmarks this task across VLMs, retrieval systems, and humans using varied front-image evidence. The strongest VLM did not significantly outperform frozen SigLIP2, and humans remained more accurate.

  • Problem

    Front-to-rear matching asks whether a vehicle can be identified after a blind interval and major viewpoint change, without intervening observations or a reliable camera transform.

    Source: §1 · p. 2

  • Method

    Front2Back-ReID evaluates seven zero-shot VLMs, four retrieval baselines, and human participants on manually verified handovers using full scenes, target crops, or silhouettes.

    Source: §1 · p. 2, §9 · p. 14

  • Results

    76.6% Rank-1 accuracy on crops for the strongest VLM was not significantly above frozen SigLIP2 at 74.0%, while humans scored 92.2–94.0%.

    Source: §1 · p. 2, §9 · p. 14

  • Takeaways & Limitations

    VLMs generally performed worse on full scenes than crops, while humans remained more reliable than the strongest VLM.

    Source: §1 · p. 2, §9 · p. 14

    The benchmark is limited to South African roads, is dominated by daytime scenes, and its input conditions change multiple factors at once.

    Source: §8 · p. 14

Abstract

from arXiv · show

Matching the same vehicle across front and rear cameras is difficult because the cameras do not share a view and the vehicle's appearance changes substantially. We introduce Front2Back-ReID, a benchmark of 500 manually verified vehicle handovers from 20 recording sequences in South Africa. Each example asks a model to match a vehicle highlighted in a front-camera image to the same vehicle among at least three candidates in a later rear-camera image. We evaluate seven zero-shot vision-language models, four image-retrieval baselines, and 25 human participants. Models are tested using full front RGB images, cropped target vehicles, and binary silhouettes. The strongest VLM achieved 76.6 percent Rank-1 accuracy on target crops, compared with 74.0 percent for the frozen SigLIP2 baseline; this difference was not statistically clear. Human participants achieved 94.0 percent accuracy with full images and 92.2 percent with target crops. Under our evaluation setup, enabling reasoning improved accuracy across all three input conditions for every model evaluated in both modes. We also found that VLMs generally performed worse on full scenes than on target crops. These results show that general-purpose VLMs do not yet consistently outperform strong visual retrieval for front-to-rear vehicle matching, while humans remain substantially more reliable.

1 Introduction

Front-to-rear vehicle matching tests identity association across a blind interval and a large viewpoint change. Front2Back-ReID compares VLMs, retrieval systems, and humans while varying the front evidence, and finds that VLMs do not consistently outperform retrieval or humans.

1 Introduction

  • Front-to-rear matching asks whether a vehicle can be identified after it leaves the front view and reappears behind the ego vehicle without intervening observations or a reliable camera transform.Viewpoint changes and new rear-scene distractors complicate matching among at least three candidates.Source: §1 · p. 2
  • 76.6% Rank-1 accuracy on crops for the strongest completed VLM was not significantly above frozen SigLIP2 at 74.0%, while humans reached 94.0% on full scenes and 92.2% on crops.Crops outperformed full scenes in every informative VLM comparison.Source: §1 · p. 2

2.1 Cross-view Vehicle Re-identification

  • Unlike fixed-camera vehicle Re-ID, Front2Back-ReID uses ego-vehicle cameras and South African road recordings, which the authors identify as the first vehicle Re-ID benchmark recorded on Southern African roads.Existing benchmarks largely use static cameras and have little to no coverage of Africa.Source: §2.1 · p. 3

2.2 Foundation Models for Visual Retrieval

  • Frozen DINOv2 and SigLIP2 serve as representation baselines for testing whether language supervision or generative multimodal models add gains beyond embedding similarity.These encoders group similar content rather than explicitly separating individual instances.Source: §2.2 · p. 3

2.3 Visual and Spatial Reasoning in VLMs

  • The study evaluates VLMs directly comparing images without video or cross-rig geometry, and tests full scenes versus crops and reasoning modes.The comparisons vary multiple factors, so they do not isolate context or reasoning alone.Source: §1 · p. 2, §2.3 · p. 4

2.4 Diagnostic Benchmarks for Visual Grounding

  • Front2Back-ReID holds the rear gallery fixed while varying front evidence among RGB, crop, and silhouette, with retrieval baselines and humans as references.This diagnostic evaluates real handovers where viewpoint reverses and traffic supplies distractors.Source: §2.4 · p. 4

3 Benchmark and Task

Front2Back-ReID is a closed-gallery task: match a highlighted front-left vehicle to its unique candidate in a later rear-left image. The benchmark uses manually verified handovers and records difficulty attributes for analysis rather than as model inputs.

3.1 Task Definition

  • Each task presents a highlighted front-left vehicle and at least three labelled candidates in a later rear-left image, with no video, camera transform, or range estimates supplied to matchers.Exactly one rear candidate matches the target.Source: §3.1 · p. 4

3.2 Data Collection and Annotation

  • Curators corrected and verified vehicle proposals and matches across 20 recording sequences, producing 500 handovers.The low-cost stereo cameras were mounted on the windscreen facing forward and the rear window facing backward; stereo depth supports difficulty analysis only.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 Construction and statistics

  • Rear-side competition is measured by Vi, the number of vehicles among rear candidates, which differs from gallery size Ki on 164 pairs.Count-stratified results use Vi; the 3, 4–5, and ≥6 groups contain 45/163/292 pairs by Vi versus 38/127/335 by Ki.Source: §3.3 · p. 6

3.4 Difficulty Stratification

  • Difficulty groups cover rear-vehicle count, target size, occlusion, and truncation, while depth is divided into five bins using the 341 pairs with valid S2M2 estimates.These attributes are used only for analysis and are not provided to models or evaluators.Source: §3.4 · p. 6, Figure 4 · p. 6

4 Evaluation Protocol

The evaluation holds the rear gallery and task rules constant while changing how the front target is presented. The three conditions differ in more than context: cropping enlarges the target, while silhouettes remove color and texture.

4 Evaluation Protocol

  • Each model receives the same rear gallery and one of three front inputs: a marked full RGB scene, an enlarged target crop, or a shape-only silhouette.The crop changes both context and target scale, while the silhouette also removes color and texture.Source: §4 · p. 7, Figure 5 · p. 7
  • All conditions share task rules and output format, while prompts change only the description of available front evidence.Rear candidates are identified by aliases, without box coordinates, detector classes, or confidence values.Source: §4 · p. 7, Figure 6 · p. 8
  • The prompt directs evaluators to choose exactly one candidate using permitted visual cues and to treat hidden features as unknown rather than mismatches.Candidate order, image position, box size, and road plausibility are explicitly excluded as identity evidence.Source: §4 · p. 8
  • RGB prompts permit color and texture cues, whereas mask prompts restrict matching to silhouette, proportions, roofline, wheel placement, cargo structure, and visible accessories.Mask prompts explicitly prohibit inferring color or texture.Source: §4 · p. 8

5 Models, Baselines, and Human Reference

The evaluation compares zero-shot VLMs with visual retrieval controls and a human reference on the same matching task. Rank-1 accuracy measures whether the correct candidate is ranked first, with chance performance set by gallery size.

5 Models, Baselines, and Human Reference

  • Frozen DINOv2 and SigLIP2 embeddings rank candidate crops by cosine similarity, while HSV matching tests color-only retrieval and uniform sampling sets chance.The four controls isolate progressively richer visual cues.Source: §5 · p. 8
  • The VLM set spans locally run small open models, hosted larger open models, and closed frontier models selected as an upper reference.The evaluated-model table lists experiments run from 24 June to 5 July 2026.Source: §5 · p. 8, Table 2 · p. 9
  • Twenty-five adults each completed 40 trials, balanced across difficulty, scene conditions, and source videos, yielding 1,000 judgments.Each pair received one judgment per condition from different participants, and no participant saw both versions of the same pair.Source: §5 · p. 9

6 Evaluation Metrics

  • Rank-1 accuracy is the primary metric: the share of batches where the highest-scoring candidate is the match; chance is 1/Ki.Because galleries are event-local, full-dataset retrieval rank is not meaningful.Source: §6 · p. 9
  • Accuracy intervals use 100,000 pair-level BCa bootstrap resamples, while model outputs that fail parsing or name an unlisted candidate count as incorrect.Human–model comparisons are paired over shared pairs; pair-level intervals may be somewhat narrow because pairs from the same recording or participant can be correlated.Source: §6 · p. 9

7 Results and Analysis

Across the tested VLMs, target crops generally outperform full scenes, and reasoning improves results for models tested in both modes, but the strongest crop result is not clearly above frozen retrieval. Humans remain more accurate, especially on small rear targets.

7 Results and Analysis

  • Without reasoning, every informative VLM except C1-only LLaVA-OneVision scores lower on full scenes than crops, while silhouettes trail crops by 27.2–42.4 points.Full-scene losses range from 6.6–16.4 points; the input changes also alter target scale or representation, so the differences do not isolate context alone.Source: Table 3 · p. 10, §7 · p. 11
  • GPT-5.5 is the best crop model with reasoning, 2.6 points above SigLIP2, but its paired 95% interval [−2.2, 7.4] includes zero.The comparison has 81 pairs correct only for GPT-5.5 and 68 correct only for SigLIP2.Source: Table 4 · p. 10, §7 · pp. 10–11
  • Every model evaluated in both modes improves with reasoning, but GPT-5.4 mini gains 11–14 points across conditions while GPT-5.5 and Gemini 2.5 Flash gain most on silhouettes.The latter models show minimal crop gains, and paired intervals crossing zero do not establish a directional gain.Source: Table 4 · p. 10, §7 · p. 11, Figure 9 · p. 12
  • Humans reach 94.0% on full RGB and 92.2% on crops; incorrect answers take median 20.0 versus 7.8 seconds on full RGB and 20.5 versus 7.0 seconds on crops.The longer times for incorrect answers may reflect harder pairs or hesitation.Source: §7 · pp. 11–12, Figure 10 · p. 12
  • On the smallest third of rear targets, pooled human accuracy stays at 93.2%, while crop accuracy without reasoning falls to 38.7% for Llama 4 Scout.Human accuracy remains high even for small or heavily occluded targets, where models degrade; the human difficulty-stratum figure pools full RGB and crop judgments.Source: Figure 7 · p. 11, Figure 8 · p. 11, §7 · p. 12
  • Five of the ten pairs missed by humans in one condition were also missed in the other, while front-target depth shows no consistent trend across five bins.Each pair received only one judgment per human condition, making the shared misses a coarse difficulty signal.Source: §7 · pp. 12–13, Figure 11 · p. 13

8 Limitations

Front2Back-ReID is limited in geographic, scene, and sensor scope, and its evidence conditions and uncertainty estimates impose additional interpretive boundaries.

8 Limitations

  • The 500 handovers come from 20 recordings in five South African cities, mostly daytime scenes, limiting broad claims about other countries, fleets, or sensors.Daytime scenes account for 97.6%; depth is available for only 341 pairs, limiting range analysis to that subset.Source: §8 · p. 14
  • Input conditions change multiple factors at once, and intervals treat pairs as independent despite possible recording or participant correlation.Each pair has one human judgment per condition; resampling by recording and collecting more judgments per pair could yield more conservative intervals and a sharper human reference.Source: §8 · p. 14

9 Conclusion

Front2Back-ReID tests front-to-rear matching with verified handovers and finds that the best VLM does not significantly exceed frozen SigLIP2, while humans score higher. Its controlled comparison is scoped to these South African recordings and current evidence conditions.

Supplementary Material

  • Front2Back-ReID contains 500 verified front-to-rear vehicle handovers from 20 South African recordings; it compares specified front evidence and model and human evaluators on the same decisions with fixed rear galleries.The 341 pairs with valid approximate front-stereo depth form five near-equal bins.Source: §9 · p. 14, S1 Dataset composition and difficulty… · p. 18

9 Conclusion

  • 76.6% crop accuracy for reasoning-enabled GPT-5.5 was not significantly above SigLIP2’s 74.0%, while humans reached 92.2–94.0%.Full-scene human accuracy was 94.0%, compared with 92.2% on crops; VLM full scenes scored below crops, a difference not attributable to context alone because cropping also changes target scale.Source: §9 · p. 14

S1 Dataset composition and difficulty definitions

  • Difficulty analyses use vehicle-only counts and scale, occlusion, truncation, and approximate-depth bins; these annotations support analysis but are not supplied to matchers.Gallery size and vehicle-only counts differ on 164 pairs; depth analyses use the 341 pairs with valid estimates.Source: S1 Dataset composition and difficulty… · pp. 18–19, Table S1 · p. 18

S2 Additional protocol details and difficult cases

The protocol holds the rear gallery fixed while varying front evidence, but crop and silhouette comparisons change multiple visual properties. Crops outperform full scenes in the reported VLM comparisons, while reasoning gains differ by model and input.

S2 Additional protocol details and difficult cases

  • The rear gallery stays fixed, but crops remove scene context and enlarge the target, while silhouettes retain shape and discard color and texture.The difficult-case examples include many candidates, a distant target, occlusion, and boundary truncation.Source: S2 Additional protocol details and… · p. 19, Figure S1 · p. 20

S3 Evidence-condition comparisons

  • Without reasoning, crops beat full RGB by 6.6–16.4 points, while silhouettes trail crops by 27.2–42.4 points; differences reflect input presentation, including target scale and available cues.LLaVA-OneVision selected C1 on every trial, so its unchanged accuracy does not indicate insensitivity to missing evidence.Source: S3 Evidence-condition comparisons · p. 19

S4 Paired reasoning comparisons

  • GPT-5.4 mini gained 14.2, 12.6, and 11.4 points on full RGB, crops, and silhouettes, respectively; evidence for reasoning gains in other matched cells was mixed.The intervals for GPT-5.5 crops and Gemini 2.5 Flash full RGB and crops crossed zero.Source: S4 Paired reasoning comparisons · pp. 19–20

S5 Human accuracy, latency, and errors

Human participants scored highly on both full-RGB and crop trials. Their decision times varied by condition and correctness, while the supplementary latency analysis filters long responses and the shared-miss examples remain limited observations.

S5 Human accuracy, latency, and errors

S5.1 Decision latency

  • Median decision time was 8.09 s for full RGB and 7.27 s for crops across all responses; incorrect answers took longer than correct ones in both conditions.The main-paper timing plot includes all 1,000 judgments; supplementary plots exclude 10 responses over 60 s and are descriptive, without adjustment for participant or question order.Source: S5.1 Decision latency · p. 21
  • The supplementary latency plots exclude 10 responses longer than 60 s, leaving 492 full-RGB and 498 crop responses.The accompanying figure shows medians and IQRs by difficulty stratum after this cutoff.Source: S5.1 Decision latency · p. 21, Figure S2 · p. 21

S5.2 Pairs missed in both human conditions

  • Ten pairs were missed in both human conditions, but with only two judgments per pair these examples do not establish majority failure or intrinsic ambiguity.The supplementary figure shows medians and IQRs by difficulty stratum, with counts pooled across input conditions.Source: S5.2 Pairs missed in both human… · p. 21, Figure S3 · p. 22

S6 Qualitative model reasoning examples

Saved model responses include plausible but misleading visual cues, but they are not verified explanations of model decisions. In one illustrated crop pair, all complete front-crop VLM runs selected the same incorrect candidate.

S6 Qualitative model reasoning examples

S7 Depth sensitivity

Depth bins show no monotonic accuracy trend, while target area has a clearer association with accuracy. The supplementary material also reports recording durations and incomplete reasoning-enabled runs.

S7 Depth sensitivity

  • SigLIP2 crop accuracy rises from 50.0% in the smallest-area quintile to 91.0% in the largest, while depth-bin accuracy is non-monotonic.GPT-5.5 changes less with target area; these binned comparisons do not isolate scale from other example properties.Source: S7 Depth sensitivity · p. 25, Figure S6 · p. 25

S8 Recording-sequence durations

S9 Excluded incomplete runs

  • Qwen 3.6 27B reasoning-enabled runs were excluded after request failures and truncated completions prevented all three input-condition runs from reaching 500 trials.The runs reached 470/500 full-RGB, 480/500 crop, and 416/500 silhouette trials; this records limits under the evaluated settings, not a provider reliability ranking.Source: S9 Excluded incomplete runs · p. 26

Related papers

Loading 2609.39492v2…