PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims

Yifan Zhang, Kai Wang

arXiv:2609.05761v1cs.CVcs.AI

TL;DR

Visual geolocation benchmarks often omit the coarse location context users provide, despite its relevance to identifying places and checking location claims. GeoContext builds a shared resource for localization and claim verification, varying reference points while holding images fixed. Models often repeat hints at similar rates across conditions even as the error cost rises with hint distance, and their verification sensitivity remains limited near the tolerance boundary.

  • Problem

    Visual geolocation benchmarks typically ask where an image was captured without accounting for the location context users often provide.

    Source: §1 · p. 1

  • Method

    GeoContext uses a context ladder of nearby reference points to evaluate open-ended localization from true, coarse hints and verification of location claims.

    Source: §1 · p. 2

  • Results

    Models repeat hints at similar rates across distance and referenceability conditions while resulting localization error grows with hint distance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that hint reliance has a growing cost and that image legibility conditions whether models can discount context.

    Source: §1 · p. 2

    Echoing was not manipulated, so its proposed link to inability to identify a scene is an interpretation rather than an experimentally established result.

    Source: §6.2.1 · p. 12

Abstract

from arXiv · show

Visual geolocation benchmarks typically ask a model where an image was captured without accounting for the location context that users often provide. We introduce GeoContext, a resource supporting two complementary tasks: GeoHint, open-ended localization given a true but coarse location hint, and GeoVerify, binary verification of whether an image was taken within 150 m of a claimed place. GeoContext constructs a context ladder by stratifying nearby reference points according to distance and referenceability, allowing the image to remain fixed while the supplied context varies. The benchmark covers 109 sites in 30 cities and evaluates five vision-language models using 21,933 GeoHint responses and 6,270 GeoVerify responses. Our evaluation reveals three main patterns. First, hint repetition varies by only 1.5 percentage points across referenceability tiers and by less than 3 points across distance bands, while the resulting localization error increases steadily with hint distance. Second, behavior depends strongly on no-context performance: at sites with low no-context accuracy, the median ratio between localization error and hint distance is approximately 1.00, whereas at higher-accuracy sites it ranges from 0.24 to 0.69. After correcting for bias introduced by the site grouping procedure, only one of the five models retains a negative accuracy estimate when given a nearby hint. Third, in GeoVerify, no model reaches d' = 1 for decoys immediately beyond the 150 m tolerance. Model rankings also change when sensitivity is separated from response bias, and 83.8% of false acceptances are reported with confidence of at least 0.8. We release the benchmark, construction pipeline, audit decisions, and scoring code.

1 Introduction

Visual geolocation can identify the right region yet name the wrong nearby entity, while ordinary coordinate-distance scores barely register that user-facing error. GeoContext studies how models use true but coarse location hints and whether they can verify location claims.

1 Introduction

  • A model can name a nearby famous building instead of the photographed one, an entity error that coordinate-distance metrics barely register.The region may be right even when the user is misled about what is in front of them.Source: §1 · p. 1
  • The understudied context is a true but coarse user hint: helpful in principle, but specific only to a neighborhood rather than a building.Prior work studied adversarial image text and captions that reveal the answer or are irrelevant.Source: §1 · p. 1
  • GeoContext provides a context ladder and supports both open-ended GeoHint localization and GeoVerify claim verification using the same resource.Its reference points are stratified by distance and referenceability, and the coordinates also serve as GeoVerify decoys.Source: §1 · p. 2
  • Hint repetition changes little across distance and referenceability, while the localization error caused by repeating a hint grows with its distance.Referenceability is legible from a place name; distance is not, but the cost of relying on it still scales.Source: §1 · pp. 1–2
  • At illegible sites answers collapse onto the hint, while at legible sites one model remains less accurate after receiving true information.Warning models that a user may be wrong helps only at legible sites; a structured evidence checklist helps in neither regime.Source: §1 · p. 2
  • GeoVerify uses coordinate-based haversine ground truth and signal detection, while geocoded scoring faults can reverse headline geolocation results.The GeoVerify measurement avoids a geocoder, judge model, or output threshold.Source: §1 · p. 2

2 Related Work

Prior geolocation work either evaluates image-only predictions or supplies text that contradicts the image or gives away the answer. GeoContext instead studies honest, coarse context and extends evaluation to verification of named location claims.

Image-only geolocation

  • Image-only distance scoring can treat a nearby wrong building as nearly correct, leaving the entity-level failure invisible.The paper retains distance thresholds for cross-site task performance and adds echo rate and error relative to hint distance.Source: §2 · p. 2

Text alongside the image

  • SIGNPOST-Bench uses adversarial place names and KoreaGEO uses irrelevant or revealing captions; neither supplies true, coarse out-of-image context.GeoContext's failures can appear at the entity level while the region stays correct.Source: §2 · p. 2

Auxiliary hints, and why they have not been studied as a variable

  • EarthWhere leaves hint distance and usefulness uncontrolled, lacks a matched no-hint condition, and scores away mere hint copying.GeoContext instead varies hint distance and referenceability so reliance can be compared with what the hint warrants.Source: §2 · p. 3

Verifying a location claim

  • Prior claim-verification approaches use physical cues or image retrieval, whereas GeoContext asks whether a general-purpose VLM can judge a named place from the scene.The closest cited benchmark uses cross-view geometry rather than a user-asserted named claim and does not measure false acceptance.Source: §2 · p. 3

3 The GeoContext resource

GeoContext is a programmatic resource that holds sites and images fixed while varying a ladder of nearby reference points. It supplies a shared audited pool for open-ended localization and location-claim verification.

3 The GeoContext resource

  • GeoContext supplies the same sites, images, and audited reference-point pool to GeoHint localization and GeoVerify claim verification.GeoHint uses a geocoder for its output threshold; GeoVerify compares known coordinates directly against its tolerance.Source: §3 · p. 3

3.1 Scope: dense, walkable urban cores

  • The dense-urban-core scope uses neighborhood-node spacing to set comparable distance bands, excluding addressing units that distort measured granularity.Car-oriented low-density areas are out of scope because “nearby” there is set by drive time; the automatic-site batch still lacks per-site verification.Source: §3.1 · pp. 3–4

3.2 Sites and images

  • The benchmark covers 109 sites in 30 cities, combining 19 hand-picked sites with 90 automatically selected sites.Site selection avoids iconic landmarks, which can saturate the baseline and undermine study of nearby-landmark substitution.Source: §3.2 · p. 4

3.2.1 Automatic site selection

  • Automatic selection screens cities for image coverage and neighborhood granularity, then randomly checks eligible Wikidata sites for street-view coverage.Random ordering avoids selecting only the most famous eligible landmarks; a narrow sitelinks band is not a reliable proxy for referenceability.Source: §3.2.1 · pp. 4–5
  • Human and model image audits agree on 343 of 429 candidates (80.0%), but Cohen’s κ = 0.185 because their judgments differ systematically.The human excludes frames with no identifiable location; the model rejects some usable views through vehicle windows, and their leakage checks catch different issues.Source: §3.2.1 · p. 5

3.3 The context ladder

  • A text-only auditor rates nearby place usability and referenceability, and its tier assignments are robust across auditor models and prompt languages.Referenceability means how often locals or tourists would use a place to describe where they are, not how famous it is.Source: §3.3 · pp. 5–6
  • The context ladder stratifies reference points into distance bands and high, medium, and low referenceability tiers, with up to 18 draws per site.Cells can run short, so per-site counts must be weighted in analysis.Source: §3.3 · p. 6

4 Task 1: GeoHint

GeoHint asks models to name a photo’s location from an image and a true but coarse hint, varying only the hint while holding the image fixed. Its geocoded metric required substantial corrections because plausible-looking fallback distances had reversed the headline result.

4 Task 1: GeoHint

  • GeoHint holds the image fixed while moving a true but coarse location hint across distance and referenceability conditions.The output schema requires city, area, and building without an “unknown” option; the permissive control schema often receives no building answer.Source: §4 · p. 6, §4.1 · p. 6

4.1 Task definition and prompts

  • The prompt conditions differ by prefix sentence, including baseline, forced-context, warned-context, and structured-checklist variants.The warning tells models to prioritize visible evidence over a possibly inaccurate user location.Source: §4.1 · p. 6

4.2 Metric

  • Haversine distance to the site coordinate scores geocoded answers, with resolution tracked from building to area to city.Ambiguous names use an anchor from the model’s answered city, not the ground truth, to avoid turning a wrong answer into a right one.Source: §4.2 · p. 6

4.3 Four faults that reversed the sign of the result

  • Corrections reduced city-level resolutions from 51.0% to 9.0% of responses, and city-level hit() results are now censored.The faults included alias fallback, linear features that require area context, fabricated global-fallback errors, and a city-centroid distance that inflated one site by 20.7 points.Source: §4.3 · p. 7

5 Task 2: GeoVerify

GeoVerify turns location prompting into a binary check of whether a photo matches a claimed destination, with a mechanically determined distance label and scoring that separates sensitivity from response bias.

5.1 Why a second task

  • GeoVerify targets visual arrival verification, a platform decision about whether a drop-off photo is consistent with its claimed destination.The paper identifies a benchmark gap for visual verification at deployment-relevant tolerances.Source: §5.1 · p. 7

5.2 Task definition

  • The model receives an image and candidate place name, then answers whether the photo was taken within τ of that place.The candidate is given by name alone, without coordinates, city, or description.Source: §5.2 · pp. 7–8
  • Ground truth uses haversine distance with τ = 150 m, avoiding a geocoder, judge model, or tunable output threshold.Image coordinates come from imagery metadata and candidate coordinates from Wikidata.Source: §5.2 · pp. 7–8
  • Decoys begin just beyond τ and are matched to signal candidates on referenceability where possible, limiting rejection based on place type rather than spatial evidence.The five decoy bands extend from 0.15–0.3 km through 3–6 km.Source: §5.2 · p. 8

5.3 Metric: sensitivity separated from bias

  • Signal detection theory separates discrimination from willingness to answer yes, unlike raw accuracy, which can reward always answering no under the sampled trial ratio.The paper uses hit and false-alarm rates to estimate sensitivity and response criterion.Source: §5.3 · p. 8
  • The discrimination radius summarizes the distance at which a model first reaches conventional discriminability, with smaller distances marking where it cannot reliably distinguish true from false drop-offs.The underlying d′ curve is reported as a function of decoy distance.Source: §5.3 · p. 8

5.4 Controls

  • False-alarm confidence is reported because low-confidence errors can be gated, whereas errors made with success-level confidence cannot be distinguished that way.The controls include mismatched-image and blank-image trials, with preregistered thresholds for task validity and informativeness.Source: §5.3 · p. 8, §5.4 · p. 9

6 Results

Across GeoHint results, model performance and hint reliance vary with image legibility, while echo rates change little across referenceability and distance. The analysis also shows that localization error rises as the supplied reference point moves farther away.

6 Results

  • The results distinguish task accuracy from reliance: the former asks whether the predicted place is close to the photograph's location, while the latter examines how answers move with the supplied hint.Source: §6.1 · p. 9, §6.2 · p. 11

6.1.1 Aggregate accuracy

  • Overall hit rates range from 52.6% for claude-opus-5 to 25.0% for claude-haiku-4-5, with the two Qwen models at 32.6% and 30.6%.For every model, hit rate is highest in the nearest distance band; part of this reflects that naming the nearby reference point can fall within the 1 km hit threshold.Source: §6.1.1 · pp. 9–10

6.1.2 Two regimes, separated by whether the image is legible

  • The 50% no-context hit-rate split assigns 65 sites to the illegible regime and 44 to the legible regime, but noisy assignment near the threshold motivates correcting baseline comparisons.Each site's no-context rate is estimated from a median of five responses; the correction is needed because those same baseline responses determine the regime.Source: §6.1.2 · p. 10, §6.1.3 · p. 11

6.1.3 The same split, per model

  • After correcting the regime-assignment bias, only gemini-flash remains less accurate with a nearby hint at legible sites, by −8.4 points; corrected gains remain large at illegible sites.An independent paired estimate across all 109 sites also finds gemini-flash alone worse with the hint, at −3.9; the correction makes surviving gains conservative.Source: §6.1.3 · p. 11

6.2.1 Echo rate

  • At illegible sites, every model echoes the reference point more often than at legible sites, with rates of 29.6% and 18.6% across contexted responses.Echoing is interpreted as a possible fallback when the schema requires a place name, but it was not manipulated and this mechanism is not experimentally established.Source: §6.2.1 · p. 12
  • Echo rates differ by just 1.5 points between low- and high-referenceability places and change by 2.8 points from 0.5 km to 6 km, non-monotonically.Pooled rates fall from 27.4% to 22.1% between the first and second distance bands, then rise to 24.6% in the third.Source: §6.2.1 · p. 12, Figure 3 · p. 12

6.2.2 Error relative to hint distance

  • At illegible sites, median error/reference-point distance is 1.00, versus 0.24–0.69 at legible sites, where the image contributes more to the answer.The ratio is computed only for responses finer than city level, excluding city-centroid fallback answers.Source: §6.2.2 · p. 13, Table 5 · p. 13

6.2.3 Distance slope

  • Across all sites, log10 errkm rises with reference-point distance (β = +0.0820); moving the hint from 0.5 km to 6 km multiplies error by 2.8.The pooled slope is 47% of the way from ignoring the hint (β = 0) to always answering with it (β = 0.176); the slope is steeper at illegible than legible sites for four of five models.Source: §6.2.3 · pp. 13–14

6.3 Interventions

Two paired prompt interventions test whether warning models about unreliable hints or requiring a visual-evidence checklist changes localization. The warning helps when images are legible but not when they are illegible; the checklist does not improve the preregistered outcomes.

6.3.1 Warning that the hint may be wrong

  • +3.94 points (p = 0.0008) is the warning’s hit-rate gain at legible sites, versus +0.46 points (p = 0.49) at illegible sites.The warning lowers echo rates in both regimes, so compliance alone does not explain why accuracy improves only with legible imagery.Source: §6.3.1 · p. 15
  • Every model gains more from the warning at legible sites than at illegible sites; qwen3-vl-235b gains +6.54 points and claude-opus-5 gains +6.17.claude-haiku-4-5 moves by +0.38 points.Source: §6.3.1 · p. 15, Table 9 · p. 15
  • Building-level answers fall from 27.9% to 24.7% with the warning, a paired difference of −3.15 points (p < 10^-4; n = 8215).The warning’s accuracy gain is therefore accompanied by coarser answers; supplying context itself raises building-level resolution above the no-context baseline in all three bands.Source: §6.3.1 · p. 15

6.3.2 Requiring a structured evidence checklist

  • The checklist changes hit rate by 0.00 points at illegible sites (p = 1.00) and +1.04 at legible sites (p = 0.38); the error-to-distance ratio remains 1.00 in the nearest illegible band.The preregistered criteria show no checklist benefit; the warning’s +3.94-point gain at legible sites provides a contrasting intervention result.Source: §6.3.2 · p. 15, Table 10 · p. 15
  • At least 99.7% of each model’s checklist responses contained an evidence object, yet the intervention left echo rates nearly unchanged.The authors interpret the null as consistent with a checklist reorganizing only evidence already extracted, which may be absent when images are illegible.Source: §6.3.2 · p. 16, Table 11 · p. 16

6.4 GeoVerify: discrimination as a function of decoy distance

GeoVerify discrimination improves as decoys move farther away, but no model reaches usable sensitivity just beyond the 150 m tolerance. Accuracy-based rankings can misstate model sensitivity because models differ in their tendency to accept claims, and false acceptances are often reported confidently.

6.4 GeoVerify: discrimination as a function of decoy distance

  • The algorithmically selected batch includes 90 sites in 30 cities, 100 images, and 6,270 scored responses across five models.One API-failed call was excluded; every returned response parsed.Source: §6.4 · p. 16
  • Swapped-continent and gray-field controls reduce false acceptances to 5.0% and 0.0%, but claude-haiku-4-5 accepts 18.1% of swapped-continent claims.That model also has d′ below 1 in every band, indicating it largely does not use the image for this task.Source: §6.4 · p. 16
  • Pooled d′ rises from 0.33 in the nearest band to 1.20 in the farthest, yet no model reaches d′ = 1 near the 150 m tolerance; gemini-flash is best there at 0.68.Figure 4 shows sensitivity increasing with decoy distance; the nearest band is the deployment-relevant comparison.Source: §6.4 · p. 16, Figure 4 · p. 17, Table 12 · p. 17
  • claude-opus-5 has a 47.2% hit rate on true claims but second-best far-band sensitivity (d′ = 1.95), so accuracy alone can invert model rankings.Its criterion is stricter than gemini-flash’s, which is biased toward accepting claims.Source: §6.4 · p. 16
  • 83.8% of false acceptances carry confidence ≥0.8, with reported rates ranging from 9.8% for claude-opus-5 to 100.0% for qwen3-vl-8b.The pooled median confidence is 0.85 for false acceptances versus 0.95 for correct acceptances.Source: §6.4 · p. 16, Table 12 · p. 17
  • By the preregistered criteria, the pooled result is inconclusive, while model-level results vary substantially: claude-opus-5 reaches d′ = 0.32 nearest and 1.95 farthest.gemini-flash reaches 2.36 in the farthest band but misses the nearest-band criterion at 0.68.Source: §6.4 · p. 18

7 Limitations

The benchmark’s coverage and measurements impose boundaries on how broadly its results can be generalized. Its imagery, reference-point sources, language, and site-level estimates each have documented constraints.

7 Limitations

  • Mapillary coverage is uneven, with mainland China the largest stated gap, and the tourist-destination candidate list does not establish broad global coverage.The authors also note that one city outside the candidate list yielded only unusable equirectangular panoramas.Source: §7 · p. 18
  • English prompts and English-language Wikidata titles leave the results for other user languages and car-oriented cities untested.Places without English-language coverage are less likely to be returned as reference points.Source: §7 · p. 18
  • Site-level no-context hit rates use a median of five responses, and sparse ladder cells require weighting; Ldist measures neighborhood granularity rather than perceived distance.Linear features can also reduce precision: Meiji-dōri + Shibuya scores 0.53 km against a true 0.05 km.Source: §7 · p. 18

8 Conclusion

GeoContext finds that models repeat location hints at similar rates across distance and referenceability, while the localization error from repeating them grows with distance. On the same images, GeoVerify remains unreliable near the deployment tolerance, and the authors release the benchmark and its supporting materials.

8 Conclusion

  • Hint repetition does not scale with distance or referenceability, while the localization error from repeating a hint grows steadily with its distance.When imagery is unreadable, the median error equals the hint distance; when it is readable, the image does more work, and one model is less accurate after receiving true information even after correcting for selection bias.Source: §8 · p. 18
  • On the same images, no model separates true claims from decoys one street away at deployment-needed tolerance; accuracy-only rankings misorder models, and most false acceptances are confident.Source: §8 · p. 18
  • The release includes the benchmark, auto-generated context ladders, auditor decisions, and scoring code.Source: §8 · p. 18

Reproducibility

The release provides benchmark data and pipeline artifacts, with image identifiers and coordinates rather than Mapillary pixels. Raw GeoHint response text is available unevenly across schemas.

Reproducibility

  • The release provides image IDs and coordinates rather than Mapillary pixels.Source: Reproducibility · p. 18
  • Released artifacts include auditor decision files, 109 ladder CSVs, district-scale and city-screening tables, parsed and geocoded GeoHint responses, and raw GeoVerify responses.The raw forced_chain responses used to compute Table 11 are also included.Source: Reproducibility · p. 18
  • Raw response text for the remaining GeoHint schemas is available from the authors on request.Source: Reproducibility · p. 19

A Prompts

The appendix reproduces the prompts used for GeoHint, GeoVerify, and the reference-point and image audits. The prompts specify what context and visual evidence models or auditors receive, and how they must respond.

A Prompts

  • The released code’s prompts are reproduced verbatim because the prompt itself is the method.The appendix ships the prompt rather than paraphrasing it.Source: A Prompts · p. 19

A.1 GeoHint

  • GeoHint prepends “I took this photo near <reference point>” to the schema; reported results use the forced schema, which disallows “unknown.”The abstention-permitting v1 schema serves as a control.Source: §A.1 · p. 20
  • GeoHint asks models to record observed infrastructure, vegetation, architecture, text script, and terrain or related cues before producing image-grounded answers.The prompt says to report “not visible” rather than guess and to base the answer on those observations.Source: §A.1 · p. 20

A.2 GeoVerify

  • GeoVerify uses one verdict-neutral prompt per trial, presenting the candidate name alone without coordinates, city, or description.The wording leaves criterion c to be measured rather than set by the prompt; responses must choose yes or no and include confidence and visible evidence.Source: §A.2 · p. 21

A.3 Reference-point audit (deepseek-v4-pro, text only)

  • The text-only reference-point audit rates whether candidate places are usable and their referenceability, meaning how often locals or tourists use them to describe where they are.The audit prompt also requests English and Chinese names where available; the Chinese field is retained in released ladder files but is not used in this paper’s English-only conditions.Source: §A.3 · p. 21

A.4 Image audit (deepseek-vision)

  • The image audit combines model review with a separate human image-only keep/exclude decision and records overlays, scene type, visual quality, and other image properties.A photo is marked unusable if it has an answer overlay, is not a street scene, or has quality at most 1; tourist infrastructure does not affect usability.Source: §A.4 · p. 22

B Why not a commercial geocoder

Wikidata misses some labels, and Nominatim resolves only a portion of those; the remaining failures include descriptive phrases, generic categories, and non-unique names. Google Places performs better, but its terms conflict with releasing resolved coordinates for reproducibility.

B Why not a commercial geocoder

  • 15 of the 100 labels Wikidata fails are resolved by Nominatim, with 10 net new labels.The remaining 85 labels fail with both geocoders.Source: B Why not a commercial geocoder · p. 22
  • Among the 85 labels both geocoders fail to resolve, 36 are descriptive phrases, 21 generic categories, and 28 plausible names, including non-unique bank chains.Eight of the plausible names are bank chains with thousands of branches.Source: B Why not a commercial geocoder · p. 22
  • Google Places performs better, but its terms restrict caching and dataset creation, conflicting with releasing resolved coordinates for reproducibility.The authors estimate the remaining ceiling at approximately 3% of records and weigh it against the cost of new cross-city fuzzy-match errors.Source: B Why not a commercial geocoder · p. 22
Loading 2609.05761v1…