PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does CLIP Learn for Regional Geolocalization? Probing Visual Cues and Scene Configuration After Adaptation

Changyu Lee, Yeonsoo Park, Abdullah Alfarrarjeh, Seon Ho Kim

arXiv:2608.21761v1cs.AIcs.CEcs.CV

TL;DR

Fine-grained regional geolocalization is difficult because nearby areas share broad visual cues, raising questions about whether pretrained CLIP features suffice and what adaptation changes. The study evaluates multiple CLIP adaptation regimes and visual interventions on eight Greater Los Angeles regions, finding substantially better nearby-region discrimination after encoder adaptation alongside greater sensitivity to scene configuration, but not structural sufficiency alone. The findings are limited to viewpoint variation near known locations.

  • Problem

    Nearby metropolitan regions share coarse geographic characteristics, so the study asks whether pretrained CLIP representations contain enough information for fine-grained regional discrimination and what adaptation adds.

  • Method

    The study compares zero-shot CLIP, frozen and adapting encoder strategies on 9,085 images from eight Greater Los Angeles regions, using cue removal, edge and blur transformations, and patch scrambling.

  • Results

    Encoder adaptation reaches 75.94–82.10% accuracy versus a 39.03% zero-shot baseline, while adapted models show higher transformed accuracy and scrambling sensitivity without improved retention after appearance reduction.

  • Takeaways & Limitations

    Adaptation is associated with greater dependence on intact scene configuration, while regional predictions remain supported by a combination of configuration and appearance cues.

  • Takeaways & Limitations

    The study covers one metropolitan area with overlapping splits, measuring viewpoint variation near known locations rather than geographically disjoint generalization.

Abstract

from arXiv · show

Large collections of street-view imagery provide rich visual information about urban environments, but extracting fine-grained geographic information from such data remains challenging. In particular, fine-grained regional geolocalization is challenging because nearby areas often share coarse geographic cues. We study regional geolocalization within a metropolitan area and ask whether pretrained CLIP features are sufficient for regional discrimination, and what visual information supports performance after adaptation. Using 9,085 street-view images from eight Greater Los Angeles regions, we compare zero-shot CLIP, frozen-encoder readouts, partial encoder updating, Low-Rank Adaptation (LoRA), and full fine-tuning. Frozen readouts remain near the 39.03% zero-shot accuracy, whereas encoder adaptation achieves 75.94-82.10%. Full fine-tuning also reduces the mean distance to the predicted region center from 12.30 km to 3.86 km. We probe these gains through semantic cue removal, appearance reduction using edge maps and blur, and scene-configuration disruption using patch scrambling. Adapted models achieve higher edge and blur accuracy and switch 42.92-45.56% of predictions after scrambling, compared with 10.79-14.60% for frozen methods. However, adaptation does not improve the fraction of performance retained after appearance reduction, while vegetation and sky remain influential. A Caltech101 control further shows that scrambling sensitivity is not unique to geolocalization. Overall, encoder adaptation substantially improves nearby-region discrimination and is associated with greater sensitivity to intact scene configuration, without evidence that coarse structure alone becomes sufficient for prediction. These conclusions concern viewpoint variation near known locations rather than geographically disjoint generalization.

I. INTRODUCTION

The study examines whether pretrained CLIP representations support fine-grained regional discrimination and what visual information accompanies gains from encoder adaptation. In eight Greater Los Angeles regions, adaptation improves performance and is associated with greater sensitivity to intact scene configuration, while appearance cues remain important.

  • Nearby metropolitan regions share coarse geographic cues, making fine-grained regional geolocalization a test of whether pretrained representations contain sufficiently discriminative information.
  • The benchmark defines regional geolocalization as classification among selected areas within one metropolitan region, using eight Greater Los Angeles classes with variations in streetscape, vegetation, and urban form.
  • 39.03% zero-shot accuracy rises to as high as 82.10% with encoder-level adaptation, while frozen readouts remain close to the zero-shot baseline.The comparison spans zero-shot CLIP, frozen readouts, partial updating, LoRA, and full fine-tuning.
  • The intervention analysis combines semantic cue removal, edge and blur transformations, and patch scrambling to distinguish cue dependence, retained appearance information, and layout sensitivity.Retention is transformed accuracy divided by original accuracy, while prediction switch rate measures how often scrambling changes predictions.
  • Adaptation increases absolute accuracy on appearance-reduced inputs and prediction changes under scrambling, but does not make structural information sufficient without appearance cues.Vegetation and sky remain influential, and a Caltech101 control indicates scrambling sensitivity is not unique to geolocalization.
  • The conclusions apply to viewpoint variation near known locations rather than geographically disjoint generalization.

II. RELATED WORK

Prior work frames visual geolocalization through classification, retrieval, coordinate prediction, and combinations of these approaches. CLIP-based systems establish geographic prediction capabilities, while this study focuses on the visual information associated with adaptation gains.

  • Visual geolocalization includes classification, retrieval, coordinate prediction, and hybrid formulations such as geographic-cell classification with retrieval or density estimation.
  • CLIP-based systems apply image–text representations to open-domain geolocalization, continuous GPS alignment, semantic geocells with retrieval refinement, and city-wide address prediction.
  • This study differs by examining the visual information associated with adaptation gains rather than introducing another city-scale application.

C. Probing Object Relations and Spatial Arrangement

The study frames regional geolocalization as sub-regional classification and probes whether scene configuration relates to broader spatial reasoning, using defined accuracy and centroid-error measures.

  • Spatial interpretation: The intervention analysis tests coarse scene-configuration sensitivity rather than relational understanding or general spatial reasoning.The probes examine selected appearance cues and scene arrangement, not individual evidence of 3-D or symbolic spatial reasoning.
  • Dataset and task: The dataset contains 9,085 images labeled across eight Greater Los Angeles regions for sub-regional classification.The regions include municipalities, neighborhoods, and districts.
  • Dataset visualization: Figure 2 presents regional sampling and representative scenes selected to include both distinctive and geographically ambiguous views.This visualization connects dataset geography with the visual ambiguity of the classification task.
  • Evaluation measures: Top-1 accuracy measures the fraction of evaluation images whose predicted region matches the ground-truth region.The evaluation set is denoted E, with predictions indexed by model and image.
  • Evaluation measures: Centroid error measures the mean distance between each image coordinate and the centroid of its predicted region.The distance is expressed in kilometers.

B. Dataset Reconstruction and Density-Based Sampling

The dataset is reconstructed from eligible street-view imagery using area-normalized sampling, while the split preserves substantial geographic overlap between training and evaluation views.

  • Density-based sampling: 9,085 images are sampled at the largest common density, ρ⋆ = 194.49 images/km2, determined by Koreatown.Sampling approximately ρ⋆A_k images per region equalizes spatial sampling intensity rather than class frequency.
  • Dataset split: The region-stratified split contains 6,359 training, 1,363 validation, and 1,363 test images.The split uses seed 42, and raw image and split metadata are retained for reproducibility.
  • Geographic overlap: 55.39% of test records share a requested coordinate with training, while 79.68% lie within 50 m of a training coordinate.Only 20.32% of test records are separated by more than 50 m.
  • Geographic overlap: The protocol measures viewpoint variation near known locations rather than geographically disjoint generalization.Different headings can reduce direct pixel correspondence while retaining local buildings, geometry, vegetation, and capture conditions.

IV. ADAPTATION METHODS

The adaptation study compares progressively more invasive changes to a pretrained image encoder, separating frozen readout learning from encoder modification.

  • Common setup: All methods use OpenCLIP’s pretrained ViT-L-14 image encoder with a default 224 × 224 input pipeline.Supervised methods train for 10 epochs with fixed optimization settings and retain the highest-validation-accuracy checkpoint.
  • Adaptation strategies: The strategy spectrum ranges from zero-shot CLIP and frozen readouts to Partial Update, LoRA, and full fine-tuning.The comparison progressively increases the extent of pretrained-model modification.
  • Adaptation strategies: Frozen readout methods learn output mappings over unchanged image features, whereas Partial Update and Full-FT modify encoder parameters.LP-T adjusts text-based class-score scale and bias; LP-C trains a linear classifier over frozen features.
  • Study question: The comparison tests whether regional discrimination is accessible through evaluated frozen readouts or benefits from encoder modification.Lower probe performance alone does not establish that frozen CLIP lacks regional information.
  • Study question: Accuracy comparisons do not identify which visual information supports adaptation gains, motivating the intervention protocol.The subsequent analysis examines visual evidence rather than treating performance alone as explanatory.

V. INTERVENTION-BASED PROBING PROTOCOL

The protocol combines semantic cue removal, appearance reduction, and patch scrambling to distinguish cue dependence, retained information, and sensitivity to intact scene arrangement.

  • Intervention families: The three intervention families test selected semantic cues, appearance-reduced information, and intact spatial arrangement.The protocol reports transformed accuracy and change relative to original-image performance.
  • Semantic cue removal: Text, vehicles, vegetation, and sky are removed because they are interpretable cues whose masking largely preserves surrounding scene context.Roads and buildings are not individually removed because inpainting could alter scene geometry.
  • Appearance and layout: Edge and blur transformations probe predictive information remaining after detailed appearance is reduced, while patch scrambling disrupts scene arrangement.These transformations are intended as complementary probes rather than perfectly selective manipulations.
  • Semantic cue removal: Vegetation and sky show the largest regional differences in mask coverage, with F = 35.04 and F = 10.93 respectively, both p < 0.001.Vehicles and text also differ significantly, with F = 5.65, p < 0.001 and F = 2.68, p = 0.009.
  • Behavioral measures: Prediction switch rate measures the fraction of changed labels and therefore captures instability rather than correctness.Cue-removal results characterize sensitivity to implemented removals rather than causal cue importance.

B. Appearance-Reduced Structure Proxies

Edge and blur transformations reduce fine appearance while preserving some coarse organization, but they do not isolate structure. Adapted models achieve higher transformed accuracy without greater retention of original performance.

  • Appearance reduction: Edge images retain text outlines, vehicle shapes, windows, and vegetation boundaries, while blur retains color and coarse texture.These transformations therefore preserve information beyond spatial structure.
  • Interpretation: Retention does not improve after adaptation, so higher transformed accuracy does not establish structural sufficiency.Retention normalizes transformed accuracy by each model’s original accuracy.

B. Variation Across Regions

Performance and intervention effects vary across regions and adaptation regimes. Environmental cues remain influential, while appearance-reduced accuracy increases without improved retention.

  • Variation Across Regions: Full-FT gains range from 6.67 points in Venice Beach to 60.56 points in Hollywood.The aggregate improvement is therefore not uniform across regions.
  • Cue-removal sensitivity: Vegetation and sky removal change more predictions than text or vehicle removal, with this ordering preserved after adaptation.
  • Cue-removal sensitivity: Removed area and detector or segmentation quality prevent causal ranking across cues, particularly because vegetation and sky masks may cover larger image fractions.
  • Appearance-reduced structure proxies: LoRA and Full-FT raise edge accuracy from 21.5% to 32.1% and 38.3%, respectively, and macro-blur accuracy from 22.1% to 42.0% and 45.2%, but retention does not improve.

E. Sensitivity to Patch Scrambling

Patch scrambling tests sensitivity to intact scene arrangement, but its effects are not uniquely geographic. Adapted geolocalization models switch predictions more often than frozen models under the same disruption.

  • Geolocalization: Under Patch-16, frozen models switch 10.79–14.60% of predictions, compared with 42.92–45.56% for adapted models.The gap persists at larger patches, while sensitivity decreases as more coherent local content is preserved.
  • Caltech101 control: Caltech101 accuracy drops by 62.19%, 31.51%, and 12.33% under Patch-16, Patch-32, and Patch-64, respectively.Scrambling therefore disrupts generic recognition as well as geolocalization.
  • Interpretation: The Caltech101 accuracy drops are not directly comparable to geolocalization switch rate, which measures any label change.

VII. DISCUSSION

Encoder adaptation improves nearby-region discrimination and is associated with greater layout sensitivity, but the evidence does not show that structural information alone becomes sufficient.

  • Effects of Encoder Adaptation: Frozen readouts remain near zero-shot accuracy, whereas PU, LoRA, and Full-FT reach 75.94–82.10%.This gap indicates that stronger discrimination is more accessible when the visual representation adapts.
  • Appearance-reduced structure proxies: Full-FT raises edge accuracy from 21.5% to 38.3% and blur accuracy from 22.1% to 45.2%, yet its retention is lower than zero-shot CLIP’s.Adapted models use more residual information without retaining a larger share of original accuracy.
  • Layout sensitivity: Patch-16 changes 42.92–45.56% of adapted predictions versus 10.79–14.60% for frozen methods.This within-task contrast supports increased layout dependence after encoder adaptation.
  • Control and scope: Caltech101 accuracy falls from 89.65% to 33.90% under Patch-16, so scrambling is not geolocalization-specific.The relevant evidence is the higher adapted-model switch rate under the same geolocalization intervention.

D. Reliability of Geographic Evidence

The evidence indicates that adapted models use both intact scene configuration and environmental appearance, while the study’s conclusions remain bounded by its intervention design and evaluation scope.

  • Semantic cues: Vegetation and sky change more predictions than text or vehicles, and this ordering persists after adaptation.The interventions must be interpreted jointly because mask-area and segmentation differences preclude causal ranking of image cues.
  • Scope and limitations: The study covers one metropolitan area and eight mixed region types, with overlapping splits testing viewpoint variation near known locations rather than geographic generalization.Edge and blur are imperfect structure proxies, scrambling introduces artifacts, and comparisons use a single seed without significance tests or confidence intervals.
  • Adaptation: 75.94–82.10% adapted accuracy versus 39.03% zero-shot accuracy shows that nearby-region discrimination emerges primarily when the visual representation adapts.Full fine-tuning also reduces centroid error from 12.30 km to 3.86 km.
  • Appearance reduction: Adaptation does not improve the fraction of original accuracy retained after edge and blur transformations.Adapted models nevertheless achieve higher edge and blur accuracy, so stronger transformed accuracy does not imply greater retention.
Loading 2608.21761v1…