PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoLEON-CLIP: Graph-Anchored Multimodal Contrastive Learning for Geographic Region Representation

Radoslaw Malek, Szymon Soltysiak, Aleksandra Kawala-Sterniuk, Piotr Szymanski

arXiv:2610.32311v1cs.CVcs.LG

TL;DR

Geographic prediction remains constrained by scarce, spatially uneven, task-specific labels. GeoLEON-CLIP aligns graph, satellite-image, and text views of H3 regions; at R@1@1, graph routing beats direct image–text retrieval in both directions.

  • Problem

    Geographic prediction still depends on scarce, spatially uneven, task-specific labels, limiting robust generalization across cities, administrative definitions, and scales.

    Source: §1 · p. 1

  • Method

    GeoLEON-CLIP aligns OSM-derived graph embeddings, satellite images, and textual region descriptions in a shared H3-indexed space using frozen encoders and trainable residual projection heads.

    Source: §1 · p. 2

  • Results

    Graph aligns strongly with image and text, and at R@1@1 a single-candidate graph detour beats direct image–text retrieval in both directions; downstream gains vary by task.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The results support graph-anchored multimodal learning as a direction for organizing satellite imagery and place descriptions in a shared H3-indexed space.

    Source: §7 · p. 10

    The random anchor-level split uses no spatial blocking and includes cities and resolutions in both partitions, so it does not measure transfer to an unseen city.

    Source: §7.1 · p. 10

Abstract

from arXiv · show

Geographic regions are multimodal: satellite imagery captures visual morphology, text conveys human-interpretable semantics, and map structures describe urban context. Existing geospatial contrastive methods typically align image-text or image-location pairs, underusing structured map data. We introduce GeoLEON-CLIP, a graph-anchored contrastive framework that aligns three views of each H3-indexed region: an OpenStreetMap-derived graph embedding, a satellite image crop, and a textual description. Frozen pretrained LEON graph embeddings provide a semantic anchor between vision and language. Frozen graph, image, and text encoders are connected by trainable residual projection heads and trained with a symmetric tri-modal contrastive objective. Experiments show strong graph-image and graph-text alignment; graph-image retrieval achieves the best cross-modal performance in a mixed-resolution pool. To evaluate the anchoring hypothesis, we propose chained cross-modal recall $\mathrm{R}@A@B$, which routes retrieval through the graph modality and compares it with direct retrieval under a matched candidate budget. At $\mathrm{R}@1@1$, image$\rightarrow$graph$\rightarrow$text and text$\rightarrow$graph$\rightarrow$image outperform direct image-text retrieval by 4.8 and 4.4 points, respectively, despite using a single intermediate cell. Downstream probes show task-dependent gains, with the largest improvements on Airbnb and Beijing housing. These results indicate that OpenStreetMap graph structure can anchor multimodal representations combining maps, imagery, and language.

1 Introduction

Geographic prediction depends on scarce, uneven, task-specific labels, while structured map data is often not treated as a central modality. GeoLEON-CLIP aligns graph, image, and text views of H3 regions and tests whether graph representations can bridge visual and textual semantics.

1 Introduction

  • Scarce, spatially uneven, task-specific labels remain a bottleneck for robust geographic prediction across cities, administrative definitions, and scales.Self-supervised learning can exploit naturally co-located observations, while map-derived information is often treated as auxiliary.Source: §1 · p. 1
  • GeoLEON-CLIP aligns OSM-derived graph embeddings, satellite imagery, and textual place descriptions in a shared H3-indexed space using frozen encoders and trainable residual projection heads.The graph embedding is pretrained LEON; the alignment uses symmetric tri-modal contrastive learning.Source: §1 · p. 2
  • The paper tests whether LEON graph embeddings can serve as a semantic anchor between visual and textual modalities, extending graph-only representation learning to multimodal alignment.The motivation is to connect graph-based urban representation learning with CLIP-style geospatial pretraining.Source: §1 · p. 2
  • The paper introduces budget-matched chained cross-modal recall to test whether routing image–text retrieval through a graph modality can outperform direct retrieval.The metric also includes alternative bridge placements as a control.Source: §1 · p. 2
  • The evaluation combines pairwise and chained retrieval, geographic-error and shared-space diagnostics, ablations, and downstream Ridge probes across five geographic prediction tasks.The dataset pairs graph, image, and text views by common H3 identifiers across district, neighborhood, and block scales.Source: §1 · p. 2

2 Related Work

Prior spatial representation work learns from maps and other geographic data, while geospatial contrastive work commonly aligns imagery with text or location. GeoLEON-CLIP positions an OSM-derived graph as an explicit modality aligned with imagery and text at the region level.

Spatial and urban representation learning

  • Spatial representation methods range from hand-crafted GIS indicators to learned representations, but map-derived methods such as Hex2Vec and S2Vec remain largely unimodal and do not explicitly align map structure with visual or textual views.H3 provides a hierarchical multi-resolution cell index.Source: §2 · p. 2

Masked autoencoding for graphs and spatial structure

  • LEON adapts graph masked autoencoding to H3-indexed spatial graphs, reconstructing masked OSM-derived node attributes from geographic adjacency context.This builds on masked prediction approaches in language, vision, and graph learning.Source: §2 · pp. 2–3

Vision-language and geospatial contrastive learning

  • Geospatial contrastive models align imagery with coordinates or text, while structured OSM-derived graph representations are usually absent or indirect.Examples include image–location models and remote-sensing vision-language models.Source: §2 · pp. 2–3

Positioning of GeoLEON-CLIP

  • GeoLEON-CLIP differs from image–text, image–location, and spatial embedding methods by aligning region-level graph, image, and text views through shared H3 identifiers.Its graph branch is an explicit OSM-derived modality rather than an indirect source of textual semantics.Source: §2 · p. 3

3 Tri-Modal H3 Dataset

The Tri-Modal H3 Dataset pairs graph, satellite-image, and text views at shared H3 anchors across metropolitan areas and three spatial scales. Its modality-specific construction and temporal choices aim to keep each view tied to the same geographic region while acknowledging that the sources are not perfectly simultaneous.

3 Tri-Modal H3 Dataset

  • The dataset fully pairs OpenStreetMap graph features, Sentinel-2 imagery, and textual descriptions at H3 anchors across 35 metropolitan areas on six continents and three scales.The scales correspond approximately to district, neighborhood, and block levels.Source: §3.1 · p. 4, §3 · p. 4

3.2 Graph Modality

  • OSM objects are aggregated into 31 functional categories per H3 cell, with ways and relations assigned by centroid and neighboring cells connected in a spatial adjacency graph.Log-transformed category counts and an empty-cell indicator form node features; centroid assignment limits each object to one cell and prevents large polygons from dominating neighboring cells.Source: §3.2 · p. 4

3.3 Image Modality

  • Sentinel-2 RGB crops cover 512×512 pixels at 10 m resolution and are centered on each H3 cell, with cloud-free observations preferred near December 2025.About 5% of cells use imagery outside the primary March 2025–February 2026 window because no qualifying scene was available within it.Source: §3.3 · p. 4

3.3.1 Hexagon-Aware Patch-Token Pooling.

  • Resolution-dependent central patch-token windows match the pooled image receptive field more closely to H3 cell area than a global crop descriptor.The method pools 14×14, 5×5, and 2×2 token windows for resolutions 7, 8, and 9, respectively.Source: §3.3.1 · p. 5
  • At resolution 9, one parent-cell image pass yields seven spatially registered child embeddings, reducing finest-resolution encoder cost by roughly a factor of seven.The seven disjoint 2×2 windows are assigned by matching child and window azimuths.Source: §3.3.1 · p. 5

3.4 Text Modality

  • Cell-level text combines OSM-associated descriptions with Wikipedia, Wikivoyage, and Reddit content, which contributes encyclopedic, travel-oriented, and informal local perspectives.External records are mapped to H3 by coordinates; finer cells without direct text inherit their parent document.Source: §3.4 · p. 5

3.1 Spatial Anchors and Study Areas

  • H3 indices join the three views to the same region, avoiding approximate joins between incompatible spatial units and providing positive views for contrastive learning.The dataset uses a hierarchical hexagonal grid, and complete graph–image–text triplets are available at every spatial resolution used.Source: §3.1 · p. 4, §3.5 · p. 6

3.5 Pairing and Temporal Consistency

  • The modalities are not perfectly simultaneous: OSM uses the December 2025 snapshot, imagery spans March 2025–February 2026, and external web text reflects sources accessed in March 2026.Imagery is selected preferentially near December 2025, while the text’s OSM-associated component follows the December 2025 object set.Source: §3.3 · p. 4, §3.4 · p. 5, §3.5 · p. 6

4 GeoLEON-CLIP

GeoLEON-CLIP maps frozen graph, image, and text encoders into a shared space using trainable projection heads. It optimizes symmetric pairwise contrastive losses for all three modality pairs, with equal weights, and evaluates using a pooled validation split.

4.1 Architecture

  • Frozen graph, image, and text encoders connect to trainable modality-specific residual projection heads; only the heads and logit scale are optimized.The best-performing model uses 4096-dimensional hidden layers, three residual layers, and a shared projection dimension of 256.Source: §4.1 · p. 6

4.2 Tri-Modal Contrastive Objective

  • The objective sums symmetric contrastive losses for graph–text, graph–image, and text–image pairs, giving each pair equal weight rather than privileging graph-involved pairs.The graph-anchored label refers to pretrained LEON embeddings and observed bridge behavior; graph and text use parent/center/child context, while images use only the center cell.Source: §4.2 · p. 6

4.3 Training Setup

  • The fixed-seed split assigns 877,889 of 975,432 anchors to training and 97,543 to validation, pooling cities and resolutions without spatial blocking.Retrieval uses a fixed 1024-anchor validation query/gallery pool dominated by resolution 9 at about 85%; the LEON encoder was pretrained on graphs from all 35 cities.Source: §4.3 · p. 6

5 Evaluation Protocol

The evaluation tests pairwise and graph-bridged retrieval, geographic error, embedding geometry, and downstream prediction. Chained recall compares two-hop retrieval with direct retrieval at a matched candidate budget.

Cross-modal retrieval

  • Pairwise retrieval evaluates all six ordered modality pairs using Recall@1, Recall@5, Recall@10, median rank, and mean reciprocal rank.Table 3 reports cross-modal retrieval for the base residual-MLP model using 1024 validation queries.Source: §5 · pp. 6–7, Table 3 · p. 7

Geographic error

  • MGE@k measures the mean distance in kilometers to the closest target centroid among the top-k retrieved cells, with lower values better.It complements rank-based metrics by distinguishing geographically nearby errors from distant ones.Source: §5 · p. 7, Table 4 · p. 7

Chained cross-modal recall

  • Chained recall retrieves top-A bridge cells and then top-B targets, counting queries whose target appears in the chained set; it is compared with direct R@(A×B).At A=B=1, both methods have an exact one-candidate budget; otherwise overlapping second-hop results can leave the chain with fewer distinct candidates.Source: §5 · p. 7

Embedding geometry

  • Embedding geometry is assessed through paired normalized-embedding cosine similarity and effective dimension from the projected covariance spectrum.Effective dimension is reported for each modality and the pooled joint space.Source: §5 · p. 7

Downstream probes

  • Ridge-regression probes test whether pre- and post-projection embeddings improve five task-specific prediction benchmarks over baseline features, comparing graph, text, image, and combined embeddings.The tasks are Airbnb prices, Beijing housing, Chicago crime, King County house sales, and Philadelphia crime; configurations are evaluated by R2.Source: §5 · p. 7

6 Results

GeoLEON-CLIP aligns graph, image, and text views, with graph–image retrieval strongest in the mixed-resolution pool. Routing image–text retrieval through the graph improves strict top-1 retrieval, while ablations, downstream probes, and qualitative arithmetic reveal additional task-dependent patterns.

6.1 Cross-Modal Retrieval

  • Graph–image is the strongest mixed-resolution pair, reaching Recall@1 of 0.800 and 0.813; graph–text follows at 0.742 and 0.741.Direct text–image retrieval reaches Recall@1 of 0.616–0.617 and Recall@10 above 0.84 in both directions.Source: §6.1 · p. 7, §6 · p. 7

6.2 Graph as a Retrieval Bridge

  • At R@1@1, I→G→T beats direct retrieval by 4.8 points (66.5% versus 61.7%), while T→G→I gains 4.4 points (66.0% versus 61.6%).Paired-bootstrap 95% intervals are [2.3, 7.3] and [1.8, 7.1] points, respectively.Source: Table 5 · p. 8, §6.2 · p. 8
  • The graph-route advantage largely disappears at wider budgets, with only I→G→T retaining a +0.8 point margin at 1@5.At resolution 9, the T→G→I interval includes zero; differing gallery difficulty prevents interpreting resolution results as a scale ranking.Source: §6.2 · p. 8

6.3 Projection Head and Training Ablations

  • The residual head is strongest in all six directions, while linear projection degrades every pair and removing modality dropout is costly for image-involved retrieval.Without modality dropout, G→I Recall@1 is 0.585 vs. 0.800, I→T is 0.366 vs. 0.617, and T→I is 0.350 vs. 0.616.Source: §6.3 · p. 8, Table 6 · p. 9

6.4 Embedding Geometry

  • Post-projection effective dimensions are 28.04 for graph, 22.93 for text, 24.05 for image, and 29.62 for all modalities pooled.Average cross-modal cosine alignment is 0.571 for text–graph, 0.442 for image–graph, and 0.302 for text–image.Source: §6.4 · p. 8

6.5 Downstream Prediction

  • Airbnb improves from R2 = 0.216 to 0.287 using post-projection all-modality features, while Beijing housing improves from 0.669 to 0.690 using pre-projection all-modality features.Smaller positive gains appear for King County and Philadelphia crime; Chicago crime has no measurable gain from pre-projection graph features.Source: §6.5 · p. 8, Table 7 · p. 9

6.6 Exploratory Cross-Modal Embedding Arithmetic

  • Qualitative arithmetic retrievals often reflect properties of the whole tri-modal object, although each expression uses one modality per input region.The observations are preliminary and qualitative, with counterexamples; a complete evaluation would require controlled benchmarks and statistical tests across city and modality combinations.Source: §6.6 · pp. 9–10
  • In the arithmetic examples, one operand tends to contribute functional or semantic character while the other tends to influence geographic placement.Text alone sometimes biases retrieval toward the corresponding city, despite being least directly tied to spatial geometry.Source: §6.6 · p. 10

7 Discussion and Conclusion

Graph aligns strongly with imagery and text, while direct image–text retrieval is harder; graph routing beats direct matching at the single-candidate setting but not generally at larger budgets. Downstream gains vary by task, and spatial generalization and dataset alignment remain bounded by stated limitations.

7 Discussion and Conclusion

  • Graph aligns strongly with both image and text, while direct image–text retrieval is consistently harder; graph–image leads in the mixed-resolution pool but can trail graph–text in per-resolution checks.The relative ordering of the two graph pairings can reverse across resolutions.Source: §7 · p. 10
  • At R@1@1, graph routing is more accurate than direct image–text matching in both directions, but direct retrieval generally regains the advantage as candidate sets grow.The graph is therefore a high-precision bridge, not a general replacement for direct retrieval.Source: §7 · p. 10
  • Adding embeddings improves several downstream tasks, especially Airbnb and Beijing housing, but no modality configuration dominates across datasets.Strong baseline features and noise or redundant signal from concatenating all modalities qualify these gains.Source: §7 · p. 10

7.1 Study Limitations

  • The dataset is limited by uneven OSM coverage, city- and language-varying text, approximate temporal alignment, and inherited parent-cell text shared by some sibling cells.Precomputed image embeddings also leave acquisition dates, cloud coverage, and mosaicking provenance outside the model and alignment analysis.Source: §7.1 · p. 10
  • The random anchor-level split includes all cities and resolutions in both partitions and does not measure transfer to an unseen city.LEON pretraining on all 35 cities and spatial autocorrelation may benefit this evaluation; per-city, spatially blocked, and leave-city-out tests remain future work.Source: §7.1 · p. 10

A Positioning Against Related Work

GeoLEON-CLIP combines map-derived graph representations with image and language alignment. Its explicit OSM graph modality distinguishes it from approaches that align imagery with location or language.

A Positioning Against Related Work

  • GeoLEON-CLIP retains OSM as an explicit graph modality to organize visual and textual region semantics in a shared H3-indexed space.This distinguishes it from the compared approaches that align imagery with location or language.Source: A Positioning Against Related Work · p. 12
  • Hex2Vec and S2Vec learn map-derived region embeddings but remain unimodal.The passage contrasts them with GeoLEON-CLIP’s graph, image, and language combination.Source: A Positioning Against Related Work · p. 12
  • SatCLIP and GeoCLIP align imagery with location, while RemoteCLIP, GeoRSCLIP, SkyScript, and UrbanCLIP connect imagery with language.GeoLEON-CLIP’s stated distinction is keeping OSM as an explicit graph modality.Source: A Positioning Against Related Work · p. 12

B OpenStreetMap Categories

The appendix identifies the OpenStreetMap categories used as graph node features and describes how map elements are assigned to locations.

B OpenStreetMap Categories

  • Points are assigned by location, whereas ways and relations are assigned using their centroids.This describes the location-assignment rule for the listed map elements.Source: B OpenStreetMap Categories · p. 12
  • Table 9 lists 31 OpenStreetMap tags aggregated into graph node features.The passage names the table’s scope but does not enumerate the individual tags.Source: Table 9 · p. 12, B OpenStreetMap Categories · p. 12
  • The listed categories are used as graph node features.The passage does not specify individual category names.Source: Table 9 · p. 12

C Chained Recall Budget Grids

The chained-recall budget grid shows that hop allocation matters in addition to the total candidate budget. Its clearest matched-budget advantage over direct image–text retrieval occurs at 1@1.

C Chained Recall Budget Grids

  • For Image–Graph–Text, every above-diagonal grid entry exceeds its transpose, so a wider second hop is consistently more productive than retrieving more graph cells first.In the reverse chain, wider second hops are preferable at small budgets, while broader first hops often help more at moderate budgets.Source: C Chained Recall Budget Grids · p. 12, Table 10 · p. 12
  • At R@1@1, the chained route improves recall over matched-budget direct image–text retrieval by 4.8 and 4.4 percentage points.The reported advantage is confined to 1@1; the largest tested setting, 10@10, reaches 94.4% and 95.6% recall.Source: C Chained Recall Budget Grids · p. 12, Table 10 · p. 12
  • At AB=4, the balanced 2@2 setting outperforms both 1@4 and 4@1 in both directions.Recall rises monotonically along both budget axes, but equal budget products need not yield equivalent candidate sets.Source: C Chained Recall Budget Grids · p. 12, Table 10 · p. 12

D Qualitative Cross-Modal Embedding Arithmetic

Figure 3 presents qualitative examples of cross-modal embedding arithmetic, combining two projected modalities and retrieving a nearest neighbor in the third. The examples suggest possible compositional structure but are not a quantified analogy benchmark.

D Qualitative Cross-Modal Embedding Arithmetic

Loading 2610.32311v1…