PaperlayerView exact revision on arXiv

Source-linked AI summary

UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, Yuxuan Liang

arXiv:2310.18340v2cs.CLcs.AI

TL;DR

Textual modality has not previously been introduced into urban region profiling, motivating investigation of whether and how it can complement satellite imagery. UrbanCLIP uses an image-to-text LLM to generate detailed satellite-image descriptions, then trains on image-text pairs with contrastive and language-modeling losses. UrbanCLIP achieves the best performance across all datasets, outperforming PGSimCLR in R^2 by 7.06%, 4.75%, 7.25%, and 5.49% in Beijing, Shanghai, Guangzhou, and Shenzhen, respectively.

  • Problem

    Textual modality has not previously been introduced into urban region profiling, motivating investigation of whether and how it can complement satellite imagery.

  • Method

    UrbanCLIP uses an image-to-text LLM to generate detailed satellite-image descriptions, then trains on image-text pairs with contrastive and language-modeling losses.

  • Results

    UrbanCLIP achieves the best performance across all datasets, outperforming PGSimCLR in R^2 by 7.06%, 4.75%, 7.25%, and 5.49% in Beijing, Shanghai, Guangzhou, and Shenzhen, respectively.

  • Takeaways & Limitations

    Textual information can serve as a complementary modality for urban visual representation learning and urban region profiling.

  • Takeaways & Limitations

    The authors identify future needs for efficient multimodal integration, improved automatic text generation and refinement, and additional downstream tasks.

Abstract

from arXiv · show

Urban region profiling from web-sourced data is of utmost importance for urban planning and sustainable development. We are witnessing a rising trend of LLMs for various fields, especially dealing with multi-modal data research such as vision-language learning, where the text modality serves as a supplement information for the image. Since textual modality has never been introduced into modality combinations in urban region profiling, we aim to answer two fundamental questions in this paper: i) Can textual modality enhance urban region profiling? ii) and if so, in what ways and with regard to which aspects? To answer the questions, we leverage the power of Large Language Models (LLMs) and introduce the first-ever LLM-enhanced framework that integrates the knowledge of textual modality into urban imagery profiling, named LLM-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining (UrbanCLIP). Specifically, it first generates a detailed textual description for each satellite image by an open-source Image-to-Text LLM. Then, the model is trained on the image-text pairs, seamlessly unifying natural language supervision for urban visual representation learning, jointly with contrastive loss and language modeling loss. Results on predicting three urban indicators in four major Chinese metropolises demonstrate its superior performance, with an average improvement of 6.1% on R^2 compared to the state-of-the-art methods. Our code and the image-language dataset will be released upon paper notification.

1 INTRODUCTION

Urban region profiling supports planning and sustainable development, but existing supervised and self-supervised approaches have limitations in labeling demands, generalizability, and modality coverage. UrbanCLIP introduces generated textual descriptions and joint contrastive-language modeling to enrich satellite-image representations.

  • 1 INTRODUCTION: Urban region profiling summarizes social, economic, and environmental attributes from web-sourced imagery for urban planning and sustainable development.Web data can provide accessible, timely, and high-resolution information compared with costly manual surveys.
  • 1 INTRODUCTION: Task-specific supervised learning requires substantial labeled data, which may limit representation generalizability and robustness across urban indicator tasks.
  • 1 INTRODUCTION: Textual modality remains relatively unexplored with urban imagery, motivating questions about whether and how language can complement satellite imagery for profiling.The paper frames these questions around effectiveness and the specific aspects improved by textual information.
  • 1 INTRODUCTION: UrbanCLIP generates detailed descriptions for satellite images with LLaMA-Adapter V2 and trains an encoder-decoder model on image-text pairs.The framework uses a visual encoder, causal-masking text decoder, decoupled decoder interaction, contrastive loss, and language-modeling loss.
  • 1 INTRODUCTION: UrbanCLIP integrates pretrained image-to-text LLM knowledge into urban region profiling and combines contrastive and generative model capacities.Experiments cover four cities and three urban indicators, with further analyses of transferability and interpretability.

2 PRELIMINARIES

Urban region profiling partitions cities into uniform regions and represents their attributes using web and spatial data. Prior work largely uses task-specific supervision or spatial-modality self-supervision, whereas UrbanCLIP adds textual information for more comprehensive representations.

  • 2 PRELIMINARIES: An urban region is formed by evenly partitioning an area of interest, while each region’s satellite image represents its observed surface structure.The paper denotes each image as I_g ∈ R^H×W×3, with H and W specifying image dimensions.
  • 2 PRELIMINARIES: Urban region profiling research includes task-specific supervised learning and self-supervised learning with spatial modalities.The related-work review identifies these as the two broad categories of existing approaches.
  • 2 PRELIMINARIES: Task-specific methods learn prediction models from data sources tailored to particular targets, whereas spatial self-supervision combines imagery with geographic or activity information.Examples include light intensity, economic indicators, satellite and street-view imagery, POIs, mobility, and knowledge graphs.
  • 2 PRELIMINARIES: Spatial-modality self-supervised learning generally generalizes better than task-specific supervised learning across urban indicator tasks.
  • 2 PRELIMINARIES: UrbanCLIP introduces textual modality alongside urban imagery to pursue more comprehensive, generalizable, and interpretable urban-region representations.This extends prior spatial-modality self-supervision with language as complementary information.

3 METHODOLOGY

UrbanCLIP integrates LLM-generated textual descriptions with satellite imagery through contrastive and language-modeling objectives. Its framework includes text generation and refinement, visual representation learning, cross-modal alignment, and optional multimodal extensions for urban profiling.

  • Text Generation and Refinement: UrbanCLIP generates detailed satellite-image descriptions with LLaMA-Adapter V2, refines them, and forms high-quality image-text pairs for training.Refinement removes or rewrites unfactual or vague information while retaining infrastructure, greenery, and activity details.
  • Single-modality Representation Learning: The visual encoder splits satellite images into 16×16 patches, linearly embeds them, adds positional information, and processes them with self-attention.A learnable [CLS] token and task-specific pooling support global and task-adapted visual representations.
  • Cross-modality Representation Learning: UrbanCLIP aligns image and text representations by contrasting matched image-text pairs against other sampled pairs, maximizing cross-modal agreement.A Transformer-based decoder provides deep inter-modal interaction over contextualized feature sequences rather than only shallow similarity.
  • Cross-modality Representation Learning: Language modeling loss trains the multimodal decoder to predict paired descriptions autoregressively at token level.The pre-training objective jointly applies image-text contrastive loss and language modeling loss.
  • Cross-modality Representation Learning: UrbanCLIP shares computation between contrastive and generative objectives through a decoupled autoregressive decoder, improving training efficiency.Causal masking allows both objectives to be computed in one forward propagation, whereas a bidirectional approach requires two passes.
  • Urban Indicator Prediction: Optional extensions support multimodal alignment with spatial data and task-specific prompting, such as directing carbon-emission prediction toward environmental information.The framework is described as having two key phases with two optional settings.

4 EXPERIMENTS

Experiments evaluate UrbanCLIP across four Chinese cities, three urban indicators, ablations, transferability tests, and an urban-insights application. UrbanCLIP consistently improves prediction and transferability, with textual refinement and joint objectives contributing to its representations.

  • 4.2 RQ1: Performance Comparison: UrbanCLIP outperforms PGSimCLR across four datasets, improving R^2 by 7.06%, 4.75%, 7.25%, and 5.49% for Beijing, Shanghai, Guangzhou, and Shenzhen.Average gains on RMSE and MAE are 7.02% and 4.27%, respectively.
  • 4.2 RQ1: Performance Comparison: UrbanCLIP performs best across carbon emission, population, and GDP prediction, with average R^2 improvements of 12.07%, 5.83%, and 0.52%, respectively.Carbon emission ranks highest, followed by population and GDP; location summaries may capture environmental POIs more readily than wealth-related information.
  • 4.3.1 Effectiveness of Textual Modality: UrbanCLIP consistently outperforms the unimodal UrbanViT variant across cities and indicators, showing that supplementary textual information strengthens visual representations.Removing textual information causes significant performance deterioration, while UrbanViT slightly exceeds ResNet-18.
  • 4.3.4 Effectiveness of Loss Design: Both contrastive-only and generative-only variants underperform the full model, while the generative objective improves text representations for multimodal visual fusion.Combining both losses yields more semantically rich satellite-image representations.
  • 4.4.1 Performance Across Cities: Across 36 source-target city pairs, UrbanCLIP achieves average R^2 around 0.411 versus 0.365 for PG-SimCLR, indicating stable transferability.UrbanCLIP leads PG-SimCLR for carbon emission, population, and GDP, with average R^2 values of 0.588, 0.384, and 0.261 versus 0.543, 0.338, and 0.215.
  • 4.4.1 Performance Across Cities: Similarity matching across Beijing, Shanghai, Guangzhou, and Shenzhen identifies regions with close urban indicators, suggesting that UrbanCLIP captures comparable spatial characteristics.The reported cross-city similarities are 0.72, 0.75, and 0.72 for the three matched images.

5 CONCLUSION AND FUTURE WORK

UrbanCLIP integrates textual modality into urban imagery profiling and reports effectiveness in experiments. The paper identifies future work in multimodal integration, text refinement, and downstream-task exploration.

  • UrbanCLIP is presented as the first framework integrating textual modality into urban imagery profiling through LLM-generated descriptions and text-image representation learning.
  • The framework is intended to support profiling urban areas across social, economic, and environmental metrics.
  • Future work includes efficient multimodal integration, improved automatic text generation and refinement, and additional downstream urban-profiling tasks.

A.1 Vision-Language Pre-Training (VLP)

Vision-language pre-training aligns visual and language representations, often leveraging frozen unimodal models. UrbanCLIP applies this scheme to satellite imagery to produce interpretable urban-region representations.

  • Vision-language pre-training seeks effective alignment between frozen vision and natural-language models.
  • CLIP exemplifies vision-language pre-training through image-text contrastive learning.
  • UrbanCLIP aligns generated descriptions with satellite images to produce interpretable representations for urban regions.

A.2 Urban Foundation Model (UFM)

Urban foundation models are pre-trained on extensive, multimodal urban data and include language- and vision-based approaches. The cited background also introduces several image-to-text foundation models used for text generation.

  • Urban foundation models pre-train on extensive urban data spanning multiple granularities and modalities.
  • Language-based urban foundation models use geo-text pre-training or adapt existing LLMs to urban scenarios, while vision-based models use pre-training or adaptation.
  • The paper introduces BLIP, Emu, and ImageBind-LLM as image-to-text foundation models used for generating text.

C DETAILS OF TEXT REFINEMENT

UrbanCLIP addresses low-quality LLM-generated descriptions, especially unfactual information and vague expressions. Its refinement pipeline combines rule-based cleaning with human counterfactual verification, while automated filtering remains unstable.

  • LLM-generated descriptions can contain unfactual information or vague expressions that weaken image-text alignment and knowledge fusion.
  • UrbanCLIP uses a two-stage heuristic refinement process consisting of text cleaning and counterfactual verification.
  • Text cleaning removes redundant and irrelevant information with rule-based regular filters and NLTK tools.
  • Counterfactual verification uses GIS- and computer-science-trained master students to check factuality, assisted by a dual-scoring mechanism for anomalous descriptions.
  • Automated BLIP-based scoring and filtering showed performance instability and low-quality filtering results compared with the heuristic approach.

D COMPLEXITY ANALYSIS

UrbanCLIP’s computational cost combines visual, textual, and cross-modal transformer operations, while excluding LLM pre-training and preprocessing from backbone training.

  • D COMPLEXITY ANALYSIS: The notation defines 𝑚1 as visual-token count, 𝑚2 as textual sequence length, 𝑑 as representation dimension, and 𝐿 as transformer depth.These quantities are used to express the visual, textual, and multimodal computational costs.
  • D COMPLEXITY ANALYSIS: Cross-modal interaction adds cross-attention with complexity O(𝐿𝑚1𝑚2𝑑).This term couples the numbers of visual and textual tokens with transformer depth and representation dimension.
  • D COMPLEXITY ANALYSIS: O(𝐿(𝑚2 2𝑑)) dominates the summed complexity for large visual and textual sequence lengths.The full expression includes visual-token, textual-token, and cross-attention terms before this approximation.
  • D COMPLEXITY ANALYSIS: UrbanCLIP excludes LLM pre-training from backbone training and performs text generation and refinement during preprocessing.The authors identify this separation as supporting practical feasibility.

E LLM LIMITATION ANALYSIS

The LLM-generated descriptions include failure cases caused by difficult-to-discern image content and hallucinations associated with LLaMA-based model capacity.

  • E LLM LIMITATION ANALYSIS: Complex residential road intersections may lead the LLM to hallucinate numerous parked cars.The authors identify this as one common bad case in generated satellite-image descriptions.
  • E LLM LIMITATION ANALYSIS: A highway crossing a residential area may be misinterpreted by the LLM as a river.This is another reported bad case attributed to limitations in interpreting satellite imagery.
  • E LLM LIMITATION ANALYSIS: These errors reflect both satellite details that are difficult for humans to discern and hallucination from LLaMA-based capacity bottlenecks.Figure 8 includes additional bad and good generated-description examples.
  • E LLM LIMITATION ANALYSIS: One example describes a heart-shaped open field surrounded by trees, with the field interpreted as containing numerous parked cars.The description illustrates the kinds of detailed scene claims produced in the analyzed cases.
Loading 2310.18340v2…