PaperlayerView exact revision on arXiv

Source-linked AI summary

Urban-ImageNet: A Large-Scale Multi-Modal Dataset and Evaluation Framework for Urban Space Perception

Yiwei Ou, Chung Ching Cheung, Jun Yang Ang, Xiaobin Ren, Ronggui Sun, Guansong Gao, Kaiqi Zhao, Manfredo Manfredini

arXiv:2605.09936v1cs.CVcs.IRcs.LG

TL;DR

Urban space perception lacks unified, theory-grounded resources connecting social-media imagery with scene, cross-modal, and instance-level evaluation. Urban-ImageNet addresses this with a multi-city Weibo dataset, HUSIC taxonomy, and shared benchmark library; results show task-dependent strengths, difficult post-level retrieval, and gains from SAM refinement.

  • Problem

    Urban perception research lacks unified datasets and tooling that connect social-media imagery with theory-grounded labels and scene classification, retrieval, and segmentation.

  • Method

    The paper constructs over 2M Weibo image–text pairs, organizes them with the theory-grounded 10-class HUSIC taxonomy, and evaluates three tasks in one library.

  • Results

    Models show task-dependent performance: EfficientNet-B4 leads T1, fine-tuned CLIP leads post-level T2, and SAM refinement improves segmentation AP by 27–40% relative to base detectors.

  • Takeaways & Limitations

    Urban-ImageNet provides a unified, theory-grounded benchmark for evaluating urban perception across modalities, scales, and task formulations.

  • Takeaways & Limitations

    The benchmark is geographically restricted to Chinese cities, and its full 2M corpus is class-imbalanced; broader generalization requires geographic expansion.

Abstract

from arXiv · show

We present Urban-ImageNet, a large-scale multi-modal dataset and evaluation benchmark for urban space perception from user-generated social media imagery. The corpus contains over 2 Million public social media images and paired textual posts collected from Weibo across 61 urban sites in 24 Chinese cities across 2019-2025, with controlled benchmark subsets at 1K, 10K, and 100K scale and a full 2M corpus for large-scale training and evaluation. Urban-ImageNet is organized by HUSIC, a Hierarchical Urban Space Image Classification framework that defines a 10-class taxonomy grounded in urban theory. The taxonomy is designed to distinguish activated and non-activated public spaces, exterior and interior urban environments, accommodation spaces, consumption content, portraits, and non-spatial social-media content. Rather than treating urban imagery as generic scene data, Urban-ImageNet evaluates whether machine perception models can capture spatial, social, and functional distinctions that are central to urban studies. The benchmark supports three tasks within one standardized library: (T1) urban scene semantic classification, (T2) cross-modal image-text retrieval, and (T3) instance segmentation. Our experiments evaluate representative vision, vision-language, and segmentation models, revealing strong performance on supervised scene classification but more challenging behavior in cross-modal retrieval and instance-level urban object segmentation. A multi-scale study further examines how model performance changes as balanced training data increases from 1K, 10K to 100K images. Urban-ImageNet provides a unified, theory-grounded, multi-city benchmark for evaluating how AI systems perceive and interpret contemporary urban spaces across modalities, scales, and task formulations. Dataset and benchmark are available at: huggingface.co/datasets/Yiwei-Ou/Urban-ImageNet and github.com/yiasun/dataset-2.

1 Introduction

Urban-ImageNet addresses gaps in urban-space perception research by unifying multi-task data, benchmarking tools, and a theory-grounded taxonomy. It contributes a large multi-city, multi-modal dataset, a unified library, and HUSIC labels for distinctions important to urban studies.

  • Existing resources isolate scene classification, image–text retrieval, and instance segmentation, preventing unified urban scene understanding.
  • Researchers lack shared tooling for data loading, fine-tuning, evaluation, and cross-dataset comparison across the three tasks.
  • Urban-ImageNet provides over 2M image–text pairs from 24 Chinese cities and 61 commercial sites across 2019–2025, with balanced 1K, 10K, and 100K splits plus a full corpus.
  • Urban-ImageNet-lib supports T1, T2, and T3 within one framework and enables comparison with established datasets.
  • HUSIC defines a 10-class taxonomy grounded in Lefebvre, Gehl, and Newman to distinguish activated spaces, publicity levels, and indoor or exterior environments.

2 Related Work

Prior urban and general-purpose benchmarks cover separate tasks or limited exterior imagery, while Urban-ImageNet combines theory-grounded labels, social-media text, longitudinal multi-city data, and segmentation support.

  • Places365, SUN, and related scene benchmarks lack theoretical grounding and textual metadata for socially differentiated urban spaces.
  • Flickr30K, MS-COCO, and LAION-5B provide captions or alt-text but do not capture authentic first-person spatial narratives from social media.
  • Cityscapes, ADE20K, LVIS, and Mapillary Vistas lack domain-specific vocabulary for commercial-space objects.
  • Urban-ImageNet’s multi-tier release enables scaling studies in the urban research domain.
  • Urban-ImageNet simultaneously provides user-generated content, theory-grounded labels, image–text pairing, multi-city longitudinal coverage, and per-class segmentation labels.

3 Dataset Description

Urban-ImageNet is built from curated Weibo imagery and text, organized through HUSIC’s theory-grounded taxonomy and standardized annotations across multiple dataset tiers. Its organization supports both UGC filtering and three benchmark tasks.

  • The crawler collected over 2 million image–text pairs from 61 location hashtags across 24 Chinese cities during 2019–2025.
  • Curation removed near duplicates, low-resolution images, inappropriate content, and repeated advertisements through a four-stage cleaning pipeline.
  • HUSIC uses Lefebvre, Gehl, and Newman to distinguish lived activity, social activation, and public-to-private spatial hierarchy.
  • HUSIC functions both as a UGC filtering pipeline and as a 10-way scene-classification benchmark.
  • The 100K balanced set was manually annotated by three researchers, with Cohen’s κ = 0.87 on a shared double-annotated subset.
  • Weibo text and images are linked through post identifiers, supporting a one-to-many structure in which one post may contain up to nine images.
  • T3 pseudo-labels were generated with Grounding DINO and SAM 2, while the evaluation subset used stricter thresholds and human review.
  • The release includes 1K, 10K, 100K balanced tiers and a class-imbalanced 2M corpus sharing consistent images, labels, and splits.

4 Benchmark Tasks and Evaluation Protocol

Urban-ImageNet-lib provides a unified evaluation framework for classification, retrieval, and segmentation, with task-specific data structures, models, metrics, and protocols. The benchmark separates category-level semantic alignment from harder post-level retrieval.

  • Urban-ImageNet-lib provides modular loaders, fine-tuning pipelines, evaluation scripts, and adapters for comparison with Places365, MS-COCO, and Cityscapes.
  • T1 predicts one of ten HUSIC labels using supervised and CLIP-based baselines, evaluated with Top-1 Accuracy, Macro-F1, and per-class metrics.
  • T2 includes category-level retrieval using HUSIC class prompts to measure zero-shot urban semantic alignment.
  • Post-level retrieval uses original Weibo texts and evaluates text-to-image and image-to-text ranking with multi-positive relevance.
  • T3 evaluates class-agnostic instance segmentation on a human-verified 10K subset using detector and SAM-refinement baselines.

5 Experimental Results

Urban-ImageNet shows strong supervised classification, near-trivial category-label retrieval after fine-tuning, and substantially harder post-text retrieval and instance segmentation. SAM box refinement improves segmentation, with Mask R-CNN+SAM the best trainable configuration.

  • Task 1: 84.9% Top-1 accuracy and 84.9% Macro-F1: EfficientNet-B4 leads Task 1 on the 100K benchmark.Supervised classifiers outperform zero-shot CLIP, whose HUSIC distinctions are not simple web categories.
  • Task 2: 94.2% and 92.7% average R@1: fine-tuned BLIP and CLIP make category-label retrieval straightforward.Zero-shot CLIP reaches 54.2% average R@1 in this setting.
  • Task 2: 8.1% R@1: fine-tuned CLIP is best for post-text retrieval, versus approximately 0.1% random chance.Appending HUSIC labels raises performance to 9.3% R@1 and 32.3% R@10.
  • Task 3: 0.373 AP: Mask R-CNN+SAM achieves the highest trainable Task 3 result after box refinement.The GT-box SAM oracle reaches AP = 0.749 but is not a trainable baseline.

6 Discussion

Discussion results indicate that each task favors a different architecture, domain fine-tuning is important, SAM refinement provides training-free segmentation gains, and performance improves across dataset scales.

  • Task-dependent architecture selection: No single model dominates all three tasks: EfficientNet-B4 leads T1, fine-tuned CLIP leads T2-Post, and Cascade Mask R-CNN with SAM refinement leads T3.The benchmark therefore supports task-appropriate architecture selection.
  • Domain gap: All zero-shot baselines perform substantially below fine-tuned counterparts, most severely for T2-Post.The discussion attributes this domain gap to colloquial Chinese post texts differing fundamentally from English captions.
  • SAM refinement: 27–40% relative AP improvement: SAM box refinement raises segmentation performance without additional training.The pipeline uses foundation-model post-processing for domain-adapted detection.
  • Scaling behaviour: 10–12%: the 1K→10K gain exceeds the 5% 10K→100K gain as all models improve monotonically across scales.The pattern is reported as consistent with standard scaling laws.

7 Conclusion

Urban-ImageNet is a large, theory-grounded Weibo benchmark that unifies urban scene classification, cross-modal retrieval, and instance segmentation. Results show consistent fine-tuning gains, difficult post-level retrieval, and strong training-free SAM refinement.

  • Dataset and benchmark: Over 2M image–text pairs from 24 Chinese cities, 61 venues, and seven years support the Urban-ImageNet benchmark.Its HUSIC framework provides theoretically principled annotations for three tasks in one unified benchmark.
  • Empirical findings: Domain fine-tuning consistently improves all baselines, while post-level social-media retrieval remains a genuinely hard open problem.The conclusion presents these as central experimental findings.
  • Empirical findings: SAM-based refinement offers a strong training-free gain for instance segmentation.The conclusion identifies this as a practical outcome alongside the retrieval challenge.

A.1 Task 1: Per-Class Diagnostics

Task 1 errors concentrate in categories distinguished by people, activation, or visually overlapping accommodation interiors, while retrieval evaluation has direction-specific structure and segmentation uses domain-specific object prompts.

  • Per-class diagnostics: Interior without People and activated/non-activated boundary pairs are the most challenging Task 1 categories.The distinguishing cue for boundary pairs is often the presence or absence of people.
  • Per-class diagnostics: Accommodation classes 4 and 5 are difficult because hotel and residential interiors visually overlap, especially at lower training scales.The confusion matrix also highlights commercial lodging versus private residential distinctions.
  • Direction-level retrieval: Category-label retrieval is symmetric by construction, so T2I and I2T values are identical for the ten fixed class prompts.Post-text retrieval is mildly asymmetric because the image-to-text candidate pool is smaller.
  • Direction-level retrieval: Post-text retrieval is multi-positive: all images sharing a Weibo Post ID count as positives for that text query.A single post can contain up to nine images, defining the evaluation structure.
  • Qualitative segmentation: SAM box refinement produces tighter, more complete masks than the standalone detector, particularly for glass surfaces and architectural structures.The per-class prompt vocabulary targets urban objects absent from general-purpose segmentation vocabularies.

B Scaling Behaviour: Full Results

Urban-ImageNet evaluates how performance changes across balanced training scales and task settings. Classification improves with more data, while retrieval becomes harder as the candidate gallery grows and computational cost limits some large-scale runs.

  • T1 scaling: All models improve monotonically across the 1K, 10K, and 100K training tiers on T1 classification.ResNet-50 rises from 66.5% to 78.1% to 83.5%, while fine-tuned CLIP rises from 70.8% to 78.0% to 82.3%.
  • Hierarchical granularity: 94% spatial-versus-non-spatial and 95% exterior-versus-interior accuracy exceed the fine-grained 10-class result of approximately 83–85%.
  • T2-Post scaling: 39.5% to 8.1% average R@1 as the retrieval pool grows from 100 to 1,000 images, showing that T2-Post becomes harder at larger gallery scale.
  • Computational cost: LLaVA-1.5 100K fine-tuning was not completed because estimated training exceeded approximately 150 GPU-hours on H100.Its per-sample cost is approximately 3,200× that of ResNet-50.

D Task 2 Retrieval Evaluation Protocol

T2-Post retrieval uses many-to-many positive matching for text-to-image retrieval and single-positive matching for image-to-text retrieval. The released evaluation script supports category-label and original-post text sources, with fine-tuning enabled through a command-line flag.

  • Positive matching: Each post text is a text-to-image query whose positive set contains all images attached to that post.
  • Positive matching: Each image is an image-to-text query with its corresponding post text as the single positive target.
  • Evaluation rule: A query counts as a hit at rank K when any positive target appears among the top-K retrieved results.
  • Reference performance: The 10K random-chance baseline is approximately 0.106% R@1, while fine-tuned CLIP reaches 8.1% average R@1.
  • Reproducibility: The evaluation script selects category prompts or original post text with --text-source and enables fine-tuning with --do-finetune.

F Extended Discussion and Limitations

Urban-ImageNet extends urban perception benchmarking with a class-agnostic segmentation setup, Chinese-city coverage, multiple dataset tiers, and explicit limitations. The HUSIC taxonomy separates spatially relevant classes receiving segmentation pseudo-labels from non-spatial classes used for classification and filtering.

  • Limitations: T3 evaluation is class-agnostic because current pseudo-label quality does not support reliable per-class breakdowns.
  • Limitations: All 61 venues lie in 24 Chinese cities, so generalization to other cultural or urban contexts remains for future geographic expansion.
  • Limitations: The full 2M corpus is class-imbalanced, whereas the 100K tier is recommended for researchers requiring balanced training at scale.
  • HUSIC organization: HUSIC classes 0–5 receive T3 instance-segmentation pseudo-labels, while classes 6–9 support T1 classification and UGC filtering.
  • Geographic design: Shared-brand sites across cities enable controlled same-brand, different-city comparisons.

I Per-Class Segmentation Vocabulary

The paper describes HUSIC’s annotation vocabulary, dataset scope, privacy controls, and benchmark contributions. Its checklist materials also report limitations and the absence of explicit error bars in the main results tables.

  • Annotation vocabulary: Each HUSIC class has dedicated Grounding DINO object prompts, and classes 0–5 feed detections into SAM 2 for instance-mask pseudo-labels.
  • Data governance: Only public Weibo content was collected, with rate limits, robots.txt directives, and platform terms of service followed.
  • Privacy protection: Released images omit usernames and unnecessary metadata, blur detected faces, licence plates, and QR codes, and have a maximum side length of 512 px.
  • Dataset scope: Urban-ImageNet covers 24 cities and 61 commercial sites organized by macro-region, city tier, and spatial typology.
  • Reporting limitations: The paper reports no error bars or confidence intervals in the main results tables, using five-fold cross-validation as partial variance information for T1.
Loading 2605.09936v1…