PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM Urban Sensing

Kaizhen Tan

arXiv:2610.00031v1cs.CV

TL;DR

Street-view prediction accuracy alone does not establish how much a photograph adds beyond data already available for the same place. The study compares images with existing urban data on shared units and tests source influence; results vary by attribute, with records or nearby observations often matching or outperforming images, while imagery is more useful for visible building attributes.

  • Problem

    Predictive accuracy alone does not show how much street-view imagery contributes beyond information already available for the same place.

    Source: §1 · p. 1

  • Method

    The study compares image predictions with existing urban data on the same units across tasks and tests source reliance using image replacements and conflicting records.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Existing data matched or outperformed image-only models for road damage, curb ramps, and house price, while imagery was stronger for building type and function and helped most with low-rise floor counts.

    Source: §5.1 · p. 5, §5.2 · p. 6, §6.4 · p. 10

  • Takeaways & Limitations

    Image value depends on visual legibility and local data coverage, so evaluating imagery within its information environment can guide collection decisions and clarify urban-map provenance.

    Source: §9 · p. 12

    The findings cover selected public datasets and three general-purpose VLMs, and image-record timing mismatches may affect features that change over time.

    Source: §8 · p. 12

Abstract

from arXiv · show

Street-view imagery is increasingly used to infer urban attributes, but predictive accuracy alone does not reveal how much a photograph contributes beyond data already available for the same place. We compare image-based predictions with existing urban data across seven attributes from five public resources and three VLMs. The same urban units are evaluated using images, task context, nearby observations, and public records, while image replacements and conflicting records test source reliance. Existing urban data matched or exceeded image-only models for road damage, curb ramps, and house price, while neighbouring official statistics nearly matched the best image result for population. Images were more informative for building type, building function, and low-rise floor count. For floor count, image advantage increased by 5.7 percentage points per doubling of distance to the nearest labelled building and declined for tall buildings whose rooflines often fell outside the frame. Models frequently followed conflicting records. OpenFACADES floor annotations were generated with OpenStreetMap floor values and showed the opposite height-dependent error pattern from image-only reruns. Street-view image value therefore depends on visual legibility and local data coverage. Comparing images with existing urban data can guide image collection and clarify the provenance of derived urban maps.

1 Introduction

The introduction asks how much street-view imagery adds beyond information already available for the same place. It motivates comparing image predictions with existing urban data, rather than treating predictive accuracy alone as evidence of image value.

1 Introduction

  • Strong image predictions may add little when nearby observations or public records already carry much of the same information.The introduction therefore frames image value as a comparison with data available for the same urban unit.Source: §1 · p. 1
  • Image value is expected to be greatest when an attribute is visually legible and existing urban data are sparse or weak.The introduction distinguishes what a photograph reveals from what labels, records, or local statistics predict without it.Source: §1 · p. 1
  • The study compares image-based predictions with alternatives available for the same urban units and tests how images and conflicting records shape model answers.The alternatives draw on prevalence, nearby labels, or public records across infrastructure, building, and socioeconomic tasks.Source: §1 · p. 2
  • The analysis asks which attributes gain information from imagery, how that contribution varies with reference-data density, and what happens when images and records disagree.These questions connect model evaluation to urban data collection and map provenance.Source: §1 · p. 2

2 Related Work

Related work documents growing use of street-view imagery and public resources for urban prediction. The section identifies a gap in evaluating what imagery contributes beyond nearby labels and public registers.

2 Related Work

  • Public datasets cover infrastructure, building, and socioeconomic attributes, but comparisons typically remain between models rather than information sources.The section names resources for accessibility, road damage, building imagery, and socioeconomic indicators.Source: §2 · p. 2
  • Urban observations are spatially related, so nearby labels and public registers can predict attributes that image-based models also target.This motivates comparing the target photograph with data available for the same place.Source: §2 · p. 2
  • The review found no evaluated building-attribute pipeline that compared imagery with predictors based on neighbouring labels or public registers.The review covered open-access street-view studies published since 2020 that predict objective building attributes.Source: §2 · p. 2

3 How the Urban Information Environment Shapes Image Value

The framework treats image value as dependent on both what a photograph makes visible and what existing data can predict. It distinguishes model response to an image from whether an image-only prediction beats the strongest tested non-image alternative.

3 How the Urban Information Environment Shapes Image Value

  • The same attribute can have different image value across places as viewpoint, image quality, record age, and local data coverage vary.A clear facade may reveal floor count where mapped neighbours are sparse, while a well-maintained register may leave the same image adding little.Source: §3 · p. 3
  • Visual legibility describes how much of an attribute is represented in a facade photograph, while non-image predictability describes how well existing data predict it without that photograph.Floor counts require enough of the elevation and roofline to be visible; population and house price have no direct visual referent in a single facade image.Source: §3 · p. 3
  • Within-model image gain measures the change when a VLM receives the target image, whereas standalone image advantage compares image-only prediction with the strongest tested non-image source.The first describes model behaviour; the second is closer to a city’s choice between information sources.Source: §3 · p. 3

4 Study Design and Data

The study evaluates images and non-image alternatives on the same urban units, then changes source inputs to test their influence. It spans public infrastructure, building, and socioeconomic resources with three general-purpose VLMs.

4.1 Comparing sources on the same urban units

  • Each source comparison uses the same evaluated units and keeps task context distinct from independently usable labels, registers, and statistics.The standalone comparison uses the strongest tested non-image source for each task.Source: §4.1 · pp. 3–4

4.2 Paired source comparisons

  • Paired runs replace a target photograph or add a conflicting register-style statement while holding the evaluated unit fixed.BuildingSense runs also remove place names or replace addresses to test responses to pixels and accompanying records.Source: §4.2 · p. 4

4.3 Public resources and urban settings

  • The design covers road damage, curb ramps, building attributes, population, and house price across public resources and three VLMs.The evaluated models are GPT-4o-mini, Gemini 2.5 Flash Lite, and Qwen3.5 Flash; Figure 1 links urban-unit information to paired source conditions and comparisons of image and record influence.Source: §4.3 · p. 4, Figure 1 · p. 5

5 Where Street-View Imagery Adds Information

Image value varied by attribute: existing sources led on several tasks, while imagery helped with visible building attributes and low-rise floor counts; place descriptions also supplied signal.

5 Where Street-View Imagery Adds Information

  • Image value varied more by attribute than by model, and Figure 3 separates within-model image gain from image-only advantage over the strongest non-image source.The two comparisons distinguish whether an image improves a VLM’s answer from whether an image-only prediction beats the available urban data.Source: §5 · p. 5, Figure 3 · p. 7

5.1 Existing urban data are strong for several attributes

  • Nearby observations or registers outperformed image-only models for road damage, curb ramps, and house price; neighbouring official statistics nearly matched the best image result for population.All three image-only models exceeded the non-image alternative for building type, while building-function gains were smaller and model-dependent.Source: §5.1 · p. 5
  • Curb-ramp images improved some VLM answers over location context but did not beat the municipal register; CityLens likewise distinguishes image gain from the value of neighbouring statistics.This contrast shows why improvement within a model does not establish an advantage over the strongest tested source.Source: §5.1 · p. 5

5.2 Floor-count value changes with building height and data coverage

  • Floor-count image advantage averaged about five percentage points near labelled buildings and exceeded 30 points beyond 400 metres; the relationship persisted after adjustments and leave-one-city-out fits.Images helped most for low-rise buildings; advantage became small or negative for tall buildings whose upper floors often extended beyond the frame.Source: §5.2 · pp. 6–7, Figure 4 · p. 8

5.3 Place descriptions can substitute for part of the image signal

  • BuildingSense images helped most when descriptions lacked points of interest, while removing points of interest reduced context-only accuracy for every model.Replacing the address changed little, indicating that place descriptions can contribute signal distinct from the image.Source: §5.3 · p. 7

6 When Images and Records Disagree

Replacing target photographs and adding conflicting records tested which source shaped predictions; image sensitivity, record influence, and annotation provenance differed across attributes and models.

6.1 Image replacements reveal attribute-specific sensitivity

  • Replacing the target photograph most affected building-type and floor-count predictions, which usually moved toward the donor building; road-damage predictions changed little, while curb-ramp responses varied by model.For population and house price, some predictions moved toward the donor area, but neighbouring official statistics remained more accurate overall.Source: §6.1 · p. 9

6.2 Supplied records often shape the prediction

  • Predictions often moved toward a conflicting supplied record across infrastructure, building, and socioeconomic tasks, and records influenced floor-count predictions most often for tall buildings.Record influence varied by model; it was less consistent for low-rise buildings, where images carried more usable evidence.Source: §6.2 · pp. 9–10, Figure 5 · p. 9

6.3 OpenFACADES annotations combine image and reference information

  • OpenFACADES floor annotations combined street images with OpenStreetMap floor values; unlike image-only reruns, their errors decreased for taller buildings, so their provenance matters when reused as image-only labels.Image-only floor-count error increased with height, consistent with upper floors leaving the frame.Source: §6.3 · p. 10

6.4 Patterns across models

  • The main attribute-level pattern recurred across all three VLMs, but image-replacement and record responses differed within tasks and across models.Existing data remained stronger for road damage, curb ramps, and house price; images were stronger for building type and function, with floor-count gains concentrated among low-rise buildings.Source: §6.4 · p. 10

7 Discussion

Image value varies with visual legibility and local data coverage; the discussion connects these differences to image collection, map provenance, and interpretation of multimodal results.

7.1 Image value is geographically uneven

  • Image advantage for floor count grew where mapped neighbours were farther away, but images helped less for tall buildings whose rooflines often fell outside the frame.The released annotations showed the opposite height-related error pattern from image-only reruns.Source: §7.1 · pp. 10–11, Figure 6 · p. 11

7.2 Implications for urban data collection and mapping

  • Compare candidate imagery with existing sources before acquisition: registers may suit curb ramps, while photographs can fill gaps for visible building attributes.Source choice also depends on update frequency: images suit visible change and condition, while stable attributes may already appear in administrative data.Source: §7.2 · pp. 11–12

7.3 Interpreting multimodal urban models and datasets

  • Map provenance should record whether predictions relied on photographs, nearby labels, or registers, alongside capture and record dates for changing assets.Multimodal models can rely heavily on accompanying records; OpenFACADES floor labels used both imagery and OpenStreetMap attributes.Source: §7.2 · p. 12, §7.3 · p. 12

8 Limitations

The findings are bounded by the selected public resources and models, and by variation in dataset geography, sampling, task definition, and image dates.

8 Limitations

  • The public resources differ in geography, sampling, task definition, and image date; temporal mismatches may affect features that changed between capture and record retrieval.Project Sidewalk panoramas were often newer than crowd labels, and OpenFACADES images and OpenStreetMap references came from different years.Source: §8 · p. 12
  • The study covers selected public datasets and three general-purpose VLMs, so specialized models, fuller registers, or different viewpoints could change the source balance.The literature review covers studies with accessible full text, and its reporting summary applies to that corpus.Source: §8 · p. 12

9 Conclusion

Across the tested tasks, imagery added information selectively, while nearby observations and public records often provided strong alternatives. Supplementary checks describe the literature corpus and test spatial, temporal, and image-replacement settings.

9 Conclusion

  • Street-view imagery added most information when attributes were visible and existing records sparse; nearby sources matched or exceeded image models for several infrastructure and socioeconomic tasks.Imagery was more useful for building form and low-rise floor count, and floor-count advantage increased in poorly mapped areas.Source: §9 · p. 12, Figure S2 · p. 15

A Literature Review and Study Selection

B Spatial and Temporal Checks

  • Road-damage accuracy declined as the gap to labelled neighbours widened but stayed above prevalence; nearby observations also identified positive damage under class-sensitive metrics.The source ordering remained the same under metrics that weighted positive damage classes more heavily.Source: B Spatial and Temporal Checks · pp. 15–16, Table S2 · p. 15, Table S3 · p. 16
  • On 100 matched Project Sidewalk clusters, the municipal register outperformed both image sets, and image date mattered for mutable street assets.The retrieved panoramas were generally newer than the labels, so the check also used archived source-era imagery.Source: B Spatial and Temporal Checks · p. 16, Table S4 · p. 16
  • Image-replacement checks sampled donor images independently while retaining target context, with task-specific separation between target and donor labels.The floor-stratified building-type sample covered six reference classes and was apartment-dominated; macro-F1 and balanced accuracy weighted observed classes equally.Source: B Spatial and Temporal Checks · pp. 16–17, Table S5 · p. 17, Table S6 · p. 17

C Floor-Count Data and Spatial Variation

Floor-count image advantage depended on both building visibility and reference-data density, while other checks examined model, attribute, annotation, and prompt settings.

C Floor-Count Data and Spatial Variation

  • The seven metropolitan areas varied in reference density and building height; camera distance increased with height, and capture year showed no monotonic relationship with accuracy.Very close tall-building views often lost rooflines, while distant views lost facade detail; reference tags were read in 2026 for images captured from 2012 to 2024.Source: C Floor-Count Data and Spatial Variation · p. 17
  • Image advantage increased across the observed range of reference density and remained positive across spatial settings; exact-count images beat interpolation for low-rise buildings but not tall ones.High-rise images often omitted rooflines or had steep angles or occlusion, consistent with the height pattern.Source: C Floor-Count Data and Spatial Variation · pp. 17–18, Table S7 · p. 18

D Supervised Replication

  • Under spatial blocking, performance declined for both sources, with a larger decline for the spatial predictor; consequently, the image advantage widened from about 12 to 21 percentage points.Blocked test areas had fewer nearby training labels, the setting where the main analysis found imagery most useful.Source: D Supervised Replication · p. 18

E Additional Attribute Comparison

F BuildingSense Class-Balanced Results

G Height Pattern in Annotation Errors

  • Image-only error increased with reference floor count, whereas released annotation error declined; models followed supplied floor values most strongly for tall buildings.This reversal is consistent with greater use of the supplied record when the image becomes difficult to read.Source: G Height Pattern in Annotation Errors · p. 20

H OpenFACADES Annotation Provenance

I Prompt Design

  • Prompts requested fixed-format answers at temperature zero, with task-specific instructions for floor count, building type, road damage, curb ramps, BuildingSense, and CityLens.The CityLens conditions included location-only runs, neighbouring official scores, and cross-city image replacements.Source: I Prompt Design · p. 22

J Additional Paired Results

The paired image-replacement results include three house-price estimates with differing uncertainty intervals. For CityLens house price, Gemini returned few answers after image replacement, and the main figure marks the estimate with an open symbol.

J Additional Paired Results

  • The companion archive provides model and source-condition results, including paired image-replacement results used in the main source-conflict figure.It also includes task-specific error, correlation, height-group, and text-comparison results.Source: J Additional Paired Results · p. 22
  • For CityLens house price, Gemini returned few answers after image replacement, and the main figure marks this estimate with an open symbol.The archive retains raw responses and condition-level counts.Source: J Additional Paired Results · p. 22
  • House-price image-replacement effects were −0.8 [−9.2, 7.5] (120), +20.0 [0.0, 50.0] (10), and +8.3 [−0.8, 17.5] (120).Positive values indicate lower accuracy after replacement; brackets give uncertainty intervals and parentheses give paired sample sizes.Source: Table S11 · p. 22

Related papers

Loading 2610.00031v1…