PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning the City : Quantifying Urban Perception At A Global Scale

Abhimanyu Dubey, Nikhil Naik, Devi Parikh, Ramesh Raskar, César A. Hidalgo

arXiv:1608.01769v2cs.CV

TL;DR

Existing urban-perception methods lack the scale and resolution needed for global city studies. This paper combines a crowdsourced dataset spanning 56 cities with neural networks trained on pairwise judgments, showing that urban perception data can be produced globally.

  • Problem

    Existing surveys, audits, and computer-vision methods lack the scale and resolution needed to quantify urban perception across cities worldwide.

  • Method

    The paper crowdsources pairwise judgments across six perceptual attributes and trains Siamese-like convolutional networks with classification and ranking losses to predict them.

  • Results

    The networks generalized across visual attributes and predicted perceived street safety in six new cities spanning six continents.

  • Takeaways & Limitations

    The resulting global urban-appearance dataset can support computational studies of perception across cities and disciplines including architecture, sociology, and economics.

  • Takeaways & Limitations

    Crowdsourcing larger datasets requires very large numbers of pairwise comparisons, with stable rankings needing 24–36 comparisons per image.

Abstract

from arXiv · show

Computer vision methods that quantify the perception of urban environment are increasingly being used to study the relationship between a city's physical appearance and the behavior and health of its residents. Yet, the throughput of current methods is too limited to quantify the perception of cities across the world. To tackle this challenge, we introduce a new crowdsourced dataset containing 110,988 images from 56 cities, and 1,170,000 pairwise comparisons provided by 81,630 online volunteers along six perceptual attributes: safe, lively, boring, wealthy, depressing, and beautiful. Using this data, we train a Siamese-like convolutional neural architecture, which learns from a joint classification and ranking loss, to predict human judgments of pairwise image comparisons. Our results show that crowdsourcing combined with neural networks can produce urban perception data at the global scale.

1 Introduction

The paper addresses the limited resolution and scale of existing urban-perception studies by introducing a globally crowdsourced dataset and neural models for predicting perceptual judgments from street-level images. The approach supports cross-attribute prediction and safety estimation in six cities excluded from training.

  • Urban appearance is widely believed to influence residents’ behavior and health, motivating policy initiatives and systematic measurement.
  • Existing field surveys, virtual audits, and crowdsourced studies lack the resolution and scale needed to exploit global Street View imagery.New York City alone has roughly one million street blocks, making exhaustive city-wide appearance datasets difficult to generate manually.
  • Streetscore cannot accurately measure perceived safety beyond its regional training data, limiting globally representative urban-appearance datasets.Its training set contained only a few thousand images from New York and Boston, whose architecture and planning may differ from other cities.
  • 1.17 million pairwise comparisons cover 110,988 images from 56 cities in 28 countries across six continents, rated by 81,630 volunteers on six perceptual dimensions.The Place Pulse 2.0 dimensions are safe, lively, boring, wealthy, depressing, and beautiful.
  • Networks trained on one visual attribute fairly accurately predict others and estimate perceived street safety in six new cities from six continents excluded from training.

2 Related Work

The paper situates its approach across four research strands and contributes a CNN-based technique that predicts human judgments of urban appearance from a global crowdsourced dataset.

  • Research context: The work connects literature on image perception, urban imagery, urban appearance and socioeconomic outcomes, and image ranking and comparison.These four strands frame the paper’s research context.
  • Predicting image perception: Prior work predicts perceptual responses including aesthetics, memorability, interestingness, virality, and perceived street-level safety.The related literature includes Streetscore, which uses generic image features and support vector regression to predict perceived safety from United States street-level images.
  • Urban imagery: Urban imagery from sources such as Google Street View and Flickr has enabled computer-vision applications in architecture, urban planning, urban economics, and sociology.Prior studies use these sources to identify geographically distinctive visual elements and analyze cities.
  • Urban appearance and outcomes: Prior research links perceived unsafety and urban appearance with residents’ health and criminal behavior, including alcoholism, obesity, sexually transmitted disease spread, and Broken Windows Theory.The cited literature examines how places perceived as unsafe or disorderly relate to socioeconomic outcomes.
  • Image ranking and comparison: Image-ranking research studies relative attributes and subjective comparisons for tasks including image classification, guided image search, and clothing-style ranking.This literature includes annotations collected through an online game and methods for generating image-comparison metrics.
  • Contribution: The paper contributes a CNN-based technique that predicts human judgments on urban appearance using a global crowdsourced dataset.This contribution extends the four related literatures identified by the authors.

3 The Place Pulse 2.0 Dataset

Place Pulse 2.0 is a global crowdsourced dataset of street-level urban-perception judgments, built from 110,988 Google Street View images across 56 cities and 1,169,078 pairwise comparisons from 81,630 online users. Its relative-comparison interface was designed to efficiently and consistently capture perceptions across six urban-appearance attributes.

  • Data collection: 110,988 Google Street View images were collected from 56 major cities in 28 countries across all six inhabited continents.Images were sampled using uniformly gridded latitude-longitude points over city-boundary polygons and captured between 2007 and 2012.
  • Comparison protocol: Users viewed a randomly chosen image pair side by side and selected one in response to a preselected perceptual question.The web interface collected relative comparisons rather than numerical scores, following prior work arguing that relative comparisons are more efficient and accurate for human rankings.
  • Perceptual attributes: The dataset measures six urban-appearance attributes: safe, lively, boring, wealthy, depressing, and beautiful.Each participant selected one of the six questions before comparing the displayed image pair.
  • Crowdsourcing: 1,169,078 pairwise comparisons were collected from 81,630 online users between May 2013 and February 2016.Users contributed 16.6 comparisons on average; 6,118 provided one comparison, while 30 provided more than 1,000.
  • Data consistency: Prior Place Pulse findings indicated no significant cultural bias, high inter-user reproducibility, and high transitivity in perceptions of urban appearance.Because these findings established consistency, demographic information was not collected for the larger Place Pulse 2.0 dataset.

4 Learning from the Place Pulse 2.0 Dataset

This section develops a neural approach that predicts pairwise urban-perception comparisons directly from image content. It replaces comparison-heavy ranking workflows with a Siamese-style CNN and an end-to-end combination of classification and ranking losses.

  • Motivation: 24 to 36 comparisons per image are needed for stable TrueSkill rankings, implying ∼1.2 to 1.9 million comparisons per question for 110,988 images.After 33 months, the dataset contained only 3.35 comparisons per image per question on average.
  • Problem Formulation: The method learns a ranking function fr(x) from raw image pixels to satisfy as many pairwise comparison constraints as possible.Each triplet specifies two images and a label indicating which image wins.
  • Problem Formulation: The ranking task is transformed into classification, where a function extracts image-level and pair-discriminating features before predicting the winner.This formulation motivates a convolutional neural network architecture for paired images.
  • SS-CNN Architecture: SS-CNN processes an image pair through two identical, disjoint, tied-weight feature extractors whose outputs feed a fusion sub-network trained with softmax loss.The fusion sub-network uses convolutional layers followed by a fully connected layer.
  • Ranking SS-CNN: RSS-CNN adds ordinal ranking to pairwise classification because binary logistic training alone does not capture global ordering or fine-grained image differences.Its objective is L = Lc(P)+λLr(P), with λ selected by grid search to maximize validation classification accuracy.

5 Experiments & Results

Experiments show that fine-tuned Siamese and ranking CNNs predict urban-safety comparisons substantially better than generic image features, with RSS-CNN using VGGNet achieving the strongest accuracy. RSS-CNN also supports prediction across six perceptual attributes and generation of stable synthetic rankings.

  • SS-CNN: 6.5% average accuracy gain followed fine-tuning across SS-CNN experiments on the Place Pulse 2.0 dataset.The improvement was attributed to end-to-end learning and the dataset’s size and diversity.
  • SS-CNN: SS-CNN (VGGNet) achieved the best performance among the three evaluated methods and network initializations.The methods were Softmax, TrueSkill, and RankSVM; the passage identifies VGGNet as the deepest architecture but does not provide the corresponding accuracy value.
  • RSS-CNN: 73.5% accuracy was achieved by RSS-CNN (VGGNet) for pairwise comparison prediction, the highest result among the tested RSS-CNN models.RSS-CNN compares the learned ranking-function outputs for the two images in each test pair.
  • Dataset comparison: 56.7% accuracy was obtained by RankSVM using GIST, Texton Histograms, and CIELab Color Histograms, below all SS-CNN variations.The best RSS-CNN (VGGNet) model reached 73.5% accuracy.
  • Other attributes: RSS-CNN (VGGNet) was evaluated on Safe, Lively, Beautiful, Wealthy, Boring, and Depressing, with performance roughly proportional to available training comparisons.The model was trained separately for each attribute and evaluated using binary prediction accuracy.
  • Synthetic rankings: 30 synthetic comparisons per image were generated with RSS-CNN (VGGNet) to produce stable TrueSkill scores for four attributes.Random image pairs were input to the trained model, and the predicted comparisons were then supplied to ranking algorithms.

6 Discussion & Concluding Remarks

The paper contributes a crowdsourced global urban-appearance dataset and a neural architecture for predicting human-labeled pairwise image comparisons. The technique generalizes to visual-attribute studies and enables research using global urban-appearance data.

  • Contributions: The study introduced a crowdsourced global urban-appearance dataset containing pairwise image comparisons and a neural network architecture predicting human-labeled comparisons.The method focuses on predicting pairwise win/loss decisions to support image ranking.
  • Limitations: 13.2% of pairwise comparisons were equal for the given perceptual attribute and were ignored because the method predicted pairwise win/loss decisions.The paper notes that incorporating prediction of equal comparisons was not included in the described approach.
  • Broader applications: The technique can generalize to computer-vision studies of image, object, and scene-category style, perception, or visual attributes.The trained networks can generate a global dataset of urban appearance for studying how urban appearance relates to residents’ behavior and health and how architectural styles are perceived.

Supplementary Material: Deep Learning the City: Quantifying Urban Perception At A Global Scale

The supplementary material identifies the paper’s authors: Abhimanyu Dubey, Nikhil Naik, Devi Parikh, Ramesh Raskar, and César A. Hidalgo.

  • Abhimanyu Dubey, Nikhil Naik, and Devi Parikh are listed as authors.
  • Ramesh Raskar and César A. Hidalgo are also listed as authors.

Supplemental Contents

The supplement covers training-data size and prediction performance, perceptual-attribute correlations, and visual examples from Place Pulse 2.0 and six additional cities.

  • Section 1: Section 1 analyzes how training data size relates to prediction performance.
  • Section 2: Section 2 discusses correlations between perceptual attributes.
  • Section 3: Section 3 presents example images and perceptual attributes for Place Pulse 2.0, plus images and maps from six additional cities excluded from training.

1 Size of Training Data and Accuracy

The study evaluates SS-CNN and RSS-CNN as training-data size varies on the Safety attribute of PP 2.0, measuring binary prediction accuracy after fine-tuning three basic networks. Training fractions increase from 10% in 10% increments, with results reported in Figure 1.

  • Figure 1 plots the fraction of training data against accuracy for SS-CNN and RSS-CNN evaluated with the three models.
  • 240,587 comparisons form the Safety training set, while 111,040 comparisons form the test set.
  • Training fractions start at 10% and increase in 10% increments, with performance measured by binary prediction accuracy.
  • SS-CNN and RSS-CNN are fine-tuned from AlexNet, PlacesNet, and VGGNet across different training-data sizes.

2 Correlation between Perceptual Attributes

The analysis measures squared Pearson correlations among TrueSkill scores for six perceptual attributes to assess their orthogonality. Safe correlates most positively with Beautiful and most negatively with Boring.

  • Correlation analysis: TrueSkill scores for all PP 2.0 images were used to compute the Squared Pearson Correlation Coefficient (R2) between attribute pairs.The six attributes are Safe, Lively, Beautiful, Wealthy, Boring, and Depressing.
  • Correlation analysis: Safe has its largest positive correlation with Beautiful and its largest negative correlation with Boring.These relationships are reported from the pairwise attribute correlations in Table 1.

3 Example Images and Perceptual Attributes

The section illustrates the visual and perceptual breadth of the Place Pulse 2.0 dataset and describes how synthetic pairwise comparisons produce TrueSkill-based image rankings. It also extends perceived-safety scoring to six cities outside the dataset.

  • TrueSkill scoring: 30 synthetic pairwise comparisons per image are generated by feeding randomly selected image pairs into a trained RSS-CNN (VGGNet).The comparisons are used to generate TrueSkill scores for all images.
  • TrueSkill scoring: TrueSkill rankings initialize every image at µ = 25 and σ = 25/3, use β = 25/3 and ε = 0.1333, and retain only µ after updates.TrueSkill models each image as a N(µ, σ^2) random variable and updates scores after every contest.
  • Safety extension: 6 cities outside the Place Pulse 2.0 dataset receive perceived-safety TrueSkill scores from 30 generated comparisons per image.For each image, 15 comparisons pair it with an image from the original dataset and 15 compare images from the new cities.
Loading 1608.01769v2…