Source-linked AI summary
Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, Li Fei-Fei
TL;DR
ACS results can lag current demographics by half a decade. This paper combines publicly available data with machine learning to estimate socioeconomic statistics and political preferences, reporting an 88% chance of Democratic voting and an 82% chance of Republican voting otherwise.
Problem
Some ACS results lag current demographics by half a decade.
Source: p. 2Method
The paper combines publicly available data with machine learning to determine socioeconomic statistics and political preferences in the US population.
Source: p. 3Results
The reported election association gives an 88% chance of Democratic voting and, otherwise, an 82% chance of Republican voting.
Source: p. 2Takeaways & Limitations
The study shows it is possible to determine socioeconomic statistics and political preferences in the US population.
Source: p. 3The cost of annotating Street View photos makes it infeasible to collect the data at the desired scale.
Source: Car Classification · p. 15
Abstract
from arXiv · showhide
The United States spends more than $1B each year on initiatives such as the American Community Survey (ACS), a labor-intensive door-to-door study that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed half a decade. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may provide a cheaper and faster alternative. Here, we present a method that determines socioeconomic trends from 50 million images of street scenes, gathered in 200 American cities by Google Street View cars. Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22M automobiles in total (8% of all automobiles in the US), was used to accurately estimate income, race, education, and voting patterns, with single-precinct resolution. (The average US precinct contains approximately 1000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a 15-minute drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next Presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographic trends may effectively complement labor-intensive approaches, with the potential to detect trends with fine spatial resolution, in close to real time.
Materials and Methods
The Materials and Methods section describes using car detection and classification on Google Street View imagery to estimate demographic and voting patterns. It covers adaptations to Street View images, vehicle-classification accuracy, and the data and models used for demographic estimation.
Materials and Methods
- The work builds on earlier methods to predict income, race, education levels, and voting patterns from cars in Google Street View images.The earlier paper was a proof of concept focused on a limited set of predictions.Source: Materials and Methods · p. 12
Dataset
- The dataset contained 50,881,098 Google Street View images across 200 cities, sampled at 25-meter GPS intervals with six images per point at different camera rotations.Images were primarily acquired between June and December 2013, with 3.1% obtained in November and December 2014.Source: Dataset · p. 13
Car Detection
- The final model’s test-set average precision was 65.7; the note also reports 33.27% accuracy across distinct categories, and a single-component, eight-part DPM detected cars on all 50 million images in two weeks.A prior improved validation-set AP by 1.92 at negligible cost.Source: Car Detection · pp. 13–15, Car Classification · p. 15
Car Classification
- The classifier combined Street View and product-shot training images because annotating enough Street View photos alone was infeasible.To balance sources, the method duplicated each Street View image 10 times, and simulated noisy detector inputs by sampling image regions according to measured bounding-box overlap.Source: Car Classification · pp. 15–16
- 33.27% accuracy was achieved at the 2,657-class level; make and model accuracy were 66.38% and 51.83%, and U.S./non-U.S. manufacturing-country accuracy was 87.71%.These are the reported accuracies for car classification at each level.Source: Car Classification · pp. 16–17
- Body-type errors tended to involve similar categories, while make errors mostly favored more popular makes and country errors often involved Japan or the USA.Examples include coupe/sedan and regular-cab/extended-cab confusions; the model-level confusion matrix had a strong diagonal.Source: Car Classification · p. 17
Demographic Estimation
- Demographic estimates use car-count, price, fuel-economy, hybrid/electric, country, foreign-car, and body-type attributes.The listed attributes include average detected cars per image, city and highway miles per gallon, and percentages of cars by country and body type.Source: Demographic Estimation · p. 17
- The feature set also includes car percentages by five model-year ranges and by each of 58 makes.Year ranges span 1990–1994 through 2010–2014; car year is selected as the minimum of its possible year values.Source: Demographic Estimation · p. 17
- The ground truth comes from ACS socioeconomic data collected in 2008–2012 and precinct-level 2008 presidential vote counts for Obama and McCain.The vote analysis counts only votes for Obama and McCain.Source: Demographic Estimation · pp. 17–18
- Models train on partitioned zip-code, precinct, and city sets; ridge regression estimates income and voter affiliation, while logistic regression estimates race and education.Five-fold cross-validation selects the regularization parameter, and the final model averages five trained models.Source: Demographic Estimation · p. 18
- Analyses restrict regions to populations of at least 500 and at least 50 detected cars, and estimate voting probabilities conditioned on whether pickups or sedans are more numerous.The probability procedure defines the pickup-to-sedan ratio and estimates Democrat/Republican probabilities for ratios above or below 1.Source: Demographic Estimation · pp. 18–19
Supporting Information · Image Data
This supporting-information section describes the image data and annotations assembled for vehicle-recognition models, including car categories, training images, category metadata, and data splits for training and evaluation.
Image Data
- Annotated image data supported both training car-detection and classification models and applying them to Street View images of cities.The section describes assembling car categories, collecting product-shot and Street View images, annotating subsets, and recording metadata for each category.Source: Image Data · p. 24
Car Categories
- The 15,213 car types initially retrieved from Edmunds.com were grouped into 3,141 categories, forming sets of visually indistinguishable classes with extremely subtle differences between fine-grained categories.Source: Car Categories · pp. 24–25
Product Shot Images
- After removing categories with fewer than three disparate data sources (one post on any website used), the final dataset contained 2,657 car categories.Images lacking one central automobile with its entire exterior depicted were filtered out.Source: Product Shot Images · p. 26
Street View Images
- Street View data collection selected GPS points, gathered images, and annotated cars with bounding boxes and car-type labels.Car experts performed the final car-type annotation.Source: Street View Images · p. 26
Selecting GPS Points
- The study selected 200 cities, sampled GPS points on 25-meter grids, kept points within 12.5 meters of the nearest road, and attempted six images per point.For a handful of cities without full coverage, GPS samples from U.S. Census road data augmented the points.Source: Selecting GPS Points · p. 27
- Of a randomly selected subset of 399,331 Street View images, 26.6% were annotated as having no visible cars, while 12.4% had more than 10 cars.Annotations involved enclosing cars with bounding boxes and labeling each car’s type; car experts performed the final step.Source: Street View Images · p. 26, Selecting GPS Points · p. 28
- The image distribution shows a sharp dropoff in cars above the horizon line.Source: Selecting GPS Points · p. 28
Expert Class Annotations
- 110 expert annotators labeled the images for approximately 2,000 hours after screening for speed of at least one car per minute and precision of at least 80%.Source: Expert Class Annotations · p. 28
- 69,562 confident car-category annotations were collected, with annotators labeling only boxes taller than 50 pixels, a criterion met by 32.89% of boxes.Source: Expert Class Annotations · pp. 28–29
- Expert labeling proceeded from make to body type, model, and then trim and year, with example images aiding fine distinctions and an option to abstain.Source: Expert Class Annotations · p. 29
Car Metadata
- Car categories covered 58 makes and 777 models, with manufacturing years represented from 1990 through 2014.Source: Car Metadata · pp. 29–30
- Metadata also included 11 body types, seven manufacturing countries, city and highway MPG, and 2012 prices.Source: Car Metadata · p. 30
- Metadata came from Edmunds.com in August 2012, with some missing prices filled by car experts and visually indistinguishable classes assigned the oldest car’s metadata.Source: Car Metadata · p. 30
Dataset Summary
- Product-shot and Street View annotations were split into training, validation, and test sets of 50%, 10%, and 40% for detector and classifier training.Source: Dataset Summary · p. 30
Demographic Data
The study used ACS data for income, education, and race, alongside precinct-level vote counts from the 2008 U.S. presidential election.
Income
- Income came from ACS variable B19013_001E for 2008–2012, measuring median household income in 2013 inflation-adjusted dollars [3].Source: Income · p. 31
Education
- Education data came from the ACS [3] and was divided into mutually exclusive categories: less than high school graduate (B06009_002E), high school graduate including equivalency (B06009_003E), some college or associate’s degree (B06009_004E), and bachelor’s degree (B06009_005E).Source: Education · p. 31
Race
- Race data from the ACS covered White alone, Black or African American alone, and Asian alone [3].The corresponding codes were B02001_002E, B02001_003E, and B02001_005E.Source: Race · p. 31
Voting
- 2008 election data supplied precinct-level Obama and McCain vote counts [26]; votes for other candidates were excluded from the total.Source: Voting · p. 31
- ≥95% of votes went to Obama in many precincts, often where African Americans were concentrated; Obama also received over 50% in most precincts.The authors note that the dataset’s precincts were in major cities, which favor Democratic candidates.Source: Voting · pp. 31–32