Source-linked AI summary
Using Deep Learning and Google Street View to Estimate the Demographic Makeup of the US
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, Li Fei-Fei
TL;DR
The work addresses the challenge of designing a system that recognizes cars in Google Street View images. It uses car detection and classification to support demographic prediction, achieving reported make, model, and origin accuracies.
Problem
The central challenge is designing an object-detection system for recognizing cars in Google Street View images.
Method
The method detects cars across 50 million images using bounding boxes and scores, then applies car classification to support demographic prediction.
Results
The system classifies car make with 66.38% accuracy, model with 51.83% accuracy, and U.S. manufacturing origin with 87.71% accuracy.
Takeaways & Limitations
The reported results show that income, race, education levels, and voting patterns can be predicted from the analyzed vehicle information.
Takeaways & Limitations
Street View annotation costs make collecting labeled photos infeasible, and the process did not provide full coverage for a handful of cities.
Abstract
from arXiv · showhide
The United States spends more than $1B each year on initiatives such as the American Community Survey (ACS), a labor-intensive door-to-door study that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed half a decade. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may provide a cheaper and faster alternative. Here, we present a method that determines socioeconomic trends from 50 million images of street scenes, gathered in 200 American cities by Google Street View cars. Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22M automobiles in total (8% of all automobiles in the US), was used to accurately estimate income, race, education, and voting patterns, with single-precinct resolution. (The average US precinct contains approximately 1000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a 15-minute drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next Presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographic trends may effectively complement labor-intensive approaches, with the potential to detect trends with fine spatial resolution, in close to real time.
Materials and Methods
The study combines Google Street View imagery with computer-vision models to detect and classify automobiles, then uses vehicle data to estimate demographic patterns. The pipeline balances accuracy and computational efficiency across 50 million images.
- Demographic inference: Detection and classification outputs were used to estimate income, race, education levels, and voting patterns from cars in Google Street View images.The paper describes this as a methodology for demographic inference based on vehicle observations.
- Image collection: 50 million Google Street View images from 200 cities were collected, with six camera rotations sampled at each GPS point.Images were primarily acquired between June and December 2013, with 3.1% collected in late 2014.
- Car detection: A deformable part model with one component and eight parts detected cars across all 50 million images in two weeks.This configuration was selected as a performance-efficiency compromise.
Image Data
The authors construct a large automobile-image dataset by grouping visually indistinguishable vehicle types, sourcing product-shot images, and collecting and annotating Street View imagery. The resulting resources support car detection and classification across U.S. cities.
- Car categories: The initial list contained 15,213 U.S. car types produced from 1990 onward, which were grouped into visually indistinguishable classes.The paper uses “car” for four-wheeled automobiles including sedans, coupes, trucks, vans, and SUVs.
- Car categories: The cleaned category-building process produced 3,141 fine-grained car categories with subtle visual differences.The final product-shot dataset later retained categories with at least three disparate data sources, resulting in 2,657 categories.
- Product Shot Images: Product-shot images were crawled from cars.com and craigslist.org, filtered for a single prominent car with its exterior fully visible, and annotated with bounding boxes.Bounding boxes isolate car appearance from extraneous background during classifier training.
- Street View annotation: 26.6% of sampled images were annotated as having no visible cars.This annotation result characterizes the prevalence of car-free images in the collected Street View sample.
Demographic Data
The study uses American Community Survey and presidential-election data to define demographic and voting outcomes, then estimates them from vehicle attributes extracted from Street View images. Vehicle classification and feature-based models support estimates of income, race, education, and voting at neighborhood and precinct scales.
- Data sources: American Community Survey data measure median household income, education, and racial demographics using specified census variables and mutually exclusive education categories.Income data cover 2008–2012 and are expressed in 2013 inflation-adjusted dollars.
- Data sources: Education categories range from less than high school through graduate or professional degree.The categories include high school, some college or an associate’s degree, and bachelor’s degree.
- Data sources: Racial outcomes use ACS categories for White alone, Black or African American alone, and Asian alone.
- Voting data: The voting outcome uses precinct-level vote counts for Barack Obama and John McCain, with other candidates excluded from the total.Most dataset precincts favored Obama, and the precincts were located in major cities that favor Democratic candidates.
- Model evaluation: Scatter plots compare ground-truth income, race, education, and Obama-vote measures with model estimates, using y=x as the perfect-estimation reference.The supplementary figures also show confusion matrices, feature weights, detection probabilities, bounding-box distributions, and dataset statistics.