Source-linked AI summary
Fine-Grained Car Detection for Visual Census Estimation
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Li Fei-Fei
TL;DR
Traditional demographic and city data collection is costly and labor intensive, motivating faster visual alternatives. The paper detects fine-grained car types in Google Street View images and uses them to predict socioeconomic and city attributes, finding strong income correlation and broader city-level associations.
Problem
Existing demographic and city information is traditionally collected through costly surveys, while image-based computational social science has been limited in scope and scale.
Method
The paper detects cars in 50 million Google Street View images across 200 cities using a fine-grained car dataset and trains models to predict socioeconomic attributes.
Results
The predictions correlate with income and other city attributes, including pollution, crime, income segregation, and vehicle registration; income reaches Pearson’s r=0.82.
Takeaways & Limitations
Cars detected from Google Street View contain predictive information about neighborhoods, cities, and their demographic makeup, enabling large-scale sociological analysis.
Abstract
from arXiv · showhide
Targeted socioeconomic policies require an accurate understanding of a country's demographic makeup. To that end, the United States spends more than 1 billion dollars a year gathering census data such as race, gender, education, occupation and unemployment rates. Compared to the traditional method of collecting surveys across many years which is costly and labor intensive, data-driven, machine learning driven approaches are cheaper and faster--with the potential ability to detect trends in close to real time. In this work, we leverage the ubiquity of Google Street View images and develop a computer vision pipeline to predict income, per capita carbon emission, crime rates and other city attributes from a single source of publicly available visual data. We first detect cars in 50 million images across 200 of the largest US cities and train a model to predict demographic attributes using the detected cars. To facilitate our work, we have collected the largest and most challenging fine-grained dataset reported to date consisting of over 2600 classes of cars comprised of images from Google Street View and other web sources, classified by car experts to account for even the most subtle of visual differences. We use this data to construct the largest scale fine-grained detection system reported to date. Our prediction results correlate well with ground truth income data (r=0.82), Massachusetts department of vehicle registration, and sources investigating crime rates, income segregation, per capita carbon emission, and other market research. Finally, we learn interesting relationships between cars and neighborhoods allowing us to perform the first large scale sociological analysis of cities using computer vision techniques.
Introduction
The paper uses fine-grained car information from Google Street View to infer socioeconomic and city attributes at large scale. It addresses the limited scope of prior visual approaches by analyzing 50 million images across 200 American cities.
- Motivation: Prior image-based studies of neighborhoods and cities were limited in scope and scale compared with the paper’s large-scale approach.Earlier work used satellite or global Google Street View features to predict neighborhood characteristics.
- Motivation: Cars provide informative socioeconomic signals because 95% of American households own automobiles and car types reflect owners’ characteristics.The paper gives the prevalence of Teslas as an indicator of a wealthy neighborhood.
- Approach and results: 50 million images across 200 cities were processed to predict income, segregation, per-capita carbon emissions, and crime rates from detected cars.The pipeline uses Google Street View as its visual source and reports Pearson’s r=0.82 for income.
- Approach and results: The analysis finds that wealthy people drive foreign-made cars, while many vans are correlated with high crime.These relationships support the paper’s sociological analysis of cars and demographics.
- Dataset contribution: The work contributes a fine-grained dataset containing 2,657 car classes and nearly 700,000 images for large-scale recognition research.The dataset combines images from commercial and classified-car websites with Google Street View and is made publicly available.
- Approach and results: The system predicts socioeconomic and city information from a single public visual source, including pollution, crime, income segregation, and vehicle registration.The paper reports correlation with socioeconomic census data and information absent from the census.
A Large Scale Fine-Grained Car Dataset
The dataset combines broad car coverage with fine-grained visual classes and annotations designed for recognizing vehicles in difficult Street View conditions. Its class hierarchy becomes especially subtle at the year and trim levels, while Street View images are distributed unevenly across classes.
- Dataset construction: The dataset begins with roughly 18,000 car types, groups indistinguishable types, and uses AMT bounding boxes to form 2,657 visual car groups.Images were gathered from edmunds.com, craigslist.com, and cars.com.
- Class structure: At the year and trim level, class differences become extremely subtle, making the lowest levels of the hierarchy hardest to distinguish.The hierarchy and class distributions are summarized in Figure 2.
- Annotations: A bounding box is a set of pixel coordinates identifying the location of the car in an image.The annotations specify the object location needed for detection.
- Dataset construction: Street View recognition is difficult because images can contain multiple cars that are occluded and low resolution, unlike centrally framed product shots.The dataset therefore includes bounding-box annotations for cars appearing in Street View.
- Class structure: Product images cover a wide variety of classes, whereas the Street View class distribution is much more skewed.The two source types therefore provide different coverage across the dataset’s classes.
Fine-Grained Detection and Classification
The paper builds a two-stage system that detects cars in Street View images and classifies them into fine-grained categories at city scale. It combines efficient detection with a location-and-size prior and evaluates classification errors at both vehicle-model and attribute levels.
- Pipeline: The pipeline first localizes likely car regions, then classifies the highest-likelihood regions with a custom CNN.The two stages separate generic car detection from fine-grained vehicle classification.
- Car Detection: A single-component eight-part DPM achieves 64.2% average precision for large-scale car detection.The design favors simplicity and efficiency for processing 50 million images, accepting a small accuracy trade-off.
- Car Detection: The detector models bounding-box center coordinates and log(area) in an 8,000-bin histogram to incorporate Street View car-distribution knowledge.The location prior is estimated from Street View training bounding-box statistics and regularized with a pseudo count of 1 per bin.
- Car Detection: The location prior improves detection AP by 1.92 at negligible time cost and yields probabilistic bounding-box outputs.DPM scores are converted to estimated probabilities using isotonic regression learned on validation data.
- Car Classification: The CNN classifies 2,657 car classes with 31.27% accuracy on ground-truth boxes and 33.27% on true-positive DPM detections.Attribute-level accuracies are substantially higher, and most body-type errors occur between similar categories such as sedans and coupes.
Visual Census of Cities and Neighborhoods
The pipeline aggregates probabilistic car detections and attributes across images to characterize cities and neighborhoods, then relates those measurements to environmental, income, segregation, and crime patterns. These analyses show that car-derived features correlate with external socioeconomic and transportation indicators.
- Car-based city measurements: City-level car attributes are computed by aggregating expected class counts across images, allowing estimates of price, make, body type, MPG, and origin.Class-level expectations can be combined to estimate attributes such as the percentage of Hummers or foreign cars.
- Car-based city measurements: Pearson correlation with Massachusetts vehicle-registration distributions was used to evaluate the accuracy of detected city-level car attributes.The comparison covered Boston, Worcester, and Springfield using publicly released Massachusetts DMV records.
- Environmental patterns: A -0.66 Pearson correlation links calculated state average MPG with 2012 transportation-sector carbon footprints, with coastal states greener than inland states.The MPG values were calculated from car attributes detected in Google Street View.
- Income and crime: Chicago is the most segregated city, with expensive and cheap cars forming large clusters on its West and East sides, respectively.Jacksonville’s Moran’s I was only 33% as large as Chicago’s, and its expensive- and cheap-car clustering was limited.
- Income and crime: An income model using 88 car-related features reaches city-level r=0.82 and zip-code-level r=0.70 against ACS median household income.Average car price correlates with income at r=0.44, while foreign-manufactured-car percentage is the most positively correlated attribute at r=0.47.
- Income and crime: Burglary-rate predictions achieve Pearson correlation 0.61, while cars per image are the strongest listed predictors of unsafe zip codes.Cars per image correlate at r=0.31 for crimes against people and r=0.36 for crimes against properties.
Conclusion
Across 50 million images in 200 cities, the study shows that Google Street View car detections contain predictive information about neighborhoods, cities, and demographic makeup. The authors suggest that increasingly available street-view and satellite imagery could support near-real-time census prediction and future prediction of additional demographic variables.
- Conclusion: Analysis of 50 million images across 200 cities shows that detected cars contain predictive information about neighborhoods, cities, and demographic makeup.The system predicted income, crime, pollution, and relationships between cars and people from a single visual source.
- Conclusion: The authors propose that street-view and satellite imagery could enable close to real-time census prediction while augmenting or supplanting survey-based demographic data collection.Future work will investigate race, education levels, and voting patterns.