GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition

1 Introduction

• Web-scraped datasets can contain gender, racial, and geographic biases, with images often concentrated in North America and Western Europe.Large-scale scraping has enabled growth in image datasets, but the paper identifies these biases as a drawback.Source: §1 · p. 1
• Unlike DollarStreet, which curates images previously collected and released on the web, GeoDE uses a collection process targeting a different object-centric image distribution.Previously available images may already have appeared in multimodal foundation-model training data, complicating train/test separation.Source: §1 · p. 2
• The authors commission object photographs from people worldwide, obtain permission and copyright, compensate creators, and check images for identifiable people or other PII.They describe the resulting data as temporarily unseen for testing.Source: §1 · p. 2
• Adding GeoDE training images improves performance by 9% on DollarStreet and 21% on GeoDE's test split, while models perform poorly on images from Africa, East Asia, and SE Asia.GeoDE contains 61,940 images across 40 object categories and six geographic regions.Source: §1 · p. 2
• GeoDE's image distribution differs from other datasets even when controlling for world region and object class, addressing selection bias tied to incentives for exciting rather than mundane web uploads.The authors report that requesting images with specific content mitigated this selection bias.Source: §1 · p. 2
• GeoDE's collection method is costly, partly because it aims to fairly compensate content creators and curators.The authors also note possible economic-diversity bias because workers need a smartphone.Source: §1 · p. 2

2 Related Work

• GeoYFCC and OpenImages Extended remain geographically concentrated, in Europe and India respectively, despite efforts to increase dataset diversity.Table 1 compares dataset collection approaches, distributions, and sizes, and notes GeoDE's permission, privacy, and balance properties.Source: §2 · p. 2, Table 1 · p. 3
• DollarStreet and GeoDE are both crowd-sourced and relatively small, but DollarStreet images illustrate everyday actions while GeoDE focuses on visual differences in objects across the world.DollarStreet repurposes images collected by Gapminder for a social-science perspective.Source: §2 · p. 3
• Participant-generated video datasets also use paid workers to obtain examples outside common web-scraped distributions, whereas GeoDE's central goal is geographic diversity.That goal creates distinct recruitment and scope challenges.Source: §2 · p. 3
• The authors situate GeoDE within work on selection bias and responsible collection practices addressing privacy, consent, copyright, and worker compensation.The passage cites prior work on these topics.Source: §2 · p. 3

3 Collecting GeoDE

• Object classes are selected using dataset tags, visual features and clustering, and manual review, then pruned to remove non-objects, region-specific wild animals, and object variants.The initial heuristics address the difficulty of identifying globally varied objects without an existing geographically diverse dataset.Source: §3 · p. 3, Table 2 · p. 4
• The collection targets six regions and three to four countries within each, chosen partly for gaps in public datasets and Appen participant availability.The authors aim for roughly even image counts across each object-class and region pair.Source: §3 · p. 4, Table 3 · p. 4
• More than 4,500 workers representing varied genders, ages, and races uploaded images for specified object classes, and Appen's QA team manually checked submissions.The collection process asks workers to provide images for a given class.Source: §3 · p. 4

Getting sufficient images of all object classes

• Clarifying “stove” as any cooking surface addressed underrepresentation caused by workers’ differing interpretations of the class label.The authors describe this as a vocabulary challenge in geographically diverse collection.Source: §4 · p. 4

Image quality and annotation

• Duplicate views and similar instances were filtered, while tree-heavy images could be tagged to support filtering or model-error analysis.Trees often appeared alongside other outdoor objects, and foreground and background objects were not always easy to distinguish.Source: §4 · p. 4

Other

• Only 0.78% of images contained identifiable information; incidental non-identifiable people were tagged, and image counts were roughly balanced by region but less evenly by country.The authors report incidental people particularly in images of larger objects such as monuments.Source: §4 · p. 4

Cost

• $0.87 per image and $54,000 overall, excluding researcher time, made this collection method expensive while compensating photographers, managers, and QA staff.The authors attribute part of the cost to aiming for fair compensation.Source: §4 · p. 4

Regional scope

• The six regions are broad, so some objects may vary within a region, although the authors did not observe this empirically in their analysis.This qualifies what regional grouping may capture about object appearance.Source: §4 · p. 4

Qualitative

• GeoDE’s sampled stoves and houses show greater variety than ImageNet’s, including more varied stove types and house materials and sizes.The stoves also appear more used than those in ImageNet.Source: §5 · p. 5

Object appearance

• DollarStreet features overlap with GeoDE more than GeoYFCC features do, but some distributions still differ, including “Stove” in the Americas.The comparison uses t-SNE visualizations of features by object class and region.Source: Figure 5 · p. 6

Statistics

• GeoDE is balanced across regions and classes, while GeoYFCC has over 1M images but most countries are from Europe and its top 20 of 1,197 tags comprise 34% of the dataset.DollarStreet has 38,479 photos across 289 classes, averaging 133 images per class; GeoDE has 61,940 photos across 40 classes, averaging 1,548 images per class.Source: §5 · pp. 5–6

Object appearance

• 96.3% accuracy distinguishing GeoYFCC from GeoDE and 96.1% distinguishing DollarStreet from GeoDE show that a classifier can distinguish each dataset pair.The analysis also compares features within matched region–object pairs; the authors note the GeoDE–GeoYFCC difference is more pronounced and likely related to web-scraping.Source: §5 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6

6 GeoDE as an evaluation dataset

• Both models are more than 7% less accurate on African than European images, and CLIP averages 82.8% accuracy overall.CLIP accuracy is especially low for dustbin (37.3%), medicine (54.1%), cleaning equipment (59.0%), and spices (63.2%).Source: §6 · p. 6, Table 4 · p. 7
• CLIP sometimes labels houses as religious buildings, particularly in images from Asia.The examples illustrate a systematic error pattern rather than only low average accuracy.Source: Figure 6 · p. 7
• 31 of 40 objects have a regional CLIP accuracy outside the corresponding 95% confidence interval, indicating substantial geographic variation.Examples include fence accuracy of 88% in Europe versus 60% in Africa, and stove accuracy of 95% in the Americas versus 67% in East Asia.Source: §6 · p. 7

7.1 Training a model with GeoDE

• The comparison uses equal-sized training sets, replacing ImageNet images with GeoDE images in the combined set.The combined set contains 29,820 GeoDE images and 8,533 ImageNet images; the ImageNet-only training set contains 38,353 images.Source: §7.1 · pp. 7–8
• For 21 of 40 objects, regional accuracy falls outside 99.9% confidence intervals after a Bonferroni correction, confirming significant geographic variation.The CLIP analysis also reports that 31 of 40 objects fall outside their corresponding 95% confidence intervals.Source: §7.1 · p. 7, Figure 8 · p. 8
• CLIP feature plots show region-specific groupings, including East and Southeast Asian monasteries and temples among religious buildings.The ImageNet-trained model and CLIP perform best on European images and worst on African images.Source: Figure 7 · p. 8, Figure 8 · p. 8

7.2 Cost-vs-Diversity tradeoffs

• GeoDE images cost more to collect than images from the standard web-scraping and crowd-sourced annotation pipeline, motivating region- and class-level trade-off analysis.The study varies data from one region while keeping dataset size and class balance constant, and compares performance on that region and Europe.Source: §7.2 · pp. 8–9

7.1 Training a model with GeoDE

• ImageNet plus GeoDE improves accuracy across all regions on both GeoDE and DollarStreet test sets, with improvements across most overlapping DollarStreet categories.The authors suggest this indicates GeoDE is more geographically diverse than ImageNet and that geographically diverse training data has an advantage.Source: §7.1 · p. 8, Table 5 · p. 9

7.2 Cost-vs-Diversity tradeoffs

• Adding regional GeoDE data improves performance in that region and in Europe, with larger relative gains in the added-data region and no observed saturation.Spices, waste container, and cleaning equipment benefit most; Figure 9 compares per-object relative AP improvement and highlights classes with the largest increases.Source: §7.2 · p. 9, Figure 9 · p. 9

8 Conclusion

• GeoDE’s crowdsourced collection avoids personally identifiable information, secures image rights and creator permission, compensates creators, and allows control over geographic diversity and object distribution.The authors report that GeoDE highlights shortcomings in common models and can improve performance when added to training data.Source: §8 · p. 9
• The authors conclude that crowd-sourcing is a viable method for creating diverse and responsible image datasets.They also describe GeoDE as useful for evaluating common models and improving performance when included in training.Source: §8 · p. 9

Appendix

• The appendix includes a datasheet, category-selection heuristics, a feature-space comparison with ImageNet, finetuning results, and further dataset counts and examples.It also provides additional experimental details.Source: Appendix · p. 11

MOTIVATION

• GeoDE was created for geographically diverse training and evaluation and to understand the process of crowdsourcing an image dataset from scratch.The dataset has been used in this paper to evaluate large-scale models for geographical bias; other possible uses include studying geodiversity and differences between web-scraped and crowd-collected images.Source: MOTIVATION · p. 12, PREPROCESSING / CLEANING / LABELING · p. 15

COMPOSITION

• GeoDE is a sample balanced across 6 regions and 40 objects selected to maximize geodiversity, with 61,940 images and image-level object labels and metadata.Metadata includes location, phone type, background people, and tree coverage; the datasheet says no people are recognizable.Source: COMPOSITION · p. 12

PREPROCESSING / CLEANING / LABELING

• GeoDE has possible object- or country-label errors affecting less than 1% of the dataset, and smartphone-only collection means it lacks economic diversity.The datasheet reports that no preprocessing, cleaning, or labeling was performed.Source: COMPOSITION · p. 13, PREPROCESSING / CLEANING / LABELING · pp. 15–16

COLLECTION

• Participants worldwide photographed specified objects for compensation, and Appen’s quality team manually verified images contained the objects.Collection took place in 2022 and cost $54,000, excluding researcher time.Source: COLLECTION · p. 14

DISTRIBUTION

• GeoDE is freely available as a .zip under a CC-BY license, with Princeton hosting it and updates planned as needed.The datasheet says older versions will not remain hosted, but update information and an applicable update script will be provided on GitHub and the website.Source: DISTRIBUTION · p. 16, MAINTENANCE · pp. 16–17

B Selecting object categories for GeoDE

• Three GeoYFCC [10] tag-selection methods identified geographically variable candidates, which were pruned and grouped into collectible object categories.Methods selected tags using regional prediction performance, region-classification accuracy, and clustering-based Gini impurity; subsequent filtering removed non-objects and wild animals and merged semantic variants.Source: B Selecting object categories for GeoDE · pp. 17–18
• GeoDE contributors included people of varying genders, ethnicities, and ages, as shown in participant demographics.Figure 10 presents the participant demographics.Source: B Selecting object categories for GeoDE · p. 18, Figure 10 · p. 18

C.1 Comparison to ImageNet

• To avoid noisy GeoYFCC [10] tags, the authors compare GeoDE with ImageNet using feature-space differences, extracting ImageNet21k features with a PASS [2] trained ResNet50 [17].The supplied passages describe the comparison setup but do not report its results.Source: §C.1 · p. 19
• 96.0% accuracy predicting whether features came from GeoDE or ImageNet showed that the datasets’ feature spaces are very different.The authors also visualize t-SNE plots of different objects in Figure 11.Source: §C.1 · p. 19

C.2 Comparison with DollarStreet

• DollarStreet’s top 40 categories averaged 382 images per category, compared with GeoDE’s 1,548, leaving fewer DollarStreet images after filtering to common categories.The authors conducted small-scale tests to examine how images from the two datasets differ.Source: §C.2 · p. 19
• CLIP scores varied across regions: spices showed high variance across all regions, stoves varied most in Africa and Southeast and East Asia, and East Asian religious buildings scored lower.These patterns are shown as box plots of CLIP probabilities for images within each class.Source: Figure 12 · p. 20
• 3,000 GeoDE training images matched the DollarStreet-trained baseline, while all 4,788 DollarStreet images failed to match the GeoDE-trained baseline, indicating higher relative value per GeoDE image.The test used 13 classes common to both datasets, with more than 100 DollarStreet images per class, and PASS features with linear classifiers.Source: §C.2 · pp. 19–20

D More results using GeoDE as an evaluation dataset

• CLIP performs worse on some object classes, including houses, spices, and medicine.The appendix visualizes probabilities assigned to images from the same class.Source: D More results using GeoDE as an… · p. 20
• Score variation is large across regions for spices, for selected regions for stoves, and especially low for East Asian religious buildings.The cited passage describes these as distinct patterns in the box plots.Source: D More results using GeoDE as an… · p. 20

E More results when training with GeoDE

• The supplementary training section adds results for incrementally adding regional GeoDE images and for fine-tuning ResNet50 instead of freezing its weights.Source: E More results when training with GeoDE · p. 20

E.1 Results from incrementally adding additional regions

• Adding regional GeoDE images increases accuracy more in that region than in the control, while accuracy on European images also rises.The figure compares incremental regional additions; the prose notes that the within-region increase exceeds the control.Source: §E.1 · p. 20, Figure 13 · p. 21

E.2 Results from finetuning a ResNet50 model

• Fine-tuned ResNet50 shows the same overall trend, but noisier results, potentially because it overfits the small training set.The model was ImageNet-pretrained and fine-tuned on fractions of ImageNet and GeoDE using SGD.Source: §E.2 · p. 20, Figure 14 · p. 21

E.3 Performance on individual countries

• Country-level accuracy differences remain for pretrained CLIP, but discrepancies within regions further reduce when training on GeoDE.The analysis includes countries with at least 25 test images and reports regional averages, standard deviations, and minimum-accuracy countries.Source: §E.3 · pp. 20–21

• Most object-region pairs have at least 150 images, though wheelbarrow, monument, boat, and flag have noted exceptions; some categories fall short of 175.Table 8 gives object counts by region and marks categories that did not reach 175 images per region.Source: F More details about the dataset · p. 21, Table 8 · p. 36
• Examples show regional variation in backyard surfaces, bicycle types, and boats, alongside GeoDE’s generally balanced object-region coverage.Backyards may be paved or lack lawns; examples also include children’s bicycles, rickshaws, motorized cycles, ships, kayaks, and canoes.Source: F More details about the dataset · p. 21, Figure 15 · p. 22, Figure 17 · p. 24
• Training on GeoDE reduces discrepancies among country accuracies within regions, whereas pretrained CLIP shows country-level differences.Table 7 reports regional average accuracy, standard deviation, and the minimum-accuracy country for countries with more than 25 images.Source: Table 7 · p. 36