Does Object Recognition Work for Everyone?

• Object-recognition systems are deployed globally, creating a responsibility to work equally well regardless of cultural background or socio-economic status.Source: §1 · p. 1
• Accuracy differs by around 15−20% between items photographed in the United States and those photographed in Somalia or Burkina Faso.Source: §1 · p. 1
• These geographical and income-related disparities are consistent across a range of commercial cloud image-recognition services.Source: §1 · p. 1
• The study uses household-item images spanning countries and income levels to illustrate classifications produced by publicly available recognition systems.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2

Dataset

• The Dollar Street dataset contains 135 household-item classes photographed in 264 homes across 54 countries.Source: §2 · p. 2
• The analysis removes abstract classes and evaluates recognition accuracy on the remaining 117 classes.Source: §2 · p. 2

Experimental setup

• Five cloud services plus a sixth object-recognition system are evaluated using location and purchasing-power-adjusted monthly household consumption income.Source: §2 · p. 2
• Human annotators assess accuracy@5 by checking whether any of the five highest-ranked predictions matches the dataset’s ground-truth class.Source: §2 · p. 2

Results

• Across all six systems, household-item recognition accuracy is substantially higher for high-income households than for low-income households.Source: §2 · p. 2

Caveat

• The study’s annotators were all based in the United States, which may bias annotations, although qualitative evaluations suggest the impact is very small.Source: §2 · p. 2

Results

• Accuracy is approximately 15% higher for household items photographed in the United States than for those photographed in Somalia or Burkina Faso.Source: §2 · p. 3, Figure 4 · p. 3

• Some Dollar Street labels reflect an object's affordance rather than its physical identity, such as “refrigerators” referring to cooling pots and pans.Source: §3 · p. 3
• Computer-vision datasets undersample visual scenes in heavily populated regions, especially Africa, India, China, and South-East Asia.Source: §3 · p. 3
• The same Flickr queries in Hindi and English return visually different results.Source: Figure 5 · p. 3, §3 · p. 3
• English-centered image collection excludes images tagged in other languages and can introduce subtler class-coverage problems.Source: §3 · p. 3
• The dataset distributions can be compared against a world population-density reference map to assess geographical representativeness.Source: §3 · p. 3, Figure 6 · p. 4

• The paper is related to broader research on fairness and representative computer-vision systems.Source: §4 · p. 4
• Existing machine-learning fairness research includes formulations such as statistical parity and equalized odds, alongside impossibility results.Source: §4 · p. 4
• This study focuses on multiclass and potentially multi-label prediction rather than binary prediction.Source: §4 · p. 4
• Because the image input lacks user or group-membership information, group-based fairness formulations are difficult to apply directly.Source: §4 · p. 4
• Related computer-vision studies have documented disparities in gender classification, image-search representation, and ImageNet class associations.Source: §4 · p. 4

• The analysis addresses representation bias and elements of measurement bias, but not historical, evaluation, or aggregation bias.Source: §5 · p. 5
• The study identifies geographical and income-related accuracy disparities without solving them.Source: §5 · p. 5
• Suggested mitigation approaches include geography-based resampling and multilingual training, but they may be insufficient on their own.Source: §5 · p. 5
• The authors suggest that broader solutions may require few-shot learning of new visual classes and reduced sensitivity to training-data variation.Source: §5 · p. 5

Annotation

• Human annotators assessed whether any of five top-ranked predictions matched each image's ground-truth class annotation.Source: Figure 7 · p. 7

Dataset

• The analysis used 117 Dollar Street classes comprising 20,455 total images.Source: Table 1 · p. 7

Income disparities

• Among the ten classes with the largest income-group accuracy discrepancies, some labels reflect affordances rather than the depicted object.Source: Classes with Largest Accuracy… · p. 7, Figure 8 · p. 8
• In India, accuracy across 2,221 images remains correlated with household income after holding country constant, suggesting location and income both partly drive the observations.Source: Decoupling Geographical Location and… · p. 7, Figure 9 · p. 8
• The supplemental figures include the ten classes with the largest differences between the highest- and lowest-income groups.Source: Figure 8 · p. 8

Geographical disparities

• A country-level Facebook-system map encodes approximately 60%, 75%, and 90% accuracy with red, yellow, and green, respectively.Source: Analysis of Facebook System · p. 7, Figure 10 · p. 8