PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Geographically Biased

Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, Stefano Ermon

arXiv:2402.02680v2cs.CLcs.AIcs.CYcs.LG

TL;DR

This paper examines how LLM biases appear when social groups are distinguished by geography, a lens applicable to human attributes projected onto space. It elicits zero-shot location ratings and finds both geospatial knowledge and geographic bias, including disadvantage toward lower-socioeconomic-condition areas on sensitive subjective topics.

  • Problem

    The paper asks how to evaluate LLM bias across social groups distinguished by location, given that many aspects of human life are meaningfully projected onto geography.

    Source: §1 · p. 1

  • Method

    The authors use modified GeoLLM prompts to elicit zero-shot location ratings and introduce a bias score using rating variability and correlation with an anchoring distribution.

    Source: §3.3 · p. 5

  • Results

    LLMs make accurate zero-shot geospatial predictions and exhibit biases across objective and subjective topics, particularly against lower-socioeconomic-condition areas on sensitive subjective topics.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors urge researchers to attend to geographic bias when constructing training corpora and training LLMs to help models eschew harmful stereotypes.

    Source: §5 · p. 9

    The evaluated set of six objective and five sensitive subjective topics may not be exhaustive, making a single comprehensive geographic bias score difficult.

    Source: §A.1 · p. 13

Abstract

from arXiv · show

Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $\rho$ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's $\rho$ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

1. Introduction

The introduction frames geography as a way to evaluate systemic errors in LLM predictions across social groups, then presents evidence of zero-shot geospatial knowledge alongside geographic bias. It highlights lower-socioeconomic-condition areas’ disadvantage on sensitive subjective topics and a proposed score for measuring bias.

1. Introduction

  • Geographic bias is defined as systemic errors in geospatial predictions, with social groups distinguished by location.The geographic lens applies to human attributes projected onto space, including culture, race, language, economics, politics, and religion.Source: §1 · p. 1
  • Zero-shot ratings show strong monotonic correlation with ground truth, reaching Spearman’s ρ of 0.89 for average temperature.The introduction also reports accurate geospatial predictions using the expected value of ratings with logprobs.Source: §1 · pp. 2–3
  • The paper evaluates LLM knowledge in a zero-shot setting because fine-tuning on specific datasets may obfuscate inherent biases.It adapts GeoLLM prompts to elicit ratings for locations around the world.Source: §1 · p. 2
  • LLMs are biased against lower-socioeconomic-condition areas on sensitive subjective topics, including attractiveness, morality, and intelligence.The introduction gives residents in Africa versus Europe on attractiveness as an example.Source: §1 · p. 3
  • The proposed bias score shows substantial variation across LLMs, and GPT-4 Turbo is described as significantly less biased than Gemini Pro.The score uses mean absolute deviation and rank correlation with an anchoring bias distribution such as infant mortality.Source: §1 · pp. 2–3

2. Related Work

Related work situates geographic bias among social-bias evaluations and geospatial uses of LLMs. The paper emphasizes global coverage and distinguishes its zero-shot ratings approach from prior geospatial work using auxiliary map data or fine-tuning.

2. Related Work

  • Prior social-bias research often focuses on the US, while globally relevant biases may be neglected; geographic bias spans social groups linked to location.The paper lists race, ethnicity, socioeconomic status, culture, and politics among these distinctions.Source: §2 · p. 3
  • Existing prompt-based bias datasets assess specific social dimensions, while this rating-based dataset is presented as the first to comprehensively evaluate geographic biases.The ratings use a 0.0–9.9 scale and the dataset follows a question-answering format.Source: §2 · p. 3
  • Earlier geospatial LLM work includes GeoLLM’s use of auxiliary map data, GeoGPT’s autonomous geospatial tool, and K2’s geoscience fine-tuning.The paper extends GeoLLM by adding a prompt prefix for zero-shot ratings to avoid fine-tuning as a confound in bias evaluations.Source: §2 · p. 3

3. Methods

The methods avoid fine-tuning when assessing zero-shot geographic predictions, then visualize ratings and define a bias measure for sensitive subjective topics.

3. Methods

  • To avoid fine-tuning as a bias-evaluation confound, prompts elicit zero-shot predictions with a task-context prefix and GeoLLM spatial context, topic, and rating scale.Fine-tuning could introduce or exacerbate biases through its procedure and data.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1. Zero-shot Geospatial Predictions with LLMs

  • Zero-shot prediction maps coordinates to topic ratings, evaluates rank agreement with Spearman’s ρ, and extracts outputs via greedy modal ratings or expected first-digit values when logprobs are available.Expected-value ratings are continuous, while greedy sampling yields discrete values.Source: §3.1 · pp. 4–5

3.2. Visualizations on Maps

  • Maps use 2,000 prompts selected by population-importance and farthest-point sampling; fractional ranks support comparison, while signed rank-error colors expose over- and underestimates.Fractional ranks are preferred because they are more consistent across models and robust to outliers.Source: §3.2 · p. 5

3.3. Bias Score for Sensitive Subjective Topics

  • For sensitive subjective topics, the bias measure combines rank correlation with an anchor distribution, rating dispersion (MAD), and squared answer rate; ideal behavior may be constant, random, or refusal.The anchor can represent socioeconomic conditions or another geographic distribution, such as population density.Source: §3.3 · pp. 5–6

4. Experiments

The experiments compare five widely used LLMs across objective, sensitive subjective, and geographically independent topics. They find accurate zero-shot predictions alongside systematic geographic patterns in errors and bias, with bias magnitude varying across models.

4.1. Topics

  • The study evaluates five models using six objective topics with ground truth, five sensitive subjective topics, and five geographically independent topics.Objective benchmarks include socioeconomic indicators and temperature and precipitation; sensitive-topic ratings are intended to be constant, while independent topics provide a comparison for model agreement.Source: §4 · p. 6, §4.1 · p. 6

4.2. Zero-shot Performance

  • LLM ratings significantly correlate monotonically with objective-topic ground truth, and log-probability expected values improve GPT-4 Turbo and GPT-3.5 Turbo performance.The results support using geospatial datasets to evaluate geographic knowledge.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3. Common Biases on Objective Topics

  • Mean rank errors show common regional overestimates and underestimates on population density, built-up ratio, and infant mortality, alongside many correctly ranked regions.Examples include underestimation of population-density ranks in Africa and India, overestimation of built-up ratios in underdeveloped regions, and underestimation of infant-mortality ranks in parts of South and Southeast Asia.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4. Common Biases on Sensitive Subjective Topics

  • Models show little agreement on geographically independent topics but consistent regional patterns on sensitive subjective topics, whose ratings correlate positively with infant survival rate.The prominent regions are primarily in Africa, parts of the Middle East, and South Asia, versus North America, Europe, Australia, and parts of East Asia; infant mortality rate is a better predictor of this bias than the other tested objective measures.Source: §4.4 · pp. 7–8, Table 2 · p. 8

4.5. Magnitude of Biases

  • The bias score combines infant-survival correlation, rating MAD, and answer rate, and indicates substantial model variation, with GPT-4 Turbo and Llama 2 70B least biased.Rating MAD decreases on sensitive topics, often to almost three times below objective-topic MAD; with MAD below 0.02, log-probability expected ratings can reveal subtle biases.Source: §4.5 · p. 8, Table 3 · p. 8

5. Conclusion

Popular LLMs can make accurate zero-shot geospatial predictions, yet exhibit geographic bias across objective and subjective topics, especially against lower-socioeconomic areas. The authors call for attention to this bias in training corpora and model development.

5. Conclusion

  • LLMs make accurate zero-shot geospatial predictions, especially when using the expected value of ratings with logprobs.The conclusion also reports geographic bias across objective and subjective topics, particularly discrimination against areas with lower socioeconomic conditions.Source: §5 · p. 9
  • Geographic bias appears across objective and subjective topics, particularly against areas with lower socioeconomic conditions.The authors define it as systemic errors in geospatial predictions that distinguish social groups by location.Source: §5 · p. 9
  • The authors urge researchers to consider geographic bias when constructing training corpora and training LLMs to avoid harmful stereotypes.They frame the concern as stereotypes affecting applications and interactions with users worldwide.Source: §5 · p. 9

A.1. Discussion of Limitations

The evaluation covers six objective and five sensitive subjective topics, but this set may not support a comprehensive measure of geographic bias. Bias may also be difficult to observe for topics independent of geography, though such topics are described as rare.

A.1. Discussion of Limitations

  • The six objective and five sensitive subjective topics may not be exhaustive, making a single comprehensive geographic-bias score difficult to establish.The paper notes that additional topics can be added, but the set may not provide a general measure.Source: §A.1 · p. 13
  • Geographic bias may be difficult to observe for geography-independent topics such as body temperature or respiration rate, although these are described as rare.Source: §A.1 · p. 13

A.2. Geographic Bias in Real Use-cases

The authors examine whether geographic bias appears in country recommendations as well as location ratings. Their examples show recommendations varying with the requested activity and report that country mentions correlate with infant survival rates.

A.2. Geographic Bias in Real Use-cases

  • The authors hypothesize that subjective location ratings reflect LLMs’ internal belief systems and may negatively influence outputs in more typical use cases.They identify perpetuating stereotypes of geographic regions and residents as the main foreseeable harm.Source: §A.2 · p. 13
  • Recommendations were more mixed for meeting diverse or kind people: Africa was preferred in 7/10 and 3/10 responses, respectively.Source: §A.2 · p. 13
  • 0/50 responses preferred Africa for meeting likable, attractive, moral, intelligent, or hard-working people, while 30/30 preferred Africa for safari, wildlife, or deserts.The subjective-topic findings were consistent with the authors’ geographic-bias ratings.Source: §A.2 · p. 13
  • 0.56 Spearman correlation: countries mentioned in open-ended recommendations were associated with infant survival rate, and mentioned countries had a median 3.15 infant deaths per 1,000 live births versus 14 for unmentioned countries.The authors report this as consistent with their rating-based findings.Source: §A.2 · p. 13
  • For the open-ended test, GPT-3.5 produced 50 recommendations across five prompts, with 10 responses collected per prompt.The prompts asked for country lists, with most countries possible candidates.Source: §A.2 · p. 13

A.3. Robustness to Changes in the GeoLLM Prompt

Prompt ablations leave objective-topic performance and sensitive-topic correlations largely consistent, but removing the full address reduces the reported correlation with socioeconomic conditions. Restoring the state and country address elements resolves that drop.

A.3. Robustness to Changes in the GeoLLM Prompt

  • 32% drop: removing the full address lowers Spearman’s ρ for sensitive subjective topics with socioeconomic conditions from 0.50 to 0.34.The prose points to Tables 4 and 5 for prompt-ablation results.Source: §A.3 · p. 14, Table 4 · p. 14, Table 5 · p. 14
  • Objective-topic performance and sensitive-subjective correlations with infant mortality remain surprisingly consistent across prompt ablations.The reported robustness applies to removing individual prompt elements; the full-address removal is an exception for the socioeconomic-condition correlation.Source: §A.3 · p. 14, Table 4 · p. 14, Table 5 · p. 14
  • Adding the state and country elements back to the address resolves the drop in the sensitive-topic correlation with socioeconomic conditions.Source: §A.3 · p. 14

A.5. Granularity

The paper shows that geographic bias in average resident intelligence ratings can appear at neighborhood scale in the Bay Area.

A.5. Granularity

  • In the Bay Area, GPT-3.5 Turbo rates lower-socioeconomic Oakland red for average resident intelligence, while wealthier areas including Mountain View, Menlo Park, Palo Alto, Stanford, and San Francisco appear green.The authors describe the red and green regions as clear geographic biases at neighborhood level.Source: §A.5 · p. 14

A.6. Extra Figures and Tables

The supplementary figures and tables show geographic rating patterns and provide additional views of bias, rating variation, and response rates.

A.6. Extra Figures and Tables

  • Figure 6 maps GPT-3.5 Turbo’s Bay Area average-intelligence ratings, with red marking lower ratings and green higher ratings.Source: Figure 6 · p. 15
  • Figure 7 provides an enlarged version of Figure 4 with a dark background.Source: Figure 7 · p. 15
  • Figure 8 uses expected ratings and displayed log probabilities to reveal bias even when ratings appear constant, enabling continuous-valued predictions.Source: Figure 8 · p. 16
  • Table 7 reports mean absolute deviation of ratings, and Table 9 reports rating Gini coefficients across models for sensitive subjective and objective topics.Source: Table 7 · p. 16, Table 9 · p. 16
  • Table 8 reports answer rates for sensitive subjective and objective topics.Source: Table 8 · p. 16
Loading 2402.02680v2…