PaperlayerView exact revision on arXiv

Source-linked AI summary

Institutional Prestige as Geographic Bias in Large Language Models: Evidence from Three Factorial Experiments with Bootstrap Confidence Intervals

Maikel Leyva-Vazquez, Florentin Smarandache

arXiv:2608.18107v1cs.CLcs.AI

TL;DR

This paper examines whether LLM candidate evaluations reflect institutional prestige, geography, or name-origin ethnicity, using factorial experiments to separate these signals. Across three studies, institutional and journal prestige produced robust evaluation advantages, with journal prestige dominating institutional prestige.

  • Problem

    Existing LLM-bias research provides limited factorial evidence separating institutional prestige from geographic stereotyping and testing whether publication venue interacts with institutional prestige.

  • Method

    Three factorial experiments vary candidate names, institutions, countries, and journals across four LLMs and five professional domains, using bootstrap confidence intervals and NBI analysis.

  • Results

    Journal prestige dominated institutional prestige by 5.7× (+1.937 vs. +0.341), while institutional prestige exceeded country-of-origin effects (+0.185 vs. +0.126).

  • Takeaways & Limitations

    The findings identify prestige recognition, especially through publication venue, as the strongest reported evaluation signal, while NBI captures inconsistency affecting low-prestige profiles.

  • Takeaways & Limitations

    Generalizability is limited by four universities in three countries, English-only credentials, institution–city confounding, and an extreme journal-prestige contrast.

Abstract

from arXiv · show

We investigate whether large language models (LLMs) systematically discriminate in candidate evaluations based on applicant name ethnicity and/or institutional prestige and geographic location. Three factorial experiments are reported (4,320 API calls, four LLMs, five professional domains). Study 1 (3x4 design) finds a statistically robust institution-tier gradient of +0.297 points on a 10-point scale (95% bootstrap CI: +0.175 to +0.422), while name-origin effects are negligible and non-significant (95% CI crosses zero). Study 2 (2x2 Prestige x Country design) breaks the prestige-geography confound: the prestige effect (+0.185; 95% CI: +0.093 to +0.275) exceeds the country-of-origin effect (+0.126; 95% CI: +0.037 to +0.218) by 1.5x. Study 3 (2x2 Journal x Institution design) reveals that journal prestige (Nature vs. a peripheral open-access journal) dominates institutional prestige by 5.7x: journal effect +1.937 (95% CI: +1.811 to +2.062) vs. institution effect +0.341 (95% CI: +0.184 to +0.504). A "rescue effect" is confirmed: publishing in Nature compensates for low institutional prestige more strongly for candidates from the University of Guayaquil (+2.127) than from MIT (+1.745). Results are quantified using the Neutrosophic Bias Index NBI<T,I,F>; the I component reveals elevated evaluation inconsistency for low-prestige profiles, an epistemic disadvantage not captured by mean-only metrics. Code and data: https://github.com/mleyvaz/geo-bias-llm

1 Introduction

This study examines whether LLMs reproduce geographic and institutional inequalities when used as automated evaluators. Three factorial experiments disentangle institutional prestige, country, and publication-venue signals, with bootstrap confidence intervals reported throughout.

  • Motivation: LLMs increasingly evaluate scholarship, hiring, credit, and research-funding candidates while encoding latent associations from training data.The introduction frames these systems as potential reproductions of existing geographic and institutional inequalities.
  • Research gap: Prior research has examined gender, race, and name-based ethnic signaling, while institutional prestige bias has emerged in peer review and university recommendations.The paper identifies a need for cleaner designs to distinguish these bias channels.
  • Study design: Three factorial studies test institution-tier, prestige-versus-country, and journal-versus-institution effects across LLM evaluations.Study 1 uses a 3×4 design; Study 2 uses 2×2 Prestige × Country; Study 3 uses 2×2 Journal × Institution Prestige, testing whether publication venue modifies institutional bias.
  • Study design: All studies report 10,000-iteration bootstrap 95% confidence intervals for their estimated effects.The confidence intervals provide the study’s stated uncertainty quantification framework.

2 Related Work

Prior work documents demographic, institutional, and geographic bias in LLM outputs, while identifying institutional prestige as a distinct bias channel. However, existing studies do not jointly test prestige and country or examine journal prestige independently.

  • Related Work: LLM bias research spans demographic, institutional, and geographic dimensions, including race, ethnicity, gender, and name-based effects in hiring and recommendations.Large-scale audits report race and ethnicity effects in hiring, although alignment-trained models show reduced name-based discrimination.
  • Related Work: 72.45% of model-generated suggestions favour top-ranked institutions despite only 8.56% of real enrolment, indicating institutional prestige overrepresentation.Peer-review simulations likewise identify institutional affiliation as the dominant bias channel, with low-prestige manuscripts facing rejection penalties.
  • Related Work: Geographic bias studies report Western defaults and country-of-origin effects, but none apply a clean Prestige × Country factorial design or test journal prestige independently.The present study addresses these three gaps.

3 Methodology

The methodology varies candidate names, institutions, and journals across controlled factorial experiments evaluated by four LLMs in five professional domains. It uses standardized 0–10 scoring and Neutrosophic Bias Index components with bootstrap confidence intervals to quantify evaluation differences and inconsistency.

  • Model and candidate setup: Four LLMs evaluate candidates whose credentials remain constant while name origin, institution, and/or journal vary across conditions.The models are Claude Haiku 4.5, GPT-4o-mini, Gemini 2.0 Flash, and Llama 3.1 8B Instruct, tested at temperature = 0.1.
  • Evaluation scenarios: Thirty scenarios cover scholarship, hiring, credit, health, and public policy, using evaluator prompts without anti-bias instructions and mandatory 0–10 SCORE extraction.Fewer than 0.5% of responses required fallback extraction.
  • Study 1 design: 1,440 API calls implement Study 1 by crossing three name origins with four institution tiers across 30 stimuli and four models.Institution tier and country co-vary in Study 1; Study 2 addresses this limitation directly.
  • Study 2 design: Study 2 uses a 2×2 Prestige × Country design comparing MIT, UNAM, Framingham State University, and Universidad de Guayaquil.Prestige and country main effects are defined as averages of the corresponding high- and low-level conditions.
  • Study 3 design: Study 3 crosses high- versus low-prestige institutions with high- versus low-prestige journals while holding candidate name, city, and credentials constant.The institutions are MIT and Universidad de Guayaquil; the journals are Nature and NCML.
  • Bias measurement: NBI = ⟨T, I, F⟩ quantifies model evaluations, with F measuring downward deviation from the Anglo-MIT reference and bootstrap 95% CIs based on 10,000 resamples.The 30 stimuli are resampled with replacement independently for each comparison, and σmax = 5 bounds I to [0, 1].

4 Results

Across three studies, institutional and journal prestige produced positive evaluation gradients, while name-origin effects were non-significant and low-prestige profiles showed greater inconsistency. Journal prestige was the dominant effect, with Nature producing the largest premium for the low-prestige institution.

  • Study 1: Institution tier: +0.297 points marked the cross-model institution-tier gradient (95% CI: [+0.175, +0.422]), whose interval was entirely positive.The T1 vs. T2 contrast was +0.189, while T2 vs. T3 was −0.011 and non-significant.
  • Study 1: Name origin: +0.094 was the Arabic–Anglo contrast and +0.073 the Latino–Anglo contrast; both 95% CIs crossed zero and neither was significant.Anglo names had the lowest cross-model mean at 7.540, compared with 7.633 for Arabic and 7.612 for Latino names.
  • Study 2: Prestige and country: +0.185 was the cross-model prestige effect versus +0.126 for country, with both 95% CIs entirely positive and prestige exceeding country by 1.5×.The critical UNAM–FSU contrast was +0.058 (95% CI: [−0.072, +0.186]), which crossed zero.
  • Study 2: Domain effects: +0.160, +0.278, and +0.201 were significant prestige effects in hiring, credit, and public policy, respectively.Country effects were significant in hiring (+0.187) and credit (+0.264), while scholarship and health effects were non-significant.
  • Study 3: Journal and institution prestige: +1.937 was the journal prestige effect versus +0.341 for institution prestige, making journal prestige 5.7× larger.All four models showed highly significant journal effects; institutional effects were significant only for Gemini and Llama.
  • Study 3: Rescue effect and inconsistency: +2.13 was the largest journal premium for the UGye + Nature rescue-effect cell, while T5 profiles had higher inconsistency values (I = 0.108–0.187) than reference profiles (I = 0.079–0.139).The higher I values indicate greater evaluation inconsistency for candidates from institutions appearing less frequently in training data.

5 Discussion

The discussion finds robust institutional-prestige bias despite negligible name-origin effects, with journal prestige exerting substantially greater influence and increasing inconsistency for low-prestige profiles. It also identifies methodological and sampling limitations that constrain interpretation.

  • Studies 1–2: +0.297 institution gradient is statistically robust, whereas name-origin effects (±0.094) are indistinguishable from zero at the 95% level.The authors argue that alignment training has eliminated measurable name-based discrimination but not institutional-prestige bias.
  • Studies 1–2: 1.5× larger prestige effects than country-of-origin effects are statistically significant and more consistent across models.The UNAM vs. FSU contrast (+0.058; CI crosses zero) is individually ambiguous, although UNAM ≥FSU in 3/4 models.
  • Study 3: 5.7× more evaluative weight is carried by journal prestige than institutional prestige.The rescue effect is stronger for Guayaquil-affiliated candidates (∆UGye = +2.128) than MIT candidates (∆MIT = +1.745).
  • Neutrosophic Bias Index: I = 0.108–0.187 for low-prestige profiles versus I = 0.079–0.139 for the reference, indicating greater evaluation inconsistency.The NBI framework captures an epistemic disadvantage that mean-only audits miss.
  • Limitations: Four limitations concern percentile-bootstrap assumptions, prestige–geography confounding, English-only credentials, and a small institution sample.The journal contrast is also extreme, comparing the world’s most cited journal with a peripheral open-access outlet.

6 Conclusion

Across three factorial experiments involving 4,320 API calls, LLMs systematically favored candidates from prestigious institutions, while name-based ethnic discrimination was statistically non-significant. The experiments used four LLMs, five professional domains, and bootstrap confidence intervals, with code and data publicly available.

  • 6 Conclusion: +0.297 points (95% CI: [+0.175, +0.422]) marked the institution-prestige gradient in candidate evaluations.This result was demonstrated across three factorial experiments with 4,320 total API calls.
  • 6 Conclusion: Name-based ethnic discrimination was statistically non-significant, with an effect of ±0.094 whose confidence interval crossed zero.The passage attributes this result to the effectiveness of alignment training in this setting.
  • 6 Conclusion: The experiments covered four LLMs and five professional domains and used bootstrap confidence intervals.Code, data, and reproducible experiments are available at https://github.com/mleyvaz/geo-bias-llm.

Conflict of Interest

The first author is Editor-in-Chief of NCML, which published an earlier Spanish-language version of the work and served as the low-prestige journal stimulus in Study 3.

  • Conflict of Interest: Maikel Leyva-Vázquez is NCML’s Editor-in-Chief; the co-Editor handled the earlier version’s editorial decision, while NCML was selected as Study 3’s low-prestige journal stimulus for ecological validity.The passage identifies NCML as a genuinely peripheral open-access venue and notes that the results reflect unfavourably on the journal.
Loading 2608.18107v1…