PaperlayerView exact revision on arXiv

Source-linked AI summary

When RAG Fails to Equalize: Geo-bias in Factual Question Answering over Public Companies

Abhinav Havaldar, Enrico Santus

arXiv:2608.25717v1cs.CLcs.AI

TL;DR

The paper asks whether RAG uniformly compensates for missing factual knowledge, especially across geographically unevenly represented public companies. It evaluates six LLMs on a controlled benchmark spanning four attributes and four context conditions, finding that retrieval gains remain tied to baseline knowledge while misleading context induces systematic errors. These results position retrieval as a conditional rather than universal corrective.

  • Problem

    It remains unclear whether retrieval genuinely compensates for missing knowledge or instead depends on uneven entity coverage and existing model representations.

  • Method

    The authors construct a benchmark of approximately 2,000 companies across 15 global equity indices and evaluate six LLMs on four attributes under four context conditions.

  • Results

    Perfect context narrows but does not eliminate geographic accuracy gaps, misleading context induces systematic errors, and larger models do not remove these structural effects.

  • Takeaways & Limitations

    RAG should be evaluated for geographic robustness and sensitivity to imperfect evidence rather than judged by average accuracy alone.

  • Takeaways & Limitations

    The benchmark uses Wikipedia-derived atomic facts and synthetic misleading context, is English-centric, and evaluates US/European models.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is widely assumed to mitigate factual errors in large language models (LLMs), but it remains unclear whether retrieval uniformly compensates for missing knowledge. We study this question in a controlled factual QA setting over public companies, constructing a benchmark of approximately 2,000 firms across global equity indices. We evaluate six LLMs on four atomic attributes under four conditions: no-context, perfect context, misleading context, and distraction context. We find strong geographic disparities in no-context accuracy, indicating uneven parametric knowledge. While perfect context improves performance, it does not eliminate these gaps: gains are correlated with baseline accuracy, suggesting retrieval effectiveness is coupled to internal representations. Under misleading context, models frequently copy incorrect information. Larger models improve overall performance but do not remove these structural effects. These results challenge the view of RAG as a universal corrective and highlight the interaction between model knowledge, context quality, and entity representation.

1 Introduction

Factual QA over public companies exposes uneven geographic knowledge and raises whether retrieval independently corrects those disparities. The paper introduces a controlled benchmark and finds that retrieval benefits are conditional, while misleading evidence creates systematic errors.

  • Motivation: Public-company questions vary in difficulty because factual coverage and parametric knowledge differ across markets and entities.Large English-dominant markets generally have richer documentation and stronger training-data representation than smaller or emerging markets.
  • Research gap: The paper asks whether retrieval compensates for missing knowledge or instead depends on existing entity representations and can reinforce disparities.Misleading context may also cause models to copy incorrect information.
  • Contributions: The benchmark covers approximately 2,000 public companies across 15 global equity indices, with paired inductive and deductive questions on four corporate attributes.The attributes are Industry, Founding Year, Headquarters, and Key People.
  • Contributions: The evaluation separates parametric knowledge from contextual effects using no-context, perfect-context, misleading-context, and distraction-context conditions.This framework tests evidence use and robustness to imperfect context.
  • Findings: No-context accuracy varies across markets, inductive questions are easier, perfect context narrows but does not eliminate gaps, and misleading context induces systematic copying errors.Larger models improve overall performance but do not remove these structural effects.
  • Implications: The findings imply that factual reliability depends on model knowledge, context quality, and entity representation rather than retrieval alone.Evaluation and deployment should assess robustness across markets and sensitivity to imperfect evidence.

2 Related Work

Prior work establishes that language models store substantial but uneven factual knowledge and that retrieval can improve knowledge-intensive QA. It also shows that models may ignore, over-copy, or follow incorrect context, motivating entity- and geography-sensitive evaluation.

  • Parametric knowledge: Pretrained models encode substantial factual knowledge in their parameters, but recall is brittle under paraphrase and prompt variation.Closed-book QA frames parameters as compressed factual memory that improves with scale.
  • Retrieval-augmented generation: RAG conditions models on external documents to address limits of parametric memory and has established strong baselines for knowledge-intensive QA.Retrieval and reader architectures underpin benchmark suites such as KILT.
  • Evidence use: Research shows that models may hallucinate, ignore, or over-copy retrieved evidence, especially when context conflicts with internal priors.Incorrect evidence can be followed when prior knowledge is weak, while strong models can also be misled.
  • Geographic bias: LLM factual recall varies systematically across countries and income groups, extending documented representation disparities into geographic knowledge.This paper shifts the focus from country-level recall to entity-level factual QA.
  • Domain gap: Financial QA benchmarks often emphasize numerical reasoning, whereas this work studies atomic company facts across global markets and their sensitivity to context.The complementary challenge is uneven coverage and contextual robustness.

3 Research Questions and Hypotheses

The study treats global market variation as a natural experiment to separate parametric knowledge, evidence use, and robustness to imperfect context. Its hypotheses test directional difficulty, geographic variation, non-uniform retrieval benefits, contextual over-reliance, and model-scale effects.

  • Framework: The framework analyzes factual QA across indices, question formulations, and context regimes to test whether retrieval acts independently of structural knowledge bias.The central comparison is independent correction versus conditioning on existing model knowledge.
  • Directional asymmetry: H1 tests whether entity-to-attribute and attribute-to-entity questions differ in difficulty and error patterns.These correspond to inductive and deductive directions.
  • Geographic disparity: H2 tests whether no-context factual accuracy varies across geographic indices and regions.This targets disparities in parametric knowledge without retrieval context.
  • Retrieval benefits: H3 tests whether perfect-context gains are uniform or correlated with baseline performance.The hypothesis directly examines dependence on prior knowledge.
  • Contextual over-reliance: H4 tests whether misleading or distracting context induces systematic errors or copying behavior.The focus is contextual over-reliance on incorrect or irrelevant evidence.
  • Model scale: H5 tests whether model scale improves helpful-context use and robustness to misleading context, particularly in underrepresented markets.The hypothesis concerns both capability and robustness.

4 Methodology

The methodology builds a geographically heterogeneous company benchmark, constructs paired multiple-choice questions, and evaluates six models under controlled context regimes. Accuracy and context-sensitivity measures are then stratified by baseline knowledge and modeled across indices and models.

  • Benchmark Construction: The benchmark contains 2,135 unique public companies mapped to 15 global equity indices, with 2,165 records reflecting multi-index membership.The indices span North America, Europe, Asia, Latin America, Africa, and Oceania.
  • Benchmark Construction: The dataset is intentionally heterogeneous in market size and information coverage, from large English-dominant markets to smaller or emerging markets.This heterogeneity supports geographically stratified analysis but makes coverage quality and distractor difficulty variable across markets.
  • Attribute Construction: Four atomic attributes are extracted from company infoboxes: headquarters, founding year, industry, and key people.They are selected for corporate relevance and broad availability, despite heterogeneous cardinality and partial missingness.
  • Question Construction: Each fact becomes paired inductive and deductive multiple-choice questions, with semantically plausible distractors selected from nearby companies.The procedure yields approximately 15,000–17,000 questions depending on attribute availability.
  • Context Regimes: Company lead paragraphs provide standardized context under no-context, perfect-context, misleading-context, and distraction-context conditions.These regimes isolate parametric recall, evidence utilization, and robustness to incorrect or irrelevant information.
  • Models and Inference: Six proprietary and open-weight models are evaluated with fixed multiple-choice prompts and consistent decoding settings.Responses are mapped to answer options and scored for correctness.
  • Evaluation: Binary correctness is summarized as accuracy with confidence intervals, while stratification uses index-level no-context accuracy as a proxy for parametric knowledge.Indices are grouped into baseline-accuracy bins for context-specific comparisons.
  • Evaluation: Correction, misleading, and distraction rates measure prediction changes relative to the no-context baseline.The analysis also compares probabilities across conditions and models to test the five hypotheses.

5 Results

Across markets and context conditions, factual QA performance reflects uneven parametric knowledge and context sensitivity. Inductive questions are easier than deductive ones, perfect context does not remove geographic disparities, misleading context causes copying failures, and scale improves performance without eliminating structural differences.

  • Question direction: Inductive questions are consistently easier than deductive questions, which require identifying the correct entity among plausible alternatives.The asymmetry is largest for smaller models; GPT-5’s lower deductive accuracy than GPT-5 mini is reported as an observed exception.
  • No-context performance: No-context accuracy varies substantially across indices, with higher performance in large, English-dominant markets and lower performance in smaller or less-represented ones.The pattern appears across proprietary and open-weight models, while larger models preserve the same relative disparities.
  • Perfect context: Perfect context improves accuracy across models but does not eliminate geographic disparities, because gains tend to track baseline parametric knowledge.This coupling appears across attributes and models, with a modest dampening effect for larger models.
  • Misleading context: Misleading context substantially reduces accuracy as models often adopt incorrect evidence instead of resisting it.Models can perform worse with incorrect context than under parametric recall alone, although stronger parametric knowledge sometimes mitigates the effect.
  • Distraction context: Distraction context degrades performance more mildly but systematically than misleading context, with larger models and stronger-baseline indices generally more robust.Distractor difficulty may be attenuated in non-English-dominant markets because distractor selection uses an English-centric embedding model.
  • Model scale: Larger models outperform smaller ones across conditions but do not remove underlying geographic and evidence-use disparities.Scale improves baseline accuracy, helpful-context gains, and robustness to misleading or distracting evidence while preserving the qualitative structure of results.

6 Discussion

The results support an interaction between parametric knowledge and contextual evidence: retrieval improves factual QA conditionally, while misleading evidence can induce errors and larger models do not remove geographic disparities.

  • All five hypotheses are supported, linking factual QA performance to interactions between parametric knowledge and contextual evidence.
  • Retrieval is a conditional amplifier rather than a universal corrective, helping more where models are already strong and less where knowledge is sparse.
  • Robust evaluation should be stratified by geography, entity coverage, and context condition, with misleading-evidence robustness treated as a core requirement.

7 Conclusion

The paper introduces a geographically heterogeneous benchmark for factual QA over public companies and finds that retrieval narrows but does not eliminate accuracy gaps, while misleading context causes systematic errors. These effects persist as larger models improve, motivating stratified and robustness-oriented evaluation; the benchmark has stated scope limitations.

  • The benchmark uses geo-economic heterogeneity among public companies to study knowledge and retrieval in large language models.
  • Correct context narrows accuracy gaps but does not eliminate them, while misleading context induces systematic errors.
  • Retrieval is not a universal corrective, so high-stakes evaluation should be stratified and robustness-oriented.
  • The benchmark relies on Wikipedia-derived fields, atomic facts, synthetic misleading contexts, English-centric evaluation, and primarily US/European models.

A.1 Dataset Retrieval and Composition

The benchmark is assembled from Wikipedia company pages and index constituent lists, normalized into four atomic corporate attributes and paired question formats. Evaluation uses multiple-choice prompts with original, misleading, and distraction contexts constructed from company summaries.

  • Dataset Retrieval and Composition: Wikipedia index pages provide constituent lists, while company pages supply lead paragraphs and selected infobox fields; duplicate entities are merged after normalization.
  • Dataset Retrieval and Composition: The benchmark retains company name, leading paragraph, Headquarters, Founding Year, Key People, and Industry fields, excluding sparse or unstable fields.
  • Dataset Retrieval and Composition: Founding years are reduced to the first valid four-digit year, while list-valued fields are delimiter-split, standardized, and canonically sorted.
  • Dataset Retrieval and Composition: Industry values are mapped to GICS sectors, normalized, deduplicated, validated, and lexically sorted.
  • Dataset Retrieval and Composition: Location strings are converted into city-country objects, with aliases and diacritics normalized to canonical names.
  • Prompt Templates and Context Construction: Multiple-choice evaluation uses fixed answer-choice prompts, with a context-enabled variant for supplied evidence.
  • Prompt Templates and Context Construction: Perfect contexts use unchanged company summaries, whereas misleading contexts substitute another entity’s plausible attribute evidence and distraction contexts retain the incorrect company name.
  • Prompt Templates and Context Construction: Perfect-context filtering excludes 61.0% of Founding Year, 29.6% of Headquarters, and 86.0% of Key People questions.

B.3 Robustness and Coverage Diagnostics

Coverage diagnostics test whether source length, distractor similarity, and company size explain geographic performance disparities. The reported checks find that index differences persist beyond these factors, while model-query and response-parsing procedures are standardized.

  • Coverage Diagnostics: Mean Wikipedia lead length ranges from 60–158 words but does not significantly predict perfect-context accuracy (Pearson r = 0.432, p = 0.11, n = 15).
  • Coverage Diagnostics: Target-distractor similarity is higher for English-market indices (mean = 0.061) than non-English-market indices (mean = 0.046), but the difference is not significant (p = 0.24, n = 15).
  • Company-Size Robustness: Revenue quintile significantly predicts accuracy in every condition, yet index fixed effects remain jointly significant after controlling for size (p < 0.001).
  • Inference and Parsing: Models are queried individually through Bloomberg-hosted OpenAI Chat Completions endpoints, and unparseable JSON responses are excluded.

C Results

The evaluation compares factual QA across baseline, perfect-context, misleading-context, and distraction-context settings, using stratified analyses by baseline no-context accuracy. Results examine performance across attributes, indices, and model-access configurations.

  • Evaluation setup: Stratified analyses group companies into bins according to baseline no-context accuracy, with Center marking each bin’s midpoint.
  • No-context results: Figure 7 summarizes no-context performance across attributes and global equity indices.
  • Perfect context: Perfect-context accuracy is reported separately for Founding Year, Industry, Key People, and Headquarters across Tables 8–11.
  • Misleading context: Misleading Rate measures the proportion of previously correct no-context predictions that become incorrect under misleading context.
  • Context robustness: Misleading-context results cover four attributes in Tables 12–15, while distraction-context results cover the same attributes in Tables 16–19.
  • Distraction context: Distraction Rate measures the proportion of previously correct no-context predictions that become incorrect under irrelevant context.
Loading 2608.25717v1…