PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

Zhuoshi Pan, Junru Lu, Yan Qian, H. Vicky Zhao, Di Yin, Xing Sun

arXiv:2608.28478v1cs.CL

TL;DR

ElephantBench addresses whether LLMs retain divergent, long-tail factual accounts rather than only canonical answers. It builds an auditable closed-book benchmark from low-exposure documents and verified conflicts, then finds that even the strongest model recovers both accounts on only slightly more than half of questions, while scaling and reasoning improve but do not eliminate incomplete recall.

  • Problem

    Existing QA and long-tail benchmarks usually assume a single canonical answer, leaving whether parametric memory preserves different verified accounts of long-tail facts insufficiently tested.

  • Method

    ElephantBench uses a graph-based pipeline to mine low-exposure documents, identify support and conflict relationships, synthesize multi-account QA records, and validate them against documents, public sources, and human review.

  • Results

    Even the strongest model achieves 52.38% complete recall, while scaling model size and inference-time reasoning improve recall without eliminating incomplete recall across verified accounts.

  • Takeaways & Limitations

    ElephantBench provides a reproducible probe that distinguishes failure to recall a low-exposure fact from incomplete recall of its documented accounts.

  • Takeaways & Limitations

    Corpus-level frequencies are only an observational proxy for model-specific training exposure, and the benchmark does not exhaustively cover long-tail knowledge, disagreement types, or all models and configurations.

Abstract

from arXiv · show

Factual question answering (QA) typically assumes a single canonical answer, obscuring whether large language models (LLMs) retain divergent accounts of long-tail facts. To address this gap, we introduce ElephantBench, a closed-book knowledge probe comprising 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against the originating documents and authoritative public web sources and is then reviewed by human annotators. Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions, while on nearly all remaining questions it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness. Corpus analysis further shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in parametric memory. More broadly, our graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.

1 Introduction

ElephantBench addresses the gap between rare-fact recall and complete recall of divergent, verified accounts in LLM parametric memory. It introduces a traceable closed-book benchmark and a corpus-grounded analysis of exposure imbalance.

  • Conventional QA and long-tail benchmarks typically assume one canonical answer, leaving recall completeness for divergent facts untested.Knowledge-conflict benchmarks generally provide conflicting passages at inference time, so they do not test whether parametric memory preserves different accounts without evidence.
  • 1,094 traceable questions form ElephantBench, a closed-book benchmark mined from a low-exposure web corpus and reviewed by humans.Records are externally verified, source-traceable, and evaluated without source documents, retrieval, or external tools.
  • The benchmark separates retention of low-exposure facts from recall of all verified accounts instead of collapsing performance into a single accuracy score.Its metrics diagnose both whether a fact is remembered and whether every documented account is recovered.
  • A two-stage graph-based pipeline clusters knowledge points, extracts entities, and uses LLMs to identify support and conflict edges efficiently.The design narrows candidate comparisons and is intended to transform low-exposure corpora into reusable knowledge probes.
  • Exposure to the prevalent account helps fact recall, whereas exposure to the less prevalent account is more closely associated with complete recall.This corpus-grounded relationship connects exposure distributions with memory completeness and can inform data curation.

2 Related Work

Prior benchmarks separately study rare-fact recall, evidence-conditioned conflicts, and answer disagreement, but ElephantBench tests naturally occurring source disagreement in closed-book parametric memory. Its items preserve multiple verified accounts for the same subject–attribute pair without imposing one as the sole ground truth.

  • Long-tail QA benchmarks test rare-fact recall but generally score a single canonical answer rather than complete recall of divergent accounts.
  • Knowledge-conflict benchmarks usually provide conflicting evidence at inference time and evaluate evidence-conditioned detection, reconciliation, or multi-hop reasoning.
  • Multi-answer and context-sensitive tasks focus on ambiguity, time, location, or relations among candidate answers.
  • ElephantBench withholds source evidence and tests whether parametric memory preserves naturally occurring, independently verified accounts for the same subject–attribute pair.

3 Tracing the Elephant

ElephantBench constructs auditable multi-account QA records from low-exposure documents by locating support and conflict relationships in a document graph. The resulting closed-book evaluation measures failed, partial, and complete recall of verified answers.

  • 3.1 Task Formulation: Dlow contains documents removed by quality filtering, preserving rare facts that standard filtering may discard and making low-exposure memory failures easier to detect.
  • 3.2 Document Graph Construction: Verified conflict edges seed local subgraphs containing both endpoints and their support neighbors for QA synthesis.
  • 3.1 Task Formulation: Each QA item pairs a question q with a verified answer set A containing k ≥2 answers, each traceable to an audited source hidden from the evaluated model.Matched named-entity and clue-based questions share the same answer set.
  • 3.2 Document Graph Construction: Documents in Dlow become graph nodes, with support edges for agreeing accounts and conflict edges for incompatible accounts about the same subject–attribute pair.The graph is an offline construction and audit structure and is never shown to evaluated models.
  • 3.2 Document Graph Construction: Knowledge-point clustering and named-entity retrieval generate candidate document pairs, avoiding exhaustive comparison of all document pairs before relation classification.
  • 3.4 Closed-Book Evaluation and Scoring: Closed-book scoring classifies responses as complete, partial, or failed recall, and reports conditional completeness among answers that recover at least one verified answer.

4 Experiments

ElephantBench evaluates whether models recall all verified accounts of long-tail facts, distinguishing complete recall from merely recalling one answer. Across 32 models, scale and reasoning improve recall but leave substantial incompleteness.

  • Evaluation Metrics: The evaluation separates complete, partial, and failed recall, with K = C/(C + P) measuring completeness among responses recalling at least one verified answer.Higher C/K and lower P/F are better.
  • ElephantBench Curation: 1,094 validated QA pairs remain after graph-based sampling, verification, human review, and deduplication from 4,127 conflict edges.The benchmark spans 22 fields; News, public information accounts for 39.4%.
  • Evaluation Results: 52.38% is the highest complete recall rate, achieved by Kimi-K3; Gemini-3.1-Pro and GPT-5.5 reach 50.37% and 50.18%, respectively.Failed recall rates for the three strongest models range from 2.19% to 2.65%, while partial recall ranges from 45.25% to 47.44%.
  • Effect of Model Scale: 5.48% average complete recall for models under 10B parameters rises to 52.38% for the best model above 1T parameters.Within Qwen3.5, scaling from 2B to 397B raises C from 1.65% to 32.27% and reduces F from 81.35% to 8.50%, while P rises from 17.00% to 59.23%.
  • Effect of Reasoning Mode: 13.99 and 12.89 percentage-point gains in complete recall occur when reasoning is enabled for GPT-5.6-Sol and GPT-OSS-120B, respectively.For Qwen3.5, reasoning raises C by 0.73–4.39 points at 9B–397B but lowers it by 0.64 and 0.37 points at 2B and 4B.
  • Open-Weight and Proprietary Models: Kimi-K3, an open-weight model, ranks first despite proprietary systems occupying most leading positions.Two of the three best-performing systems are proprietary.

5 Analysis

Analysis links recall completeness to source exposure, knowledge domain, model complementarity, question formulation, and perplexity. Minority-side exposure is associated with more complete recall, while domain and evaluation patterns reveal substantial variation.

  • Factors Affecting Model Performance: A larger Nmaj −Nmin gap generally raises partial recall, whereas larger Nmin is associated with higher complete recall and lower partial recall.A one-standard-deviation increase in majority-side exposure is associated with +14.18 percentage points in P and −10.17 in F; the same minority-side increase is associated with +15.13 in C and −15.41 in P.
  • Performance across Knowledge Domains: Complete recall ranges from 38.7% in People, Organizations, and Events to 6.2% in Consumer Products and Services.Government and Civic Affairs reaches 12.0%, while partial recall dominates the two lowest-performing domains at 55.6% and 62.1%.
  • Cross-Model Oracle Coverage: An oracle pool raises complete recall from 52.4% with Kimi-K3 to 81.2% across 32 configurations, saturating after 23 configurations.The pool reduces failed recall from 2.4% to 0, but 18.8% of questions remain partial because no configuration recovers all verified accounts.
  • Question Formulation: Named-entity and clue-based questions differ by only 0.65 percentage points across eight frontier models, with no significant difference.Named-entity questions perform better for four models and clue-based questions for the other four.
  • Evaluation Proxy: For Qwen3.6-27B, higher conditional perplexity bins contain fewer complete responses and more failures.The alignment between perplexity bins and graded outcomes supports conditional PPL as a lower-cost evaluation proxy.

6 Conclusion

ElephantBench reveals that single-answer QA can hide incomplete recall of divergent long-tail accounts. Complete recall remains limited even for strong models, and minority-account exposure is more closely associated with completeness.

  • Conclusion: Even the strongest model recalls all verified accounts for only slightly more than half of the questions.Complete recall is more closely associated with exposure to the less prevalent account than with overall fact frequency.
  • Conclusion: Scaling model size and increasing inference-time reasoning improve long-tail fact recall but do not eliminate incomplete recall across verified accounts.The benchmark separates failure to recall a low-exposure fact from failure to recall all documented accounts.

Limitations

The study’s exposure analysis relies on corpus frequencies rather than model-specific training data, and its benchmark and model coverage are necessarily bounded. These limits constrain how broadly exposure associations and evaluation findings should be generalized.

  • Limitations: Corpus-level frequencies are only an observational proxy because evaluated models generally do not disclose their pretraining corpora.Exposure-related findings should therefore be interpreted as associations rather than direct measurements of model-specific training data.
  • Limitations: The 1,094-question benchmark across 22 knowledge fields cannot exhaustively represent long-tail knowledge or the diversity of source disagreements.The authors suggest expanding scale, domain coverage, and disagreement types.
  • Limitations: Evaluation of 32 representative open-weight and proprietary models excludes later releases, every model size, and every reasoning configuration.The reported model comparisons therefore do not cover the full available model and configuration space.

Ethics Statement

ElephantBench’s retained accounts are externally supported and screened for harmful content, while its construction pipeline uses structured model outputs and reduces pairwise comparison cost through clustering. The benchmark is intended for factual-knowledge research and model evaluation.

  • Ethics Statement: Each retained account is independently supported by authoritative public web evidence, though source error or misinformation may remain possible.Human review removes unnecessary personal information, sensitive attributes, harmful allegations, and objectionable content.
  • Benchmark Construction: The construction prompts cover knowledge-point labeling, support and conflict induction, QA synthesis, and external verification.At every stage, the model receives relevant full documents and returns structured outputs for the next stage.
  • Candidate-Pair Reduction from Knowledge-Point Clustering: Knowledge-point clustering removes at least 93.4% of the naive pair space and reduces pairwise LLM judgments by at least 15.2×.This comparison isolates the computational gain from clustering before pairwise relation judgment.

C Evaluation Settings

ElephantBench evaluates models in a closed-book setting: models answer from parametric memory, and a judge semantically grades whether they recover all verified accounts. Human annotation checks support the judge’s reliability.

  • Target-model and judge inputs: Models receive only the benchmark question and a closed-book instruction, without source documents, reference answers, or scoring rubrics.The target-model prompt permits reporting multiple values when remembered but prohibits search, browsing, and external tools.
  • Target-model and judge inputs: The GPT-5.6-Sol judge receives the question, verified answers, and model response, but neither source documents nor external tools.It applies a four-step semantic-coverage rubric to the recorded response.
  • Judge reliability: Three graduate annotators independently evaluated frontier-model responses using the same three-way criteria as the judge.Table 2 reports exact agreement of 90.13–93.36% and Cohen’s κ of 0.815–0.877.
  • Judge reliability: The judge’s model ranking matches the ranking from human annotations.This supports using the judge for the benchmark’s semantic evaluation.

E Robustness to Sampling Temperature

The section examines robustness to sampling temperature and relates exposure imbalance to recall outcomes. It also shows that disagreement mechanisms differ substantially in difficulty and that reasoning effects can vary by model scale.

  • Robustness to sampling temperature: All 1,094 questions were evaluated under T=0 and T=1 while checkpoint, prompt, reasoning configuration, output budget, and other sampling parameters remained fixed.The same GPT-5.6-Sol judge scored every condition.
  • Conflict mechanisms: 38.9% complete recall and 13.6% failed recall make measurement conflicts easiest, while temporal and reporting conflicts have complete recall of 18.3% and 18.0%.Partial recall reaches 56.9% for temporal conflicts and 56.5% for reporting disagreements.
  • Conflict mechanisms: 4.12 percentage points is the average reasoning gain for temporal conflicts across 22 matched comparisons.Positive gains occur for 15 pairs, with Wilcoxon p = 0.011; gains for other mechanisms range from 1.27 to 1.94 points and are not statistically significant.
  • Exposure analysis: Larger dominant-side exposure is associated with more partial recall, whereas larger minority-side exposure is associated with higher complete recall and lower partial recall.The regression standardizes log-transformed supporting-document counts and interprets coefficients as percentage-point changes per standard deviation.
  • Exposure analysis: The exposure coefficients represent observational associations rather than causal effects.Each side’s association is estimated while holding the other side’s exposure fixed.

I Illustrative Cases across Knowledge Domains

The examples show that model recall varies across knowledge domains and that shared blind spots can persist across all evaluated models. Conditional perplexity tracks these recall outcomes as an auxiliary evaluation signal.

  • Domain-level variation: Models more often achieve complete recall for distinctions such as birth versus baptism, alternative pandemic-mortality estimates, and global versus regional scope.These patterns are aggregated over 32 primary model configurations.
  • Domain-level variation: Questions about fine-grained prices or event dates produce almost exclusively partial or failed recall.This contrasts with cases where the disagreement involves broader conceptual or scope distinctions.
  • Shared blind spots: Four representative cases show that none of the 32 evaluated models recovers all verified accounts.The cases involve a corrected vote tally, cause-of-death reports, prison-sentence interpretations, and regional release-date scope.
  • Conditional perplexity: Lower conditional PPL corresponds to complete recall, while partial and failed recall show progressively higher PPL.The loss is computed over the two verified answer spans while the full response remains autoregressive context.
  • Conditional perplexity: The authors propose conditional PPL as a computationally efficient proxy for generation-based evaluation.This conclusion follows from outcome-conditioned PPL patterns across four models.
  • Benchmark curation: The benchmark retains only items whose answers are source-supported, independently verified, and free of answer leakage or question–fact mismatch.Final review was conducted by three author-annotators.
Loading 2608.28478v1…