Source-linked AI summary
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs
Muhammed Saeed, Simon Razniewski
TL;DR
Fixed-question benchmarks probe only experimenter-selected slices of LLM knowledge, while fluent long-form prose remains largely unmeasured. LLMPEDIA materializes about 1.3M articles from three models’ parametric memory without retrieval and audits sampled atomic claims. On a uniform random sample, 68.4% of GPT-5-mini claims were supported, with 30.5% insufficient and 1.2% refuted, and the live site exposes these results through stable, browsable comparison views.
Problem
Fixed-question benchmarks probe only experimenter-selected slices of LLM knowledge, leaving fluent long-form prose unmeasured.
Method
LLMPEDIA materializes about 1.3M articles from three models’ parametric memory without retrieval and audits sampled atomic claims against external evidence.
Results
68.4% of GPT-5-mini claims were supported, 30.5% insufficient, and 1.2% refuted on a uniform random sample.
Takeaways & Limitations
The live encyclopedia makes claim-level factuality, knowledge divergence, and framing effects browsable through stable URLs and comparison views.
Takeaways & Limitations
Single-pass generation samples what a model surfaces rather than the full extent of its parametric knowledge, and an omitted fact is not evidence the model lacks it.
Abstract
from arXiv · showhide
Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ~1.3M articles from three model families' parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world's largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net
1 Introduction
LLMPEDIA addresses the availability bias of fixed-question benchmarks by materializing and auditing model-generated encyclopedia prose, then making its claims and divergences browsable. Its central result is that unverifiability dominates refutation, while stable URLs and comparison views expose what single-score leaderboards hide.
- Motivation: Fixed-question benchmarks inspect only experimenter-selected subjects and questions, leaving broad regions of fluent long-form model knowledge unmeasured.This limitation includes knowledge that may be weak, strong, or simply unverifiable.
- Approach: LLMPEDIA materializes an encyclopedia from parametric memory alone and audits sampled atomic claims to make latent knowledge browsable and fact-checkable.The demonstrator presents this as an interactive artifact complementing the companion paper’s methodology and findings.
- Findings: 68.4% of GPT-5-mini claims were supported, 1.2% refuted, and 30.5% unverifiable on a uniform random sample.The site colors these verdicts and places evidence beside each audited claim.
- Interactive resource: Five live views support link exploration, claim-level factuality inspection, cross-model comparison, persona comparison, and guided topic analysis.Each entity has a stable, shareable URL, and the site is presented as a static million-page resource.
- Interactive resource: Cross-model and cross-persona views expose knowledge divergence and framing effects that a single-score leaderboard hides.All prompts, articles, verdicts, and code are live at llmpedia.net.
2 LLMPEDIA in One Minute
LLMPEDIA grows a browsable encyclopedia from model-generated articles and verifies sampled claims against external evidence. Its pipeline preserves the model’s own associations while distinguishing support, contradiction, and insufficient evidence.
- Article generation: A single seed entity initiates breadth-first article expansion through sanitized, normalized, filtered, and deduplicated wikilinks.Each surviving entity is enqueued once under its first parent, so the browsed structure follows the model’s associations.
- Claim verification: Claim verification decomposes audited articles into atomic claims and checks them against evidence retrieved using only the subject name.Claims receive supported, refuted, or insufficient labels, separating contradiction from silence.
- Claim verification: The public audit covers 2,010 subjects and 20,092 claims rather than the full million-article corpus.This stratified audit makes per-claim verification feasible at million-article scale.
3 Corpus and Numbers a Visitor Can Check
LLMPEDIA materialized roughly 1.3M articles across three model corpora and audits claims using stratified sampling. Its visitor-checkable result is a 68.4% true rate, with unverifiability dominating falsehood among non-true outcomes.
- Corpus: ∼1.3M articles span GPT-5-mini, DeepSeek-V3.2, and Llama-3.3-70B, including general-domain and topic-focused corpora.GPT-5-mini reaches ∼1M general-domain articles, while DeepSeek-V3.2 and Llama-3.3-70B reach ∼120K each.
- Results: 68.4% is GPT-5-mini’s true rate on a uniform random sample.The same sample reports 30.5% unverifiable claims and 1.2% refuted claims.
- Results: 30.5% unverifiability is the dominant non-true outcome, exceeding the 1.2% refuted rate.The audit distinguishes unverifiability from falsehood rather than combining them.
- Audit: Claims are audited on a stratified sample rather than across the full corpus.Table 1 explicitly distinguishes the audit sample from the full corpus.
4 Web Provision and Access
LLMPEDIA is a static site that serves roughly 1M HTML articles with client-side search and stable URLs. Its pages expose genealogy and expansion-funnel provenance for browsing and citation.
- Access: ∼1M articles render as plain HTML without a server-side database, while search uses a compact in-browser inverted index.This architecture is intended to keep the site responsive at million-article scale.
- Access: Each entity has a stable URL in the form llmpedia.net/<model>/<Subject>.html for deep linking and citation.Visitors can reach entities through seed links, search, or direct URLs.
- Provenance: The Article Genealogy panel shows bfsParent and bfsLayer, while the Expansion Funnel reports how outbound links flowed.These provenance views are available on each page.
5 Demonstration Experience
The demonstrator lets visitors browse model-generated encyclopedias, inspect claim-level evidence, compare models and personas, and drill into controlled topic runs. These views expose factuality, knowledge divergence, framing effects, and generation-path boundaries that fixed benchmarks do not show.
- 5.1 Link-Based Knowledge Exploration: Visitors navigate model-generated entities through searchable, linked pages and genealogy panels that trace associations and parent paths back toward the seed.Pages expose construction provenance through bfsParent and bfsLayer, while outbound links support graph exploration.
- 5.2 Inspecting Factuality at the Claim Level: Claim-level audits label atomic claims supported, refuted, or unverifiable and let visitors inspect the evidence behind each verdict.The audit covers 2,010 subjects and 20,092 claims, while increasing BFS depth lowers true rate from 94.0% to 56.0% as unverifiability rises.
- 5.3 Cross-Model Analysis: Cross-model pages reveal different outlines, infobox schemas, BFS paths, and entity coverage for the same subject across GPT-5-mini, DeepSeek-V3.2, and Llama-3.3-70B.Only 7.3% of subjects appear in all three corpora, with entity Jaccard 0.17–0.22.
- 5.4 Persona Comparison: Persona comparison displays the same subject under left-leaning, scientific-neutral, and conservative framings, with classifier-highlighted lexical matches and per-column counts.The Malay people example shows 13 left-lexicon hits versus 2 in the neutral column, while factual precision remains unchanged.
- 5.4 Persona Comparison: Persona effects alter article emphasis more than correctness: 37 significant effects appear in expected directions, while precision changes by no more than 3.6 pp within a cell.On neutral and control topics, the contested axes largely collapse to 6 significant shifts versus 37.
- 5.5 Topic Analysis: A Guided Drill-Down: Topic analysis crosses fixed seed lists with topic, persona, and model selections, producing controlled small encyclopedias that isolate each axis.Visitors drill down in three clicks: topic, then persona, then model.
6 Related Work
Prior work typically probes selected facts or materializes triples, while LLMPEDIA extends parametric-memory materialization to readable, discourse-level encyclopedia articles.
- LAMA and MMLU measure factual knowledge only through experimenter-selected questions.
- Recursive elicitation and GPTKB established materialization approaches for exposing model knowledge beyond fixed probes.
- LLMPEDIA extends materialization from triples to discourse-level articles that users can read.
- Unlike retrieval-grounded STORM, LLMPEDIA is purely parametric; Grokipedia operates at scale but discloses no methodology.
7 Conclusion
LLMPEDIA materializes encyclopedia content from model parametric memory and audits sampled claims, revealing that unverifiability dominates falsehood among non-true outcomes.
- LLMPEDIA materializes an encyclopedia from three model families’ parametric memory and audits a sampled subset claim by claim.
- Unverifiability, not falsehood, is the dominant failure revealed by the audit.
8 Limitations
The study’s conclusions are bounded by single-pass generation, sampled and judge-dependent verification, changing evidence, unequal model scale, entity ambiguity, and a narrow proposition-level metric.
- Single-pass, temperature-0 generation samples surfaced knowledge rather than the full extent of a model’s parametric knowledge.
- Verification covers 2,010 subjects and 20,092 claims rather than all ∼1.3M articles, and residual judge error remains.
- The January–March 2026 snapshot may change as sources, models, and Wikipedia evolve, while 28.2% of frontier subjects lack usable web evidence.
- GPT-5-mini has ∼1M articles versus ∼120K for each open-weight model, making deep-hop and long-tail analyses GPT-5-mini specific.
- Recursive expansion can produce hallucinated or conflated subjects, so page presence does not establish existence or accuracy.
- Distinct entities sharing an unqualified surface form may merge before context-aware, sense-specific linking resolves them.
- The metric measures evidence support for individual propositions, not coherence, neutrality, completeness, salience, or omission.
9 Ethics Statement
LLMPEDIA is presented as an auditing artifact rather than a reference source, with explicit safeguards and boundaries around accuracy, profiling, persona interpretation, and reuse.
- LLMPEDIA probes what models surface from parametric memory and identifies weak, unverifiable, or wrong knowledge without treating benchmark saturation as broad reliability.
- LLMPEDIA is an AI-generated research and auditing artifact, not a reference encyclopedia, and users should consult authoritative sources for consequential domains.
- The project excludes detected private information where possible and is not intended for profiling individuals or decisions about them.
- Persona outputs are controlled framing interventions, not the authors’ views, and may reproduce ideological or cultural biases in training data.
- Prompts, articles, verdicts, and code are released under CC BY 4.0 with attribution and AI-generation marking requirements.