PaperlayerView exact revision on arXiv

Source-linked AI summary

What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models

Dustin Wright, Sarah Masud, Jared Moore, Srishti Yadav, Maria Antoniak, Peter Ebert Christensen, Chan Young Park, Isabelle Augenstein

arXiv:2510.04226v7cs.CLcs.AIcs.CYcs.IRcs.LG

TL;DR

The paper addresses the lack of systematic evidence on LLM claim diversity across time and cultural contexts, despite LLMs’ growing use for knowledge-intensive tasks. The study quantifies claim diversity with Hill-Shannon diversity across 27 LLMs, 155 topics, and 12 countries, producing a dataset of 70M semantically clustered claims. Epistemic diversity has increased, but all studied systems remain less diverse than search, RAG can improve diversity, larger models are less diverse, and parametric claims reflect English more than local languages.

  • Problem

    The paper addresses the lack of systematic evidence on LLM claim diversity across time and cultural contexts, despite LLMs’ growing use for knowledge-intensive tasks.

    Source: §1 · p. 1

  • Method

    The study quantifies claim diversity with Hill-Shannon diversity across 27 LLMs, 155 topics, and 12 countries, producing a dataset of 70M semantically clustered claims.

    Source: §1 · p. 2

  • Results

    Epistemic diversity has increased, but all studied systems remain less diverse than search, RAG can improve diversity, larger models are less diverse, and parametric claims reflect English more than local languages.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support focusing diversity measurement and improvement on the breadth of communities represented, while diversified RAG sources can increase diversity.

    Source: §7 · p. 9

    The study’s RAG setup is intended to simulate real-world use, but implementations vary, so its utility should be benchmarked across several implementations.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as primary knowledge sources, yet their epistemic diversity - defined as the diversity of real-world claims in their outputs - has never been measured. Low epistemic diversity would pose a risk of knowledge collapse as homogeneous LLMs mediate a shrinking in the range of accessible information over time. The dominant paradigm is that overall LLM diversity is low, but this is always with respect to a single point in time, with no reference baseline or consideration for variation across countries. We address this gap in knowledge by performing the first systematic study of epistemic diversity in LLMs across time and cultural context, testing 27 LLMs on 155 topics covering 12 countries, resulting in 1.7M responses and 70M individual claims. We find that epistemic diversity has increased substantially over the past three years, a positive counter to recent diversity pessimism. However, despite progress, we find that every system is less diverse than a search baseline. This gap is not uniform: RAG can improve diversity, while large models are counterintuitively less diverse than smaller ones. Moreover, LLM parametric knowledge systematically reflects English over local-language knowledge for country specific topics. Together, these results demonstrate that while progress on epistemic diversity is tangible, it is insufficient and unevenly distributed.

1 Introduction

The paper introduces an empirical measure of epistemic diversity and examines LLM claims across models, time, and country-specific topics. Diversity has risen, but remains below search and varies with generation setting, model size, and whose knowledge is represented.

1 Introduction

  • The study analyzes 1.7 million responses and 70 million claims from 27 LLMs across 155 topics and 12 countries.Responses are decomposed into claims, clustered by semantic equivalence, and measured with Hill-Shannon diversity.Source: §1 · p. 2
  • Epistemic diversity is generally increasing, yet every studied system remains less diverse than a weak traditional search baseline.The authors describe the remaining gap as especially concerning amid shifts toward AI-exclusive search designs.Source: §1 · p. 2
  • RAG has a statistically significant positive impact on diversity, while model size has a statistically significant negative impact, with smaller models generating more diverse knowledge.RAG benefits vary by country; the USA benefits more, which the authors link to greater diversity in its RAG sources.Source: §1 · p. 2
  • For country-specific topics, LLM parametric knowledge reflects English-language claims more than local-language claims.The authors conclude that progress is tangible but insufficient and unevenly distributed.Source: §1 · p. 2

2 Background

The paper situates its study at the intersection of LLM homogenization and concerns about knowledge collapse. It distinguishes its focus on broad claim diversity across time and countries from prior work on narrower forms of diversity.

Knowledge Collapse

  • The paper connects LLM use in knowledge-intensive tasks with a societal concern: reduced diversity in AI outputs could also reduce diversity in collective knowledge.Source: §2 · p. 2

LLM Homogenization

  • LLM homogenization has been studied across lexical, stylistic, semantic, creative, conceptual, recommendation, coding, and perspective diversity.The paper positions its work closest to perspective diversity, extending the focus to semantically equivalent classes of claims in open-ended responses.Source: §2 · p. 2

Knowledge Collapse

  • Knowledge collapse is framed as a narrowing of ideas that may affect sources such as Wikipedia, minoritized knowledge, scientific discovery, political discourse, and writing ideation.The paper assesses this risk by comparing LLM outputs across models, sources, and time with traditional knowledge systems.Source: §2 · p. 3

3 Problem Setup and Notation

The paper operationalizes epistemic diversity as diversity in the distribution of claims generated for open-ended prompts. Its notation represents each model-topic sample as a categorical distribution over meaning classes.

3 Problem Setup and Notation

  • Epistemic diversity is measured over claims generated in response to open-ended prompts about different topics.The methodology decomposes responses into claims, groups semantically equivalent claims into meaning classes, and measures their Hill-Shannon diversity.Source: §3 · p. 3
  • For a model m and topic t, each meaning class has an empirical frequency and a probability given by its relative frequency in the claim sample.Source: §3 · p. 3
  • P_mt denotes the categorical distribution whose outcomes are the meaning classes and whose probabilities are their class probabilities.Source: §3 · p. 3

4 Data Collection

The data pipeline elicits responses, decomposes them into claims, and clusters semantically equivalent claims into meaning classes. The authors evaluate decomposition and clustering choices before constructing the claim distributions.

4 Data Collection

  • The data collection pipeline generates free-text responses, decomposes them into claims, and clusters semantically equivalent claims to calculate the distribution.Source: §4 · p. 3

4.1 Generation and Decomposition

  • The study samples 200 information-seeking templates from 479 selected templates and uses each sampled template for every topic.Llama 3.1 70B decomposes responses in non-overlapping three-sentence chunks, retaining claims explicitly relevant to the topic.Source: §4.1 · p. 3
  • The final decomposition prompt is P3, selected to prioritize claim fidelity over missing claims.The authors evaluate decomposition prompts using an LLM-as-a-judge setup.Source: §4.1 · pp. 3–4

4.2 Clustering

  • Claim clustering uses mutual entailment in a greedy bootstrap process, narrows comparisons to similar claims, and then splits larger clusters with DBSCAN.Source: §4.2 · p. 4
  • 4.17 cluster quality score and 13.79% missing rate indicate generally strong clustering with a relatively small proportion of unclustered claims missed.The evaluation uses automated labels for cluster quality and whether claims should have been clustered.Source: §4.2 · p. 4

5 Measuring Epistemic Diversity

Hill-Shannon diversity measures claim diversity while accounting for how claims are distributed, enabling meaningful comparisons across settings. The method adjusts comparisons for unequal coverage because equally sized samples can miss rare claims unevenly.

5 Measuring Epistemic Diversity

  • Hill-Shannon diversity accounts for claim frequencies and supports meaningful comparison across settings, unlike richness and entropy alone.It is the exponentiated entropy of the categorical distribution and balances emphasis between frequent and rare claims.Source: §5 · p. 4
  • Under a uniform distribution, Hill-Shannon diversity is maximized and can be interpreted as the effective number of meaning classes.A distribution with Hill-Shannon diversity 2K̂ is effectively twice as diverse as one with diversity K̂.Source: §5 · pp. 4–5
  • Equal-sized claim samples can overestimate a lower-diversity model relative to a higher-diversity model because rare claims take longer to uncover.The higher-diversity model may reveal new meaning classes as more claims are sampled, while the lower-diversity model may already be well characterized.Source: §5 · p. 5
  • The method estimates coverage from meaning-class frequencies, then rarefies each model’s claims to the minimum coverage achieved across models.This accounts for differences in how completely each sample covers its underlying distribution.Source: §5 · p. 5

6 Empirical Results

Across 27 LLMs and 155 topics, epistemic diversity generally rose over time but remained below search. The results also vary by generation setting, model size, and country representation.

6.1 Models, Settings, and Topics

  • The study tests 27 LLMs across 155 topics spanning 12 countries, producing 1.7M responses and 70M decomposed claims.A 50,000-claim calibration found that 98.4% of mutually entailed matches appeared among the first six candidates; claim frequencies also fell rapidly across the top clusters.Source: §6.1 · pp. 5–6, Figure 2 · p. 6

6.2 Epistemic Diversity Across Time

  • At least 18.7% more diverse, search exceeds GPT-5, the best model (average HSD 3110 vs. 2621); every studied model trails this weak baseline.The authors describe the comparison as a lower bound: search used up to 40 results per topic and was only rarefied when coverage exceeded the minimum achieved by a system.Source: §6.1 · p. 5, Figure 3 · p. 6, §6.2 · p. 7
  • With the exception of Qwen, epistemic diversity increased over time; Gemma 3 and GPT-5 had the largest single-generation gains.Figure 3 plots topic-averaged HSD against release date, connecting models of approximately the same size across versions.Source: Figure 3 · p. 6, §6.2 · p. 7

6.3 Epistemic Diversity Across Settings

  • RAG and Search yield significantly more diverse outputs than parametric IFT, at least for the tested topics.The comparison uses a linear mixed-effects model with generation setting as a fixed effect and model as a random effect.Source: Table 2 · p. 7, §6.3 · p. 7

6.4 Epistemic Diversity Across Models

  • Model size has a statistically significant negative effect on HSD, with smaller models generating more diverse claims.The authors hypothesize that greater model capacity increases memorization, narrowing output-token distributions; they note that performance gains from larger models may come with reduced diversity.Source: Table 3 · p. 7, §6.4 · p. 8

6.5 Epistemic Diversity Across Countries

  • RAG gains track search-source diversity (r=0.73, p < 0.01, N=13), and are larger for the USA, India, Russia, France, China, and general topics.The authors suggest US-region retrieval may under-represent topic-specific information from some countries.Source: Figure 4 · p. 8, §6.5 · p. 8
  • Five of eight countries show a significant English-over-local representation gap, and none shows significantly greater local-language representation.Representation for USA-specific topics is significantly greater than for every other country.Source: Figure 5 · p. 8, §6.5 · p. 9

7 Discussion and Conclusion

The study finds that epistemic diversity has improved, but important gaps remain: LLM outputs are less diverse than web search, and country-specific claims favor English over local-language knowledge.

7 Discussion and Conclusion

  • All models remain less diverse than traditional web search, while the study’s results show epistemic diversity has improved in recent years.The authors conclude that relying on LLMs as a primary knowledge source means accepting less diverse information than a simple web search could reveal.Source: §7 · p. 9
  • Scaling alone does not improve epistemic diversity and can harm it, while RAG with diversified sources can increase diversity.The choice of RAG database is critical for country-level representation, which correlates strongly with source diversity.Source: §7 · p. 9
  • For country-specific topics, generated claims reflect English-language claims more than local-language claims.The authors call for measuring and improving diversity for the breadth of communities that use LLMs.Source: §7 · p. 9

Limitations

The study’s scope and measurement choices constrain how broadly its findings can be generalized, including its topic selection, clustering method, and RAG setup.

Limitations

  • Manual topic selection may limit generalizability, and the country-level measure of whose knowledge is represented is coarse-grained.Some topics were selected from an outsider perspective; the study also excludes very long-tail and niche topics through its Wikipedia content-rating criterion.Source: Limitations · p. 9
  • The study uses only 200 prompt templates because of computational constraints, limiting prompt coverage.The authors note that broader coverage could be achieved with more templates.Source: Limitations · p. 9
  • The decomposition and clustering algorithms are imperfect, and strict semantic-equivalence clustering may miss useful partial overlap between claims.The authors use a validated, more efficient BERT-based approach instead of stronger NLI methods, accepting some accuracy cost.Source: Limitations · p. 10
  • The selected RAG setup is intended to simulate a real-world scenario, but actual implementations vary; diversity also needs context alongside factuality and relevance.The authors suggest systematically benchmarking multiple RAG implementations.Source: Limitations · p. 10

A Full Evaluation Details

The appendix evaluates claim decomposition and clustering, specifies the prompts and clustering procedure, and lists the topics used in the study.

A.1 Decomposition

  • P3 was chosen for final decomposition to prioritize claim fidelity; human annotations and G-Eval were used to evaluate prompt variants.The annotators showed Pearson correlations of 0.74 for claim quality and 0.58 for missing claims; G-Eval correlations were 0.60 and 0.68, respectively.Source: §A.1 · p. 13

A.2 Clustering

  • The clustering evaluation found generally cohesive clusters, with a 4.17 cohesion score and a 13.79% missing-singleton rate.The semantic-similarity-only DBSCAN comparison had a lower cohesion score of 3.48.Source: §A.2 · p. 13

B Prompts

C Clustering Algorithm

  • The clustering procedure assigns decomposed claims to semantically equivalent meaning classes and breaks up large clusters with DBSCAN.The algorithm compares a claim with similar prior claims using mutual entailment before assigning it to a cluster.Source: C Clustering Algorithm · p. 14, D Topics · p. 15

D Topics

  • The listed topics include country-specific people and historical events, alongside general subjects such as democracy, climate change, and artificial intelligence.The country-specific lists include topics associated with the USA, Argentina, Ethiopia, and other countries.Source: D Topics · pp. 14–15

E Comparison to Semantic Similarity Only

The section argues that semantic similarity is not a reliable stand-in for whether two claims convey the same knowledge. Similarity-only clustering can combine distinct or opposing claims and thereby underestimate epistemic diversity.

E Comparison to Semantic Similarity Only

E.1 LLM Tropes (Wright et al., 2024)

The section compares semantic-similarity clustering with the paper’s claim-clustering approach. The similarity-based approach yields fewer missed singletons but lower cluster cohesion.

E.1 LLM Tropes (Wright et al., 2024)

  • Following Wright et al. (2024), the comparison embeds decomposed claims and uses DBSCAN to group them into classes.The selected parameters were eps 0.2 and minimum size 2, chosen for a cluster count comparable to the semantic-equivalence approach; tuning could improve results.Source: §E.1 · p. 16
  • 3.48 vs 4.17 cluster cohesion and 0.4% vs 13.8% missing singleton rate show higher recall alongside lower cohesion for semantic-similarity clustering.The comparison is with the paper’s algorithm; the authors describe the result as a clustering-quality tradeoff.Source: §E.1 · pp. 16–17
  • The evaluation uses the same LLM-as-a-judge assessment to compare cluster cohesion and claims missed from clusters.Source: §E.1 · p. 16

E.2 Artificial Hivemind (Jiang et al., 2025)

This appendix compares cosine similarity between response embeddings with the paper’s claim-based measure of epistemic diversity. The cosine-based view misses several patterns visible in the claim-based results and is harder to interpret.

E.2 Artificial Hivemind (Jiang et al., 2025)

  • The comparison applies average pairwise cosine similarity to sentence embeddings of each model’s original responses within each topic.This reproduces the procedure used by Jiang et al. (2025), with a different embedding model from the original paper.Source: §E.2 · p. 17
  • Cosine similarity shows little change over time except for Gemma 1 and GPT-5, no RAG or model-size differences, and search more diverse than LLMs.Figure 8 plots average within-topic pairwise cosine similarity by model release date; lower scores indicate greater diversity, but trends seen with epistemic diversity are lost.Source: §E.2 · p. 17, Figure 8 · p. 18
  • The cosine axis is less interpretable than claim-based Hill-Shannon diversity, which supports a more concrete comparison of GPT-5 and GPT-4o outputs.The paper reports that cosine similarity decreased by 0.2, while effective claim diversity for GPT-5 was more than 5× greater than for GPT-4o on its topics.Source: §E.2 · p. 17

F Model Details

  • Table 4 describes four model families and distinguishes small, medium, and large model sets.Source: F Model Details · p. 17

G InfoGap Details

  • For InfoGap, the study decomposes English and local-language Wikipedia pages, retrieves multilingual candidate matches, and uses GPT-5 to judge matches to generated English claims.RAG-generated claims are excluded to avoid including Wikipedia text in the generation context.Source: G InfoGap Details · p. 17

H Generation Details

  • The study retrieves 40 Google results per topic and caps generation at 2,100 tokens per prompt variation, with up to 1,000 extra context tokens for RAG.Across 155 topics, the 6,200 pages average 12,836 tokens and have a median of 2,132 tokens.Source: H Generation Details · p. 17

I Artifacts Statement

J Use of AI in Research Statement

Loading 2510.04226v7…