PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking News Out of the Filter Bubble: Generative AI Search Diversifies Collective Attention and Raises Shared Information Consumption

Heeseung Andrew Lee, Dokyun Lee, Gwanhoo Lee, Dongwon Lee

arXiv:2609.38946v1cs.CYcs.HCcs.IRcs.MM

TL;DR

Whether AI-answer search narrows news exposure and weakens shared information is consequential because shared topics support public deliberation. In a randomized field experiment with 37,561 Washington Post readers using the same archive, shared information consumption rose 69%, even as attention became less concentrated and shifted toward less-popular topics.

  • Problem

    The paper examines whether AI-answer search narrows news exposure and weakens shared information, a consequential concern because shared topics support public deliberation and consensus.

    Source: Introduction · p. 1

  • Method

    A 38-day randomized field experiment with 37,561 Washington Post readers compared conventional search with the same results plus AI answers citing articles from the same archive.

    Source: Introduction · p. 2

  • Results

    Shared information consumption per reader rises by 69%, while topic concentration declines and consumption shifts toward less-popular topics at individual and aggregate levels.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Generative AI search can diversify collective attention while increasing shared information consumption.

    Source: Discussion · p. 5

    Because the experiment added searches within one publisher’s archive, its effects may differ for third-party, multi-publisher services.

    Source: Discussion · p. 5

Abstract

from arXiv · show

Generative AI search and AI overviews are transforming access to information and news, renewing concerns that readers will encounter a narrower range of topics and have less in common. We examine these concerns via a randomized field experiment with 37,561 readers at The Washington Post. Both groups searched the same archive, but treatment readers also received AI answers with article citations above conventional results. Measuring consumption across displayed answers and opened articles, we find that AI search expands the reach of widely read topics and increases overlap in readers' topic consumption. At the same time, consumption becomes less concentrated and shifts toward less-popular topics, both within readers and across the audience. AI answers account for most of the increase in shared information, delivering it without requiring article clicks and broadening exposure beyond the articles readers open. Cited articles also contribute to the shift toward less-popular topics. Readers shift from conventional-result clicks and browsing toward cited articles and follow-up searches. More frequent searching offsets lower article consumption per search, producing a small increase in article consumption per reader. Total information consumption per minute also rises. Generative AI search can thus diversify collective attention while strengthening the information readers have in common.

Introduction

In a randomized field experiment, adding AI answers to search increased shared information and topic overlap while dispersing consumption toward less-popular topics; AI answers supplied much of the shared information without article clicks.

Introduction

  • The study tests filter-bubble concerns in a 38-day randomized Washington Post search experiment with 37,561 readers, measuring shared information, topic concentration, and popularity.Both groups searched the same archive; Treatment readers saw AI answers citing up to five source articles above the conventional ranked results.Source: Introduction · pp. 1–2

RAG search increases shared information consumption

  • Treatment readers consumed 69% more shared information than Control readers, with AI answers accounting for most of the increase and topic consumption more overlapping across Treatment pairs.The share of readers consuming shared information rose from 37% to 55%.Source: RAG search increases shared… · p. 2, Figure 1 · p. 3

AI answers and cited articles change how readers consume

  • Treatment readers consumed 82% more information overall, from 29% more searches and 41% more information per search; consumption per minute rose 62%.Conventional-result clicks and browsing fell by 21 and 5 percentage points, while cited-source clicks and follow-up searches without article clicks rose by 14 and 11 points. More searching offset 18% lower article consumption per search, yielding 6% higher article consumption per reader.Source: RAG search increases shared… · p. 2, Figure 1 · p. 3, AI answers and cited articles change… · p. 4

Consumption disperses toward less popular topics

AI answers and cited articles change how readers consume

  • Including AI answers raised Treatment readers’ shared-information reach from 35.1% to 55.4%, while cited-source articles’ aggregate consumption was less concentrated and favored less-popular topics than other Treatment articles.AI answers broadened total information beyond opened articles; the analyses found no evidence that Treatment readers searched for less-popular topics or that cited-source lists offered less-popular articles than conventional results.Source: Introduction · p. 2, AI answers and cited articles change… · pp. 3–4

Discussion

  • The experiment tests one publisher’s archive, and its answer-topic measures infer answer content from cited articles; third-party search may shift follow-up searches off-publisher.The discussion also notes that broader topic coverage does not guarantee accurate or unbiased summaries, and calls for research distinguishing topic diversity from publisher diversity.Source: Discussion · p. 5

Setting and design

A randomized Washington Post field experiment compared conventional search with search augmented by AI answers, using the same article corpus and retrieval infrastructure. The analysis included 37,561 readers assigned at their first on-site search from October 1 to November 7, 2024.

Setting and design

  • The experiment included 37,561 readers: 26,639 assigned to Control and 10,922 to Treatment.Readers were assigned by browser cookie at their first on-site search between October 1 and November 7, 2024, before the public launch of the publisher’s RAG search feature.Source: Setting and design · pp. 5–6
  • Treatment readers saw an AI answer citing up to five articles above conventional results; Control readers saw conventional results, with both groups using the same corpus and retrieval infrastructure.Source: Setting and design · pp. 5–6
  • Causal interpretation assumes same-day Control and Treatment readers were otherwise comparable after measured pre-search characteristics were accounted for.Measured characteristic differences were small but jointly statistically significant; adjustment for these characteristics and week of first search yielded estimates close to the original group differences.Source: Setting and design · p. 6

Topic model … Inference

The methods map article opens and displayed AI answers onto a shared 58-topic model, then define measures of consumption, shared information, topic concentration and popularity, search behavior, and statistical inference.

Topic model

  • BERTopic represents 370,654 archived articles across 58 retained topics after four routine-service topics are excluded and the remaining topic shares are rescaled.The same model is applied to Control and Treatment groups.Source: Topic model · p. 6

Measuring consumption

  • Article consumption sums topic shares from opened articles, while total information consumption also includes estimated topic shares from displayed AI answers.Each answer’s topic distribution is the equal-weight average of its cited articles’ distributions; opens or displays without topic estimates remain in engagement records but are excluded from topic-based consumption. These measures capture information provided, not how fully it was read or understood.Source: Measuring consumption · p. 6
  • Search efficiency divides total information consumption by searches, and consumption efficiency divides it by recorded minutes on site.Source: Measuring consumption · p. 6

Shared information consumption

  • The shared core is the 14 topics in the top quartile ranked by distinct Control readers opening an article mainly about each topic; shared and non-core consumption sum topic shares across those respective sets.The ranking covers October 1 to November 7, 2024, including reading before first search, and the same cutoff defines the core for both groups. The analysis also measures the proportion of readers opening an article or receiving an AI answer mainly about a core topic; alternative cutoffs and a pre-search-only ranking are checked.Source: Shared information consumption · p. 6
  • Across all 58 topics, reader-pair similarity is measured by cosine similarity, while shared information is the amount of overlapping consumption summed across topics.Both measures are averaged over all pairs, with zero assigned when either reader lacks measured topic consumption, so averages reflect both measured-consumption prevalence and overlap.Source: Shared information consumption · p. 6

Topic concentration and popularity

  • Topic concentration uses effective topic count and normalized Gini, while popularity uses Control-group article-consumption ranks, weighted mean rank, and top-10 popular share.Individual measures are averaged among readers with at least two article opens or AI answer displays; aggregate measures are calculated after combining each group’s consumption. Rank 1 is most popular, and higher ranks are less popular.Source: Topic concentration and popularity · p. 6

Search behavior

  • The next action after each search is classified as a cited-source article open, conventional-result article open, browsing, follow-up search, or session end.An open counts as cited-source when an AI answer in that session cited the article at or before the open; action shares use all searches, and aggregate concentration and popularity are compared for cited-source versus other article consumption in Treatment.Source: Search behavior · p. 7

Inference

  • Differences use reader-resampled 95% confidence intervals and same-day-stratified permutation tests, with Holm-adjusted P values for the shared-information, engagement, concentration, and popularity comparisons.The authors used Claude Fable 5 and GPT-6 via Codex to assist with topic modeling, replication analyses, and text editing, and verified all analyses.Source: Inference · p. 7

Ethics · Supplementary material

The study used de-identified reader records, received an exempt determination after limited IRB review, and reports no competing interests. Supplementary material outlines additional results and methods questions, while reader-level data remain unavailable for public sharing under the data agreement.

Ethics

  • The University of Texas at Dallas IRB deemed the secondary analysis of de-identified reader records exempt under limited review, and the authors were not involved in the publisher’s experiment.The records lacked direct identifiers and were not readily identifiable; the review was under IRB-26-577.Source: Ethics · p. 7

Competing interests

  • The Washington Post conducted the randomized test, while the authors retained responsibility for the analyses, interpretation, and manuscript and reported no competing interests.Publisher review before publication was limited to identifying proprietary information; no author was employed by or held a financial interest in The Washington Post.Source: Competing interests · p. 7

Data, materials, and software availability

  • Reader-level clickstream data cannot be shared publicly, but the authors plan to release analysis code and may provide de-identified summary data subject to agreement with The Washington Post.The data were provided under a Data Use and Research Collaboration Agreement executed in 2024; the code is intended to reproduce reported figures and tables from the analytic dataset.Source: Data, materials, and software… · p. 7

Interfaces and Results

  • The supplementary results index covers what readers saw and examines shared and non-core consumption, reach across readers, and how definitions or political topics affect results.It also lists analyses of AI-answer contributions and reader-pair information overlap.Source: Interfaces and Results · p. 9
  • Other supplementary results questions address consumption per search and minute, text and time with news, robustness to sample and topic-model choices, and what changes when only opened articles count.The index also lists analyses of AI answers, cited-source articles, query topics, post-search behavior, and how more searches add up to consumption.Source: Interfaces and Results · p. 9

Experimental Design, Measurement, and Inference

  • The supplementary methods index includes reader assignment and sample definition, topic-model curation, measurement of article and answer topics, time, shared information, and reader-pair comparisons.It also lists procedures for search windows and cited-source opens, concentration and popularity, multiple comparisons, and the 58 topic labels.Source: Experimental Design, Measurement, and… · p. 9

S1. What did readers in each group see?

The experiment held the archive and conventional results constant while adding an AI answer and cited-article links for Treatment readers.

S1. What did readers in each group see?

S2. Does RAG search increase shared and non-core consumption? · S3. Does shared information reach more readers?

RAG search increases consumption of both shared and non-core information, while AI answers expand shared information’s reach beyond the readers who open relevant articles. These findings persist under alternative definitions of shared topics and after political topics are excluded.

S2. Does RAG search increase shared and non-core consumption?

  • Shared consumption rises from 0.676 to 1.144 and non-core consumption from 0.687 to 1.339 per assigned reader (both Holm-adjusted P < 0.001).Consumption includes the portions of articles and answers devoted to each topic set.Source: S2. Does RAG search increase shared… · p. 10
  • Shared consumption includes article and answer portions devoted to the 14 topics read by the largest shares of Control article readers; non-core consumption covers the remaining topics.For example, an item that is 60% about shared topics contributes 0.6 to shared consumption.Source: S2. Does RAG search increase shared… · p. 10

S3. Does shared information reach more readers?

  • Shared-information reach rises by 18.5 percentage points, from 36.9% to 55.4%, when AI answers are counted; article-only reach is 36.9% in Control and 35.1% in Treatment.The increase in reach comes from including answers, rather than more readers opening articles on shared topics.Source: Table S2 · p. 11, S3. Does shared information reach more… · p. 11

S4. Does the increase depend on how shared information is defined, or on political topics?

  • The increase in shared-information consumption and reach holds when shared topics are defined as the top 10, 14, or 18 topics.The top 10 and top 18 alternatives test whether results depend on the boundary being set at 14 topics.Source: Table S3 · p. 11, S4. Does the increase depend on how… · p. 11
  • Shared-information consumption and reach also increase when topics are ranked using only Control reading before the first search (all P < 0.001).This definition is formed before readers are exposed to the search interface.Source: Table S3 · p. 11, S4. Does the increase depend on how… · p. 11
  • Shared-information consumption increases after political topics are excluded, whether remaining topics are kept or shared information is redefined among them.The exclusions range from U.S. National Politics alone to a broadest set of 10 topics; only consumption on excluded topics is subtracted.Source: S4. Does the increase depend on how… · p. 11

S5. How much of the increase comes from AI answers? · S6. Do pairs of readers share more information? · S7. Does consumption per search and per minute increase?

AI answers account for nearly all of the increase in shared information consumption, while reader pairs share more across topics and total consumption rises per search and per minute.

S5. How much of the increase comes from AI answers?

  • AI answers account for 98.0% of the increase in shared information consumption and 89.2% of the increase in non-core information consumption.These are answer contributions divided by the corresponding total increases; the decomposition does not isolate answers' causal effect from the rest of the interface.Source: S5. How much of the increase comes… · p. 12, Table S5 · p. 12

S6. Do pairs of readers share more information?

  • Cosine similarity and shared information per reader pair both increase across all topics, including topics outside the shared set (Holm-adjusted P < 0.001).Both measures include articles and AI answers and average all assigned reader pairs.Source: S6. Do pairs of readers share more… · p. 12, Table S6 · p. 12
  • The pairwise overlap measures average all assigned reader pairs, and pairs with no measured consumption are counted as zero.The last two table rows divide shared amounts by topic set and have no separate tests.Source: S6. Do pairs of readers share more… · p. 12
  • Pairs involving a reader with no measured topic consumption contribute zero, so the increases reflect participation in measured consumption as well as overlap.Source: S6. Do pairs of readers share more… · p. 12

S7. Does consumption per search and per minute increase?

B. Shared information consumption per minute … S11. Do the results hold under a different topic model?

Shared information consumption per minute is higher with AI search, while checks using alternative topic models, political-topic exclusions, and broader reader samples preserve the shift toward less-popular topics and lower concentration.

B. Shared information consumption per minute

  • +0.0412 shared information consumption per minute (95% CI [0.0372, 0.0451]) separates Treatment from Control.The measure divides each group’s shared information consumption by its total observed time on site.Source: B. Shared information consumption per… · p. 13

S8. Does higher consumption also mean more text and more time spent with news?

S9. Do the individual-level results depend on consumption volume or who is included?

B. Popularity when readers with only one article or answer are also included

S10. Do the concentration and popularity results hold without political topics?

Table S12 (continued)

  • The reported aggregate-level Gini coefficient was −0.0305, with bracketed bounds [−0.0402, −0.0208] and P < 0.001.Source: Table S12 (continued) · p. 16

S11. Do the results hold under a different topic model?

Table S14 (continued) … Answer

Across alternative answer measurements and weighting choices, shared information consumption increases, while total consumption remains less concentrated and shifts toward less-popular topics. The continued results also report increases in effective topics and reductions in the top-10 popular share across two topic specifications.

Table S14 (continued)

  • Across the 58-topic and 72-topic specifications, Effective number of topics increases by +1.27 [0.63, 1.82] and +1.13 [0.38, 1.82], respectively.Mean popularity rank is +0.98 [0.69, 1.27] and +1.12 [0.78, 1.47]; Top-10 popular share (%) is −3.8 [−4.8, −2.7] and −2.5 [−3.5, −1.6], respectively.Source: Table S14 (continued) · p. 17

S12. Do the results depend on how answers are measured or weighted?

  • Across alternative answer measurements, shared information consumption increases, while total consumption remains less concentrated and shifts toward less-popular topics.The size of the differences depends on how answers are measured.Source: S12. Do the results depend on how… · p. 17

Answer

  • Robustness checks compare answer-text and cited-source measurements, citation-order weighting, the same answers across methods, and half or double answer weights.All tested differences have unadjusted P < 0.001; the last column has no separate test.Source: S12. Do the results depend on how… · p. 17

Table S16 (continued)

  • For the same 10,948 answers measured using cited sources, Aggregate Gini is −0.0335 [−0.0430, −0.0236] (P < 0.001).The table reports this comparison under content measurement, using the cited-sources main measure.Source: Table S16 · p. 17, Table S16 (continued) · p. 18

B. Topic popularity

representation

  • The findings persist when answers are measured differently or counted as less or more than one article.Source: representation · p. 18

S13. What changes when only opened articles are counted? … Table S25 (continued)

Opened-article consumption, AI answers, and cited-source articles shift attention toward less-popular topics, while adding answers broadens aggregate topic consumption. Treatment readers search more often and shift their next actions toward cited articles and follow-up searches, producing higher per-reader information consumption.

S13. What changes when only opened articles are counted?

S14. How do AI answers broaden consumption and add less-popular topics?; B. Within the Treatment group in aggregate

B. Within the Treatment group in aggregate

S15. How does cited-source article consumption differ from other article consumption?; B. Individual popularity of Treatment cited articles and Control articles

S16. Do readers ask about, or does search offer, less-popular topics?

S17. What do readers do next after a search?

S18. How do more searches add up to more consumption?

  • Searches per reader rose 29.4%, article consumption per search fell 18.2%, and per-reader article consumption rose 5.9%; including answers, total information consumption rose 82.2% per reader and 40.8% per search.Source: S18. How do more searches add up to… · p. 22

Table S25 (continued)

  • Per-reader means include all assigned readers, while per-search measures divide group consumption by all searches; the ratios describe observed totals, not the causal contribution of extra searches.Source: Table S25 (continued) · p. 23

Experimental Design, Measurement, and Inference … S28. How are concentration and popularity measured?

The study compares cookie-assigned readers within entry days and defines topic-based consumption, shared information, concentration, and popularity using explicit measurement rules. Robustness checks show that the reported findings persist after adjustment for reader characteristics and entry timing, and under an alternative popularity ranking.

S19. How were readers assigned, and who is in the analytic sample?

  • Readers were assigned by browser cookie at their first on-site search, and the main comparisons preserve each entry day’s group sizes because treatment shares vary across days.The sample preparation screens for automated traffic and excludes readers whose first-search query could not be reconstructed; the assignment code, allocation logs, and intended ratio were not disclosed.Source: S19. How were readers assigned, and… · p. 23

S20. Were the two groups alike before the first search?; Table S27 (continued)

Experimental Design, Measurement, and Inference; S21. Do the results hold after adjusting for reader characteristics and entry timing?

S22. How was the topic model chosen and curated?; S23. How much of the observed consumption has topic information?

  • BERTopic’s minimum cluster size of 700 produced 62 topics; after content exclusions, the 58 remaining topics define news-consumption measures, with similar topic-estimate coverage across groups.The model uses all-MiniLM-L6-v2 embeddings, UMAP, and HDBSCAN; comparing models on the same articles and answers preserves the concentration and popularity findings.Source: S22. How was the topic model chosen… · p. 25, S23. How much of the observed… · p. 26

S24. How are article consumption and answer topics measured?; S25. How are time on site and efficiency measured?

  • Article opens and substantive AI-answer displays each contribute one consumption count divided across estimated topics, while per-search and per-minute measures divide group totals by searches and observed time on site.Answer topic shares average those of cited articles with equal source weights; session time ends at departure or the last recorded activity, and departure times are available for about 97% of sessions in each group.Source: S24. How are article consumption and… · p. 26, S25. How are time on site and… · pp. 26–27

S26. How are shared information and reader-pair comparisons defined?

  • Shared information is defined as consumption on the 14 topics in the top quartile of Control-reader reach, while reader-pair overlap and cosine similarity compare consumption distributions across all topics.Per-reader measures include all assigned readers for reach and consumption; pair averages include all assigned pairs, and topic overlap does not establish that readers encountered the same articles, facts, or viewpoints.Source: S26. How are shared information and… · p. 27

S27. How are search windows and cited-source article opens defined?

  • Search windows group successive article opens until a new query, home or section visit, new visit, or session end; immediate post-search actions distinguish cited-source opens from conventional-result opens.A cited-source open is an article previously cited in an AI answer during the same session; follow-up searches have no intervening article open or other page view.Source: S27. How are search windows and… · p. 27

S28. How are concentration and popularity measured?

  • Concentration is measured with effective topic count and normalized Gini, while popularity uses mean popularity rank and top-10 share based on Control article consumption.The pre-search Control ranking closely matches the main ranking (Spearman correlation 0.981), and the shift toward less-popular topics persists with either ranking.Source: S28. How are concentration and… · pp. 27–28, Table S34 · p. 28

S29. How are the shared-information tests adjusted for multiple comparisons? … S32. How are the supplementary comparisons tested?

The supplementary sections specify how multiplicity, uncertainty, and dependence among reader- or query-level measurements are handled. They also distinguish adjusted tests from unadjusted intervals and supplementary P values.

S29. How are the shared-information tests adjusted for multiple comparisons?

  • After Holm adjustment across 11 shared-information and engagement comparisons, the ten tabulated comparisons remained P < 0.001 when the unreported conditional-cosine comparison was omitted.The 11 comparisons were selected before adjustment but were not preregistered; tests used 100,000 permutations, except shared information per reader pair, which used 10,000.Source: S29. How are the shared-information… · p. 28
  • Rebuilding the shared-topic definition in each permutation also yielded P < 0.001 for shared consumption, reach, and non-core consumption.The main tests retained the original shared-topic definition; the conditional-cosine comparison was unreported because its direction depended on topic representation.Source: S29. How are the shared-information… · p. 28

S30. How are intervals and permutation tests computed?

  • Reader-bootstrap intervals use 10,000 resamples and are unadjusted per estimate, while permutation tests reassign labels within each of 38 first-search days, preserving group sizes.Descriptive text and time comparisons in section S8 use 5,000 reader resamples.Source: S30. How are intervals and permutation… · p. 28

S31. How are the concentration and popularity tests adjusted?

  • Table 1 applies Holm adjustment across eight concentration and popularity comparisons using 100,000 permutations, rebuilding Control-reader popularity ranks per permutation while bootstrap intervals fix observed ranks.These comparisons were chosen before adjustment but were not preregistered.Source: S31. How are the concentration and… · p. 28
  • Each political exclusion, alternative topic model, and article-only analysis uses its own eight-comparison Holm adjustment; other supplementary permutation tests use 10,000 permutations and unadjusted P values unless specified.The eight Table 1 comparisons were chosen before adjustment but were not preregistered.Source: S31. How are the concentration and… · p. 28

S32. How are the supplementary comparisons tested?

  • Within-reader and article-route comparisons resample paired contributions together; Table S18 also uses two-sided paired t tests, while answer-versus-Control-article comparisons resample groups independently.Table S21, panel B, uses Welch tests with Holm adjustment across two measures within each inclusion rule.Source: S32. How are the supplementary… · p. 28
  • For Table S11’s equal-consumption check, each eligible reader’s two-article or two-answer concentration is averaged over 100 draws, then bootstrapped 10,000 times without repeating draws.Table S23 uses 10,000 paired query-bootstrap resamples for mean-difference intervals and two-sided Wilcoxon signed-rank P values, which need not yield matching conclusions.Source: S32. How are the supplementary… · p. 28

S33. What are the 58 topics and their labels?

The supplement lists 58 topics with model terms and descriptive labels, which name content categories without changing the analysis. It also explains how Control reach and consumption share are calculated and identifies table annotations for Fig. 1A and shared information.

S33. What are the 58 topics and their labels?

  • Table S35 lists 58 topics with five leading model terms, descriptive labels, Control reach, and Control consumption share.Labels draw on model keywords and representative article content reviewed with generative AI assistance; they name content categories and do not change the analysis.Source: S33. What are the 58 topics and their… · p. 29
  • Control reach measures the share of Control article readers opening an article mainly about a topic, while consumption share allocates all Control article consumption across topics.Both measures cover the whole study window, including reading before the first search; reach percentages can overlap, while consumption shares sum to 100% before rounding.Source: S33. What are the 58 topics and their… · pp. 29, 31
  • Bold labels appear in Fig. 1A, and shaded table rows mark topics identified as shared information.Source: S33. What are the 58 topics and their… · p. 29

Related papers

Loading 2609.38946v1…