PaperlayerView exact revision on arXiv

Source-linked AI summary

One Polluted Page Is Enough: Evaluating Web Content Pollution in LLM Recommenders

Minghao Luo, Liang Chen

arXiv:2606.13610v2cs.CLcs.AI

TL;DR

Search-augmented LLM recommenders may treat GEO-polluted web pages as credible evidence, potentially promoting fake products. The paper introduces FORGE, which locally rewrites real brands in frozen retrieved pages and measures fake recommendations across products, scenarios, and models. All evaluated models are vulnerable, while tested defenses are inadequate and the benchmark’s attack and defense designs are not optimized.

  • Problem

    Search-augmented recommenders retrieve live web content, creating a measurement gap about whether GEO-polluted pages are consumed as credible evidence.

  • Method

    FORGE locally rewrites dominant real-brand mentions in frozen, quality-gated retrieved documents while preserving their surrounding retrieval and document properties.

  • Results

    All 12 evaluated LLMs are vulnerable across 225 products and 15 categories: a single rank-1 polluted page fools up to 27% of cells, while top-3 replacement reaches 73.8%.

  • Takeaways & Limitations

    Web-content pollution is a practical failure mode for search-augmented recommenders, and the tested defenses do not provide adequate protection.

  • Takeaways & Limitations

    The attack and defenses are not optimized, so the reported results should be read as lower bounds on attack effectiveness.

Abstract

from arXiv · show

Search-augmented LLMs increasingly mediate everyday consumer recommendations by retrieving live web content. This creates a new risk: LLM recommenders may consume web content that Generative Engine Optimization (GEO) operators have polluted to mislead them. We ask: to what extent do they become unwitting promoters of fake products? We introduce FORGE (Fake Online Recommendations in Generative Environments), which locally rewrites real products in a frozen set of retrieved web pages into fake ones and measures how often the LLM recommends the fake product, across 225 real products in 15 categories and 5 consumer scenarios. Across 12 commercial and open-weights LLMs, all models are vulnerable: a single polluted page yields fooled rates of up to 27%, while the full top-3 replacement raises this to 73.8%. Vulnerability varies across categories, increasing when models lack stable prior knowledge of the products. Reasoning does not mitigate this vulnerability; instead, it often generates spurious social proof to justify false recommendations. None of the four defenses is adequate: the skepticism prompt can exacerbate vulnerability much like reasoning, the two consensus filters risk suppressing legitimate products, and credibility re-ranking helps every model but removes only a sixth of the fakes. We release the FORGE benchmark and the evaluation code at https://github.com/leoluolol/forge-benchmark.

1 Introduction

Search-augmented recommenders shift part of their trust boundary to live web content, creating a measurement gap around GEO pollution. FORGE addresses this gap by locally rewriting retrieved evidence and evaluating fake-product recommendations across diverse models and products.

  • GEO pollution differs from adjacent robustness threats because plausible web text preserves on-task, policy-compliant recommendations and weakens common detection cues.The attack operates through the open web rather than a controlled channel.
  • FORGE locally rewrites dominant real-brand mentions in frozen retrieved documents while preserving rank, URLs, attribution, length, style, and context.This isolates the brand swap as the source of recommendation changes without polluting the live web.
  • The benchmark covers 225 products in 15 categories and evaluates 12 commercial and open-weights LLMs, with Chinese experiments and an English replication.Its product coverage ranges from brand-concentrated markets to fragmented long-tail categories.
  • 27% of cells are fooled by a single rank-1 polluted document, while top-3 replacement reaches fooled rates of 13.3%–73.8% across models.The vulnerability rises near-monotonically with the number of polluted pages.

2 Background and Preliminaries

The paper models LLM recommendation as a pipeline in which search retrieves top-K web pages and the model conditions its answer on the resulting context. GEO pollution replaces the clean web with operator-authored pages that may be retrieved and treated as credible evidence.

  • Top-K search results E = S(q; W) provide the web context on which the LLM conditions its recommendation.The model concatenates retrieved context and the query before generating a brand name.
  • GEO operators add fake pages Wfake to the open web to promote fake brands and influence downstream LLM recommendations.These pages are designed to be indexed and surfaced by mainstream search engines.
  • The pollution succeeds when the model’s generated recommendation belongs to the promoted fake-brand set.FORGE measures this outcome after polluted pages enter the retrieved context.
  • 74% of queries contain a UGC page in the top three, while rank 1 is UGC in over half of queries.Unclassified long-tail hosts are counted as commercial, making these UGC shares lower bounds.

3 The FORGE Benchmark

FORGE constructs a reproducible benchmark from quality-filtered live search results, simulates controlled pollution locally, and evaluates whether fake brands appear in ranked recommendations. It also records recommendation placement and validates that fooled cells represent endorsements rather than incidental mentions.

  • Benchmark Construction: 225 products span five consumer scenarios, with three categories of 15 products in each scenario.The scenarios include Digital Products, Local Life, Health & Personal, Fashion Accessories, and Sports & Outdoor.
  • Benchmark Construction: Each query uses a manually crafted scenario-matched template, and each frozen evidence bundle contains the first K=10 quality-gated search results in original rank order.Bundles exclude errored, garbled, boilerplate, and video-platform pages and are fixed across attack conditions and models.
  • Threat Model: The threat model allows an adversary to publish ordinary user-generated pages that rank in the retrieved top-K without model, index, training-data, or prompt access.The pages must pass upstream filtering without injected instructions.
  • Pollution Simulation: FORGE tests entity replacement, passage injection, and full synthesis, spanning increasingly realistic ways to introduce fake-brand content into retrieved documents.Entity replacement preserves URLs and surrounding context, while full synthesis replaces the document body under a same-domain URL.
  • Pollution Simulation: FORGE rewrites frozen bundles locally rather than polluting the live web, because live seeding could harm real users and become irreversible once indexed.This makes the experiment reproducible while limiting the benchmark’s correspondence to live pollution.
  • Evaluation Metric: 57% of fooled cells place the fake brand at rank 1, and 84% place it within the top three when the model is fooled at all.Across models, the unconditional top-1 rate ranges from 5% to 53%.

4 Experiment

FORGE evaluates recommendation pollution across twelve models, 225 products, and fifteen categories, finding broad vulnerability shaped by retrieval placement, attack intensity, category, and reasoning. A single top-ranked polluted page can suffice, while multiple pages and stronger synthesis attacks amplify the effect.

  • 4 Experiment: Vulnerability varies sharply by category, with everyday-consumption categories most exposed and technical-product categories least exposed.The category variation is statistically significant (Friedman χ2(14)=99.4, p < 10^-14), and dining is the most-fooled category for two thirds of models.
  • 4 Experiment: All twelve models are vulnerable under top-3 replacement, with vulnerability overlapping across closed-source and open-weights systems.Within families, larger models can be more vulnerable: Gemini 3.1 Pro is fooled roughly three times as often as Gemini 3 Flash.
  • 4 Experiment: Reasoning increases vulnerability by 18 percentage points on Qwen3.5-9B and 9 points on GLM-4.6V-Flash in paired experiments.The within-model comparison holds architecture, weights, training, and decoding constant, supporting a causal link in these experiments.
  • 4 Experiment: 27% of cells fool the two most-vulnerable models when one polluted page occupies retrieval rank 1.The same page placed at ranks 2–10 produces only 1–4% fooled rates, making the first retrieved page disproportionately influential.
  • 4 Experiment: Adding polluted pages raises fooled rates near-monotonically, with the most-vulnerable models crossing 50% after as few as three pages.The experiment uses six open-weights models on the Digital Products subset.
  • 4 Experiment: Full synthesis is the strongest attack, entity replacement is effective, and passage injection is weakest because surviving real-brand mentions can pull recommendations back.The three attack styles span low, middle, and high category examples; wholesale replacement is correspondingly more dangerous.
  • 4 Experiment: The pattern generalizes to English evidence: 8 of 12 models remain within ±10 percentage points of their Chinese rate.The replication covers smartphones, skincare, and SF restaurants using US-region search results.

5 Analysis

The analysis links pollution vulnerability to unstable brand recommendations and examines how models respond when resisting or accepting fake brands. Low cross-model agreement predicts greater vulnerability, while fooled outputs often add unsupported social proof.

  • 5 Analysis: Categories with lower cross-model agreement on brand recommendations are more vulnerable to polluted bundles.Mean pairwise Jaccard agreement predicts category vulnerability with Pearson r = −0.65, p < 0.01, and the relationship is direction-stable across models.
  • 5 Analysis: Models often resist by noticing and rejecting the fake brand rather than ignoring it.Responses that mention and reject the fake brand have reasoning traces roughly six times longer than fooled or never-mentioned responses.
  • 5 Analysis: Fooled outputs add social-proof language absent from the polluted documents.Across the population, fooled outputs trigger 1.5–11× more social-proof markers and fewer hedging markers than resisted outputs.

6 Defenses

The evaluated defenses fail in different ways: skepticism can increase vulnerability, consensus filters remove legitimate recommendations, and credibility re-ranking reduces but does not eliminate fake recommendations.

  • 6 Defenses: Across all twelve models, the skepticism prompt raises pooled fooled rate by 10.5 percentage points instead of reducing vulnerability.Closed-source models worsen by 24 points on average, while open-weights models are roughly flat or slightly helped at −3 points.
  • 6 Defenses: Skepticism backfires most in low-baseline categories, including smartphones (+32 pp), bags (+19), and makeup (+18).It is neutral in dining (+6) and helps only in skincare (−11); across closed-source models it hurts in all fifteen categories.
  • 6 Defenses: The prior filter removes planted fakes in 95% of cells and the agreement filter catches them in 90%, but both discard substantial legitimate recommendations.The filters therefore trade fake removal for recommendation utility.
  • 6 Defenses: Credibility re-ranking lowers pooled fooled rate from 50.4% to 42.1% across 1,350 paired cells.It helps every open-weights model and removes 17% of fake recommendations net, but 38–59% remain fooled where it helps most.

7 Related Work

FORGE addresses a gap left by benchmarks of prompt injection, closed-corpus poisoning, simulated-catalog attacks, and adversarial SEO: whether LLMs treat polluted live-web evidence as credible recommendation evidence.

  • Adjacent robustness settings: Existing benchmarks study prompt injection, closed-corpus RAG poisoning, simulated-catalog poisoning, and ranking manipulation rather than adversarially corrupted retrieved evidence.These settings differ from FORGE’s focus on search-augmented recommenders operating over fresh web content.
  • Distinct threat model: FORGE’s pollution remains on-task and policy-compliant, weakening detection cues such as anomalous instructions, out-of-distribution passages, trigger tokens, and refusal breakage.The unresolved question is whether retrieved polluted pages are consumed as credible evidence.
  • Why pollution can work: Minimal brand swaps preserve relevance and coherence, properties linked more closely to downstream success than factuality in generated knowledge.This provides a mechanism for fake brands to remain persuasive without obvious textual disruption.
  • Relation to prior findings: FORGE’s category pattern matches fragile long-tail entities and shows a pure primacy effect rather than the U-shaped pattern reported for long-context question answering.Fooled outputs also add social proof, characterized as a form of confabulation.
  • Benchmark contribution: The paper presents FORGE as the first Chinese vulnerability benchmark testing retrieval-time pollution, whereas existing Chinese-inclusive benchmarks use clean inputs.

8 Conclusion

FORGE demonstrates that web-content pollution can make search-augmented LLM recommenders promote fake products, especially when product knowledge is unstable. The tested defenses do not provide an adequate solution, motivating stronger evidence-level defenses and the released benchmark.

  • Conclusion: A single top-ranked polluted page can induce fake-product recommendations across commercial and open-weights LLMs, while a small number of polluted pages can make the effect widespread.
  • Conclusion: Vulnerability is strongest when models lack stable prior product knowledge, and failures often include spurious social proof that makes fake products appear credible.
  • Conclusion: None of the four tested defenses is usable as it stands: one raises fooled rates and two suppress most legitimate recommendations.The authors call for evidence-level defenses stronger than those evaluated and release FORGE as a testbed.

Limitations

The evaluation uses unoptimized attacks and defenses, uneven coverage across secondary analyses, primarily Chinese-language Shenzhen data, and a single frozen retrieval snapshot.

  • Attack and defense design: The attack and defense designs are not optimized, so the reported vulnerability should be read as a lower bound on attack effectiveness.Unstudied stronger attacks could combine domain-tailored templates, query-aware paragraphs, and adversarial-SEO techniques.
  • Attack and defense design: Optimization-based defenses could be adapted to this setting but would introduce additional computational overhead.
  • Experimental coverage: Secondary analyses use narrower subsets: some cover only open-weights models, three Digital Products categories, or six models, leaving closed-source generalization of the process signature for future work.
  • Language and region: Main results are Chinese-language with Local Life fixed to Shenzhen, while full multilingual and multiregion evaluation remains future work.An English replication preserves the low–mid–high category ordering across three matched categories.
  • Data snapshot and selection: Evidence bundles come from a single frozen retrieval snapshot, so per-category vulnerability rates may shift as the underlying corpus evolves.The authors expect structural findings such as per-model variation, polluted-page count, and primacy to be more stable.

Ethical Considerations

The paper frames FORGE as a controlled, reproducible study of an already operational dual-use threat, using locally rewritten frozen web evidence rather than polluting the live web.

  • Dual-use rationale: The authors publish because GEO-based fake-brand promotion is documented in commercial deployment and corresponding enforcement has begun.
  • Dual-use rationale: A controlled measurement framework is intended to help model developers, platforms, and users understand and mitigate exposure to web-content pollution.
  • Scope and mitigation: FORGE adds no novel attack capability, uses fake-brand prefixes unlikely to collide with real brands, and describes the method for academic reproduction rather than plug-and-play deployment.
  • Evaluation setup: The study uses a shared recommendation system prompt and scenario-specific user templates across its evaluation queries.
  • Evaluation setup: Retrieved evidence is collected through live search, filtered for errors and unsuitable pages, manually quality-reviewed, and frozen as the first 10 passing documents.
  • Brand extraction: The target brand is selected through LLM proposal, rule-based extraction, and human verification; cumulative recall reaches 100% after all three stages.
  • Brand extraction: Human review verifies candidate lists by accepting, replacing, or correcting the proposed brand, with reviewer agreement assessed on a stratified sample.

B Qualitative Case Study

The qualitative case study shows models reproducing a fake brand from polluted top-ranked documents and inventing unsupported social proof, while audits confirm these are recommendations rather than incidental mentions.

  • Qualitative Case Study: Table 6 places the fake brand Langyu only in the three replaced documents at ranks 1–3, while ranks 4–10 contain real brands.The outputs compare fooled and resisted cases under top-3 entity replacement for screen-protector recommendations.
  • Qualitative Case Study: Models generated social-proof claims about Langyu that were absent from all three polluted documents.The claims included frequent recommendations, high sales or reputation, and other positive review-style assertions.
  • Endorsement Audit: 99.0% of 1,154 fooled cells placed the fake brand inside the prompted numbered recommendation list.Only 11 cells mentioned it outside the list, making in-list placement an operational recommendation criterion.
  • Endorsement Audit: 0.9% of fooled cells contained a warning marker, and manual inspection found none of the eight highest-confidence flagged cells genuinely warned against the fake brand.The flagged markers appeared in positive contexts or negations of negative outcomes.
  • Predictors: Brand-pool richness and alignment with cross-model consensus predict category-level fooled rates, with the label-free composite reaching leave-one-out R2 = 0.672.Adding label-using features raised leave-one-out R2 to 0.727, while model fixed effects alone explained R2 = 0.434.

K Attack Realism Comparison

The attack-realism comparison finds full document synthesis most effective, entity replacement already sufficient, and passage injection weaker on average but more effective for some closed-source models. English replication preserves the vulnerability ordering, while re-ranking has a limited net benefit.

  • Attack Realism Comparison: 78% / 38% / 25% are the grand fooled rates for full synthesis, entity replacement, and passage injection, respectively.Full synthesis was strongest on 11 of 12 models and reached at least 70% on 9 of 12.
  • Attack Realism Comparison: Full synthesis produces fewer fake recommendations than entity replacement for Claude Sonnet 4.6, at 24% versus 47%.The paper characterizes this as model-specific behavior likely tied to content-style filtering of synthetic articles.
  • Attack Realism Comparison: Passage injection is weaker on average because real-brand mentions remain visible, whereas full synthesis removes that corroboration.The study cannot fully separate mixed-brand protection from the lower fake-brand density of passage injection.
  • English Cross-Lingual Replication: English replication preserves the low–mid–high category ordering: Smartphones 43% < Skincare 58% < SF Restaurants 87%.The matched Chinese ordering was 23% < 57% < 82%, and the pipeline required no Chinese-specific component.
  • English Cross-Lingual Replication: Across models, English-minus-Chinese shifts span −20 to +40 percentage points, with Gemini 3.1 Pro showing +40 and Claude Sonnet 4.6 showing −20.Eight models remain within ±10 percentage points of their Chinese baselines.
  • Defense Implication: Re-ranking flips 219 fooled cells to resisted and 106 resisted cells to fooled, producing a net reduction of 17%.The gross reduction is about 32%, but promoting editorial and commercial pages introduces new fooled cases.

N Defense Details

The evaluated defenses expose distinct weaknesses: skepticism can increase vulnerability, consensus filters suppress legitimate recommendations, and reasoning itself can amplify susceptibility to polluted evidence.

  • Skepticism Prompt: +10.5 pp pooled fooled-rate increase makes the skepticism prompt a net amplifier across 12 models.Closed-source models averaged +24 pp, while open-weights models averaged −3 pp.
  • Agreement Filter: 90% fake-brand catch at agreement threshold τ=4 costs 63% pooled utility, while stricter τ=5 costs 74%.At τ=3, the filter leaves the fake brand untouched and still incurs a 49% utility cost from acting on real brands.
  • Consensus Filters: Prior and agreement filters remove fake brands but suppress 62–79% and 52–73% of legitimate recommendations, respectively.Any threshold strict enough to catch a 3-of-10-document plant also removes most legitimate recommendations from the unfiltered model.
  • Skepticism Prompt: 14 of 15 categories worsen under skepticism, with the largest backfires in phone/PC (+32 pp), bags/shoes (+19), makeup (+18), and hospitality (+16).Skincare is the lone exception, reaching −11 pp because the open-weights subgroup improves strongly.
  • Reasoning: Reasoning increases vulnerability: disabling it reduces fooled outputs in paired experiments that hold models, prompts, evidence, and decoding constant.The paired design isolates reasoning as a causal driver in the two tested open-weights models.
  • Reasoning: Reasoning-trace length separates resisted-and-rejected cells from both fooled and never-mentioned cells by roughly sixfold.The rejected group has mean reasoning-share 0.88 versus 0.58 in the other groups, while fooled and never-mentioned cells are near-identical.
Loading 2606.13610v2…