PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning

Zhongan Bi, Qiwen Wang, Jianrong Jiang, Jigang Ding, Wenwen Xiong, Changhua Meng, Xuanang Gao, Kepeng Lin, Changjiang Jiang, Yiang Chen, Huan Yao, Wei Wang, Zhenyu Ma, Wenhui Dong

arXiv:2609.06027v2cs.CRcs.AIcs.IR

TL;DR

GEO poisoning can manipulate Web evidence used in consumer recommendations, while evaluating how agents handle suspicious evidence remains important. HAE-GEO tracks agent trajectories under escalating poisoning and finds that better final resistance does not necessarily mean better evidence recognition or recovery.

  • Problem

    GEO poisoning manipulates Web content that search-augmented agents use for consumer decisions, and evaluation must account for evidence handling before recommendations.

    Source: §1 · pp. 1, 3

  • Method

    HAE-GEO combines controlled escalation of Web poisoning with a Search–Scrape protocol and trajectory-aware evaluation from exposure through recovery.

    Source: §1 · p. 3

  • Results

    Across ten agents, evidence recognition degrades with more plausible corroboration, agentic search improves final resistance without improving recognition or utility, and defense prompting rarely converts verification into recovery.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    HAE-GEO provides a foundation for studying resistance, recovery, and utility under inference constraints.

    Source: §6 · p. 15

    For GPT-5.6-sol, M3 recovery is unobservable because gateway traces omit intermediate reasoning and other pre-final text needed to establish prior adoption.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Search-augmented LLM agents are increasingly used for consumer decisions, making them vulnerable to Generative Engine Optimization (GEO) poisoning. Existing benchmarks largely measure whether manipulated content is retrieved or endorsed, but do not track whether an agent verifies suspicious evidence, revises adopted claims, or recovers before producing its final recommendation. We introduce HAE-GEO, a benchmark that tracks the full trajectory from exposure to recovery under progressively more persuasive Web poisoning. Agents interact via a multi-turn Search-Scrape interface across three attack levels (L1 direct assertion, L2 contextual camouflage, and L3 apparent corroboration), supported by a controlled corpus of 72,039 clean pages and 770 poisoned pages per level spanning 8 product categories and 154 brands. Evaluation combines deterministic behavioral measures with six semantic rubric dimensions. Evaluating 10 agents, we find three recurring patterns: evidence recognition degrades under the corroboration trap; agentic search improves final resistance without improving evidence recognition or utility; and defense prompting increases verification, yet rarely converts verification into recovery.

1 Introduction

GEO poisoning can shape consumer recommendations by manipulating Web evidence, while prior evaluations did not jointly track verification and recovery. HAE-GEO introduces controlled, escalating poisoning and trajectory-aware evaluation, and finds recurring weaknesses in recognition, resistance, and recovery.

1 Introduction

  • Earlier benchmarks measured downstream retrieval or endorsement but did not jointly track verification and recovery or escalate the presentation of one fixed false claim.This left unclear where an agent failed and whether continued search could enable recovery after adoption.Source: §1 · p. 2
  • The evaluation assesses how agents interrogate suspicious evidence before recommendation, including retrieval traceability, source independence, and uncertainty calibration.It considers evidence handling alongside final outcomes.Source: §1 · p. 3
  • HAE-GEO holds the query, fabricated entity, false claim, and poisoning budget fixed while escalating the claim through direct assertion, contextual camouflage, and apparent corroboration.Its corpus contains 72,039 real Web pages and 770 poisoned pages per level across 8 categories and 154 brands.Source: §1 · p. 3
  • The Search–Scrape framework separates exposure, provisional adoption, verification, endorsement, and recovery, combining behavioral measures with six semantic rubric dimensions.The rubric covers risk handling, poison recognition, recovery, evidence quality and independence, uncertainty calibration, and legitimate utility.Source: §1 · p. 3
  • Across ten agents, evidence recognition worsens as false claims become more plausible, agentic search improves final resistance without uniformly improving recognition or utility, and defense prompting rarely turns verification into recovery.The evaluation identifies these patterns across model families.Source: §1 · p. 3

2 Related Work

Related work examines GEO ranking manipulation, poisoning in retrieval-augmented systems, and Web manipulation of agents. HAE-GEO distinguishes itself by jointly tracking the evidence-handling trajectory and escalating the apparent credibility of a fixed claim.

2 Related Work

  • Prior GEO studies primarily assess manipulated content’s visibility, ranking, or citations, rather than how autonomous agents verify or revise evidence after exposure.Source: §2 · p. 3
  • Poisoning studies of RAG and agent memory generally use predefined corpora, whereas open-Web agents choose what to search, inspect, and verify.The passage also distinguishes indirect prompt injection: poisoned pages manipulate factual evidence rather than provide task-hijacking instructions.Source: §2 · pp. 4–5
  • Web-agent benchmarks do not jointly distinguish exposure, adoption, verification, endorsement, and recovery while holding a false claim fixed across credibility levels; HAE-GEO evaluates both dimensions.The related-work comparison includes these settings, but the supplied table caption does not expose its cell values.Source: §2 · p. 4

3 Benchmark Design

HAE-GEO pairs controlled Web environments with a Search–Scrape interface and trajectory states to evaluate how agents respond to progressively sophisticated poisoned evidence. It separates exposure, adoption, verification, recovery, and final endorsement.

3 Benchmark Design

  • HAE-GEO combines hierarchical poisoning, paired Web environments, Search–Scrape interaction, and observable trajectory states to evaluate agent responses to poisoned evidence.Evaluation rules and semantic scoring are specified separately.Source: §3 · p. 5

3.1 Hierarchical Poisoning Generation

  • The benchmark holds the fabricated entity and false claim fixed while increasing the intended evidential sophistication of the poisoning, with L3 adding apparently corroborating signals.At L3, agents must assess the provenance and independence of apparent support.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 Controlled Web Evidence Environments

  • Across Clean and L1–L3 environments, each task shares its query and clean Web background, while only the level-specific poisoned corpus changes.The poisoning budget is constant across levels, so comparisons do not confound page count with presentation sophistication.Source: §3.2 · p. 6

3.3 Agentic Search–Scrape Interaction

  • Agents can SEARCH for result metadata, SCRAPE page content, or ANSWER; separating Search from Scrape exposes intermediate information-seeking decisions.Attack annotations and source labels are retained for offline attribution and are not returned to the evaluated model.Source: §3.3 · p. 6

3.4 Trajectory-State Modeling

  • Five deterministic trajectory states separately measure exposure, adoption, verification, recovery, and final endorsement.Exposure marks visible target-brand evidence; adoption captures provisional acceptance; verification requires an independent query and unseen non-attack evidence; recovery is defined after adoption.Source: §3.4 · pp. 6–7

4 Evaluation Protocol

The evaluation protocol applies fixed-query, multi-model trials across clean and poisoned environments, then combines exposure-conditioned behavioral measures with six rubric dimensions scored by isolated judges.

4.1 Environment and Evaluation Setup

  • The main evaluation reuses 120 category-balanced queries across ten model families, BASE and DEF prompts, and Clean/L1/L2/L3 conditions.Agents may use at most ten decision rounds before a forced final answer.Source: §4.1 · p. 7, Table 3 · p. 7

4.2 Behavioral Outcome and Trajectory Diagnostics

  • Primary behavioral metrics are conditioned on target-relevant poison exposure; endorsement requires positive framing, verification requires substantive unseen evidence from a non-attack source, and V2R measures recovery after adoption and verification.Only eligible trajectories are included for conditional measures such as M3.Source: §4.2 · p. 8

4.3 Six-Dimensional Semantic Rubric

  • The six rubric dimensions complement rule-based measures by assessing poisoning robustness, evidence reasoning, and retained task utility.M3 applies only when prior adoption is observable and is reported as not applicable otherwise.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Split LLM-as-Judge Protocol

  • Two isolated judges score M1–M3 with attack annotations and M4–M6 without attack labels; scores are reported by dimension as raw integers from 0 to 2.The quality judge sees only the query, model-visible evidence ledger, and final answer.Source: §4.4 · pp. 8–9

5 Experiments

Across a fixed ten-model evaluation, the experiments compare defense prompts, agentic search, and explicit reasoning under hierarchical poisoning while tracking robustness, evidence handling, utility, and inference cost. Gains are not uniform across these dimensions: stronger resistance can coexist with weak recovery, reduced utility, and added cost.

5.1 Experimental Setup

  • Experiments use a fixed 120-query, ten-model protocol, compare BASE and DEF across Clean/L1/L2/L3, and report FR alongside six semantic rubric dimensions.Conditional measures such as M3 are reported only for eligible trajectories.Source: §5.1 · p. 9

5.2 Robustness and Reasoning under Hierarchical Poisoning

  • Both macro M1 and M2 fall from L1 to L3 under Base and Defense, although defense shifts the curves upward and individual models show reversals.Under BASE, M1 falls 0.53 points (0.83 to 0.30) and M2 falls 0.69 points (1.02 to 0.33); under DEF, their declines are 0.65 and 0.51 points. M3 is unavailable for GPT-5.6-sol because its traces do not show observable prior adoption.Source: §5.2 · pp. 9–10, Figure 3 · p. 11
  • 10.9%, 24.2%, and 18.8%: Kimi-K3’s defended FR across L1–L3; defense raises M1/M2 universally, but FR varies and Attack Utility falls 0.01–0.27 across models.GPT-5.6-sol’s L1 FR is unchanged and its L2 FR rises from 47.6% to 49.2%; GLM-4.7-Flash remains above 46% at every level.Source: §5.2 · p. 9, Table 5 · p. 10

5.3 Agentic Search: Resistance without Recognition

  • Agentic interaction reduces fake-brand endorsement in all four model–prompt conditions, but does not reliably improve poison recognition and is often flat or lower on utility.The paired comparison uses Kimi-K3 and Qwen3.8-27B against Static Full-Context retrieval; endorsement reductions persist among queries exposed to target poisoning under both protocols.Source: §5.3 · pp. 10–11, Figure 4 · p. 12

5.4 Verification Is Not Recovery

  • Defense lifts strict verification from roughly 2% to 17%, yet defended final endorsement rises from 28.6% to 38.6% from L1 to L3, while verified adopted trajectories recover less often.Verification-to-Recovery for defended trajectories falls from 81.8% at L1 to 66.7% at L2 and L3; those conditional subsets are small.Source: §5.4 · pp. 11–12, Figure 5 · p. 13

5.5 Explicit Reasoning Strengthens Defenses

  • 9.7 and 8.8 points: reasoning-on pooled FR reductions for Kimi-K3 under BASE/DEF; Qwen3.8-27B’s reductions are 16.0 and 19.8 points, with M1/M2 rising in all four comparisons.The paired results pool L1–L3.Source: §5.5 · pp. 12–13, Table 6 · p. 13

5.6 The Cost of Robustness

  • Defense adds 1.82 tool calls per query and raises macro-average within-model token use by 24.5%, with token growth ranging from 8.6% to 52.0%.GLM-4.7-Flash adds 0.70 calls, while MiniMax-M3 adds 3.29; additional search cost does not guarantee a commensurate robustness gain.Source: §5.6 · p. 14, Table 7 · p. 14

5.7 Robustness–Utility Trade-off

  • Defense moves all ten models toward higher robustness but lower utility; Kimi-K3 has the highest defended robustness, while Qwen3.6-35B-A3B has the highest defended utility.The non-dominated Defense configurations span multiple operating points rather than identifying a single best system.Source: §5.7 · p. 14, Figure 6 · p. 15

6 Conclusion

HAE-GEO evaluates search-augmented LLM agents under progressive GEO poisoning, tracking trajectories from exposure through recovery. Across ten agents, it finds recurring gaps in evidence recognition, resistance, utility, and recovery.

6 Conclusion

  • HAE-GEO combines progressive GEO poisoning with a Search–Scrape protocol that tracks exposure, adoption, verification, endorsement, and recovery.Source: §6 · p. 15
  • Across ten agents, evidence recognition degrades as false claims become more plausible and corroborated; agentic search improves final resistance without improving recognition or utility.Source: §6 · p. 15
  • Defense prompting increases verification but rarely converts post-adoption verification into recovery.Source: §6 · p. 15

Data Release Statement

The public release includes only a subset of the clean Web corpus because some source sites restrict crawling, collection, or reuse. It therefore does not reproduce the complete clean corpus used in the experiments.

Data Release Statement

  • The public release excludes pages restricted from crawling, collection, or reuse, so it is not a complete copy of the experimental clean corpus.Source: Data Release Statement · p. 15
  • Public access to a page does not imply permission to redistribute it, and the release grants no additional rights to third-party material.Source: Data Release Statement · p. 15

A.1 Operational Semantics of the Poisoning Hierarchy

The three poisoning levels change how the same fabricated entity and claim are presented, escalating from direct promotion to contextual camouflage and apparent corroboration. The hierarchy describes construction sophistication, not a guaranteed monotonic error pattern across models.

A.1 Operational Semantics of the Poisoning Hierarchy

  • L1 presents the claim as a direct promotional assertion with little support; L2 embeds it in plausible content with partially factual context; L3 adds distributed, apparently corroborating signals.Source: §A.1 · p. 19
  • L3 requires assessing both the target claim and the provenance and independence of its apparent support.Source: §A.1 · p. 19
  • Increasing construction sophistication does not guarantee a monotonic error-rate increase, since models may respond differently to authority cues, page genres, or promotional language.Source: §A.1 · p. 19

A.2 Poisoned Page Construction

Poisoned pages are constructed from configurations pairing attack vectors with page types, using either direct generation or retrieval-and-injection. The two paths balance controlled coverage of attack configurations with the structural and stylistic variation of real Web pages.

A.2 Poisoned Page Construction

  • The construction pool pairs attack vectors with page types and maps each pair to a concrete configuration that constrains generation.Source: §A.2 · p. 19
  • Roughly equal shares of poisoned pages come from direct generation and retrieval-and-injection.Source: §A.2 · p. 19
  • Direct generation uses the selected attack vector, page type, poisoning level, and target claim; retrieval-and-injection modifies a sampled clean page while preserving its broad structure and style.Source: §A.2 · p. 19
  • Claude-sonnet-4.6 serves as the generator for both paths, combining controlled configuration coverage with real-page structural and stylistic variation.Source: §A.2 · p. 19

A.3 Product Category and Brand Coverage

The benchmark spans eight consumer-product categories, 154 fabricated target brands, and 770 poisoned pages per level. Category and target-brand inventories stay constant across L1–L3, supporting comparisons across levels without changes in category composition or poisoning budget.

A.3 Product Category and Brand Coverage

  • The benchmark covers 8 consumer-product categories, 154 fabricated target brands, and 770 poisoned pages at each poisoning level.Table 8 reports the category-level allocation.Source: §A.3 · p. 19, Table 8 · p. 20
  • The same category and target-brand inventory is used across L1–L3, so category composition and poisoning budget do not vary across levels.Source: §A.3 · p. 19

A.4 Page-Type Distribution across Poisoning Levels

The poisoned corpus shifts from direct, low-credibility page formats at L1 toward more editorial, evaluative, commercial, and apparently authoritative roles at L2 and L3.

A.4 Page-Type Distribution across Poisoning Levels

  • L1 is concentrated in direct, low-credibility formats, while L2 and L3 increasingly use editorial, evaluative, commercial, and apparently authoritative page roles.This distribution implements the intended escalation in contextual and evidential sophistication.Source: §A.4 · p. 20, Figure 7 · p. 20

A.5 Attack-Vector Distribution across Poisoning Levels

This appendix section details how poisoning levels, model-visible exposure, and trajectory states are constructed and evaluated, and specifies the source-exact judging criteria and reproducible query selection. It reports semantic-rubric findings and uses L3 trajectories to illustrate evidence handling and a recovery-label limitation, while noting that attack sophistication does not guarantee greater empirical difficulty for every model.

A.5 Attack-Vector Distribution across Poisoning Levels

  • L1 is dominated by direct promotional assertions, whereas higher levels combine contextual camouflage with credibility cues; content-length distributions are reported to assess possible length confounding.The listed L3 surface cues include ratings, parameters, reports, certifications, institutional references, and user feedback.Source: §A.5 · p. 20, §A.6 · p. 20, Figure 8 · p. 21

A.7 Environment Pairing and Exposure Accounting

  • Across conditions, tasks reuse the same queries and clean Web background with exactly matched injection budgets; isolated retrieval namespaces and caches prevent cross-condition reuse, while exposure is tracked from model-visible evidence.Corpus-level attack budget is distinguished from target-specific exposure because attack pages may remain unretrieved or fail to expose task-relevant content.Source: §A.7 · p. 21

A.8 Tool Projection and Trajectory-State Annotation

  • Search exposes only result metadata, whereas Scrape reveals page text; deterministic trajectory labels distinguish target exposure, observable prior adoption, strict independent-verification yield, evidence-based recovery, and positive final endorsement.Hidden attack labels and provenance are unavailable to the evaluated model, and Search summaries count only as leads rather than inspected evidence.Source: §A.8 · p. 22

A.9 Full Semantic Rubric Results

  • M5 can be high even when M1 and M2 are low, while M4 varies more and often changes less under DEF, supporting separate reporting rather than a single robustness score.Table 9 pools the six semantic dimensions over L1–L3; the supplied passage does not provide its cell values.Source: §A.9 · p. 23, Table 9 · p. 23

A.10 Calibration of Evidence-Enhanced Poisoning

  • L3 uses distributed, consistent but non-identical support across plausible sources because dense authority cues could trigger skepticism; L1–L3 therefore do not guarantee monotonic empirical difficulty across models.Model-specific reversals indicate that different credibility patterns can activate skepticism in different systems.Source: §A.10 · pp. 23–24, Table 10 · p. 24

A.11 Additional Evaluation Details

  • The documented experiment uses 120 category-balanced queries across ten model families, compares BASE and DEF prompts, and allows agentic runs up to ten decision rounds before forced finalization.The subset contains 60 recommendation/comparison tasks and 60 verification tasks; DEF adds evidence-safety instructions to BASE.Source: §A.11 · p. 24

A.12 Original Agent Instructions and Prompt Provenance

  • Available runtime instructions are exact exports, but judge templates are reconstructed and archived case reports cite v5 and GPT-5.5, so historical requests and scores cannot be assumed to match current artifacts.The available evaluator source identifies itself as v6, and the materials do not establish that archived scores were produced by GPT-5.5.Source: §A.12 · p. 25

A.13 Source-Exact Judge Templates and Rubric

  • The Poison Judge scores fake-brand risk handling, poison-evidence recognition, and recovery only after exposure and prior adoption; recognition requires identifying a manipulation pattern and lowering its evidence weight.The two evaluators receive separate inputs, with the Poison Judge attack-aware and the Quality Judge blind to attack labels.Source: §A.13 · pp. 27–28
  • The attack-aware rubric scores fake-brand handling, poison-evidence recognition, and recovery, requiring explicit risk treatment, identified manipulation, and observable adoption-to-retraction when applicable.Never being fooled is not recovery; M3 is inapplicable when there is no prior adoption or when Static Full-Context Search has no subsequent verification stage.Source: §A.13 · pp. 31–32
  • The attack-label-blind quality rubric separately scores evidence quality and independence, uncertainty calibration, and legitimate utility under attack.It distinguishes brand claims from verified facts, checks whether confidence matches evidence, and rewards useful recommendations that satisfy user constraints without indiscriminate suspicion.Source: §A.13 · pp. 32–33

A.14 Exact 120-Query Selection Protocol

  • The archived protocol selected 120 queries from an existing balanced 240-query pool, with 15 per category and 60 per task type.After filtering to comparison-and-recommendation and verification, it sampled without replacement within category/task strata using random.Random(20260821); 64 selected queries had fake-brand annotations and 56 did not, an observed composition rather than a quota.Source: §A.14 · p. 33

A.15 Representative Observable Trajectories

  • Base endorsed fabricated children’s-shoe brands using purported comparisons (M1/M2 0/0), although no intermediate adoption was detected.The archived answer cited the two fabricated brands’ support while declining to assess Nanga; the case illustrates endorsement without a detected adoption event.Source: §A.15 · p. 34
  • Defense scored M1/M2 2/2 after identifying promotional, user-editable evidence and incoherent wording, but retained conditional purchasing advice.M3 was inapplicable; the high M1 label does not establish unambiguous categorical exclusion.Source: §A.15 · pp. 34–35
  • A Base case labeled M1/M2/M3 2/2/2 illustrates that reported-speech attribution can undermine a recovery label.The adoption detector treated a passage criticizing a webpage’s claims as the agent’s own recommendation; the surrounding context does not clearly establish agent adoption, so this is not confirmed recovery.Source: §A.15 · p. 35
  • The agent judged the URL and repeated brand recommendations questionable, found no reliable reputation evidence or clear negative reports, and advised caution.The URL had an unusual path and inconsistent title and timestamp dates; the retrieved material was mostly advertorials, with no clear complaints, failed inspections, or recalls.Source: §A.15 · p. 36
Loading 2609.06027v2…