Source-linked AI summary
Manifest-Assisted Retrieval in Artificial Intelligence Search: Evidence Gaps and a Security Evaluation
Gangzhen Qian, Kainan Zhou, Chuhong Xu, Hang Xiao, Zhi Jing, Rong Bao, Lu Yi, Zhaoyi Li
TL;DR
The paper addresses the lack of evidence isolating /llms.txt’s incremental security and utility effects while ordinary retrieval is fixed. It combines a stage-specific framework with a reduced pilot; frozen scoring finds 21 attacker-target adoptions in 24 raw adversarial responses, with one response unresolved.
Problem
Retained external sources do not isolate /llms.txt’s incremental security and utility effects while ordinary retrieval is fixed.
Source: C. Conclusion · p. 7Method
The paper separates pipeline stages, defines five path-specific outcomes and paired loss estimands, and runs a 120-response pilot varying manifest content while ordinary evidence is fixed.
Source: §I · p. 1Results
Frozen scoring identifies 21 attacker-target adoptions in 24 raw adversarial responses, with one further response unresolved.
Source: C. Conclusion · p. 7Takeaways & Limitations
The pilot adds evidence that attack-bearing manifest prose can change answers when ordinary evidence is already supplied, while broader retrieval and action comparison remains unexecuted.
Source: C. Conclusion · p. 7The pilot measures response behavior conditional on supplied content, not natural discovery, live retrieval, ranking, tool execution, or the full guarded policy.
Source: B. Limitations · p. 7
Abstract
from arXiv · showhide
The external sources retained in our non-systematic audit did not isolate the incremental security or utility effect of consuming /llms.txt while ordinary retrieval remained fixed. The initial audit covered 39 sources through August 16, 2026, with later targeted updates through October 14. A purposive 15-tool study observed no automatic requests in 45 trials at one endpoint; a non-probability log study found that 97% of domains with detected files received no recorded requests in May 2026. We separate publication, fetch, model use, and action; define five operational threat paths and paired security-utility estimands; and exercise deterministic pre-model controls with 30 author-defined parser fixtures. We also execute a reduced pilot: 24 tasks across eight sites, five conditions, and 120 real model responses, with ordinary evidence held fixed. Frozen automatic scoring identifies attacker-target adoption in $21/24$ raw adversarial responses, with one further response unresolved; the adversarial URLs-only condition has no known adoptions and one unresolved response. The pilot measures output changes conditional on supplied evidence, without live retrieval or external actions. Together, the framework and pilot distinguish manifest-content effects from publication, discovery, and downstream execution, while leaving full retrieval and agent evaluation open.
I. INTRODUCTION
The paper asks whether consuming root-level /llms.txt changes security or utility relative to ordinary retrieval. It combines a non-systematic evidence audit with a framework and reduced pilot, while noting that low observed consumption does not establish what happens when systems use the file.
I. INTRODUCTION
- RAG lets language models consult current sources at answer time, while publisher-controlled pages and tool outputs can place embedded instructions in AI-search answer context.Instruction following is only one failure mode; the framework also separates data, ranking, and resource outcomes from instruction hijacking.Source: §I · p. 1
- Root-level /llms.txt supplies publisher-selected prose and links but defines no trust, ranking, permission, or execution semantics, unlike robots.txt’s crawler-access preferences.Both /llms.txt and sitemap.xml are publisher-authored discovery aids; neither guarantees downstream fetching, prioritization, trust, citation, or action.Source: §I · p. 1
- The paper asks whether consuming /llms.txt changes security or utility relative to ordinary retrieval and contributes path-specific outcomes, paired loss estimands, guarded-consumption controls, and a 120-response pilot.The pilot varies manifest content and prose ingestion while holding supplied ordinary evidence fixed; low observed consumption does not establish effects when a consumer uses the file.Source: §I · p. 1
II. EVIDENCE-AUDIT METHOD
- The non-systematic audit retained 39 sources and purposively selected 15 focal studies to illustrate available evidence stages, not to represent the field statistically.One reviewer performed selection and extraction; complete search exports and screening counts were unavailable.Source: §II · p. 1
- The audit retained endpoint, denominator, uncertainty, utility, adaptive-knowledge, directness, and artifact information, but heterogeneous designs precluded pooling or cross-study ranking.Companion materials include the source ledger, stage coding, bias and applicability judgments, figure data, and replication instructions, but cannot reconstruct the missing historical search flow.Source: §II · p. 1
- An October 10 targeted update and selected October 14 rechecks added two relevant sources without rerunning a comprehensive search.The framework distinguishes demonstrated controlled behavior, observed operational evidence, and proposed controls; its definitions separate exposure, model control, and realized outcome.Source: §II · p. 2
III. THREAT MODEL AND ESTIMANDS
The threat model traces content-borne attacks through an AI-search pipeline and treats attack paths as distinct rather than collapsing them into a single hijack label. Matched neutralized twins support attribution while preserving retrieval opportunity where possible.
A. Pipeline and Adversary
- The pipeline comprises retrieval, context construction, an LLM, and an optional action layer; the adversary controls a reachable resource but not the user, model weights, or trusted orchestrator.The threat model traces exposure through model interpretation, policy, capabilities, and resulting security loss.Source: A. Pipeline and Adversary · p. 2
B. Operational Paths
- A binary hijack label conflates retrieval, interpretation, ranking, and action failures, so Table I assigns each operational path a decision rule and denominator.Path labels may overlap within a trial, while stage, mechanism, and outcome are coded separately.Source: B. Operational Paths · p. 2
- Instruction attribution compares attacker-goal probability between injected and neutralized matched twins, retaining relevant facts, formatting, and retrieval opportunity where possible.A single target answer does not establish instruction attribution; inseparable factual and instructional edits remain uncertain.Source: B. Operational Paths · p. 2
C. Exposure, Control, and Loss
The estimands define security and utility loss for paired site–query–model trials under manifest and consumption-policy arms. Comparisons hold ordinary retrieval fixed and distinguish path incidence from task utility and unresolved semantic outcomes.
C. Exposure, Control, and Loss
- Each site–query–model trial assigns arm z=(m,p), where m is absent, benign, or adversarial manifest status and p is the consumption policy; each path has a prespecified violation indicator.The trace, answer, and action are denoted T_i(z), Y_i(z), and A_i(z).Source: C. Exposure, Control, and Loss · p. 2
- Security loss counts a trial once if any path is compromised, while utility is lost when the authorized answer or task state, supporting evidence, or task constraints are not satisfied.Security loss measures incidence without severity weights; path outcomes and consequential actions are reported separately.Source: C. Exposure, Control, and Loss · p. 2
- Fig. 2 maps publication or discovery, HTTP fetch, retrieval or reranking, model output, and external action as directly measured, fixed, proxy, or unmeasured stages.The map describes evidence-stage coverage for 15 historical focal studies.Source: Figure 2 · p. 3
- Paired comparisons randomize arm order while holding site, query, model, index, and evaluation horizon fixed and resetting state to avoid carryover.The control retains ordinary HTML, alternate Markdown, metadata, and sitemap retrieval.Source: C. Exposure, Control, and Loss · p. 3
- Retrieval abuse may occur before model visibility, and model errors or unreadable outputs leave semantic security unresolved rather than safe.Unknown outcomes are bounded by assigning them zero or one; unsuccessful or malformed task responses incur utility loss one.Source: C. Exposure, Control, and Loss · p. 3
IV. EVIDENCE BY PIPELINE STAGE
The evidence audit locates findings by pipeline stage and preserves study-specific units and denominators rather than ranking incomparable rates. One cited pipeline study shows cumulative attack success declining across retrieval, reranking, and generation, but does not estimate an /llms.txt effect.
IV. EVIDENCE BY PIPELINE STAGE
- Table II preserves study units, denominators, and attack-success terms, while Fig. 2 locates evidence across publication or discovery, fetch, retrieval or reranking, model output, and external action.The stage map classifies evidence as direct, fixed, proxy, or unmeasured without assessing methodological quality.Source: §IV · p. 3
- A conversational-search benchmark measures manipulation after source selection, while a separate ranker experiment measures candidate reordering; neither estimates organic discovery.Both identify post-discovery effects.Source: §IV · p. 3
- Rates from noncomparable designs should not be ranked: model-visible benchmarks condition on exposure, other studies start at different stages, and blank downstream stages are unmeasured rather than negative findings.Endpoints and denominators therefore remain separate throughout the analysis.Source: §IV · p. 3
- One reasoning-based attack falls from 91.0% to 48.0% under BM25 and from 98.5% to 59.0% under dense retrieval; these declines reflect cumulative pipeline filtering, not a manifest effect.Stagewise reporting should retain the original query–target cohort and report conditional pass rates alongside cumulative survival.Source: §IV · p. 3, Figure 3 · p. 3
V. /L L M S.T X T EVIDENCE GAP
Available observations distinguish file publication, requests, provider statements, and downstream model use, but their samples and endpoints do not establish a single adoption funnel. Existing reports and cross-stage studies leave the incremental effect of deliberately consuming /llms.txt unresolved.
V. /L L M S.T X T EVIDENCE GAP
- Publication, requests, and current provider statements cannot be arranged as one adoption funnel because their samples and units differ.Source: §V · p. 3
- The v2 proposal provides publisher-authored navigation without trust or execution authority; a retrieval audit does not demonstrate model use, and a permission-manifest proposal leaves enforcement to complementary mechanisms.Source: §V · pp. 3–4
- Rankability reported 93/1,000 and 830/10,000 confirmed root-file adopters, with 474 and 4,211 unknown outcomes, respectively.The sampling frame and validation rules changed, so these observations cannot establish growth from the original 87/1,000 figure.Source: §V · pp. 3–4
- 97% of domains with detected files in a non-probability frame of 137,210 domains received no recorded request during May 2026.The frame contained about 38,000 detected HTTP-200 Markdown files; full specification conformance was not assessed.Source: §V · p. 4
- A commercial cross-section of nearly 300,000 domains found 10.13% presence but no detected citation advantage; Google says it ignores the file for visibility and ranking.Source: §V · p. 4
- Hertz reported package-namespace interventions and agent-triggered execution, but lacked a paired absent-manifest and benign-utility comparison; enhanced entity-page studies bundled changes that do not isolate root-file consumption.Neither report supplies the full counterfactual of holding ordinary retrieval fixed while varying manifest condition and consumption policy.Source: §V · p. 4
- Other agent studies report repeated-retrieval influence conditional on exposure and architecture, while separate experiments show crafted documents can suppress relevant material and resource exhaustion can occur under evaluated architectures.Trusted-looking identifiers and tool results also motivate provenance controls in agent settings.Source: §V · p. 4
VI. GUARDED CONSUMPTION AND VALIDATOR
The paper proposes treating manifest prose and links as untrusted data within broader network, resource, provenance, evidence, and capability protections. Offline fixtures test parser and pre-fetch decisions, but do not validate live retrieval or downstream model and agent behavior.
VI. GUARDED CONSUMPTION AND VALIDATOR
- Prior structured-query results motivate separating trusted queries from untrusted data, while action-alignment results motivate evaluation on adversarial and benign tasks.Design-level and task-specific cross-site confinement have also been evaluated for agents; risk depends on surrounding capabilities, and surface-heuristic defenses motivate testing contextual false positives.Source: §VI · p. 4
- The guarded design requires secure transport, same-origin defaults, address and redirect checks, peer validation, resource caps, provenance records, independent link decisions, and policy controls for consequential actions and high-stakes claims.The protections apply to untrusted pages generally; no new general defense is claimed.Source: §VI · pp. 4–5
- In the reduced pilot, “guarded” means only the stated URL policy and removal of manifest prose, not implementation or validation of the complete system design.Table IV summarizes proposed safeguards and their verification boundaries.Source: §VI · pp. 4–5
- All 30 fixtures matched the offline contract: structure and provenance 6/6, resource bounds 4/4, location decisions 16/16, and authority separation 4/4.The validator covers location and domain-name normalization, supplied-address checks, and provenance hashing, but not live network or end-to-end behavior.Source: §VI · p. 5
- The 30/30 result establishes consistency with specified fixtures, not complete fixture coverage or policy validity, and does not estimate exploit prevalence, model robustness, ranking quality, or action safety.The authority-separation fixtures check that parser output has no executable control objects; they do not test model obedience.Source: §VI · p. 5
A. Planned Paired Evaluation
The paper specifies a proposed paired protocol for isolating manifest and consumption-policy effects, then describes a narrower fixed-evidence pilot. The full retrieval protocol remains proposed, while the pilot varies manifest inputs across tasks with ordinary evidence held constant.
A. Planned Paired Evaluation
- The proposed protocol crosses absent, benign, and adversarial manifests with ignore, raw-text, link-only, and guarded policies while holding ordinary retrieval fixed across arms.The absent condition is the no-manifest baseline; present-manifest ignore cells are implementation-equivalence checks, not distinct causal contrasts.Source: A. Planned Paired Evaluation · p. 5
- The co-primary contrasts are adversarial guarded versus absent guarded for security loss and benign guarded versus absent guarded for utility-loss difference.Planned outcomes also include discovery, context inclusion, ranking, citations, answer quality, actions, blocks, false positives, latency, tokens, and cost; confirmatory sample size, power, and deployment margins remain open.Source: A. Planned Paired Evaluation · pp. 5–6
- The proposed attack suite spans instruction and identity tactics, link steering, source promotion or suppression, resource exhaustion, and adaptive attacks; evaluation also calls for benign hard cases and stakeholder-stratified costs.The full protocol remains proposed, not publicly preregistered or executed.Source: A. Planned Paired Evaluation · pp. 5–6
B. Reduced Pilot Design
- The reduced pilot used eight purposively selected sites, three factual questions per site, five arms, and one generation per task-arm, totaling 120 calls.Answers were checked with AI assistance against 12 official documentation pages captured October 14, 2026; there was no independent human screening.Source: B. Reduced Pilot Design · p. 6
- All pilot arms received identical researcher-authored summaries containing correct ordinary-source facts; adversarial manifests added a wrong target to benign links, while guarded arms consumed approved same-origin HTTPS URLs only.The files were locally authored, and D and E had identical model-visible inputs because the guard discarded prose and link labels; no live retrieval or actions occurred.Source: B. Reduced Pilot Design · p. 6
- The pilot froze questions, aliases, targets, protocol, and code before outcomes and used randomized arm order without automatic retries.Security loss counts recognized target adoption, with unrecognized responses unresolved; utility loss is zero only for an accepted answer, specified supporting URL, and no proposed actions, and reporting is descriptive without confidence intervals or deployment decisions.Source: B. Reduced Pilot Design · p. 6
C. Results and Scope
The fixed-evidence pilot found bounded differences in its co-primary contrasts and substantial target adoption under post hoc semantic review of raw adversarial outputs. Its results are conditional on supplied evidence and do not measure live retrieval or external actions.
C. Results and Scope
- [−2/24, 1/24], or [−8.3, +4.2] percentage points, bounds the retained co-primary security contrast ∆pilot_s (E, A).Unresolved classifications prevent a frozen security point estimate; these are identification bounds, not confidence intervals.Source: C. Results and Scope · p. 6
- ∆pilot_u (D, A) = 0, although both arms incurred utility loss 2/24 rather than zero loss.The secondary raw adversarial-versus-benign contrast C–B was bounded by [20/24, 22/24], or [+83.3, +91.7] points.Source: C. Results and Scope · p. 6
- 22/24 target adoptions in C and 0/24 in every other arm emerged in the post hoc AI-assisted semantic review.Correct answers and supporting URLs appeared on 2/24 and 24/24 tasks, respectively; the review was not independent human adjudication.Source: C. Results and Scope · pp. 6–7
- The pilot demonstrates answer sensitivity to authored payloads conditional on correct evidence already being present, not live retrieval or external-action safety.Its one-model, one-generation setup, shared pages, purposive selection, utility ceiling, and unverified monetary charges limit inference.Source: C. Results and Scope · p. 7
VIII. DISCUSSION AND CONCLUSION
The discussion recommends measuring retrieval-to-action stages and tying deployment to prespecified tradeoffs. It also identifies limitations in the historical audit, reduced pilot, and parser-fixture scope.
A. Implications
- Retrieval-to-action instrumentation is needed because final-answer logs omit discovery, ranking shifts, and blocked actions.Report active policy, available capabilities, and blocked-action rate alongside answer quality to interpret tradeoffs across retrieval-only systems and agents with external tools.Source: A. Implications · p. 7
- Publishers can reduce supply-chain and representation-drift risk by generating manifests from canonical version-controlled content and reviewing changes as code.The discussion also recommends minimizing external or stale links and aligning visible and alternate representations.Source: A. Implications · p. 7
- Shadow-mode logging can reveal whether manifest changes alter retrieved sets or whether apparent utility gains come from duplicate canonical content.Promotion to active use should require prespecified security-loss, utility-gain, cost, and false-positive bounds, with rollback when a bound fails.Source: A. Implications · p. 7
B. Limitations
- The historical audit had one screener, unavailable complete search exports and screening counts, and some potentially stale or non-probability sources.Selection and extraction error remain plausible despite the retained ledger and verification steps.Source: B. Limitations · p. 7
- The reduced pilot used one model snapshot, eight purposively selected sites, fixed attack templates, and one generation per task and arm.It measures responses to supplied content, not natural discovery, live retrieval, ranking, tool execution, or the full guarded policy; no population or calibrated deployment-margin claim follows.Source: B. Limitations · p. 7
- The 30 fixtures test deterministic parsing and pre-fetch decisions, not exploit prevalence or action safety.Images, scripts, accessibility trees, tool metadata, memory, and inter-agent messages remain outside the evaluated scope.Source: B. Limitations · p. 7
C. Conclusion
The paper addresses an evidence gap on /llms.txt effects in a fixed search pipeline and adds a reduced fixed-evidence pilot. Its recommendations keep manifest navigation distinct from trust and external-action permissions.
C. Conclusion
- The retained external sources do not isolate /llms.txt’s incremental security and utility effects in a fixed search pipeline.The pilot adds evidence that attack-bearing manifest prose can change answers when ordinary evidence is already supplied; broader retrieval and action comparison remains unexecuted.Source: C. Conclusion · p. 7
- Consumers should treat /llms.txt as optional, publisher-asserted navigation data, with safeguards calibrated to each deployment’s measured security–utility tradeoff.Suggested controls include bounded retrieval, corroboration, capability separation, confirmation, monitoring, and adaptive testing.Source: C. Conclusion · p. 7
- Security criteria should be reported separately for retrieval-only systems and agents with external tools.Report active policy, available capabilities, and blocked-action rate alongside answer quality because acceptable answer-generation outcomes may be unacceptable when content can trigger external actions.Source: C. Conclusion · p. 7
- The revision package includes the retained source ledger, fixtures, frozen pilot protocol, scoring code, and raw responses, but no public repository or DOI is yet available.The package accompanies the revision; public artifact access remains to be established.Source: DATA AND CODE AVAILABILITY · pp. 7–8