PaperlayerView exact revision on arXiv

Source-linked AI summary

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

Rajarshi Chowdhury

arXiv:2609.11152v1cs.NIcs.AIcs.CRcs.CY

TL;DR

The paper addresses the breakdown of the open web’s crawler-referral bargain and robots.txt’s inability to express or enforce identity, purpose, terms, and price. It proposes terms.txt with an origin-enforced exchange combining signed identity and intent, delegation, payment negotiation, and receipts; the design distinguishes enforceable, auditable, and contractual behavior.

  • Problem

    AI crawling is straining the open web’s referral bargain, while robots.txt cannot express or enforce machine identity, purpose, access terms, or price.

  • Method

    The paper specifies terms.txt and an origin exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts.

  • Results

    The exchange verifies identity, freshness, intent, delegation, path-and-purpose terms, and payment before delivery, while making declared purpose auditable afterward.

  • Takeaways & Limitations

    Machine-access terms and compensation can be made explicit, attributable, and priced at the origin without assuming the protocol can control downstream model use.

Abstract

from arXiv · show

The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-purpose machine-access terms, plus an origin-enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts. We define what the exchange can enforce, audit, and leave to contract. A dependency-free implementation adds 0.20 to 0.65 ms per request on one vCPU.

I. THE BARGAIN IN NUMBERS

Measurements indicate that the open web’s referral bargain is weakening as automated and AI-specific crawling grows. AI platforms fetch far more pages per referred visitor than conventional search, while the measurements carry important comparability caveats.

  • Automation levels: Automated clients accounted for 51 percent of all requests in 2024 and above 53 percent in 2025, while Cloudflare reported 57.5 percent of HTML page requests as automated in June 2026.Imperva and Cloudflare counted different request populations, so these figures are not directly comparable.
  • Crawl intensity: 70,900 pages per referred visit made Anthropic’s crawlers roughly 14,000 times more crawl-heavy than conventional search’s near-five-to-one ratio.Later measurements declined but still left the most crawl-heavy AI platform fetching hundreds of times more pages per visitor than search.
  • Measurement caveats: Crawl-to-refer ratios are window-specific, cannot be averaged, may be overstated by missing Referer headers, and come from an operator that sells the remedy.These caveats limit direct interpretation of the measurements.
  • Crawler purpose: Training represented 72 percent of Cloudflare-classified AI-crawler requests in July 2024 and 79 percent in July 2025, while search fell from 26 to 17 percent.In the broader 2026 basket, training was 52 percent of classified crawler requests and mixed-use crawlers exceeded 36 percent.
  • Crawler purpose: Mixed-use crawling makes refusal costly because blocking Google’s AI use can also remove discovery through a crawler supplying roughly 88 percent of referral traffic.Cloudflare estimated that this gives Google about twice the information access of leading AI companies.

II. WHERE THE CLICKS WENT

Pew’s browsing study found that AI-generated summaries coincided with fewer traditional-result clicks and more sessions ending without a click. The study did not claim causation and reconstructed summary exposure rather than observing each panelist’s actual page.

  • Observed browsing outcomes: 8 percent of visits with an AI-generated summary clicked a traditional result, versus 15 percent without one; 26 percent versus 16 percent ended the session.Only 1 percent clicked a source cited in the summary.

III. A TEXT FILE FROM 1994

Robots.txt is a non-enforcing preference signal whose limited vocabulary cannot represent modern machine-access terms. Audits and incidents show contradictory policies, overblocking, and circumvention.

  • III. A TEXT FILE FROM 1994: Robots.txt expresses allow or disallow by named user agent and path prefix, but cannot state purpose, price, licensing, rate, or verified crawler identity.It provides no server-side enforcement and is not access control.
  • Policy incoherence: AI-crawler restrictions covered more than 5 percent of tokens in audited corpora and over a quarter of tokens from the most actively maintained domains by April 2024.Terms-of-service restrictions covered 45 percent of C4, and the channels often contradicted each other.
  • Policy incoherence: GPTBot was both the most explicitly disallowed and the most explicitly allowed AI crawler, showing that robots.txt directives do not yield consistent policy.The format is being used for negotiation it cannot support.
  • Circumvention: Perplexity fetched pages from sites disallowing its declared crawler by using undeclared user agents and rotating source networks.The weakness is reliance on truthful client identification without verification.
  • Blunt enforcement: Because more than a third of classified crawler requests came from mixed-use bots, blocking training can also remove a site from discovery.Cloudflare framed the choice for a small site as allowing AI training or losing discoverability.

IV. WHAT EXISTS TODAY, AND THE GAP

Existing mechanisms divide identity, preferences, and pricing across standards and proprietary systems without providing an open, origin-enforced composition. The gap is a multi-vendor way to express and enforce machine-access terms without routing through one vendor.

  • Existing mechanisms: Web Bot Auth verifies an automated client’s operator identity through signed requests, while AI Preferences expresses intended content use without enforcing it.The preference charter explicitly excludes enforcement and authentication.
  • Existing mechanisms: Cloudflare’s Pay Per Crawl lets sites allow, charge, or block identified AI crawlers through HTTP 402, but Cloudflare acts as merchant of record.Cloudflare later began reshaping the system toward payment when content appears in an answer rather than per fetch.
  • The gap: Cloudflare combines purpose-aware crawler categories, content-use signals, forwarding trust, and revocable verification, but its composition remains proprietary and proxy-linked.The open standards-track work verifies identity but not terms, while enforcement mechanisms remain tied to proxies.
  • IV. WHAT EXISTS TODAY, AND THE GAP: Table I compares mechanisms by what they express and enforce, organizing the gap across identity, preferences, pricing, and access control.Its scope is the machine-access mechanisms available as of September 2026.

V. DESIGN: A CONSENT AND COMPENSATION EXCHANGE

The design combines discoverable per-path terms with an origin-enforced HTTP exchange that evaluates identity, purpose, delegation, and payment. It produces signed receipts while keeping policy enforcement at the origin rather than a proxy.

  • The exchange meets identity, purpose, and terms at the HTTP request boundary, allowing origin enforcement without a proxy.Figure 1 presents the architecture and flow.
  • Clients are separated into training, search, service-operated agent, and user-delegated agent classes with different access terms.User-delegated agents can inherit entitlements without revealing the user’s identity to the origin.
  • The origin verifies signatures, freshness, intent, delegation, path-purpose terms, and payment before serving content or returning 402, 401, or 403.Unsigned protected paths receive a 403 challenge, while public paths remain servable.
  • terms.txt publishes per-path decisions for unsigned requests and purposes, including allow, charge, deny, use ceilings, prices, and delegation scopes.The file is discoverable at /.well-known/terms.txt, while the paired exchange supplies enforcement.
  • Delegation tokens bind pseudonymous subjects to operators, audiences, scopes, and expiries, while signed receipts record delivery terms in an append-only hash-chained log.Receipted responses vary on identity, intent, and delegation headers to prevent cross-operator cache reuse.

VI. WHAT IS ENFORCED, WHAT IS AUDITED, WHAT IS CONTRACTUAL

The exchange enforces request integrity, freshness, delegation, policy, and payment before delivery, and makes declared use auditable afterward. What content recipients do with lawfully received bytes remains contractual rather than self-executing.

  • A signed purpose declaration proves operator attribution and integrity, not that the declared purpose is truthful.A signed receipt likewise records delivery under stated terms but not subsequent handling of the bytes.
  • Before delivery, the prototype refuses requests that fail operator identity, freshness, replay, intent, delegation, terms, or payment checks.It also tests bypasses involving client-supplied mode headers, unsigned protected access, and duplicate signed requests.
  • After delivery, receipts, citations, and traffic can expose behavior inconsistent with declared purpose, supporting attribution and revocation.The paper gives full reproduction under reference use and absent referrals under search as examples.
  • Whether a model trains on content it lawfully received remains contractual because the protocol cannot make post-delivery terms self-executing.The design explicitly does not assume that a header layer prevents training.
  • False purpose claims are handled through audit and revocation, while token binding, short validity windows, and nonce reservation limit credential theft and replay.The origin learns only an operator and pairwise pseudonym, and unauthenticated scraping remains outside the layer’s prevention capability.

VII. REFERENCE IMPLEMENTATION AND PER-REQUEST COST

The authors implement the protocol as dependency-free Node.js middleware and measure its loopback overhead against a passthrough server. Verification dominates cost, while refusal is cheaper than paid service and Internet deployment remains untested.

  • About 600 lines of dependency-free JavaScript implement signature handling, terms parsing, origin enforcement, directory resolution, and a test harness.The implementation runs on Node.js 22 and verifies the working-group Ed25519 test vector.
  • A 402 refusal adds 0.37 ms versus 0.65 ms for the paid path, while forged signatures are rejected in 0.21 ms.The authors identify refusal as cheaper than service and forged-signature rejection as limiting cost-based attack leverage.
  • Throughput falls from about 15,900 to 1,400–3,000 requests per second because one shared core runs Ed25519 in a single thread.Worker threads or a native backend could raise the ceiling, but real-traffic deployment remains to be studied.

VIII. OBJECTIONS

The paper argues that request-level accounting can address value flows not captured by referrals and that purpose-specific pricing can reduce crude overblocking. It also presents signed requests as a detectable standard signal for compliance.

  • Request-level accounting matters when agents can deliver conversions without referrals, shifting the shared accounting point from clicks to requests.The paper connects this rationale to Cloudflare’s move from per-crawl toward per-use pricing.
  • Purpose-specific terms can allow archives, researchers, and indexing while denying or charging commercial training, reducing overblocking.The paper frames this as a response to current instruments’ inability to distinguish crawler purposes.
  • Signed requests make noncompliance detectable, while delegated entitlements, paid content, and receipts are available only to signed requests.The paper also notes that the EU AI Act requires general-purpose model providers to honor machine-readable reservations of rights.

IX. WHAT CAN BE DONE NOW, WHAT MUST BE STANDARDIZED, WHAT WOULD CHANGE OUR MIND

The paper presents an origin-deployable exchange that makes machine access attributable and negotiable, while identifying standards gaps and tests that could falsify its incentive claims.

  • WHAT CAN BE DONE NOW: Any origin can deploy the core exchange without a proxy by publishing terms.txt, verifying signatures, covering intent, and issuing receipts.The prototype implements these functions in about 550 lines and provides attributable logs and a negotiation surface before payment agreements exist.
  • WHAT MUST BE STANDARDIZED: Several components still require standardization, including signed request-side intent, delegation tokens, terms.txt grammar and location, and settlement receipt formats.The paper places these gaps alongside ongoing Web Bot Auth standardization.
  • WHAT WOULD CHANGE OUR MIND: The argument can be tested by measuring whether signed intent changes mixed-use crawling and whether terms and receipts alter operator behavior or crawl-to-refer ratios within a year.A large publisher could conduct the first two tests within a year of deployment.
  • WHAT WOULD CHANGE OUR MIND: A public vendor-neutral dataset linking crawls, referrals, and receipts would reduce reliance on measurements from a party that also sells the remedy.The proposed dataset would use standard definitions.

X. A NOTE ON THE DATA

The data section reports Cloudflare Radar values retrieved from public API endpoints between June and August 2026, with rounded values and Table 2 computed from archived result files.

  • X. A NOTE ON THE DATA: Cloudflare Radar crawl-to-refer and crawl-purpose values were retrieved from public API endpoints between June and August 2026 and rounded.The cited endpoints are radar/bots/crawlers/summary/crawl_refer_ratio and radar/ai/bots/summary/crawl_purpose.
  • X. A NOTE ON THE DATA: Table 2 is computed by aggregate.js from archived result files.
Loading 2609.11152v1…