PaperlayerView exact revision on arXiv

Source-linked AI summary

Consent in Crisis: The Rapid Decline of the AI Data Commons

Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, Sandy Pentland

arXiv:2407.14933v2cs.CLcs.AIcs.LG

TL;DR

The paper asks how web consent mechanisms handle public web data’s growing role in AI training. It audits the domains behind three open corpora over time, finding rapidly rising restrictions that may narrow and skew available training data.

  • Problem

    The paper examines how well existing web protocols communicate creators’ preferences as internet content is reused for AI training.

    Source: §1 · p. 2

  • Method

    The authors audit 14,000 domains underlying C4, RefinedWeb, and Dolma, tracking consent signals and related properties over time.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    ~5% of tokens in C4 and other major corpora have recently become restricted by robots.txt, while nearly 45% carry some Terms of Service restrictions.

    Source: §4 · p. 11

  • Takeaways & Limitations

    If restrictions are respected or enforced, available data may become less diverse, representative, and fresh, challenging data scaling for frontier capabilities.

    Source: §4 · p. 11

    The usage comparison relies on WildChat, which may not represent interactions and covers one specific ChatGPT instance.

    Source: §3.4 · p. 10

Abstract

from arXiv · show

General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AI-specific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research.

1 Introduction

The paper examines whether web consent protocols can communicate creators’ preferences as public web data is repurposed for AI. It audits the domains behind three open training corpora and identifies rapid restrictions, inconsistent signals, and mismatches between web data and conversational AI uses.

1 Introduction

  • In 2023–2024, robots.txt restrictions affected ~5%+ of tokens across C4, RefinedWeb, and Dolma, and ~25%+ from their most critical domains.The paper also forecasts a year-over-year decline in unrestricted web data.Source: §1 · p. 2
  • Websites increasingly restrict AI-related crawling in both robots.txt and Terms of Service, while the two mechanisms can contradict one another.The paper also reports developer-specific differences and omissions in robots.txt preference signals.Source: §1 · p. 2
  • The largest public web sources contain higher rates of user, multimodal, and monetized content than the long tail, but slightly less sensitive or explicit content.The head includes more news, encyclopedias, and social media; the tail includes more organizational, blog, and ecommerce sites.Source: §1 · p. 2
  • The paper finds a mismatch between web-crawled training sources and the real-world uses of conversational AI.The authors say this mismatch may have implications for alignment, data collection, and copyright.Source: §1 · p. 2

2 Methodology

The audit combines historical website records, domain samples from three open corpora, and human and automated annotation. It measures crawler restrictions and terms-of-service constraints across domains and over time.

2 Methodology

  • For each corpus, the researchers selected its top 2,000 domains by token count and formed a 3,950-domain union called HEADAll.The head sample represents the largest, most actively maintained, and critical domains for AI training.Source: §2 · p. 3
  • A separate random sample drew 10,000 domains from the three corpora’s intersection, with 2,000 selected for human annotation.The shared intersection may skew toward more widely used or higher-quality domains.Source: §2 · p. 3
  • Human annotators labeled website modalities, purposes, monetization, Terms of Service, and other metadata for HEADAll and RANDOM2k.The study also used an automatic pipeline to classify Terms of Service restrictions and crawler policies.Source: §2 · pp. 3–4
  • The audit tracks homepage, robots.txt, and Terms of Service versions monthly from January 2016 to April 2024 using the Wayback Machine.It records historical versions as well as present-day information.Source: §2 · pp. 3–4
  • The audit classifies robots.txt restrictions for AI developers and archival organizations, and defines restricted tokens by crawler or Terms of Service limits.For robots.txt, restricted tokens come from domains fully restricting at least one AI Organization crawler; for Terms of Service, they are unusable under terms precluding crawling or AI.Source: §2 · p. 4

3 Findings

The findings document a rapid rise in restrictions on web data, alongside inconsistencies in how websites signal consent for AI crawling. The most actively used web domains also differ from the long tail in composition and from observed ChatGPT uses.

3.1 The Rise of Restrictions on Open Web Data

  • By April 2024, 5–7% of full-corpus tokens and 20–33% of HEADAll tokens were restricted, up from about 1% and less than 3% a year earlier.The reported full-corpus shares cover C4, RefinedWeb, and Dolma; HEADAll comprises the largest, most actively maintained domains.Source: §3.1 · p. 5, Figure 2 · p. 6
  • Terms pages show a 26–53% relative increase in crawling restrictions in one year, and 45–55% of tokens across the three corpora have a Terms of Service data-use restriction.Terms pages specifically mentioning and restricting generative-AI use emerge more widely in 2024.Source: §3.1 · p. 6, Figure 2 · p. 6
  • By April 2025, forecasts add 2–4% absolute robots.txt restrictions and 6–10% absolute Terms of Service restrictions across full-corpus tokens.For the head distribution, the forecast also projects an additional 7–11% of highest-quality tokens becoming restricted.Source: §3.1 · pp. 6–7

3.2 Inconsistent and Ineffective Communication on AI Consent

  • OpenAI and Common Crawl agents are restricted in 91.5% and 83.4% of cases where any listed organization is restricted, while omissions and contradictory signals also occur.The paper reports unrecognized crawler names and frequent mismatches between robots.txt and Terms of Service, including anti-crawling terms without a robots.txt file.Source: §3.2 · pp. 7–8

3.3 Correlating Features of Web Data

  • HEADAll is dominated by news, social media/forums, and encyclopedias (72.9%), while RANDOM2k is dominated by personal or organization sites, blogs, and e-commerce (97%).The three corpora also differ in source composition despite deriving from Common Crawl snapshots.Source: §3.3 · p. 9
  • Compared with the tail, HEADAll domains have more ads (+47.5%), paywalls (+24.1%), restrictions, and image, video, and audio content.The reported differences are +22.5% for robots.txt restrictions, +35.3% for Terms of Service restrictions, +4.4% for image, +39.8% for video, and +38.4% for audio.Source: §3.3 · p. 9

3.4 Misalignment between Real-world AI Usage and Web Data

  • Over 30% of WildChat conversations request creative compositions, which are poorly represented in web training data; sexual role-play is 12% of interactions while sensitive content is under 1% of HEADC4 domains.News accounts for nearly 40% of HEADC4 tokens, but fewer than 1% of ChatGPT queries concern news or current affairs.Source: §3.4 · pp. 9–10
  • The WildChat comparison may not represent broader language-model use: it covers one ChatGPT instance, a potentially technical user base, and a specific collection period.The authors also note that their manually developed use taxonomy has limited granularity.Source: §3.4 · p. 10

4 Discussion

The discussion finds rapidly rising web-data restrictions and argues that, if respected or enforced, they would shrink and skew the sources available for AI. It also highlights unresolved consent protocols and possible effects on non-commercial research, content creation, and copyright analysis.

4 Discussion

  • ~5% of tokens in C4 and other major corpora became restricted by robots.txt in less than a year, while nearly 45% carry some Terms of Service restrictions.The availability of high-quality pretraining sources would rapidly diminish if restrictions are respected or legally enforced.Source: §4 · p. 11
  • Restrictions may skew training data away from news and forums, reduce diversity and representativeness, and bias it toward older, less fresh content.The discussion says this could challenge data scaling laws [46], even as declining consent protects content creators’ intentions.Source: §4 · p. 11
  • Separating retrieval-time crawling from training may help developers preserve attribution and freshness, but creators may see it as violating opt-out intent.The discussion says accusations of bypassing robots.txt opt-outs remain unconfirmed.Source: §4 · p. 11
  • The REP burdens website owners with anticipating crawler agents; the discussion proposes signaling permitted downstream uses instead of specifying agents.Suggested categories include search, non-commercial AI, and AI that attributes outputs to sources; extended restrictions could help protect data freshness.Source: §4 · p. 11
  • Wildcard blocks and restrictions on Common Crawl or the Internet Archive can also constrain non-commercial AI and academic research.Common Crawl is reported to be cited in 10,000+ research articles across fields.Source: §4 · p. 11
  • Small content providers may leave the open web or move behind logins or paywalls if they lack resources to protect against unwanted crawling.The discussion links these possible responses to economic fears where generative AI competes with original content sources.Source: §4 · p. 12
  • News domains dominate training sources, while ChatGPT use is more often creative writing, role-play, brainstorming, or general information than news.The authors caution that this mismatch does not establish a stronger case for fair use and requires further study.Source: §4 · p. 12

5 Related Work

Prior work audits AI training-data provenance, quality, bias, and characteristics across text and multimodal datasets, and studies legal, data, and crawler issues. This work identifies limited research on how AI-data consent signals change over time.

5 Related Work

  • Prior audits examine provenance, quality, bias, and characteristics of pretraining text [32] [55] [57], finetuning text [66], and multimodal datasets [11] [13] [12] [109].Other work addresses data-development challenges [89].Source: §5 · p. 12
  • Related research studies non-copyrighted data [77], fair-use implications [44] [61], future data constraints, and evolving crawler behavior.Additional work examines web link decay and Common Crawl collection.Source: §5 · p. 12
  • The authors identify little prior work measuring how consent signals for AI data evolve over time.News reports on crawler blocking are described as an initial foundation for more rigorous analysis.Source: §5 · p. 12

6 Conclusion

The conclusion presents a 14,000-domain audit of changing crawlable content and consent norms, finding weaknesses in how web protocols signal or enforce preferences. It also reports mismatches between documented AI uses and the data underlying those systems.

6 Conclusion

  • The audit covers 14,000 web domains and tracks changing crawlable content and consent norms for sources underlying general-purpose AI training corpora.The authors describe it as the first large-scale audit of these web sources.Source: §6 · p. 12
  • Inconsistencies and omissions between robots.txt and Terms of Service suggest the data ecosystem is ill-equipped to signal or enforce preferences.Source: §6 · p. 12
  • The audit uncovers distributional mismatches between documented real uses of AI systems and their underlying data.The authors release their collected annotations and analysis for further investigation of provenance, consent, and composition.Source: §6 · p. 12

Impact & Ethics Statement

Website consent signals have practical and ownership limits: administrators may not represent copyright holders, and the legal and ethical rules for enforcing preferences remain unresolved. The paper describes these signals without prescribing answers to those broader questions.

Impact & Ethics Statement

  • Website administrators are often imperfect proxies for copyright holders, especially where sites host content created or owned by others.The authors also note that tools are insufficient to attribute all content or distinguish consenting from non-consenting uses.Source: Impact & Ethics Statement · p. 13
  • Robots.txt and Terms of Service are practical status-quo signals, but have limits in ownership and in communicating intentions effectively.Source: Impact & Ethics Statement · p. 13
  • Which data-preference signals should be enforceable, and how, remains an open legal and ethical question; the authors do not prescribe answers.They describe the evolving signals and leave adherence to readers, developers, and legislators.Source: Impact & Ethics Statement · p. 13

Contributions

The contributions list assigns leads and team members across the paper’s annotation, data-collection, analysis, legal, visualization, and writing work. Contributors are listed alphabetically, except that team leads appear first.

Contributions

  • Contributors are listed alphabetically, with team leads placed first.Source: Contributions · p. 14
  • The listed work spans annotation design and review, data and Wayback collection, longitudinal and comparative analyses, forecasting, legal analysis, visualizations, and writing.The list names leads and contributors for each workstream.Source: Contributions · p. 14

A Human Annotation Methodology Details

The appendix describes crowdworker training and compensation, then details how authors annotated website characteristics and content. Its guidelines distinguish a site’s functional service from its content domain and specify labels for user content and other attributes.

A.1 Details on Crowdworkers

  • Fourteen crowdworkers from six countries annotated 14,228 rows, with iterative training, regular quality checks, and pay of approximately $25–$30 per hour.The total payment was $6,972, averaging $498 per worker.Source: §A.1 · p. 26

A.2 Human Annotation Guidelines

  • The guidelines cover annotations conducted by authors and crowdworkers for pretraining-data collection.Source: §A.2 · p. 26

A.2.1 Web Source Annotations (Authors)

  • User-content labels distinguish weakly moderated third-party content from strongly moderated or absent user content, which the paper groups together as not significant.Examples of weakly moderated sites include social media, forums, and review sites.Source: §A.2.1 · p. 26
  • Authors annotated website purpose and content domain separately, developing service categories after reviewing and clustering hundreds of websites.Service type concerns website function, while content domain describes topics; the service annotation was intended to relate site function to user conversations and copyright-market questions.Source: §A.2.1 · p. 27
  • The annotation guidelines cover sensitive content, including pornography, drugs, violence, illegal activity, and hate speech, and exclude sites that were sold or shut down.The sensitive-content label applies when such material is more than minimal.Source: §A.2.1 · pp. 26–28

A.2.2 Pretraining Datasets (Crowdworker)

The pretraining-dataset annotation instructions ask annotators to inspect sites and classify monetization, content modalities, advertisements, and relevant terms links. They provide operational criteria and suggested checks for these labels.

A.2.2 Pretraining Datasets (Crowdworker)

  • Annotators were instructed to inspect websites thoroughly, including pages beyond the homepage, to find features such as ads, paywalls, video, and audio.Source: §A.2.2 · p. 28
  • Terms-link annotation seeks relevant use, data, content, privacy, and copyright policies, with checks for links and searches to avoid missing main terms pages.Source: §A.2.2 · pp. 28–29
  • Paywall labels distinguish no paywall, some content paywalled, and all main content paywalled.Suggested checks include browsing while logged out and searching whether the site has a paywall.Source: §A.2.2 · p. 29
  • Content modalities are text, images, videos, and audio; annotators should not count automatic advertisements as site content.The guidelines provide criteria for identifying text, images, and embedded videos.Source: §A.2.2 · pp. 29–30
  • Advertisement labels record whether automatic third-party ads appear, excluding self-promotion, and direct annotators to search site pages for them.Source: §A.2.2 · p. 30

B Automatic Annotation Methodology Details

The appendix details how the audit classified robots.txt and Terms of Service restrictions and used GPT-4o to extract policy evidence. It also describes the agents analyzed and prompt refinement.

B.1 Robots.txt Taxonomy

  • Robots.txt restrictions were classified from no file or unrestricted access through sitemap, crawl-delay, search/query and directory limits to full-agent blocks.The audit tracked wildcard rules and agents for six organizations commonly associated with AI training.Source: §B.1 · pp. 30–31

B.2 Robots.txt Agents

  • OpenAI agents are nearly always disallowed when any AI organization is disallowed, while the reverse is less frequent.Figure 5 compares conditional restriction percentages across AI companies, archives, and Google Search.Source: §B.2 · p. 31, Figure 5 · p. 33

B.3 Terms of Service Taxonomy

  • Terms of Service categories distinguish commercial-use permissions, crawling and AI restrictions, and clauses against competing services or redistribution.The taxonomies use multiple levels of restrictiveness to represent nuanced clauses rather than a binary policy label.Source: §B.3 · pp. 34–35

B.4 Prompt engineering

  • GPT-4o prompts returned a taxonomy verdict with directly quoted evidence, and iterative refinement raised average accuracy against annotations to at least 85%.Prompt revisions targeted false negatives and false positives.Source: §B.4 · p. 35

B.5 Annotating and scoring

The authors manually evaluated GPT-4o’s WildChat service-type predictions and audited ToS annotations against human-reviewed examples. The section describes the service categories and ToS scoring approach.

B.5 Annotating and scoring

  • For each ToS indicator, 100 randomly sampled URLs were manually reviewed, and micro-average precision and recall were computed against the resulting annotations.The indicators were competing services, license type, and crawling and AI policies; Table 7 reports the scores.Source: §B.5 · p. 38, Table 7 · p. 38

B.6 WildChat Annotation

  • WildChat service categories include creative and academic composition, coding, planning, explanation, self-help, sexual content, news, e-commerce, translation, and organization information.Prompts could be assigned one or more categories.Source: §B.6 · pp. 38–39
  • 18% of GPT-4o’s service-type predictions were errors in a manual review of 50 randomly sampled WildChat prompts.The evaluation assessed whether each prompt’s predicted service type was correct.Source: §B.6 · p. 39

C Forecasting Method

The forecasting method uses SARIMA to model the share of restricted tokens across the three corpora. The related-work section reviews data documentation, audits, web changes, and transparency risks.

C Forecasting Method

  • The SARIMA forecast uses non-seasonal order (2, 1, 2) and seasonal order (1, 1, 1, 6), selected by optimizing the Akaike information criterion.The seasonal order reflects a recurring pattern every 6 periods, and both components are differenced once.Source: C Forecasting Method · pp. 39–40, Table 8 · p. 40

D Extended Related Work

  • Prior documentation work proposes datasheets, data statements, data cards, and model cards for disclosing dataset or model information [78].The discussion also cites frameworks for dataset life-cycle navigation and documentation standards [28] [68].Source: D Extended Related Work · pp. 40–41
  • Earlier audits examine training datasets and modalities, including Common Crawl text corpora and the geo-diversity of vision datasets [22] [13] [11] [40] [69].The section also notes dataset-finding tools and the growing practice of combining and repackaging data sources [121] [38] [91] [123] [65].Source: D Extended Related Work · p. 41
  • Prior web audits report that 38% of webpages present in 2013 were inaccessible a decade later, while studies found GPTBot blocking more common than Google-Extended blocking.A survey of news publishers found 53%+ blocked GPTBot and 41.3% blocked Google-Extended.Source: D Extended Related Work · p. 41
  • The related work links weak data transparency to reduced understanding of training data and risks including leakage, PII exposure, and bias [33] [17].It also notes data-governance efforts such as BigScience and the Public Data Trust [49] [24].Source: D Extended Related Work · p. 41

Related papers

Loading 2407.14933v2…