PaperlayerView exact revision on arXiv

Source-linked AI summary

Host-Aware Web Crawling for LLM Pretraining Data

Malte Ostendorff

arXiv:2610.13321v1cs.CL

TL;DR

The paper asks how a crawler should allocate its limited budget across and within hosts to collect high-quality pretraining data. It combines document-level classifiers, graph metrics, and crawl-coverage data to compare host-ordering and depth-allocation policies. In simulations, sampling a few pages and then deepening high-scoring hosts substantially outperforms uniform depth, while Common Crawl’s observed coverage is uncorrelated with quality.

  • Problem

    The paper asks how crawlers should allocate a limited budget across and within hosts for high-quality pretraining data, given that pipelines discard many crawled pages.

    Source: §1 · p. 1

  • Method

    The study combines document-level classifier scores, web-graph centrality, and Common Crawl coverage to evaluate host ordering and crawl-depth policies.

    Source: §1 · p. 1, §3.4 · p. 4

  • Results

    Sample-then-deepen reaches up to 7.8 times the yield of uniform breadth, while Common Crawl coverage is uncorrelated with quality (Spearman -0.08) under every classifier.

    Source: §6 · p. 9

  • Takeaways & Limitations

    A few sampled pages can guide deeper crawling toward higher-scoring hosts; centrality may help group hosts for sampling but should not alone judge their content quality.

    Source: §1 · p. 2, §6 · p. 9

    Classifier-based allocation may reduce language, regional, or content-type representation, so higher yield need not mean a more representative or diverse corpus.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more high-quality pretraining data. Combining document-level quality classifier scores, web-graph centrality metrics, and Common Crawl's per-host crawl coverage, we find that roughly half of the variation in classifier scores occurs between hosts. In the crawls we analyze, a few pages from a host also provide a useful estimate of the quality of its remaining pages. Common Crawl uses harmonic centrality, a graph centrality metric, to guide its crawl allocation. In our simulations, using centrality to choose which hosts to crawl yields limited gains over random ordering and performs worse for some classifiers. For deciding how deeply to crawl each host, sampling a few pages and then fetching more from hosts with higher-scoring samples collects several times more high-quality pages per fetch than uniform depth or centrality-based allocation. Finally, Common Crawl's observed per-host coverage has little correlation with the evaluated quality signals. We estimate that tens of millions of high-quality pages remain among known but uncrawled URLs in each crawl examined, with most concentrated on a small share of hosts.

1 Introduction

The paper asks how crawlers can allocate a limited budget across and within hosts to collect more high-quality pretraining data. It finds host-level quality signal, limited gains from centrality-based ordering, and stronger depth allocation from sampling hosts before fetching more.

1 Introduction

  • Because pretraining pipelines discard many crawled pages as low quality, crawling policy is also a data-selection decision.The introduction notes that crawling has received comparatively little attention in pretraining data-selection research.Source: §1 · p. 1
  • The study asks how to allocate crawl budgets across and within hosts, and whether Common Crawl’s allocation aligns with pretraining quality selection.It contrasts host-level allocation with page-level schedulers that fetch, extract, and classify candidate pages for their scheduling signal.Source: §1 · p. 1
  • Classifier-score variation is moderately concentrated by host, with ICC1 values of 0.43–0.62, motivating a host-aware policy and informing its sample size.The introduction frames this as evidence that a small host sample may characterize the remaining pages.Source: §1 · p. 1
  • Graph-centrality ordering yields limited gains over random ordering and performs worse for some classifiers, with language and heterogeneous hub composition explaining the negative result.The analysis spans ten classifiers.Source: §1 · p. 1
  • A three-page sample can guide deeper crawling toward hosts with higher-scoring samples, while centrality-biased depth provides little to no improvement.The sample-then-deepen policy approaches the oracle policy.Source: §1 · pp. 1–2
  • Common Crawl’s per-host crawl budget is uncorrelated with quality under every tested classifier, leaving tens of millions of known high-quality URLs uncrawled on identifiable hosts.The paper examines whether observed crawl allocation aligns with pretraining quality.Source: §1 · p. 2

2 Related Work

Prior work uses content classifiers and graph signals for pretraining-data selection or crawl scheduling, but the paper studies host ordering and depth allocation during crawling. It relates its findings to earlier negative results for centrality while distinguishing its host-level setting.

LLM Pretraining Data

  • Pretraining pipelines commonly use content-internal classifiers, including fastText models trained on English reference corpora and multilingual classifiers, rather than host or graph features.The passage identifies FineWeb-Edu as using a more compute-intensive Transformer model and notes domain blocklists and social-graph seeds as graph-adjacent signals.Source: §2 · p. 2

Graph Signals in Web IR

  • Classical graph methods and newer graph models use link structure for tasks such as spam detection and source credibility, while LinkBERT mines QA pairs from Wikipedia hyperlinks.The passage distinguishes these uses from document filtering.Source: §2 · p. 2

URL-level Crawl Policy

  • Craw4LLM’s indegree-prioritized crawl reached 69.5% of oracle downstream performance, versus 78.1% for random and 95.3% for fastText-driven crawling.The authors report a weak indegree–DCLM-score correlation and reproduce a negative overall HC correlation on Common Crawl, while finding PR positively correlated with quality within TLDs.Source: §2 · p. 2

Graph-based Data Selection

  • The paper evaluates host ordering and crawl-depth allocation under a fixed fetch budget, unlike WebGraphMix’s evaluation of mixtures of already-collected documents.It notes that DCLM found no PageRank bin outperformed random sampling and suggests pooling hosts across TLDs may obscure within-TLD correlations.Source: §2 · p. 2

3 Data and Methods

The study combines classified WARC samples, Common Crawl graph data, and full-crawl host-index and quality annotations to analyze host-level variation, crawl yield, crawl depth, and coverage. It compares allocation policies using fixed-fetch-budget yield and evaluates observed coverage against host quality density.

3 Data and Methods

  • The classified-sample regime extracts and classifies documents from 224 WARC files, while the full-crawl regime uses Common Crawl’s host index and GneissWeb annotations without reclassification.The regimes support different research questions, including full-crawl replication of depth analysis and crawl-budget analysis.Source: §3 · pp. 2–3

3.1 Classifying Web Documents

  • The primary classified sample contains 3.01M documents from 532,628 hosts, scored with ten fastText classifiers whose host medians represent quality or topic scores.The sample includes hosts with at least three documents; the paper defines classifier-predicted usefulness for pretraining as a proxy, not intrinsic document quality.Source: §3.1 · p. 3

3.2 Web Graph

  • The graph analysis uses Common Crawl’s public host-level hyperlink graph and its published harmonic centrality and PageRank measures, excluding fewer than 0.3% of hosts missing from the graph.The graph release is aligned to cover the document snapshots.Source: §3.2 · p. 3

3.3 Crawl Budget and Quality Density

  • The full-crawl analysis joins the host index’s known and fetched URL counts with GneissWeb annotations to obtain each host’s fraction of crawled pages retained as high quality.This independent full-crawl measure replicates depth analysis and grounds crawl-budget analysis; the study also examines six crawls for temporal coverage patterns.Source: §3.3 · p. 3

3.4 Metrics

  • Depth simulations count pages above a 0.5 quality-score threshold and compare a k-page sample’s quality density with that of the host’s remaining pages before using it to set crawl depth.The analysis uses FineWeb-Edu scores on the classified sample and GneissWeb scores on the full crawl.Source: §3.4 · p. 4
  • Policies receive the same fixed fetch budget and are scored by high-quality pages collected per page fetched; within a host, expected high-quality count depends on its quality density.The policy comparison includes uniform breadth, HC depth, sample-then-deepen, and an oracle that knows true host quality density.Source: §3.4 · p. 4
  • The crawl-budget analysis estimates high-quality uncrawled URLs by multiplying known-but-uncrawled URL counts by observed host quality density, assuming uncrawled pages match fetched pages in quality likelihood.It defines hidden gems as below-median-coverage hosts in the top decile of quality density.Source: §3.4 · p. 4

4 Results

A small per-host sample can guide depth allocation more effectively than uniform or centrality-based crawling, while graph centrality and Common Crawl’s observed coverage provide weak signals of page quality. Quality varies substantially between hosts, and the estimated uncrawled high-quality pages are concentrated on hosts with low crawl coverage.

4.1 RQ1: Quality Is a Host-Level Property

  • Across ten classifiers, ICC1 is 0.43–0.62, meaning roughly half of classifier-score variation lies between hosts rather than within them.This host-level consistency motivates testing whether a small sample predicts the quality of a host’s remaining pages.Source: §4.1 · pp. 4–5, Figure 1 · p. 5

4.2 RQ2: Graph Centrality Is a Weak Crawl-Yield Signal

  • At a 10% crawl budget, HC ordering collects 0.7–1.8 times as many high-quality pages as random ordering, versus 4.0–8.0 times for an oracle, and performs worse than random for DCLM.HC ordering also roughly halves the number of represented top-level domains compared with random ordering.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3 RQ3: A Small Per-Host Sample Decides Crawl Depth

  • A 3–5-page sample already captures most of the signal for ranking hosts by remaining-page quality: ρ = 0.53 at k=3 and ρ = 0.57 at k=5.Correlation rises with sample size and plateaus around k ≈15–20, with diminishing gains from additional pages.Source: §4.3 · p. 6, Figure 4 · p. 6
  • Up to 7.8× more high-quality pages per fetched page at small budgets (falling to 1.7× as budgets grow), sample-then-deepen outperforms uniform breadth and HC depth while approaching the oracle.The policy samples and classifies a few pages per host, then spends depth on hosts with higher-scoring samples.Source: §4.3 · p. 6, Figure 5 · p. 6

4.4 RQ4: Common Crawl’s Budget Is Blind to LLM Data Quality

  • 58M high-quality pages are estimated among uncrawled known URLs in CC-MAIN-2024-10; half lie on hosts crawled below 20%, under the assumption that uncrawled pages match fetched pages’ host-level quality density.The estimate is not a direct measurement of the unfetched pages’ quality.Source: §4.4 · p. 7, Figure 7 · p. 7
  • Across six crawls from 2021–2024, coverage–quality correlations remain near zero and negative; coverage is also uncorrelated with every tested classifier at |ρ| ≤0.07.Nemotron and FineWeb-Edu agree with GneissWeb, while DCLM is the lone non-agreeing classifier.Source: Table 1 · p. 7, §4.4 · pp. 7–8, Figure 8 · p. 8

5 Discussion

Host-level sampling offers a lower-compute complement to page-level scheduling, but graph centrality is entangled with language and host quality. Common Crawl’s broad-sampling objective and the fixed-URL scope of the simulations limit what these findings establish.

5 Discussion

  • A small sample can guide how deeply to crawl each host, complementing page-level schedulers that score every candidate page.Because a few sampled pages predict the host’s remaining quality, the scheduling signal need not be computed for every page in the crawler’s hot path.Source: §5 · p. 8
  • Centrality is a weak quality signal: language differences can reverse observed correlations, and high-centrality hosts mix high- and low-quality pages.DCLM’s pooled correlation with harmonic centrality is negative but positive within English hosts; its scores also differ by host language.Source: §5 · p. 8
  • Common Crawl’s quality misalignment reflects its goal of broad, general-purpose Web coverage rather than LLM-oriented crawling.The analysis estimates tens of millions of high-quality pages among known but uncrawled URLs per crawl, assuming uncrawled pages match fetched pages from the same host.Source: §5 · pp. 8–9
  • The simulations allocate budget over already-fetched pages and do not test whether centrality helps discover useful URLs or hosts.Combining centrality-based exploration with sample-guided depth allocation remains unevaluated.Source: §5 · p. 9

6 Conclusions

The conclusions favor sample-guided host depth allocation over centrality as a quality signal, while identifying live crawling and downstream model effects as open tests. The estimated missed URLs concern known URLs, and the quality-density analysis relies primarily on one filter.

6 Conclusions

  • Between-host variation accounts for about half of document-quality variation, supporting the use of a small sample to represent a host.The conclusion reports that ordering hosts by graph centrality collects at most 1.8 times as many high-quality pages as random, and fewer for DCLM.Source: §6 · p. 9
  • A three-page sample followed by deeper crawling of higher-scoring hosts yields up to 7.8 times the high-quality yield of uniform breadth, while centrality-biased depth offers little to no improvement.Centrality ordering collects at most 1.8 times as many high-quality pages as random ordering, and fewer for DCLM; an oracle collects 4.0–8.0 times as many.Source: §6 · p. 9
  • Common Crawl coverage is uncorrelated with quality under every tested classifier, leaving an estimated 58M known high-quality URLs uncrawled in one crawl.The conclusion estimates that 62% of these URLs are on 369,534 under-crawled hosts.Source: §6 · p. 9
  • The proposed policy samples a few pages per host, scores them, and fetches more from hosts with higher-scoring samples; centrality may group hosts for sampling but should not judge content quality alone.Source: §6 · p. 9

6.1 Future Work

  • None of the research questions tests a newly deployed crawler under production conditions such as latency, freshness, politeness limits, or discovery of new URLs and hosts.Source: §6.1 · p. 9
  • The study measures crawl yield and coverage, not trained-model performance; testing whether yield gains transfer requires downstream model ablations.The quality-density signal also uses primarily one filter because precomputed annotations are available for it.Source: §6.1 · p. 9

Limitations

The reported limitations concern classifier-based quality proxies, estimates of uncrawled pages, and the scope and representativeness of the sampled crawls. The results do not establish that three-page sampling generalizes beyond the observed Common Crawl pages and hosts.

Limitations

  • Classifier-based quality density inherits model biases, and no evaluated model is ground truth.The RQ4 result is also shown under Nemotron and FineWeb-Edu, but this does not establish ground truth.Source: Limitations · p. 9
  • Uncrawled high-quality URL counts assume uncrawled pages resemble crawled ones, making them estimates rather than exact counts.The estimate also omits crawl overlap, politeness, and content freshness.Source: Limitations · p. 10
  • The host-consistency and sample-size analyses cover pages selected by Common Crawl, with WARC subsampling and minimum-document thresholds further limiting coverage.The full-crawl GneissWeb analysis removes WARC subsampling but still reflects Common Crawl’s page selection.Source: Limitations · p. 10
  • The results do not establish that three pages suffice for arbitrary hosts or pages outside the observed crawls.The scope boundary follows from the study’s Common Crawl page selection and sampling choices.Source: Limitations · p. 10
  • Higher classifier-based yield may reduce representation of languages, regions, or content types that classifiers score poorly, and broader-exploration measures remain unevaluated.The authors suggest reserving budget for exploration and monitoring coverage, but their effectiveness remains to be evaluated.Source: Limitations · p. 10

A Temporal Stability of Graph Ranks

Host harmonic centrality and PageRank rankings remain strongly correlated with rankings in later Common Crawl graph releases, supporting use of one reference graph for the study’s document snapshots.

A Temporal Stability of Graph Ranks

  • Above 0.85: host harmonic centrality and PageRank ranks correlate with later Common Crawl graph releases up to approximately 15 months later.Table 2 reports Spearman correlations of 0.851–0.923 across comparisons.Source: Table 2 · p. 12, A Temporal Stability of Graph Ranks · p. 12
  • Table 2 compares each host’s reference-graph centrality rank with its rank in later graph releases; 1.0 denotes identical ordering.Both HC and PR stay above 0.85 even approximately 15 months apart.Source: Table 2 · p. 12
  • This stability supports pairing one aligned graph with document snapshots from different crawls instead of rebuilding the graph each time.Source: A Temporal Stability of Graph Ranks · p. 12, Table 2 · p. 12

B Crawl-Yield Detail (RQ2)

At a fixed 10% crawl budget, centrality ordering yields limited gains over random and can perform worse, while oracle ordering collects several times more high-quality documents. Centrality ordering also narrows top-level-domain representation.

B Crawl-Yield Detail (RQ2)

  • At 10% budget, centrality ordering gains at most 1.8× over random and falls below 1 for DCLM, while the oracle reaches 4.0–8.0×.The values are high-quality documents collected as multiples of a random crawl (1.0).Source: Table 3 · p. 12, B Crawl-Yield Detail (RQ2) · p. 12
  • At the same budget, harmonic-centrality ordering roughly halves represented top-level domains versus random, narrowing crawl diversity.Table 3 reports the per-classifier comparison; Figure 9 shows the full budget sweep.Source: Table 3 · p. 12, B Crawl-Yield Detail (RQ2) · p. 12

C Full Graph-Classifier Correlation Tables

C.1 HC and PageRank Correlations Across Classifiers

Across classifiers, host-level centrality correlations are modest and language composition helps explain their signs; the appendix tables also report host-level quality variation across sampling thresholds.

C.1 HC and PageRank Correlations Across Classifiers

  • Across twelve classifiers, Spearman correlations of host-level harmonic centrality and PageRank range from −0.086 to +0.207, and their signs agree across correlation measures.Table 4 reports Spearman ρ, Kendall τ, and weighted Spearman ρ; HC and PR have the same sign and similar magnitudes for each classifier.Source: Table 4 · p. 12, §C.1 · p. 12

C.2 Language-Stratified Correlations (log HC, n ≥3)

  • Within-language centrality-quality correlations tend to be more positive than pooled correlations; JQL-Edu and FineWeb2-HQ retain positive signals on non-English hosts.Table 5 gives the classifier correlations for all hosts and separately for English- and non-English-primary hosts.Source: §C.2 · p. 12, Table 5 · p. 13

C.3 TLD Macro-Category Breakdown

  • HC–DCLM correlation is negative within academic and commercial TLD categories, where non-English hosts that DCLM scores highly concentrate.The breakdown localizes the language confound to domains where language and centrality covary most.Source: Table 6 · p. 13, §C.3 · p. 13

C.4 Score Distributions by Language Stratum

  • DCLM’s median score is substantially higher on non-English than English hosts, while FineWeb-Edu scores the groups similarly.These score distributions are consistent with language composition contributing to the pooled centrality-quality correlation.Source: Table 7 · p. 13, §C.4 · p. 13

C.5 Full ICC1 Table

  • Between-host quality variation is near one-half and stable across minimum thresholds of n ≥3, 5, and 10 documents per host.Table 8 reports the full ICC1 decomposition and supports using a small per-host sample to stand in for a host.Source: Table 8 · p. 13, §C.5 · p. 13

D Datasets and Models

The section inventories the study’s datasets by data regime and lists the page-classification models used in the classified-sample regime.

D Datasets and Models

  • Table 9 maps datasets to data regimes and research questions, distinguishing WARC crawls scored by the authors from Common Crawl’s precomputed host-index and GneissWeb artifacts.The study combines public Common Crawl artifacts and Hugging Face models across the two data regimes.Source: D Datasets and Models · p. 13, Table 9 · p. 14
  • Table 10 lists the exact Hugging Face models used to classify pages in the classified-sample regime.Source: Table 10 · p. 14
  • FineWeb-Edu is a compute-efficient fastText reimplementation with comparable accuracy to the original Transformer; JQL-Edu and FineWeb2-HQ use frozen embeddings and lightweight per-language heads on non-English documents.Source: Table 10 · p. 14

E Supplementary Figures

The supplementary figures show centrality-based crawl ordering against random and oracle policies, the persistence and composition of missed high-quality URLs, and agreement patterns among classifiers.

How the classifiers relate

  • Across 462,494 documents, DCLM is largely orthogonal to other quality signals and correlates −0.459 with Nemotron-Q, while GneissWeb topic scorers and Nemotron heads form positive blocks.FineWeb-Edu aligns most closely with GneissWeb-Q; FineWeb2-HQ correlates weakly or negatively with English-trained classifiers.Source: E Supplementary Figures · p. 15, Figure 12 · p. 18

Crawl yield across budgets

  • Across crawl budgets, harmonic centrality, PageRank, and Common Crawl ordering stay near random and below the oracle; DCLM centrality is worse than random.The centrality-to-oracle gap is widest at small budgets and closes as the crawl approaches completion.Source: E Supplementary Figures · p. 15, Figure 9 · p. 16

Persistence across crawls

  • Across six crawls from 2021–2024, coverage–quality correlations range from −0.057 to −0.151, and each crawl leaves tens of millions of high-quality URLs uncrawled.The disconnect persists across crawls rather than appearing in only one snapshot.Source: E Supplementary Figures · p. 15, Figure 10 · p. 16

Where the missed budget sits

  • Country-code publisher TLDs are over-represented among missed high-quality URLs; .uk holds 23% of the missed budget from 6.1% of hosts.Among keyword-labeled hosts, news/media and scholarly sites dominate, while about 80% are uncategorized; the labeled categories are only a lower bound.Source: E Supplementary Figures · p. 15, Figure 11 · p. 17
Loading 2610.13321v1…