PaperlayerView exact revision on arXiv

Source-linked AI summary

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan

arXiv:2610.31636v1cs.CL

TL;DR

FineWeb2’s shared multilingual recipe leaves language-specific defects difficult to anticipate, while manual audits do not scale and LLM classification incurs inference costs. This paper uses a tool-using agent to discover corpus defects and compile them into tests validated by a native speaker. On Slovak FineWeb2, the audit found seven findings across four phenomena, with blinded labels supporting six; the best static baseline validated more selectors per run than any agent run.

  • Problem

    FineWeb2 applies one recipe across more than a thousand languages, but its authors could not test each language individually, and Slovak’s closely related neighbour presents a language-identification challenge.

    Source: §1 · p. 1

  • Method

    A tool-using agent explores Slovak FineWeb2 through a read-only API, compiles findings into executable tests, and has a native speaker validate sampled documents before model-free deployment.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Seven findings across four phenomena emerged; blinded labels supported 6 of 7 findings at .53–1.00 defect rates against .08 in the control, while the best static baseline validated more selectors per run than any agent run.

    Source: §1 · p. 2, §5 · p. 5, §6 · pp. 8–9

  • Takeaways & Limitations

    The workflow turns a one-off audit into validated executable tests, and its suite can be run on other corpora implementing the adapter interface.

    Source: §1 · p. 1, §3 · p. 3, §6 · p. 9

    Validation covers 255 documents labelled by one native speaker, limiting precision estimates and leaving agreement on ambiguous Czech/Slovak text unmeasured.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing \$2.21-\$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against \$12k-\$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.

1 Introduction

Multilingual corpus pipelines make large-scale data feasible but leave language-specific defects difficult to anticipate and review. The paper uses an LLM as a corpus debugger that turns discoveries into native-speaker-validated tests, rather than as a classifier.

1 Introduction

  • FineWeb2 applies one configurable recipe across more than a thousand languages, while its authors could not test each language individually.Its high-affinity wordlist filter targets languages with measured contamination above 10%, not mid-resource Slovak despite its closely related neighbour.Source: §1 · p. 1
  • Manual audits consume expert attention linearly and do not compose into repeatable checks, while classifying the corpus with an LLM incurs inference costs without explaining defects.The paper frames these as limitations of the two established approaches to reviewing a language pipeline.Source: §1 · p. 1
  • A tool-using agent explores the corpus through a read-only API and produces findings with executable selectors, whole-corpus prevalence, and precision for human validation.The prompt names no expected defect or neighbouring language; tests can be rerun on future crawls or pipeline revisions.Source: §1 · p. 1
  • The audit targets Slovak FineWeb2, a 26.5M-document slice that is too large to read and lacks a dedicated audit despite Slovak’s closely related neighbour.The study asks whether an agent can surface native-speaker-confirmed defects, compile them into cheap tests, and find slices outside static audits.Source: §1 · p. 2
  • Seven findings across four phenomena emerged from the Slovak audit, including a 5.32% content farm that passed the pipeline’s filters.The agent’s contribution is measurement, characterisation, and tests; its findings were compared with static sampling strategies.Source: §1 · p. 2

2 Related Work

Prior corpus audits and pipeline studies document multilingual web-corpus failure surfaces, while work on behavioural testing and data-quality assertions motivates repeatable corpus tests. This paper focuses on making audits executable and grounded in language-community validation.

2 Related Work

  • Existing audits cover many corpora and individual languages, while pipeline studies examine related failure surfaces; this work asks how audits can become repeatable and executable.The paper situates Slovak within established cleaning practice and known Czech/Slovak language-identification difficulty, without promising downstream gains.Source: §2 · p. 2
  • Behavioural testing and declarative data-quality assertions motivate corpus tests, but the assertions here must come from people who know the language.The agent writes tests that label the corpus; it is not presented as a language classifier.Source: §2 · p. 2
  • Unlike curation agents that optimise data-selection policies, this agent outputs a test suite and optimises no downstream metric.The paper also distinguishes its work from LLM-based corpus-linguistic discovery.Source: §2 · p. 2

3 The Audit Environment

The audit environment gives an agent a read-only view of corpus metadata and sampled text, then separates discovery, human validation, and model-free deployment. Its artefacts and evidence are released for replay and reuse.

3 The Audit Environment

  • The adapter separates exact corpus-wide prevalence over metadata rows from text inspection through a deterministic 1-in-13 sample of 2,034,894 documents.Metadata aggregates over 26.5M rows return in under a second, unlike scanning 100 GB of text; the schema contains no named language pair or failure mode.Source: §3 · p. 3
  • Eleven budgeted tools support aggregation, sampling, text search, language identification, detector evaluation, and saving findings; each distinct document read costs one inspection.Calls are logged for replay, and the three protocol runs’ aggregate queries returned counts rather than document text.Source: §3 · p. 3
  • The agent follows one trajectory from orientation and hypothesis formation through evidence gathering, counter-evidence, prevalence measurement, selector checks, and finding records.Its prompt specifies tools, the finding schema, and standards for reproducible, document-grounded findings without naming a defect or neighbouring language.Source: §3 · p. 3
  • Discovery produces candidates, a native speaker labels blinded document samples for validation, and deployment runs approved tests without a model.Only an adopted verdict assigns status; accepted tests carry selectors, assertions, and evidence, while other tests remain candidates.Source: §3 · p. 3, Figure 1 · p. 3
  • The release includes the environment, runner, prompt, suite, and inputs to the paper’s reported numbers, with tables generated from those artefacts.The repository includes a suite of seven Slovak slice guards and two metadata invariants.Source: §3 · pp. 3–4

4 Experimental Setup

The experiment audits the Slovak FineWeb2 slice using three budgeted agent runs and matched-budget static sampling baselines, then evaluates candidate selectors through blinded native-speaker labels and adjudication.

4 Experimental Setup

  • The audited Slovak FineWeb2 slice contains 26,470,482 documents from 351,844 hosts and 96 Common Crawl snapshots spanning 2013–2024.All documents meet the published language_score threshold of 0.667, while auxiliary identifiers disagree with the pipeline label on 3.12%.Source: §4 · p. 4
  • Each of three agent runs was capped at 300 document inspections and 250 turns; each baseline was capped at six turns.All runs used claude-sonnet-5 with default sampling settings and no set temperature.Source: §4 · p. 4
  • Static baselines show the same model 300 selected documents as metadata and their first 400 characters, without tools or exploration.Strategies include random sampling, low language confidence, LID disagreement, and high duplication; comparison therefore changes both adaptivity and tool access.Source: §4 · p. 4
  • A native speaker blindly labelled sampled documents without provenance, URL, or metadata; defect labels include Czech, mixed, other-language, or junk documents.The 358-document draw included 40 uniform controls, and slice precision estimates how often selected documents are defects, not recall.Source: §4 · p. 4
  • Of 358 drawn documents, 255 received labels, with none marked uncertain; the shortfall resulted from two-pass labelling and dropped documents.Adjudication grouped overlapping selectors and used a model to tabulate evidence and propose verdicts, which the annotator adopted without independently deliberating each decision.Source: §4 · pp. 4–5

5 Results

Across three runs, the auditor identified seven findings grouped into four phenomena, including a 5.32% Firebase content farm; blinded labels supported six findings. The compiled tests run quickly, while the best static baseline validated more selectors at under half the cost.

5 Results

  • 1,408,437 documents (5.32%) form the largest finding, a Firebase and web.app content farm among seven findings across four phenomena.The farm contributes 12.3% of all running words and rises sharply from late 2022.Source: §5 · p. 5, Table 2 · p. 6
  • The metadata-only suite scans all 26,470,482 documents in 0.7 seconds without a GPU, versus $12,309–$90,456 per LLM classification pass.The tests reuse columns already computed in corpus-wide passes; all nine ran unchanged on an independent 28,720-document held-out split.Source: §5 · pp. 5, 6, 8
  • 6 of 7 findings received human-judged defect rates of .53–1.00, versus .08 for the random control; the slo.* finding was unsupported.The slo.* finding was released as a candidate.Source: §5 · p. 5, Figure 4 · p. 6
  • Every validated phenomenon was retrievable through a shipped signal: LID disagreement, profanity counts, or uniform sampling.All G04 selectors fall inside the n_bad_words ≥3 slice (0.39%); whether an agent would find it through profanity-ranked sampling was untested.Source: §5 · pp. 6–7
  • 86.6% of the content farm passed stricter static thresholds, and no farm document reached the duplication threshold.All three auxiliary identifiers agree with the Slovak label on 87% of farm documents, so these signals do not isolate the farm.Source: Figure 6 · p. 7, §5 · p. 7
  • 4 selectors validated by LOWCONF exceeded every agent run’s maximum of 3, across three groups and at under half the cost.Pooled, baselines and agents each had six validated selectors, but covered different groups.Source: Figure 7 · p. 8, Table 3 · p. 8, §5 · p. 8

6 Conclusion

The Slovak audit found that the agent was less effective than a well-chosen static sample at validating selectors, while producing a workflow that turns findings into executable tests. The tests run without a model, and the workflow has also been applied to other FineWeb2 configurations.

6 Conclusion

  • The agent validated fewer selectors per run than a well-chosen static sample, and every validated defect was reachable by a static strategy.The authors did not run the model on the profanity-ranked sample that reaches adult spam.Source: §6 · pp. 8–9
  • The workflow turns a one-off audit into executable, validated tests and can reject a finding, as it did for one candidate.The rejected finding was shipped as a candidate.Source: §6 · p. 9
  • The framework already runs on eleven other FineWeb2 configurations, which re-ran tests written for Slovak.Other communities can also point the adapter at their own corpus and run the suite.Source: §6 · p. 9

Limitations

The study demonstrates agentic discovery on one language, one corpus, and three runs of one model; its validation and comparisons also have important constraints. The released guards do not establish broad recall or stability over time.

Limitations

  • Agentic discovery was tested only on Slovak, one corpus, and three runs of one model; other FineWeb2 configurations only reran Slovak tests.A fourth, stronger-model run reproduced the largest phenomena, but its extra findings are unvalidated.Source: Limitations · p. 9
  • Validation labels cover 255 documents from one native speaker, limiting precision estimates and leaving inter-annotator agreement on ambiguous Czech/Slovak text unmeasured.The study measures selector precision, not recall, and does not measure downstream language-model effects.Source: Limitations · p. 9
  • Baseline comparisons confound adaptivity with tool access, and the profanity-ranked baseline was not evaluated with the model.Two baseline verdicts were later corrected after labels were restored, moving the best baseline from a tie to exceeding the agent.Source: Limitations · p. 9
  • Natural variation within a single crawl can trip all seven prevalence guards and produce false alarms; some host patterns may drift as the web changes.The guards are calibrated for corpus-wide evaluation and do not guarantee stability.Source: Limitations · p. 9

Ethical Considerations

The paper describes safeguards for handling web text, a read-only audit API with bounded document inspection, and an agent prompt that requires evidence-backed, quantified findings. The prompt avoids naming expected defects, and the recorded runs used only the audit tools.

Ethical Considerations

  • The annotator was warned about explicit material and could skip any document or stop at any time; released excerpts and trajectories were redacted for third-party personal data.Real email addresses were replaced at the tool boundary, while anonymisation placeholders were retained; the paper does not republish the residual addresses.Source: Ethical Considerations · p. 9

A Audit Tool API

  • Aggregation, prevalence measurement, and host and crawl statistics are free, while each distinct document read costs one inspection from the fixed budget.The API also supports sampling, text search, sentence-level language identification, and detector evaluation; calls are logged for replay.Source: A Audit Tool API · p. 11

B Agent Prompt

  • Findings must be systematic, reproducibly retrievable, grounded in inspected documents, and quantified; the agent is told to prefer a few strong findings and conserve inspection budget.It should use metadata-only selectors when similarly accurate and report evidence against its findings.Source: B Agent Prompt · p. 12
  • The prompt directs the agent to form falsifiable hypotheses, seek counter-evidence, and reject hypotheses when evidence does not support them.It also asks the agent to estimate prevalence, check selector precision, and record likely false positives.Source: B Agent Prompt · p. 12

C What the Agent Was Told

  • The prompt names no defect or neighboring language, and a 41-term check found no defect or failure-mode terms, including those later reported.It does state the Slovak corpus claim and pipeline steps, and the authors note that the data dictionary makes the profanity count visible.Source: C What the Agent Was Told · pp. 12–13
  • Recorded streams show that all 227 calls were audit API calls except one harness tool-selection call per run; no shell, file, network, or built-in retrieval tools were invoked.Source: C What the Agent Was Told · p. 13

D Why the Czech Selector Needs Three Identifiers

A conjunction of three language identifiers is highly precise on CHALIS's single-label Czech/Slovak sentences, whereas each identifier alone performs poorly on the Slovak side. The benchmark does not replace human validation because it uses short sentences and contains many ambiguous cases.

D Why the Czech Selector Needs Three Identifiers

  • CHALIS separates hard Czech/Slovak sentences with a single correct label from sentences valid in both languages, allowing calibration without asking a person.Source: D Why the Czech Selector Needs Three… · p. 13
  • 99.6% precision: requiring fastText, langdetect, and Lingua all to say Czech identifies 256 correct sentences among 257 fired cases out of 527 single-label sentences.Source: D Why the Czech Selector Needs Three… · p. 13
  • On 36 single-label Slovak sentences, fastText is correct 22.2% of the time, langdetect 69.4%, and Lingua 66.7%.The majority class is Czech, and the Slovak side is where the identifiers fail.Source: D Why the Czech Selector Needs Three… · p. 13
  • 55.4% of benchmark Czech/Slovak material is ambiguous, and the conjunction fires on 28.5% of those jointly valid sentences.Because the selector scores documents and the defect criterion extends beyond language, a speaker must judge the two slices' precision.Source: D Why the Czech Selector Needs Three… · p. 13

E The Written Tests on Other Languages

The Slovak tests and framework were checked on upstream Slovak and eleven other FineWeb2 configurations, without running an agent on those languages. Missing upstream metadata was treated as unmeasurable, while two metadata-only findings reproduced and language-neutral invariants were also checked.

E The Written Tests on Other Languages

  • On 11 foreign corpora, all seven Slovak tests were correctly reported out of scope, while the two language-neutral invariants ran.Identifier uniqueness passed everywhere; the invalid-confidence invariant failed in 11 of 12 configurations.Source: E The Written Tests on Other Languages · p. 13
  • On upstream Slovak, five of seven slice guards were reported unmeasurable rather than passing at zero prevalence.The content farm reproduced at 3.72% and the machine-translation farm at 0.258% using metadata alone.Source: E The Written Tests on Other Languages · p. 13
  • The invalid-confidence defect followed fastText’s log(p + 10^-5) convention, and the pipeline stored that score unchanged.The maximum was float32(1.00001) in 8 of 11 affected configurations and within 2 × 10^-6 in the other three.Source: E The Written Tests on Other Languages · p. 13

F More on the Held-out Split

Held-out and crawl-level checks show that some findings persist while others vary sharply over time, and that patching the document slices leaves metadata defects visible. Replaying the agent’s calls was mostly reproducible, and a pilot run volunteered counter-evidence against its own hypothesis.

F More on the Held-out Split

  • 40.23% of held-out data failed the invalid-confidence invariant, while identifier uniqueness passed despite 13 collisions in 26.5M documents.The collisions were in one crawl, with both rows from the same WARC file but different extracted text; joining on id therefore fans out.Source: F More on the Held-out Split · pp. 13–14

G Where the Slices Live in Time

  • The Czech slices appeared in all 96 crawls at roughly 0.06–0.40%, while the content farm was 21.6 times larger in the eight newest crawls than earlier.The farm peaked at 38.7% in the 2023-06 snapshot and remained at 0.39% in the last crawl; adult spam peaked in 2020 and nearly disappeared.Source: G Where the Slices Live in Time · p. 14

H Findings as Patches

  • Filtering all seven retained selectors would remove 6.10% of documents and 16.31% of words; rerunning the suite confirmed zero matches for patched guards.The two metadata invariants still failed, as intended; removing spam raised the invalid-confidence share from 40.4% to 41.5%.Source: H Findings as Patches · p. 14

I Trajectory Replay

  • 144 of 147 logged calls returned the same rows as a set on replay, and 139 were byte-identical.The three set differences were LIMIT queries with ties at the cutoff.Source: I Trajectory Replay · p. 14

J Counter-evidence the Agent Volunteered

  • In a pilot run, the agent concluded that roughly half of a strict triple-agreement slice was not defective and lowered its expected precision to 0.5.It attributed this to diacritic-free informal Slovak fooling all three auxiliary identifiers into voting Czech.Source: J Counter-evidence the Agent Volunteered · p. 15

K Cross-model Robustness

A fourth run with a stronger model reproduced the largest content-farm finding exactly, but used broader or different selectors for other known findings. It also proposed new, unvalidated phenomena at higher cost.

K Cross-model Robustness

  • The fourth run reproduced the Firebase/web.app content farm exactly: the same 1,408,437 documents, with Jaccard 1.00 overlap.It also found the machine-translated health farm with a broader selector (147,318 documents; Jaccard 0.31) and Czech mislabelling with a different selector (23,076 documents; Jaccard 0.09).Source: K Cross-model Robustness · p. 15
  • The run proposed four further phenomena that remain unvalidated, including sk.-locale subdomains, flat-HTML farms, replaced email addresses, and inserted advertising boilerplate.It also verified the metadata invariant arithmetically and released it as a test.Source: K Cross-model Robustness · p. 15
  • The stronger model proposed more candidates at 2.9–3.7 times the cost of a primary run.The fourth run used the same environment, prompt, and 300-inspection budget, but was outside the frozen three-run protocol.Source: K Cross-model Robustness · p. 15

L The Test Suite

The suite turns findings into retraceable guards or invariants, with accepted findings distinguished from a candidate unsupported by blinded labels. The examples span content farms and Czech text, while the annotation guidance treats usefulness—not readability alone—as the criterion for corpus text.

L The Test Suite

  • Findings compile into prevalence guards for future growth and invariants for any occurrence, with evidence and expected false positives attached as appropriate.A guard’s bound is 1.2 times measured prevalence; an invariant instead records the mechanism producing the violation.Source: L The Test Suite · p. 16
  • 0.387% (102,474 documents) matched adult/porn word-salad spam with incoherent generated text and repeated boilerplate.The finding was accepted, with a defect rate of 0.84 (n = 19).Source: L The Test Suite · p. 16
  • 0.167% (44,251 documents) matched Czech-language or heavily code-switched documents labelled Slovak, often from shared commerce catalogs and mixed-language forums.The finding was accepted; mutually intelligible Czech and Slovak can lead the pipeline and auxiliary identifiers to agree incorrectly.Source: L The Test Suite · p. 16
  • 0.120% (31,795 documents) matched a systematic slice of substantially or entirely Czech text that the corpus labelled Slovak.Examples include product data, forum posts, cookie notices, thesis metadata, and Pinterest boards; the finding was accepted.Source: L The Test Suite · pp. 16–17
  • 5.321% (1,408,437 documents) matched the Firebase content farm, which passed language filters with high confidence and evaded near-duplicate deduplication.Its pages were stitched, multi-topic text on randomly named Firebase subdomains; the finding was accepted.Source: L The Test Suite · p. 17
  • 0.198% (52,354 documents) matched a distinct pornography doorway-page network whose machine-translated tag phrases were keyword-stuffed rather than coherent prose.The finding was accepted, with a defect rate of 1.00 (n = 20).Source: L The Test Suite · p. 17
  • 0.359% (94,946 documents) matched a separate network of incoherent adult doorway pages, many using the same literal CMS template header.The finding was accepted, with a defect rate of 0.83 (n = 12).Source: L The Test Suite · p. 17

M Validation Details and Annotation Guidelines

  • The slo.* machine-translated farm remained a candidate, with a 0.07 defect rate (n = 14), rather than a finding supported by blinded labels.The annotation guidance says fluent Slovak can still be junk when a page exists to place links or keywords; the 300 most profane documents sampled separately all belonged to G04 findings.Source: L The Test Suite · pp. 16–17, M Validation Details and Annotation… · p. 18
Loading 2610.31636v1…