PaperlayerView exact revision on arXiv

Source-linked AI summary

AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser

Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng, Linfeng Hou, Mengjie Liu, Lindong Lu, Wenchang Ning, Jia Yu, Rui Min, Jin Shi, Haojiong Chen, Peng Zhang, Wenjian Zhang, Qian Jiang, Zengjie Hu, Guoqiang Yang, Zhenxiang Li, Fukai Shang, Runyuan Ma, Chenlin Su, Zhongying Tu, Wentao Zhang, Dahua Lin, Conghui He

arXiv:2511.16397v2cs.CL

TL;DR

HTML extraction is an overlooked step in web-data curation, and existing heuristic tools can lose document structure and corrupt structured elements. The paper introduces MinerU-HTML, a semantic-aware extraction pipeline, and reports better extraction and pretraining results than Trafilatura-based alternatives. Its controlled comparison finds an average-accuracy advantage for AICC over TfCC when filtering and deduplication are identical.

  • Problem

    HTML extraction has received comparatively little attention, although heuristic tools can fail to preserve document structure and structured elements such as formulas, code, and tables.

    Source: §1 · p. 2

  • Method

    MinerU-HTML uses a compact language model to classify HTML blocks as main content or boilerplate, producing Main-HTML for downstream formatting.

    Source: §1 · p. 2, §2.1.1 · p. 5

  • Results

    AICC-trained models achieve 50.82% average accuracy across 13 benchmarks, 1.08pp above TfCC under identical filtering and deduplication.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The controlled AICC–TfCC comparison supports extraction quality as an important component of web-corpus construction and model performance.

    Source: §1 · p. 3

    The pretraining comparison excludes datasets using model-based quality filtering because that would confound extraction quality's specific contribution.

    Source: §4.2 · p. 15

Abstract

from arXiv · show

While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.

1 Introduction

Web corpora depend on HTML extraction before filtering, yet extraction has received less attention despite its difficulty preserving document structure and technical elements. The paper introduces semantic-aware MinerU-HTML and reports extraction and pretraining results against existing approaches.

1 Introduction

  • MinerU-HTML recasts extraction as semantic-aware sequence labeling, and the work releases the extractor, benchmark, and AICC corpus.The authors describe model-based extraction as scalable with more data and stronger base models.Source: §1 · pp. 2–3
  • 0.8182 ROUGE-N F1 on WebMainBench, versus Trafilatura's 0.6358, across 7,887 annotated pages.The benchmark evaluates main-content extraction.Source: §1 · pp. 2–3
  • 0.9093 code-block and 0.9399 formula edit similarity for MinerU-HTML exceed Trafilatura's 0.1305 and 0.6107, respectively.Source: §1 · p. 3
  • 72.0% of 10,000 Common Crawl document-pair comparisons preferred AICC over TfCC in an LLM-as-judge evaluation.This corpus-level comparison is reported as generalizing the benchmark extraction improvements.Source: §1 · p. 3
  • 50.82% average accuracy across 13 benchmarks for AICC-trained models exceeded TfCC's 49.74% by 1.08pp under identical filtering and deduplication.Source: §1 · p. 3
  • AICC significantly outperformed FineWeb on reading comprehension, scoring 42.37% versus 36.68%, a 5.69pp difference.The paper reports improvements across all task categories.Source: §1 · p. 3

2 MinerU-HTML: HTML Parsing for AI-ready CC

MinerU-HTML extracts content from raw HTML as Main-HTML, then converts it into formats such as Markdown. Its model-based classification pipeline is paired with template-based scaling and a typed intermediate representation for formatting.

2 MinerU-HTML: HTML Parsing for AI-ready CC

  • MinerU-HTML first extracts a cleaned Main-HTML subset, then converts it into AI-ready formats such as Markdown.Source: §2 · p. 3

2.1.1 Core Extraction Pipeline

  • The core pipeline classifies simplified HTML blocks as main content or boilerplate, then projects labels onto Mapping HTML to assemble Main-HTML as a valid DOM subtree.Pre-processing creates compact Simplified HTML and structure-preserving Mapping HTML; constrained output supports faithful reconstruction.Source: §2.1.1 · pp. 4–5, Figure 1 · p. 4
  • The classifier treats preprocessed HTML as a sequence of simplified blocks and predicts a binary main-content or boilerplate label for each block.The approach is framed as sequence labeling rather than free-form text generation.Source: §2.1.1 · pp. 4–5

2.1 Main-HTML Extraction

  • For web-scale processing, structurally similar pages are clustered, model decisions on representative pages distilled into XPath-based rules, and those rules propagated to other pages.This template-aware strategy reduces language-model inference while retaining the core extraction pipeline's quality.Source: §2.1.2 · p. 6

2.1.2 Scaling to Common Crawl

  • Only 0.4% of 300 billion Common Crawl pages required GPU inference: subdomain clusters used one representative page and rules for the rest while preserving equivalent extraction quality.The analysis found approximately 1.2 billion distinct clusters.Source: §2.1.2 · p. 7

2.2 Document Formatting

  • The formatting pipeline uses a semantically rich, compact intermediate representation to make extracted content more useful for downstream AI applications.Markdown is the primary output format for language-model training.Source: §2.2 · pp. 7–8

2.2.1 HTML to Content List Conversion

  • Main-HTML is parsed into a typed JSON content list before rendering, enabling semantic-type filtering and format-specific pruning.Specialized detectors classify elements such as code, tables, and formulas; complex tables can remain HTML to preserve structure.Source: §2.2 · p. 8, §2.2.1 · p. 8

2.2.2 Content List to Markdown Conversion

  • The second formatting stage applies type-specific rendering rules to convert the structured content list into Markdown.Source: §2.2.2 · p. 8

3 Evaluation of MinerU-HTML

MinerU-HTML is evaluated on WebMainBench for main-content extraction and preservation of structured elements, then tested for generalization on WCEB. It outperforms the strongest baseline on both benchmarks and substantially reduces extraction input size.

3.1.1 Dataset Overview

  • WebMainBench contains 7,887 annotated pages, while its 545-page structured subset targets formulas, code blocks, and tables.Pages are sampled 90% randomly from Common Crawl and 10% from high-traffic sites; the structured subset includes ground-truth Markdown preserving elements.Source: §3.1.1 · p. 9, §3.1.2 · p. 9, §3.1.3 · p. 9

3.1.4 Annotation Principles

  • Annotation guidelines prioritize context-integral, human-generated content while excluding unrelated sidebars, advertisements, and auto-generated metadata.Examples include retaining academic-paper abstracts, author information, and references while excluding algorithmic recommendations.Source: §3.1.4 · pp. 9–10

3.2.1 Evaluation Metrics

  • ROUGE-N F1 (N=5) is the primary metric for overall main-content extraction, measuring n-gram overlap; ROUGE-L is avoided because its O(n·m) cost is prohibitive on long documents (average 2,304 tokens).For tables, TEDS measures structural similarity by computing tree edit distance on the HTML DOM representation [32].Source: §3.2.1 · p. 10

3.3.1 Main Content Extraction

  • On conversational pages, MinerU-HTML scores 0.7671 versus 0.5750 for the best baseline, and leads on pages containing tables and mathematical equations.The reported scores for pages containing tables are 0.7693 versus Trafilatura’s 0.5505; for mathematical equations, 0.8889 versus 0.7327.Source: §3.3.1 · pp. 10–11
  • 0.8182 overall ROUGE-N F1 on WebMainBench puts MinerU-HTML ahead of Trafilatura (0.6358, Html+MD mode).The benchmark compares extraction across content types and difficulty levels.Source: §3.3.1 · p. 10, Table 1 · p. 11

3.3.2 Efficiency Analysis

  • 5,735 input tokens and 383 output tokens per document remain after MinerU-HTML preprocessing, versus 44,706 and 2,304 for direct raw-HTML processing.These are 12.83% and 16.64% of the respective naive generative-baseline lengths.Source: §3.3.2 · p. 11

3.3.3 Structured Element Extraction

  • 0.9093 code edit similarity, 0.9399 formula edit similarity, and 0.7388 table TEDS show stronger structured-element preservation than either baseline.Complex tables with merged cells, nested structures, and inconsistent markup remain challenging; MinerU-HTML nevertheless preserves row-column structure and cell alignment.Source: §3.3.3 · pp. 11–12, Table 2 · p. 12

3.4 Generalization to WCEB

  • 0.8002 overall on WCEB exceeds the strongest baseline, Trafilatura, at 0.7833.The authors report strong performance across WebMainBench and WCEB.Source: Table 3 · p. 12, §3.4 · p. 12

4 Dataset Construction and Pretraining Experiments

The authors construct AICC with MinerU-HTML and compare it with Trafilatura-extracted TfCC under controlled processing. AICC is preferred in most judged comparisons and yields higher average accuracy across 13 benchmarks, with gains over TfCC persisting through training.

4.1.3 Quality Filtering and Deduplication

  • Both corpora are extracted from the same two Common Crawl snapshots, then undergo identical filtering and deduplication before training.Post-processing yields 372B tokens for AICC and 317B for TfCC; models are trained from scratch on 62B-token subsets and evaluated across 13 benchmarks.Source: §4.1.1 · p. 13, §4.1.3 · p. 15, §4.2 · p. 15

4.1.1 Extraction and Initial Processing

  • AICC extracts 1.16× more content on average than TfCC, according to the sampled document-pair length comparison.The length ratio compares extracted lengths from the two datasets.Source: §4.1.1 · p. 13, Figure 3 · p. 13

4.1.2 Extraction Quality Analysis

  • AICC wins 72.0% of judged document comparisons overall, with preference switching at length ratio zero.TfCC is preferred in 51–92% of comparisons when AICC extracts less; AICC is preferred in 75–98% when it extracts more, and some extreme negative-ratio cases reflect missed main content.Source: §4.1.2 · p. 14, Figure 4 · p. 14
  • Near-equal document lengths can still favor AICC when Trafilatura mishandles lists or misses titles and authors.In some cases where TfCC is longer, it retains boilerplate that MinerU-HTML excludes.Source: §4.1.2 · p. 14

4.3 Downstream Evaluation Results

  • 50.82% average accuracy at 63B tokens puts AICC ahead of TfCC (49.74%), RefinedWeb (49.13%), and FineWeb (49.61%).The 1.08pp gain over TfCC is reported for datasets differing in extraction method but undergoing identical post-processing; AICC remains superior or competitive across 15 checkpoints.Source: §4.3 · pp. 15–16, Figure 5 · p. 16
  • Against TfCC, AICC improves average accuracy by 1.93pp on General Knowledge, 0.49pp on Reasoning, and 0.35pp on Reading Comprehension.AICC leads FineWeb on General Knowledge and Reading Comprehension, while FineWeb is slightly better on Reasoning.Source: §4.3 · p. 16

5 Conclusion

The conclusion presents extraction quality as a critical, improvable part of web corpus construction, with controlled experiments and corpus-scale comparisons supporting AICC’s gains. It also identifies model-based extraction as scalable and releases the corpus, benchmark, and extraction system.

5 Conclusion

  • AICC achieves the best performance on General Knowledge and Reading Comprehension, outperforming FineWeb on Reading Comprehension by 5.69pp.Figure 6 compares AICC, TfCC, RefinedWeb, and FineWeb at 63B tokens.Source: Figure 6 · p. 17
  • AICC improves accuracy by 1.08pp over TfCC under identical post-processing, is preferred in 72.0% of 10,000 document comparisons, and preserves 1.16× more content judged valuable.The conclusion reports validation on a 4.52-trillion-token corpus and says AICC outperforms RefinedWeb and FineWeb across multiple benchmarks.Source: §5 · p. 17
  • Model-based extraction can improve systematically with better data and models, unlike heuristic extraction, whose rule updates often conflict.The conclusion frames extraction quality as a continuously improvable component and lists future directions including JavaScript rendering and larger-scale validation.Source: §5 · p. 17
  • The authors release MinerU-HTML, WebMainBench, and AICC to support research on semantic-aware extraction and its impact on pretraining data quality.Source: §5 · p. 17

A.1.1 Training Data Construction

The training dataset was designed to reflect variation in web layouts, languages, and document formats. Its construction combines structural sampling, language and format balancing, and block-level annotation.

A.1.1 Training Data Construction

  • The training dataset uses a three-stage curation pipeline to capture diversity in page layout, language, and document format.Source: §A.1.1 · p. 21
  • DBSCAN clustering of Common Crawl DOM features yielded about 40 million layout-distinct candidates, from which language and format classification selected a balanced 1 million pages.Source: §A.1.1 · p. 21
  • Block-level annotations were generated with MinerU-HTML’s simplification algorithm and an LLM, producing 870,945 annotated samples after filtering.Source: §A.1.1 · p. 21

A.1.2 Training MinerU-HTML-Classifier

The classifier was fine-tuned on 870K samples using Qwen3-0.6B. The document-formatting appendix illustrates content-list conversion to Markdown and provides an example spanning raw source, Main-HTML, Markdown, and metadata.

A.1.2 Training MinerU-HTML-Classifier

  • Qwen3-0.6B was fine-tuned on the full 870K-sample dataset for four epochs.The base model has a 32K context window and supports over 100 languages.Source: §A.1.2 · p. 21

A.2 Document Formatting

  • Figure 8 presents pseudocode for converting the content list to Markdown.The text directs readers to the figure for the conversion.Source: §A.2 · p. 21, Figure 8 · p. 22
  • Figure 9 shows an example with raw source, ground-truth Main-HTML, its Markdown conversion, and fine-grained metadata.Source: Figure 9 · p. 22

B.1 Benchmark Construction

WebMainBench combines broad Common Crawl sampling with popular websites and defines main content through contextual relevance and human authorship. Pages receive multi-stage review and metadata for fine-grained analysis.

B.1 Benchmark Construction

  • The benchmark draws 90% of pages randomly from Common Crawl and 10% from top-ranking websites, covering 5,434 top-level and 5,904 second-level domains.Source: §B.1 · p. 23
  • Annotation includes article-integral material such as tables of contents and references, excludes independent sidebars and copyright footers, and targets human-generated content rather than associated auto-generated metadata.Source: §B.1 · p. 23
  • Each page passes an initial annotation, a second annotator’s review and correction, and a senior inspector’s final quality check; pages uninterpretable due to rendering issues are discarded.Source: §B.1 · p. 23
  • Metadata records language, conversational or normal style, complexity based on four normalized measures, and rich-content tags for tables, code, and formulas.Complexity categories use the 30th and 70th percentiles as dynamic thresholds.Source: §B.1 · p. 23
  • The structured benchmark table reports element counts and subtype breakdowns for formulas, code blocks, and tables across 545 pages.Source: Table 5 · p. 24

C Pretraining Details

The paper specifies the pretrained model architecture, training run, and optimizer configuration used in its experiments.

C Pretraining Details

  • The model has 24 layers, hidden size 2048, intermediate size 5504, 16 attention heads, 16 KV heads, head dimension 128, and 1,525,516,288 total parameters.Its RoPE theta is 1000000.0.Source: C Pretraining Details · p. 24
  • Pretraining uses a global batch size of 64, context length 4,096, 240,000 training steps, and 62B consumed tokens.Source: C Pretraining Details · p. 24
  • The training setup uses the Qwen3 tokenizer [27] with a 151,936-token vocabulary, a 1 × 10−4 learning rate, and AdamW with specified weight decay and beta values.The optimizer settings are Weight_decay = 0.01, β1 = 0.9, β2 = 0.95, and ϵ = 10−8.Source: C Pretraining Details · p. 24

D Evaluation Details

The evaluation details specify few-shot demonstrations and report performance across training checkpoints and task categories; the case study compares extraction of structured content across tools.

D Evaluation Details

  • Table 6 lists the number of few-shot in-context demonstrations used for each downstream task, with CSQA denoting CommonsenseQA.Source: Table 6 · p. 25, D Evaluation Details · p. 25
  • AICC maintains superior average accuracy to TfCC across 15 checkpoints spanning 4B to 63B training tokens.The full average-accuracy results are identified as Table 7.Source: Table 7 · p. 25
  • At the 63B-token final checkpoint, AICC performs best on General Knowledge and Reading Comprehension, while FineWeb leads on Reasoning.Table 8 compares AICC, TfCC, FineWeb, and RefinedWeb across the three task categories.Source: Table 8 · p. 25

E Structured Element Extraction Case Study

  • In the code examples, MinerU-HTML preserves indentation and code-block structure, whereas Trafilatura and Resiliparse lose whitespace or delimiters.Source: Figure 10 · p. 26, Figure 11 · p. 26
  • For mathematical formulas, MinerU-HTML preserves complete structure and LaTeX syntax, while the baselines omit formulas or lose formatting markers.Source: Figure 12 · p. 27, Figure 13 · p. 27
  • For tables, MinerU-HTML retains rows, columns, headers, and cell relationships that Trafilatura and Resiliparse collapse into unstructured text.Source: Figure 14 · p. 28, Figure 15 · p. 28
  • The annotation-tool screenshot shows main-content selection highlighted in blue alongside a real-time preview.Source: Figure 16 · p. 29

F AICC-TfCC pairwise document comparison

The pairwise document-comparison examples show preferences for AICC or TfCC across documents of differing or similar lengths, using a rating prompt that evaluates eight content-module types.

F AICC-TfCC pairwise document comparison

  • AICC is preferred in four illustrated comparisons, including cases where its document is longer, TfCC's is longer, or lengths are similar.Source: Figure 17 · p. 30, Figure 18 · p. 30, Figure 20 · p. 31
  • In one example where TfCC is longer than AICC, the TfCC document is preferred.Source: Figure 19 · p. 31
  • The rating prompt asks a judge to compare eight content-module types—code, math, table, image, list, title, paragraph, and other—and return a structured JSON score.Source: Figure 22 · pp. 33–34
Loading 2511.16397v2…