PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, Daniel Gomes, João Magalhães

arXiv:2609.07699v3cs.CLcs.AI

TL;DR

Regional-language web corpora are difficult to curate when standard filters confuse PT-PT with PT-BR and discard valid text. This work builds a PT-PT corpus from Arquivo.pt with a pipeline that cleans text before filtering. The post-scraping block increases final document yield by 19.04%.

  • Problem

    Standard web-scraping and language-identification filters can confuse PT-PT and PT-BR and discard valid PT-PT content, while large-scale documented PT-PT web-archive pipelines are lacking.

    Source: §1 · p. 1, European Portuguese Language Resources · p. 3

  • Method

    The paper processes Arquivo.pt web archives through a six-stage pipeline that includes a custom post-scraping block before filtering, deduplication, and quality classification.

    Source: §3.2 · p. 3, §3.2.2 · p. 4

  • Results

    19.04% more final documents were obtained through the post-scraping intervention, and the pipeline produced a 41-billion-token high-quality European Portuguese corpus.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The resulting PT-PT corpus is intended to support pre-training and mid-training of language models focused on European Portuguese.

    Source: §1 · p. 2

    Document-level heuristic filters remove whole documents when any quality threshold is exceeded, discarding some useful content mixed with low-quality text.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with rigorous language identification, weighted fuzzy deduplication, and neural quality classification, our pipeline offers a scalable framework and a clean, representative corpus optimized for LLM pre-training.

1 Introduction

The paper addresses the difficulty of building high-quality European Portuguese resources amid PT-BR dominance and filtering that can discard valid PT-PT text. It presents a large-scale curation pipeline whose post-scraping block raises retained-document yield and produces a corpus for language-model training.

1 Introduction

  • The pipeline processes Arquivo.pt collections representing approximately 411 TB of raw WARC data spanning 1997–2024.It combines geographical filtering, neural language identification, heuristic filtering, fuzzy deduplication, and neural quality classification.Source: §1 · p. 1
  • Standard scraping and language-identification filters can confuse PT-PT with PT-BR and discard valid PT-PT content at rigid heuristic boundaries.The paper situates this challenge amid limited resources for regional language variants.Source: §1 · p. 1
  • 19.04% more high-quality documents were retained after adding the post-scraping block, which removes short and duplicate lines before heuristic filters.The block is reported to rescue valuable text that standard filters would otherwise reject.Source: §1 · p. 1
  • The resulting corpus contains 41 billion tokens of high-quality European Portuguese text optimized for LLM pre-training and mid-training.The pipeline overview describes six sequential curation stages applied to 71 collections.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work established large-scale web-corpus processing and multilingual dataset construction, but regional variants can remain blurred or underrepresented. For European Portuguese, the paper identifies a gap in openly documented, large-scale web-archive processing pipelines.

2 Related Work

  • FineWeb and related pipelines combine filtering and deduplication, building on approaches from MassiveText and C4.FineWeb released an 18.5-trillion-token dataset alongside the Datatrove processing library.Source: §2 · p. 2
  • Web-corpus deduplication includes URL-level, exact, semantic, and MinHash near-duplicate methods, with some pipelines also using model-based filters.Prior work describes deduplication as improving model quality and reducing memorization.Source: §2 · p. 2
  • Multilingual web datasets have grown, but can blur regional variants and language-specific characteristics, motivating variant-specific corpus efforts.The cited work also emphasizes language-specific pipelines for quality.Source: §2 · p. 2

European Portuguese Language Resources

  • Earlier PT-PT resources include a 0.8-billion-token Arquivo.pt subset of .pt news sites; the paper addresses the lack of a large-scale, openly documented PT-PT web-archive pipeline.The earlier subset contained 1.5 million documents and used basic heuristic cleaning and exact-duplicate removal.Source: European Portuguese Language Resources · pp. 2–3

3 Corpus Creation Methodology

The methodology processes Arquivo.pt collections through a six-stage Datatrove-based pipeline. Its custom post-scraping block precedes filtering, while staged deduplication and quality classification produce the final corpus.

3.1 Data Collection

  • The team processed 71 collections totaling roughly 411 TB of WARC archives, selected across content types and crawl periods to limit duplicates while maintaining coverage.The collections span 1997–2024 and include frequently crawled, general-web, and smaller curated collections.Source: §3.1 · p. 3

3.2 Data Processing

  • The six-stage Datatrove pipeline comprises WARC processing, text post-scraping, data filtering, MinHash deduplication, quality classification, and cross-collection deduplication.It adapts FineWeb2’s Portuguese threshold configuration and adds custom processing components.Source: §3.2 · pp. 3–4

3.2.1 WARC Processing

  • WARC processing excludes .br or PT-BR-like URLs and blacklisted content, then extracts text while preserving page titles as metadata.The pipeline does not restrict collection to .pt domains because that would discard valid PT-PT content hosted elsewhere.Source: §3.2.1 · p. 4

3.2.2 Text Post-Scraping

  • The custom post-scraping block increased the final number of documents in the test collection by 19.04%, removing lines with fewer than 10 alphabetic characters and retaining one instance of duplicate lines per document.Source: §3.2.2 · p. 4
  • Post-scraping removes lines with fewer than 10 alphabetic characters and within-document duplicate lines before downstream heuristic filtering.The block retains one instance of each repeated line.Source: §3.2.2 · p. 4

3.2.3 Filtering

  • Filtering applies GlotLID with a 0.799 PT-PT probability threshold, followed by repetition and quality heuristics, formatting correction, and personal-data redaction.The quality heuristics assess n-gram repetition, line endings and newline ratios, word-count and word-length bounds, formatting markers, alphabetic content, and Portuguese stop words.Source: §3.2.3 · p. 4

3.2.4 Deduplication and Rehydration

  • Per-collection MinHash deduplication clusters duplicate documents and retains one representative; rehydration weights retained documents by cluster size, with caps for very large clusters.The implementation can alternatively retain duplicates and annotate cluster identifiers and sizes.Source: §3.2.4 · p. 5

3.2.5 Quality Classification

  • EuroFilter classifies documents after per-collection deduplication, and low-quality entries are discarded before the next stage.This classification step runs outside Datatrove.Source: §3.2.5 · p. 5

4 Data Curation Results

Tests on the FAWP21 collection show that post-scraping increases retained documents without changing downstream filters; across 71 collections, the pipeline produces a 41B-token corpus after cross-collection deduplication.

4 Data Curation Results

  • Pipeline design decisions are validated through experiments on a single collection, FAWP21, composed primarily of Portuguese news media, frequently updated websites, and government and public body pages, crawled between April and June 2015.Source: §4 · p. 5

4.1 Post-Scraping Steps

  • 10.94 million documents remain after both post-scraping steps, a 19.04% increase over baseline without reducing downstream quality checks.Short-line removal alone yields 9.92 million documents (+7.94%); 56.7% of documents reach the final Gopher Quality stage with both steps, versus 47.2% without post-scraping.Source: §4.1 · p. 6

4.2 Model-Based Quality Pre-Filtering

  • 0.4% more documents survive when EuroFilter quality classification precedes heuristic filtering, a marginal gain that does not justify its added processing time.The pipeline therefore uses heuristic-only filtering.Source: §4.2 · p. 6

4.3 Final Corpus Statistics

  • From approximately 5.4 billion raw entries, URL filtering leaves 3.4 billion, extraction leaves 1.7 billion, heuristic filtering leaves 731 million, and single-collection MinHash leaves 148 million.The last step reduces entries by about 80% from the previous stage and about 97% from raw input.Source: §4.3 · p. 6, Table 2 · p. 6
  • Cross-collection deduplication leaves 24.6M documents and 41B tokens, reducing the 176B-token corpus by 76.6%.Compared with the Arquivo.pt data used in GlórIA, the corpus has 16.4× more documents and 51.3× more tokens.Source: §4.3 · p. 6, Table 3 · p. 6
  • Media and blogs account for 41% and 43.7% of tokens, respectively, and over 90% of documents contain fewer than 4k tokens.Figure 4 presents the top 10 domains by entry count across categories; Figure 5 shows document token-length distribution.Source: Figure 4 · p. 7, §4.3 · p. 7, Figure 5 · p. 7

5 Experiments and Results

The experiments assess filter-stage text quality, quality thresholds, memorization, and PT-BR contamination. They report minimal verbatim memorization and find PT-PT remains the predominant variant in the final corpus.

5.1 Filter Stage Quality Ablations

  • FineWeb Quality and Gopher Quality removals have higher bits-per-byte than the final output across four models, while Gopher Repetition removals have the lowest bits-per-byte and highest top-1 accuracy.Figure 6 compares top-1 next-token accuracy and bits-per-byte on documents removed at each filter stage and the final output; repetitive text is inherently predictable.Source: §5.1 · pp. 7–8, Figure 6 · p. 8

5.1.1 Quality Thresholds Comparison

  • AMALIA achieves top-1 accuracy averages of 0.68 on medium-quality documents and 0.63 on high-quality documents.The evaluation compares the top 57–20% medium-quality split, top 20% high-quality split, and combined final output.Source: §5.1.1 · p. 8, Figure 7 · p. 8

5.2 Memorization Experiments

  • Over 99% of samples yield six or fewer consecutive matching words, and BLEU-4 remains below 0.169, indicating minimal memorization.AMALIA’s memorization metrics remain comparable to EuroLLM, its base model.Source: Table 4 · p. 8, §5.2 · p. 9, Figure 8 · p. 9

5.3 Brazilian Portuguese Contamination

  • 84.65% of the final corpus is estimated as PT-PT and 15.35% as PT-BR, averaging two variety classifiers.The individual classifiers assign 86.4% and 82.9% to PT-PT, respectively.Source: §5.3 · p. 9

6 Conclusion

The pipeline processes 411 TB of Arquivo.pt WARC archives and produces a 41-billion-token European Portuguese dataset. Its post-scraping block increases the final document count by 19.04%.

6 Conclusion

  • The pipeline combines URL, language, and heuristic-based filtering, model-based quality classification, and MinHash fuzzy deduplication to create a 41-billion-token European Portuguese dataset.It processes 411 TB of raw WARC archives spanning 1997 to 2024.Source: §6 · p. 9

Limitations

The corpus is skewed toward news and frequently updated websites, while document-level filtering trades some useful content for reduced risk of noisy text entering training.

Limitations

  • News media and frequently updated websites dominate the processed Arquivo.pt collections, potentially leaving academic, technical, and literary text underrepresented.Including general AWP and smaller curated collections was used to improve topical diversity.Source: Limitations · p. 10
  • Document-level heuristic filters discard an entire document when any quality threshold is exceeded, including documents with mostly high-quality text.Sentence- or paragraph-level filtering could recover some useful content, but would add complexity.Source: Limitations · p. 10
  • The document-level approach was deliberately chosen to minimise the risk of noisy or malformed text influencing LLM training, despite discarding some useful content.Source: Limitations · p. 10

Ethical Considerations

The pipeline followed stated ethical and legal guidelines, including removal of personal data and respect for copyright, licensing, and robots.txt directives. Collection selection was tailored to the different contents and crawling patterns of Arquivo.pt collections.

Ethical Considerations

  • Personal data was removed from the final corpus, and collection followed publicly available information, copyright and licensing restrictions, and each domain’s robots.txt directives.Source: Ethical Considerations · p. 10

A Collection Selection Details

  • Because Arquivo.pt collection types vary in content, focus, and crawling frequency, each required examination and a tailored processing procedure.Source: A Collection Selection Details · p. 13
  • FAWP selections covered nearly every quarterly collection from 2010–2016 and 2020–2024, while AWP selections sampled roughly one similar-sized collection per year from 2008–2024.FAWP includes frequently updated websites; AWP consists of complete Portuguese Web crawls, mainly from .pt, with blogs and general content.Source: A Collection Selection Details · p. 13
  • Smaller collections included curated, donated, special-event, and ArchivePageNow pages from 1997–2024 to improve variety and quality.Source: A Collection Selection Details · p. 13

B Dropped Samples per Stage

Documents preserved in early stages still face subsequent heuristic filtering, and Figure 9 shows how samples change across pipeline stages.

B Dropped Samples per Stage

C Rescued Samples by the Post-Scraping Stage

Table 5 compares raw scraped documents with their post-scraping versions and illustrates cases that would otherwise be discarded by downstream heuristic filters.

C Rescued Samples by the Post-Scraping Stage

D Removed Samples

The section presents examples of discarded documents by filter and examines exceptional cases of model memorization.

E Memorization Example

  • Figure 9 compares dropped-document counts across post-scraping approaches at each filtering stage.Source: Figure 9 · p. 14
  • Table 5 compares rescued documents before and after post-scraping, labeling the issue that would otherwise have triggered heuristic filtering.Source: Table 5 · p. 14

D Removed Samples

E Memorization Example

  • Among the five most verbatim AMALIA reproductions, the longest exact continuation is 26 consecutive words.Table 7 marks exact reproductions in green, divergences in red, and the ground-truth continuation in blue.Source: E Memorization Example · p. 13, Table 7 · p. 16
Loading 2609.07699v3…