PaperlayerView exact revision on arXiv

Source-linked AI summary

FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition

Jonas Golde, Patrick Haller, Alan Akbik

arXiv:2512.13884v1cs.CL

TL;DR

Multilingual NER resources have often lacked both broad language coverage and rich label sets. FiNERWEB uses automated passage selection and multilingual LLM annotation to build a dataset spanning 91 languages and 25 scripts. Models trained on selected FiNERWEB splits achieve comparable or improved performance to current state-of-the-art models, and annotation analyses report strong cross-lingual consistency.

  • Problem

    Existing multilingual NER resources tend to offer either broad language coverage or rich label sets, while universal NER models mostly focus on English.

    Source: §1 · p. 1

  • Method

    FiNERWEB rates passages with multilingual LLMs, trains a regression model to select NER-relevant text, then annotates selected passages and translates labels.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    Models trained on selected FiNERWEB splits achieve comparable or improved performance to current state-of-the-art models, and annotation analyses find strong cross-lingual consistency.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The released dataset variants and passage-selection models serve benchmarking and large-scale data creation.

    Source: §5 · p. 9

    Annotation quality and coverage are constrained by uneven multilingual LLM accuracy across low-resource languages, scripts, and domains, which may bias evaluations.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Recent multilingual named entity recognition (NER) work has shown that large language models (LLMs) can provide effective synthetic supervision, yet such datasets have mostly appeared as by-products of broader experiments rather than as systematic, reusable resources. We introduce FiNERweb, a dataset-creation pipeline that scales the teacher-student paradigm to 91 languages and 25 scripts. Building on FineWeb-Edu, our approach trains regression models to identify NER-relevant passages and annotates them with multilingual LLMs, resulting in about 225k passages with 235k distinct entity labels. Our experiments show that the regression model achieves more than 84 F1, and that models trained on FiNERweb obtain comparable or improved performance in zero shot transfer settings on English, Thai, and Swahili, despite being trained on 19x less data than strong baselines. In addition, we assess annotation quality using LLM-as-a-judge and observe consistently high scores for both faithfulness (3.99 out of 5) and completeness (4.05 out of 5), indicating reliable and informative annotations. Further, we release the dataset with both English labels and translated label sets in the respective target languages because we observe that the performance of current state-of-the-art models drops by 0.02 to 0.09 F1 when evaluated using target language labels instead of English ones. We release FiNERweb together with all accompanying artifacts to the research community in order to facilitate more effective student-teacher training for multilingual named entity recognition.

1 Introduction

Existing multilingual NER resources tend to offer either broad language coverage or rich label sets, while universal NER models mostly focus on English. FiNERWEB combines automated passage selection and LLM annotation in a scalable pipeline, releasing a dataset covering 91 languages and 25 scripts.

1 Introduction

  • LLMs can supply synthetic NER supervision for smaller student models, but prompting can require separate label-specific prompts or postprocessing.The cited work describes these students as distilling LLM knowledge and sometimes outperforming their larger teachers.Source: §1 · p. 1
  • Existing multilingual resources often provide broad language coverage or rich label sets, while universal NER models mostly focus on English.PAN-X covers over 150 languages with coarse entity types, whereas DynamicNER supports over 100 label types across 12 comparatively high-resource languages.Source: §1 · p. 1
  • FINERWEB provides an LLM-annotated dataset spanning 91 languages and 25 scripts, addressing a gap in multilingual NER resources.The release includes regression models and their training data, and the paper reports releasing all accompanying artifacts.Source: §1 · pp. 1–2
  • The pipeline rates passages for NER usefulness, trains a regression model to filter FineWeb-2, then annotates selected passages with multilingual LLMs.The selected passages are annotated using GPT-4o mini and Gemma3-27B; their annotations are merged and English labels are translated into target languages.Source: §1 · p. 2
  • Experiments assess training on selected dataset splits, annotation quality with LLM-as-a-judge, label translation, and the distribution of entity types.The introduction reports comparable or improved performance against current state-of-the-art models, without specifying the metrics in these passages.Source: §1 · p. 2

2 FiNERweb

FiNERweb filters FineWeb-2 passages for NER suitability, then annotates them with two multilingual LLMs and merges their outputs. The resulting resource spans 91 languages and 25 scripts.

2.1 Dataset Source

  • FineWeb-2 is restricted to languages used to train XLM-RoBERTa, yielding 91 languages in their native scripts across 25 scripts.This choice supports compatibility with XLM-RoBERTa and similar models.Source: §2.1 · pp. 2–3

2.2 Stage 1: Selecting High-Quality Passages

  • LLMs rate 1,000 passages per language on a four-point NER-usefulness scale; a regression model learns to predict those scores from passages.Training minimizes mean squared error between predicted and LLM-assigned scores.Source: §2.2 · pp. 3–4
  • 84.1 F1 (binary, GPT-4o mini annotations, XLM-RoBERTa) exceeds 77.3 F1 (binary, Gemma3 annotations, XLM-RoBERTa), and the former model is selected for filtering.The regression models are evaluated using macro-averaged precision, recall, and F1 across languages.Source: Table 1 · p. 3, §2.2 · p. 4

2.3 Stage 2: Filtering FineWeb-2

  • For FineWeb-2, the selected regression model computes usefulness scores on 256-token chunks and retains chunks scoring > 0.5 (equivalent to passage score >= 3), subject to language-identification quality control.Source: §2.3 · p. 5

2.4 Stage 3: Dataset Annotation

  • The annotation pipeline reconciles overlapping spans, merges sufficiently similar labels, retains annotations from both LLMs, and translates label sets when supported by Google Translate.For overlapping spans of at least 50%, labels with similarity above 0.75 are merged.Source: §2.4 · pp. 5–6
  • FiNERweb averages more distinct annotations per sentence and covers more distinct entity types than NuNER, PileNER, and Euro-GLiNER-X.Table 3 compares these universal NER datasets.Source: Table 3 · p. 5, §2.4 · p. 6

3 Ablations

FiNERWEB supports zero-shot transfer to human-labeled English, Thai, and Swahili benchmarks, with performance comparable to or better than multilingual GLiNER baselines despite limited training data. Analyses also find high annotation quality, reduced label separability with translated labels, and a long-tail distribution of model confidence.

3.1 Downstream Performance

  • Comparable or improved performance on CoNLL-2003, ThaiNER, and MasakhaNER was achieved with 7.5k multilingual or 2.5k per-language passages, though joint training reduced Thai performance.The authors attribute the Thai decrease to differences in positive-to-negative example ratios and Binder’s lack of a word-segmentation requirement.Source: §3.1 · p. 6

3.2 Annotation Quality

  • Only 21 of 91 evaluated languages averaged below 4.0 on both faithfulness and completeness; across-language error analysis found 6.12% of entities missing and 5.97% incorrectly labeled.The authors identify under-annotation, rather than hallucination, as the primary source of error.Source: §3.2 · p. 7

3.3 Label Similarity

  • Translated labels show higher semantic overlap than English labels, and GLiNER performance consistently decreases on translated evaluations; mutually exclusive losses treat equivalent translations as hard negatives.The authors suggest objectives that account for semantic overlap, including shared concept identifiers or multiple equivalent positives.Source: §3.3 · p. 8, Figure 5 · p. 8

3.4 Confidence-Based Partitioning

  • Approximately 50% of predicted gold spans receive confidence above 0.97, while confidence varies by entity type, with “scientific concept” at 0.532 in one example.The analysis collects confidence scores for predicted gold spans at confidence >= 0.5 across five folds and three seeds.Source: §3.4 · p. 8

4 Related Work

Prior work uses LLM prompting for zero-shot NER and synthetic data to train more efficient student models. Existing systems and datasets differ in label coverage, while FiNERWEB contributes multilingual training data and confidence-based splits.

Zero-Shot NER

  • LLM prompting has enabled zero-shot NER, while synthetic LLM-generated data supports distilling knowledge into more efficient student models.The related work notes that prompting may require separate prompts per label type or postprocessing to align generated annotations with source text.Source: §4 · pp. 8–9

Synthetic Dataset Generation

5 Conclusion

FiNERWEB is a multilingual NER dataset suite accompanied by passage-selection models and training data. Its analyses assess quality and design choices, finding reduced label separability with translated labels and under-annotation as a key improvement target.

5 Conclusion

  • FiNERWEB spans more than ninety languages and includes 12 variants differing in annotation schema, label set, and annotator model.The release also includes regression models and training data for selecting passages, supporting dataset scaling, benchmarking, and large-scale data creation.Source: §5 · p. 9
  • The analyses find that translated labels reduce separability under classical losses and identify under-annotation and label translation as directions for improvement.The paper also reports confidence-based subsets and LLM-as-a-judge assessment of annotation faithfulness and completeness.Source: §5 · p. 9

Limitations

FiNERweb’s annotations are constrained by uneven multilingual LLM capability, a 91-language scope, and a release size fixed for manageability and verification.

Limitations

  • Uneven LLM accuracy across low-resource languages, scripts, and domains may bias evaluations toward languages where annotator models are more reliable.The quality and coverage of annotations inherit this unevenness.Source: Limitations · p. 10
  • The 91-language set excludes some linguistic diversity, including languages with limited digital presence, endangered languages, and languages using poorly supported scripts.Extending coverage would require additional annotation resources and quality control.Source: Limitations · p. 10
  • The 250k-sample release was fixed for manageability and thorough quality verification, not as the pipeline’s upper bound.The annotation and processing steps are designed to scale further if broader coverage or more samples are needed.Source: Limitations · p. 10

A Dataset Overview

The appendix provides a per-language overview using the main section’s metrics and distinguishes selected metrics by annotation model.

A Dataset Overview

  • Tables 8 and 9 give a per-language overview with the main section’s metrics, differentiating selected metrics by annotation model.Source: A Dataset Overview · p. 14

B Entity Extraction Algorithm

The entity extraction algorithm grounds LLM-produced entity mentions in the original text by locating exact matches and recording typed character spans.

B Entity Extraction Algorithm

  • The algorithm takes the original text and LLM-annotated entities, each specified by surface form and entity type.Source: B Entity Extraction Algorithm · p. 14
  • For each mention, it finds every exact string occurrence and records its start and end offsets, matched text, and entity type.The search advances one character at a time to capture all occurrences.Source: B Entity Extraction Algorithm · p. 14
  • It sorts the resulting spans by starting position, making the grounding procedure deterministic and computationally efficient.Source: B Entity Extraction Algorithm · p. 14

C Selected Multilingual Examples

The multilingual qualitative analysis finds that the regression models distinguish passages with clear, contextually relevant entity mentions from noisy or off-topic reviews beyond English.

C Selected Multilingual Examples

  • Across the analyzed languages, high-scoring passages contain clear, contextually relevant entity mentions, while low-scoring examples are dominated by noisy or off-topic reviews.The authors report that this distinction generalizes beyond English.Source: C Selected Multilingual Examples · p. 14

D Prompts

The appendix presents prompts and procedures for filtering NER-useful passages, extracting and typing entities, and assessing annotation quality. It also includes qualitative examples and tables describing FiNERweb.

D.1 Preference Data Prompt

  • The preference-data prompt is used to train the regression model that filters for high-quality passages.Figure 8 shows the prompt; Table 7 presents examples with regression scores above 3.5 for high-quality passages and below 0.5 for low-quality passages.Source: §D.1 · p. 14, Table 7 · p. 15, Figure 8 · p. 16
  • FindEntitySpans takes text and typed entity strings, records matching spans with offsets, and returns the spans sorted by start position.The output schema lists start, end, text, and type for each span.Source: §D.1 · p. 15

D.2 Entity Extraction Prompt

  • The entity-extraction prompt asks for all named entities in a passage and their types, seeking a diverse and comprehensive set.Figure 9 identifies the prompt used on filtered FineWeb-2 passages.Source: §D.2 · p. 15, Figure 9 · p. 17, §D.3 · p. 17

D.3 Quality Assurance Prompt

  • The NER-usefulness prompt applies a four-point additive rubric based on entity clarity and context, cleanliness, domain diversity, and suitability for NER training or evaluation.The rubric awards points for these properties in sequence.Source: §D.3 · p. 16
  • The annotation-quality prompt asks a model to score faithfulness and completeness and list missing, extra, and incorrect annotations.Figure 10 describes its inputs as a sentence and proposed annotations, with JSON as the output format.Source: Figure 10 · p. 18
  • The appendix includes full-overview tables for FiNERweb, with headings for language, script, document count, average length, labels per document, and unique labels.Tables 8 and 9 are identified as the full overview and its continuation.Source: §D.3 · p. 19, Table 8 · p. 19, Table 9 · p. 20
Loading 2512.13884v1…