CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

1 Introduction

• Dataset documentation is fragmented and inconsistent, hindering discovery, reuse, and reproducibility while raising licensing, provenance, and responsible-use concerns.Machine-readable metadata standards are motivated by format differences, limited interoperability, and sparse descriptions of data collection and use.Source: §1 · p. 1
• Despite Croissant adoption by major platforms, creating its metadata remains manual and time-consuming, while existing automated approaches cover few fields and struggle with scientific-paper variability.The paper notes more than 500,000 public Hugging Face datasets and over 32,000 created monthly in 2025.Source: §1 · pp. 1–2
• CroissantMiner pairs 602 dataset papers with the full 30-field Croissant 1.1 schema, including core and Responsible-AI fields, enabling end-to-end evaluation on real papers.The benchmark includes 102 human-validated gold annotations and 500 LLM-generated silver annotations.Source: §1 · p. 2
• Single-pass extraction with Claude Sonnet 4.6 and a canonical prompt scores highest; all four evaluated agentic architectures underperform the same-backbone single-pass systems.The gap is most pronounced on long-form RAI fields that require integrating information across multiple paper sections.Source: §1 · p. 2
• The evaluation compares frontier and open-weight models and four agentic architectures using rule-based metrics and a human-audited LLM judge.The paper also analyzes annotation reliability and field-level extraction difficulty.Source: §1 · p. 2
• The authors release open-source software and a demo for generating Croissant metadata drafts for authors, reviewers, and users to inspect and refine.The data-creation pipeline uses a single-pass LLM to produce silver annotations and gold-split pre-fills, which annotators rate and adjudicate.Source: Figure 1 · p. 2, §1 · p. 3

From Documentation Frameworks to Machine-Readable Standards

• Datasheets, Data Statements, Model Cards, Data Cards, and Data Nutrition Labels offer systematic documentation, but their natural-language outputs are costly to create and difficult to standardize and reuse at scale.Croissant adds a machine-readable standard with a Responsible-AI extension.Source: §2 · p. 3
• Platforms can infer structural metadata from data files, but contextual and RAI information remains embedded in dataset documentation, motivating extraction directly from scientific papers.Benchmarks for metadata extraction remain limited and schema-specific.Source: §2 · p. 3

Automated Metadata and Documentation Generation

• Existing tools generate free-text cards, extract attributes for alternative schemas, or infer structural fields from dataset files, rather than producing Croissant-conformant contextual and RAI metadata.Examples include CardGen, Datadoc Analyzer, MOLE, MeXtract, and platform tools such as Hugging Face.Source: §2 · p. 3

3 The CroissantMiner Benchmark

• The benchmark pairs 602 ML dataset papers with the full 30-field Croissant 1.1 schema, including 102 human-validated gold and 500 silver papers.The corpus prioritizes highly downloaded Hugging Face datasets and spans vision, NLP, multimodal learning, audio, code, robotics, and medical AI.Source: §3 · p. 3

3.1 Gold Dataset: Human Validation Protocol

• The gold annotations cover 10 core descriptive and 20 Responsible-AI fields, and mark undocumented values as absent even when those values exist elsewhere.Annotators verified LLM pre-fills; 9,595 ratings covered 3,060 fields across 102 papers, and majority voting resolved 95.0% of cells.Source: §3 · p. 3, §3.1 · p. 4
• Gold-label agreement is high overall, but varies by field: constrained fields reach AC1 = 0.96, while open-ended RAI fields score as low as 0.45.Disagreement on sc:publisher (AC1 = −0.09) reflects ambiguity over whether the field refers to a dataset host, venue, or author affiliation.Source: §3.1 · p. 4

3.2 Silver Dataset: Annotations at Scale

• The 500-paper silver split uses Sonnet 4.5 pre-fills without human review, so its labels are not equivalent to gold.It is intended to support model development, large-scale corpus analysis, and metadata-trend studies beyond the manually validated subset.Source: §3.2 · p. 4

3.3 Dataset Statistics and Coverage

• Across the 602 papers, 19.2 of 30 fields are documented on average, with coverage higher in gold than silver: 21.7 versus 18.7.Gold cells are undocumented 27.7% of the time; long-form RAI fields often require cross-section synthesis and show low annotator agreement and large system errors.Source: §3.3 · p. 5

4 The CroissantMiner System

• The system extracts PDF text, removes reference sections, prompts an LLM to produce the 30-field record, and validates the JSON against the schema.Schema violations occurred in fewer than 1% of extractions.Source: §4 · p. 5

4.1 System Architectures

• The four agentic architectures cover role-based specialists, plan-and-critique, retrieval-based context narrowing, and dynamic tool use.Their prompts received comparable refinement effort, including anti-hallucination guidance and, where applicable, verify-and-correct passes.Source: §4.1 · p. 6
• The single-pass baseline uses a structured prompt with field-specific definitions, examples, and instructions to return null when information is absent.The prompt forbids external knowledge, and temperature, token limit, and model version are fixed for determinism.Source: §4.1 · p. 6
• Parallel Specialists partitions the 30 fields across five parallel calls, retries fields returned empty, then merges their outputs.The disjoint field groups avoid key conflicts, but the architecture provides no guidance about where fields are likely to appear.Source: §4.1 · p. 6
• Triage & Critique predicts which field groups are likely documented, extracts all fields with supporting quotes, then checks empty values and verifies long-form RAI quotes.It can fill some empty core fields from Hugging Face and Semantic Scholar metadata.Source: §4.1 · p. 6
• Locator-Extractor locates relevant paper sections before field-group extraction, then verifies empty fields and can enrich them from external sources.ReAct instead dynamically chooses fields and tools over multiple reasoning steps, with full paper text available at each turn.Source: §4.1 · pp. 6–7

5.1 Experimental Setup

• The 88-paper held-out test follows a 14-paper development split; structured fields use rule-based scoring, while long-form fields use an LLM judge.Source: §5 · p. 7, §5.1 · p. 7
• GLM-5 was selected as judge for its human agreement: κ = 0.890 initially, then κ = 0.708 with the final rubric and audited gold.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 Performance Analysis of Evaluated Systems

• No configuration exceeds 0.71 composite, and most systems score lower on long-form RAI fields than on core fields.Source: §5.2 · pp. 7–8
• Across evaluated systems, single-pass extraction consistently beats same-backbone agentic architectures, while composite scores range from 0.39 to 0.71.Source: §5.2 · p. 8, Table 2 · p. 8
• With Claude Sonnet 4.6, ReAct scores 0.652 (∆= −0.057), Parallel Specialists 0.647 (∆= −0.061), Triage + Critique 0.624 (∆= −0.085), and Locator-Extractor 0.566 (∆= −0.143).All four differences are statistically significant; single-pass also scores higher than every agentic architecture on GPT-5.4 and Gemini 3.1 Pro.Source: §5.2 · p. 8

5.3 Costs & Error Analysis

• On documented RAI cells, single-pass leaves 2.4% empty versus Parallel Specialists 10.9%, Locator-Extractor 12.9%, ReAct 13.4%, and Triage + Critique 14.8%.Triage + Critique’s explicit-evidence and quote checks are intended to prevent unsupported answers, at the cost of sometimes leaving documented fields empty.Source: §5.3 · p. 9
• Among flagged pre-fill ratings, incomplete extraction (43.0%) and wrong-section assignment (19.5%) dominate; sparse documentation, not page count, predicts single-pass difficulty.Errors cluster in RAI attributes requiring multi-step reasoning or aggregation; structured core fields are generally extracted reliably.Source: §5.3 · p. 9, Figure 3 · p. 10
• Agentic configurations cost 1.2× to 7.8× as much per paper as matched-backbone single-pass extraction without improving composite scores.Under the paper’s token-accounting assumptions, this makes the tested agentic approaches suboptimal across the evaluated budget tiers.Source: §5.3 · p. 9

• CroissantMiner benchmarks metadata extraction on 602 dataset papers annotated with the full 30-field Croissant schema.Source: §6 · p. 10
• Across evaluated systems, single-pass extraction consistently outperforms the tested agentic architectures.Source: §6 · p. 10
• The gap is largest for long-form RAI fields that require integrating information across multiple paper sections.Source: §6 · p. 10

Limitations

• Scores entangle extraction capability with documentation completeness; 27.7% of gold cells have no documented value, so low scores on sparse fields are not extraction failures alone.Source: Limitations · p. 10
• Tier-2 scores rely on an LLM judge whose agreement with humans is imperfect, especially on difficult fields, and do not replace full human evaluation.Source: Limitations · p. 10
• Gold labels began as LLM pre-fills; re-annotation left rankings stable but favored the seed model, so some pre-fill anchoring cannot be ruled out.Source: Limitations · p. 10
• Some test fields have as few as two documented cells, and pretraining contamination cannot be ruled out; the benchmark covers English-language ML papers and Croissant 1.1.Extension to other languages, domains, and future schema versions is left to future work.Source: Limitations · p. 10

A.1 Croissant Field Definitions

• Croissant 1.1 defines 30 targets: 10 core descriptive fields and 20 RAI fields, scored through constrained, short-text, or LLM-judge metrics.Source: §A.1 · p. 15

A.2 Annotation Pipeline Phases

• Human annotation used calibration, a full-grid pass, redistribution and clean-up, then senior adjudication; final gold values came from majority votes over 9,595 ratings.Every cell was brought to at least three distinct annotators before adjudication of cells without a majority.Source: §A.2 · pp. 15–16

A.3 Annotation Reliability

• Annotation agreement was high overall (95.0% majority consensus) but varied by field, with constrained fields showing higher agreement than open-ended fields.Figure 4 reports Gwet’s AC1 with bootstrap confidence intervals and reports Krippendorff’s α alongside it to account for skewed label distributions.Source: Figure 4 · p. 17, §A.3 · p. 17

A.4 Robustness to the Seed Model

• System rankings remain similar across checks: re-annotation gives Spearman ρ = 0.903, human-rewritten cells ρ = 0.882, and paraphrased gold ρ = 0.98.The re-annotation involved 10 papers; paraphrasing changed wording while preserving meaning.Source: §A.4 · pp. 17–18
• Seed models benefit from their own pre-fills: GPT-5.4 shifts from 0.673 to 0.930 and Sonnet 4.5 from 0.824 to 0.634; the 10-paper re-annotation cannot support field-level conclusions.A smaller seed-family effect also warrants care in cross-family comparisons; single-pass extraction still beats agentic architectures for every backbone in the pilot.Source: §A.4 · p. 18

B Synthesis and Implications

• CroissantMiner is positioned as a draft-then-review workflow that flags low-confidence fields for author checks rather than unattended metadata production.Per-field accuracy and inter-annotator agreement can guide review, while fields with low agreement should require explicit author sign-off.Source: B Synthesis and Implications · pp. 18–19

C.1 From Documentation Frameworks to Machine-Readable Standards

• Croissant combines dataset, resource, record-structure, and ML-semantic metadata with 20 RAI attributes, but platform-generated metadata covers structural fields while contextual RAI information remains in papers.Croissant is described as adopted across platforms and required for NeurIPS dataset submissions.Source: §C.1 · p. 19

C.2 Automated Metadata and Documentation Generation

• Existing systems produce free-text cards or use schemas other than Croissant, while platform tools supply structural metadata complementary to CroissantMiner’s contextual and RAI extraction.The benchmark is presented as the first to pair scientific papers with Croissant-RAI-conformant gold annotations.Source: §C.2 · p. 20

C.3 LLM-Based Structured Extraction and Evaluation

• Prior work on structured extraction motivates detailed per-field guides, while LLM-judge research informs evaluation beyond exact match.The cited work includes JSON extraction from scientific text, conversational prompt engineering, schema-conformant generation, and key-level or structured-output benchmarks.Source: §C.3 · pp. 20–21

D.1 Canonical Extraction Prompt

• The canonical extraction prompt combines expert-role framing, accuracy-first instructions, Croissant field definitions, and the schema in an approximately 2,100-word prompt.Figure 5 shows the prompt used to extract Croissant metadata fields.Source: Figure 5 · p. 21, §D.1 · p. 22

D.2 LLM Judge Prompt (Tier-2)

• For each paper-field cell, GLM-5 receives the human-validated reference and candidate extraction, then returns a structured 1–3 score with a written reason.The prompt is reproduced in Figure 6; parse failures affected fewer than 1% of cells in released runs.Source: §D.2 · p. 22, Figure 6 · p. 22
• The deployed judge achieves 71.5% exact agreement with human consensus (κ = 0.663), and its audited system means correlate with consensus at ρ = 0.746.Of 57 disagreements, 45 involved a more lenient judge rating; the audit covered 200 RAI cells and does not exactly match the full reported lineup.Source: §D.2 · p. 22, Table 6 · p. 23

E System Implementation Details

• Tier 1 scores seven enum-like fields by exact match and three short-text fields by token F1; Tier 2 uses a GLM-5 judge for the remaining long-form RAI fields.System comparisons use paper-clustered bootstrap intervals and paired tests, with null-null cells skipped and hallucinated or missing values scored zero.Source: E System Implementation Details · p. 23

F.1 Per-Field Results

• The per-field results report ten core fields under rule-based metrics and twenty RAI fields under the GLM-5 judge; fields with small support have unstable scores.The tables cover the 88-paper test split and give the number of papers whose gold annotation documents each field.Source: §F.1 · p. 24, Table 7 · p. 24, Table 8 · p. 24

F.2 Silver split tracks the gold split

• Gold and silver per-field fill rates correlate at Pearson r = 0.839 and Spearman ρ = 0.823; cr:isLiveDataset is a notable exception because the silver model emits it only when text mentions live updates.The reported comparison covers 30 fields; the silver split is a Sonnet 4.5 extraction without human verification.Source: §F.2 · pp. 24–25, Figure 8 · p. 26

F.3 PDF Parser Ablation

• With Docling full export, parser scores stay within 0.01 overall for each model, while its body-only export costs 0.10 core for Sonnet 4.6 and 0.12 for GPT-5.4, mostly on datePublished and citeAs.A legacy Type 3 font report is mishandled by Docling’s default parser; the study retains the default parser because it tests Docling as shipped.Source: §F.3 · pp. 25–26, Table 9 · p. 26

F.4 Architectural Failure-Mode Analysis

• In sampled agentic errors, leaving documented fields empty is most common for ReAct, Parallel Specialists, and Triage + Critique, while incomplete answers dominate Locator-Extractor errors.Some empty fields contain the needed information under another field; one exhibit shows Locator-Extractor returning only Web Scraping while omitting other documented collection steps.Source: §F.4 · pp. 27–28

F.5 Popularity, Recency and Documentation Sparsity

• Score correlates with documentation sparsity (ρ = −0.40 for Sonnet 4.6; ρ = −0.61 for GPT-5.4), but not paper length; the analysis finds no evidence of popularity or recency effects within the tested scope.The authors note that these analyses cannot rule out pretraining contamination entirely.Source: §F.5 · p. 29

F.6 Scaling Within a Model Family

• Qwen3 composite scores rise from 0.376 at 4B parameters to 0.528 at 32B, with most gains in long-form RAI fields, yet Qwen3-32B remains below frontier single-pass systems.The RAI score rises from 0.272 to 0.484, while constrained core scores stay within a narrow band.Source: §F.6 · p. 29, Table 11 · p. 30

G Per-System Cost

• Per-system costs are estimated from recorded token usage at standard public list prices, excluding unrecorded usage and accounting for caching only where cached-token counts are available.The cost table compares systems with single-pass extraction on the same backbone, with stated exceptions for hybrid and self-hosted systems.Source: G Per-System Cost · p. 30, Table 12 · p. 30

• The headline isolation analysis pins all four agentic patterns to Sonnet 4.6; cross-backbone variants are reported where compute and time allowed.Source: H Agentic Configurations: Full… · p. 31
• Across backbones, all four agentic patterns scored below matching single-pass systems.Source: H Agentic Configurations: Full… · p. 31
• Triage + Critique composites were 0.667 (Sonnet 4.5), 0.624 (Sonnet 4.6), 0.557 (GPT-5.4), and 0.436 (Gemini 3.1 Pro).The Sonnet 4.5 run was gold-seed and excluded from the headline ranking.Source: H Agentic Configurations: Full… · p. 31
• Locator-Extractor composites were 0.556 (Sonnet 4.5), 0.566 (Sonnet 4.6), 0.502 (GPT-5.4), 0.448 (Gemini 3.1 Pro), and 0.478 (hybrid).Sonnet 4.5 was a gold-seed run; the locator-buffer approach was the most decomposed pattern and had the largest accuracy loss.Source: H Agentic Configurations: Full… · p. 31
• Parallel Specialists scored 0.647 (Sonnet 4.6), 0.589 (GPT-5.4), and 0.539 (Gemini 3.1 Pro), below matching single-pass systems on every backbone.Source: H Agentic Configurations: Full… · p. 31
• ReAct scored 0.652 (Sonnet 4.6), 0.631 (GPT-5.4), and 0.582 (Gemini 3.1 Pro), below matching single-pass systems on every backbone.The released ReAct variant is [v3]; earlier Sonnet 4.6 variants [v1] and [iter-5] were excluded from the headline ranking.Source: H Agentic Configurations: Full… · p. 31

PDF Parser Noise

• PDF extraction linearizes tables and figures and omits equations; Docling showed no consistent gain on table-heavy fields.The authors attribute errors on those fields to evidence reporting and interpretation rather than text extraction; vision-language preprocessing remains untested.Source: I Limitations and Future Work · p. 31

English-only Gold Split

• The 102 gold papers were selected by Hugging Face download rank, which favors English-language datasets despite language-agnostic schema fields.Reporting conventions differ across language communities, and multilingual annotation would require recruiting annotators with relevant language coverage.Source: I Limitations and Future Work · p. 31

Benchmark Scale and Contamination Caveat

• The 102-paper gold split supports per-field estimates but is too small for tight domain stratification, while the 500-paper silver split cannot replace gold evaluation.The authors also cannot rule out training-data contamination because most gold papers predate model release windows and model-side training disclosures are unavailable.Source: I Limitations and Future Work · p. 31

Gold-seed Dependency

• Audit revised 191 of 3,060 cells, but the gold retains Claude Sonnet 4.5’s filled-versus-null pattern, inflating its composite and excluding it from headline ranking.Re-annotating 10 papers from GPT-5.4 pre-fills left other systems’ rankings stable, while the seed model and, to a lesser degree, its family gained.Source: I Limitations and Future Work · pp. 31–32

Single LLM Judge

• The deployed single GLM-5 judge achieved κ = 0.708 on the 60 development cells and κ = 0.663 on a separate 200-cell audit; no ensemble was used.The authors propose evaluating whether an ensemble improves human agreement and whether rankings are sensitive to judge choice.Source: I Limitations and Future Work · p. 32

Residual LLM Non-determinism

• Temperature-zero runs remain nondeterministic for frontier models on long contexts; bootstrap intervals omit run-to-run variability, which was not separately quantified.Model versions and prompt hashes were pinned to mitigate reproducibility concerns.Source: I Limitations and Future Work · p. 32

Architecture Coverage

• Although architectural-isolation analysis pins the four agentic patterns to Sonnet 4.6, matching single-pass systems outperform them in all 12 architecture-backbone cells.The cross-backbone variants cover GPT-5.4 and Gemini 3.1 Pro, plus a Gemini-3.1-Pro and GPT-5.4-Mini locator hybrid.Source: I Limitations and Future Work · p. 32

Field-category Scope

• Structural fields and multiresource datasets with separate sub-resource metadata fall outside the 30-field Croissant 1.1 benchmark scope.The authors note that extending the schema would proportionally increase per-paper extraction work.Source: I Limitations and Future Work · p. 32