PaperlayerView exact revision on arXiv

Source-linked AI summary

Croissant: A Metadata Format for ML-Ready Datasets

Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Joan Giner-Miguelez, Pieter Gijsbers, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Michael Kuchnik, Satyapriya Krishna, Sylvain Lesage, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Hamidah Oderinwale, Pierre Ruyssen, Tim Santos, Rajat Shinde, Elena Simperl, Arjun Suresh, Goeffry Thomas, Slava Tykhonov, Joaquin Vanschoren, Susheel Varma, Jos van der Velde, Steffen Vogler, Carole-Jean Wu, Luyao Zhang

arXiv:2403.19546v3cs.LGcs.AIcs.DBcs.IR

TL;DR

ML data work faces friction from varied formats, limited interoperability, and difficulty discovering and combining datasets. Croissant provides a layered metadata vocabulary for ML datasets; human raters found its metadata readable and understandable, and the paper concludes that it was also complete and concise.

  • Problem

    ML data work is hindered by varied data formats, limited interoperability between tools, and difficulty discovering and combining datasets.

    Source: §1 · p. 1

  • Method

    Croissant is a community-driven, Schema.org-based dataset metadata vocabulary organized into dataset metadata, resource, structure, and semantic layers.

    Source: §3 · p. 3

  • Results

    Human raters found Croissant metadata readable, understandable, complete, and concise.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Croissant is already adopted by popular dataset repositories and frameworks, while its longer-term success depends on further adoption and broad availability and support.

    Source: §6 · p. 10

    The usability study drew annotators from the Croissant community, which may have introduced bias, and the authors call for evaluation with non-expert users.

    Source: §5 · p. 10

Abstract

from arXiv · show

Data is a critical resource for machine learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that creates a shared representation across ML tools, frameworks, and platforms. Croissant makes datasets more discoverable, portable, and interoperable, thereby addressing significant challenges in ML data management. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, enabling easy loading into the most commonly-used ML frameworks, regardless of where the data is stored. Our initial evaluation by human raters shows that Croissant metadata is readable, understandable, complete, yet concise.

1 Introduction

Croissant is introduced as a shared metadata format intended to make ML datasets easier to discover, use, and share across tools and platforms. The paper also reports a preliminary usability evaluation involving annotations of ten widely used datasets.

1 Introduction

  • Croissant records dataset attributes, resources, structure, and semantics in a shared format designed to improve discoverability, portability, reproducibility, and interoperability.Its ML-specific metadata enables direct loading into ML frameworks and tools.Source: §1 · p. 1
  • Croissant provides a unified view across common ML data types and layouts without requiring changes to the underlying data representation.It adds semantic descriptions and ML-specific information to existing datasets.Source: §1 · p. 1
  • The preliminary usability evaluation gathered practitioner annotations of ten widely used language, vision, audio, and multimodal datasets and analyzed response consistency and feedback.The study assessed metadata creation with Croissant.Source: §1 · p. 2

2 Related Work

Prior dataset metadata standards and documentation efforts offer useful foundations, but the paper identifies gaps in ML-specific support and machine readability. Croissant is presented as a standardized, machine-readable framework that builds on these efforts.

Standards for Catalogs and Metadata

  • Existing metadata standards often lack ML-specific support, compatibility with existing ML tools, or responsiveness to evolving needs such as responsible ML.The paper notes that standards including DCAT and schema.org serve broader cataloging and metadata needs, while tabular-data and domain-specific efforts address narrower settings.Source: §2 · p. 2

Operationalizing Responsible ML through Data Work

  • Dataset documentation efforts such as Datasheets for Datasets, Data Statements, Data Cards, and Data Nutrition Labels address responsible data work, but typically rely on natural language rather than a standard machine-readable representation.Repositories including Kaggle, OpenML, and Hugging Face have also developed documentation efforts.Source: §2 · p. 3
  • Croissant fills the machine-readability gap with standardized dataset documentation intended to provide semantic consistency and integrate with ML community tools and frameworks.The paper connects dataset documentation to discoverability, interoperability, and usability.Source: §2 · pp. 2–3

3 The Croissant Format

Croissant describes datasets through layered metadata, resource, structure, and semantic descriptions, with an extension for responsible-AI information. Its tools and integrations connect the format to repositories, search, and ML frameworks.

3 The Croissant Format

  • Croissant is a community-driven vocabulary built on Schema.org, organized into dataset metadata, resource, structure, and semantic layers.The section also introduces its Responsible AI extension and supporting integrations.Source: §3 · p. 3

3.1 The Dataset Metadata Layer

  • Dataset metadata builds on schema.org/Dataset, specifying required, recommended, and optional properties and adding properties for snapshots, live datasets, and citations.The PASS example illustrates dataset metadata and resources.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2 The Resources Layer

  • FileObject describes an individual resource file, while FileSet describes sets of files and can use inclusion and exclusion filters.The PASS example includes a metadata CSV, an image archive, and a FileSet for data files.Source: §3.2 · p. 4

3.3 The Structure Layer

  • RecordSet provides a common structure for multi-field records across modalities, including joins between images and structured metadata.Fields identify their data sources; nested RecordSets and extraction or transformation methods support additional structures.Source: §3.3 · pp. 4–5

3.4 The Semantic Layer

  • The semantic layer types dataset splits, labels, and common ML data such as bounding boxes, categorical values, and segmentation masks.It uses Croissant primitives and can map fields to schema.org data types such as GeoCoordinates.Source: §3.4 · p. 5

3.5 The Croissant-RAI Extension

  • Croissant-RAI extends Croissant to help publish, discover, and reuse responsible-AI metadata, with properties developed and evaluated against RAI use cases.Its use cases include data life cycles, labeling and participatory processes, AI safety, fairness assessments, and regulatory compliance.Source: §3.5 · p. 5

3.6 Croissant Tools and Integrations

  • Hugging Face Datasets, Kaggle Datasets, and OpenML together describe over 400,000 datasets in Croissant, and Google Dataset Search offers a filter for Croissant datasets.The Python reference library supports validation and data loading integrations; TensorFlow Datasets and PyTorch DataPipes provide framework paths, while the Editor supports visual metadata creation and validation.Source: §3.6 · pp. 5–6

3.7 The Croissant Working Group

  • The Croissant Working Group includes stakeholders and domain experts from academia, industry, research organizations, and collaborative networks, and the format is designed to be modular and extensible.The group discusses use cases and collects feedback for future versions.Source: §3.7 · p. 6

4 Croissant Evaluation: A User Study with ML Practitioners

The study assessed Croissant metadata through practitioner annotations of ten ML datasets and evaluations of five vocabulary criteria. Ratings were generally positive, while text-attribute agreement varied by attribute type and between Croissant and Croissant-RAI.

4.1 The User Study Process

  • Nine Croissant-community volunteers produced 30 annotations across ten datasets, using a 16-attribute JSON template and BLEU to assess agreement on text fields.Datasets spanned language, vision, and audio; selection required an existing Croissant description, an HF dataset card, and a publication describing dataset creation. The study selected ten Croissant and six Croissant-RAI attributes for manual specification.Source: §4.1 · pp. 6–7

4.2 Mapping Evaluation Criteria to Croissant

  • The evaluation mapped five vocabulary criteria to Croissant-specific questions about agreement, missing attributes, redundancy, intuitive names, and understandability.Annotators used Croissant specifications when assessing understandability.Source: §4.2 · pp. 7–8

4.3 Results and Discussion

  • 25 of 30 annotations found that Croissant attributes captured important dataset information.Figure 5 presents answers to the completeness question.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Around 60% of annotations found none or few attributes redundant, while about 23% flagged some as redundant or not definable, often Croissant-RAI attributes.Annotators sometimes lacked documentation for these fields or found the information difficult to extract; Figure 6 presents conciseness answers.Source: §4.3 · pp. 8–9, Figure 6 · p. 8
  • About 83% found attribute names intuitive and around 90% found the specifications understandable, though sc:creator and sc:publisher were commonly confused.Figures 7 and 8 present the readability and understandability answers.Source: §4.3 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
  • More than 75% of annotations received very confident correctness ratings, and around 80% of annotators rated their dataset understanding four or five out of five.Annotations with moderate confidence had lower agreement on attribute values.Source: §4.3 · p. 9
  • Croissant attributes averaged 0.55 BLEU versus 0.41 for Croissant-RAI attributes.The paper attributes the gap partly to free-form RAI responses and the greater effort needed to find RAI information in publications.Source: §4.3 · p. 9

5 Limitations and Future Work

Croissant's structure may be difficult for users unfamiliar with the format, and its editor remains in an early stage. Planned work includes documentation, tool enhancements, and support for complex data structures.

5 Limitations and Future Work

  • Croissant's structure may challenge users unfamiliar with the format, potentially limiting broader adoption.The authors identify comprehensive documentation and extended tools as ways to make the format easier to use.Source: §5 · pp. 9–10
  • Future Croissant versions may formalize some free-text attributes.Source: §5 · p. 9
  • The Croissant editor is still early-stage; planned additions include file archiving, nested fields, dataset examples, and community guidelines.The guidelines are intended to account for domain-specific needs.Source: §5 · p. 10
  • A planned Geo-Croissant extension will demonstrate handling complex data structures and formats such as HDF5 and Zarr.Source: §5 · p. 10

6 Conclusion

Croissant is a standardized metadata format intended to improve ML dataset discoverability, portability, and interoperability. Its adoption and extensibility support continued use across repositories, tools, and communities.

6 Conclusion

  • Croissant provides a standardized representation intended to make ML datasets more discoverable, portable, and interoperable.It addresses data-management challenges across repositories, tools, frameworks, and platforms.Source: §6 · p. 10
  • Croissant has been adopted by popular dataset repositories and frameworks, and human raters judged its metadata readable, understandable, complete, and concise.The conclusion notes that future success depends on further adoption, dataset availability, and support from ML tools and frameworks.Source: §6 · p. 10
  • Croissant’s extensibility and broad data coverage allow communities to adapt it to their needs, as illustrated by the Croissant-RAI extension.The paper says this can streamline collaboration between ML and other fields.Source: §6 · p. 10

Checklist

The checklist describes documentation and accessibility materials expected for dataset submissions, alongside reproducibility expectations for benchmarks.

Checklist

  • Dataset submissions should provide documentation and intended uses, a reviewer-accessible dataset and Croissant metadata record, responsibility and licensing statements, and a hosting and maintenance plan.The checklist also names documentation frameworks such as datasheets, dataset nutrition labels, and data statements.Source: Checklist · p. 15
  • Supplementary materials should link to datasets and metadata, explain how to read the data, describe long-term preservation, and specify an explicit license.Structured metadata using Web standards and persistent identifiers are also recommended.Source: Checklist · p. 16
  • Benchmark results should be reproducible through accessible datasets, code, and evaluation procedures; these supplementary-material requirements do not apply to papers introducing dataset-creation best practices.The checklist marks several experimental-reporting items as not applicable for this paper.Source: Checklist · pp. 15–16

7 Appendix

The appendix illustrates how Croissant files and libraries can be used to load datasets, prepare ML workflows, and visualize COCO bounding boxes.

7.1 Code Examples

  • The code examples demonstrate workflows using the mlcroissant and TensorFlow Datasets libraries.They cover loading a Croissant file, using JSON-LD in TFDS, loading a Hugging Face dataset into a TFDS workflow, and visualizing COCO bounding boxes.Source: §7.1 · p. 17

7.1.1 Loading a Dataset from a Croissant File

  • The Croissant-file example creates a dataset with mlcroissant, prints metadata, and iterates over its records.The snippet points to a remote Croissant metadata file.Source: §7.1.1 · p. 17

7.1.2 Loading data from a Croissant JSON-LD file in an ML workflow by using TFDS

  • The TFDS JSON-LD example uses a Croissant metadata URL to prepare data and print a dataset example.The snippet imports TensorFlow Datasets and calls download_and_prepare.Source: §7.1.2 · p. 17

7.1.3 Using Croissant into ML-workflow by loading into TFDS Data loader for HF Datasets

  • The Hugging Face workflow example inspects Croissant metadata, creates a TFDS CroissantBuilder, prepares data, and includes a training/testing split step.The code uses a Hugging Face Croissant endpoint.Source: §7.1.3 · p. 17

7.1.4 Visualizing Bounding Boxes in Croissant using the COCO 2014 dataset

  • The COCO example creates a Croissant JSON-LD file for a bounding-box record set and retrieves its first record for visualization.The snippet uses image identifiers and bounding-box values to locate an image and draw a rectangle; it notes that COCO uses XYWH while PIL uses XYXY.Source: §7.1.4 · pp. 18–19

7.2 Croissant Health Metrics

Croissant Health crawls repository-hosted Croissant files and computes metrics from crawl responses and metadata structures. Its reported coverage and file validity differ across Hugging Face and OpenML, and OpenML datasets have more complex structures.

7.2 Croissant Health Metrics

  • Croissant Health automatically scrapes repository-hosted Croissant files and computes metrics from crawl responses and counts of FileObjects, FileSets, RecordSets, and Fields.It is implemented for Hugging Face Datasets and OpenML, with more detailed statistics planned for the future.Source: §7.2 · p. 19

7.2.1 Croissant Statistics for Hugging Face Datasets

  • Over 100k Hugging Face Croissant datasets were successfully downloaded, while 25% of Croissant files were invalid.The statistics are intended to help identify and fix Croissant-generation issues.Source: §7.2.1 · p. 19, Figure 9 · p. 20
  • Hugging Face datasets averaged fewer than 10 resources, RecordSets, and Fields.Figure 10 illustrates the mean and standard deviation statistics.Source: §7.2.1 · p. 19, Figure 10 · p. 20

7.2.2 Croissant Stastistics for OpenML Datasets

  • OpenML had about 4k datasets, with around 25% of Croissant files invalid due to authentication issues accessing private datasets.Figure 11 shows adoption, and Figure 12 illustrates dataset statistics.Source: Figure 11 · p. 20, Figure 12 · p. 20, §7.2.2 · p. 20
  • OpenML Croissant datasets were more structurally complex than Hugging Face datasets, with many containing more FileObjects, RecordSets, and Fields.Croissant Health’s checks currently cover only Hugging Face and OpenML datasets.Source: §7.2.2 · p. 20

7.3 User Study

Participants generally reported high confidence in their Croissant annotations, while their understanding of the datasets varied more; most took 15–30 minutes to create a description.

7.3 User Study

7.4 Semantic search with Croissant

Croissant’s unified dataset descriptions can be scraped and embedded to support semantic search; the paper illustrates a pipeline that explores dataset proximity in an embedding space.

7.4 Semantic search with Croissant

  • Croissant descriptions can be scraped and projected for semantic search, with an example visualizer for exploring dataset proximity in the embedding space.Source: §7.4 · p. 24
  • The example pipeline reads more than 5,000 OpenML Croissant descriptions, extracts descriptions and URLs, and embeds them with a sentence-transformer encoder.Source: §7.4 · p. 24
  • The pipeline projects embeddings into three dimensions using PCA and t-SNE.Source: §7.4 · p. 24

7.5 Croissant Editor for Dataset Authors

The open-source Croissant editor helps dataset publishers generate metadata through a familiar interface, inferring dataset structure and guiding users through metadata fields.

7.5 Croissant Editor for Dataset Authors

  • The open-source editor generates Croissant metadata for dataset publishers through a familiar interface that abstracts the format’s syntax.Source: §7.5 · p. 24
  • Users can start creating a Croissant dataset by dragging and dropping files.Source: §7.5 · p. 24
  • The editor infers resources and structure definitions from the data, then guides users through required and optional fields.Source: §7.5 · p. 24
  • The editor runs locally or through the Hugging Face interface and incorporates Croissant Core and Croissant RAI attributes.Source: §7.5 · p. 24, Figure 21 · p. 26
Loading 2403.19546v3…