Source-linked AI summary
HEPData: a repository for high energy physics data
Eamonn Maguire, Lukas Heinrich, Graeme Watt
TL;DR
HEPData addresses the need for an open repository of experimental particle-physics data underlying published papers. The paper describes a complete software rewrite using modern technologies and a customized Invenio v3 overlay. The resulting site replaces the old platform, migrates all records, supports external submissions, and provides a FAIR-oriented web platform.
Problem
HEPData provides open access to scattering data underlying several thousand experimental particle-physics publications.
Method
The software was rebuilt predominantly in Python and JavaScript as a customized overlay on Invenio v3, with YAML-based migration and a role-based submission workflow.
Results
The transition to the new CERN-hosted site was effectively completed, all data records were migrated, and the new submission system had supported external submissions since January 2017.
Takeaways & Limitations
The new HEPData site provides a web platform for making particle-physics data Findable, Accessible, Interoperable, and Reusable under FAIR principles.
Takeaways & Limitations
Use beyond experimental particle-physics papers may require further development for low-energy data and publications absent from Inspire HEP.
Abstract
from arXiv · showhide
The Durham High Energy Physics Database (HEPData) has been built up over the past four decades as a unique open-access repository for scattering data from experimental particle physics papers. It comprises data points underlying several thousand publications. Over the last two years, the HEPData software has been completely rewritten using modern computing technologies as an overlay on the Invenio v3 digital library framework. The software is open source with the new site available at https://hepdata.net now replacing the previous site at http://hepdata.cedar.ac.uk. In this write-up, we describe the development of the new site and explain some of the advantages it offers over the previous platform.
1. Introduction
HEPData is a long-established open-access repository for scattering data from experimental particle-physics papers. A complete rewrite replaced its legacy platform with a modern, open-source site hosted at CERN.
- HEPData contains data related to several thousand publications, including measurements from the Large Hadron Collider.
- The repository provides Level 1 data, complementing the CERN Open Data Portal’s focus on Levels 2 and 3.
- The new site was completely rewritten over two years, hosted on CERN OpenStack, and offers advantages and new features over the old site.
- The HEPData code is open source and available through a dedicated GitHub organisation.
2. Migration
The migration replaced database-centered storage and an ad hoc input process with human-readable YAML text files and validation-oriented tooling.
- The new system stores records as text files because only metadata, rather than data points, needs to be searchable.
- HEPData introduced a YAML submission format instead of retaining the old ad hoc input format.YAML is described as a human-readable superset of JSON.
- Existing records were exported from the old database into YAML for migration to the new system.
3. Software
The rewritten HEPData software uses Python and JavaScript on the Invenio v3 framework, supporting standardized data conversion and interactive visualization.
- The software was rewritten predominantly in Python and JavaScript as a highly customized overlay on Invenio v3.
- Standardized storage enables automatic conversion of data points into multiple formats and their visualization.
- Custom JavaScript using D3.js renders tables, scatter plots, and heatmaps with controls for switching plot elements on and off.
4. Discoverability
The new HEPData platform improves discovery through metadata search, semantic enrichment, versioning, and visual interfaces for records and queries.
- PostgreSQL and Elasticsearch provide fast, powerful searches across HEPData metadata fields.
- Users can perform keyword and faceted searches, while the site provides screenshots of typical records and searches.
- HEPData content is semantically enriched with Schema.org vocabulary to improve how search engines understand it.
- Records are versioned similarly to arXiv papers, allowing corrections to mistakes in original submissions.
5. Conversion
The new HEPData site provides JSON access to searches, records, and tables, while converting tables into YAML, CSV, ROOT, or YODA formats through an extensible converter.
- JSON equivalents expose search results, record displays, and data tables for programmatic access via scripts or applications.The JSON representation is requested by adding format=json to the URL.
- Data tables use YAML natively and can be converted to CSV, ROOT, and YODA formats.CSV supports widely used applications; ROOT and YODA outputs use analysis-specific objects selected according to table structure.
- ROOT conversion writes TGraphAsymmErrors for numeric data and uses TH1F, TH2F, or TH3F objects according to bin widths and independent-variable count.
- HEPData DOIs are embedded in exported formats, preserving the origin of downloaded data points.A web-service package runs the converter in a Docker container with dependencies such as ROOT and YODA.
- Predictable URLs support CSV, JSON, ROOT, YAML, and YODA downloads, including all tables or the latest version when parameters are omitted.For whole-submission JSON, light=True omits the data tables.
6. Submission
The new submission workflow assigns distinct coordination, uploading, and review responsibilities, supports sandbox testing, and allows records to be prepared before the corresponding paper has an Inspire record.
- Roles and workflow: The workflow defines Coordinator, Uploader, and Reviewer roles for initiating, preparing, and approving submissions.The Coordinator may also act as Reviewer, and current submissions are visible to these roles through their Dashboard.
- User tools: Users can watch records for email notifications about future revisions and ask submission participants questions through each record.
- Roles and workflow: A Coordinator assigns an Uploader and Reviewer, after which the Uploader uploads the submission and the review process proceeds.
- Integration and timing: Records can be initiated and reviewed without an Inspire record number, allowing preparation alongside the paper and DOI citation in its first arXiv version.Paper metadata is pulled directly from Inspire, and the DOI is minted when the record is finalised.
- Submission format: Submissions use submission.yaml metadata plus YAML or JSON data files for each table, uploaded together with optional auxiliary files in an archive.
- Sandbox: Sandbox records let any logged-in user test uploads and share persistent, non-searchable records without special permissions.Creators can remove Sandbox records when they are no longer required.
7. Future plans
Future development targets broader input and dataset types, including ROOT-based numerical extraction, HistFactory configurations, and simplified likelihoods for reinterpretation packages.
- HEPData could store numerical theoretical predictions without changes to its software or submission workflow, while low-energy data may require further development.Additional work may be needed for low-energy keywords and records whose publications are absent from Inspire HEP.
- A planned mixed YAML/ROOT input format would keep metadata in YAML while extracting numerical values from ROOT objects into standard YAML.
- The project is expanding native support beyond simple tables to richer datasets such as HistFactory configurations and simplified likelihoods.Lossless archival of likelihood data could support various reinterpretation packages.
8. Summary
The HEPData software was completely rewritten and migrated all records to a CERN-hosted new site; the platform is presented as supporting FAIR data practices for particle physicists.
- The software was rewritten over two years in Python and JavaScript as a highly customised overlay on Invenio v3.
- All records were migrated from the old site to the new HEPData site hosted at CERN on OpenStack and managed remotely from Durham.
- The new submission system had successfully handled external data submissions from January 2017 onward.
- The new site provides a web platform for making particle-physics data Findable, Accessible, Interoperable, and Reusable under the FAIR principles.