Source-linked AI summary
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
TL;DR
NLG evaluation depends on datasets and metrics that can evolve, but static benchmarks can preserve outdated choices and overlook qualities not captured by leaderboard metrics. GEMv2 provides modular infrastructure for documenting, loading, and evaluating datasets; its initial release supports 40 datasets in 51 languages and provides online evaluation.
Problem
Static benchmarks can impede adoption of new datasets and metrics, while leaderboard metrics may miss NLG qualities that require evolving automatic and human evaluation.
Source: §1 · p. 1Method
Results
The initial release supports 40 datasets in 51 languages, with 23 datasets improved through configurations, filtering, or re-splitting and 17 having challenge sets.
Source: §6 · p. 7Takeaways & Limitations
GEMv2 offers a consistent workflow across supported datasets and metrics, and makes submitted model outputs available for download by evaluation researchers.
Source: §6 · p. 7Some supported languages lack tested NLG metrics, tokenization may be inappropriate, and combining any dataset with any metric may produce flawed evaluations.
Source: C Limitations · p. 15
Abstract
from arXiv · showhide
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
1 Introduction
Static benchmarks can preserve outdated NLG evaluation choices even as datasets, metrics, and human evaluation improve. GEMv2 addresses this with modular infrastructure, improved dataset documentation, and online evaluation, expanding support to 40 tasks and 51 languages.
1 Introduction
- Static benchmarks can block adoption of new datasets and metrics, reinforce outdated evaluation designs, and miss properties not captured by leaderboard metrics.These limitations are especially pertinent to NLG, where quality relies on evolving automatic and human evaluation approaches rather than accuracy alone.Source: §1 · p. 1
- GEMv1’s living benchmark let contributors propose datasets and metrics while model developers chose among tasks and evaluation suites.Those suites tested robustness and fairness aspects of models.Source: §1 · p. 1
- GEMv2 data cards organize dataset information into a header, a Quick-Use section, and expandable detailed sections.The Quick-Use section summarizes languages, PII, and licensing information.Source: Figure 1 · p. 2
- GEMv1’s centralized data management, single metric framework, and limited documentation guidance hindered scaling and adoption.The single framework also created dependency issues and challenges for participants with limited compute resources.Source: §1 · p. 2
- GEMv2 expands support from 13 to 40 tasks and from 18 to 51 languages, and adds online evaluation that collects model outputs and computes metrics for all datasets.Its data-card collection and rendering tools are intended to make documentation easier for creators and readers.Source: §1 · p. 2
2 Features and Functionality
GEMv2 provides a modular workflow for documenting, selecting, loading, and evaluating datasets, with an extensible metric framework and online access to model outputs for further analysis.
2 Features and Functionality
- GEMv2’s modular infrastructure supports dataset and metric additions that remain compatible with the other features, without requiring model developers to change their existing setup.Data cards can be completed through an interactive form with field-level instructions and loaded for updates.Source: §2 · p. 2
- Dataset loaders use a common code path, while standardized inputs support evaluation across more than 100 data splits.Datasets expose string targets and reference lists; unique GEM IDs let the evaluation library unshuffle outputs.Source: §2 · p. 3
- Metrics implement a compute() function, while the library handles caching, parallelism, and tokenization; metrics may specify Docker environments to avoid dependency conflicts.Source: §2 · p. 3
- Online evaluation submissions join a growing corpus of model outputs that researchers can use to develop metrics or conduct analyses.Submissions also appear in the result exploration tool.Source: §2 · p. 3
3 Dataset Selection and Loading
GEMv2 selected datasets through community submissions subject to consent, open permissive licensing, and text-to-text requirements. The release spans varied tasks, languages, configurations, and challenge sets, with some language diversity concentrated in a few datasets.
3 Dataset Selection and Loading
- GEMv2 selected 27 new tasks alongside 13 carried over from GEMv1, requiring author consent, permissive open licensing, and a text-to-text formulation.The community solicitation yielded 40 submissions; dataset quality was not an additional selection criterion.Source: §3 · p. 3
- Data-to-text and summarization are the most common task types, and summarization datasets tend to be larger; median input length is generally under 500 tokens, with summarization and some sports data-to-text tasks as exceptions.Four dataset configurations are hidden by Figure 2’s axis truncation.Source: Figure 2 · p. 4, §3 · p. 4
- GEMv2 supports languages across resource classes, but English appears in far more datasets and most language diversity comes from XLSum, WikiLingua, and the surface realization shared task ’20.Excluding those three, 13 datasets support non-English languages, 9 exclusively.Source: §3 · p. 4
- 40 datasets are supported; 14 have multiple configurations and 17 include challenge splits.WikiLingua was redesigned to support splits from and to any of its 18 languages, enabling crosslingual evaluations; the weather challenge split selects examples with complex discourse relations.Source: §3 · p. 4
4 Data Cards
GEMv2’s data cards document dataset creation, use, risks, and maintenance, while an interactive tool supports creating and navigating cards. Their structured format also allows analysis of dataset practices, including PII reporting, licensing, and sourcing.
4 Data Cards
- The card template addresses accountability, evaluation utility, completeness, curation impact, PII risks, and comparison with datasets serving similar goals.The revised template follows telescope, periscope, and microscope question groupings based on expected answer length.Source: §4 · p. 5
- An interactive tool supports creating or updating cards, shows completion progress, and conditionally displays questions based on earlier answers.Card answers are converted into structured JSON, then rendered as Markdown for loaders and as a web viewer.Source: §4 · p. 5
- The three-column viewer organizes card information from high-level summaries on the left to additional detail on the right.The columns correspond to telescope, periscope, and microscope questions.Source: §4 · p. 5
- Twenty cards report PII as unlikely or definitely absent, while 10 report it as likely or definitely present.Among explanations for absent PII, seven cite restrictions on data format or domain.Source: §4 · p. 5
- Most datasets use Creative Commons variants (22); four use MIT and three use Apache 2.0, while eight restrict use to non-commercial cases.The licensing distribution may be skewed by selecting only publicly available datasets.Source: §4 · p. 6
- Dataset sourcing is split almost evenly among automatic, crowdworker, and expert creation, with crowdworker-created datasets slightly more common.Amazon Mechanical Turk was the most frequently named crowdworking platform; the authors note that expert hiring through such platforms may confound the categories.Source: §4 · p. 6
5 System Design
GEMv2 hosts dataset, submission, evaluation, and results resources on the Hugging Face Hub. Its online workflow runs submitted model outputs against dataset references, computes NLG metrics, and publishes results for visualization.
5 System Design
- The system uses Spaces for submission, result downloads, and performance visualization, while dataset repositories host datasets, submissions, evaluations, and results.AutoTrain runs evaluation jobs using Hugging Face Benchmarks.Source: §5 · p. 6
- GEM-metrics computes common metrics such as BLEU and ROUGE as well as model-based metrics such as BLEURT through Docker integration.The metrics run within the evaluation workflow using GEM-metrics.Source: §5 · p. 6
- Submitted outputs are stored in a Hub repository, trigger an AutoTrain evaluation job, and are evaluated against dataset reference splits before results are published to the Hub.The published metrics supply visualizations on the GEM website and Space.Source: §5 · p. 6
6 Conclusion
GEMv2 unifies generation-research infrastructure with a workflow for documenting, choosing, loading, and evaluating mutually compatible datasets and metrics. Its initial release supports 40 datasets in 51 languages and includes dataset improvements, challenge sets, and an online submission tool.
6 Conclusion
- 40 datasets in 51 languages are supported in GEMv2’s initial release.The release also includes 23 datasets improved through configuration, filtering, and re-splitting, and 17 datasets with challenge sets.Source: §6 · p. 7
- GEMv2 proposes a consistent workflow for documenting and choosing datasets, then loading and evaluating them while keeping supported datasets and metrics compatible.The conclusion frames this as unified infrastructure for generation research.Source: §6 · p. 7
- A submission tool computes metrics and makes model outputs available for evaluation researchers to download.Researchers interested in integrating datasets are invited to contact the authors for support.Source: §6 · p. 7
A Dataset Overviews
Appendix A provides an overview of supported datasets, with input and output lengths reported using the mT5 tokenizer.
A Dataset Overviews
- Table 3 provides a detailed overview of the supported datasets.When a dataset has multiple configurations, the reported sizes and lengths are their medians.Source: A Dataset Overviews · p. 12
- Input and output lengths are reported in tokens according to the mT5 tokenizer.Source: A Dataset Overviews · p. 12
- For datasets with multiple configurations, the overview reports median sizes and lengths.Source: A Dataset Overviews · p. 12
B Changes to Datasets
This section describes dataset-specific changes to BiSECT and FairytaleQA, including challenge-set construction and task-oriented data preparation.
B.1 BiSECT
- BiSECT retains its original splits and adds a 359-pair HSPLIT-Wiki challenge set and a 1,798-pair Type-3 challenge set.The second set keeps Type-3 pairs after filtering pairs with significant length differences.Source: §B.1 · p. 12, §1 · p. 13
1. Direct Insertion: when a long sentence l con-
- The BiSECT classification distinguishes direct insertion, changes near the split, and changes across sentences; only Type 3 enters the second challenge set.The second set is filtered to remove pairs with significant length differences.Source: §B.1 · p. 12, §1 · p. 13
B.2 FairytaleQA
- FairytaleQA is organized for question generation, using expert-labeled answers and related story sections as input and the corresponding question as target.For this task, GEMv2 uses only the first of the two answers in the evaluation and test splits.Source: §B.2 · p. 13
B.3 MLB Data to Text
These dataset updates adapt serialization, splits, and task configurations across several GEMv2 datasets, while also adding challenge data and training-set controls.
B.3 MLB Data to Text
- MLB Data to Text serializes home-team, visiting-team, and player records, followed by innings in chronological order.The format follows Puduppully and Lapata (2021).Source: §B.3 · p. 13
B.4 Opusparcus
- Opusparcus provides regular evaluation sets containing paraphrases and full sets retaining all successfully annotated pairs, including rejected paraphrases.The validation and test sets are each available in both versions.Source: §B.4 · p. 13
- Its quality parameter selects nested training subsets with estimated true-paraphrase proportions from 60% to 100%; higher quality yields smaller, cleaner subsets.Training-set paraphrase status is unknown because those sets were not manually annotated.Source: §B.4 · p. 13
B.5 ROTOWIRE_English-German
- ROTOWIRE_English-German linearizes home-team records before visiting-team records, then lists players by team and descending points, with relative team rank included.The serialization follows a format similar to Saleh et al. (2019) and adds relative ranks following Puduppully et al. (2019b).Source: §B.5 · p. 13
B.6 SciDuet
- SciDuet combines paper and slide information into complete input-output instances and adds a challenge set excluding slides whose titles match paper section headers.Inputs include paper title, abstract, section headers and content, plus slide title; outputs contain slide text.Source: §B.6 · p. 14
B.7 SIMPITIKI
- SIMPITIKI receives independent train, validation, and test splits for both XML versions, plus challenge sets with transformations unseen during training and source-based splits.Version 2 has better sentence boundaries, and the two versions have different splits.Source: §B.7 · p. 14
B.8 SportSett Basketball
- SportSett linearizes game date and venue, team line-scores and next-game information, then home- and visiting-team player box-scores.Source: §B.8 · p. 14
B.9 squad_v2
- SQuAD2.0's new test split is created from its train split and added to the squad_v2 dataset.The original SQuAD2.0 release had training and development splits only.Source: §B.9 · p. 14
B.10 Taskmaster-3
This section describes the contents and task-specific configurations of Taskmaster-3, Turku Hockey, Turku Paraphrase, and WikiLingua in GEMv2.
B.10 Taskmaster-3
- Taskmaster-3 contains 23,789 movie-ticketing dialogs created by one crowdsourced worker writing both customer and agent turns.Customers decide on ticket details or opt out of the transaction.Source: §B.10 · p. 14
B.11 Turku Hockey
- Turku Hockey offers event-level generation from separately linearized game events alongside game-level generation from an entire game's structured data.Game-level generation produces descriptions for all relevant events.Source: §B.11 · p. 14
B.12 Turku Paraphrase
- Turku Paraphrase offers plain, classification, and generation configurations; generation filters unsuitable examples and orders directional pairs from more detailed input to more general output.Classification also reverses pair order, flipping directional labels when needed.Source: §B.12 · p. 14
B.13 WikiLingua
- WikiLingua aligns document-summary pairs across its 17 non-English languages through English, enabling crosslingual loading for any language pair.The release also includes randomly sampled 3,000-instance evaluation subsets and removes thumbnail payloads and summaries longer than 60% of the input document.Source: §B.13 · p. 15
C Limitations
GEMv2 limits dataset curation decisions, identifies unresolved issues in multilingual metrics and human evaluation, and documents contributor roles.
C Limitations
- GEMv2 limits dataset inclusion decisions to licensing and consent, leaving usefulness judgments to the research community.This shifts the importance of documenting dataset issues and alternatives to data cards.Source: C Limitations · p. 15
- Combining every dataset with every metric may produce flawed evaluations, while some supported languages lack tested metrics or appropriate tokenization schemes.Some datasets' specific metrics are not yet supported.Source: C Limitations · p. 15
- The overview omits detailed discussion of human evaluation; it does not scale, and each task requires its own approach.The authors say they are developing better human evaluation for a subset of tasks and will release instructions separately.Source: C Limitations · p. 15
D Contribution Statements
- The contribution statements credit community members for dataset suggestions, discussion, proofreading, infrastructure, loaders, data cards, metrics, and documentation.The statements also identify specific contributors to the submission infrastructure and card-rendering tools.Source: D Contribution Statements · pp. 15–16