PaperlayerView exact revision on arXiv

Source-linked AI summary

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, Jiawei Zhou

arXiv:2102.01672v3cs.CLcs.AIcs.LG

TL;DR

NLG research lacks a multilingual, consistently evaluated testbed and relies on divergent datasets and flawed automated metrics. GEM addresses this gap with a living benchmark spanning varied tasks, languages, datasets, challenging test sets, and evaluation strategies, providing a repeatable environment for clearer progress tracking.

  • Problem

    NLG evaluation relies on divergent, mostly English datasets and automated metrics that can poorly reflect human judgments or reward undesirable outputs.

  • Method

    GEM constructs a living benchmark spanning eleven datasets, 18 languages, varied generation challenges, challenging test sets, and NLG-specific data cards.

  • Results

    GEM provides a repeatable, consistent, and more interpretable environment for evaluating systems across many datasets and tracking progress toward NLG goals.

  • Takeaways & Limitations

    GEM offers a shared testbed for evaluating NLG systems and metrics across multilingual tasks while supporting future benchmark expansion.

  • Takeaways & Limitations

    GEM currently focuses on deterministic input-to-output tasks, leaving personalized and controlled NLG evaluation as a challenge.

Abstract

from arXiv · show

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.

1 Introduction

GEM is a living benchmark designed to evaluate diverse NLG challenges through human and automatic evaluation while adapting as datasets, metrics, and models improve. Its initial release spans varied datasets, communicative goals, languages, and challenging test sets to expose model shortcomings beyond standard leaderboard progress.

  • Motivation: ROUGE can reward fluent but meaningless or unfaithful summaries and tends to favor longer outputs because it measures lexical overlap with references.The benchmark argues that NLG progress must be assessed against multiple goals simultaneously.
  • Contribution: GEM enables in-depth evaluation of model outputs across human and automatic methods to uncover shortcomings and opportunities for progress.The benchmark is intended to avoid encouraging leaderboard hill climbing and to evolve as datasets, metrics, and models improve.
  • Benchmark design: The initial GEM release includes eleven datasets covering generation challenges such as content selection, planning, surface realization, paraphrasing, and simplification.The datasets also vary in communicative goals, languages, data noisiness, and resource availability.
  • Benchmark design: 18 languages are represented across GEM’s tasks, including two datasets without English, while dataset sizes range from 5k to 500k data points.More than half of the datasets were post-processed to improve data quality.
  • Benchmark design: GEM introduces ten types of challenging test sets that probe specific modeling aspects and help assess performance robustly against modeling shortcuts.These test sets complement the benchmark’s diverse tasks and datasets.

2 Benchmarks in NLG

The section argues that NLG benchmarks improve comparability but can mischaracterize progress when they reduce evaluation to a single metric, discourage dataset scrutiny, or neglect multilingualism and reproducible human-evaluation standards. GEM addresses these limitations as a living, multilingual NLG benchmark and testbed for richer, adaptable evaluation.

  • NLG benchmarks enable unified evaluation and fairer model comparisons, making them critical for measuring modeling progress.
  • Single-number leaderboards and single-metric evaluations can encourage blind optimization and mischaracterize system performance.This may obscure considerations such as model size and fairness.
  • A living benchmark can adjust datasets and evaluation metrics as limitations emerge, making it more powerful and long-lived.The section contrasts this flexibility with static evaluation and highlights human-in-the-loop data expansion as an example.
  • Increasing multilingualism of NLG research: GEM promotes multilingual NLG by including non-English datasets and sustaining evaluation of datasets created for multilingual shared tasks.The motivation includes differences in morphology and word order that make cross-language transfer and modeling more complex.
  • Providing a testbed for automated evaluation: GEM provides a multi-task testbed for evaluating automated metrics across tasks, setups, and languages, supported by released system outputs and human annotations.It also permits additional metrics to be integrated into the living benchmark system for faster adoption.
  • Developing reproducible human evaluation standards: NLG human-evaluation methodologies differ drastically, with inconsistent or unspecified definitions of criteria such as “accuracy” and “fluency,” motivating reproducible reporting standards.

3 Dataset Selection

GEM selects datasets through a proposal, criteria, and voting process that maximizes utility under resource constraints. The resulting benchmark emphasizes diverse, multilingual, clean, and resource-varied datasets, with strong preferences for Summarization, Data-to-text, and Dialog.

  • Selection Methodology: GEM’s selection methodology elicited dataset proposals, established hard and soft criteria, and used member voting to maximize utility under constrained resources.Hard criteria determine inclusion or exclusion, while soft criteria inform the utility of remaining tasks.
  • Selection Methodology: The survey received 28 responses, favoring a median of 10 tasks or average of 12, with about one-third featuring non-English language.The criteria survey also collected ratings, hard-criterion judgments, soft-criterion preferences, and open-text feedback.
  • Selected Criteria: Selection principles prioritize diverse high-level tasks, clean data, mixed resource levels, interesting test sets, and multi-reference datasets, without requiring established evaluation strategies.The benchmark focuses on multiple datasets per high-level task and avoids conflating model mistakes with learned noise.
  • High-Level Tasks: Summarization, Data-to-text, and Dialog received strong preference, while image-input tasks, Question Answering, and creative generation tasks were excluded because they complicate or undermine evaluation.Creative tasks such as story and poetry generation were considered especially affected by inadequate evaluation approaches.
  • Specific Datasets: The final 11-dataset selection spans 18 languages and dataset sizes from 5,000 examples to 1.5M, with two datasets excluding English entirely.The highest-ranked datasets were E2E with 0.577, XSum with 0.538, and ToTTo with 0.461; non-English datasets ranked lower overall.

4 Experimental Setup

GEM releases validation-set baseline results because its test sets and final metrics were not yet available. The setup evaluates diverse model architectures with an expanding, multi-faceted metric suite designed to reduce metric overfitting and characterize outputs and systems.

  • Experimental setup: GEM reports validation-set baseline results using publicly available data and metrics because the test sets and final metric selection were not yet released.These baselines are intended to indicate model performance while supporting correct training and evaluation.
  • Experimental setup: The setup applies architectures varying in size, complexity, and training because some GEM tasks may not benefit from language-model encoders.The pretrain-then-finetune paradigm has generally improved NLP results, but some GEM inputs are not natural language.
  • Metric evaluation: Metrics are drawn from a constantly expanding list, while hidden test metrics help avoid overfitting and will expand the incomplete baseline results.The metric set is computed through the GEM shared-task framework.
  • Metric evaluation: Reference-based evaluation includes lexical metrics BLEU, ROUGE-1/2/L, and METEOR, alongside semantic metrics BERTScore and BLEURT.BERTScore baselines use RoBERTa-large and mBERT, while BLEURT uses English-only BLEURT-base-128.
  • Metric evaluation: The suite also probes faithfulness with QA-based approaches, uses NUBIA for decomposable entailment and similarity scores, and measures diversity and system characteristics.System characterization includes parameter counts, output vocabulary size |V|, and mean output length.

5 Results

GEM’s Results section avoids reducing complex interactions among data, metrics, and model outputs to a single score or traditional leaderboard. Instead, it presents an interactive exploration system and shared-task evaluation framework for nuanced, transparent analysis.

  • Results: GEM does not distill interactions among data, metrics, and model outputs into a single number or traditional leaderboard.The section instead describes an interactive result exploration system.
  • Results: The parallel coordinates plot displays results across many metrics and makes patterns spanning multiple metrics, systems, or tasks easier to spot than tables.Lines intersect parallel axes at corresponding mapped values, and filters can be applied along axes.
  • Results: For T5 on ToTTo, diversity metrics score higher than other systems while reference-based metrics score lower.Because ToTTo has a single baseline, the comparison remains unclear in the supplied passage.
  • Results: The completed system will provide model cards, related metadata, and exemplary outputs to improve transparency and inform users considering particular systems.These features are intended to support analysis of quantitative and individual outputs.
  • Results: The shared-task evaluation part will release all system outputs and metrics so participants can run their own evaluations and conduct analyses.This complements the interactive, self-directed result exploration.

6 Submitting to the benchmark

GEM submissions may cover a subset of tasks because computation resources can be limited, but all predictions must be formatted in a single file. After submission, GEM runs automated evaluations, collects human annotations, and develops reproducible human-evaluation strategies alongside model-card documentation.

  • Submission requirements: Submissions need not include predictions for every test and challenge set, but all model predictions must be formatted in one file.Submitters are encouraged to cover as many tasks as possible while accounting for computation-resource restrictions.
  • Submission requirements: Every submitter must answer questions for a model card covering social-impact concerns and information needed to replicate experiments.Model information may remain anonymous during required anonymization periods.
  • Evaluation: After submission, GEM runs its evaluation suite on submitted outputs and additionally collects human annotations.
  • Human Evaluation GEM: Human Evaluation GEM develops reproducible and consistent strategies by defining measurable text quantities, annotation schemes, rater guidelines, and annotation infrastructure.
  • Human Evaluation GEM: The human-evaluation setups target summarization, dialogue, simplification, and data-to-text, following established taxonomies, reporting strategies, and an evaluation datasheet.
  • Human Evaluation GEM: Shared-task participants must provide gold annotations on system outputs to assess the consistency of crowdsourced annotations.

7 Next Steps

GEM’s next steps focus on expanding inclusiveness, addressing personalized NLG, and sustaining the benchmark through regular updates and community participation.

  • Inclusiveness: GEM aims to collect datasets that better represent languages and cultures beyond its initial American and British English focus.The benchmark is intended as a starting point for broader, more inclusive data collection.
  • Personalized NLG: GEM currently targets deterministic input-to-output transformations, leaving personalized NLG systems challenging to enable and evaluate.Relevant directions include dialect, formality, writing-style transfer, and personalization around individual traits.
  • Personalized NLG: The benchmark will incorporate user-focused NLG through developments spanning datasets and evaluation.This direction extends GEM beyond its current task focus.
  • Living benchmark: GEM commits to regular updates that incorporate submitted model outputs, analyses, and metrics into its online results.Dataset selection updates will also consider input from the wider NLG research community through a yearly selection process.

8 Conclusion

The paper introduces GEM as a living NLG benchmark focused on evaluation. GEM provides a testbed for principled, repeatable, consistent, and more interpretable experiments across many datasets while elevating evaluation approaches.

  • GEM is introduced as a living natural language generation benchmark focused on evaluation.
  • GEM provides an environment in which NLG systems can be tested in a principled manner.
  • GEM enables experiments across many datasets with repeatable, consistent, and more interpretable evaluation.

9 Contribution Statements

GEM is organized as a large, decentralized effort with task-specific subgroups whose members contributed to coordination, dataset preparation, evaluation, infrastructure, and related activities. The section records these contributions across the benchmark’s steering, task, evaluation, technical, and community groups.

  • Organization: GEM uses a decentralized organization divided into task-specific subgroups to acknowledge contributors across the effort.The contribution statements list responsibilities for all groups.
  • Steering Committee: The steering committee provides feedback and direction, while Sebastian Gehrmann coordinates and leads GEM and the ACL 2021 workshop.The committee also acts as conference organizers for the workshop.
  • Task Groups: Task groups prepared dataset statements, data cards, loaders, baselines, code, and preprocessing for summarization, dialog, data-to-text, table-to-text, and simplification.Examples include MLSum, XSum, Wiki-Auto, DialoGPT, Schema-Guided Dialog, E2E NLG, Czech Restaurants, WebNLG, CommonGen, DART, and ToTTo.
  • Evaluation: Automated evaluation contributors integrated BLEU, Meteor, ROUGE, BLEURT, BERTScore, SARI, and referenceless and source-based metrics, alongside reference-set preparation and code refactoring.These contributions covered the GEM evaluation framework and reference sets for several datasets.
  • Supporting Activities: Additional groups established human-evaluation criteria, built the website and visualization system, developed model-fine-tuning infrastructure, designed data and model statements, and planned challenge and crowdsourced datasets.The authors also participated in discussions, surveys, feedback, and paper writing.

A Task Suggestion Categories · 1. Dataset Name · 2. Reference

GEM dataset suggestions required participants to provide information in specified categories.

  • A Task Suggestion Categories: Participants had to provide information in the categories required for suggesting a dataset for GEM.

B Considered datasets … 30. WikiBio (Lebret et al., 2016)

GEM proposed a broad, evolving collection of datasets for NLG research, spanning multilingual generation, dialogue, summarization, question generation, data-to-text, compression, and creative writing. The considered resources include established benchmarks alongside newer and specialized datasets.

  • B Considered datasets: Overall, GEM’s considered-dataset list was intended to support broad evaluation across diverse NLG tasks rather than a single dataset or application.The proposal explicitly introduces the resources as datasets considered for inclusion in GEM.
  • B Considered datasets; 11. ELI5; 30. WikiBio: Other proposed resources address image-to-text, visual question generation, and narrative or creative generation, including Alex Context NLG, Visual Question Generation, ELI5, and Writing Prompts.The list includes context-based generation, visual question generation, long-form explanation, and prompted writing.
  • B Considered datasets; 7. CoQA; 12. Hindi Poetry 14; 5. Chinese Poetry; 13. LCSTS: The proposal includes multilingual and domain-specific resources such as Bangla image-to-text, Czech restaurant data, Hindi poetry, Chinese poetry, and LCSTS.These datasets represent image captioning, task-oriented generation, poetry, and short-text summarization.
  • 2. AmbigQA/AmbigNQ; 7. CoQA; 18. NarrativeQA: The considered benchmarks cover question answering and question generation, including AmbigQA/AmbigNQ, CoQA, SQuAD question generation, and NarrativeQA.The list spans ambiguity resolution, conversational question answering, question generation, and narrative comprehension.
  • 15. MLSUM; 30. WikiBio; 28. WebNLG: The benchmark considers summarization and compression resources such as MLSUM, XSum, WikiSum, MSR Abstractive Text Compression, PubMed, and Arxiv.These datasets cover multilingual, extreme, encyclopedic, scientific, and abstractive compression-oriented summarization.
  • 19. PersonaChat; 17. MultiWOZ 2.2; 7. CoQA: Dialogue-oriented candidates include PersonaChat, Schema-Guided Dialogue, Ubuntu Dialogue Generation, MultiWOZ 2.2, and CoQA.The proposal spans persona-based, schema-guided, technical-support, multi-domain, and conversational question-answering settings.

C Task and Criteria Selection Survey

The GEM selection survey asked members to rate task inclusion and decide which task, dataset, language, resource, diversity, and evaluation criteria should guide benchmark composition.

  • C Task and Criteria Selection Survey: Members rated each suggested task’s inclusion in GEM on a 5-point Likert scale from strongly against to strongly in favor.The scale ranged from 1 to 5.
  • C Task and Criteria Selection Survey: The survey asked whether to exclude tasks associated with shared tasks held in 2021, since 2020, or ever.Each exclusion criterion used a yes/no response.
  • C Task and Criteria Selection Survey: It also assessed excluding paid-license, non-downloadable, or non-text datasets and tasks.Examples of non-text inputs included images or graphs.
  • C Task and Criteria Selection Survey: Members evaluated desired benchmark size, non-English coverage, task diversity versus focus, noisy versus clean data, and low- versus high-resource data.Task count ranged from 2 to 20, while non-English coverage ranged from 10 to 100%.
  • C Task and Criteria Selection Survey: The survey rated preferences for challenge-oriented or non-iid test sets, multiple references, multiple datasets per NLG task, unclear evaluation strategies, and reliable automatic metrics.These preferences used 5-point Likert scales, except the task-level composition questions that used the specified 10-point or 5-point ranges.
Loading 2102.01672v3…