PaperlayerView exact revision on arXiv

Source-linked AI summary

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, Yejin Choi

arXiv:2201.05955v5cs.CL

TL;DR

Large-scale crowdsourcing can produce NLP datasets dominated by repetitive patterns, motivating a more diverse creation process. WANLI combines dataset cartography, GPT-3 generation, automatic filtering, and human revision and labeling; its 107,885 examples improve performance across eight out-of-domain NLI test sets despite being 4 times smaller than MultiNLI.

  • Problem

    Crowdsourced NLP datasets often contain repetitive and spurious patterns because workers face difficulty creating diverse examples at scale, limiting linguistic coverage.

  • Method

    WANLI uses dataset cartography to identify challenging patterns, GPT-3 to generate similar examples, automatic filtering, and human labeling and revision.

  • Results

    WANLI’s 107,885 NLI examples improve performance on eight out-of-domain test sets when replacing MultiNLI supervision, despite being 4 times smaller.

  • Takeaways & Limitations

    The results support using generative models to aid dataset creation while shifting human work toward evaluating and revising generated content.

  • Takeaways & Limitations

    The pipeline is evaluated only on English natural language inference, with broader applications left to future work.

Abstract

from arXiv · show

A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluative strength of humans. Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns. Machine generated examples are then automatically filtered, and finally revised and labeled by human crowdworkers. The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets. Remarkably, training a model on WANLI improves performance on eight out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI, compared to training on the 4x larger MultiNLI. Moreover, it continues to be more effective than MultiNLI augmented with other NLI datasets. Our results demonstrate the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.

1 Introduction

WANLI addresses repetitive patterns in crowdsourced NLP datasets by combining language models’ generation with human evaluation and revision. Applied to NLI, the pipeline produces a smaller dataset that improves out-of-domain performance and reduces some documented spurious correlations.

  • Motivation: Crowdsourced NLP datasets often overrepresent repetitive and spurious patterns, limiting linguistic diversity and making models brittle outside standard benchmarks.Workers can reliably write correct examples, but producing diverse examples at scale is difficult, encouraging limited writing strategies.
  • Approach: WANLI combines language-model generation with human evaluation to create examples that preserve valuable linguistic patterns without requiring the model to understand the task itself.The approach uses models for efficient, diverse generation and humans to assign labels and optionally revise examples for quality.
  • Approach: The pipeline identifies challenging reasoning patterns with dataset cartography, prompts GPT-3 to generate similar examples, filters them automatically, and sends them to crowdworkers for labeling and revision.The filtering metric is inspired by data maps and estimates which generated examples are most ambiguous for a trained task model.
  • Results: WANLI contains 107,885 NLI examples created from MultiNLI and demonstrates effectiveness across out-of-domain evaluations.The authors apply the method to natural language inference, where models determine whether a premise entails a hypothesis.
  • Results: Replacing MultiNLI supervision with WANLI improves performance on eight out-of-domain test sets despite WANLI being 4 times smaller.The result includes test sets converted from downstream tasks such as question-answering and fact verification, and remains when MultiNLI is augmented with other datasets.
  • Results: WANLI has fewer previously documented spurious correlations than MultiNLI, although its GPT-3-generated content may contain distinct lexical correlations.The authors also report that the approach can improve performance on certain in-domain test sets.
  • Implications: The worker-AI framework is presented as more scalable than traditional crowdsourcing and potentially generalizable to classification tasks whose performance stagnates on popular benchmarks.The authors frame dataset curation as a setting where generative algorithms can assist human annotation.

2 Worker-AI Collaborative Dataset Creation for NLI

WANLI creates NLI data through a four-stage collaboration between task models, GPT-3, and human crowdworkers. The pipeline identifies ambiguous reasoning patterns, generates and filters similar examples, then uses human review to revise and label the retained data.

  • Overview: The four-stage pipeline collects challenging examples, generates pattern-matched examples with GPT-3, filters them automatically, and has crowdworkers revise and label the results.It begins with an existing dataset and ends with human-reviewed data.
  • Stage 1: Seed Selection: Dataset cartography identifies ambiguous examples, which vary in model confidence across training epochs and are treated as valuable for generalization.The method focuses on the most ambiguous examples rather than easy or consistently misclassified examples.
  • Stage 1: Seed Selection: For each seed, the method selects k = 4 same-label nearest neighbors using [CLS] representations to provide GPT-3 with examples sharing a reasoning pattern.The resulting representation-space similarity tends to correspond to human-interpretable similarity in the reasoning required.
  • Design Considerations: The approach automatically discovers augmentation targets without prior knowledge of challenging patterns, while revision must balance machine-example quality against reintroducing annotation artifacts.The authors note that exemplar collection could also use expert curation or category labels.
  • Stage 2: Overgeneration: GPT-3 receives grouped examples from a particular task slice and generates new examples in the same slice, rather than directly solving new unlabeled instances.The prompt separates premise and hypothesis using label-specific terms such as “Implication,” “Possibility,” and “Contradiction.”
  • Stage 3: Automatic Filtering: Estimated max variability filters generated examples by measuring the worst-case spread of model predictions across training checkpoints.The metric enables ambiguity estimation for unlabeled generated examples, under the assumption that each example would negligibly affect the checkpoints.

3 Training NLI Models with WANLI

WANLI-trained models generalize better than MultiNLI-trained models across the paper’s OOD NLI challenge sets, including when compared with larger or augmented training sets. Adding WANLI to ANLI also improves ANLI in-domain accuracy.

  • Out-of-domain evaluation: Models trained on WANLI outperform MultiNLI-trained models on every evaluated OOD test set.The evaluation uses RoBERTa-large and challenge sets constructed independently of MultiNLI and WANLI.
  • Out-of-domain evaluation: 11% on HANS, 9% on Adversarial NLI, and 4% on Diagnostics are the reported WANLI-over-MultiNLI improvements.WANLI is 4× smaller than MultiNLI and contains primarily machine-written examples.
  • Comparison with combined training sets: WANLI-trained models outperform baselines combining MultiNLI with other NLI datasets and augmentation sets in every OOD setting.This includes comparison with MNLI + SNLI + ANLI, which uses 9× more data.
  • Comparison with combined training sets: WANLI alone outperforms combining WANLI with MultiNLI, indicating that additional data is not necessarily beneficial when it predominantly contains easy-to-learn examples.The comparison concerns training-set composition rather than simply dataset size.
  • In-domain evaluation: Augmenting ANLI with WANLI improves ANLI in-domain test accuracy by 1.4% while also greatly aiding OOD performance.Table 4 evaluates whether including WANLI in ANLI training improves in-domain performance.

4 Artifacts in WANLI

WANLI reduces several known artifact signals relative to MultiNLI, although it retains distinct lexical correlations and remains imperfectly robust to partial-input shortcuts. Its premise–hypothesis similarity distributions also provide less label information than MultiNLI’s.

  • Lexical correlations: WANLI contains fewer previously known spurious correlations than MultiNLI but retains distinct lexical correlations that may reflect GPT-3 artifacts.The WANLI correlations generally involve different, less interpretable words than MultiNLI correlations, aside from “not” and “no”.
  • Partial-input models: The partial-input analysis assumes that a model seeing only one input should have no information about the correct NLI label.This assumption motivates testing whether individual premises or hypotheses carry label-correlated artifacts.
  • Partial-input models: 41.6% hypothesis-only accuracy on WANLI versus 49.6% on MultiNLI indicates weaker partial-input label signal under matched dataset size.A premise-only WANLI classifier achieves 42.9% accuracy; the MultiNLI premise-only comparison has a structural guarantee of no label information because each premise has all three labels.
  • Premise–hypothesis semantic similarity: In WANLI, neutral and contradiction examples have nearly indistinguishable premise–hypothesis similarity distributions, unlike the better-separated MultiNLI distributions.Entailed examples remain most similar to their premises in both datasets, but similarity provides less label signal in WANLI.

5 What does WANLI show about the human machine collaboration pipeline?

Human review reveals that WANLI revisions primarily improve fluency and relational clarity, while disagreement often reflects genuine ambiguity. GPT-3 also produces creative outputs beyond patterns that crowdworkers may readily create, but generation and labeling introduce a strong contradiction-label imbalance.

  • Revision patterns: 74% of both premise and hypothesis revisions change word count by only −1 to +2 words, with edits mainly targeting fluency or relational clarity.Fluency edits address redundancy and self-contradiction; clarity edits resolve ambiguities that obscure the entailment relationship.
  • Annotator disagreement: Annotator disagreements often represent genuinely ambiguous examples rather than careless mislabels.These cases provide a test bed for studying how ambiguity surfaces in classification tasks.
  • Generation and label distribution: Only 15% of human-labeled WANLI examples receive the contradiction label despite balanced generation seeds.Contradiction patterns are generally more difficult for GPT-3 to copy, and abstract reasoning patterns are harder to mimic than simpler transformations.
  • Generation and label distribution: Even when GPT-3 fails to copy the intended examples, diverse in-context examples yield creative outputs that may challenge human crowdworkers to produce.This highlights a complementary role for generation rather than requiring perfect pattern replication.

6 Related Work

Related work identifies persistent weaknesses in crowdsourced, adversarial, and fully automatic dataset creation. WANLI instead positions human–machine collaboration as a collaborative route to more robust labeled classification data.

  • Crowdsourcing: Crowdsourced datasets scale broadly but may fail to isolate the key linguistic phenomena required by their tasks.NLI crowdsourcing is especially associated with well-studied annotation artifacts, and protocol changes have produced negative results.
  • Crowdsourcing: This remains a hard problem requiring greater innovation in dataset collection.The related-work discussion frames artifact reduction and linguistic coverage as unresolved challenges.
  • Adversarial data collection: Adversarial collection increases annotator effort and may reduce diversity or fail to improve generalization on non-adversarial tests.Its outcomes can also depend strongly on the chosen adversaries, complicating fair evaluation.
  • Dataset generation: Fully automatic generation may limit examples to phenomena already accessible to the generating model, contrary to the goal of teaching more diverse phenomena.Related approaches also leave automatic identification of under-represented slices as an open issue when slices are specified by labels or metadata.
  • Human-machine collaboration: Human–machine collaboration extends prior human validation of generated text to labeled datasets for robust classification models.WANLI uses collaboration rather than adversarial interaction between machines and humans.

7 Conclusion

WANLI presents worker–AI collaboration as an alternative to free-form crowdsourcing for distilling human linguistic competence into learnable data. Its empirical utility supports asking workers to revise and evaluate generated content while developing stronger generative assistance.

  • Conclusion: The paper frames dataset creation as distilling human linguistic competence into data that models can learn from.Its proposed worker-and-AI collaborative approach changes how that competence is elicited at scale.
  • Conclusion: WANLI’s empirical utility suggests asking workers to revise and evaluate generated content rather than write all examples freely.The authors hope this encourages generative algorithms that aid dataset creation and re-imagine human annotation.

8 Ethics Statement

The paper addresses risks to people and data quality arising from language-model generation and crowdsourcing. It mitigates some risks through annotation procedures and worker protections, while acknowledging that harmful content may remain.

  • Risks of Generated Text: Generated text may perpetuate social harms and contain toxic language, including subtle biases that annotators could miss.Annotators were asked to discard offensive examples, but harmful examples may still enter the final dataset.
  • Worker Relations: The authors recognize the asymmetrical relationship between crowdsourcing requesters and workers.They report taking care to pay fair wages and respond to worker feedback and questions.
  • Privacy and Oversight: The study collects only Amazon Mechanical Turk worker IDs and will not release them.The annotation effort received an IRB exemption.

9 Limitations

The paper’s evidence is bounded by its single-task, single-language evaluation, unablated pipeline, and reliance on singly labeled examples. Additional analyses also expose possible generated-data artifacts and methodological costs.

  • Scope: The pipeline is evaluated only on English natural language inference, leaving broader applications for future work.Its generality across languages and tasks is therefore not established here.
  • Generated-Data Artifacts: Generated datasets may contain artifacts, including social biases that accuracy-based evaluation may not capture.The authors specifically investigate possible over-representation of Western entities in WANLI.
  • Ablations: The authors cannot isolate the effectiveness of individual pipeline components because sufficiently large ablations would be prohibitively expensive.Unexamined variations include exemplar-group collection methods and GPT-3 prompting templates.
  • Label Ambiguity: WANLI uses one label per example even when annotators disagree for valid reasons.The authors encourage future work on ambiguity and the limits of forcing such instances into n-way classification.
  • Evaluation Constraints: The variability correlation analysis uses a 99%/1% MNLI split rather than repeatedly retraining after holding out each example.This approximation yielded Pearson’s r = 0.527 with p-value 7 × 10^-281.

C.3 Dataset sizes at each stage

The construction pipeline expands a seed set through GPT-3 generation, filters the outputs, and then selects a balanced subset for human review. The reported intermediate sizes quantify substantial overgeneration and filtering.

  • Generation: Stage 1 selects 98,176 seed examples, and generating five examples per seed produces 372,404 examples in Dgen.Examples with improper premise–hypothesis formatting are skipped during generation.
  • Filtering: After initial filtering, the dataset contains 287,241 examples, including removals for identical pairs and copied in-context examples.The discarded examples include 79,278 identical premise–hypothesis pairs and 4,732 copied examples.
  • Selection: The pipeline retains the half with the highest estimated max variability and balances intended label classes, producing Dfiltered with 143,619 examples.Human review is not recruited for every example in Dfiltered.

D.3 Revision Analysis

Human revisions were generally limited in scope but often targeted coreference, while annotation disagreements frequently reflected genuine ambiguity. The resulting WANLI-trained model outperformed size-matched MultiNLI baselines across all reported test sets.

  • Revision Scope: 74% of both premise and hypothesis revisions changed word count by only −1 to +2 words.The authors characterize revisions as generally targeted yet effective.
  • Coreference: Pronoun changes occurred in 11.6% of premise revisions and 20.6% of hypothesis revisions, often clarifying coreference.These revisions changed the set of pronouns present in the text.
  • Label Disagreement: Authors agreed on labels only 49% of the time for a sample of examples on which two annotators disagreed.In 12% of cases, all three labels appeared among the four annotations, suggesting true ambiguity.
  • Label Disagreement: WANLI retains examples with disagreement because one of multiple reasonable labels can still provide valuable training signal.The dataset therefore preserves some inherently ambiguous examples despite using one training label.
  • Baseline Comparison: A WANLI-trained model outperforms size-matched MultiNLI baselines on every test set.The compared baselines include AFLite-filtered, most-ambiguous, and randomly downsampled MultiNLI subsets.

E.2 Evaluation on MultiNLI

WANLI preserves near-MultiNLI in-domain accuracy when mixed with MultiNLI, but WANLI alone performs worse on MultiNLI development while improving out-of-domain evaluation. These robustness gains also transfer to T5-base, which outperforms MultiNLI on every reported test set.

  • MultiNLI development evaluation: ∼1% in-domain accuracy change results when WANLI is mixed into MultiNLI training through swapping or augmentation.Training on WANLI alone lowers MultiNLI development performance by ∼10%.
  • MultiNLI development evaluation: Higher out-of-domain performance despite the ∼10% MultiNLI development drop suggests MultiNLI evaluation is not a definitive signal of model ability.The comparison concerns training on WANLI alone versus MultiNLI development evaluation.
  • Finetuning T5: 4% on NLI Diagnostics, 10% on HANS, and 8% on Adversarial NLI are the T5-base gains from training on WANLI rather than MultiNLI.T5-base outperforms MultiNLI on every test set reported in Table 11.
  • Finetuning T5: WANLI’s robustness improvements generalize to T5-base, an architecture never used in the data curation pipeline.This provides an architecture-transfer check beyond the model used during curation.
Loading 2201.05955v5…