PaperlayerView exact revision on arXiv

Source-linked AI summary

UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi

arXiv:2103.13009v1cs.CL

TL;DR

Commonsense research lacks organized evidence about generalization and transfer across diverse tasks. The paper introduces RAINBOW and cost equivalent curves, uses extensive transfer experiments to derive cross-task findings, and presents UNICORN’s state-of-the-art results across eight benchmarks.

  • Problem

    Commonsense benchmarks and models have lacked an organized effort to evaluate transfer and generalization across diverse tasks and datasets.

  • Method

    The paper introduces the RAINBOW multitask benchmark, cost equivalent curves for data-efficiency comparisons, extensive transfer experiments, and the UNICORN model.

  • Results

    UNICORN achieves state-of-the-art performance across eight commonsense benchmarks, while experiments find QA-based commonsense datasets transfer well and knowledge graphs do not.

  • Takeaways & Limitations

    Transfer can improve or match performance when following a particular recipe, and larger models benefit more from transfer than smaller ones.

  • Takeaways & Limitations

    Because T5 was pretrained on GLUE and SUPERGLUE, the experiments cannot determine whether RAINBOW’s gains reflect novelty or greater generality.

Abstract

from arXiv · show

Commonsense AI has long been seen as a near impossible goal -- until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose two new ways to evaluate commonsense models, emphasizing their generality on new tasks and building on diverse, recently introduced benchmarks. First, we propose a new multitask benchmark, RAINBOW, to promote research on commonsense models that generalize well over multiple tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how the choice of source datasets, pretrained language models, and transfer learning methods impacts performance and data efficiency. We perform extensive experiments -- over 200 experiments encompassing 4800 models -- and report multiple valuable and sometimes surprising findings, e.g., that transfer almost always leads to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well with each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit more from transfer than smaller ones. Last but not least, we introduce a new universal commonsense reasoning model, UNICORN, that establishes new state-of-the-art performance across 8 popular commonsense benchmarks, aNLI (87.3%), CosmosQA (91.8%), HellaSWAG (93.9%), PIQA (90.1%), SocialIQa (83.2%), WinoGrande (86.6%), CycIC (94.0%) and CommonsenseQA (79.3%).

1 Introduction

The paper addresses the lack of organized evaluation of commonsense transfer across diverse tasks by introducing RAINBOW and cost equivalent curves, then uses these tools to study transfer and build UNICORN.

  • 1 Introduction: The work responds to growing interest in commonsense AI while targeting pretrained models’ continuing commonsense errors and the absence of organized cross-benchmark transfer research.
  • 1 Introduction: RAINBOW is proposed as a multitask benchmark for evaluating commonsense models that generalize across multiple tasks and datasets.
  • 1 Introduction: Cost equivalent curves evaluate how source datasets, pretrained language models, and transfer methods affect target-task performance and data efficiency.
  • 1 Introduction: The study finds that transfer can improve or match performance with a particular recipe, QA datasets transfer well together, knowledge graphs do not, and larger models benefit more.
  • 1 Introduction: UNICORN achieves state-of-the-art performance on eight commonsense benchmarks, including αNLI (87.3%), HELLASWAG (93.9%), and CommonsenseQA (79.3%).The reported benchmarks also include COSMOSQA, PIQA, SOCIALIQA, WINOGRANDE, and CYCIC.

2 Cost Equivalent Curves

Cost equivalent curves compare the target-data requirements of single-task and transfer-based methods at matched performance, making data-efficiency trade-offs explicit.

  • 2 Cost Equivalent Curves: Cost equivalent curves measure how many target-dataset examples a transfer method needs to match the single-task baseline’s performance.
  • 2 Cost Equivalent Curves: The curves are constructed by mapping each baseline dataset size to its performance and finding the matching target-data requirement on the transfer method’s curve.
  • 2 Cost Equivalent Curves: The method assumes performance varies continuously and strictly monotonically with cost, enabling isotonic-regression estimates of learning curves.
  • 2 Cost Equivalent Curves: The evaluation reframes success from maximizing fixed-dataset performance to improving the cost-benefit trade-off, with cost potentially defined beyond data.

3 RAINBOW

RAINBOW is a suite of six multiple-choice commonsense benchmarks spanning narrative, reading-comprehension, physical, social, and coreference reasoning, evaluated with transfer comparisons.

  • 3 RAINBOW: The suite is restricted to multiple-choice question-answering datasets to keep evaluation clean-cut.
  • 3 RAINBOW: αNLI selects the most plausible middle sentence connecting a story’s beginning and ending, while COSMOSQA tests commonsense reading comprehension.
  • 3 RAINBOW: HELLASWAG selects plausible context completions, PIQA tests physical commonsense solutions, and SOCIALIQA evaluates reasoning about social situations.
  • 3 RAINBOW: WINOGRANDE consists of Winograd-schema-inspired problems requiring reasoning about social and physical interactions.

4 Empirical Insights

Across extensive transfer experiments, sequential training generally provides the strongest or most reliable data-efficiency gains, while transfer effectiveness depends on task family, data regime, and model size. RAINBOW transfers well among commonsense QA tasks, larger models benefit more, and knowledge-graph multitasking has little impact.

  • 4.1 What’s the Best Approach for Transfer?: Sequential training generally matches or beats other transfer approaches, using fewer examples to reach comparable performance than multitask training or the single-task baseline.This result holds across the six RAINBOW tasks, with sequential training often slightly more data-efficient than multitask fine-tuning.
  • 4.1 What’s the Best Approach for Transfer?: Sequential and multitask fine-tuning uniformly outperform the single-task baseline across RAINBOW tasks and dataset sizes, while sequential training rarely harms performance.Multitask learning can instead help mainly when data is scarce and hurt when data is plentiful; WinoGrande falls from 77.0% to 72.1% under multitask learning.
  • 4.2 What Transfers Best for Common Sense?: RAINBOW tasks transfer better to one another than GLUE or SUPERGLUE transfer to RAINBOW, and sequential training improves data efficiency and performance for every task with T5-BASE or larger.This uniform improvement does not hold for multitask training or the smallest T5-SMALL model.
  • 4.3 Does Model Size Affect Transfer?: Larger pretrained models benefit more from transfer, saving more baseline examples, while sequential training remains comparable to or better than alternative methods across model sizes.Cost equivalent curves make transfer effects comparable despite different baseline performance levels.
  • 4.3 Does Model Size Affect Transfer?: Knowledge-graph multitasking shows little impact: only SocialIQa benefits, possibly because ATOMIC contributed to its construction.The authors suggest that generative knowledge-graph completion and non-QA serialization may impede transfer to discriminative tasks.

5 UNICORN

UNICORN is a universal commonsense reasoning model pretrained through multitask learning and fine-tuned for downstream commonsense tasks. It achieves new state-of-the-art performance across all six RAINBOW datasets and two additional benchmarks.

  • Model: UNICORN is designed as a pretrained commonsense reasoning model for fine-tuning on downstream commonsense tasks.Its training goal is analogous to multitask pretraining on GLUE and SUPERGLUE.
  • Training: The model is first multitasked on RAINBOW and then trained separately for each downstream task, with special handling for WINOGRANDE.
  • SOTA on RAINBOW: UNICORN establishes new state-of-the-art results on all six RAINBOW datasets.Reported scores are αNLI (87.3%), COSMOSQA (91.8%), HELLASWAG (93.9%), PIQA (90.1%), SOCIALIQA (83.2%), and WINOGRANDE (86.6%).
  • SOTA beyond RAINBOW: UNICORN also achieves state-of-the-art results on CYCIC (94.0%) and COMMONSENSEQA (79.3%), extending its results beyond RAINBOW.

6 Related Work

Related work situates commonsense benchmarking within broader efforts on scalable modeling, benchmark construction, transfer learning, and cost-aware evaluation. It also notes adversarial filtering as a response to statistical shortcuts in commonsense datasets.

  • Scaling Laws: Modern scaling work reports predictable performance gains from increasing data, parameters, and training compute.
  • Commonsense Benchmarks: Commonsense benchmarking has expanded because neural models can exploit dataset artifacts instead of understanding the intended task.
  • Commonsense Benchmarks: Adversarial filtering removes easily predicted examples to reduce the impact of statistical biases and annotation artifacts in commonsense benchmarks.Besides COSMOSQA, all RAINBOW tasks use adversarial filtering.
  • Transfer Learning: Transfer learning evolved from unsupervised word representations to contextualized neural language-model representations that can be fine-tuned across varied tasks.
  • Data Efficiency & Evaluation: Prior evaluation research emphasizes cost-benefit trade-offs, including compute-performance effects from hyperparameter tuning and training examples saved by qualitative knowledge.

7 Conclusion

The paper concludes that cost-equivalent curves support data-efficiency comparisons, RAINBOW organizes commonsense transfer experiments, and UNICORN improves state-of-the-art results across eight benchmarks.

  • Conclusion: The study introduces RAINBOW, cost-equivalent curves, and UNICORN as resources for evaluating and improving general commonsense reasoning.The authors state that these resources are intended to support future work.
  • Cost Equivalent Curves: Cost-equivalent curves compare innovations through cost-benefit trade-offs, such as dataset size or monetary cost versus accuracy.The framework treats single-task training as a control and multitask or sequential training as a treatment.
  • Cost Equivalent Curves: The curves assume a continuous, strictly monotonic relationship between cost and benefit and estimate trade-offs with isotonic regression.
  • Cost Equivalent Curves: The cost-equivalent mapping composes the control cost-benefit curve with the inverse treatment curve to identify equal-benefit costs.
  • Implementation: The implementation estimates the inverse curve by switching regression inputs and outputs in scikit-learn’s isotonic regression.

B Datasets

The paper reviews diverse commonsense tasks, benchmark collections, and knowledge graphs used to study transfer. RAINBOW combines six multiple-choice datasets spanning event, physical, social, and linguistic commonsense.

  • Additional Datasets: The study also evaluates COMMONSENSEQA, JOCI, and CYCIC alongside the six RAINBOW datasets.CYCIC uses five-way questions involving commonsense reasoning and knowledge about arithmetic, logic, time, and locations.
  • Tasks: The evaluated tasks span abductive explanation, in-context commonsense inference, plausible text completion, physical problem solving, social-emotional reasoning, and coreference resolution.αNLI selects the more plausible middle of a three-sentence story, while the other tasks use their respective multiple-choice formats.
  • Transfer Sources: Transfer sources include RAINBOW, GLUE, SUPERGLUE, and commonsense knowledge graphs such as CONCEPTNET and ATOMIC.CONCEPTNET links concepts through canonical relations, whereas ATOMIC links events through preconditions, postconditions, and plausible inferences.
  • RAINBOW: RAINBOW combines αNLI, COSMOSQA, HELLASWAG, PIQA, SOCIALIQA, and WINOGRANDE to measure different aspects of commonsense reasoning.The suite covers likely event sequences, reading-between-the-lines inference, physical affordances, social reasoning, and coreference.

C Training and Evaluation

The experiments use T5 as a text-to-text model, with task-specific preprocessing and standard transfer-training procedures. Leaderboard evaluations use T5-11B, while most investigations use T5-LARGE and evaluate with accuracy.

  • C.1 Model and Implementation: T5 frames NLP tasks as mapping input text to output text with an encoder-decoder architecture.The model is pretrained using masked language modeling and multitask training.
  • Implementation and Reproducibility: The implementation extends the original T5 code only with dataset preprocessing, reading, and task mixing, and releases experiment results for reproducibility.Experiments were run on TPU-backed Google Cloud infrastructure.
  • C.2 Preprocessing: Each example is converted into text by adding a dataset prefix, tagging features, and joining them with new-line characters.Figure 6 illustrates this preprocessing for WINOGRANDE.
  • C.1 Model and Implementation: Leaderboard submissions use T5-11B, whereas other experiments generally use T5-LARGE unless model size is under study.All tasks are converted to text-to-text, trained with teacher forcing, and decoded greedily.
  • Leaderboard Submissions: For leaderboard evaluations, UNICORN is first multitasked on RAINBOW and then trained separately on each end task with checkpoint-based early stopping.Dev-set accuracy selects the best checkpoint for evaluation.
  • Investigatory Experiments: Investigatory experiments run for 50,000 updates with batch size 16 and grid-search learning rates of 4e-3, 1e-3, and 2.5e-4.Two-stage transfer methods reuse first-stage hyperparameters in the second stage, and accuracy is used for all tasks.

D Leaderboards

UNICORN’s leaderboard evaluation reports state-of-the-art performance across multiple commonsense benchmarks. The comparisons include prior systems, accuracy-based results, and a distinction between models that do and do not use CONCEPTNET for COMMONSENSEQA.

  • Leaderboard Results: UNICORN achieves state-of-the-art performance across a number of popular commonsense benchmarks.The appendix presents these results alongside previous state-of-the-art systems and other baselines.
  • Reported Benchmarks: The appendix includes leaderboard tables for αNLI, COSMOSQA, HELLASWAG, PIQA, SOCIALIQA, WINOGRANDE, CYCIC, and COMMONSENSEQA.It also reports hyperparameters for several submissions and provides code and downloadable experimental results.
  • Leaderboard Results: 90.1% is reported for UNICORN in a leaderboard comparison alongside BERT-LARGE, ROBERTA-LARGE, and UNIFIEDQA-3B.The supplied passage does not identify the benchmark associated with this comparison.
  • COMMONSENSEQA: COMMONSENSEQA leaderboard submissions are divided into models that use CONCEPTNET and models that do not.The distinction matters because CONCEPTNET-based systems can eliminate human-authored distractor options.
  • COMMONSENSEQA: UNICORN is reported as the state-of-the-art model among COMMONSENSEQA systems that do not use CONCEPTNET.The state-of-the-art CONCEPTNET-using model combines the knowledge graph with ALBERT and scores 79.5% accuracy.

E.1 Transferring to the RAINBOW Tasks

The appendix presents additional experiments using RAINBOW as the set of end tasks. These experiments compare transfer settings and report dev accuracy.

  • Experimental Scope: The additional figures and tables use RAINBOW for the end tasks.

E.2 Transferring to Other Tasks

The appendix examines transfer beyond the core RAINBOW-task experiments, including transfers to COMMONSENSEQA and JOCI, model-size effects, and knowledge-graph sources. The figures and tables compare transfer methods and report dev accuracy, but the supplied passages do not provide their numerical outcomes.

  • E.2 Transferring to Other Tasks: Experiments target COMMONSENSEQA and JOCI as transfer destinations.The appendix includes both full-data performance and learning-curve comparisons for these tasks.
  • Model Size: Model-size experiments compare transfer to COMMONSENSEQA using T5-SMALL, T5-BASE, and T5-LARGE.The associated tables include full-task performance and learning curves for different transfer approaches.
  • Transfer Sources and Methods: Transfer methods from GLUE, SUPERGLUE, and held-out RAINBOW tasks are compared for RAINBOW targets with T5-LARGE.The held-out setup uses the other five RAINBOW tasks for transfer.
  • Reported Analyses: The appendix reports learning curves and full-data performance for transfer experiments across the RAINBOW tasks.The tables separately cover ordinary transfer, transfers to other tasks, model-size experiments, and knowledge-graph transfer.
  • Knowledge-Graph Transfer: Knowledge-graph transfer experiments use CONCEPTNET, ATOMIC, or both as sources for RAINBOW tasks.The comparisons include multitask training with and without simultaneous multitasking on RAINBOW.
Loading 2103.13009v1…