UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

1 Introduction

• Despite pretrained language models’ broad empirical strengths, their silly and nonsensical mistakes have renewed interest in commonsense knowledge and reasoning.The introduction also notes that commonsense benchmarks and models are nascent, without an organized effort to administer diverse benchmarks and investigate transfer across them.Source: §1 · pp. 1–2
• The authors propose RAINBOW, a multitask benchmark for commonsense models to generalize across tasks and datasets, and cost equivalent curves to compare transfer choices’ performance and data efficiency.The curves ask how much target-dataset training data transfer saves relative to a non-transfer baseline; lower curves indicate more successful transfer.Source: Figure 1 · p. 1, §1 · p. 2
• Cost equivalent curves target data efficiency because fine-tuning state-of-the-art models requires substantial training data, while collecting new task or domain datasets can cost tens of thousands of dollars.The goal is for models to generalize systematically across datasets rather than rely solely on target-dataset data.Source: §1 · p. 2
• Experiments find that intermediate-task transfer can yield better or equivalent performance with a particular recipe; QA-based commonsense datasets transfer well, knowledge graphs do not, and larger models benefit more.These findings come from reevaluating intermediate-task transfer with the proposed evaluation tools.Source: §1 · p. 2
• UNICORN establishes new state-of-the-art performance across eight commonsense benchmarks.Results are αNLI (87.3%), COSMOSQA (91.8%), HELLASWAG (93.9%), PIQA (90.1%), SOCIALIQA (83.2%), WINOGRANDE (86.6%), CYCIC (94.0%), and COMMONSENSEQA (79.3%). Ablations show improved data efficiency at all training dataset sizes.Source: §1 · p. 2

2 Cost Equivalent Curves

• Cost equivalent curves define cost as target-dataset training examples and show how many examples a transfer approach needs to match the single-task baseline’s performance.The comparison measures data efficiency as the amount of target data varies.Source: §2 · p. 2
• For the single-task baseline’s performance on ∼10k examples, RAINBOW multitasking requires about 5k examples, while GLUE multitasking requires more than 10k.Curves below the y = x diagonal indicate improvement over the baseline.Source: §2 · p. 2
• The curve construction assumes performance varies continuously and strictly monotonically with cost, then uses isotonic regression to estimate learning curves and match performance levels (Kaplan et al. 2020).The procedure maps each baseline dataset size to its performance and finds the number of examples the new method needs to match it.Source: §2 · p. 2
• Cost equivalent curves reframe evaluation from maximizing performance on a fixed-size benchmark to efficiently reaching a given performance, and can use costs such as GPU compute.This work focuses on data efficiency.Source: §2 · p. 2

• RAINBOW includes only multiple-choice question-answering datasets to keep evaluation clean-cut.Source: §3 · p. 2
• COSMOSQA tests commonsense reading comprehension of everyday narratives, while HELLASWAG asks models to choose the most plausible ending to a short context.Source: §3 · p. 2
• PIQA evaluates physical commonsense reasoning, while SOCIALIQA evaluates commonsense reasoning about social situations and interactions.Source: §3 · p. 2
• WINOGRANDE is a large-scale collection of Winograd schema-inspired problems requiring reasoning about social and physical interactions.Source: §3 · p. 3

4.1 What’s the Best Approach for Transfer?

• Sequential training generally matches or beats other transfer recipes and is often slightly more data-efficient than multitask fine-tuning.Both approaches often use fewer examples than multitask training or the single-task baseline to reach the same performance; they perform similarly on αNLI and SocialIQA, while improving data efficiency on the other tasks.Source: §4.1 · pp. 3–4
• Sequential and multitask fine-tuning outperform the single-task baseline across all RAINBOW tasks and dataset sizes, while sequential training rarely harms performance elsewhere.The finding holds for full datasets as well as smaller training sets, and mostly extends to other source and target tasks.Source: §4.1 · p. 4
• Multitask training initially needs fewer examples than the single-task baseline but can require more when data is plentiful, including on HellaSWAG and WinoGrande.It hurts full-dataset performance on CosmosQA, HellaSWAG, and WinoGrande; WinoGrande accuracy drops from 77.0% to 72.1%.Source: §4.1 · p. 4

4.2 What Transfers Best for Common Sense?

• RAINBOW tasks transfer better to one another than GLUE or SUPERGLUE do across all six RAINBOW tasks and training-set sizes.The result also holds for CommonsenseQA with multitask training, but RAINBOW may not help or may slightly reduce data efficiency when multitasking with JOCI (Zhang et al. 2017).Source: Figure 3 · p. 4, Table 2 · p. 4, §4.2 · p. 5
• With sequential training and T5-BASE or larger, RAINBOW improves data efficiency and performance for every task considered, unlike the other multisets.No multiset uniformly improves performance with multitask training; T5-SMALL does not always benefit, while GLUE and SUPERGLUE often have little effect or slightly decrease data efficiency.Source: §4.2 · p. 5
• T5’s multitask pretraining included GLUE and SUPERGLUE, so the experiments cannot determine whether RAINBOW’s gains reflect novelty to T5 or more general information.The pretrained weights mix many tasks with a language-modeling objective (Raffel et al. 2019).Source: §4.2 · p. 5

4.3 Does Model Size Affect Transfer?

• Larger models appear to gain more from RAINBOW transfer to CommonsenseQA, saving more training examples than their respective baselines.The comparison holds gradient updates fixed across model sizes; whether the trend generalizes to other contexts remains open.Source: §4.3 · p. 5, Figure 4 · p. 5
• Across all three tested model sizes, sequential training appears at least as effective as the other transfer approaches.This extends the finding that sequential training generally matches or beats other recipes.Source: Figure 4 · p. 5, §4.3 · p. 6

• UNICORN’s primary goal is to provide a pretrained commonsense reasoning model ready for fine-tuning on downstream commonsense tasks.The authors frame this as analogous to multitasking off-the-shelf T5 models on NLP benchmarks during pretraining.Source: §5 · p. 7
• For its performance-limit evaluation, the authors multitasked T5-11B on RAINBOW, then trained on each task individually, handling WINOGRANDE separately.WINOGRANDE followed multitasking on the other five RAINBOW datasets before training on WINOGRANDE; checkpoints were selected by dev accuracy.Source: §5 · p. 7
• UNICORN establishes new SOTA on αNLI (87.3%), COSMOSQA (91.8%), HELLASWAG (93.9%), PIQA (90.1%), SOCIALIQA (83.2%), and WINOGRANDE (86.6%).Source: §5 · p. 7
• UNICORN also achieves SOTA on CYCIC (94.0%) and COMMONSENSEQA (79.3%), showing strong performance beyond RAINBOW.Source: §5 · p. 7

Scaling Laws

• Scaling studies find gains from more data, parameters, and training reliable and predictable (Hestness et al. 2017; Sun et al. 2017; Rosenfeld et al. 2020; Kaplan et al. 2020), motivating cost-benefit evaluation.Sun et al. (2017) found that accuracy improvements require exponentially more data.Source: §6 · p. 7

Commonsense Benchmarks

• Commonsense benchmarks use adversarial filtering to remove easily predicted examples (Zellers et al. 2018; Le Bras et al. 2020), a technique used by all RAINBOW tasks except COSMOSQA.Other commonsense benchmarks beyond those explored include Roemmele, Bejan, and Gordon (2011), Levesque, Davis, and Morgenstern (2011), and Mostafazadeh et al. (2016).Source: §6 · p. 7

Transfer Learning

• Transfer learning progressed from unsupervised word representations (Mikolov et al. 2013) to contextual language models and task fine-tuning, with T5 framing tasks as text-to-text (Raffel et al. 2019).Radford et al. (2018) demonstrated fine-tuning language models on tasks encoded as text; Devlin et al. (2019), Yang et al. (2019), Liu et al. (2019b), and Lan et al. (2019) improved on the technique.Source: §6 · p. 7

Data Efficiency & Evaluation

• Clark and Matwin (1993) counted saved training examples, whereas this work plots example counts achieving equivalent performance; Dodge et al. (2019) examined compute-performance trade-offs from hyperparameter tuning.Schwartz et al. (2019) also noted the importance of cost-benefit trade-offs in evaluation.Source: §6 · p. 7

7 Conclusion

• UNICORN improves state-of-the-art results across eight benchmarks, building on the paper’s RAINBOW commonsense dataset collection and empirical study.The paper presents UNICORN as a model trained using lessons from its large-scale study of transfer on pretrained language models.Source: §7 · p. 7

A Cost Equivalent Curves

• Cost equivalent curves compare how an innovation changes a cost-benefit trade-off, extending beyond dataset-size and performance comparisons.The paper introduces them to facilitate comparisons across contexts that traditional learning curves make difficult to compare.Source: §7 · p. 7, A Cost Equivalent Curves · p. 11
• The framework treats the baseline as the control, the innovation as the treatment, the outcome measure as benefit, and resources spent as cost.Examples include comparing sequential with single-task training by data usage at a target accuracy, or optimization methods by hyperparameter evaluations at a target perplexity.Source: A Cost Equivalent Curves · p. 11
• Curve construction assumes cost and benefit have a continuous, strictly monotonic relationship and uses isotonic regression to fit control and treatment cost-benefit curves.The paper notes that expected cross-entropy versus parameters, data, and compute empirically satisfies this assumption in machine learning.Source: A Cost Equivalent Curves · p. 11
• The cost equivalent curve maps each control cost to the treatment cost yielding the same benefit, by composing the control curve with the inverse treatment curve.The inverse exists under the assumed strict monotonicity.Source: A Cost Equivalent Curves · p. 11
• The implementation estimates the inverse curve by swapping regression inputs and outputs, using scikit-learn’s isotonic regression implementation.The paper provides code in the RAINBOW repository.Source: A Cost Equivalent Curves · p. 11

B Datasets

• The study investigates transferring commonsense from dataset collections to end tasks; the appendix describes the tasks, multisets, and knowledge graphs used.B.1 covers evaluated end tasks, B.2 expands on RAINBOW and other multisets, and B.3 details the knowledge graphs.Source: B Datasets · p. 11

B.1 Tasks

• The six RAINBOW tasks test explanation selection, passage-based commonsense comprehension, plausible story endings, physical problem-solving, social-emotional reasoning, and commonsense coreference.αNLI selects a story’s most plausible middle; COSMOSQA asks about inferences from a passage; HELLASWAG uses adversarially selected distractor endings; PIQA asks for solutions to physical goals; SOCIALIQA tests social and emotional commonsense; WINOGRANDE uses sentence blanks requiring physical or social commonsense (Bhagavatula et al. 2020; Huang et al. 2019; Zellers et al. 2019; Bisk et al. 2020; Sap et al. 2019b; Sakaguchi et al. 2020).Source: §B.1 · p. 11
• The additional end tasks are COMMONSENSEQA, CYCIC, and JOCI, testing fine-grained world knowledge, commonsense and topic knowledge, and graded likely inferences, respectively.JOCI is treated as five-way classification and evaluated with accuracy to align it with the other end tasks (Talmor et al. 2019; Zhang et al. 2017).Source: §B.1 · p. 12

B.2 Multisets

• For transfer, the study uses two additional multisets alongside RAINBOW; GLUE (Wang et al. 2019b) measures natural language understanding by evaluating models on a suite of classification tasks.Source: §B.2 · p. 12
• RAINBOW combines six multiple-choice benchmarks spanning event sequences, physical instrumental knowledge, and social common sense: αNLI, COSMOSQA, HELLASWAG, PIQA, SOCIALIQA, and WINOGRANDE.The six datasets also serve individually as end tasks in the experiments (Bhagavatula et al. 2020; Huang et al. 2019; Zellers et al. 2019; Bisk et al. 2020; Sap et al. 2019b; Sakaguchi et al. 2020).Source: §B.1 · p. 11, §B.2 · p. 12

B.3 Knowledge Graphs

• CONCEPTNET combines curated and crowdsourced concepts connected by canonical relations; its commonsense subset focuses especially on objects and things (Speer, Chin, and Havasi 2017).The passage gives “water” ATLOCATION “river” as an example relation triple.Source: §B.3 · p. 12
• ATOMIC represents event relationships and commonsense inferences, including preconditions, postconditions, and plausible consequences (Sap et al. 2019a).One example links a person making another person’s coffee to the recipient feeling grateful.Source: §B.3 · p. 12

C Training and Evaluation

• The appendix provides technical details of the training and evaluation setup to help reproduce the experiments.Source: C Training and Evaluation · p. 12

C.1 Model and Implementation

• T5 is an encoder-decoder text-to-text model pretrained with masked language modeling and multitask training, trained by teacher forcing and greedily decoded at test time.The available sizes were small (60M parameters), base (220M), large (770M), 3B, and 11B; the experiments used the original T5 implementation, extended for preprocessing, data reading, and task mixing.Source: §C.1 · pp. 12–13

C.2 Preprocessing

• Preprocessing prepended a dataset-identifying string, wrapped each feature in uniquely tagged XML-like brackets, and joined the features with newlines.Figure 6 illustrates the format with a WINOGRANDE example.Source: §C.2 · p. 13, Figure 6 · p. 13

C.3 Training and Hyper-parameter Tuning

• UNICORN’s leaderboard and test-set evaluations used T5-11B, first multitasked on RAINBOW for 25,000 updates, then fine-tuned on each end task for 25,000 updates with dev-accuracy early stopping.The grid searched learning rates 4e-3, 2e-3, 1e-3, and 5e-4, and batch sizes 16 and 32.Source: §C.3 · p. 13
• Investigatory experiments generally started from T5-LARGE and trained for 50,000 updates, using batch size 16 and reporting best dev scores to avoid test leakage.Full-data multitask experiments explored equal and dataset-size-weighted mixing; knowledge-graph transfer also explored forward, backward, and bidirectional prediction, while learning curves used equal mixing and forward prediction. Two-stage transfer reused first-stage hyperparameters, and all tasks used accuracy.Source: §C.3 · p. 13

C.4 Hardware, Software, and Compute

• The compute setup used two Google Cloud VMs and TPUs, with v3-8 for T5-LARGE or smaller and v2-256 or v3-256 for T5-11B.The 11B runs used model parallelism 16; training usually took several hours, and runs were parallelized with GNU Parallel.Source: §C.4 · pp. 13–14

D Leaderboards

• UNICORN achieves state-of-the-art performance across a number of popular commonsense benchmarks, with the appendix comparing its results against previous leaders and other baselines.The supplied passages do not give the values from the leaderboard tables, so no additional benchmark scores are reported here.Source: D Leaderboards · p. 14
• The appendix includes leaderboard submissions for αNLI, COSMOSQA, and HELLASWAG.Source: Table 4 · p. 14, Table 5 · p. 14, Table 6 · p. 14
• Leaderboard submissions are also tabulated for PIQA, SOCIALIQA, and WINOGRANDE, whose table caption defines AUC as area under the dataset-size versus accuracy learning curve.Source: Table 7 · p. 14, Table 8 · p. 14, Table 9 · p. 14
• The appendix includes leaderboard submission tables for CYCIC and COMMONSENSEQA.Source: Table 10 · p. 14, Table 11 · p. 14
• 79.5% accuracy is reported for the ConceptNet-using ALBERT model, while UNICORN is state-of-the-art among COMMONSENSEQA models that do not use CONCEPTNET.The leaderboard separates systems by ConceptNet use because using the graph can provide an advantage by eliminating human-authored distractor options; the cited ALBERT model combines the knowledge graph with ALBERT (Lan et al. 2019)11.Source: D Leaderboards · pp. 14–15
• Submission hyperparameters vary by benchmark, with learning rates from 5e-4 to 2e-3 and batch sizes of 16 or 32.WINOGRANDE settings additionally vary by size: xs and s use 2e-3 with batch size 16, m uses 5e-4 with batch size 32, l uses 1e-3 with batch size 16, and xl uses 2e-3 with batch size 16.Source: D Leaderboards · p. 15

E Experiments

• The experiments appendix provides additional figures and experiment tables, and makes the code, models, CSV experimental results, and further figures available through the RAINBOW repository.Source: E Experiments · p. 15

E.1 Transferring to the RAINBOW Tasks

• The figures and tables use RAINBOW for the end tasks.Source: §E.1 · p. 15

E.2 Transferring to Other Tasks

• The other-task experiments target COMMONSENSEQA and JOCI.Source: §E.2 · p. 15

E.3 Effect of Size

• The model-size experiments use COMMONSENSEQA as the transfer target.Source: §E.3 · p. 15

E.4 Transferring Knowledge Graphs

• Additional RAINBOW transfer comparisons cover multisets, GLUE, SUPERGLUE, and held-out RAINBOW tasks using T5-LARGE.The captions specify dev accuracy for these comparisons without showing which transfer source performs best.Source: Figure 7 · p. 16, Figure 8 · p. 16, Figure 9 · p. 17

E.2 Transferring to Other Tasks

• Transfer to COMMONSENSEQA and JOCI is examined through multitask training, with full-data baselines and learning curves reported for both targets.The provided captions identify the comparisons and their scope but do not give performance values.Source: Table 22 · p. 21, Table 21 · p. 21, Table 23 · p. 21

E.3 Effect of Size

• Transfer to COMMONSENSEQA is compared across T5-SMALL, T5-BASE, and T5-LARGE, with full-task performance and learning curves reported by model size and transfer approach.The figures and tables specify evaluation scope, not comparative outcomes.Source: Figure 14 · p. 22, Table 25 · p. 22, Table 26 · p. 22

E.4 Transferring Knowledge Graphs

• Knowledge-graph transfer to RAINBOW is evaluated with ATOMIC, CONCEPTNET, and their combination, both with and without joint RAINBOW multitasking.The supplied captions also identify full-data performance and learning-curve tables, but provide no results.Source: Figure 15 · p. 23, Figure 17 · p. 24, Figure 16 · p. 24