PaperlayerView exact revision on arXiv

Source-linked AI summary

LMentry: A Language Model Benchmark of Elementary Language Tasks

Avia Efrat, Or Honovich, Omer Levy

arXiv:2211.02069v2cs.CLcs.AIcs.LG

TL;DR

Large, complex benchmarks can make it harder to inspect fundamental language capabilities and robustness, while their size can impede replication and analysis. LMentry evaluates 25 tasks designed to be trivial for humans, measuring accuracy and robustness in a quick, interpretable benchmark. Even its strongest tested model underperformed humans, and models showed brittleness to simple input changes.

  • Problem

    Complex benchmarks make fundamental capabilities and systematic robustness harder to inspect, and their size can put replication and analysis out of reach for many researchers.

    Source: §1 · p. 1

  • Method

    LMentry evaluates 25 human-trivial language tasks in zero-shot settings, measuring accuracy and robustness to simple input perturbations.

    Source: §1 · p. 1, §2 · pp. 2–3

  • Results

    TextDavinci002 scored 66.1% on LMentry versus 100% human performance, and even 175B-parameter models underperformed humans and were brittle to inconsequential input perturbations.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    LMentry offers quick, informative insights that complement complex benchmarks, whose size can make replication and analysis out of reach for many researchers.

    Source: §1 · p. 1

    TextDavinci002's instruction-finetuning details were not publicly available at the time of writing.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

As the performance of large language models rapidly improves, benchmarks are getting larger and more complex as well. We present LMentry, a benchmark that avoids this "arms race" by focusing on a compact set of tasks that are trivial to humans, e.g. writing a sentence containing a specific word, identifying which words in a list belong to a specific category, or choosing which of two words is longer. LMentry is specifically designed to provide quick and interpretable insights into the capabilities and robustness of large language models. Our experiments reveal a wide variety of failure cases that, while immediately obvious to humans, pose a considerable challenge for large language models, including OpenAI's latest 175B-parameter instruction-tuned model, TextDavinci002. LMentry complements contemporary evaluation approaches of large language models, providing a quick, automatic, and easy-to-run "unit test", without resorting to large benchmark suites of complex tasks.

1 Introduction

Complex benchmarks can make fundamental capabilities and robustness harder to inspect systematically, while their size puts replication and analysis out of reach for many researchers. LMentry tests basic language abilities and reveals substantial performance and robustness gaps, even in leading models.

1 Introduction

  • Perplexity offers no specific, interpretable insights into capabilities, while complex benchmarks make systematic robustness analysis and replication harder.The large size of many benchmarks makes replication and analysis out of reach for many researchers.Source: §1 · p. 1
  • LMentry comprises 25 tasks of basic language ability, such as writing a sentence containing a specified word or choosing the longer of two words.The benchmark measures accuracy and robustness to trivial input perturbations and provides an aggregate LMentry score.Source: §1 · p. 1
  • 175B TextDavinci002 scored 66.1% on LMentry, versus 100% human performance; the next-best InstructGPT 175B scored 48.4%.No other model scored higher than 41%; instruction finetuning improved accuracy, but neither it nor scaling appeared to increase robustness.Source: §1 · p. 1
  • LMentry exposes failure cases that challenge even the strongest contemporary language models despite being immediately obvious to humans.It is designed to provide quick, automatic insights that complement existing evaluation approaches.Source: §1 · p. 1

2 LMentry

LMentry is a zero-shot benchmark of 25 short tasks intended to be trivial for humans. It combines accuracy with robustness to simple input changes and uses automatic evaluation across diverse task formats.

2 LMentry

  • LMentry comprises 25 short tasks, including word-in-sentence, category identification, length comparison, and rhyme tasks.The benchmark assesses language-model performance in a zero-shot setting and measures both accuracy and robustness to simple input perturbations.Source: §2 · p. 2
  • Tasks follow guidelines requiring elementary-school solvability, quick automatic evaluation, short form, and isolated measurement of a language-understanding aspect.Authors review one another's tasks for adherence, discarding tasks without complete agreement before selecting a subset for interpretable analysis.Source: §2 · pp. 2–3
  • The LMentry score multiplies mean accuracy across tasks by mean robustness across four aspects, expressing accuracy penalized by brittleness.Each aspect averages over applicable tasks; task-level robustness is 100 minus the largest accuracy gap between cases.Source: §2 · p. 3
  • The four robustness aspects measure accuracy gaps across argument order, argument content, templates, and adjacent tasks.Not every aspect applies to every task; for example, a one-argument sentence-writing task cannot measure argument-order robustness.Source: §2 · p. 3, Figure 1 · p. 3
  • The benchmark contains 110,703 examples, with three templates and 3,000 examples per task; six tasks also have separate argument-content analyses.Arguments are consistent across templates, enabling template comparisons with the same arguments.Source: §2 · p. 3
  • An official evaluation suite scores outputs against approximately 10 regular-expression patterns per task, with Cohen’s kappa above 0.99 against manual ratings.The comparison used 100 sampled predictions per task from five strong baselines, totaling approximately 12,000 examples; a human study found no incorrect answers in 250 sampled examples.Source: §2 · pp. 3–4

3 Experimental Settings

The experiments evaluate 16 models from OpenAI decoder-only and T5-based encoder-decoder families. They use zero-shot inputs, greedy generation, and the LMentry score alongside separate accuracy and robustness results.

3 Experimental Settings

  • The study tests 16 models from OpenAI decoder-only and T5-based encoder-decoder families.The OpenAI models include GPT-3 and InstructGPT variants, while the T5-based set includes T0 and Tk variants.Source: §3.1 · p. 4

3.1 Baselines

  • The baselines include four GPT-3 and four InstructGPT sizes from 0.35B to 175B parameters, plus TextDavinci002 and T5-based models.T0 and Tk variants differ in finetuning datasets or the information and demonstrations included in instructions.Source: §3.1 · p. 4

3.2 Evaluation Protocol

  • Evaluation is zero-shot, with one example per input; predictions use greedy decoding for up to 50 tokens.The study reports the LMentry score as well as individual accuracy and robustness results for each model.Source: §3.2 · p. 4

4 Results

LMentry scores remain well below human performance, and the results show that instruction finetuning improves accuracy while robustness remains vulnerable to trivial changes. Robustness varies inconsistently with scale and across input perturbations.

4 Results

  • TextDavinci002 scores 66.1% on LMentry versus estimated human performance of ~100%; InstructGPT 175B scores 48.4%, and no other model exceeds 41%.The authors attribute the underwhelming scores to both limited accuracy and lack of robustness.Source: §4 · pp. 4–5, Table 2 · p. 5

4.1 Accuracy

  • Instruction finetuning significantly boosts accuracy across tested models; at 175B, InstructGPT improves 66% over GPT-3 and is more accurate on 23 of 25 tasks.Instruction finetuning typically outperforms scaling the same model: finetuning GPT-3 6.7B yields more than training GPT-3 175B from scratch.Source: §4.1 · p. 5, Figure 2 · p. 5
  • Further instruction finetuning in T0 and Tk can improve accuracy, though prior work found more detailed instructions could hurt performance on models below 11B.Most T0 and Tk 11B models fall between InstructGPT 6.7B and InstructGPT 175B in accuracy.Source: §4.1 · p. 5

4.2 Robustness

  • Scaling shows no clear correlation with increased robustness, and TextDavinci002 remains highly sensitive across all four robustness aspects despite typically scoring better.The accuracy difference between TextDavinci002 and InstructGPT 175B exceeds their robustness difference in both absolute and relative terms.Source: §4.2 · p. 6, Table 3 · p. 6
  • InstructGPT models and TextDavinci002 are generally more accurate when the correct answer appears first, while T0 models show the opposite pattern.For TextDavinci002 on alphabetical order, accuracy is 91.1% when the answer is first and 55.5% when it is second.Source: §4.2 · p. 6, Table 4 · p. 7
  • For any words from category, InstructGPT 175B is 100% accurate when all five words do not belong, but only 4% when one does; all T0 models output no regardless of category-word count.The authors say this aspect best demonstrates that accuracy or robustness alone can paint an incomplete picture.Source: §4.2 · p. 6
  • Template rankings are unstable across scale: scaling InstructGPT from 1.3B to 6.7B changes rankings in 18 tasks and the best template in 16.Scaling from 6.7B to 175B also changes rankings in 18 tasks, with the first-place template changing in 15.Source: §4.2 · pp. 6–7, Table 6 · p. 7
  • TextDavinci002 has adjacent-task accuracy gaps above 30 points, while InstructGPT 175B is the most robust InstructGPT model on this aspect, mostly due to gains on negation tasks.Adjacent-task robustness measures accuracy gaps between similar tasks that differ in a specific way.Source: §4.2 · p. 7

5 Related Work

The section surveys perplexity, complex downstream benchmarks, and simple-task evaluations as approaches to assessing language models. It positions LMentry among prior simple-task efforts while distinguishing its task format and ready-to-use design.

Perplexity and complex benchmarks

  • Perplexity tracks overall model quality but offers limited fine-grained insight, while next-token prediction may not align with human behaviors, expectations, or values.Complex downstream benchmarks offer more isolated testbeds, but have grown larger and more complex as model performance improves.Source: §5 · p. 8

Large Complex Benchmarks

  • LMentry avoids the benchmark-complexity arms race with trivial tasks that still provide interpretable insights into capabilities and robustness.Its compact size also makes it accessible to labs with fewer resources.Source: §5 · p. 8

Simple Tasks

  • Unlike oLMpics’ short classification tasks for masked language models, LMentry uses generative tasks and evaluates free-form generations; humans score 100% on LMentry without external resources.The section contrasts this with human performance of 85%–91% on five oLMpics tasks, even with web search; bAbI emphasizes multi-step reasoning and commonsense, HANS focuses on three permutation types, and CheckList is an evaluation framework.Source: §5 · p. 8
  • CheckList helps users create examples for exposing failure cases, but requires creativity and manual work, whereas LMentry is a concrete ready-to-use benchmark.Source: §5 · p. 8

6 Discussion

LMentry tests language models on tasks that are trivial for humans. The reported results show substantial underperformance and brittleness even on these tasks, motivating further research into model behavior and the grounding of complex-task performance.

6 Discussion

  • Even 175B-parameter models significantly underperform humans on tasks trivial to people and are highly brittle to inconsequential input perturbations.The discussion suggests informative insights need not require complex benchmarks, whose size can put them out of reach for many researchers.Source: §6 · p. 8
  • The authors hope the discrepancy between strong performance on complex tasks and weaker LMentry performance encourages research into the principles governing model behavior.They frame LMentry as a first step toward checking whether performance on complex user-facing tasks stands on solid ground.Source: §6 · p. 8

A Tasks

Appendix A presents LMentry’s 25 human-trivial tasks through prompt templates, using basic-level English words for single-word arguments. The tasks cover generation, word and letter properties, comparisons, sound, and sentence context.

A Tasks

B Task-Level Results

Appendix B presents baseline accuracy by task across two tables.

B Task-Level Results

C Robustness Aspects

LMentry assesses robustness across argument order, argument content, template, and adjacent tasks, with coverage varying by task form. The measures capture accuracy gaps under specified changes to prompts, arguments, or task pairs.

Argument Order Robustness

  • Argument-order robustness measures performance on tasks where either of two given arguments can be the correct answer.The listed task pairs include choosing any versus all words from a category and the most versus least associated word.Source: C Robustness Aspects · p. 22

Argument Content Robustness

  • Argument content robustness is the accuracy gap between trivial-from-a-human-perspective argument subsets, measured on six tasks with 3,000 examples per subset.Table 36 lists the subsets used for each task.Source: C Robustness Aspects · p. 22, Table 36 · p. 23

Template Robustness

Adjacent Tasks Robustness

  • Adjacent-task robustness is the mean accuracy gap across pairs of similar tasks that differ in a specific aspect.Examples include tasks about more versus fewer letters and sentence inclusion versus exclusion.Source: C Robustness Aspects · p. 22

D Task Creation Guideline

Figure 3 presents the guideline used during the creation of LMentry’s tasks.

  • Figure 3 presents the guideline used during the creation of LMentry’s tasks.Source: Figure 3 · p. 23

E Automatic Evaluation Study

Table 37 reports agreement and Cohen’s kappa between LMentry’s automatic evaluation and manual human evaluation on predictions from five strong baseline models. Predictions are scored 0 or 1, and kappa is undefined when all predictions receive the same score.

  • Table 37 reports agreement and Cohen’s kappa between automatic and manual human evaluation on predictions from five strong baseline models.Source: Table 37 · p. 24
  • Each LMentry prediction is scored either 0 or 1.Source: Table 37 · p. 24
  • Cohen’s kappa is undefined when all predictions receive the same score, even though agreement is complete.Source: Table 37 · p. 24

Related papers

Loading 2211.02069v2…