PaperlayerView exact revision on arXiv

Source-linked AI summary

oLMpics -- On what Language Model Pre-training Captures

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant

arXiv:1912.13283v2cs.CLcs.AIcs.LG

TL;DR

The paper addresses limited evidence about whether pre-trained LM representations support symbolic reasoning or whether performance comes from fine-tuning. It introduces eight reasoning tasks and a protocol combining zero-shot evaluation with controls, finding qualitatively different, context-dependent abilities and complete failure on many tasks.

  • Problem

    It remains unclear whether pre-trained LM representations capture symbolic reasoning capabilities, including comparison and conjunction, rather than merely supporting fine-tuning.

  • Method

    The paper evaluates eight reasoning tasks using zero-shot tests and learning-curve comparisons with multiple controls to separate pre-training from fine-tuning.

  • Results

    Models differ qualitatively: ROBERTA-L can solve some tasks in zero-shot settings, but success is context-dependent and all models fail on four of eight tasks.

  • Takeaways & Limitations

    The findings and released infrastructure support evaluating LM reasoning and guiding future work on datasets, objectives, and model design.

  • Takeaways & Limitations

    Reasoning successes are not abstract or compositional: performance depends on context and can drop sharply when inputs depart from training-distribution patterns.

Abstract

from arXiv · show

Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a fine-tuned LM to the learning curve of multiple controls, which paints a rich picture of the LM capabilities. Our main findings are that: (a) different LMs exhibit qualitatively different reasoning abilities, e.g., RoBERTa succeeds in reasoning tasks where BERT fails completely; (b) LMs do not reason in an abstract manner and are context-dependent, e.g., while RoBERTa can compare ages, it can do so only when the ages are in the typical range of human ages; (c) On half of our reasoning tasks all models fail completely. Our findings and infrastructure can help future work on designing new datasets, models and objective functions for pre-training.

1 Introduction

The paper introduces reasoning probes and an evaluation protocol that separates capabilities in pre-trained representations from skills learned during fine-tuning. Across eight tasks, models differ qualitatively, succeed only in context-dependent settings, and often fail.

  • Research gap and approach: The paper proposes probes for symbolic reasoning skills and controls for distinguishing pre-trained representations from fine-tuning.The protocol combines zero-shot evaluation with comparisons against controls and learning curves.
  • Research gap and approach: Eight reasoning tasks test operations such as conjunction, comparison, and composition.The probes are designed for reasoning capabilities that may be difficult to capture through a language-model objective focused on word co-occurrence.
  • Findings: ROBERTA-L can solve some tasks perfectly in zero-shot settings while other models perform near random, revealing qualitative differences among similar models.The comparison includes models based on BERT and RoBERTa.
  • Findings: ROBERTA-L compares ages nearly perfectly for values 15-105 but fails outside that range, while birth years from 1920-2000 remain reliable.Performance depends on both the numeric range and linguistic context rather than generalizing to arbitrary scenarios.
  • Resources: The evaluation infrastructure includes code and models intended to support future probe design and analysis of pre-trained language models.The paper reports releasing code and infrastructure for designing and testing probes on many pre-trained LMs.

2 Models

The models section describes BERT-based encoders and two probing setups: masked-token multiple choice for constrained answer sets and question answering for flexible answers. It also defines non-contextualized lower-bound baselines.

  • Model architecture: BERT and related models encode token sequences as contextualized representations using Transformer layers and masked-language modeling pre-training.The encoder produces a representation h for each input token.
  • Probing setups: MC-MLM is used for small, consistent single-token answer sets, whereas MC-QA supports variable answer sets and multi-token answers.MC-QA represents each question-candidate pair and scores candidates with a dedicated QA head.
  • Probing setups: MC-MLM masks one token and scores a restricted set of candidate vocabulary tokens using the masked token’s representation.The candidate set contains the correct token and K −1 distractors, with K ∈ {2, 3, 4, 5}.
  • Probing setups: The MC-MLM setup can obtain reasonable performance without training by reusing pre-trained representations and masked-language-model head weights.This makes zero-shot evaluation possible for suitable probes.
  • Baselines: Two non-contextualized baselines provide lower bounds for MC-MLM and MC-QA using GLOVE representations and task-specific architectures.The MLM baseline uses a feed-forward head over concatenated token vectors, while the MC-QA baseline uses ESIM.

3 Controlled Experiments

The controlled evaluation combines zero-shot masked-language-model testing with fine-tuning learning curves and controls to distinguish pre-trained representations from learning during fine-tuning. AGE-COMPARE shows strong but language- and context-sensitive performance for ROBERTA-L.

  • Zero-shot evaluation: Zero-shot MC-MLM evaluates masked-token predictions using pre-trained representations and the existing MLM head without fine-tuning.The task is formatted as a statement with one masked token and a small candidate set.
  • Zero-shot evaluation: 98% accuracy: ROBERTA-L compares ages correctly, versus 70% for BERT-WWM and 50% random performance for BERT-L.The MLM-BASELINE is also random because its MLP_MLM weights are randomly initialized.
  • Learning curves: Learning curves vary training examples from 62 to 4K, while fixed representations are used and only MLP_MLM parameters are fine-tuned.MAX summarizes peak accuracy, while WS weights low-example performance more heavily.
  • Generalization and setup: Testing ages 15–38 after training on ages 43–120 shows that ROBERTA-L generalizes to unseen values, while ROBERTA-L remains best with the fewest MC-QA examples.MC-QA requires more examples because its MLP_QA cannot be initialized from pre-trained weights, making zero-shot evaluation uninformative.
  • Controls: The NO LANGUAGE control removes all tokens except the mask and task arguments, testing whether performance depends on language input.Similar full-input and reduced-input curves indicate low language sensitivity and weaker evidence that pre-trained language representations explain performance.
  • Controls: ROBERTA-L is highly language-sensitive: its control performance becomes random without language and decreases substantially when key words are replaced with nonsense.BERT-WWM shows lower language sensitivity in the NO LANG. control.

4 The oLMpic Games

The oLMpic Games probe symbolic reasoning skills such as numeric comparison, conjunction, and multi-hop composition across automatically constructed tasks. Results show that models differ qualitatively and that ROBERTA-L’s age comparisons depend strongly on the numerical context.

  • Probe design: The probes test symbolic operations including number comparison, shared properties, and multi-hop composition, while some tasks also require background knowledge.Automatically generated examples and manually selected phrasing can interact with language abilities.
  • Age comparison: ROBERTA-L and BERT-WWM handle AGE-COMPARE better than controls, but different LMs with similar architectures show clear qualitative differences.ROBERTA-L and, to some extent, BERT-WWM perform well under the controls.
  • Original comparison: 98% zero-shot accuracy: ROBERTA-L almost perfectly predicts “younger” and “older” in the original AGE-COMPARE experiment.Figure 3A corresponds to this original comparison setup.
  • Birth-year perturbation: 76% accuracy: given birth years, ROBERTA-L correctly flips the prediction so that earlier birth years indicate older age.For example, it judges a person born in 1980 older than one born in 2000.
  • Age-value perturbation: 12% accuracy: with typical ages in the same statement, ROBERTA-L consistently outputs the opposite prediction.The model appears to compare values while disregarding the language in this condition.
  • Argument-value sensitivity: Swapping ages with typical birth years makes ROBERTA-L always predict “older”, emphasizing sensitivity to argument values.The perturbations therefore distinguish numerical-value sensitivity from robust age reasoning.

Can Language Models compare object sizes?

The size-comparison probe tests whether LMs combine property knowledge with comparison. Its prompts compare manually sampled objects using masked “larger” or “smaller” answers.

  • A neutral-context preference for “older” over “younger” could potentially explain one comparison result.
  • The probe requires both knowledge of an object’s numeric property value and the ability to compare values.
  • Probe Construction: Prompts state that one object is usually much [MASK] than another, with “larger” and “smaller” as candidate answers.
  • Probe Construction: Objects are manually sampled from lists spanning animal and other domains.

4.2 Do LMs know “always” from “often”?

The ALWAYS-NEVER probe asks LMs to distinguish event frequencies from masked statements. Models perform poorly, and fine-tuning appears to account for much of the limited success rather than pretrained representations.

  • The “Always-Never” task: The probe asks models to select frequency quantifiers such as “never” or “always” for statements about events.
  • 1,300 examples were collected, with 1,000 used for training and 300 for evaluation.
  • 56% of examples involve sizes that can overlap, requiring models to determine whether object sizes are non-overlapping.
  • Results: 20% random accuracy and 35.5% majority-vote accuracy provide the task baselines.
  • Results: In zero-shot evaluation, accuracy is below random; fine-tuning reaches 57% in BERT-L, similar to MLM-BASELINE.
  • Results: MLM-BASELINE consistently outperforms ROBERTA-L, while language controls show little performance degradation.
  • Analysis: Models rarely predict “often” and “rarely,” despite those answers being gold labels in 19% of training examples.
  • Analysis: The analysis reports that models always predict “sometimes” for the template involving a dish with a food type.

4.3 Do LMs Capture Negation?

The negation probe examines whether LMs use “not” when context contains antonyms and distinguish it from intensifiers in synonym contexts.

  • The probe tests whether the presence of “not” affects masked-token prediction in synonym-versus-antonym contexts.
  • It targets whether models properly use negation while distinguishing taxonomic relations between synonyms and antonyms.

Do LMs Capture the Semantics of Antonyms?

The antonym-negation task requires selecting “not” for antonym contexts and an intensifier such as “very” for synonym contexts. Reported results compare model learning curves and language controls.

  • Antonym contexts should elicit “not,” whereas synonym contexts should elicit a word such as “very.”
  • Correct performance requires knowledge of taxonomic relations and reasoning about negation usage in context.
  • Figure 4 presents learning curves for two tasks, showing each best-performing LM alongside NO LANG. and baseline controls.
  • MLM-BASELINE reaches a comparable Language Sensitivity of 67 and MAX of 80%.
  • The table reports antonym-negation accuracy over two answer candidates, with random accuracy at 50%.

4.4 Can LMs handle conjunctions of facts?

The probes test whether LMs can combine multiple facts, either by finding concepts satisfying two properties or by identifying shared taxonomic properties. Results show some task capability, but language sensitivity and baseline proximity complicate attribution to pre-training.

  • Property conjunction: Property conjunction asks models to identify a concept satisfying two CONCEPTNET properties while rejecting distractors satisfying only one.Examples are automatically generated from predicate-object pairs, with disjoint concepts in training and evaluation.
  • Property conjunction: The parallel “but not” version raises MAX results to 79-96%, while improvement over ESIM-Baseline and language sensitivity remain low.The authors interpret the high MAX results as evidence that relevant information is present, but pre-training attribution remains unclear.
  • Overall assessment: Overall, performance near baselines and low language selectivity make it difficult to determine whether conjunction ability existed before fine-tuning.The same attribution concern applies to the property-conjunction results.
  • Taxonomy conjunction: Taxonomy conjunction tests whether models can find a mutual hypernym for two concepts, using distractors that are hypernyms of only one entity.Evaluation focuses on food and animal taxonomies with disjoint taxonomic trees between training and evaluation.
  • Taxonomy conjunction: A common taxonomy error is choosing the hypernym closer to one entity rather than the gold answer shared by both entities.For example, models may select “bird” for a crow and horse pair instead of the more general answer “animal.”

4.5 Can LMs do multi-hop reasoning?

The multi-hop probes test whether LMs can compose facts or perform sequential comparisons. Models generally fail to demonstrate reliable multi-step reasoning, despite isolated partial success during probing.

  • Probe design: The paper introduces compositional question answering and multi-hop comparison probes to test whether LMs can combine previously observed capabilities.The probes build on the observation that some models can compare ages.
  • Encyclopedic composition: Encyclopedic composition requires resolving a relative-clause reference and then answering a question using multiple linked facts.Examples include asking when the band where John Lennon played first formed.
  • Encyclopedic composition: All models achieve low accuracy on encyclopedic composition, while the baseline performs best with a MAX of 54%.Language sensitivity is small, and the models are first fine-tuned on the required single-hop facts before composition.
  • Multi-hop comparison: Multi-hop comparison requires finding the oldest among several ages and returning that person’s ordinal position.The probe uses “first,” “second,” and “third” as answers for age arguments in the AGE-COMPARE ranges.
  • Multi-hop comparison: ROBERTA-L’s multi-hop comparison accuracy is almost random zero-shot, while BERT-WWM reaches a MAX of 65% after about 4,000 examples.All LMs appear to learn the task during probing, and the models do not show clear multi-step capability.

5 Medals

Across the oLMpic Games, pre-training yields limited symbolic-reasoning success. ROBERTA-L is strongest on several tasks, but perturbations expose inconsistent generalization and abstraction.

  • Overall medals: LMs generally do not demonstrate strong pre-training capabilities across the symbolic reasoning tasks.BERT-WWM shows partial success in a few tasks, while ROBERTA-L performs highly on ALWAYS-NEVER, OBJECTS COMPARISON, and ANTONYM NEGATION.
  • Overall medals: ROBERTA-L emerges as the most promising LM, but perturbed inputs reveal inconsistent generalization and abstraction.The result contrasts high performance on selected tasks with sensitivity to changes in the probe context.

6 Discussion

The paper finds that reasoning abilities differ qualitatively across similarly structured LMs and that successful behavior is context-dependent rather than abstract. Its probes and released infrastructure are intended to support further evaluation and model development.

  • Findings: Even models with identical structure and objective functions differ both quantitatively and qualitatively in their reasoning abilities.ROBERTA-L shows abilities absent from other models.
  • Findings: With appropriate data and optimization, models can acquire skills from an LM objective that may initially seem surprising.This conclusion follows the paper’s comparison of reasoning abilities across models.
  • Limitations: When LMs succeed, their reasoning is context-dependent: age comparison works when the numbers are typical ages, while distributional discrepancies cause large performance drops.The paper states that current LMs do not reason through abstraction and composition as humans perceive them.
  • Implications: The released code and data are intended to help evaluate reasoning abilities, design new probes, and guide future pre-training, objective-function, and model-design work.The stated goal is to address capabilities that current models lack.
Loading 1912.13283v2…