Source-linked AI summary
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
TL;DR
Language models perform impressively on standard tasks, but it remains unclear whether this reflects transferable reasoning or specialized procedures learned during pretraining. The paper evaluates default and counterfactual variants across 11 tasks, finding nontrivial but consistently and substantially degraded counterfactual performance. The authors conclude that observed task performance should be interpreted separately from abstract task ability, while noting that counterfactual performance can underestimate reasoning ability when variants add difficulty.
Problem
The paper asks whether language-model abilities generalize to new tasks through task-general reasoning or instead depend on recognizing specific tasks seen during pretraining.
Method
The paper alters default task conditions while preserving the general reasoning procedure, and compares model performance across default and counterfactual variants on 11 tasks.
Results
Across 11 tasks, models show above-random counterfactual performance for most tasks, but performance consistently and substantially degrades relative to default conditions.
Takeaways & Limitations
Standard-benchmark success is not sufficient evidence that language models possess full general capacity for the evaluated tasks.
Takeaways & Limitations
Counterfactual performance may not precisely reflect an LM’s true reasoning ability because some variants introduce additional task difficulty, as in melody retrieval across keys.
Abstract
from arXiv · showhide
The impressive performance of recent language models across a wide range of tasks suggests that they possess a degree of abstract reasoning skills. Are these skills general and transferable, or specialized to specific tasks seen during pretraining? To disentangle these effects, we propose an evaluation framework based on "counterfactual" task variants that deviate from the default assumptions underlying standard tasks. Across a suite of 11 tasks, we observe nontrivial performance on the counterfactual variants, but nevertheless find that performance substantially and consistently degrades compared to the default conditions. This suggests that while current LMs may possess abstract task-solving skills to an extent, they often also rely on narrow, non-transferable procedures for task-solving. These results motivate a more careful interpretation of language model performance that teases apart these aspects of behavior.
1 Introduction
The paper asks whether language-model abilities reflect transferable task-general reasoning or recognition of task-specific procedures, and evaluates this through counterfactual task variants. Across 11 tasks, counterfactual performance is above random for most tasks but consistently and substantially below default performance.
- 1 Introduction: The paper distinguishes generalization to new task variants from generalization to unseen instances of known tasks.This targets whether models deploy task-general reasoning skills or recognize and recall specific tasks seen during pretraining.
- 1 Introduction: The evaluation alters task conditions or rules while preserving the general reasoning procedure and changing the input-output mapping.These variants are called counterfactual tasks because they deviate from default task conditions.
- 1 Introduction: The authors release code, synthetically generated data, and language-model interactions for the evaluation.The released interactions include prompts and responses.
- 1 Introduction: The study evaluates GPT-4, GPT-3.5, Claude, and PaLM-2 under both default and counterfactual conditions across 11 tasks.The task suite spans traditional NLP, code generation, drawing, and spatial reasoning.
- 1 Introduction: Counterfactual performance is above random for most tasks but consistently and substantially degrades relative to default performance.The result indicates some task generalizability alongside reliance on default-condition-specific behaviors.
2 Counterfactual Tasks
The framework models tasks as functions evaluated under world models, then compares default and counterfactual worlds using zero-shot prompts and task-specific metrics. Comprehension checks help distinguish failure to follow counterfactual instructions from failure on the variant task itself.
- 2 Counterfactual Tasks: A task is represented as a function mapping an input to an output under a world model that specifies evaluation conditions.Examples include Python indexing and operator precedence or the number base used in arithmetic.
- 2 Counterfactual Tasks: Traditional evaluations may be confounded by web-text contamination because pretraining can expose models to evaluation instances and support memorization.The framework therefore examines generalization to new task variants rather than only new inputs.
- 2 Counterfactual Tasks: The framework evaluates the same task under default and counterfactual world models to assess whether performance is specific to default conditions.When task difficulty is controlled across worlds, the performance difference can be attributed to overfitting to the default instantiation.
- 2 Counterfactual Tasks: Counterfactuals are defined as variations on default task conditions rather than worlds wholly outside human experience or guaranteed to be absent from pretraining.Base-9 addition illustrates a familiar concept used under altered conditions.
- 2 Counterfactual Tasks: Models receive the task, instance, and world model in zero-shot prompts, after which outputs are parsed and compared with ground-truth labels.The framework uses approximate decoding and task-specific metrics.
- 2 Counterfactual Tasks: Counterfactual comprehension checks test whether prompts specifying altered world conditions are understood.A simpler control task can indicate whether the counterfactual world prompt effectively changes model behavior.
3 Tasks
The paper applies counterfactual evaluation across language, programming, logic, spatial, visual, musical, and game-related tasks. Examples alter conventions such as indexing, premise plausibility, coordinate systems, tunings, keys, and chess starting positions.
- 3 Tasks: The task suite covers deductive reasoning, code generation, drawing, spatial reasoning, music, chess, and the SET card game.It includes both traditional NLP tasks and non-standard or extralinguistic tasks.
- 3.2 Programming: Programming is tested with ThonPy, a fictional language using 1-based indexing while otherwise matching Python.The evaluation uses HumanEval, and the comprehension check uses simpler list-indexing programs.
- 3.4 Natural Language Reasoning with First-Order Logic: Natural-language reasoning uses FOLIO entailment examples whose premises are manually rewritten to violate common sense.The study tests whether performance changes with premise truthfulness, while the comprehension check asks which premise version is true.
- 3 Tasks: Spatial reasoning asks for object coordinates under conventional versus counterfactual cardinal-direction systems.The task tests whether models can apply conceptual spaces to reason about object locations.
- 3.7 Music: Chord placement evaluates fret locations for standard and altered string tunings, including special instruments with one or two altered strings.Drop-D guitar tuning is included to study whether counterfactual-tuning frequency affects results.
- 3.7 Music: Melody note retrieval compares retrieving the n-th note in a canonical key with retrieving it after transposition to another key.The requested output is the n-th note of the scale of the given key.
- 3 Tasks: Chess evaluation changes the initial positions of knights and bishops using the Chess 960 setup before testing four-move opening legality.The task probes understanding of chess rules under an altered starting configuration.
4 Results
Across four evaluated language models, counterfactual performance is consistently below default-task performance, even when models often understand the counterfactual setting. Large gaps, including near-perfect to near-zero drops in some cases, indicate reliance on default-condition-specific behaviors.
- GPT-4, GPT-3.5, Claude, and PaLM-2 were evaluated with and without zero-shot chain-of-thought prompting.The evaluated PaLM-2 model was the second-largest publicly accessible version.
- Counterfactual task performance is substantially worse than default performance both with and without 0-shot CoT.The comparison is reported across the main results in Figures 2 and 3.
- Most counterfactual results remain above random performance, indicating some degree of the targeted ability.This interpretation is strongest when counterfactual comprehension-check accuracy is high.
- When CCC accuracy is high, default-to-counterfactual gaps demonstrate limitations in models’ abstract capacity to solve the target task.Lower CCC can confound this conclusion, but the reported gaps can still be very large.
- Models sometimes fail the counterfactual comprehension check while succeeding under default conditions, pointing to default-condition-specific overfitting.The paper gives arithmetic, programming, and drawing as examples.
5 Analysis
The analysis examines how commonness, proximity to default conditions, task and instance factors, prompting, and demonstrations shape counterfactual performance. Across these analyses, models retain some transferable ability but remain sensitive to default-condition familiarity and often show persistent performance gaps.
- 5.1 “Commonness” of Counterfactual Conditions: More common counterfactual conditions generally produce smaller performance degradations, suggesting a memorization-like effect.Spatial direction swaps and drop-D guitar tuning yield comparatively strong counterfactual performance, while commonness can matter more than proximity in arithmetic.
- 5.2 Proximity between Default and Counterfactual Conditions: LM correctness in FOLIO increases with more true premises and conclusion-truth matches, but decreases with more false or uncertain premises.These statistically significant trends indicate that reasoning accuracy is affected by the distance between the LM-believed real world and the reasoning world.
- 5.2 Proximity between Default and Counterfactual Conditions: Performance generally improves when counterfactual task variants are closer to the default instantiation, although arithmetic is especially sensitive to condition commonness.For guitar and ukulele tunings, performance usually decreases monotonically with increasing distance from the original tuning.
- 5.3 Relationship between Default vs. Counterfactual Performance: Default and counterfactual performance are usually correlated across task variants, instances, and models, preserving traditional evaluations’ usefulness.The correlation indicates some transferable reasoning, but the relationship can reverse in spatial reasoning, where GPT-4 is strongest by default yet suffers the largest degradation.
- 5.4 0-Shot Chain-of-Thought Prompting: 0-shot chain-of-thought prompting generally helps, but it can hurt PaLM-2’s addition and GPT-4 and GPT-3.5’s default-tuning chord performance.The authors suggest overthinking or reliance on memorized input-output mappings as possible explanations for these exceptions.
- 5.5 Few-Shot Demonstrations: The default-counterfactual gap in two-digit addition decreases with more demonstrations but remains substantial for bases 9, 11, and 16.Accuracy improvement plateaus toward 16-shot, so adding demonstrations alone is unlikely to eliminate the gap for arithmetic.
- 5.6 Qualitative Analysis of Drawing Results: In drawing, models may understand counterfactual transformations yet produce untransformed, simplified, lower-quality, or syntactically invalid outputs.GPT-4 passed the comprehension check for flip and rotation instructions, while GPT-3.5 produced more syntactically invalid programs in counterfactual cases.
6 Discussion
The discussion argues that counterfactual performance probes general reasoning beyond familiar task conditions, while highlighting interpretive and evaluation caveats.
- Human performance may also decline under unfamiliar counterfactual conditions when time is limited, despite potentially having the competence to generalize with more reasoning time.The authors frame this as a competence–performance distinction and a hypothesis about humans.
- GPT-3.5 generated parseable programs for 99% of default drawings, versus 62%, 71%, and 75% under vertical-flip, 90°-rotation, and 180°-rotation conditions.These figures summarize the average parseability drop across the three transformations.
- The authors caution that human behavior is not necessarily the target for language-model design, so human counterfactual competence is not itself a model requirement.They state that replicating or modeling human intelligence need not be a primary goal.
- Task-specific reasoning can be useful on familiar tasks, but memorization is an overlooked confound when interpreting language models’ reasoning abilities.The discussion distinguishes usefulness on default tasks from the broader goal of general reasoning.
- The study treats counterfactual tasks as useful proxies for model generalizability and underlying reasoning capabilities, including in practically relevant variants such as arithmetic in different bases.The authors do not claim that counterfactual worlds must be wholly unfamiliar or useless.
- More informative prompts may reduce the default–counterfactual gap, but the authors report that zero-shot chain-of-thought and in-context learning did not fully eliminate it.More advanced prompting methods remain future work.
7 Limitations
The limitations concern whether counterfactual performance cleanly measures transferable reasoning, because variants may differ in difficulty, be encountered during pretraining, or permit shortcuts.
- Counterfactual performance may not precisely reflect an LM’s underlying reasoning ability because several evaluation factors can affect it.The authors introduce the limitations as constraints on interpreting the measured performance.
- 7.1 Underestimation: The intended difficulty match between default and counterfactual tasks is not always achievable, and an objective difficulty measure may not exist.Base-9 and base-11 addition illustrate how altered conditions could independently change task difficulty.
- 7.1 Underestimation: Melody retrieval in an uncommon key may require first recalling a canonical-key melody and then transposing it, making the counterfactual task harder than the original.This two-step strategy can lower counterfactual performance independently of overfitting, although reliance on it may itself indicate canonical-key overfitting.
- 7.1 Underestimation: The authors cannot know how rarely particular counterfactual conditions appeared during pretraining, so observed gaps may underestimate overfitting for genuinely unseen conditions.Online text may already contain examples such as rotated drawings.
- 7.2 Overestimation: Some perturbations permit mapping inputs back to default conditions, and this shortcut was not successfully prohibited in the logical-reasoning task.The authors explicitly disallowed the shortcut for drawing but not for logical reasoning, potentially contributing to high counterfactual performance there.
- 7.2 Overestimation: Non-perfect counterfactual-comprehension accuracy cannot perfectly separate failure to understand the altered condition from failure to perform the counterfactual task.The authors note that CCC itself can involve nontrivial computation, such as program evaluation in ThonPy.
8 Related Work
Related work has shown that language models can encode some grounded conceptual structures, whereas this paper examines whether they can transfer reasoning procedures across altered task conditions.
- Prior studies generally find that language models’ representations of concepts such as color and size can plausibly mirror grounded-world structures.These results concern conceptual structures acquired through text-only training.
- A cited line of prior work finds grounded conceptual structures in language models, while this study reports incomplete acquisition of transferable reasoning procedures.The contrast is between conceptual representation and reasoning-process generalization.
- The paper’s evaluation targets reasoning-process generalization rather than generalization to new concepts or conceptual structures.This distinguishes the present work from prior studies of conceptual representation.
- The paper relates its perturbations informally to interventions in causal-inference frameworks used for reasoning, interpretability, spurious-correlation detection, and fairness.This is presented as a conceptual relationship rather than the paper’s primary evaluation framework.
- Earlier counterfactual work in NLP often concerns alternative events licensed by a default world, unlike this paper’s counterfactual world states that alter default conditions.The paper positions its task-variant framework as a different use of counterfactual evaluation.
9 Conclusion
The conclusion reports consistent performance degradation across 11 counterfactual tasks and attributes the gap to overfitting on default variants. It recommends separating abstract task ability from observed benchmark performance and studying whether grounding improves robustness.
- Across 11 counterfactual tasks, language-model performance consistently and substantially degraded under counterfactual conditions.This is the paper’s central conclusion.
- The authors attribute the default–counterfactual gap to overfitting on default task variants.They connect this interpretation to the need for more careful analysis of benchmark performance.
- Future LM analyses should explicitly consider abstract task ability separately from observed task performance, especially when evaluated variants may be abundant in pretraining data.The recommendation is motivated by possible frequency-related overfitting.
- The authors propose investigating whether more grounded language models are more robust to task variations.They mention grounding in the real world or in a semantic representation as possible directions.
A.2 Programming
The programming evaluations test whether language models can adapt code generation and execution to altered indexing rules while preserving comparable task structure. The execution setup excludes PaLM-2 because its context limit produces truncated, unparseable outputs.
- HumanEval supplies short Python programs for code-generation evaluation against input-output test cases.
- The execution task gives models ground-truth programs and asks them to predict test-case outputs under altered indexing conditions.
- The main execution analysis retains 150 programs and 969 test cases after removing incompatible programs, zero-indexed cases, and indistinguishable outputs.
- The same counterfactual prompt is used for generation and execution, but CCC is included only for execution to preserve comparability with prior generation work.
- PaLM-2 is omitted from execution because its maximum 1,024-token output context causes frequent truncation and unparsable results, especially with 0-shot CoT.
A.3 Basic Syntactic Reasoning
The basic syntactic reasoning evaluations use synthetic word-order variants and counterfactual logical premises to test whether models follow altered structural rules rather than default English or common-sense assumptions.
- Basic Syntactic Reasoning: Synthetic English variants cover all six subject–verb–object orderings by moving subject and object subtrees while preserving other dependents.
- Basic Syntactic Reasoning: The main task identifies a sentence’s main verb and single-word main subject, retaining only examples whose SVO form matches the original English sentence.
- Basic Syntactic Reasoning: The CCC asks models to reconstruct SVO sentences from five counterfactual word orders, testing whether they identify subject, verb, and object.
- Basic Syntactic Reasoning: Because swapped nouns remain equally plausible in the generated sentences, reconstruction has two natural English candidates and cannot rely on a bag-of-words heuristic.
- Basic Syntactic Reasoning: The logical-reasoning study uses 81 manually rewritten counterfactual FOLIO instances and instructs models to avoid common sense or world knowledge.
- Basic Syntactic Reasoning: The authors do not release LM interaction data for FOLIO because subsequent dataset updates were not public at the time of the paper.
- Basic Syntactic Reasoning: Spatial reasoning changes the direction–unit-vector mapping across swapped, rotated, and randomly permuted worlds, measuring room-level accuracy requiring all three objects to be correct.
A.6 Drawing
The drawing and music-related evaluations ask models to transform familiar objects, chords, and melodies under altered orientations, tunings, or keys. Drawing quality is assessed by humans and CLIP, while chord and melody tasks use exact musical criteria.
- Drawing: The drawing task selects 100 Emoji objects that multiple authors judged feasible to render as Processing code.
- Drawing: Counterfactual drawing requires vertical flips or 90° and 180° rotations while prohibiting transformation functions as shortcuts.
- Drawing: Human annotators judge object identity and orientation, averaging results over four annotators and separating objects with or without canonical orientations.
- Drawing: CLIP judgments align with human annotations 84% of the time across sampled default and counterfactual drawings.
- Music: The chord-fingering task evaluates whether fret placements produce exactly the notes of specified guitar and ukulele chords under altered string tunings.
- Music: CCC chord tests ask models to infer notes from all-zero, all-one, or all-two fret sequences, with accuracy computed at the fret level.
- Music: Melody retrieval uses eight famous melodies, asking for notes 1 through 7 in C major by default and in other keys counterfactually.
- Games: The chess task swaps bishops and knights in the initial position and tests legality judgments on 400 procedurally generated four-move openings.
C Raw Results
The raw-results section compiles prompts and results across arithmetic, programming, syntactic and logical reasoning, spatial reasoning, drawing, music, chess, and SET. Tables report task-specific accuracy or pass-rate metrics, with several breakdowns by condition or analysis.
- C Raw Results: Tables 18–34 report numeric results for the evaluated tasks.
- C Raw Results: Prompt tables document default, CCC, and 0-shot-CoT variants across arithmetic, programming, reasoning, drawing, chord, melody, chess, and SET tasks.
- C Raw Results: Arithmetic results are reported as accuracy percentages, including a separate table for GPT-4 analyses with 0-shot CoT.
- C Raw Results: Programming execution results use accuracy, while generation results use pass@1 and pass@10 and distinguish the full HumanEval set from a counterfactual-compatible subset.
- C Raw Results: Syntactic, logical, spatial, drawing, chord, melody, chess, and SET results are reported in task-specific accuracy tables.
- C Raw Results: Spatial results separate room-level tests requiring all objects from object-level accuracy and label swapped, rotated, and random-permutation conditions.
- C Raw Results: Drawing results include both automated accuracy and human-evaluation breakdowns for objects with or without canonical orientation.