PaperlayerView exact revision on arXiv

Source-linked AI summary

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, Jenia Jitsev

arXiv:2406.02061v5cs.LGcs.AIcs.CL

TL;DR

The paper examines whether benchmark performance adequately establishes robust LLM reasoning, given prior reports of failures whose causes remained unclear. It tests models with a simple Alice-in-Wonderland math problem and controlled variations, finding fragile performance across variants and arguing for benchmarks that better expose generalization deficits.

  • Problem

    Earlier reports of LLM failures often lacked systematic controls, leaving unclear whether they reflected core reasoning deficits or fixable formulation and intervention issues.

    Source: §1 · p. 2

  • Method

    The authors use a concise, human-solvable Alice-in-Wonderland problem and vary its natural-number values to test models across structure-preserving instances.

    Source: §1 · p. 2

  • Results

    SOTA LLMs show low average accuracy and strong fluctuations across AIW variants, while control problems rule out basic parsing and arithmetic issues as explanations.

    Source: §5 · pp. 32–33

  • Takeaways & Limitations

    The authors propose novel benchmarks using template variations to reveal generalization and core-function weaknesses that current benchmarks may miss.

    Source: §5 · p. 35

    Earlier studies using similar simple math word problems did not test models at the current frontier, limiting evidence about those models in that setting.

    Source: §4 · p. 32

Abstract

from arXiv · show

Large Language Models (LLMs) are often described as instances of foundation models that possess strong generalization obeying scaling laws, and therefore transfer robustly across various conditions in few- or zero-shot manner. Such claims rely on standardized benchmarks that suppose to measure generalization and reasoning, where state-of-the-art (SOTA) models score high. We demonstrate here a dramatic breakdown of generalization and basic reasoning of all SOTA models claiming strong function, including large scale advanced models like GPT-4 or Claude 3 Opus, using a simple, short common sense math problem formulated in concise natural language, easily solvable by humans (AIW problem). The breakdown is dramatic as it manifests on a simple problem in both low average performance and strong performance fluctuations on natural variations in problem template that do not change either problem structure or its difficulty at all. By testing models on further control problems with similar form, we rule out that breakdown might be rooted in minor low-level issues like natural language or numbers parsing. We also observe strong overconfidence in the wrong solutions, expressed in form of plausible sounding explanation-like confabulations. Various standard interventions in an attempt to get the right solution, like chain-of-thought prompting, or urging the models to reconsider the wrong solutions again by multi step re-evaluation, fail. We use these observations to stimulate re-assessment of the capabilities of current generation of LLMs as claimed by standardized benchmarks. Such re-assessment also requires common action to create standardized benchmarks that would allow proper detection of such deficits in generalization and reasoning that obviously remain undiscovered by current state-of-the-art evaluation procedures, where SOTA LLMs manage to score high. Code: https://github.com/LAION-AI/AIW

1 Introduction

The paper tests unresolved questions about whether reported LLM failures reflect fundamental deficits or issues that simple interventions can fix. Using a concise common-sense math problem and controlled variations, it finds low accuracy and fragile performance across tested models, motivating reassessment of benchmark-based capability claims.

1 Introduction

  • Instantiating different natural numbers in the concise AIW template creates controlled variations for testing sensitivity to problem-irrelevant changes.The authors use repeated trials to test whether models solve the same simple problem consistently across variations.Source: §1 · p. 2
  • All tested SOTA models show low average accuracy and strong fluctuations across AIW variants that preserve problem structure and difficulty.The authors report that even advanced models sometimes produce correct reasoning, but its frequency varies substantially across natural variations.Source: §1 · p. 3
  • Successful performance on AIW Light controls rules out low-level parsing, family-structure, attribute-binding, or elementary-arithmetic issues as explanations for the observed failures.The authors conclude that the failures have a more generic character.Source: §1 · p. 3
  • Strong fluctuations also appear on other AIW versions and in recent reasoning models, indicating the pattern is not unique to the original formulation.The tested versions include DeepSeek-R1 and o1-mini.Source: §1 · p. 3
  • Wrong answers often come with persuasive, overconfident explanations, and models also fail to detect and revise mistakes during self-verification.For variations with low accuracy, wrong answers are prominent, limiting majority voting as a remedy.Source: §1 · p. 3
  • The findings challenge the adequacy of standardized benchmarks for revealing core-function deficits and motivate new benchmarks designed to detect generalization failures.The authors note that many models score highly on standardized benchmarks despite near-zero accuracy on some simple AIW variations.Source: §1 · p. 3

2 Methods & Experiment Setup

The study evaluates how SOTA and reasoning models respond to AIW variants that preserve the problem’s structure, using wording and number changes, targeted controls, and benchmark comparisons to examine robustness and consistency. It also considers model comparisons and reasoning-trace approaches across these variants.

2.1 Simple common sense reasoning problems and their variations

  • The original AIW question asks how many sisters Alice’s brother has; under the shared-parents assumption, the answer is C = M + 1, with N and M varied up to 7.The authors use four number variations intended to preserve the problem structure and difficulty.Source: §2.1 · p. 4

2.1.1 Control AIW Light problems

  • The AIW Light controls modify only the final question to isolate family-relation, arithmetic, and total-girls operations while matching correct answers to the original variations.The three controls ask how many brothers Alice’s sister has, how many siblings Alice has, or how many girls are in total; the latter two require arithmetic, while the family control does not.Source: §2.1.1 · pp. 4–5

2.2 Prompt types, providing input and response parsing

  • The researchers use RESTRICTED, STANDARD, and THINKING prompts to test whether observations persist across output constraints and encouragement to reason.STANDARD requests a natural-number final answer, THINKING encourages careful reasoning, and RESTRICTED permits only the final number; responses are formatted with an answer marker for parsing, with unformatted controls checking that this instruction does not affect behavior.Source: §2.2 · pp. 5–6

2.3 Evaluating model responses

  • Correct responses are treated as Bernoulli trials, with X/n estimating a model’s correct-response rate and a Beta-Binomial approach used to represent the rate distribution.The authors estimate the mean and variance of the response probability and use the resulting Beta distribution to simulate data for plots.Source: §2.3 · p. 6

2.4 Selecting models for evaluation and conducting experiments

  • The study selects prominent SOTA models, tests at least 30 trials per model, variation, and prompt combination, and releases raw responses, estimates, and experiment routines.Models were selected from popular leaderboard rankings, with multiple scales from the same families included where possible.Source: §2.4 · p. 6, results · p. 7

3.1 Humpty Dumpty sat on a wall: breakdown of SOTA LLMs on the simple AIW problem

  • Averaged across RESTRICTED, STANDARD, and THINKING prompts, four models crossed p = 0.3; GPT-4o scored p = 0.649 and Claude 3 Opus p = 0.431, despite strong fluctuations across unchanged AIW variations.For stronger performers, correct response rates could fluctuate from close to 1 to close to 0 across variations that did not change problem structure or difficulty; fluctuations appeared independent of prompt type.Source: §3.1 · pp. 7–8, Figure 3 · p. 8

3.1.1 Control experiments using AIW Light problems

  • Models that struggled on AIW performed strongly across the AIW Light controls, supporting the conclusion that basic family relations, attribute binding, and elementary arithmetic were not the source of failure.The controls separately tested sibling counting, family relations, and total-girls counting; their success, together with the original task’s fluctuations, supports a problem-unspecific generalization and reasoning deficit rather than a low-level parsing or operation failure.Source: Figure 5 · p. 9, Figure 6 · p. 10, Figure 7 · p. 10

3.1.2 Discrepancies between claimed strong function and observed breakdown

  • High MMLU scores did not reliably predict AIW performance or distinguish models’ basic reasoning capabilities.Models with similar high MMLU scores could have dramatically different AIW results, and models such as Command R+ scored zero on AIW despite high MMLU scores.Source: Figure 8 · p. 11, §3.1.2 · p. 11
  • Smaller models generally score below p = 0.1 on AIW, near the paper’s proposed random-guessing level; Llama 2 7B’s higher average comes from one outlier variation.The authors infer that responses around or below p = 0.1 may reflect guessing, since the answer could plausibly be narrowed to 0–10.Source: §3.1.2 · pp. 11–12

3.2 Curiouser and curiouser

  • Adding the redundant statement “Alice is female” raises average correct-response rates for some models, but strong variation-to-variation fluctuations persist.GPT-4o is close to 1 on variations 2 and 4 but drops heavily on variations 1 and 3; adding 60 to the instantiated numbers likewise preserves the breakdown pattern.Source: Figure 9 · p. 12, §3.2 · p. 13, Figure 10 · p. 14

3.2.1 Persisting fluctuations and breakdown on various AIW versions

  • AIW Extended and AIW Friends show low correct-response rates and fluctuations across structure-preserving variations, with performance also differing across these similarly difficult problem versions.The versions change which entities and relations must be handled, but the authors argue the structural and difficulty differences are mild enough that intact generalization should not produce such pronounced performance changes.Source: §3.2.1 · pp. 13, 14, 16, Figure 11 · p. 15

3.2.2 Harder problem versions: further breakdown of advanced LLMs

  • On AIW+, GPT-4/4o and Claude 3 Opus average below p < 0.2, while o1-preview stays close to 1; on Colleague Circles, o1-preview drops below p = 0.8 and fluctuates across number variations.On Colleague Circles, o1-preview is below p = 0.6 on variation 1 but above p = 0.8 on variation 4.Source: §3.2.2 · pp. 17–18, Figure 13 · p. 17

3.3.1 Dominance of wrong responses assessed by frequency distribution of natural numbers on output.

  • For low-performing AIW variations, wrong answers at M dominate output distributions, so majority voting cannot recover the correct answer M+1; stronger variations often peak at M+1.The distribution shape and peak locations are proposed as indicators of model capability and a basis for ranking models.Source: §3.3.1 · pp. 18–19, Figure 14 · p. 19, Figure 15 · p. 19

3.3.2 Overconfident tone and confabulations accompanying wrong answers

  • Models often express strong confidence in wrong solutions, including in THINKING, SCIENTIST, and CONFIDENCE prompts.Examples include Claude 3 Opus claiming its reasoning has no mistakes and Command R+ describing a wrong conclusion as clear and unambiguous.Source: §3.3.2 · p. 20
  • Wrong answers were often accompanied by plausible-sounding confabulations, including nonsensical logic, social or cultural rationales, and refusals based on invented ethical concerns.The passage gives Command R+ as an example of social or cultural confabulations and CodeLLama-70B-instruct as an example of ethical-concern refusals.Source: §3.3.2 · p. 21

3.3.3 Further relevant observations.

  • Follow-up probes found failed self-correction, frequent SQL and parameterized-AIW failures, continued base-model collapse, and no transfer from in-context examples to AIW-Param.Some models, including Mistral/Mixtral, frequently produced correct relational SQL forms, unlike many smaller and some larger models.Source: §3.3.3 · p. 21

3.4.1 Debunking strong function claims

  • Claude 3.5 Sonnet performed near perfectly on most original AIW variations but broke down on structurally similar AIW Ext; NuminaMath-7B also collapsed on AIW despite strong AIW Light control performance.Fine-tuning Llama 3.1 8B on original AIW examples similarly raised original-version performance while leaving AIW Ext weak, illustrating version-specific improvement without robust generalization.Source: §3.4.1 · pp. 21–22, Figure 18 · p. 22, Figure 19 · p. 23

3.4.2 Model comparison and ranking

  • Across AIW original, Ext, and Friends, GPT-4o-mini and Claude 3 Sonnet averaged below p = 0.1, versus above p = 0.3 for GPT-4o and Claude 3 Opus.The smaller models’ gap from their larger counterparts was more pronounced than on MMLU, and AIW variation distributions exposed differences in robustness.Source: §3.4.2 · p. 25, Figure 22 · p. 26
  • The unified robustness score R combines average correct response rate with variation consistency, lowering Llama 2 70B’s rank for relying on one outlier and placing Llama 3.1 70B higher.Average correct response rate alone ranked Llama 2 70B above Llama 3.1 70B, while R placed Llama 2 70B alongside Llama 2 7B.Source: Figure 26 · p. 28, §3.4.2 · p. 28
  • Only o1-preview reached R = 0.9; every other model remained below R = 0.5, with models grouped into mid, low, and very-low performance classes.The ranking used AIW Ext, Friends, Plus, and Circles Colleagues, omitting AIW original because of evidence of possible data leakage.Source: Figure 27 · p. 29, §3.4.2 · p. 29

3.4.3 Model comparison: advanced reasoning models

  • Most reasoning models fluctuated across AIW variations despite strong reasoning-benchmark scores; o1-preview was a notable exception, and reasoning models improved over conventional SOTA LLMs.The evaluation used AIW Friends, AIW Plus, and AIW Circles Colleagues to reduce potential test-data leakage; o1-preview still fluctuated and averaged lower on Circles Colleagues.Source: §3.4.3 · p. 29
  • Across AIW Friends, Plus, and Circles Colleagues, reasoning models show strong response-rate fluctuations across slight template variations, including frontier models on Circles Colleagues.On Friends and Plus, o1-preview and o3-mini are exceptions to the reported lack of robustness; on Circles Colleagues, fluctuations include o1-preview and o3-mini.Source: Figure 28 · p. 30, Figure 29 · p. 30, Figure 30 · p. 31
  • SFT-only distilled models, including R1-Distilled-Llama-70B, OpenThinker-32B, S1.1-32B, and LIMO-32B, improve strongly over their base instruct models, reaching levels comparable to DeepSeek R1 671B and outperforming o1-mini.Larger-scale SFT-only distilled models show response-rate distribution shapes remarkably similar to DeepSeek-R1 and RL-trained o1-mini and o3-mini.Source: §3.4.3 · pp. 30–31
  • OpenThinker-Unverified-32B gains a substantial boost from unverified reasoning traces, while verified traces add further boost.The passage presents this as a hint that unverified traces may suffice to improve problem solving over standard SOTA LLMs.Source: §3.4.3 · p. 31

4 Related work & limitations

Prior work evaluates reasoning with standardized benchmarks while also documenting failures and concerns about whether benchmark scores reflect robust reasoning. The paper situates its findings among those studies and identifies limited variation diversity as a current constraint.

Benchmarking LLM capabilities

  • Standardized reasoning benchmarks span tasks from elementary reasoning to olympiad- and graduate-level problem solving, including ARC, GSM8K, MMLU, AIME24/AIME25, MATH500, and GPQA Diamond.The paper notes that such benchmarks are widely used to measure and improve LLM reasoning performance.Source: §4 · p. 31

Recent reasoning models

  • Larger distilled reasoning models reach mid-range AIW accuracy and robustness, matching or outperforming DeepSeek-R1 and o1-mini, while standard SOTA models and smaller distilled models remain below 0.2.The comparison suggests SFT on reasoning traces can reach performance comparable to approaches employing RL.Source: Figure 31 · p. 32

Related work

  • Earlier studies report reasoning breakdowns and formulation sensitivity; prior simple math-word-problem work did not test current frontier models, while GSM8K-template variations did not conclusively show frontier-model breakdown.The authors contrast their findings of robustness deficits in SOTA and advanced reasoning models with those earlier results [13] [51] [52] [53] [54] [55] [56] [57] [61].Source: §4 · p. 32

Limitations

  • A key limitation is the insufficient diversity and number of AIW variations; the authors propose systematic procedural instance generation for a larger, more varied evaluation set.Other studies have also examined failures, including possible test-data contamination and efforts to build benchmarks intended to assess generalization beyond memorization [14] [15] [48] [50].Source: §4 · p. 32

5 Discussion & Conclusion

The discussion reports severe, fragile performance on simple AIW problems despite benchmark-based capability claims, and argues that current evaluations can miss generalization weaknesses. It identifies leakage, scale-related concerns, safety implications, and the need for open, reproducible research and better benchmarks.

5 Discussion & Conclusion

  • SOTA models show low average correctness and strong fluctuations across AIW variations that preserve problem structure and difficulty.The authors say the fluctuations reveal severe generalization deficits and contrast with claims of robust higher-level problem solving.Source: §5 · pp. 32–33
  • GSM-S fluctuations were 0.07 versus 0.67 on AIW for GPT-4o, while average performance was >0.69 versus <0.15 on AIW for Llama 3 8B.The authors argue these AIW variations reveal deficits that remain hidden on GSM-S; they hypothesize GSM8k-type training-data leakage may help explain the contrast.Source: Figure 32 · p. 33
  • AIW Light controls argue against parsing or elementary-arithmetic problems, while related AIW versions show similar breakdowns and standardized benchmarks miss severe weaknesses.The paper names AIW Ext, AIW Friends, AIW+, and AIW Circles Colleagues as related versions.Source: §5 · p. 33
  • Some advanced and newer reasoning models show correct reasoning or improvement, but strong fluctuations persist across structure- and difficulty-preserving variations.The authors say reasoning is not entirely absent, but remains fragile and cannot be accessed robustly in the tested setting.Source: §5 · pp. 33–34
  • The authors hypothesize that benchmark leakage helps explain strong benchmark results; AIW fine-tuning improved performance on seen variations but collapsed on closely related versions.They argue benchmarks for web-scale-trained models should be immune at least to direct test-data leakage.Source: §5 · p. 34
  • Smaller models show severe AIW breakdown and a large performance gap from larger models, raising concerns that leaky benchmarks distort model appraisal and comparison.The authors hypothesize that current benchmarks cannot properly support model comparisons or scaling-law measurements.Source: §5 · p. 34
  • The paper calls for a fully open, reproducible model-creation and evaluation pipeline, while noting AIW testing still depends on problems absent from unknown training data.Unknown training datasets make that absence hard to ensure.Source: §5 · p. 34
  • The authors warn that persuasive, overconfident wrong answers may obscure errors, especially when users cannot readily check a real-world problem’s solution.They say similar robustness failures are highly likely to manifest on variations of real-world problems, though this is an extrapolation from the simple AIW scenario.Source: §5 · pp. 34–35

A Additional details on performed experiments

This section introduces further details on the procedures used in the experiments.

A Additional details on performed experiments

A.1 Models selected for experiments

Experiments use a defined core-model set under shared inference settings, alongside an overview of AIW versions, prompt types, and selected light-problem variants.

A.1 Models selected for experiments

  • All tested models used T = 0.1 and top-p = 1.0; Table 1 lists the core models’ names, origins, and versions.The paper reports control experiments in which different settings did not change the main behavior pattern.Source: §A.1 · p. 42, Table 1 · p. 42

B Problem versions, prompt types and variations

C Model performance and behavior on AIW problem versions

This section points to supplementary material detailing performance coverage, evaluation behavior, and related AIW variants.

C Model performance and behavior on AIW problem versions

  • Experiments used local deployment via vLLM or API-based liteLLM and TogetherAI.The passage describes these as the execution options used for experiments.Source: C Model performance and behavior on… · p. 44
  • The supplementary overview of average correct response rates includes models scoring zero.The supplied passage does not provide the rates themselves.Source: C Model performance and behavior on… · p. 44
  • Supplementary figures cover trial counts, average output lengths, RESTRICT behavior, prompt-type comparisons, and AIW Ext fluctuations.The passage points to Figures 42, 43, 34, 35, and 41 for these details, respectively.Source: C Model performance and behavior on… · p. 44
  • Supplementary Tables 5–9 cover AIW Light Arithmetic Total Girls, Alice Female Power Boost and Original, Exaggerated Numbers, AIW Ext, and AIW Friends.The captions specify variations 1–4 for Tables 5–8; Table 9 is titled AIW Friends variations 1–4.Source: Table 5 · p. 45, Table 6 · p. 45, Table 7 · p. 46

C.1 Standardized benchmarks failure.

Across several standardized reasoning benchmarks, the tested models’ high scores contrast with their low AIW correct-response rates, including zero responses for some models.

C.1 Standardized benchmarks failure.

  • High scores on MATH, ARC-c, GSM8K, and Hellaswag coincide with low AIW correct-response rates, sometimes zero.The paper visualizes this mismatch in Figures 36–39 and reports it for many models.Source: §C.1 · p. 47
  • Command R+ cannot solve a single AIW problem instance despite scoring high on standardized benchmarks.The paper cites Supplementary Table 12 for this example.Source: §C.1 · p. 47
  • The authors conclude that standardized benchmarks fail to properly reflect models’ generalization and reasoning capabilities.They say the disparity challenges the claim that these benchmarks correctly measure current models’ core functionality.Source: §C.1 · pp. 47, 49

C.2 Boosting by redundant information and persisting fluctuations: Alice female power boost

Adding the redundant statement that Alice is female raises average correct-response rates for many models, but variation-to-variation fluctuations remain.

C.2 Boosting by redundant information and persisting fluctuations: Alice female power boost

  • Adding the fully redundant “Alice is female” information raises average correct-response rates across AIW variations 1–4.The increase is reported for most models that had non-negligible correct-response rates on the original AIW problem.Source: §C.2 · p. 49
  • The boost is significant for most models with non-negligible correct-response rates on the original AIW problem.The redundant information does not remove the strong fluctuations across variations.Source: §C.2 · p. 49
  • The authors interpret this performance change from irrelevant information as further evidence of deficits in generalization and basic reasoning.Strong fluctuations across variations remain after the information is added.Source: §C.2 · p. 49

C.3 Parameterized AIW problem (AIW-Param)

The parameterized AIW task tests whether models can handle a more generic formulation without explicit natural numbers. Models also perform poorly on this version, while responses require manual checking for correctness and reasoning.

C.3 Parameterized AIW problem (AIW-Param)

  • The parameterized problem asks how many sisters Alice’s brother has, with the correct answer M + 1.It omits explicit natural numbers in its generic formulation.Source: §C.3 · pp. 49–50
  • Only five models exceed a 0.2 correct-response rate on AIW; many others perform below 0.15 or collapse to zero.Rates are averaged across variations and THINKING, STANDARD, and RESTRICTED prompts.Source: Figure 33 · p. 50
  • Models also perform poorly on the generic version, whose responses require manual review for answer and reasoning correctness.Automated parsing can misclassify answers when the task uses no explicit natural numbers, and can mark correct answers despite incorrect reasoning.Source: §C.3 · p. 50

D Examples of correct and failed responses

The supplied comparisons show that prompt format affects average AIW accuracy, while performance varies across problem variations and diverges from benchmark scores. The authors also provide collected responses and examples.

D Examples of correct and failed responses

  • The authors provide collected model responses and examples of correct and incorrect answers in the AIW repository.Examples are referenced in supplementary figures 45–49.Source: D Examples of correct and failed… · p. 50
  • RESTRICTED-prompt average accuracy resembles other prompt types, but stronger models show greater variation across AIW variations.RESTRICTED requires only the final answer; the authors use THINKING in the main experiment to avoid disadvantaging models.Source: Figure 34 · p. 51
  • THINKING prompts yield better average AIW accuracy than STANDARD, while THINKING and THINKING v2 behave similarly across tested models.THINKING v2 differs by explicitly saying “step by step.”Source: Figure 35 · p. 51
  • AIW accuracy often diverges from GSM8k, MATH, and Hellaswag scores, with several models scoring high on benchmarks but poorly on AIW.For example, Command R+ has zero correct responses on AIW despite high GSM8k and MATH scores.Source: Figure 36 · p. 52

E Base model experiments

Base models, like instruction-tuned models, perform poorly on AIW, with no remarkable overall differences reported between the groups.

E Base model experiments

  • Base models perform poorly on AIW, with no remarkable difference from the main instruction-model results; Mistral 7B base has the highest average correct-response rate among tested base models.The tested base models include Mixtral 8x7B and 8x22B, Mistral 7B, and LLaMA 2 7B, 13B, and 70B.Source: E Base model experiments · p. 52

F Confabulations and overconfident tone accompanying wrong answers

The authors report that models often defend wrong AIW answers with high confidence and persuasive pseudo-explanations. These confabulations can invoke nonsensical reasoning or irrelevant social and cultural context.

F Confabulations and overconfident tone accompanying wrong answers

  • Models most often express high certainty even when their AIW answers are completely wrong, and may resist revising them when challenged.The authors describe this resistance as “stubbornness” and report it in multi-turn interactions.Source: F Confabulations and overconfident… · p. 53
  • The paper terms persuasive-sounding pseudo-explanations backing incorrect AIW answers “confabulations.”Some contain mathematical calculations or logic-like expressions that make little or no sense for the problem.Source: F Confabulations and overconfident… · p. 53
  • Some confabulations invoke social or cultural context irrelevant to solving the problem, including invented ethical concerns in a CodeLlama-70B-instruct example.The example introduces a person with Down syndrome and argues that the question should be modified.Source: F Confabulations and overconfident… · pp. 53–54
  • Command R Plus gives wrong answers while exploring non-binary identity and culture-dependent family identification despite the problem’s ordinary use of “brother” and “sister.”The paper says these detours generate confusion while retaining a persuasive tone.Source: F Confabulations and overconfident… · p. 54

G Inability to revise wrong solutions

The paper finds that both prompted self-checks and multi-turn user feedback usually fail to make models detect and correct their wrong AIW answers. Revisions occur only rarely, while performance also varies across equivalent problem formulations.

G Inability to revise wrong solutions

  • AIW correct-response rates strongly diverged from ARC-c average scores; Table 12 averages AIW rates across four variations and three prompt types.Source: Figure 39 · p. 55, Table 12 · p. 56
  • Adding the redundant statement “Alice is female” increased average correctness, although the original pronoun “she” already specified her gender.Source: Figure 40 · p. 55
  • Self-verification through THINKING prompts, customized requests for multiple solutions, or unprompted checks lengthened narration but did not reliably revise wrong answers.The authors report poor self-checking across these cases.Source: G Inability to revise wrong solutions · pp. 54–56
  • In multi-turn checks, models usually agreed to re-evaluate but failed to detect or correct errors, often retaining their initial answers with confidence.Some insisted their solution was correct; rare revisions followed repeated user insistence.Source: G Inability to revise wrong solutions · p. 56
  • Correct response rates fluctuated across equivalent AIW formulations, including versions asking about Bob’s sisters versus Alice’s brothers.The cited figure notes that Claude 3.5 Sonnet and Qwen 2 72B Instruct dropped on the latter version.Source: Figure 41 · p. 57
  • Wrong answers were revised only in very rare cases, after users repeatedly insisted that they were mistaken.Source: G Inability to revise wrong solutions · p. 57

H Reformulation of AIW problem as relational SQL database problem

The SQL experiments test whether models can represent the AIW problem’s relational structure and use that representation to reach the right answer. Correct SQL reformulation did not reliably predict correct AIW solutions.

H Reformulation of AIW problem as relational SQL database problem

  • Mixtral 8x22B Instruct v0.1 generated SQL queries that would correctly solve an AIW variation but still gave the wrong final answer.In the example, its answer was 4 instead of 5.Source: H Reformulation of AIW problem as… · p. 57
  • When given only its SQL queries, Mixtral often produced the correct answer; when given the queries alongside its natural-language text, it often answered incorrectly.The authors hypothesize that the model attends more to natural-language text than to the SQL queries.Source: H Reformulation of AIW problem as… · pp. 57, 60
  • Models with fewer than 100 responses per prompt type were excluded from further analysis because their correct-response rates were negligible; Phi-2 had the longest responses.The figure caption attributes Phi-2’s response length to its being a base model less capable of following instructions.Source: Figure 42 · p. 58, Figure 43 · p. 59
  • On AIW-param, older GPT-4 versions had the highest correct-response rates, while GPT-4o fell below p = 0.05 and Claude 3 Opus below p = 0.1.Other tested models had rates close to zero, according to the figure caption.Source: Figure 44 · p. 60
  • Correct SQL reformulation was not predictive of AIW accuracy: GPT-4o performed better than Mistral on AIW but failed the SQL task, while smaller models such as Mistral-7B could not solve it.The authors suggest the smaller models may not discover the AIW problem’s structure.Source: H Reformulation of AIW problem as… · p. 60

I In-context learning experiments

The in-context learning experiments test whether examples teach models the AIW problem’s general solution or only a shortcut. Models often answered examples correctly but failed when that shortcut no longer applied or when asked for a variable-based solution.

I In-context learning experiments

  • After seeing a few solved AIW examples, base models readily produced correct answers on new instances, consistent with using the shortcut M + 1.Source: I In-context learning experiments · p. 60
  • When the query changed so M + 1 no longer applied, models still used that shortcut, suggesting the examples had not induced proper reasoning.The altered query asks for the number of brothers of Alice’s sister.Source: I In-context learning experiments · p. 60
  • On AIW-param, models frequently failed to give the generalized response M + 1, instead producing incorrect variable expressions or explicit numbers.Llama 3 70B sometimes generated the correct response, but the authors found no indication that ICL instantiates better reasoning.Source: I In-context learning experiments · p. 62

Related papers

Loading 2406.02061v5…