PaperlayerView exact revision on arXiv

Source-linked AI summary

On Memorization of Large Language Models in Logical Reasoning

Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, Ravi Kumar

arXiv:2410.23123v2cs.CL

TL;DR

The paper asks whether strong LLM reasoning benchmark performance reflects memorization, a question raised by the coexistence of impressive reasoning and basic mistakes. It measures memorization with a perturbation-based score and a dynamically generated Knights and Knaves benchmark. Fine-tuning leads to heavy memorization while generalization improves, and analyses indicate that models develop reasoning skills alongside memorization.

  • Problem

    LLMs can perform well on challenging reasoning benchmarks yet make basic mistakes, leaving unclear how they solve reasoning tasks and whether benchmark success reflects memorization.

    Source: §1 · p. 1

  • Method

    The authors define LiMem using inconsistency under local perturbations and study LLMs with a dynamically generated Knights and Knaves benchmark.

    Source: §1 · pp. 1–2

  • Results

    Fine-tuning produces heavy memorization of training puzzles while generalization improves, and analyses indicate that models develop reasoning skills alongside memorization.

    Source: §1 · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The feature-rich benchmark could support future studies of LLM logical reasoning.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We find that LLMs could interpolate and memorize the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet they struggle with slight variations of these puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. Through in-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers, we establish that LLMs develop reasoning skills on K&K puzzles alongside memorization. Finally, our analysis based on a per-sample memorization score sheds light on how LLMs switch between reasoning and memorization when solving logical puzzles. Our code and data are available at https://memkklogic.github.io.

1 Introduction

The paper asks whether LLM reasoning success reflects memorization, given that strong benchmark performance coexists with basic mistakes. It introduces a perturbation-based measure and a Knights and Knaves benchmark to study memorization alongside reasoning.

1 Introduction

  • The paper examines an apparent contrast between LLMs’ impressive performance on challenging reasoning problems and mistakes on basic problems.The authors say understanding how LLMs solve reasoning tasks matters for learning and generalization behavior, as well as robust real-world applications.Source: §1 · p. 1
  • LiMem treats high accuracy paired with inconsistent performance on local perturbations as evidence of memorization.The perturbations require similar underlying reasoning principles.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The K&K benchmark generates puzzles at different difficulty levels, perturbs existing puzzles at the problem level, and synthesizes detailed reasoning steps.The study evaluates 17 off-the-shelf LLMs and fine-tunes Llama3-8B and GPT4o-mini.Source: §1 · p. 2
  • Fine-tuning can produce heavy memorization of training puzzles while generalization performance also improves as memorization grows.The authors investigate this pattern through perturbation tests, cross-difficulty transfer, internal probing, and fine-tuning with wrong answers.Source: §1 · p. 2

2 Measuring Memorization in Reasoning

The section defines memorization as accurate performance on observed puzzles coupled with inconsistency under minimal perturbations, then introduces a dynamic Knights and Knaves benchmark for measuring it.

2.2 Knights and Knaves Reasoning Benchmark

  • The benchmark’s abstract and natural-language modules generate puzzles and reasoning steps, then perturb mathematical structure or wording and recompute answers.Abstract perturbations replace a statement or leaf node and ensure the solution changes; language perturbations diversify names and templates.Source: Figure 2 · p. 3, §2.2 · pp. 3–4

2.1 Memorization Metrics for Reasoning Tasks

  • LiMem combines accuracy with inconsistency under local perturbations; a high score indicates memorization, while a low score signals reasoning only when accuracy is also high.The perturbations preserve the underlying solving principle and a similar difficulty level.Source: §2.1 · p. 3

2.2 Knights and Knaves Reasoning Benchmark

  • The released core dataset contains 1,000/100/50 train/test/validation puzzles for each 2–8-person task, with six perturbed variants per puzzle.Default puzzles use maximum tree width and depth of 2; the variants include two problem-level and four language-level perturbations.Source: §2.2 · p. 4
  • Except for o1, the best models reach at most 70% accuracy on 2-person puzzles, falling to 11% on 8-person puzzles.The benchmark evaluates 17 leading models; prompting techniques do not fundamentally improve performance on challenging tasks.Source: §2.2 · p. 4

3 Quantifying Memorization in Reasoning

The section measures memorization after fine-tuning K&K puzzles and finds that training-set interpolation coexists with perturbation inconsistency and transfer to unseen difficulty levels.

Fine-tuning setup

  • The study fine-tunes Llama3-8B and GPT4o-mini separately by task, using either question-answer pairs or questions with synthetic reasoning steps and answers.The models are fine-tuned on training puzzles disjoint from the test set.Source: §3 · p. 4

Results

  • Fine-tuning on M-person puzzles generally improves test accuracy on N-person puzzles, including when N differs from M.Most accuracy-improvement grid values are above zero, indicating transfer across unseen tasks.Source: Figure 6 · p. 5
  • GPT4o-mini nearly reaches 100% training accuracy with both direct-answer and chain-of-thought fine-tuning.Fine-tuned models show high training accuracy on tasks such as 3- and 5-person puzzles.Source: §3 · p. 5
  • LiMem reaches ∼50% on the 8-person training task, with larger scores on harder tasks and generally lower scores for GPT4o-mini than Llama3-8B.The gap between original and consistently solved perturbed samples indicates heavy reliance on memorization.Source: §3 · p. 5
  • Math-level perturbations yield higher memorization scores than language-level perturbations; role-flipping leaves Llama3-8B near zero accuracy and about 80% LiMem.The authors suggest models may disregard the altered statement because of an internal bias associating knights with truthfulness.Source: §3 · p. 5

4 Learn to Reason by Fine-tuning With Answers Only

Direct fine-tuning on question-answer pairs improves K&K reasoning even as models memorize training puzzles. Transfer, probing, and incorrect-answer tests provide evidence of learning beyond memorization.

4 Learn to Reason by Fine-tuning With Answers Only

  • Fine-tuned models can both memorize K&K puzzles and improve their reasoning capability as memorization increases.Direct fine-tuning can teach models to reason from question-answer pairs without detailed reasoning steps.Source: §4 · p. 5

4.1 Reasoning capabilities of Direct FT-ed model

  • Training on M-person puzzles generally improves test accuracy on N-person puzzles, including harder tasks, with larger gains for N ≤6.More epochs improve generalization, especially for Llama3-8B; when fine-tuned on 10k 8-person puzzles, GPT4o-mini reaches about 90% test accuracy on moderately difficult 4/5-person puzzles.Source: §4.1 · pp. 5–6, Figure 9 · p. 6
  • On correctly solved training puzzles, inconsistency decreases over epochs even as the memorization score rises; role-flipping yields a higher score, possibly reflecting a bias that knights are truthful.The inconsistency reduction aligns with the improved transferability reported across difficulty levels.Source: §4.1 · pp. 5–6, Figure 7 · p. 6
  • 52% test accuracy on 8-person tasks is reached by GPT4o-mini Direct FT on 10k puzzles, where the unfine-tuned model scores near zero.The training uses answers only; CoT fine-tuning is generally more effective with 10k samples, except on the 2-person task, where a distribution gap can lead to looping responses.Source: §4.1 · p. 6

4.2 Probing Direct FTed models

  • Probing accuracy rises in deeper layers and peaks around layers 14–15, indicating that internal representations distinguish true from false puzzle statements.The probing accuracy is higher than in the unfine-tuned model, and puzzles with more people reach 85% probing accuracy at later layers.Source: §4.2 · p. 6, Figure 8 · p. 6, Figure 10 · p. 7

4.3 Direct FT with Wrong Answers

  • Direct fine-tuning with incorrect answers still yields gradual test-accuracy improvements, which cannot be explained by pure memorization of the fully mislabeled training set.Each wrong answer retains some correct role assignments, which may help the model learn to reason.Source: §4.3 · p. 7

5 Learn to Reason by Fine-tuning with CoTs

Fine-tuning with reasoning steps can improve accuracy on unseen K&K puzzles, particularly when model capacity and training data are sufficient. Even corrupted reasoning steps can yield gains, though they hinder convergence and generalization compared with clean reasoning chains.

5 Learn to Reason by Fine-tuning with CoTs

  • CoT fine-tuning improves test accuracy on unseen puzzles, and can outperform Direct FT, especially with larger training sets.Llama3-8B struggles with CoT fine-tuning, likely because its capacity is insufficient to learn CoT skills from ≤1K samples.Source: §5 · p. 7
  • Fine-tuned models show lower inconsistency despite increased memorization and positive transfer to easier and harder tasks.CoT fine-tuning can yield much higher test accuracy than Direct FT, especially with a larger training set.Source: §5 · p. 7
  • Even fully corrupted CoT data can improve test accuracy over training epochs, but incorrect or shuffled steps slow convergence and reduce accuracy versus clean CoT.Shuffling the reasoning steps further harms convergence and generalization.Source: §5 · p. 7, Figure 12 · p. 7

6 Distinguish Memorization from Reasoning

The paper tests whether puzzle and model features can distinguish examples solved consistently from those solved inconsistently. These indicators are informative but imperfect, with model embeddings outperforming puzzle-based indicators in one reported setting.

6 Distinguish Memorization from Reasoning

  • The per-example memorization score treats a correctly solved puzzle as memorized when it becomes unsolvable after local perturbation.Examples the model does not solve initially are excluded because their consistency ratio is undefined.Source: §6 · p. 7, Figure 13 · p. 8
  • 0.629/0.787 test AUC for Direct/CoT FT-ed GPT4o-mini and 0.627 for Direct FT-ed Llama3-8B show puzzle-based indicators are informative but imperfect.The indicators include TF-IDF, bag-of-words, word length, and character length from puzzle questions, synthetic reasoning steps, or model responses.Source: §6 · p. 8, Figure 13 · p. 8
  • The best embedding-based indicator reaches 0.70 AUC, outperforming the puzzle-based indicator at 0.627 AUC on 3-person puzzles.Lower-layer features poorly distinguish reasoning from memorization, while higher-layer features are more informative; fine-tuned model features are consistently more informative than unfine-tuned ones.Source: §6 · p. 8

Related Work

The paper distinguishes its focus on memorization in reasoning from prior work on verbatim regurgitation, privacy, and copyright. It also situates its benchmark among synthetic logical-reasoning evaluations, especially K&K puzzles.

Related Work

  • Unlike prior work focused on near-verbatim training-text regurgitation, this paper measures memorization in reasoning tasks using local puzzle perturbations.The cited contexts for prior memorization research include privacy and copyright concerns.Source: Related Work · p. 8
  • Compared with TruthQuest’s 3–6-person K&K puzzles and answers, this benchmark supports dynamic generation of puzzles, perturbations, solutions, and detailed reasoning steps.The paper also defines and measures memorization and examines its relationship to reasoning.Source: Related Work · p. 8

Conclusion

The conclusion summarizes LiMem and the dynamic Knights-and-Knaves benchmark for quantifying memorization, alongside analyses of reasoning skills and puzzle perturbations. It focuses primarily on benign perturbations, treats role-flipping as an adversarial reference, and leaves open how reasoning skills develop and how to encourage reasoning over memorization.

Conclusion

  • LiMem measures memorization through inconsistency on locally perturbed puzzles, and analyses find reasoning skills emerge alongside memorization as LLMs learn training examples.The study also examines local perturbations, transferability, intermediate-output probing, fine-tuning with wrong answers, and signals associated with reasoning versus memorization.Source: Conclusion · p. 8

A Discussion and Future Work

  • Open questions include how reasoning develops from question-answer fine-tuning, why test accuracy remains below 100%, and whether interventions can favor reasoning over memorization.The authors also ask whether models learn shortcut rules that solve only a subset of puzzles.Source: A Discussion and Future Work · p. 14

B Extended Related Work

  • The dynamic K&K benchmark extends prior work with automatically generated perturbations, solutions, and reasoning steps, and supports scalable variation in difficulty and dataset size without extra human effort.It enables measuring memorization in reasoning tasks; its mathematical-level perturbations change the ground-truth solution while keeping puzzles superficially similar and comparably difficult.Source: B Extended Related Work · pp. 14–16
  • Fine-tuned GPT4o-mini and Llama3-8B generalize to unseen in-distribution and out-of-distribution K&K problems, while analyses relate their developing reasoning skills to grokking.The discussion notes that reasoning improves during fine-tuning as models memorize more puzzles, but the development mechanism remains unclear.Source: A Discussion and Future Work · p. 14, B Extended Related Work · p. 15

C.2 The Abstract Puzzle Module: Generator

  • The generator samples puzzles from a random seed and difficulty specification (N, W, D), while the abstract representation encodes people and statements with integers, tuples, and logical operators.Invalid or uninteresting sampled cases are skipped and resampled.Source: §C.2 · p. 16, §C.1 · p. 16

C.3 The Abstract Puzzle Module: Solver and Reasoner

  • The Solver retains only puzzles with exactly one Boolean-assignment solution, while the Reasoner generates stepwise explanations by proposing roles, checking contradictions, and backtracking.The Reasoner can reorder the examination queue to better match human behavior.Source: §C.3 · p. 17

C.4 The Abstract Puzzle Module: Perturber

  • Abstract perturbations make local changes while preserving the puzzle’s underlying principle and roughly maintaining difficulty, with a unique solution different from the original.The supported operations are replacing a person's statement tree or changing one leaf in an existing tree.Source: §C.4 · p. 17

C.7 The Natural Language Module: NL-Perturber

  • The natural-language module converts abstract puzzles and Reasoner outputs into text, while its language-level perturber changes wording without changing the underlying puzzle or solution.The NL-Generator uses sampled names and statement templates to render claims.Source: §C.5 · p. 18, §C.6 · p. 18, §C.7 · p. 19
  • The NL-Perturber’s benign variations replace character names or role labels and reorder each person’s statement.Role-label pairs include saint/sinner and hero/villain; uncommon names are randomly sampled.Source: §C.7 · p. 19
  • Role-flipping causes many model mistakes despite producing valid, unambiguous problems, so it is retained as a reference rather than a main focus.The perturbation reverses the usual truth conditions: knaves tell the truth while knights lie.Source: §C.7 · p. 19

C.8 Dataset Generation

The dataset uses K&K puzzles with bounded statement width and depth across two to eight people, and provides locally perturbed variants. The section also reports perturbation validity and identifies the models evaluated.

C.8 Dataset Generation

  • The dataset uses maximum statement width W = 2 and depth D = 2, with puzzle sizes from 2 to 8 people.Training and test length distributions are described as similar.Source: §C.8 · p. 19
  • Leaf perturbation changes one truth-value leaf, while statement perturbation changes an entire statement; both examples change the answer.The section also compares math-level with language-level perturbations in Figure 20.Source: §C.8 · p. 19, Figure 20 · p. 20
  • Figures 14–19 show generated questions, answers, synthetic reasoning traces, and training-data length distributions.The question and reasoning-trace examples cover 3-, 5-, and 8-person tasks.Source: Figure 14 · p. 20, Figure 15 · p. 20, Figure 17 · p. 20
  • Leaf perturbations were valid for 76% of 2-person samples, rising to 100% for 7/8-person tasks under a 2,000-attempt limit.The reported rates for 3-, 4-, 5-, and 6-person tasks were 93.4%, 95.4%, 98.8%, and 99.5%, respectively.Source: §C.8 · pp. 19–20

D.1 Models

  • Table 1 presents a 2-person puzzle example in original, leaf-perturbed, and statement-perturbed forms.Table 2 lists evaluated model links or endpoint specifications.Source: Table 1 · p. 21, Table 2 · p. 21

D.2.1 EVALUATION

The evaluation uses task-specific prompts and deterministic decoding, with correctness assessed from the model’s final conclusions. The section also describes fine-tuning, probing, sample classification, and computing resources.

D.2.1 EVALUATION

  • Evaluation uses greedy decoding at temperature t = 0, with a maximum output length of 2048 tokens, and keyword-matches each person’s ground-truth identity in the conclusion.The default prompt asks for reasoning followed by a clearly formatted conclusion; the appendix also explores 0-shot CoT and 1-shot direct/CoT prompts.Source: §D.2.1 · pp. 21, 22, 24

D.2.2 FINE-TUNING

  • Direct FT trains on instructions, questions, and conclusions, while CoT FT additionally includes synthetic reasoning steps.Llama3-8B uses LoRA with a 5e-5 learning rate; GPT4o-mini uses the fine-tuning API defaults and is trained for five epochs.Source: §D.2.2 · p. 24

D.2.3 PROBING

  • Probing trains logistic-regression classifiers on MLP outputs from transformer blocks to distinguish correct from incorrect statements.The examples pair K&K puzzles with conclusions about characters’ roles.Source: §D.2.3 · p. 25

D.2.4 DISTINGUISHING MEMORIZATION FROM REASONING

  • The memorization-versus-reasoning classifier uses correctly answered training samples, labels consistency under perturbation, then splits samples 80%/20% for binary classification..Source: §D.2.4 · p. 25

D.2.5 COMPUTATION RESOURCES

  • Fine-tuning runs on two 80-GB NVIDIA A100 GPUs, while evaluation can run on one 48-GB NVIDIA RTX A6000 GPU.Source: §D.2.5 · p. 25

E.1 Benchmark Performance of Off-the-shelf Models

Off-the-shelf models perform poorly on K&K puzzles, and prompting methods do not substantially improve their results. During fine-tuning, memorization grows while inconsistency on correctly solved training puzzles declines; reported analyses compare perturbation types and train/test consistency.

E.1 Benchmark Performance of Off-the-shelf Models

  • Open-source models show poor K&K accuracy, especially as puzzle size increases, and prompting methods do not significantly improve performance.Figure 21 reports test accuracy and LiMem across prompt formats.Source: §E.1 · p. 26, Figure 21 · p. 26
  • Self-consistency gives limited improvement on the 3-person task and fails to enhance GPT4o-mini’s accuracy on the 8-person task.The evaluation queries each puzzle 40 times at temperature 0.7.Source: Table 3 · p. 26, §E.1 · p. 26

E.2 Memorization Measurement

  • As fine-tuning epochs increase, inconsistency on correctly solved training puzzles tends to decrease even as training-set LiMem increases.The trend is reported for both Direct FT and CoT FT GPT4o-mini.Source: §E.2 · p. 27, Figure 22 · p. 27
  • Fine-tuned models have higher memorization scores on training puzzles than test puzzles, especially on hard tasks, and show stronger memorization under math-level than language-level perturbations.Combining math-level and language-level perturbations can produce higher scores than language-level perturbations alone.Source: Figure 23 · p. 27, Figure 25 · p. 28
  • Consistency ratios are generally higher on solved test puzzles than solved training puzzles for challenging 5/8-person tasks, but remain comparable for 3-person puzzles.Consistency ratios are generally higher on easy than hard tasks.Source: Figure 24 · p. 28

E.3.1 LLAMA3-8B

For Llama3-8B, transfer from 8-person training is strongest with 10k puzzles, but prolonged training can reduce test accuracy on easier 2-person puzzles.

E.3.1 LLAMA3-8B

  • Figure 26 reports Llama3-8B train and test accuracy across training epochs under different evaluation configurations.Source: Figure 26 · p. 29, §E.3.1 · p. 29
  • Prolonged fine-tuning on 10k samples can lower Llama3-8B test accuracy, especially on 2-person puzzles, potentially through overfitting to the harder 8-person task.Source: §E.3.1 · p. 29
  • Figures 27 and 28 show transfer across problem sizes and training epochs, presenting accuracy gains over the unfine-tuned baseline and absolute accuracy, respectively.Source: §E.3.1 · p. 29, Figure 27 · p. 30, Figure 28 · p. 31
  • 10k 8-person puzzle fine-tuning yields 87% accuracy on 3-person tasks, versus nearly 0 for the unfine-tuned model.The models train on answers without reasoning steps; the results suggest that training on the hardest tasks transfers underlying rules to easier ones.Source: §E.3.1 · p. 29, Figure 29 · p. 31

E.3.2 GPT4O-MINI

For GPT4o-mini, matching fine-tuning and evaluation paradigms usually gives the best accuracy, while 10k-puzzle training transfers strongly at early epochs. Across additional analyses, fine-tuning with incorrect answers can still improve performance, and puzzle and model features distinguish consistent solving under perturbation.

E.3.2 GPT4O-MINI

  • Using the same training and evaluation paradigm usually gives GPT4o-mini its best accuracy on both training and test data.Source: §E.3.2 · p. 32
  • In the shown CoT-FT failure case, GPT4o-mini loops through assumptions and contradictions without reaching a conclusion.Source: Figure 32 · p. 33, §E.3.2 · p. 34
  • GPT4o-mini reaches high accuracy across tasks at early epochs when trained on 10k 8-person puzzles, but CoT fine-tuning has poor test accuracy on 2-person puzzles.Source: §E.3.2 · p. 32, Figure 33 · p. 33

E.4.1 LLAMA3-8B

E.4.2 GPT4O-MINI

E.5 Probing

E.6 Distinguishing Memorization from Reasoning

  • Length-related puzzle features help distinguish consistent solving under perturbations, and fine-tuned Llama3-8B embeddings provide more distinguishable signals for memorized samples than base-model embeddings.For GPT4o-mini, test AUC is generally higher after CoT fine-tuning than after direct fine-tuning.Source: §E.6 · p. 40, Figure 39 · p. 41, Figure 40 · p. 42
Loading 2410.23123v2…