PaperlayerView exact revision on arXiv

Source-linked AI summary

Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting

Lizhe Zhang, Wentao Chen, Li Zhong, Letian Peng, Zilong Wang, Jingbo Shang

arXiv:2503.02296v2cs.AI

TL;DR

Overlap-based evaluations can mistake benign code reuse for harmful memorization and do not test whether models solve semantically changed tasks. The paper introduces code rewriting and MRI to assess that behavior, finding that harmful memorization generally decreases with scale on simpler tasks, while SFT raises accuracy and amplifies memorization and PPO offers a more balanced trade-off.

  • Problem

    Existing overlap-based evaluations conflate benign code reuse with harmful recall and do not assess task correctness under semantic variation.

    Source: §1 · p. 1

  • Method

    The paper rewrites solutions into semantically different, similarly difficult tasks and uses MRI to combine answer similarity to the original solution with the performance drop under rewriting.

    Source: §1 · pp. 2–3

  • Results

    On MBPP+ and BIGCODEBENCH, harmful memorization generally decreases with model scale on simpler tasks but persists on harder ones; SFT raises accuracy while inflating MRI, whereas PPO offers a better risk–accuracy trade-off.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The findings frame fine-tuning choice as a trade-off between prioritizing accuracy and managing memorization risk.

    Source: §5.2 · p. 8

    The paper uses “memorization” specifically for high similarity to the original solution together with performance drops under semantically altered code rewriting.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language models (LLMs) have recently demonstrated exceptional code generation capabilities. However, there is a growing debate whether LLMs are mostly doing memorization (i.e., replicating or reusing large parts of their training data) versus generalization (i.e., beyond training data). Existing evaluations largely proxy memorization with surface/structural similarity, thereby conflating benign reuse of repeated code with harmful recall and neglecting task correctness under semantic variation. We define harmful memorization behaviorally as failure at high similarity and introduce a semantic perturbation code rewriting, which rewrites a semantically different answer at a similar difficulty level for a given coding task, then reverse-engineers a novel coding task. We further propose Memorization Risk Index (MRI), a normalized score that combines two signals: (i) how similar the model's answer for the rewritten task is to the original ground-truth solution, and (ii) how much performance drops from the original task to its rewritten counterpart. MRI is high only when both conditions hold -- when the model outputs similar code but fails the perturbed task -- thereby capturing harmful memorization rather than benign reuse of repeated code. Empirical evaluations on code generation benchmarks MBPP+ and BigCodeBench reveal that (1) memorization does not increase with larger models and in many cases alleviates as they scale; (2) supervised fine-tuning (SFT) improves accuracy while introduces memorization; (3) reinforcement learning with proximal policy optimization (PPO) achieves a more balanced trade-off between memorization and generalization.

1 INTRODUCTION

The paper argues that overlap-based evaluations conflate benign code reuse with harmful memorization and miss whether models solve tasks after semantic changes. It introduces code rewriting and MRI, then compares memorization trends across model scales and post-training strategies.

1 INTRODUCTION

  • Existing evaluations use surface or structural overlap as a proxy for memorization, conflating benign repeated-code reuse with harmful recall and not testing success under semantic variation [Yang et al., 2024; Riddell et al., 2024; Dong et al., 2024].Source: §1 · p. 1
  • MRI combines answer similarity to the original solution with the performance drop on the rewritten task, capturing harmful memorization when both similarity and failure are present.Source: §1 · p. 2
  • The paper uses “memorization” specifically for high similarity to the original solution alongside performance drops under semantically altered code rewriting.Source: §1 · p. 2
  • Across MBPP+ and BIGCODEBENCH, harmful memorization generally falls with scale on simpler tasks but persists on harder ones; SFT raises accuracy while amplifying it, whereas PPO offers a more balanced risk–accuracy trade-off.The evaluation covers introductory MBPP+ problems and more difficult BIGCODEBENCH tasks.Source: §1 · pp. 2–3
  • Code rewriting creates a semantically different answer at similar difficulty, then reverse-engineers a novel coding question; a judge checks the rewritten task’s quality.Source: Figure 1 · p. 2, §1 · p. 3

2 RELATED WORK

Prior work highlights LLM advances in code generation while noting that static benchmark performance does not assess responses to prompt variation. Memorization research has examined regurgitation and contamination, but overlap alone does not distinguish harmful recall from valid reuse.

2.1 CODE GENERATION WITH LLMS

  • Code-focused LLMs have advanced natural-language-to-code generation, while static benchmark evaluations can overlook memorization under prompt variations.Source: §2.1 · p. 3

2.2 MEMORIZATION IN CODE GENERATION

3 METHODOLOGY

The methodology creates semantically altered coding tasks while preserving surface format, then measures harmful memorization by combining solution similarity with performance loss. It also uses meaning-preserving perturbations and fine-tuning procedures to study robustness and memorization.

3.1 CODE REWRITING

  • Code rewriting changes one logic while preserving the function signature, then derives a semantically different prompt with similar syntax.GPT-5 checks code-prompt match and difficulty alignment; two Python programmers review a random 10% sample.Source: §3.1 · pp. 3–4

3.1.1 METRIC-MEMORIZATION RISK INDEX

  • MRI multiplies similarity between the rewritten-task output and original solution by relative accuracy drop, identifying high risk when similar code fails.Its multiplicative design requires both high similarity and high performance drop.Source: §3.1.1 · p. 4

3.2 MUTATION AND PARAPHRASE

  • Mutation adds textual noise, while paraphrase rewords the prompt; both preserve the original task semantics.Mutation examples include word scrambling, random capitalization, and character noise.Source: §3.2 · p. 5

3.2.1 METRIC—ROBUSTNESS RELATIVE ACCURACY DROP

  • Robustness Relative Accuracy Drop tracks solved-task retention after meaning-preserving changes; zero indicates no accuracy loss, while larger values indicate degradation.Source: §3.2.1 · p. 5

3.3 FINE-TUNING METHODS

3.3.1 SUPERVISED FINE-TUNING

  • SFT trains coding prompts against code solutions and evaluates the checkpoint where validation loss begins rising to distinguish memorization from overfitting.Source: §3.3.1 · p. 5

3.3.2 REINFORCEMENT LEARNING

  • PPO uses code correctness as its reward and selects the checkpoint with the highest validation reward.Source: §3.3.2 · p. 6

4 EXPERIMENT SETUP

Experiments use MBPP+ and BigCodeBench, with a train/test split for fine-tuning and broader model-scale evaluations across Qwen and Llama model series.

4.1 DATASETS

  • MBPP+ has 378 tasks and BigCodeBench 1140; fine-tuned models use a 4:1 train/test split, with MBPP+ test estimates potentially imprecise (n = 78).Models without fine-tuning are evaluated on the complete benchmark sets.Source: §4.1 · p. 6

4.2 MODELS

  • Scale-up experiments cover Qwen-2.5, Qwen-2.5-Coder, Llama-3.1, and Llama-4 series; fine-tuning uses Qwen-2.5-7B and Qwen-2.5-Coder-7B.Training and inference use four NVIDIA A100 GPUs, with approximately 40 GPU hours.Source: §4.2 · p. 6

5 RESULT ANALYSIS

Across Qwen and Llama families, scaling is associated with lower memorization risk most clearly on MBPP+, while risk persists more on BIGCODEBENCH. SFT raises accuracy alongside MRI, whereas PPO offers a better risk–accuracy trade-off; semantic-altering rewrites produce larger accuracy drops than semantic-preserving perturbations.

5.1 MEMORIZATION ANALYSIS ON INSTRUCT MODELS

  • On MBPP+, Qwen-Instruct MRI falls from 0.0722 at 0.5B to 0.0000 at 32B, while on BIGCODEBENCH it drops from 0.1740 to 0.0841 at 14B before rising to 0.1143 at 32B.Qwen-Coder MRI also declines on BIGCODEBENCH from 0.1778 to 0.1178 between 0.5B and 32B; scaling effects are milder on the harder benchmark.Source: §5.1 · p. 6, Figure 2 · p. 7
  • Llama results show task-dependent risk: MRI is near zero on MBPP+ but remains non-zero on BIGCODEBENCH, where RADrew stays high at comparable similarity levels.For Llama-3.1, MRI changes from 0.0020 to 0.0000 on MBPP+ and from 0.0947 to 0.0884 on BIGCODEBENCH; the table covers Llama-3.1 and Llama-4 instruct models.Source: §5.1 · p. 7, Table 1 · p. 7

5.1.1 ADDITIONAL FINDINGS

  • On harder BIGCODEBENCH tasks, memorization risk persists even at larger scales; Qwen-2.5-32B-Instruct has MRI 0.1178.By contrast, Qwen-2.5-Instruct MRI falls from 0.0722 at 0.5B to effectively zero at 32B on introductory MBPP+ tasks.Source: §5.1.1 · p. 8
  • On BIGCODEBENCH, Qwen-Coder has higher Sim(Trew) than Qwen-Instruct, 0.3237 ± 0.0086 versus 0.2670 ± 0.0268, but only slightly higher MRI, 0.1367 ± 0.0224 versus 0.1142 ± 0.0247.The reported means and standard deviations are over 6 seeds.Source: §5.1.1 · p. 8

5.2 IMPACT OF FINE-TUNING STRATEGIES ON MEMORIZATION

  • On BIGCODEBENCH, SFT raises Qwen-2.5-7B accuracy from 0.3158 to 0.3772 while MRI rises from 0.0799 to 0.1747; PPO yields accuracy 0.3509 and MRI 0.0795.For the coder variant, SFT raises accuracy from 0.3684 to 0.4079 and MRI from 0.1392 to 0.1921; PPO yields accuracy 0.3728 and MRI 0.1336.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 ROBUSTNESS TO SEMANTIC-PRESERVING PERTURBATIONS

  • On BIGCODEBENCH, mutation has RADmut = 0.20 ± 0.12 and paraphrase RADpar = 0.06 ± 0.04, versus rewriting RADrew = 0.46 ± 0.09.On MBPP+, RADmut = 0.10±0.08 and RADrew = 0.10±0.09, while RADpar = 0.01 ± 0.01.Source: §5.3 · p. 9, Table 2 · p. 9
  • Mutation accuracy drops decrease with model size on both benchmarks, while paraphrase effects remain near zero; on BIGCODEBENCH, Qwen-2.5-Coder averages RADmut = 0.25 ± 0.12 versus 0.20±0.13 for instruction models.Llama-3.1 and Llama-4 have lower BIGCODEBENCH RADmut values of 0.1050 and 0.1206, respectively; on MBPP+, Llama-4 has RADmut = 0.0578.Source: §5.3 · p. 9

6 CONCLUSION AND FUTURE WORKS

The conclusion frames harmful memorization as high similarity to an original solution combined with performance drops on semantically modified tasks. Experiments show scaling-related reductions particularly on simpler tasks, higher MRI with SFT, and a better risk–accuracy trade-off with PPO; future work includes mitigation and transferability beyond code.

6 CONCLUSION AND FUTURE WORKS

  • The paper’s conclusion identifies further research on reducing memorization’s impact and testing whether its code-generation metrics apply to domains such as mathematical reasoning.The reported findings include lower harmful memorization with scale on simpler tasks, persistence on harder tasks, and a better risk–accuracy trade-off for PPO than SFT.Source: §6 · p. 10

ETHICS STATEMENT

The appendix describes ethical safeguards, GPT-5’s limited assistance, and the prompts and validation procedures used to generate and assess task variants.

ETHICS STATEMENT

  • The datasets are intended for evaluating memorization in LLM code generation and educational or scientific use, while harmful or malicious use is discouraged.The authors say they remove harmful or offensive content from variant datasets to mitigate potential risks.Source: ETHICS STATEMENT · p. 10

A USE OF LARGE LANGUAGE MODELS (LLMS)

  • GPT-5 assisted with task generation and manuscript language; the authors say methodological design, filtering, integration, research ideas, and analysis remained theirs.They state that research design, analysis, and interpretation did not otherwise depend on LLM assistance.Source: A USE OF LARGE LANGUAGE MODELS (LLMS) · p. 14

B PROMPTS FOR TASK GENERATION

  • The appendix provides the full GPT-5 prompts used to generate mutation, paraphrase, and code-rewriting variants.The prompts include the system and user messages passed to the API.Source: B PROMPTS FOR TASK GENERATION · p. 14

B.1 CODE-REWRITING EVOLUTION

  • Code rewriting changes one core logic or structure while keeping task difficulty equivalent and the prompt syntactically similar to the original.The rewritten prompt must reflect the code change and retain the original input signature; the entry point may be updated when the old one would mislead.Source: §B.1 · pp. 14–15

B.2 CODE-REWRITING EVOLUTION LLM JUDGE

  • The LLM judge checks whether rewritten code matches its prompt and whether the evolved task has difficulty comparable to the original.Its criteria include prompt-code alignment, input/output consistency, algorithmic thinking, complexity, and programming concepts.Source: §B.2 · pp. 15–16

B.3 MUTATION EVOLUTION

  • Mutation adds word scrambling, random capitalization, and character noise while perturbing the prompt’s surface text.The number of operations scales with prompt length at approximately four operations per five words, with at least one of each type.Source: §B.3 · p. 16

B.4 PARAPHRASING EVOLUTION

  • Paraphrasing changes wording while preserving task meaning, requirements, input/output specifications, code-related tokens, and structural cues.The instructions allow synonym substitution, sentence reordering, and clause rephrasing without adding or dropping information.Source: §B.4 · p. 17

C EXAMPLES OF CLEARER PARAPHRASED PROMPTS

  • The examples illustrate paraphrases that restate prompts for reversing words, computing Jacobsthal numbers, and retrieving the smallest k tuple elements.The authors also report validating rewritten test cases and solutions, using official MBPP+ inputs and generated expected outputs, and following Zhuo et al. (2024) guidelines for BigCodeBench.Source: C EXAMPLES OF CLEARER PARAPHRASED… · pp. 17–18, §B.4 · p. 17

D EXAMPLES OF REGRESSED TASKS

The appendix presents sampled cases where Qwen2.5-Coder-32B-Instruct passed original tasks but failed rewritten versions, alongside examples of harmless implementation similarity.

D EXAMPLES OF REGRESSED TASKS

  • The case studies report that performance regression was not caused by higher difficulty on rewritten tasks.The authors randomly selected five passed-original/failed-rewritten tasks from each benchmark and provide prompts, solutions, and GPT-5 alignment and difficulty analyses.Source: D EXAMPLES OF REGRESSED TASKS · p. 18
  • Figures 4–8 show five MBPP+ task examples, and Figure 9 shows one BigCodeBench example, all where Qwen2.5-Coder-32B-Instruct passed the original task but failed in code rewriting.Source: Figure 4 · p. 18, Figure 5 · p. 19, Figure 6 · p. 19

E HARMLESS INTERNAL LOGIC MODIFICATIONS

  • Two Llama3.1-8B-Instruct cases preserve function and variable names while changing internal implementation, illustrating harmless syntactic similarity.The examples include replacing a list comprehension with a loop and replacing a reverse-slice search with binary search.Source: E HARMLESS INTERNAL LOGIC MODIFICATIONS · p. 24

E.1 TASK MBPP/18: LIST COMPREHENSION ¡-¿ EXPLICIT LOOP

The examples show Llama3.1-8B-Instruct retaining names while changing implementation logic in two MBPP tasks.

E.1 TASK MBPP/18: LIST COMPREHENSION ¡-¿ EXPLICIT LOOP

  • In MBPP/18, the rewritten answer replaces a list comprehension with an explicit loop while preserving the function name and variables.The rewritten loop traverses the input in reverse, accumulates retained characters, then reverses the result before joining it.Source: §E.1 · p. 24

E.2 TASK MBPP/793: REVERSE-INDEX LOOKUP BINARY SEARCH

  • In MBPP/793, the rewritten answer replaces reverse-slice lookup with iterative binary search while retaining last, arr, and x.The original answer uses a reverse slice and index lookup; the rewritten answer searches with low and high bounds.Source: §E.2 · p. 24

F TRAINING DETAILS REGARDING SFT/RL

The appendix details the SFT and PPO training setup, including prompting, checkpoint selection, and optimization settings.

F.1 FINE-TUNING DETAILS

  • Training used Verl with PyTorch FSDP on one machine with two GPUs, and PPO rollouts used VLLM while optimization used AdamW.The dataset followed Verl’s format, used a 4:1 train/test split, and paired problem descriptions with code solutions as targets.Source: §F.1 · pp. 24–25

F.1.1 SUPERVISED FINE-TUNING (SFT)

  • SFT ran for 20 epochs with a default learning rate of 1 × 10^-5, and the selected checkpoint was taken before observed overfitting.The authors also used the epoch-20 checkpoint as the fully overfitting comparison point.Source: §F.1.1 · p. 25

F.1.2 PROXIMAL POLICY OPTIMIZATION (PPO)

  • PPO used a binary reward for passing all tests and selected the checkpoint with the highest test reward.The critic and actor learning rates were 1 × 10^-5 and 1 × 10^-6, respectively.Source: §F.1.2 · p. 26

G EVOLVED-TASK GENERATION (GPT-5)

Evolved-task generation used the gpt-5-2025-08-07 API, with the prompt template in Appendix B. Generation used default temperature and top-p, a 1080-token maximum, and regex clean-up; the estimated cost was approximately 450 USD for one round of each evolution type across both benchmarks.

G EVOLVED-TASK GENERATION (GPT-5)

Loading 2503.02296v2…