PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory

Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, James Zou

arXiv:2504.07952v1cs.LGcs.CL

TL;DR

Language models usually handle queries independently without retaining prior successes or mistakes. Dynamic Cheatsheet adds a selectively curated external memory at inference time, and experiments report substantial gains across reasoning and knowledge tasks, though smaller or specialized models show limited or inconsistent improvements.

  • Problem

    Current language models typically retain no memory of prior questions, successes, or mistakes during inference, repeatedly re-deriving insights and errors.

  • Method

    Dynamic Cheatsheet lets black-box LLMs iteratively store and refine compact heuristics, strategies, solution sketches, and code snippets in external memory without modifying model weights.

  • Results

    DC improved performance across challenging reasoning and knowledge benchmarks, including Claude 3.5 Sonnet’s AIME 2024 accuracy rising from 23% to 50% and GPT-4o’s Game of 24 success rising from 10% to 99%.

  • Takeaways & Limitations

    DC offers a practical framework for continuous test-time learning, with benefits linked to model capacity, task similarity, retrieval precision, and offloading repeated computations to external tools.

  • Takeaways & Limitations

    Smaller and some specialized models show limited or inconsistent gains because they may generate unreliable strategies or struggle to retrieve and interpret stored heuristics.

Abstract

from arXiv · show

Despite their impressive performance on complex tasks, current language models (LMs) typically operate in a vacuum: Each input query is processed separately, without retaining insights from previous attempts. Here, we present Dynamic Cheatsheet (DC), a lightweight framework that endows a black-box LM with a persistent, evolving memory. Rather than repeatedly re-discovering or re-committing the same solutions and mistakes, DC enables models to store and reuse accumulated strategies, code snippets, and general problem-solving insights at inference time. This test-time learning enhances performance substantially across a range of tasks without needing explicit ground-truth labels or human feedback. Leveraging DC, Claude 3.5 Sonnet's accuracy more than doubled on AIME math exams once it began retaining algebraic insights across questions. Similarly, GPT-4o's success rate on Game of 24 increased from 10% to 99% after the model discovered and reused a Python-based solution. In tasks prone to arithmetic mistakes, such as balancing equations, DC enabled GPT-4o and Claude to reach near-perfect accuracy by recalling previously validated code, whereas their baselines stagnated around 50%. Beyond arithmetic challenges, DC yields notable accuracy gains on knowledge-demanding tasks. Claude achieved a 9% improvement in GPQA-Diamond and an 8% boost on MMLU-Pro problems. Crucially, DC's memory is self-curated, focusing on concise, transferable snippets rather than entire transcript. Unlike finetuning or static retrieval methods, DC adapts LMs' problem-solving skills on the fly, without modifying their underlying parameters. Overall, our findings present DC as a promising approach for augmenting LMs with persistent memory, bridging the divide between isolated inference events and the cumulative, experience-driven learning characteristic of human cognition.

1. Introduction

Dynamic Cheatsheet gives black-box LLMs a persistent, selectively curated memory for reusing strategies, code, and insights across inference queries. Across reasoning and knowledge tasks, it improves performance while smaller models benefit less and curated memory avoids full transcript growth.

  • Current LLMs typically process each query independently, retaining no explicit or implicit memory of prior questions, successes, or mistakes.
  • Dynamic Cheatsheet lets black-box LLMs store, discard, and refine compact reusable strategies, solution sketches, and code snippets at inference time without labels or human feedback.
  • DC retrieves prior insights before generating a solution and curates the memory afterward, preserving useful approaches while revising or pruning errors.
  • 23% to 50% accuracy on AIME 2024 more than doubled Claude 3.5 Sonnet’s baseline score through retained algebraic and combinatorial insights.
  • 10% to 99% success on Game of 24 shows GPT-4o benefiting from retrieving a stored Python brute-force solver instead of manual arithmetic.
  • Smaller models benefit less because they produce fewer reliable strategies and struggle to refine stored content, while DC’s curation favors concise transferable knowledge over full transcripts.

2. Dynamic Cheatsheet (DC) Methodology

Dynamic Cheatsheet gives black-box LLMs an evolving external memory that stores reusable solutions and refines them during inference. Its variants combine generation, curation, and retrieval while contrasting curated memory with one-off, empty-memory, full-history, and retrieval-only baselines.

  • Core framework: DC maintains a non-parametric memory that tracks successes and failures without modifying the LLM’s underlying parameters.It stores heuristics, strategies, and short textual artifacts for future inference.
  • Solution generation: The generator receives a new query and current memory, producing a candidate solution conditioned on previously stored solutions, insights, techniques, or heuristics.The workflow is represented as ỹ_i = Gen(x_i, M_i).
  • Memory curation: The curator updates memory by assessing answer usefulness and generalizability, refining or removing entries, and consolidating the contents for clarity and compactness.Curation does not rely on ground-truth labels and may be performed by one model or multiple tools and models.
  • DC-Cu: DC-Cu generates from the current memory before cumulatively expanding and refining memory, but it does not include retrieval.Its post-query update prevents incorporating current-query insights during the response and omits direct reuse of historical input-output pairs unless retained.
  • DC-RS: DC-RS retrieves the top-k similar prior inputs and outputs, gives them with recent memory to the curator, and then generates the response from the updated memory.This changes the update order and adds retrieval of prior cases.
  • Baselines: The evaluation compares DC variants with baseline prompting, empty memory, full-history appending, and dynamic retrieval without curation.Full-history appending can exceed context limits, while dynamic retrieval pastes similar interactions without abstracting generalized solutions.

3. Experimental Setup

The experiments target challenging algorithmic, mathematical, logical, and domain-specific reasoning tasks where iterative memory refinement may improve performance. Models use structured answer parsing and task-specific accuracy metrics across AIME, GPQA-Diamond, Game of 24, Math Equation Balancer, and MMLU-Pro subsets.

  • Task selection: The study prioritizes tasks requiring multi-step reasoning, heuristic search, strategic adaptation, and cumulative learning rather than near-saturated benchmarks.The selected datasets are intended to stress-test refinement of reasoning over time.
  • Tasks and datasets: The evaluation includes AIME 2024, AIME 2025, and AIME 2020–2024 exam questions covering algebra, combinatorics, number theory, geometry, and probability.AIME 2024 and AIME 2025 each contain 30 questions, while AIME 2020–2024 contains 133 questions.
  • Tasks and datasets: GPQA-Diamond contains 198 expert-validated natural-science questions spanning biology, chemistry, and physics.The questions are designed to test difficult, multi-hop reasoning that non-experts often miss.
  • Tasks and datasets: Game of 24 uses 100 four-number arithmetic examples requiring an expression that evaluates to 24 using each number exactly once.The task emphasizes systematic search, strategic reasoning, and pattern recognition.
  • Tasks and datasets: Math Equation Balancer contains 250 expressions requiring operators to be inserted so each equation becomes valid.The task emphasizes sequential operator placement in elementary arithmetic.
  • Tasks and datasets: MMLU-Pro Engineering and Physics are multiple-choice subsets sampled at 250 questions each from larger professional-level domains.The original datasets contain 969 engineering and 1,299 physics questions.
  • Models and evaluation: The evaluation covers GPT-4o, Claude 3.5 Sonnet, GPT-4o-mini, Claude 3.5 Haiku, and DeepSeek R1.Final answers are wrapped in XML-style tags for consistent parsing before task-specific accuracy evaluation.
  • Accuracy metrics: Soft Match ignores minor formatting differences for GPQA-Diamond and MMLU-Pro, whereas Functionally Correct checks task constraints despite slight numeral or formatting differences for Game of 24, Math Equation Balancer, and AIME.The metrics are tailored to each dataset’s answer requirements.

4. Main Results

Dynamic Cheatsheet substantially improves reasoning performance by retaining and reusing solution strategies, especially when models discover reliable algorithmic procedures. Gains span mathematical, knowledge-intensive, and tool-oriented tasks, but depend on model capability and curated memory.

  • Game of 24: GPT-4o’s Game of 24 accuracy increased from 10% baseline to 99% under DC-RS through retrieval and reuse of a Python brute-force solution.With empty memory, DC-∅ reached 19%, highlighting the contribution of storing and reusing solutions.
  • Knowledge-intensive tasks: Claude 3.5 Sonnet improved on GPQA-Diamond from 59.6% to 68.7% under DC-RS, whereas GPT-4o increased only from 57.1% to 58.1%.The results indicate that retrieval and curation benefits vary across models and tasks.
  • Math Equation Balancer: Claude 3.5 Sonnet’s Math Equation Balancer accuracy rose from 44.8% to 98–100%, while GPT-4o improved from 50.0% to 99–100% with DC.Both models repeatedly retrieved algorithmic or Python-based balancing routines after discovering them.
  • Knowledge-intensive tasks: Claude gained up to 8.0% in MMLU-Pro Physics, rising from 74% to 82%, as compact engineering and physics reference guides were stored and retrieved.GPT-4o instead showed slight decreases on these tasks.
  • Memory curation: Selective memory curation outperformed full-history retention, which can overwhelm context with unfiltered dialogue and increase inference costs over time.DC retains concise, transferable problem-solving tips and code snippets rather than entire transcripts.

GENERAL META-REASONING STRATEGIES <memory_item>

The curated memories capture general problem-solving strategies and can transfer across structurally similar tasks. However, smaller models often gain less because they generate fewer reliable strategies and struggle with retrieval and refinement.

  • GENERAL META-REASONING STRATEGIES: Curated AIME memories encode reusable strategies such as identifying patterns, decomposing problems, verifying examples, and choosing computational approaches.The framework explicitly includes specialized guidance for grid and combinatorial problems.
  • Model-scale limitations: GPT-4o-mini showed slight declines under some DC variants and remained largely stagnant on GPQA-Diamond, indicating difficulty leveraging stored information effectively.The limitation is not universal: some smaller-model settings still provide utility from retrieval-based adaptation.
  • Model-scale limitations: Smaller models often populate memory with incorrect or partial attempts, causing iterative refinement to stall.Their weaker long-context understanding and retrieval can also produce irrelevant memory use or misapplied stored knowledge.
  • GENERAL META-REASONING STRATEGIES: DC is most effective when test examples share structural similarities, allowing discovered geometry, combinatorics, or algorithmic strategies to transfer across subsequent questions.Related questions presented early may accelerate test-time learning.

5. Additional Analyses and Discussions

DC reduces repeated reasoning overhead by reusing curated strategies and code, but its benefits depend on model capability, retrieval quality, and reliable memory maintenance. Additional analyses also identify limitations involving sequential processing, truncated updates, and error propagation.

  • Reasoning and information efficiency: DC reduces reasoning overhead and token usage by reusing established techniques across subsequent queries.The initial discovery and curation of a robust approach remain non-trivial.
  • Comparison with majority voting: MV produced no improvement over single-shot inference on AIME 2024 or AIME 2025, whereas DC improved performance.MV remained at 23.3% on AIME 2024 and 6.7% on AIME 2025.
  • Clustering of errors and corrections: Faulty heuristics can be amplified across related queries, making careful curation and pruning necessary.Errors and corrections often cluster in latent embedding space, enabling transfer to nearby queries.
  • Retrieval bottlenecks and noise: Poor retrieval choices can reduce performance, as observed in GPT-4o’s occasional dip on GPQA-Diamond.The analysis motivates denser search and stronger ranking methods.
  • Efficiency and memory maintenance: Sequential memory curation can challenge large-scale parallel or batch inference, while truncated updates may degrade stored heuristics over time.The latter occurs when models abbreviate or reference prior memory instead of explicitly rewriting it.
  • Model capability and transferability: DC can amplify capable models but provides limited gains when smaller models generate or interpret too few reliable strategies.This limitation was observed for GPT-4o-mini and similarly for some R1 models.

A.1. Test-time learning (online learning)

Test-time learning incorporates information encountered during inference without conventional offline fine-tuning. DC applies this idea to black-box LLMs by updating external memory rather than model parameters.

  • Test-time learning: Test-time learning updates a model’s predictions using information seen during inference without full-scale offline fine-tuning.Earlier approaches often refit or reweight parameters for each test instance or batch.
  • Motivation: Direct test-time weight updates are costly and difficult for black-box language-model APIs.This motivates parameter-free adaptation through structural changes to model inputs.
  • Dynamic Cheatsheet: DC updates an external memory of newly found solutions, errors, and strategies without touching model weights.The approach does not require a new training loop for each batch or scenario.

A.2. Test-time compute and reasoning

Inference-time reasoning methods can improve individual answers but usually impose additional per-instance computation and do not preserve their benefits across later queries. DC instead stores reusable solutions so computational gains can transfer across interactions.

  • Test-time compute: Inference-time reasoning strategies improve LLM capability but increase computational overhead for each test instance.Examples include chain-of-thought, tree-of-thought, and majority-vote sampling.
  • Persistence: Most inference-time expansions are ephemeral, so later tasks typically do not benefit from earlier computation without manual prompt sharing.DC targets repeated overhead across multiple test instances.
  • Tool usage: External Python, symbolic, and specialized tools can offload complex computations from language models.Under DC, a discovered Python brute-force algorithm can be stored and repeatedly retrieved for arithmetic puzzles.
  • Persistent reuse: DC treats consecutive tasks as opportunities to refine a persistent external store of reusable lessons, heuristics, and code blocks.This distinguishes it from methods that combine multiple inference passes for only one question.

A.3. Memory-augmented generation and reasoning

Memory-augmented generation commonly retrieves external reference knowledge, while newer methods also retain reasoning traces or thought templates. DC emphasizes selective, external, training-free storage of reusable insights and heuristics.

  • External memory: Modern retrieval-augmented LLMs generally consult external document corpora for reference knowledge.This represents one major line of research on augmenting language models with memory.
  • Reasoning memory: Thought-Retriever and Buffer-of-Thoughts retain reasoning processes or distilled thought templates for analogous future problems.These approaches extend memory beyond ordinary reference documents.
  • Dynamic Cheatsheet: DC selectively stores relevant insights and heuristics rather than accumulating full transcripts or ephemeral reasoning expansions.The framework remains fully external and training-free, supporting plug-and-play use.

B.1. Performance Comparison of Baseline and DC-RS Approaches

Figures 8 and 9 compare baseline prompting with Dynamic Cheatsheet using Retrieval & Synthesis (DC-RS) for Claude 3.5 Sonnet and GPT-4o.

  • Figure 8 reports Claude 3.5 Sonnet’s overall performance under baseline prompting and DC-RS.
  • Together, the figures provide model-specific comparisons between minimal-instruction inference and retrieval-and-synthesis memory augmentation.
  • Figure 9 reports GPT-4o’s overall performance under baseline prompting and DC-RS.

B.2. Clustering of Errors and Corrections

The GPQA-Diamond question embeddings show that correct and incorrect answers often cluster in latent space. Dynamic Cheatsheet can transfer strategies within clusters, but uncurated memories may also propagate erroneous heuristics.

  • Correct and incorrect answers often cluster in the latent embedding space of GPQA-Diamond questions.
  • Dynamic Cheatsheet can transfer learned strategies among questions within these embedding clusters.
  • Erroneous heuristics may also spread without careful memory curation, requiring refinement and strategy verification.

B.3. Evolution of Memory Content under Dynamic Cheatsheet

Dynamic Cheatsheet memory changes across inference while its prompts support structured solving, tool use, and selective retention of reusable strategies. Under DC-RS, the curator maintains a concise, evolving repository, while DC-Cu shows somewhat greater late-stage fluctuations.

  • Memory evolution: DC-Cu and DC-RS memory states become highly stable after the first few Game of 24 iterations.Stability is measured by word-level LCS similarity between consecutive memory states.
  • Memory evolution: DC-Cu experiences slightly greater memory fluctuations than DC-RS during the second half of inference.
  • Prompt design: The baseline prompt asks for an answer with minimal instructions and no structured reasoning or tool-use encouragement.
  • Prompt design: Non-baseline approaches use structured instructions for solution development, strategy selection, Python code generation, and execution.The same structured generator prompt is used for DC-∅, DR, FH, DC-Cu, and DC-RS.
  • Memory curation: The cheatsheet is updated after each input by synthesizing lessons, insights, tricks, and errors from prior problems.
  • Memory curation: The DC-RS curator selectively retains, refines, and organizes strategies, code snippets, and heuristics into concise, generalizable memory.Its stated goal is to improve long-term performance while filtering redundant or trivial information.
Loading 2504.07952v1…