PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Experience for Continual LLM Improvement

Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie, Shen Yan

arXiv:2608.18027v1cs.CL

TL;DR

Conventional LLM evaluations treat inference as isolated, overlooking whether models can improve from experience during interaction. This paper introduces Chain-of-Experience, an iterative feedback framework, and finds consistent cross-domain gains, with self-feedback averaging 62.9% to 71.0% over existing test-time scaling approaches.

  • Problem

    Conventional LLMs treat inference as isolated events and provide limited evidence about continuous improvement through test-time interaction and feedback.

  • Method

    Chain-of-Experience has models iteratively solve problems while reusing interaction histories and feedback from models, executors, correctness signals, or none.

  • Results

    Across math, coding, and knowledge tasks, feedback-based CoE consistently improves performance, with self-feedback averaging 62.9% to 71.0% over existing test-time scaling approaches.

  • Takeaways & Limitations

    CoE demonstrates feedback-driven test-time learning across tasks, while complementary feedback channels provide additional gains whose benefits depend on task difficulty.

  • Takeaways & Limitations

    The study does not update model parameters, so its improvements reflect contextual reuse of experience rather than persistent internal learning.

Abstract

from arXiv · show

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

1 Introduction

Chain-of-Experience (CoE) frames test-time improvement as iterative problem-solving with accumulated experience and feedback, addressing the limitation that deployed LLMs treat inference as isolated events. Across math, coding, and knowledge tasks, experiments with eight state-of-the-art models show that feedback-based CoE consistently improves performance and can substantially outperform test-time scaling approaches.

  • 1 Introduction: Contemporary LLMs are deployed in a fixed state, treating each inference as an isolated event and ignoring accumulated experience.The introduction contrasts this behavior with humans’ continuous learning from successes and failures.
  • 1 Introduction: Existing search-based methods often consolidate temporary model experience into one answer and discard it, requiring models to restart each problem-solving process.These methods include heuristic strategies and majority voting that use verifiers over parallel candidate generations.
  • 1 Introduction: Chain-of-Experience (CoE) uses iterative problem-solving with feedback to study how LLMs improve during test time through accumulated interaction experience.The framework explores four feedback types: none, model, executor for code, and correctness for general tasks.
  • 1 Introduction: Eight state-of-the-art models, including GPT-5, o3, Gemini-2.5 Pro, and Claude-4.5 Sonnet, are evaluated across math, coding, and knowledge domains.Experiments compare four feedback settings: none, executor, model, and correctness feedback.
  • 1 Introduction: Feedback-based CoE consistently yields notable improvements and substantially outperforms test-time scaling approaches.The introduction summarizes results from extensive experiments across the four feedback types and refers to Figure 1.

2 Related Work

Related work spans training-free test-time reasoning strategies and broader approaches to learning from experience. These include Chain-of-Thought variants, search-based reasoning, verifier-based selection, and reinforcement-learning-based improvement.

  • Training-free Test-time Strategies: Chain-of-Thought prompting improves arithmetic, commonsense, and symbolic tasks, motivating a family of Chain-of-X methods.Representative variants include contrastive CoT, least-to-most prompting, Chain-of-Explanation, Chain-of-Note, and Chain-of-Knowledge.
  • Training-free Test-time Strategies: Tree-of-Thought explores multiple reasoning paths via search and outperforms CoT on planning tasks.The related work also notes the emergence of large reasoning models such as OpenAI’s o series and DeepSeek R1, alongside verifier-based methods.
  • Learning from Experiences: Reinforcement learning formalizes learning from experience through policy gradients and actor–critic methods, supporting advances in games, robotics, control, alignment, and reasoning.Recent approaches include post-training on online generations to improve alignment and reasoning.

3 Model Improvement via CoE

Chain-of-Experience (CoE) extends single-turn generation into iterative problem solving, where models condition each response on prior attempts and feedback. The framework studies four feedback settings—none, execution, model, and correctness—to shape experience across test-time iterations.

  • CoE formulation: CoE models sequential decision-making: each response depends on the question, prior responses, and corresponding environmental feedback.Each experience consists of an action-feedback pair (a_i, f_i).
  • CoE formulation: The iterative loop has models generate actions conditioned on accumulated experience, receive environmental feedback, and use the resulting experience for subsequent actions.The environment may be a model simulation or coding environment.
  • Feedback mechanisms: The study compares four feedback types: none, execution, model, and correctness feedback.These span implicit absence of evaluation to explicit signals from execution environments, auxiliary critics, or domain-specific verifiers.
  • Feedback mechanisms: Execution feedback supplies traces, errors, runtime logs, or test outcomes produced by running a response in an executable environment.This feedback is defined as the environment’s response to the question and model-generated answer.
  • Feedback mechanisms: Correctness feedback provides binary oracle-like evaluation, with f_i = 1{a_i is correct} ∈ {0, 1}, serving as a high-signal reference setting.The paper notes that such verification can be costly or unavailable in real-world settings.

4 Experiments

Across math, coding, and knowledge benchmarks, Chain-of-Experience with feedback consistently improves reasoning-model performance over feedback-free and conventional baselines. Feedback also improves efficiency, while stronger zero-shot models exhibit greater test-time learning gains.

  • Experimental setup: Experiments cover six benchmarks spanning math, coding, and knowledge tasks: AIME 2025, OmniMath, LiveCodebench (V6), LiveBench (Code), EvaLearn, and GPQA Diamond.
  • Performance: Across six benchmarks, self feedback and executor/correctness feedback improve eight reasoning models by 5.6% and 11.1%, respectively, over no-feedback counterparts.
  • Performance: ICL, ACE, and DC average 62.1%, 64.0%, and 62.7%, below the without baseline at 66.8%, whereas self feedback reaches 71.0% and the best feedback signal reaches 79.3%.
  • Efficiency: Feedback-based variants generally achieve higher accuracy at lower API cost than no-feedback baselines, with self feedback a cost-effective compromise on the two coding tasks.Feedback iterations shift compute toward iterative interaction and produce less verbose outputs rather than simply inflating prompt length.
  • Improving capability: Base performance and improving capability show a positive trend, including correlations of r = 0.97 on LiveBench (Code) and r = 0.83 on LiveCodeBench (V6).Improving capability is defined from initial zero-shot performance and peak accuracy under CoE with model self feedback.

5 Further Discussion and Conclusion

The discussion shows that Chain-of-Experience remains useful under misleading feedback, with improvements driven largely by feedback interpretation and early iterative gains. Complementary feedback channels improve performance, whereas memory-based selection does not consistently surpass pure self-feedback.

  • Learning from Spurious Feedback: 7.6% average degradation: Spurious feedback generally reduces performance, but SelMV helps models maintain or recover accuracy under adversarial signals.On GPQA-Diamond, incorrect-feedback SelMV surpasses model feedback by 0.9% (79.4% → 80.3%).
  • Analysis of Improvement Patterns: 47.7% of improvements are feedback-driven, while coding gains also reflect specification recall and model-generated feedback contributes 58.7% versus 41.1% for other sources.The analysis covers 6,630 incorrect-to-correct flips across five models and uses GPT-5 to classify improvement causes.
  • Dual Feedback CoE: Dual feedback reaches 76.7% on AIME 2025 and 81.2% on LiveBench (Code), exceeding correctness-only, model-only, or executor-only feedback.Dual feedback combines model feedback with correctness signals for math and executor signals for coding.
  • CoE with Principled Experience Selection: Pure self-feedback outperforms Dynamic CheatSheet and SimpleMem on AIME 2025, achieving 60.0% versus 50.0% (+DC) and 56.7% (+SimpleMem).The comparison uses a within-task protocol with no cross-task leakage.
  • Conclusion: CoE enables LLMs to improve during inference through iterative feedback and accumulated experience across math, coding, and knowledge tasks.The conclusion characterizes CoE methods as improving performance and efficiency and reports positive correlation between model ability and improvement capacity, with most gains emerging early.

A Detailed Experimental Setup · B Averaged Results

The setup varies reasoning effort and decoding parameters across model families. Averaged best scores over 20 iterations show that feedback improves performance across all six benchmarks, while ACE and DC perform poorly on this testing suite.

  • A Detailed Experimental Setup: OpenAI models use the default reasoning_effort settings “low” and “high” to vary reasoning level.These settings are applied as the two reasoning variants.
  • A Detailed Experimental Setup: Claude 4.5 Sonnet disables thinking mode and uses a 10,000-token thinking budget for its low and high reasoning variants.The passage specifies these settings as the respective reasoning variants.
  • A Detailed Experimental Setup: OpenAI models use default decoding with temperature 1.0.This decoding configuration applies to the OpenAI models.
  • A Detailed Experimental Setup: Gemini and Claude models use temperature 0.2 for decoding.The passage groups Gemini and Claude series under this decoding setting.
  • B Averaged Results: 20 iterations are used to compute the averaged best scores for different methods in Table 3.The reported scores are averaged best scores over the iteration process.
  • B Averaged Results: Feedback improves LLM performance on all six benchmarks.This conclusion is drawn from the averaged results in Table 3.
  • B Averaged Results: Existing self-improving algorithms ACE and DC do not perform decently on this testing suite.The passage contrasts these methods with the benefit observed from feedback.

C Evaluated Benchmarks … F.4 Analysis of Improvement Patterns

The paper evaluates continual improvement across diverse math, coding, and knowledge benchmarks using established test-time baselines and task-specific evaluation procedures. Further analyses show that models recover from spurious feedback, gain most benefits early, can benefit from external model feedback, and improve through several identifiable patterns.

  • C Evaluated Benchmarks: AIME 2025 contains 30 challenging 2025 AIME cases, while OmniMath contributes 200 sampled problems from its 4,428-problem olympiad-level collection.Both benchmarks support evaluation across mathematical difficulty levels.
  • C Evaluated Benchmarks: LiveCodeBench V6 uses 175 challenging code-generation samples, LiveBench-Code uses 128 coding samples, EvaLearn includes 648 experience-learning examples, and GPQA Diamond contains 198 expert-authored science questions.The benchmark suite spans coding, experience learning, and biology, chemistry, and physics.
  • D Baselines: The baselines include OpenAI built-in reasoning, few-shot ICL with retrieved demonstrations, Dynamic CheatSheet with distilled strategies, and ACE with evolving playbooks.These methods represent native reasoning scaling, retrieval-based demonstrations, adaptive memory, and iterative context adaptation.
  • E Benchmark Input and Answer Examples: The benchmark examples cover step-by-step mathematics, ideal-gas science questions, tree algorithms, and public/private coding test cases.These examples illustrate the input formats used across the evaluated domains.
  • E Benchmark Input and Answer Examples: Evaluation uses exact match for AIME 2025 and GPQA-Diamond, LLM-as-a-Judge for OmniMath, and a Python interpreter for LiveBench-Code and LiveCodeBench.The coding evaluations therefore use executable task feedback rather than text-only judging.
  • F Full Discussions: The full discussions analyze model behavior under spurious feedback, extended iterations, external model feedback, and identifiable improvement patterns.These analyses constitute the paper’s extended discussion section.
  • F.1 LLM with All Spurious “Correct” Feedback: Models initially drop under uniformly “correct” feedback but quickly recover and improve as they adapt to the inconsistent supervision pattern.GPT-5 mini and o4-mini show larger gains in the SelMV-augmented results.
  • F.2 Extended Rounds of Iterations: 16.7% > 2.2% on AIME 25 and 21.2% > 3.5% on OmniMath show that most gains occur within the first 20 of 50 iterations, with later improvements marginal.The pattern holds across models, indicating rapid internalization and consolidation of useful feedback signals.

F.5 Human–GPT Judge Agreement · G BrowseComp-Plus

GPT-5’s automatic attribution labels show substantial agreement with human judgments on sampled improvement trajectories, while BrowseComp-Plus reveals that self feedback can reduce performance on out-of-distribution knowledge tasks. The findings validate GPT-5 as a human-attribution proxy but highlight limits of relying solely on self feedback beyond training-data knowledge.

  • F.5 Human–GPT Judge Agreement: 100 incorrect-to-correct trajectory pairs were sampled, with 25 examples drawn from each of four improvement categories.The categories were Feedback Fidelity, Self Reflection, Specification Recall, and Random.
  • F.5 Human–GPT Judge Agreement: Two human annotators independently classified each trajectory using the GPT-5 judge’s criteria, and Cohen’s Kappa compared GPT-5 labels with the majority human label.This procedure evaluated agreement between automated and human attribution judgments.
  • F.5 Human–GPT Judge Agreement: 0.768 overall κ indicates substantial agreement between the GPT-5 judge and human annotators.The result supports GPT-5 as a reliable proxy for human attribution in this task.
  • F.5 Human–GPT Judge Agreement: Disagreements concentrate in Random and Self Reflection, where distinguishing stochastic drift from genuine self-correction can be ambiguous even for human raters.The ambiguity affects both automated and human attribution judgments.
  • G BrowseComp-Plus: BrowseComp-Plus evaluates deep research systems while isolating retriever effects with a local database, using 200 sampled examples for faster evaluation.The benchmark is sourced from BrowseComp and targets knowledge-intensive evaluation.
  • G BrowseComp-Plus: BrowseComp-Plus requires knowledge beyond the models’ training data, unlike the coding and math tasks.This benchmark therefore tests an out-of-distribution knowledge scenario.
  • G BrowseComp-Plus: For most models, incorporating self feedback causes a performance decline relative to the no-feedback setting on BrowseComp-Plus.The result highlights the limitation of relying solely on self feedback for out-of-distribution knowledge tasks.

H Full Results of Performance and Efficiency

Full results across eight reasoning LLMs and six benchmarks show that feedback-based generation techniques generally outperform baseline strategies, while self-feedback lags when tasks require external search-based knowledge.

  • Evaluation scope: The full evaluation reports model performance and API costs for eight reasoning LLMs over six benchmarks.Model performance is presented in Figure 3, while API costs are presented in Figure 12.
  • Model performance: On BrowseComp-Plus, self-feedback models fall behind when the task requires external search-based knowledge.This result highlights a limitation of self-feedback on search-dependent tasks.
  • Model performance: Feedback-based generation generally achieves higher model accuracy than baseline strategies across eight state-of-the-art LLMs and six benchmarks.The evaluated feedback levels include correctness/executor, self, and binary executor feedback.

I Token Analysis

CoE with feedback achieves higher accuracy at comparable token budgets than other multiround methods, indicating greater return per token rather than merely longer prompts.

  • I Token Analysis: CoE with feedback achieves higher accuracy at comparable token budgets than other multiround methods.The comparison is based on aggregated token counts and accuracy reported in Table 5.

J Dual Feedback and Principled Selection · K Prompt for Improvement Pattern Analysis

Dual feedback improves performance when its channels are complementary, while memory-based selection can lose critical reasoning preserved by full experience trails. The accompanying analysis prompt separates why a model changes from what changes and requires concise causal attribution of the observed learning pattern.

  • J Dual Feedback and Principled Selection: Dual feedback combines model feedback with correctness or executor signals within each iteration.The experiments use correctness feedback for math and executor feedback for coding, alongside model feedback.
  • J Dual Feedback and Principled Selection: Memory-based selection with Dynamic CheatSheet and SimpleMem consistently underperforms pure model feedback.The methods are integrated within the same task without cross-task retrieval.
  • J Dual Feedback and Principled Selection: Dual feedback surpasses either channel alone on AIME 2025 and LiveBench (Code).On OmniMath, correctness feedback alone matches dual feedback, indicating limited benefit from adding model feedback when the primary signal is already strong.
  • J Dual Feedback and Principled Selection: Detailed feedback, including self feedback, achieves decent results with fewer API costs than feedback-free CoE.Figure 12 compares total dollar API cost with best model performance over 20 iterations.
  • J Dual Feedback and Principled Selection: Aggressive summarization or retrieval may discard critical intermediate reasoning steps that full experience trails preserve.This explains why memory-based compression fails to add benefit beyond full experience traces in these experiments.
  • K Prompt for Improvement Pattern Analysis: The attribution judge identifies both the change driver and change manifestation between consecutive attempts.It does not recompute overall correctness; correctness labels are provided separately.
  • K Prompt for Improvement Pattern Analysis: Change drivers include feedback fidelity and self-reflection or internal reasoning.Feedback fidelity captures direct use of provided feedback, whereas self-reflection captures internally identified errors or alternative approaches without explicit feedback.
  • K Prompt for Improvement Pattern Analysis: Change manifestations include structural plan or algorithm revision and local step soundness or invariant fixes.The prompt distinguishes high-level reformulation from local logic, variable, arithmetic, or invariant corrections.

L Limitations · M Declaration of AI Tool Usage

The evaluation is concentrated on controlled math, knowledge, and coding benchmarks, while longer-horizon interaction settings and parameter updates remain outside the study. The manuscript reports limited GPT-5 use for language refinement, without using AI for original content, analysis, or core scientific ideas.

  • L Limitations: The evaluation focuses primarily on math, knowledge, and coding benchmarks.These domains provide controlled settings for probing iterative improvement.
  • L Limitations: Long-horizon interaction-intensive scenarios are not evaluated, although the CoE paradigm should naturally extend to them.The passage situates these scenarios as an important area beyond the current evaluation.
  • L Limitations: The study does not update model parameters, isolating Chain-of-Experience as a test-time paradigm.This design choice separates the mechanism from parameter learning.
  • L Limitations: Observed improvements therefore arise from contextual reuse of experience rather than true learning.The passage identifies parameter updates to internalize experience as an important next step.
  • M Declaration of AI Tool Usage: GPT-5 was used only for minor language refinement and smoothing during manuscript preparation.The stated use concerns wording rather than research development.
  • M Declaration of AI Tool Usage: AI tools were not used to generate original content, conduct data analysis, or formulate core scientific ideas.The declaration limits AI involvement in substantive manuscript and research contributions.
  • M Declaration of AI Tool Usage: Conceptual development, experimentation, and interpretation were conducted independently without reliance on AI tools.This statement covers the study’s core research activities.
Loading 2608.18027v1…