PaperlayerView exact revision on arXiv

Source-linked AI summary

CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning

Linas Nasvytis, Simon Jerome Han, Ben Prystawski, Satchel Grant, Noah D. Goodman, Judith E. Fan

arXiv:2605.28742v2cs.AI

TL;DR

Language models often need substantial data and compute to improve from verifiable rewards. CORE contrasts successful and unsuccessful reasoning traces to generate compact, utility-tested insights, and outperforms competing methods across diverse tasks while using fewer rollouts and less evaluation-time context.

  • Problem

    Existing methods for learning from verifiable rewards often require large amounts of data and computation, motivating more human-like efficiency from only a handful of practice problems.

  • Method

    CORE contrasts successful and unsuccessful rollouts to generate, verify, store, and retrieve concise natural-language insights about general reasoning strategies or constraints.

  • Results

    CORE achieves the highest mean held-out accuracy in 9 of 12 task-by-data-regime conditions across four reasoning tasks.

  • Takeaways & Limitations

    Storing and reusing contrastive insights rather than whole reasoning traces can improve sample, rollout, and context efficiency across reasoning, planning, and problem-solving tasks.

  • Takeaways & Limitations

    CORE assumes verifiable rewards, limiting its applicability to verifiable domains, and its experiments leave performance in more open-ended environments unresolved.

Abstract

from arXiv · show

Language models can use verifiable rewards to improve at a wide variety of reasoning tasks. However, both parametric (e.g. RLVR) and non-parametric (e.g. prompt optimization) approaches to doing so typically require hundreds of training samples and thousands of model rollouts, making them expensive in the best case and intractable in the worst. To address this challenge, we introduce Contrastive Reflection (CORE), a non-parametric learning algorithm that compares past reasoning traces to generate insights: short natural-language descriptions of reasoning strategies and constraints that capture differences between successful and unsuccessful problem attempts. Across four reasoning tasks, we demonstrate that CORE enables more rapid improvement than both parametric (GRPO) and non-parametric (GEPA, episodic RAG, and MemRL) methods, while using fewer rollouts. Under fixed rollout budgets with as few as five training samples, CORE achieves the strongest performance in most task-data regimes. Finally, we highlight how CORE is substantially more context-efficient than non-parametric baselines, requiring fewer prompt tokens while storing learned knowledge as compact, interpretable natural-language insights. Our results therefore suggest that distilling contrasts between successful and unsuccessful reasoning traces into abstract and useful insights can provide a more efficient and interpretable route to model self-improvement than weight updates, prompt optimization, or direct reuse of stored reasoning traces.

1 Introduction

CORE addresses the high data and compute demands of learning from verifiable rewards by using contrastive insight discovery from past successes and failures. It is introduced as a non-parametric method that improves efficiency, interpretability, and performance through accumulated natural-language insights.

  • Motivation: CORE targets the high data and compute demands of existing methods, which can require hundreds of thousands of rollouts or hundreds of training and validation samples [10] [3].The introduction contrasts these requirements with humans’ ability to improve substantially from limited experience.
  • Motivation: Contrastive reflection is motivated by evidence that comparing successes with failures yields more abstract, explicit, concise, and reusable principles than reflecting on experiences in isolation.The approach also draws on selective insight application based on current relevance or prior usefulness.
  • Method: CORE is a non-parametric algorithm that enables language models to learn from verifiable rewards by generating and accumulating insights.It uses external memories for generated insights and past rollouts, updating insight memory when contrastive reflection helps solve a problem.
  • Results: Across logic, planning, and problem-solving tasks, CORE outperforms parametric and non-parametric baselines while learning faster and using substantially less evaluation-time context.The claimed advantage holds regardless of the number of available training samples.
  • Interpretability and analysis: CORE insights are interpretable natural-language learning artifacts with empirical utility estimates, while ablations isolate contributions from contrastive reflection and utility-aware retrieval.These properties aim to reduce unwanted behaviors associated with opaque parameter updates.

2 Related Work

Related work spans parametric and non-parametric learning from verifiable rewards, external memory systems, and methods targeting sample and rollout efficiency. These approaches differ in whether they update model weights or context, what they store and retrieve, and how they trade off learning-efficiency dimensions.

  • Learning from Verifiable Rewards: Learning from verifiable rewards divides existing methods into parametric approaches that update model weights and non-parametric approaches that improve context while keeping the model frozen.Examples include STaR [32] and GRPO [5] [25] [10] for parametric learning, and MIPRO and GEPA [22] for non-parametric learning.
  • External Memory Systems for Language Models: External memory systems vary in what they store, from raw reasoning traces and reflections [27] to executable programs [30], distilled behavior descriptions [7] [23], and procedural guidance such as AGENTS.md [3].They also differ in how stored items are selected for retrieval, with both dimensions affecting sample efficiency.
  • Learning Efficiency: Learning efficiency has distinct sample-efficiency and rollout-efficiency dimensions, and improving one can come at the expense of the other.Sample efficiency concerns the number of distinct training problems, whereas rollout efficiency concerns the number of attempts required for those problems.

3 CORE: Contrastive Reflection

CORE is a non-parametric method that improves a frozen language model by converting contrasts between successful and failed reasoning traces into compact, utility-aware natural-language insights. It retrieves and tests these insights during training, then freezes its learned memories for held-out evaluation.

  • CORE: Contrastive Reflection: CORE improves a frozen language model by generating short insights that hypothesize which reasoning strategies or constraints distinguish successful from unsuccessful rollouts.These insights are not summaries of prior rollouts; they function as credit-assignment hypotheses for future problems.
  • Insight retrieval: CORE retrieves insights by combining local utility on semantically similar training problems with exploration bonuses, then updates retrieved insights using baseline-relative rewards.The baseline normalization gives positive evidence only when an update-conditioned attempt improves over the problem’s expected performance, while retrieved insights receive group-level credit.
  • Contrastive reflection: After a training failure, CORE contrasts the failed rollout with a semantically similar correct rollout and prompts the same frozen model to propose meaningful candidate insights.The positive rollout may come from the same problem when a previous correct solution exists, while semantic similarity helps expose subtle success and failure factors.
  • Contrastive reflection: Candidate insights are individually admission-tested before entering memory, and only candidates that satisfy the admission criterion are retained with utility evidence from their trial.In the experiments, one admission sample and zero margin mean an insight is admitted when it enables a first-attempt solution on its originating problem.
  • Evaluation: During evaluation, CORE freezes both memories and uses exploitation-only retrieval from training-problem rollouts, with no reflection, admission testing, exploration, or memory updates.Held-out test rollouts are never added to either memory.

4 Evaluation

Across four verifiable reasoning tasks, CORE improves rapidly and achieves the strongest results in most training-data regimes while using substantially less evaluation-time context than competing methods. Its learned insights are compact and functionally organized, with a small subset driving the largest gains.

  • Evaluation setup: Evaluations use 5, 10, or 100 training problems and separate 100-problem held-out sets, with three independent runs reporting mean verifier accuracy.The benchmark suite spans algorithmic, arithmetic, logical, and symbolic reasoning: Tower of Hanoi, MathGAP, ZebraLogic, and Matchstick arithmetic.
  • Rollout efficiency: By 350 training rollouts, CORE exceeds every baseline’s best evaluation performance and reaches higher final performance despite using 2,100 versus baselines’ 4,000 rollouts.Across tasks, held-out accuracy rises 59.9% from 0.445 at rollout 0 to 0.712 at rollout 350, reaching 0.717 at rollout 2,100.
  • Sample efficiency: CORE achieves the highest mean held-out accuracy in 9 of 12 task-by-data-regime conditions, improving over no learning by 54.8%, 56.2%, and 52.3% with 5, 10, and 100 examples.MemRL leads Tower of Hanoi with 5 and 100 examples, while GEPA leads ZebraLogic with 100 examples.
  • Context efficiency: CORE adds 0.92k context tokens per evaluation item, versus 33.6k for Episodic RAG, 32.7k for MemRL, and 1.29k for GEPA.The method’s gains therefore come from compressing training experience into abstract, reusable insights rather than retrieving large reasoning traces during evaluation.
  • Insight analysis: Most admitted insights have non-negative utility, while high-utility insights organize search spaces, track intermediate states, or verify and validate constraints.Non-negative weighted utility exceeds 91% for every task, and a smaller subset of high-utility insights drives the largest gains.

5 What Accounts for Learning from Contrastive Reflection?

CORE’s gains come from both contrastive insight generation and utility-aware insight reuse. Across four tasks, full CORE performs best among its variants and outperforms GEPA, with utility-aware retrieval improving final mean improvement from 0.227 to 0.268.

  • Insight generation: Contrastive reflection yields the strongest improvement over rollout 0 at every nonzero evaluation checkpoint, exceeding reflection from only incorrect or only correct traces.CORE compares failed and successful reasoning traces from the same or similar problems, while the alternatives use only one trace type.
  • Insight reuse: 0.268 versus 0.227: removing utility-aware retrieval reduces final mean improvement, showing that semantic relevance alone is insufficient for best performance.Full CORE combines semantic relevance with learned utility estimates when retrieving insights.
  • Ablation results: Full CORE achieves the largest final held-out-accuracy gains, outperforming GEPA and all ablated variants across the four-task evaluation.The comparison uses the 10-training-example setting and aggregates results across all four tasks.

6 Discussion

CORE improves sample, rollout, and context efficiency by storing contrastive insights about successful and failed reasoning traces rather than entire rollouts or their summaries. Its applicability is limited by reliance on verifiable rewards, coarse credit assignment, added inference cost, and evaluation focused on reasoning, planning, and problem-solving tasks.

  • CORE improves sample, rollout, and context efficiency over strong baselines across four logic, planning, and problem-solving tasks.
  • CORE stores insights that contrast successful and failed rollouts, changing what experience is stored and reused compared with episodic-memory methods.Standard episodic-memory methods store and retrieve entire rollouts or summaries of rollouts.
  • Future directions include combining CORE with RLVR-style training, continual learning through insight accumulation and retrieval, and extending CORE beyond single-turn reasoning.
  • CORE assumes verifiable rewards, limiting its applicability to verifiable domains, while its utility update gives all retrieved insights the same outcome.Finer-grained credit assignment among multiple insights remains unresolved.
  • Reflection and admission testing add inference cost, and the experiments focus on reasoning, planning, and problem-solving tasks.

Appendix Outline

The appendix provides pseudocode for CORE and analyzes how its insight memory grows across training rollouts, including the distribution of insight utilities.

  • Appendix Outline: The appendix presents pseudocode for the CORE algorithm.
  • Appendix Outline: It examines the growth of the insight memory across training rollouts.
  • Appendix Outline: It reports distributions of insight utilities.

A Pseudocode for the CORE algorithm

Algorithm 1 specifies CORE’s training procedure: it initializes rollout and insight memories, samples problems with failure bias, retrieves insights, evaluates generated solutions, and updates memory statistics. When a rollout fails, the procedure retrieves a positive rollout for contrastive processing.

  • A Pseudocode for the CORE algorithm: CORE initializes empty rollout and insight memories, estimates no-memory baselines for training problems, and iterates over training steps.The procedure requires a training set, frozen model, verifiers, rollout memory, and insight memory.
  • A Pseudocode for the CORE algorithm: At each step, CORE failure-biased samples a problem, retrieves top-K insights from neighboring rollouts, generates and verifies a solution, and computes utility as reward minus baseline.The generated rollout is stored in rollout memory, while retrieved insight statistics are updated using the computed utility.
  • A Pseudocode for the CORE algorithm: For failed solutions, CORE retrieves a positive rollout from rollout memory for contrastive reflection and insight admission testing.The supplied pseudocode identifies failure handling as part of a broader procedure that includes insight retrieval, contrastive reflection, and admission testing.

B Growth of the insight memory across training rollouts

Figure 5 tracks the number of admitted insights across training rollouts for each task, with uncertainty shown by 95% confidence-interval ribbons across runs.

  • B Growth of the insight memory across training rollouts: Figure 5 reports how the number of admitted insights changes across training rollouts for each task.
  • B Growth of the insight memory across training rollouts: The analysis compares insight-memory growth across tasks over the course of training rollouts.
  • B Growth of the insight memory across training rollouts: Ribbons represent 95% confidence intervals across the runs.

C Distributions of insight utilities

This section examines insight utilities across the four tasks and situates them alongside the growth of the insight memory during rollouts.

  • C Distributions of insight utilities: Figure 5 tracks the number of stored insights accumulated during training across rollouts for each task, with 95% confidence intervals across runs.
  • C Distributions of insight utilities: Figure 6 shows the distribution of estimated insight utilities across the four tasks.
Loading 2605.28742v2…