PaperlayerView exact revision on arXiv

Source-linked AI summary

COEVO: Co-Evolving Context and Parameters for Recursive Self-Improvement

Siwei Chen, Xinping Bao, Xinyu Cai, Yuan Cao, Wan Jiang, Shaohong Chen

arXiv:2609.33398v1cs.AI

TL;DR

Existing RSI approaches commonly adapt model parameters or external context separately, despite their interaction in shaping learning and context use. COEVO jointly updates parameters from on-policy experience and adapts context using policy entropy and attention. It improves task performance over fixed-context RL and achieves stronger worst-case performance under prompt changes in the reported evaluation.

  • Problem

    Parameter learning and context optimization are often studied separately, although context shapes learning experience and policy updates change how that context is interpreted.

    Source: §1 · p. 1

  • Method

    COEVO couples on-policy parameter learning with context adaptation guided by policy entropy and prompt-conditioned attention.

    Source: §1 · p. 2

  • Results

    COEVO improves task performance over fixed-prompt RL, with worst-case Average@12 of 43.06%, exceeding the strongest fixed-context baseline by 2.78 percentage points.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    The results support treating external context as an adaptive component of RSI rather than only a fixed interface to an LLM.

    Source: §7 · p. 9

    The study uses bounded adaptation: parameters and editable strategy context change while the task objective, optimizer, evaluator, and execution interface remain fixed.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Recursive self-improvement (RSI) seeks to move large language models beyond static training pipelines toward systems that can participate in improving their own future behavior. Existing approaches largely follow two directions: updating model parameters through online learning, or improving the external context through search, reflection, and prompt optimization. Although both mechanisms can support continued improvement, they are typically studied independently. This separation overlooks an important interaction: the context shapes the experience from which a model learns, while an evolving model may interpret and utilize the same context differently over time. We therefore formulate RSI as a problem of parameter--context co-evolution, where model parameters and the learning context adapt within a shared feedback loop. We introduce COEVO, a framework that updates model parameters from on-policy experience while adapting contextual guidance according to the state of the evolving policy. Policy entropy and prompt-conditioned attention are used as complementary signals to guide this adaptation. Experiments show that COEVO consistently improves task performance over fixed-context reinforcement learning and produces policies that are more robust to changes in system prompts. More broadly, our results suggest that external context should be viewed not merely as a fixed interface to a large language model, but as an adaptive component of recursive self-improvement.

1 Introduction

RSI methods often adapt parameters or external context separately, although each changes the conditions under which the other operates. COEVO couples on-policy parameter learning with policy-aware context refinement, and improves task performance over fixed-prompt RL in the evaluated settings.

1 Introduction

  • RSI aims to close the feedback loop between a model’s state, the experience it generates, and the mechanisms that produce subsequent improvements.On-policy RL lets models generate trajectories, receive task-level feedback, and update parameters from that experience.Source: §1 · p. 1
  • Parameter-learning approaches typically keep training context fixed, while context-optimization approaches generally treat the underlying model policy as fixed.The former can internalize strategies; the latter can alter how existing capabilities are elicited.Source: §1 · p. 1
  • The context shapes learning experience, while parameter updates change how the policy interprets and uses that context.This coupling motivates treating RSI as parameter–context co-evolution.Source: §1 · p. 1, Figure 1 · p. 2
  • Unlike reward-driven prompt search, COEVO uses exploration state and context utilization to guide targeted context adaptation.Reward-only search can rank candidate prompts but offers little information about why a context is ineffective or how to change it.Source: Figure 1 · p. 2
  • COEVO couples on-policy parameter updates with context refinement guided by policy entropy and prompt-conditioned attention, improving task performance over fixed-prompt RL in evaluated settings.Ablations also show that these policy-aware signals improve over reward-only prompt evolution.Source: §1 · p. 2
  • The paper frames RSI as a unified co-evolution problem in which model updates change contextual interpretation and context updates reshape on-policy experience.It also identifies complementary exploration and context-utilization signals, with ablations showing distinct failure modes when either is removed.Source: §1 · p. 2

2 Problem Formulation and Design Hypotheses

The paper frames recursive self-improvement as a coupled process: context shapes the experience used for learning, and policy changes alter how context is used. Its design hypothesis is that context usefulness varies with policy state, motivating interventions based on both exploration and context utilization.

2.1 Context affects the learning distribution

  • Figure 2 shows prompt-section attention shifting during training and entropy declining under fixed detailed context, while adaptive context maintains broader exploration.These observations motivate using both utilization and exploration signals.Source: Figure 2 · p. 3
  • Changing training context changes the trajectory distribution contributing to the next policy update, even with the task reward and optimizer fixed.This is distinct from selecting an inference wrapper after learning.Source: §2.1 · pp. 2–3

2.2 Context Usefulness May Be Policy-Dependent

  • The same scaffold may help an early policy but later become redundant or restrictive, and a different policy may use an initially ignored instruction.The paper hypothesizes that scaffold usefulness depends on current policy state, not the prompt alone.Source: §2.2 · p. 3
  • Context usefulness is relational: it depends on how the current policy uses guidance and how broadly it is exploring, not on a universal ranking of prompt styles.Figure 2 observations motivate state-dependent context revision.Source: §2.2 · p. 3

2.3 Context revision as a state-dependent intervention

  • Policy entropy represents behavioral exploration, while section-level attention serves as an operational proxy for context utilization.Together, the coordinates distinguish policy states that may have similar outcome feedback but call for different interventions.Source: §2.3 · p. 3
  • The framework tests candidate contexts with frozen-weight comparisons, then evaluates state-conditioned revision through subsequent learning under shared evaluation contexts.Attention is treated as a utilization proxy and the reference band as a calibration device, not as an optimal-state detector.Source: §2.3 · p. 3

3 Co-Evolving Context and Parameters

COEVO alternates policy learning under a fixed prompt with context adaptation informed by the evolving policy. Entropy and attention characterize exploration and scaffold use, and verified context changes shape the on-policy experience used in subsequent learning.

3.1 Parameter–Context Co-Evolution

  • COEVO represents RSI as joint evolution of model parameters and a prompt comprising fixed task constraints plus an editable strategy scaffold.The task objective is expected reward over tasks and trajectories generated by the current policy conditioned on that prompt.Source: §3.1 · p. 4
  • Reward remains the optimization objective; entropy and attention characterize policy state and guide context evolution rather than replace reward.The feedback loop runs both ways: context changes future training experience, while parameter changes alter how context is used.Source: §3.1 · p. 4
  • Training alternates between on-policy parameter updates under a fixed prompt and context adaptation using the updated checkpoint.The task objective, optimizer, evaluator, and execution interface remain fixed, making this a bounded form of RSI.Source: §3.1 · p. 4

3.2 Policy-State Characterization

  • Policy entropy measures exploration, with lower values indicating concentration on fewer continuations and higher values indicating broader exploration.Entropy is interpreted relative to its historical mean, not as a globally fixed optimum.Source: §3.2 · p. 4
  • Section-level attention estimates utilization by measuring response attention to semantic scaffold sections split at sentence boundaries.Utilization is length-normalized and interpreted relative to measurements from the current checkpoint.Source: §3.2 · pp. 4–5
  • Entropy and utilization jointly guide the direction and form of context intervention, while attention remains a proxy rather than evidence of causal influence.The reference ranges calibrate both signals to the current policy.Source: §3.2 · p. 5, §3.3 · p. 5, Table 1 · p. 5

3.3 Policy-Aware Context Evolution

  • Entropy and attention define four intervention regimes; context is preserved when exploration is in range or utilization estimates are unreliable.When multiple sections show utilization deviations, the controller selects the section with the strongest policy-relative mismatch.Source: §3.3 · pp. 5–6, Figure 3 · p. 5
  • A constrained editor proposes local scaffold revisions, which are accepted only when frozen-checkpoint tests shift the relevant state as intended while preserving task performance within tolerance.Accepted revisions generate the next on-policy experience, closing the loop between context evolution and parameter learning.Source: §3.3 · p. 6

4 Experiments

The experiments compare COEVO with fixed-prompt GRPO across model scales and math and code tasks, using shared evaluation prompts to assess performance and prompt sensitivity.

Models and domains

  • Evaluations cover four Qwen model checkpoints on mathematical reasoning and code generation, with LoRA-based, GRPO-style parameter updates.Math uses DAPO-Math-17k and AIME 2025; code uses a temporal split derived from LiveCodeBench, starting from a common SFT checkpoint trained on 300 verified trajectories.Source: §4 · p. 6

Baselines

  • Fixed-prompt GRPO uses Simple, Medium, and Detail contexts, while shared inference prompts separate training-context effects from evaluation-context differences.E-SPL-style comparisons are restricted to the ablation setting because full RL training is computationally costly.Source: §4 · p. 6, Figure 4 · p. 7

Evaluation

  • Average@12 and Pass@12 measure mean per-problem success and the fraction of problems solved by at least one sample, respectively.Code results are macro-averaged over three evaluation prompts and use the local temporal split, not the official LiveCodeBench benchmark.Source: §4 · p. 6

Baselines

  • COEVO is compared with fixed prompts of increasing specification: Simple, Medium, and Detail.Figure 4 spans Qwen3.5 model scales and task domains.Source: Figure 4 · p. 7

5 Results and Analysis

COEVO improves math performance and worst-case performance across inference prompts. Case studies show policy-state-dependent, selective context changes, while ablations indicate that entropy and attention provide complementary guidance.

5.1 Main Results

  • COEVO improves both Average@12 and Pass@12 on Math at 4B and 9B, achieving the strongest overall performance across the compared prompt settings.Increasing fixed-prompt detail can improve one metric while degrading another; Figure 4 compares the approaches across model scales and task domains.Source: §5.1 · p. 7
  • 43.06% worst-case Average@12 and 73.33% worst-case Pass@12 make COEVO strongest under both metrics across inference prompts.These are 2.78 and 6.66 percentage points above the strongest fixed-context baseline, respectively; Pass@12 is identical across all three inference prompts.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Case Study

  • In Case I, 11 of 20 revisions were accepted, with low exploration and utilization prompting added structure and high utilization with low exploration prompting relaxed requirements.High-exploration states prompted stronger organization or consolidation; Case II removed or rewrote redundant operational details while retaining core behavioral requirements, so evolution was not simply cumulative.Source: §5.2 · pp. 7–8

5.3 Ablation and Mechanism Analysis

  • Attention-only lacks an explicit exploration signal, while entropy-only cannot localize scaffold use; together, entropy and attention provide complementary information for context interventions.Attention-only's entropy decreases substantially over training, while entropy-only leads to weaker downstream performance.Source: §5.3 · p. 8
  • The E-SPL-style baseline's stronger on-policy performance coincides with held-out Average@12 of 41.61% and Pass@12 of 66.67%, trailing COEVO by 2.70 and 6.66 points.The gap coincides with greater entropy and output-diversity reduction, consistent with premature exploitation; COEVO conditions revision on exploration and utilization.Source: §5.3 · pp. 8–9

6 Related Work

Related work studies parameter adaptation, external-context optimization, and broader forms of recursive self-improvement. COEVO uses policy entropy and prompt-conditioned attention to guide context adaptation during reinforcement learning.

Recursive self-improvement

  • Online learning and reinforcement learning update model parameters, while reflection, prompt search, evolutionary optimization, and linguistic memory adapt external context.The paper characterizes these as complementary approaches that are typically considered separately.Source: §6 · p. 9

Reinforcement learning and policy-state characterization

  • COEVO uses policy entropy and prompt-conditioned attention as complementary descriptors of exploration and instruction utilization to guide context evolution during RL.Unlike work that modifies programs, algorithms, or agent code, this approach focuses on policy-aware parameter–context co-evolution while keeping surrounding optimization and evaluation machinery fixed.Source: §6 · p. 9

7 Conclusion

The conclusion frames recursive self-improvement as coordinated evolution of model parameters and external context, studied in a bounded and controllable setting.

7 Conclusion

  • Parameter–context co-evolution offers a perspective for analyzing how model updates change context use and context updates reshape subsequent learning experience.The framework is presented as a foundation for studying more general self-improving systems and how adaptive mechanisms may jointly contribute to recursive capability development.Source: §7 · p. 9

A.1 Models and tasks

The appendix specifies the Math and Code task data and experimental settings, then gives separate trajectory limits for the two domains.

A.1 Models and tasks

  • Math uses AIME 2025 for evaluation, while the Code experiments use a temporal split derived from LiveCodeBench release-v6.The Math training set contains 17,867 unique problems; the Code train, calibration, development, and final-test splits contain 400, 60, 100, and 200 disjoint tasks.Source: §A.1 · p. 12

A.2 RL configuration

  • Math trajectories allow at most six assistant turns, with tool-call and token limits that vary by model scale.The maximum trajectory length is 8192 tokens for 4B and 32768 for 9B and 27B; code-interpreter calls are capped at four for 4B and three for 9B and 27B.Source: §A.2 · p. 12
  • Code trajectories allow at most two assistant turns and one isolated, stateless Python call.They are capped at 10,240 tokens per response, 20,480 total trajectory tokens, and 512 tool-response tokens.Source: §A.2 · p. 12

A.3 Evaluation protocol

The appendix defines evaluation measures and protocols, describes when and how COEVO edits prompts, and reports selected training and checkpoint results.

A.3 Evaluation protocol

  • Average@12 is the mean success rate across twelve responses per task, while Pass@12 is the fraction of tasks with at least one successful response.For Code, success requires the submitted program to pass the full judge.Source: §A.3 · p. 13

A.4 Context-Evolution Algorithm

  • COEVO attempts context evolution every five RL steps, after sampling on-policy trajectories and updating the model parameters.The fixed task specification and interface remain unchanged; only the strategy scaffold is selected to change.Source: §A.4 · pp. 13–14
  • At an edit boundary, the controller compares latest-step entropy and section utilization with their cumulative historical means to determine edit direction.The means include the current step, and the first-boundary exploration and utilization states are initialized to Low.Source: §A.4 · p. 13
  • The frozen current checkpoint compares the incumbent prompt with valid edits on the same latest task examples, then selects the candidate ranked best in the requested direction.The reported algorithm generates eight candidate edits; the incumbent and candidates use fresh responses under the same checkpoint.Source: §A.4 · p. 14

B.1.1 Math prompts

  • The reported Math prompts instruct models to solve step by step and use a Python interpreter when it helps with calculation, symbolic manipulation, or enumeration.The final answer must use a boxed format, and a tool call must stop before the final-answer turn.Source: §B.1.1 · p. 15

B.1.2 Code prompts

  • The Code prompt variants distinguish concise reasoning with one focused check from a detailed workflow that derives, implements, and reviews a solution.The detailed workflow also checks examples and boundary or adversarial cases when useful.Source: §B.1.2 · p. 16

C.1.2 Qwen3.5-9B

  • For the Qwen3.5-9B math training trajectory on AIME 2025, COEVO reports 65.56% Average@12 and 93.33% Pass@12, achieving the highest reported aggregate.Source: §C.1.2 · p. 18, Table 9 · p. 19

C.2 Code Checkpoint Selection

  • Code checkpoints are selected on a separate 100-task development split using three inference prompts, with Mean, then Worst, then Case as the selection order.The selected steps differ by prompt and scale: 40, 100, and 100 for 4B-Code; 100, 60, and 60 for 9B-Code.Source: §C.2 · p. 18

C.3 Final Code Evaluation

The final Code summaries report COEVO's success metrics at both model scales.

C.3 Final Code Evaluation

  • At 4B-Code, COEVO reaches 30.40% Average@12 and 43.33% Pass@12, the highest success rate reported.The final summaries are reported in Table 12.Source: §C.3 · p. 19
  • At 9B-Code, COEVO reaches 34.12% Average@12 and 55.67% Pass@12, the highest success rate reported.The final summaries are reported in Table 12.Source: §C.3 · p. 19
  • Table 12 summarizes final Code results on fixed-prompt checkpoints.Source: §C.3 · p. 19

C.4 Aggregate Results

The aggregate results align principal Math and Code settings in one table and describe the evaluation scope for those entries.

C.4 Aggregate Results

  • 44.31% Average@12 and 73.33% Pass@12 are the reported COEVO results for 4B-Math.These entries follow the main Figure 5 results.Source: §C.4 · p. 19
  • Table 13 reports Math and Code results in a single aligned table across the principal settings.Its caption identifies the table as aggregate results across principal Math and Code settings.Source: §C.4 · p. 19, Table 13 · p. 21
  • Fixed-prompt Code entries in the aggregate table are macro-averaged over the Simple, Medium, and Detail inference prompts.Source: §C.4 · p. 19

Related papers

Loading 2609.33398v1…