PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Knowledge-Model Co-Evolution via Consistency Regularization

Sirui Chen, Zhining Liu, Chao Zhang, Jingrui He, Zheng Li

arXiv:2610.30790v1cs.AI

TL;DR

Different analyses of the same experience can yield guidance with substantially different utility, making reliable knowledge evolution a challenge for LLM agents. SPLICE uses consistency of utility gains across perturbed views to co-evolve knowledge and model parameters. Across benchmarks and model scales, it outperforms baselines and its evolved knowledge supports more efficient task solving and transfers across models.

  • Problem

    Different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, including plausible guidance that underperforms using no knowledge.

    Source: §1 · p. 2

  • Method

    SPLICE selects knowledge updates using common gains across perturbed evolution views and interleaves those updates with reinforcement learning and gated contrastive self-distillation.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    Across four benchmarks, SPLICE improves average accuracy over the strongest baseline by 3.0 points with Qwen3.5-4B and 4.6 points with Qwen3.5-9B.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Its evolved knowledge requires up to 44% fewer output tokens per correct solution than the strongest baseline and transfers across model sizes and families.

    Source: §1 · p. 2

    The improvement bound assumes independent, identically distributed utility gains from evolution views.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

LLM agents can evolve by learning from accumulated experience. Existing approaches extract reusable external knowledge, such as skills and memories, from experience to guide subsequent task solving, with some further internalizing this knowledge through model training. However, different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Inspired by consistency regularization, we identify consistent utility gains across different ways of analyzing the same experience, rather than relying on any single way of analyzing it. We introduce SPLICE, a framework for consistency-regularized knowledge-model co-evolution. It constructs evolution views by perturbing how the same experience is analyzed, producing different knowledge candidates. Knowledge updates are then guided by common gain, which accounts for utility gains and their consistency across views. During model evolution, self-distillation internalizes the evolved knowledge alongside reinforcement learning, using knowledge associated with consistent utility degradation as a contrastive reference. The two processes interleave, with model updates driving further knowledge evolution. Experiments across diverse domains show that SPLICE outperforms knowledge evolution and model training baselines by up to 7.1 and 4.6 points in average accuracy, while its evolved knowledge reduces output tokens per correct solution by up to 44% and transfers across model sizes and families.

1 INTRODUCTION

Agents can turn accumulated experience into reusable knowledge and internalize that knowledge through training, but the utility of guidance can depend on how the experience is analyzed. SPLICE uses consistency of utility gains across different analyses to guide knowledge evolution.

1 INTRODUCTION

  • Different analyses of the same experience can produce reasonable-looking guidance with substantially different utility.The introduction notes that plausible knowledge can even underperform using no knowledge at all.Source: §1 · p. 2
  • Consistency across evolution views helps distinguish gains that generalize beyond a single analysis from view-specific gains.SPLICE treats experience-to-knowledge transformation as parameterized by textual instructions and perturbs them to define different views.Source: §1 · p. 2
  • SPLICE perturbs how experience is analyzed, scores knowledge candidates by common gain, and interleaves knowledge updates with model training.It uses self-distillation alongside reinforcement learning to internalize evolved knowledge, with noisy knowledge from consistent utility degradation as a contrastive reference.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe an agent that uses external knowledge while generating task trajectories, then updates both its knowledge and model through an interleaved process. They also introduce GRPO as a policy-optimization method.

2 PRELIMINARIES

  • An agent’s experience comprises its task, generated trajectory, environment feedback, and reward, while external knowledge can include reusable skills, tools, memories, or environment information.The agent generates trajectories conditioned on its current model, task history, feedback, and knowledge.Source: §2 · p. 2
  • Knowledge evolution updates external knowledge from experience, and model evolution trains the policy to internalize knowledge-guided behavior.The updated model generates new experience that can change knowledge utility and inform subsequent knowledge evolution.Source: §2 · pp. 2–3
  • SPLICE’s overview divides the framework into knowledge evolution, which selects experience using common gain across views, and model evolution, which combines GRPO with gated contrastive self-distillation.Noisy knowledge serves as the contrastive reference for internalizing evolved knowledge.Source: Figure 2 · p. 3
  • GRPO optimizes the policy using group-relative advantages, with a clipped policy objective and a KL penalty toward a frozen reference policy.The penalty is weighted by β, and ε is the clipping threshold.Source: §2 · p. 3

3 SPLICE: KNOWLEDGE-MODEL CO-EVOLUTION

SPLICE co-evolves external knowledge and model parameters by selecting knowledge updates with consistent utility gains across perturbed evolution views, then internalizing the adopted knowledge. The updated model generates experience for subsequent rounds.

3 SPLICE: KNOWLEDGE-MODEL CO-EVOLUTION

  • SPLICE interleaves knowledge evolution and model evolution, using the updated model’s new experience to support further knowledge updates.Knowledge utility is evaluated with respect to the current model as training proceeds.Source: §3 · p. 3, §3.2 · p. 5

3.1 CONSISTENCY-REGULARIZED KNOWLEDGE EVOLUTION

  • Textual revisions create distinct evolution views that analyze the same experience and produce separate knowledge candidates.Candidates are evaluated by replaying tasks; common gain combines mean utility gain with an inconsistency penalty, and selected trajectories support subsequent updates.Source: §3.1 · pp. 3–4
  • Common-gain selection favors tasks with gains supported across views, discounting isolated high gains; the resulting knowledge update is retained only if held-out validation performance improves.The bound described in the section depends on mean gain and utility inconsistency; at the same positive mean gain, lower inconsistency gives a stronger lower bound on improvement under a new view.Source: §3.1 · p. 4

3.2 KNOWLEDGE-GUIDED MODEL EVOLUTION

  • For contrastive self-distillation, SPLICE extracts noisy knowledge from tasks with utility degradation consistently supported across views.The noisy knowledge is constructed anew when model evolution starts with the adopted knowledge.Source: §3.2 · pp. 4–5
  • SPLICE weights distillation by each task’s observed benefit from knowledge and gates noisy-knowledge contributions at the token level.When evolved and noisy knowledge affect a token’s probability in the same direction, the gate is zero; opposing effects determine the noisy reference’s contribution.Source: §3.2 · p. 5

4 EXPERIMENTS

Across four benchmarks, SPLICE improves on knowledge-evolution and model-training baselines. Its analyses examine consistency-based selection, efficiency, transfer across models, and how knowledge utility changes during co-evolution.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers LiveMath, LiveCode, SpreadSheet, and WebShop using Qwen3.5-4B and Qwen3.5-9B, with knowledge-only and co-evolution variants of SPLICE.Baselines include training-free knowledge evolution and methods that train or co-evolve model parameters.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 MAIN RESULTS

  • SPLICE (Know. Evo) matches or outperforms every knowledge-evolution baseline, improving Avg. over SkillOpt by 6.6 points with Qwen3.5-4B and 7.1 points with Qwen3.5-9B.Its largest gains are on WebShop: 13.9 and 11.1 points, respectively, where tasks require long-horizon interaction.Source: §4.2 · p. 6, Table 1 · p. 6
  • SPLICE (Co-Evo) exceeds the strongest model-training baseline, ReSkill, by 3.0 and 4.6 points in Avg. with Qwen3.5-4B and Qwen3.5-9B, respectively.The results also note that skill-augmented baselines do not always improve over GRPO on complex tasks.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 ANALYSIS

  • Replacing common gain with mean gain drops Test Acc. by 13.8 and 6.1 points for Qwen3.5-4B and Qwen3.5-9B, and validation Acc. is maintained or improved in only 29% of rounds.This ablation tests whether selecting task evidence by average utility gain alone suffices.Source: §4.3 · p. 7, Table 2 · p. 7
  • SPLICE achieves the highest Test Acc. across both model sizes on SpreadSheet and WebShop, outperforming Best branch and Merge-all strategies that also use multiple evolution views.Best branch performs worst; Merge-all may include redundant or distracting evidence, indicating that gains are not due to additional computation alone.Source: §4.3 · p. 7, Figure 4 · p. 7
  • SPLICE has the lowest output tokens per correct attempt on WebShop and LiveCode for both model sizes, reducing this cost by up to 44% relative to the strongest baseline.On WebShop it produces 4.98k and 6.07k knowledge tokens for Qwen3.5-4B and Qwen3.5-9B, respectively, and requires 8.33 and 8.64 interaction steps.Source: Table 3 · p. 7, §4.3 · p. 8, Figure 5 · p. 8
  • Knowledge evolved with Qwen3.5-9B yields the largest mean gain across all tested target-model settings, including other sizes and model families, without further evolution.On LiveMath, gains are 31.13 points for Qwen3.5-4B and 33.86 points for Gemma-4-12B-IT, versus baseline gains of at most 17.38 and 12.42 points.Source: §4.3 · p. 8, Figure 6 · p. 8
  • On LiveMath, trained models without knowledge outperform base models with final knowledge for both sizes; Qwen3.5-9B reaches 67.88% versus 52.32%.Knowledge gains shrink as models improve, and early knowledge no longer improves the trained 4B model, motivating validation with the current model before training.Source: §4.3 · pp. 8–9, Figure 7 · p. 9

5 RELATED WORK

Prior work studies external knowledge evolution, knowledge internalization and co-evolution, and consistency regularization. SPLICE draws on these strands by using utility consistency to guide knowledge updates and by using consistently degrading knowledge as a contrastive reference.

Knowledge evolution for LLM agents

  • Knowledge-evolution work develops external guidance from experience, but recent studies report that accumulated knowledge does not reliably improve performance and can mislead future tasks.Some approaches retain only candidate updates with positive estimated gain; SPLICE instead uses utility consistency across evolution views.Source: §5 · p. 9

Knowledge internalization and co-evolution

  • Prior work internalizes knowledge through reinforcement learning and self-distillation, and co-evolves skills with policies; SPLICE forms its contrastive reference from knowledge that consistently degrades utility.It gates that reference at the token level to retain guidance shared with evolved knowledge.Source: §5 · p. 9

Consistency regularization

  • Consistency regularization encourages consistent predictions under perturbations, a principle extended from semi-supervised learning to supervised training and large-model fine-tuning.SPLICE applies consistency to the utility of evolved knowledge.Source: §5 · p. 9

6 CONCLUSION

SPLICE co-evolves knowledge and model parameters using utility gains consistent across evolution views. Across four benchmarks and two model scales, it outperforms knowledge evolution and model training baselines, while its knowledge supports efficient task solving and transfer.

6 CONCLUSION

  • Across four benchmarks and two model scales, SPLICE outperforms knowledge evolution and model training baselines, while its knowledge supports efficient task solving and transfer.The conclusion also reports that consistent gains across different analyses provide an effective signal for turning experience into generalizable knowledge.Source: §6 · p. 9
  • SPLICE updates knowledge using gains consistent across evolution views and internalizes it through reinforcement learning and gated contrastive self-distillation.The processes interleave, with the updated model generating experience for further knowledge evolution.Source: §6 · p. 9

A.1 BENCHMARKS

This appendix specifies the benchmark tasks, models, evaluation protocol, and sequential and asynchronous procedures for co-evolving knowledge and models.

A.1 BENCHMARKS

  • Training tasks supply experience, validation tasks evaluate proposed knowledge updates, and test tasks are reserved for evaluation.Table 4 summarizes the data splits and training settings; counts refer to tasks or, for WebShop, shopping sessions.Source: §A.1 · p. 15, Table 4 · p. 15
  • The four benchmarks cover multiple-choice mathematics, code generation, spreadsheet editing, and shopping through textual product pages and purchase actions.LiveMath uses March–June 2026 subsets; LiveCode combines v5 and v6 with contest-separated splits, SpreadSheet uses Verified-400, and WebShop uses its official 1K-product environment.Source: §A.1 · p. 15

A.2 MODELS AND BASELINES

  • Experiments use Qwen3.5-4B and Qwen3.5-9B task models, with an external LLM as the knowledge optimizer, and compare knowledge and training baselines.Baselines include Base, ReasoningBank, ACE, EvoSkill, SkillOpt, GRPO, SkillRL, and ReSkill.Source: §A.2 · pp. 15–16

A.3 EVALUATION PROTOCOL

  • Accuracy is the percentage of test tasks solved under each benchmark’s task-specific success criterion; SPLICE uses the latest knowledge passing held-out validation with the final model, or no knowledge if none passes.Training-free methods are averaged over three seeds.Source: §A.3 · p. 16

B.1 DESIGN CONSIDERATIONS

  • Co-evolution must evaluate knowledge with the model that uses it and keep the positive and contrastive knowledge fixed while the model internalizes them.Sequential evolution meets both requirements but makes knowledge evolution and model training wait for each other; asynchronous evolution relaxes the first requirement to use useful knowledge sooner.Source: §B.1 · p. 17

B.2 SEQUENTIAL CO-EVOLUTION

  • Sequential co-evolution alternates knowledge updates with the model fixed, then trains with positive and contrastive knowledge fixed before resuming evolution using new experience.Knowledge evolution stops after M consecutive rejected updates; the rejection rounds provide a shared baseline for constructing contrastive knowledge, and the sequential phases leave each other idle in turn.Source: §B.2 · p. 17

B.3 ASYNCHRONOUS CO-EVOLUTION

  • Asynchronous co-evolution evolves a separate knowledge copy during training and adopts it only when it improves held-out validation with the current model.It performs self-distillation when the reward gap exceeds a threshold and otherwise uses GRPO; training uses stage limits and can restart knowledge evolution after L stages without adopted knowledge.Source: §B.3 · pp. 17–18
  • Asynchronous knowledge evolution replays experience under its generating knowledge, then merges common-gain updates into the latest evolving knowledge rather than restarting from each batch.This lets knowledge evolution accumulate improvements even when the knowledge used to generate experience differs from the current evolving copy.Source: §B.3 · p. 18

B.4 DISCUSSION: RELAXING R1

Asynchronous replay can mix knowledge gains with model changes, shifting which tasks pass the common-gain threshold. The discussion describes design choices that limit this effect and relates SPLICE’s consistency criterion to prior work.

B.4 DISCUSSION: RELAXING R1

  • Because asynchronous replay uses a newer model, model improvement can admit tasks whose gains came from training, while model degradation can exclude useful evidence.Recent experience keeps the model change small, and knowledge is adopted for training only after validation with the current model; the remaining cost is potentially less-informative selections and rejected updates.Source: §B.4 · p. 19

C MORE RELATED WORK

  • SPLICE differs from prediction-consistency methods by requiring agreement in knowledge utility gains, not in the content produced by perturbed views.Related approaches also evolve external knowledge and internalize it through policy training or self-distillation; SPLICE derives its negative reference from knowledge associated with consistent utility degradation.Source: C MORE RELATED WORK · p. 20

D ADDITIONAL RESULTS

Additional WebShop analyses find that SPLICE’s proposed updates are comparatively stable and improve both error correction and preservation of existing successes. LiveMath examples show that knowledge utility changes as the model improves.

Stability of proposed updates

  • On WebShop, SPLICE’s proposed updates mostly stay near zero or positive validation-accuracy change and have the smallest mean absolute change for both model sizes.Baselines frequently propose updates that lower validation accuracy by 10 points or more, especially SkillOpt and EvoSkill; Figure 8 covers the first 50 evolution rounds, including updates later rejected.Source: Figure 8 · p. 21, D ADDITIONAL RESULTS · p. 21

Error correction and success preservation

  • 23.60% error correction and 75.94% success preservation with Qwen3.5-4B, and 17.75% and 77.47% with Qwen3.5-9B, are SPLICE’s highest pooled rates for both metrics.The best baseline preservation rates are 68.40% and 64.79%, respectively; Figure 9 reports proposed updates from the first 30 WebShop rounds, excluding rounds without an update.Source: Figure 9 · p. 21, D ADDITIONAL RESULTS · p. 22

Training dynamics of co-evolution

  • On LiveMath, knowledge helps early but its benefit shrinks as the model improves without it, and some updates are then rejected.Figure 10 tracks validation accuracy with and without the latest knowledge; it also illustrates knowledge that helps the 4B model at step 44 but causes an error at step 56, and a 9B model that accepts a useful execution-feedback check.Source: Figure 10 · p. 22, D ADDITIONAL RESULTS · p. 22

E PROOF OF PROPOSITION 1

The proof establishes that positive common gain implies positive mean utility gain. In an illustrative selection model, common-gain selection can outperform single-view and mean-gain selection when view-specific gains fall within a particular range.

E PROOF OF PROPOSITION 1

  • Positive common gain C guarantees positive mean gain µ, since C ≤ µ under the stated integrability and identical-distribution assumptions.The proof uses independent, identically distributed view gains and assumes E|∆| < ∞.Source: E PROOF OF PROPOSITION 1 · p. 23

F.1 ILLUSTRATIVE COMPARISON OF SELECTION STRATEGIES

  • When aq^2 < d < aq, common-gain selection has strictly higher expected held-out utility than single-view and mean-gain selection for sufficiently large N.The comparison assumes view-specific gains do not transfer to held-out tasks and treats replay gains as exact expected reward differences, without estimation noise.Source: §F.1 · pp. 24–25
  • Common gain discounts view-specific gains: it approaches gS + aq^2 as N increases, while mean gain approaches gS + aq.When aq^2 < d < aq, this difference changes which task evidence is selected for the knowledge update.Source: §F.1 · p. 25
  • The selection advantage is limited to a specific regime: if d > aq, both multi-view strategies select R with probability approaching one, while if d < aq^2, both select S.The analysis does not establish universal superiority over mean-gain selection or merging all evidence.Source: §F.1 · p. 25

F.2 ROBUSTNESS TO SHIFTS IN EVOLUTION VIEWS

The view-shift result bounds expected utility under a changed distribution of evolution views using the original mean gain and common gain. For shifts with 0 ≤ ρ ≤ 1, positive common gain guarantees positive expected utility gain under the new distribution.

F.2 ROBUSTNESS TO SHIFTS IN EVOLUTION VIEWS

  • The shifted expected utility is bounded below by (1 − ρ)µ + ρC, where ρ controls how strongly the relative frequencies of evolution views can change.The bound applies when Q is absolutely continuous with respect to P and the stated finite-mean assumption holds.Source: §F.2 · pp. 25–26
  • When 0 ≤ ρ ≤ 1, positive common gain guarantees positive expected utility gain under the shifted view distribution Q.The bound concerns knowledge evolution on a fixed task, not transfer to arbitrary new tasks or preservation after the optimizer consolidates updates.Source: §F.2 · pp. 25–26
  • At a fixed mean gain, lower utility inconsistency µ − C provides a stronger guarantee under a change in the view distribution.The possible reduction in expected utility is controlled by µ − C.Source: §F.2 · p. 26

G.1 EVOLVED KNOWLEDGE

The examples contrast knowledge updates that improve task execution with noisy guidance that can derail it. Consolidated evolved knowledge resolves repeated product-search loops, whereas a misleading instruction can exhaust the task budget without a purchase.

G.1 EVOLVED KNOWLEDGE

  • Different candidate instructions each raise replay reward from 0 to 1, and the consolidated color-selection rules yield a successful purchase where the original knowledge led to repeated visits and no purchase.The consolidated rules prioritize inspecting another product before using a closest-color fallback and avoiding revisits after confirming a color is absent.Source: §G.1 · p. 27

G.2 NOISY KNOWLEDGE

  • With existing knowledge, all three required options are selected and the purchase succeeds.Source: §G.2 · p. 28
  • Noisy guidance that treats three required options as completed when only two appear in recent history leads to a stalled search and no purchase.The example notes that two visible selections do not establish that the third option was selected.Source: §G.2 · p. 28

Related papers

Loading 2610.30790v1…