PaperlayerView exact revision on arXiv

Source-linked AI summary

SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization

Jiarui Yuan, Tailin Jin, Weize Chen, Zeyuan Liu

arXiv:2602.04811v2cs.CLcs.AIcs.LG

TL;DR

SE-Bench addresses the lack of a clean measurement for whether agents internalize novel knowledge rather than relying on prior knowledge or difficult reasoning. It obfuscates NumPy into a pseudo-novel package and evaluates simple coding tasks without documentation, revealing distinct outcomes for Closed-Book SFT, RL, and self-play. The benchmark supports knowledge internalization as a diagnostic capability for self-evolving agents.

  • Problem

    Existing evaluations confound knowledge internalization with pre-training exposure and reasoning complexity, limiting rigorous measurement of this capability.

  • Method

    SE-Bench obfuscates NumPy functions and documentation into randomized identifiers, trains agents with documentation, and tests them on simple undocumented coding tasks.

  • Results

    Closed-Book SFT internalizes knowledge, standard RL fails, and SFT-based self-play succeeds on noisy self-generated tasks.

  • Takeaways & Limitations

    SE-Bench provides a controlled diagnostic platform and unit test for knowledge internalization in self-evolving agents.

  • Takeaways & Limitations

    The study leaves the concrete mechanisms underlying RL's internalization failure as an open direction for future work.

Abstract

from arXiv · show

True self-evolution requires agents to act as lifelong learners that internalize novel experiences to solve future problems. However, rigorously measuring this foundational capability is hindered by two obstacles: the entanglement of prior knowledge, where ``new'' knowledge may appear in pre-training data, and the entanglement of reasoning complexity, where failures may stem from problem difficulty rather than an inability to recall learned knowledge. We introduce SE-Bench, a diagnostic environment that obfuscates the NumPy library and its API doc into a pseudo-novel package with randomized identifiers. Agents are trained to internalize this package and evaluated on simple coding tasks without access to documentation, yielding a clean setting where tasks are trivial with the new API doc but impossible for base models without it. Our investigation reveals three insights: (1) the Open-Book Paradox, where training with reference documentation inhibits retention, requiring "Closed-Book Training" to force knowledge compression into weights; (2) the RL Gap, where standard RL fails to internalize new knowledge completely due to PPO clipping and negative gradients; and (3) the viability of Self-Play for internalization, proving models can learn from self-generated, noisy tasks when coupled with SFT, but not RL. Overall, SE-Bench establishes a rigorous diagnostic platform for self-evolution with knowledge internalization. Our code and dataset can be found at https://github.com/thunlp/SE-Bench.

1 Introduction

SE-Bench targets knowledge internalization as a core form of self-evolution by separating memorization from prior knowledge and reasoning difficulty. It introduces a controlled benchmark and uses it to reveal how training paradigms affect retention.

  • Motivation: Current evaluations cannot cleanly measure knowledge internalization because prior knowledge and reasoning complexity confound failures.A model may already know supposedly novel information, or fail because the task is difficult rather than because memory is insufficient.
  • Benchmark: SE-Bench obfuscates NumPy functions and documentation into a pseudo-novel package, making tasks impossible without the mapping but trivial with it.Test-time solutions must use the obfuscated package rather than the original NumPy library.
  • Purpose: The benchmark provides a controlled testbed for studying whether SFT and RL genuinely internalize new knowledge.It is designed both as a metric and as an environment for dissecting self-evolution mechanisms.
  • Findings: Closed-Book Training improves retention by removing documentation during parameter updates and forcing external logic into model weights.Reference material available during updates can inhibit long-term retention.
  • Findings: Standard RL fails to fully internalize knowledge, whereas SFT supports internalization and SFT-based self-play can learn from noisy self-generated tasks.The paper identifies PPO clipping and negative gradients as factors affecting RL internalization.

2 SE-BENCH

SE-Bench creates an out-of-distribution package by obfuscating NumPy while preserving its logic, then constructs and verifies coding tasks that test API knowledge without documentation. Its design separates reasoning ability from knowledge retention through controlled training and testing conditions.

  • Motivation: Knowledge internalization is difficult to measure with existing libraries because models may have encountered them during pre-training.Newly released libraries can also become obsolete as model knowledge cutoffs advance.
  • SE-BENCH Design: SE-Bench maps NumPy functions to randomized identifiers while preserving their underlying logic, creating a structurally realistic but training-distribution-alien package.The design aims to guarantee zero-shot impossibility while keeping documented tasks algorithmically trivial.
  • Construction Pipeline: The benchmark is constructed through Obfuscation, Question Generation, and Filtering stages.The pipeline is described as a three-stage process.
  • Obfuscation: The obfuscation stage implements ZWC as a wrapper around 268 common NumPy functions with semantically void names such as zwc.kocito.The wrapper functions internally call corresponding NumPy functions, while documentation describes the new package.
  • Task Construction: Claude-4.5-sonnet generates simple coding problems and at least eight test cases from the original NumPy functions, avoiding hallucinated obfuscated syntax.Training uses single-function tasks, while testing includes single-function retention and multi-function compositional generalization.
  • Evaluation: Testing removes API documentation, and AST verification requires passing all tests, using ZWC APIs, and importing no NumPy.The protocol includes special checks against bypass behaviors.
  • Dataset: The dataset contains 1,417 tasks: 718 for training and 699 for testing, including 259 single-function and 440 multi-function test tasks.Both splits maintain broad coverage of the ZWC API surface.
  • Validation: Qwen3-8B exceeds 90% accuracy with standard NumPy, reaches 0% zero-shot accuracy with ZWC, and recovers performance when relevant documentation is provided.The results support reasoning-triviality, prevention of pre-training leakage, and solvability with information.

3 Experiment

Experiments validate SE-Bench as a knowledge-internalization benchmark and compare memory-based, SFT, RL, and hybrid strategies. Closed-SFT is the key successful parameter-update approach, while most alternatives remain incomplete or fail.

  • Design Validation: Table 1 uses Qwen3-8B Pass@64 to test whether standard NumPy tasks are reasoning-trivial and whether ZWC prevents pre-training leakage.The in-context condition provides a solvability ceiling when documentation is available without training.
  • Baselines: Memory-based methods achieve non-trivial results across model sizes, but remain far from perfect despite Expel leading the Multi-Function split.The results suggest autonomous memory management remains limited.
  • Parameter Optimization: Only Closed-SFT and the hybrid Closed-SFT-RL achieve success among parameter-update methods; all other methods fail completely.Removing documentation during updates distinguishes Open-SFT from Closed-SFT and forces logic encoding into model weights.
  • Parameter Optimization: The Open-Book Paradox shows that identical training trajectories can produce different outcomes depending on whether documentation remains available during parameter updates.The contrast is presented as evidence of internalization rather than simple alignment to the test-time prompt distribution.

4 Analysis and Insight

SE-BENCH isolates knowledge internalization from prompt dependence and reasoning difficulty, revealing distinct roles for SFT, RL, and self-play. Closed-book SFT internalizes knowledge, standard RL struggles to create foundational memories but can improve their use, and self-play succeeds when paired with SFT.

  • RQ1: SFT and Internalization: SE-BENCH tests whether agents internalize knowledge by separating documentation-free recall from task reasoning and prompt consistency.The benchmark’s analysis examines whether parameter updates create robust knowledge rather than transient context dependence.
  • RQ1: SFT and Internalization: Closed-SFT outperforms Open-SFT even when documentation is available at test time, confirming that withholding documentation during training drives parametric internalization.This rules out prompt-distribution consistency as the explanation for Closed-SFT’s advantage.
  • RQ2: RL and Internalization: Closed-RL achieves zero performance when rollouts use documentation but training does not, whereas an SFT-like RL configuration recovers Closed-SFT performance.The ablation indicates that the RL framework itself is not the issue; PPO-style clipping and negative advantages are implicated instead.
  • RQ2: RL and Internalization: PPO clipping suppresses the large probability shifts needed for new vocabulary, while negative advantages can erase tentative associations during early memorization.These mechanisms are presented as the two specific safety mechanisms isolating standard RL’s internalization failure.
  • RQ3: Self-Play and Internalization: Closed-SFTself reaches 22.5% performance while Absolute Zero fails completely, showing that self-generated data can support internalization when optimized with SFT.The result attributes standard self-play’s failure to RL optimization rather than to the self-play paradigm or the model’s inability to generate valid data.
  • RQ4: Knowledge Evolution from SFT to RL: The hybrid Closed-SFT-RL achieves state-of-the-art performance among parameter-update methods, suggesting RL amplifies knowledge after SFT establishes foundational internalization.SE-BENCH is positioned as a controlled environment for studying these distinct mechanisms.
  • RQ4: Knowledge Evolution from SFT to RL: RL improves utilization after SFT by reducing ZWCArray Attribute Hallucination from 37.0% to 10.0%, but does not significantly reduce function-name or signature errors.The error shift suggests RL replaces uncertain calls with valid alternatives while leaving fundamental memory errors unresolved.
  • Discussion: Connections to Recent Advancements: Removing documentation during SFT is described as a functional requirement that forces external logic to be compressed into model parameters.The authors connect this mechanism to knowledge-internalization strategies such as Deliberative Alignment.

5 Related Work

Prior work advances agent capabilities through memory, retrieval, refinement, and tool use, but does not adequately assess whether agents internalize knowledge into underlying capabilities.

  • LLM-powered agents perform deep research and code engineering tasks, but continual improvement still relies largely on scaling human-annotated data.This dependence is described as inefficient and costly.
  • Self-evolving agents are designed to learn continuously from their own trajectories, with memory engineering offering one methodological approach.
  • Existing evaluations emphasize code generation, search, and tool use rather than the foundational ability to memorize and leverage knowledge.Current memory benchmarks often rely on weakly verifiable user feedback.

6 Conclusion

SE-BENCH is presented as a diagnostic testbed for knowledge internalization, revealing distinct effects of training access, reinforcement learning, and self-play. The authors position it as a unit test for whether agents genuinely learn from experience.

  • SE-BENCH obfuscates NumPy to test whether agents internalize knowledge rather than rely on familiar APIs.
  • The Open-Book Paradox indicates that accessible reference knowledge inhibits retention, whereas Closed-Book Training supports true retention.
  • The RL Gap shows that standard reinforcement learning can optimize behavior without fully internalizing new facts.
  • Self-play can distill knowledge from noisy self-generated curricula when seeded with supervised fine-tuning.
  • The benchmark is proposed as a critical unit test for future self-evolving agents’ ability to learn from experience.

A Limitation

The paper identifies scope and interaction-setting limitations in SE-BENCH. Future extensions should test broader domains and richer, longer-horizon interactions to examine the generality of the findings.

  • SE-BENCH currently focuses on code generation, providing a precise and verifiable setting but limiting demonstrated domain coverage.
  • Extending the benchmark to factual knowledge and other domains would test whether the findings generalize to more diverse, realistic settings.
  • Most evaluated baselines use single-turn interactions, leaving the role of long-horizon, multi-turn interaction in knowledge internalization underexplored.
  • Broader task coverage and richer interaction settings are identified as opportunities to deepen understanding of knowledge internalization.

B The Effect of Question and Trajectory Diversity

The diversity experiments separately vary question and response diversity under Closed-SFT. Question diversity strongly affects training efficiency and final performance, whereas response diversity has little effect once enough correct responses are available.

  • Experimental design: Closed-SFT varies question counts and responses per question to isolate question diversity from response diversity.
  • Question diversity: Fewer distinct questions reduce training efficiency and final performance, indicating that question diversity is crucial for knowledge internalization.
  • Response diversity: Training efficiency and final performance remain largely consistent across response-diversity levels once a sufficient number of correct responses is reached.
  • Comparison: Question diversity has a substantially larger influence on knowledge internalization than response diversity.
  • Implication: The results prioritize the quality and diversity of questions when studying knowledge internalization in self-evolution.

C Scaling to Different Model Family and Larger Model

Experiments across larger and different model families support the generality of the Open-Book Paradox and RL Gap, while memory-based Expel remains effective. Continued RL can also improve specific API-use behaviors after prior internalization.

  • Expel remains effective on Llama models and the larger Qwen3-30B-A3B, supporting autonomous memory management approaches.
  • Closed-SFT successfully internalizes knowledge, whereas Open-SFT, Open-RL, and Closed-RL fail across different model families and larger-scale models.The authors report this as consistent with the Open-Book Paradox and RL Gap discussed earlier.
  • RL after SFT-Internalized can further improve the degree of knowledge internalization.A case study analyzes how RL contributes to this improvement.
  • After RL, the agent avoids assuming that ZWCArray provides a nonexistent sum() method and instead uses suitable Python built-in operations.
  • In Multiple tests, RL helps the agent explore ZWC APIs and apply them correctly in more complex multi-API scenarios.

E.1 Hyper Parameters of Main Experiment

The experiments use distinct configurations for memory-based, SFT-based, and RL-based methods, with Open and Closed prompts defining whether API documentation remains available during parameter updates.

  • ACE uses temperature 0.6, response length 8192 tokens, up to three sandbox turns per query, independent query insights, and an aggregated skillbook.
  • ExpeL uses Qwen3 8B, 4B, and 1.7B backbones for both policy and insight extraction, with direct generation replacing iterative ReAct.
  • SFT uses batch size 32, temperature 1.0, learning-rate warmup from 1 × 10^-6 to 1 × 10^-5, cosine decay, and context length 16384.
  • RL uses batch size 32, temperature 1.0, fixed learning rate 1 × 10^-6, and GRPO rollout number n = 8.
  • E.3 Difference between Open and Closed: In the Open setting, parameter updates retain API documentation, whereas the Closed setting removes it and forces reliance on internalized parameters.
  • Single tasks use one core ZWC API, while Multiple tasks compose several ZWC APIs and format inputs and outputs as lists.
  • ZWC maps same-named NumPy functions from main and linalg namespaces to different obfuscated names, ensuring one-to-one semantic correspondence.

G Example of Error Types

The paper categorizes failed trajectories into five error types, illustrated with representative examples covering hallucinated APIs, misinterpreted outputs, signature errors, and incompatibility with native Python.

  • The error taxonomy includes ZWCArray Attribute Hallucination, Return Value Misinterpretation, Parameter Signature Misalignment, Native Python Incompatibility, and ZWC Function Hallucination.The examples were randomly sampled from failed evaluation trajectories and manually verified.
  • ZWCArray Attribute Hallucination is illustrated in Table 15.
  • Return Value Misinterpretation is illustrated in Table 16.
  • Parameter Signature Misalignment is illustrated in Table 17.
  • Native Python Incompatibility and ZWC Function Hallucination are illustrated in Tables 18 and 19, respectively.
Loading 2602.04811v2…