Source-linked AI summary
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses
Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao
TL;DR
Current evaluations of skill evolution can confound capability abstraction with solution leakage, obscuring what downstream performance measures. EngramBench enforces capability overlap without solution overlap and tests agents on interactive transfer tasks. Across the reported evaluations, skill frameworks provide a positive learning delta and an execution compass, while exact implementation remains constrained by base-model reasoning.
Problem
Current skill-evolution evaluations confound genuine capability abstraction with solution leakage, making downstream performance difficult to interpret.
Source: §1 · p. 2Method
EngramBench pairs feedback-rich learning tasks with isolated, multi-turn transfer projects under a two-phase protocol that freezes the generated skill bank after learning.
Source: §1 · pp. 2–3Results
Skill-evolution frameworks produce a positive learning delta on complex, multi-turn interactions, while static skill banks do not remove the base model’s last-mile implementation bottleneck.
Source: §1 · p. 3, §8 · p. 13Takeaways & Limitations
Static skill banks serve as an execution compass that can steer agents away from token-heavy trial-and-error, while EngramBench measures cross-domain capability transfer.
Source: §8 · p. 13The deep ablation study covers 48 controlled runs across four representative tasks because a run can consume up to 66 million tokens and requires manual trajectory auditing.
Source: §7 · p. 13
Abstract
from arXiv · showhide
While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning, complex state management, and continuous cross-domain abstraction. However, current evaluations of skill evolution in autonomous agents suffer from a critical identifiability problem: they structurally confound genuine capability abstraction with rote solution leakage (i.e., copying highly similar code from historical training data). To resolve this, we introduce EngramBench, a rigorous, capability-grounded benchmark governed by the strict axiom of capability overlap without solution overlap. Comprising 30 diverse learning tasks and 13 unseen transfer tasks, EngramBench challenges agents to navigate interactive, multi-hour development cycles driven by LLM-simulated users. Our extensive evaluation across 48 multi-hour execution trajectories -- corroborated by human-expert validation -- reveals a profound insight into procedural memory. We demonstrate that static skill banks do not magically bypass the "last mile" of exact code implementation, which remains bottlenecked by the base model's inherent reasoning limits. However, they serve as an indispensable execution compass. By navigating agents away from catastrophic, token-heavy trial-and-error, genuine capability abstraction slashes redundant context bloat and reduces overall coding time by over 55%. Ultimately, EngramBench shifts the evaluation paradigm from trivial pattern matching to the verifiable measurement of deep, cross-domain capability transfer.
1 Introduction
EngramBench addresses the confounding of genuine skill abstraction with solution leakage by enforcing capability overlap without solution overlap. Its 30 feedback-rich learning tasks and 13 isolated, interactive transfer projects provide a setting for measuring whether skills support sustained execution.
1 Introduction
- Current evaluations confound genuine abstraction with solution leakage or irrelevant history, making failures difficult to diagnose.The design rule is to preserve shared high-level challenges while separating task-specific implementations.Source: §1 · p. 2
- EngramBench introduces a capability-grounded benchmark governed by capability overlap without solution overlap.The boundary excludes project-specific implementations, constants, and exact API paths while expecting reusable workflows and engineering principles.Source: §1 · p. 2
- The benchmark provides 30 learning tasks across six business domains and 13 transfer projects with no shared business logic or APIs.Learning histories include feedback-rich trajectories, while transfer projects require combining skills across domains.Source: §1 · pp. 2–3
- The benchmark’s many-to-many design treats experience from different learning tasks as reusable skills supporting multiple transfer tasks, not direct reuse of project code.The capability relationships shown are illustrative.Source: §1 · p. 2, Figure 1 · p. 3
- Transfer projects use multi-turn interaction with an LLM-simulated user, who asks follow-up questions, checks requirements, and rejects incomplete drafts.The two-phase protocol freezes the generated skill bank after learning and standardizes the base model and learning trajectories across methods.Source: §1 · pp. 3–4
- A skill-evolution framework produces a positive learning delta, guiding agents through complex, multi-turn user interactions where base models struggle without prior knowledge.The evaluation establishes a baseline for future research.Source: §1 · p. 3
2 Related Work
Prior programming evaluations range from isolated code generation to repository-level issue resolution, but neither alone establishes whether agents can abstract reusable skills. EngramBench targets that gap by separating cross-domain capability transfer from retrieval of similar historical code.
2.1 Code Generation vs. Software Engineering
- Short-horizon benchmarks such as HumanEval (Chen et al. 2021) and MBPP (Austin et al. 2021) assess standalone function synthesis, while software engineering requires iterative, feedback-driven work across modules.EngramBench addresses the shift toward sustained execution through long, interactive development lifecycles.Source: §2.1 · p. 4
2.2 Current Agent Benchmarks and Their Limitations
- Repository-level benchmarks such as SWE-bench (Jimenez et al. 2023) challenge agents with real-world GitHub issues, but intra-repository skill-evolution tasks can reward retrieval of similar code and APIs.EngramBench instead enforces cross-domain sequences with capability overlap but no solution overlap.Source: §2.2 · p. 4
2.3 Procedural Memory and Skill Evolution in LLMs
- Procedural-memory frameworks distill trajectories into reusable guidelines or code, but their efficacy remains confounded when evaluations do not separate abstraction from the availability of similar in-domain data.EngramBench provides infrastructure to evaluate those frameworks under capability-grounded transfer.Source: §2.3 · pp. 4–5
3 Constructing EngramBench
EngramBench is a requirement-driven, long-horizon sandbox for testing cross-domain engineering skill transfer. It pairs diverse learning and transfer tasks with full-stack constraints and interactive, multi-layer feedback.
3 Constructing EngramBench
- EngramBench uses requirement-driven projects to test agents building complete business systems across cross-layer dependencies, environmental constraints, and interactive feedback.Unlike benchmarks built around isolated GitHub issues, its projects start from lightweight API contracts.Source: §3 · p. 5
3.1 Task Curation & Domains
- 30 learning tasks span six business domains, with recurring engineering capabilities and feedback-rich trajectories supporting abstraction and transfer.The dataset archives 583 interactive turns, and human experts verified task feasibility and engineering utility.Source: §3.1 · p. 5, Figure 2 · p. 6
- Representative transfer requirements are linked to reusable mechanisms and concrete implementation, repair, and verification episodes in learning histories.These mechanism-level mappings trace capability overlap through recorded trajectories rather than task-name similarity.Source: §3.1 · p. 5, Table 1 · p. 6
3.2 Codebase Complexity and Cross-Layer Implementation
- Transfer tasks require full-stack implementations spanning database migrations, background dispatchers, OpenAPI integrations, and browser-observable UI states.CapacityLease also requires capacity conservation, atomic admission, concurrent-race handling, and crash recovery.Source: §3.2 · p. 7
3.3 Environment Capabilities and Multi-Layered Feedback
- EngramBench provides an isolated, stateful OCI-compliant Docker sandbox with real-world execution feedback, simulated-user multi-turn progress checks, and a two-stage evaluation pipeline: public interface checking followed by frozen-code hidden testing.Source: §3.3 · p. 7
4 Rigorous Evaluation Protocol
The protocol aims to isolate skill abstraction from test-set or solution leakage through granular scoring, adversarial hidden tests, and physical separation of learning and transfer environments.
4 Rigorous Evaluation Protocol
- Physical isolation boundaries underpin a multi-dimensional protocol intended to distinguish capability gains from context matching or test-set overfitting.Source: §4 · p. 8
4.1 The Confirmed-Case Pass Fraction (CCPF)
- CCPF measures the fraction of applicable hidden cases passed, offering a finer-grained view than binary task success.Applicable cases are equally weighted; unresolved, crashed, or unreached cases remain in the denominator, and agent-authored tests are ignored.Source: §4.1 · p. 8
4.2 Multi-Dimensional Verification and Hidden Stress Testing
- Hidden evaluators stress-test five engineering dimensions, including contracts, concurrency, recovery, UI consistency, and operational compatibility.CapacityLease checks 64 concurrent requests for one lease and event, tests worker recovery after SIGKILL, and rejects tests that pass with broken dependencies.Source: §4.2 · pp. 8–9
4.3 Anti-Cheating and Sandboxing Protocols
- The two-phase protocol freezes a hashed skill bank, then evaluates transfer tasks in an independent environment without access to learning workspaces or histories.Skill evolution is disabled during transfer execution, and hidden evaluators are mounted only after the public interface check and commit freeze.Source: §4.3 · p. 9
5 Experimental Setup
The experiment compares skill-bank configurations across audited long-horizon transfer runs, measuring functional correctness alongside computational cost and agent behavior.
5 Experimental Setup
- 48 audited execution trajectories come from four transfer tasks, four skill configurations, and three independent trials per configuration-task pair.The coding agent is GPT-5.5 Medium, while DeepSeek V4 Pro simulates the user under fixed requirements and a frozen plan.Source: §5 · p. 9
5.1 Baselines: Native Skills vs. Ablations
- Four configurations compare zero-shot behavior with complete memory, memory distilled from filtered histories, and an exploratory core-procedure ablation.S-Full-20 contains 20 consolidated skills; S-Ablated-20 uses histories with 69.66% less raw text, while Coreless-3 retains three basic interface skills.Source: §5.1 · p. 10
5.2 Evaluation Metrics
- Evaluation reports micro-average CCPF, total tokens, pure coding-agent development time, and per-turn behavioral actions.Resource reductions are measured relative to the No Skills baseline.Source: §5.2 · p. 10
6 Experimental Results & Analysis
Across 48 runs, the analysis finds that static skills can improve correctness and sharply reduce execution costs, but they do not remove the base model’s last-mile implementation bottleneck. Ablations and behavioral audits link effective transfer to abstracted engineering capabilities and show how their absence coincides with costly trial-and-error.
6.1 Aggregate Performance: The “Last Mile” Bottleneck
- S-Full-20 improved aggregate CCPF by 4.62 percentage points, but gains varied by task, including +14.81 pp for EscrowGuard and a regression for CapacityLease.The comparison uses 12 runs each for No Skills and S-Full-20.Source: §6.1 · p. 11, Table 2 · p. 11
6.2 The Efficiency Paradigm: Memory as an Execution Compass
- 55.3% less total coding time and 80.91% fewer non-cached input tokens accompanied S-Full-20, while completion turns fell only 1.85%.The reported saving was over 104 minutes per run; reduced external wait times accounted for only about 6 minutes.Source: §6.2 · pp. 11–12, Table 3 · p. 11
6.3 Ablation Insights: Capability Trumps History Volume
- Despite 69.66% less historical text, S-Ablated-20 achieved the highest CCPF (52.60%), with coding time 8.6% higher than S-Full-20.The authors conclude that a substantially reduced evidence base can sustain transfer when distillation abstracts the correct engineering principles.Source: §6.3 · p. 12, Table 4 · p. 12
- Coreless-3 fails to preserve S-Full-20’s correctness–time profile: CCPF drops to 48.36%, and average coding time exceeds 190 minutes, worse than the No Skills baseline.Coreless-3 removes targeted core software-engineering skills from the supplied memory.Source: §6.3 · p. 12
6.4 Behavioral Audit: Decisive Execution vs. Trial-and-Error
- S-Full-20 used 29.8% fewer patch invocations while adding 3.66% fewer lines; Coreless-3 had 68.97% more tool invocations and 131.76% more reasoning tokens than S-Full-20.Coreless-3 generated nearly 700k reasoning tokens, 14.76M characters of error logs, and 26 context compactions; the authors characterize this as blind exploration.Source: Table 5 · p. 12, §6.4 · p. 13
7 Limitations and Future Work
The evaluation is bounded by computational scale and a restricted experimental setup. The authors identify broader task coverage, dynamic memory retrieval, and additional model architectures as future work.
Computational Scale
- A single run can consume up to 66 million tokens, limiting the deep ablation study to 48 controlled runs across 4 representative tasks.Scaling the high-resolution audit to all 13 transfer tasks remains ongoing.Source: §7 · p. 13
Evaluation Scope and Infrastructure
- The study evaluates a strictly frozen skill bank, omitting dynamic memory retrieval, and grounds its empirical findings in GPT-5.5 Medium.The authors identify both as boundaries of the current evaluation.Source: §7 · p. 13
Future Work
- Future work will test diverse open-weights architectures and enforce stricter, air-gapped container boundaries for hidden evaluators.Source: §7 · p. 13
8 Conclusion
EngramBench addresses the confounding of capability abstraction and solution leakage with a benchmark that requires capability overlap without solution overlap. Across 48 multi-hour trajectories, static skill banks guide execution and reduce coding time, while exact implementation remains constrained by base-model reasoning.
8 Conclusion
- EngramBench addresses the confounding of genuine capability abstraction with rote solution leakage through capability overlap without solution overlap.Source: §8 · p. 13
- By navigating agents away from token-heavy trial-and-error and redundant code modifications, capability abstraction reduces context bloat and overall coding time by over 55%.Static skill banks guide execution but do not bypass the last mile of exact code implementation, which remains bottlenecked by base-model reasoning limits.Source: §8 · p. 13
- EngramBench shifts agent evaluation from pattern matching toward verifying deep, cross-domain capability transfer and efficient engineering.Source: §8 · p. 14
A Learning Evidence Coverage Across the Transfer Set
Table A.1 maps requirements across the transfer tasks to reusable mechanisms, learning episodes, and evaluation dimensions. Its coverage is task-level rather than an exhaustive audit of every README clause, and a mapping alone does not establish skill extraction or successful use.
A Learning Evidence Coverage Across the Transfer Set
- Table A.1 links each public requirement to a reusable engineering mechanism, a recorded learning episode, and an evaluation dimension across all 13 transfer tasks and a shared cross-layer mapping.Coverage is complete at the task level, not exhaustive across README clauses; a mechanism-level link does not establish that a skill bank extracted or an agent used the mechanism successfully.Source: A Learning Evidence Coverage Across… · p. 15, Table A.1 · p. 15
- The evidence notation distinguishes repair loops from implementation followed by verification, and separates observed HTTP responses from assertion-based verification.A repair loop records failure or an incorrect result, a change, and subsequent executed verification.Source: A Learning Evidence Coverage Across… · p. 15
- Case IDs refer to the evaluator version audited on 30 September 2026, not historical pass results.Source: A Learning Evidence Coverage Across… · p. 15