PaperlayerView exact revision on arXiv

Source-linked AI summary

GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks

Leijun Zhou, Zhihao Liu, Xiang Qu, Chenxu Liu, Yifei Liu, Yanke Yu, Jingzhe Xu, Xuejun Wu, Buyue Qian, Xi Chen, Yaowei Zheng, Junhao Hu

arXiv:2608.03764v1cs.AI

TL;DR

Existing benchmarks offer limited coverage of valuable enterprise workflows and weak attribution of gains from training experience. GDPevo addresses this with rule-hybridized task splits and automated generation, finding that self-evolution improves held-out accuracy by up to 16.44 percentage points.

  • Problem

    Existing evolution-native benchmarks provide limited coverage of economically valuable enterprise workflows and often lack train–test relationships that attribute gains to transferable experience.

  • Method

    GDPevo decomposes workflows into business rules, recombines them across held-out tasks, and uses an automated pipeline to evaluate transferable self-evolution.

  • Results

    Every evolved agent combination improves over its base, with held-out accuracy gains of up to 16.44 percentage points.

  • Takeaways & Limitations

    GDPevo supports reproducible, economically grounded evaluation of whether agents learn from prior experience and transfer knowledge to new tasks.

  • Takeaways & Limitations

    The experiments evaluate only non-parametric skill-based evolution, although GDPevo supports other persistent-state forms.

Abstract

from arXiv · show

Agent self-evolution updates an agent's persistent state from prior experience and reuses it to solve related tasks more effectively. Evaluating self-evolution is difficult: existing benchmarks provide limited coverage of economically valuable task domains, do not always design training and test tasks such that test-time gains can be attributed to training experience, and remain vulnerable to data contamination. We present GDPevo, an evolution-native benchmark grounded in GDP-related enterprise workflows, together with the fully automated data pipeline that generates it. Its core mechanism, rule hybridization, decomposes each enterprise workflow into atomic business rules, distributes subsets of these rules across training tasks, and recombines them in held-out test tasks so that test-time gains are attributable. GDPevo spans CRM, ERP, finance, healthcare, legal, and data-centric workflows. Its V1 release contains 120 tasks in 12 groups, with five training and five held-out test tasks per group. Full automation enables the pipeline to expand the suite to 240 tasks in 24 groups (V2) within two days, providing a practical response to contamination. Using GDPevo, we evaluate four agents, each comprising a harness and a model, under four supervision types. Self-evolution consistently improves held-out accuracy by up to 16.44 percentage points. But the best evolved agents remain far below the fully informed oracle ceiling of 91.6%, indicating that the self-evolution ability of current agents remains far from fully realized. We publicly release the pipeline, benchmark, and full evaluation results at https://github.com/Prism-Shadow/GDPevo.

1 Introduction

GDPevo is an evolution-native benchmark for testing agent self-evolution on GDP-related enterprise workflows, using rule hybridization to make training-to-test generalization concrete and attributable. Its automated pipeline spans diverse domains, supports reproducible evaluation, and enables systematic comparison across agents and supervision types.

  • Benchmark scope: GDPevo evaluates self-evolution on GDP-related tasks spanning CRM, ERP, finance, healthcare, legal, and data-centric workflows.It targets economically valuable enterprise workflows governed by business-specific rules and deterministic correctness criteria.
  • Benchmark design: Rule hybridization decomposes workflows into atomic business rules, distributes subsets across five training tasks, and recombines them across five held-out test tasks.This design requires agents to infer reusable rules during training and apply them compositionally at test time, making gains attributable to training experience.
  • Scale and release: 120 tasks in 12 groups comprise V1, while full automation expands GDPevo to 240 tasks in 24 groups for V2 within two days.Each V1 group contains 5 training and 5 held-out test tasks.
  • Evaluation: Deterministic rule-based grading makes scores reproducible and traces failures to violated rules, while evaluation also reports tokens, agent turns, and monetary cost alongside accuracy.The pipeline converts rubric points into code-based test cases rather than relying on an LLM judge.
  • Experimental setup: Four agents are evaluated under four supervision types: no-evolution, fewshot, reflect, and self, with skill-based evolution used throughout.Fewshot uses gold answers, reflect uses scores from the agent’s own attempts, and self uses only the training questions.

2 Background and Related Work

Agent self-evolution can update different persistent states, but this paper evaluates skill-based non-parametric evolution while keeping GDPevo agnostic to the state representation. Prior benchmarks vary in whether they explicitly construct related train–test tasks, motivating GDPevo’s rule-based transfer evaluation and broader enterprise focus.

  • Persistent State and Evolution: Agent behavior depends on model parameters θ and context C, which can be changed through parametric learning or prompt, context, and harness engineering.The paper frames behavior as sampling from P(y | C, θ).
  • Persistent State and Evolution: The experiments use skills: a skill creator learns from training tasks, and the resulting skill library transfers to test tasks.This setup does not limit GDPevo, which can evaluate memories, prompts, skills, harness code, or model parameters.
  • Evolution Benchmarks: Evolution benchmarks either omit explicit train–test splits or construct them explicitly to test transfer from prior experience to unseen related tasks.EdgeBench and RSIBench exemplify the first category, while other benchmarks define explicit train–test relationships.
  • Evolution Benchmarks: Train–test tasks must be distinct but related: identical tasks risk memorization, whereas unrelated tasks provide no relevant training support.Existing examples include EvoAgentBench’s 528/267 split, SkillFlow’s 166 tasks across 20 workflow families, and SEA-Eval’s success and token-consumption metrics.
  • GDPevo’s Positioning: Existing benchmarks offer limited coverage of difficult enterprise workflows and often derive transferable relationships post hoc, whereas GDPevo constructs them from the ground up using diverse business rules.GDPevo is explicitly designed as an evolution-native benchmark for transfer from experience.
  • Benchmark Requirements: Evolution benchmarks should resist contamination, use deterministic rule-based graders, and track tokens, agent turns, and monetary cost.These properties complement enterprise coverage and train–test splits designed around transferable rules.

3 The GDPevo Benchmark

GDPevo evaluates self-evolution through task groups that pair shared enterprise environments with training and held-out test tasks. Its automated construction uses rule hybridization, calibration, and review to create diverse, measurable workflows whose test-time gains reflect learned business rules.

  • Benchmark unit: Each task group contains one shared business environment, 5 training tasks, and 5 held-out test tasks; accuracy gain over no training measures improvement.Agents process training tasks, update persistent state, and then solve held-out tasks.
  • Automated construction: The pipeline generates task groups from seed scenarios and associated examples, combining one shared environment with 10 related tasks split into 5 training and 5 held-out test tasks.Seed scenarios originate from existing domain-specific benchmarks, while agent-based generation expands them into concrete task groups.
  • Scale and contamination: 120 tasks in 12 task groups comprise V1, while full automation produced another 120 tasks in 12 task groups for V2 in under two days.Rapid regeneration is intended to counter data contamination when a released version risks exposure.
  • Rule hybridization: Rule hybridization decomposes business logic into atomic, independently checkable rules, scatters subsets across training tasks, and recombines them across held-out tests.Because the hidden rules are absent from world knowledge, supervision during evolution helps agents infer and record them before generalization.
  • Quality control: Calibration targets roughly 40–60% for test tasks without training and about 0.1–0.3 improvement after fewshot evolution, while acceptance requires at least 5 of 6 reviewer votes.Reviewers inspect completeness, evaluation format, and rubric design.

4 Evaluation

GDPevo evaluates self-evolution on 240 rule-recombination tasks using deterministic grading across agent, harness, model, and supervision configurations. Fewshot evolution produces the strongest accuracy gains and can reduce or preserve costs, but evolved agents remain well below the 91.6% oracle ceiling and often transfer poorly across domains.

  • Dataset: The evaluation uses 240 tasks in 24 groups, each containing five training and five test tasks that recombine hidden business rules.Training tasks expose rule fragments, while held-out tests recombine those rules.
  • Metrics: GDPevo scores tasks with deterministic, rule-based graders and uses aggregate comparisons, per-task rubric breakdowns, and transfer heatmaps.The grading pipeline converts rubric points into code-based test cases rather than relying on an LLM judge.
  • Agents and supervision: The study compares Codex and Claude Code harnesses with GPT-5.5, Opus-4.8, GLM-5.2, and DeepSeek-V4-Pro-Preview under four supervision types.The harness controls tools and execution, while the model supplies the reasoning policy.
  • Accuracy and cost: 2.59 to 16.44 pp: every agent surpasses its base accuracy with supervision, while fewshot achieves the highest accuracy for all four agents.DeepSeek-V4-Pro-Preview fewshot reaches 48.79% versus 49.37% for GPT-5.5 base at approximately 1/28 of the amortized end-to-end cost.
  • Accuracy and cost: 15.14 pp: GPT-5.5 fewshot raises accuracy while reducing test-time cost by 20.88%, with amortized cost nearly matching base.Its amortized cost is $1.296 per task versus $1.294 for base; Opus 4.8 has essentially unchanged test-time cost (−0.57%).
  • Limits and transfer: 91.6%: the fully informed oracle ceiling remains substantially above the best evolved configuration, while fewshot transfer across domains is often harmful.Five of six off-diagonal transfer cells are negative, indicating source-domain overfitting.

5 Conclusion

GDPevo is an evolution-native benchmark with a fully automated pipeline for testing whether agents learn from prior experience and transfer that knowledge to new GDP-related tasks. Its rule-hybridization design makes this generalization concrete and test-time gains attributable.

  • Benchmark purpose: GDPevo evaluates whether agents learn from prior experience and transfer that knowledge to new tasks.It is designed as an evolution-native benchmark for agent self-evolution.
  • Benchmark scope: GDPevo is the first benchmark to evaluate agent self-evolution on GDP-related tasks.The benchmark targets GDP-related enterprise workflows.
  • Evaluation design: Rule hybridization makes generalization from training to test concrete and test-time gains attributable.GDPevo is the first benchmark to introduce this mechanism.
  • Data pipeline: GDPevo includes a fully automated data pipeline that enables rapid benchmark regeneration.The supplied passage identifies rapid regeneration as a consequence of full automation.

A Reproducibility and Artifact

GDPevo is designed for reproducibility through public release of its pipeline, benchmark, graders, evaluation runs, and task-level artifacts. Its evaluation workspace is natural-language-driven and packages training, test, and scoring resources for each task group.

  • Public Release: The full GDPevo pipeline, 240-task benchmark, rule-based graders, and released evaluation runs are publicly available.The release includes the benchmark infrastructure and complete evaluation artifacts.
  • Task Artifacts: Each task group includes a shared environment, 5 training tasks, 5 held-out test tasks, and per-task evaluation scripts.The scripts contain rubrics enumerating the weighted scoring points described in Section 4.1.
  • Evaluation Workspace: The evaluation workspace is natural-language-driven, using Markdown skills and guides to launch experiments.An orchestration agent opens the skills-and-guides folder before launching an experiment.

B Task Group Details

GDPevo’s released suite comprises 24 task groups spanning six enterprise domains, with training and held-out test tasks in every group. Groups distribute atomic business rules across training tasks and recombine them in tests, enabling independently checkable, traceable scoring.

  • Suite composition: 240 tasks span 24 groups across CRM, ERP, Finance, Healthcare, Legal, and Data Analysis, with five training and five held-out test tasks per group.V1 contains the CRM, ERP, and Finance groups; V2 adds Healthcare, Legal, and Data Analysis groups.
  • Rule hybridization: The CRM lead-capture group distributes subsets of sponsor-status priority, blacklist exclusion, contact deduplication and normalization, and follow-up scheduling rules across training tasks, then recombines them in tests.Per-task rubrics independently check each atomic rule, supporting traceable and reproducible scoring.

C Evaluation Results Details

The evaluation details provide complete leaderboard results across four supervision types and separately quantify one-time skill-generation overhead. The released machine-readable reports preserve task-group metrics and generated skill artifacts underlying both tables.

  • Leaderboard results: Table 3 reports model–harness composition, reasoning setting, accuracy, uncertainty, lift over base, and test-time efficiency for all four supervision types.Each test task runs three times; ACC and STD use task-group macro-averaging, while lift is the absolute percentage-point change from the matching base row.
  • Skill-generation overhead: Table 4 reports one-time skill-generation cost separately from Table 3’s test-time efficiency measures.Values average three independent generation attempts and task groups 001–024; costs are in USD and tokens are in millions.
  • Skill-generation overhead: Generated skills are reused across five held-out tasks per group, making amortized per-test-task overhead one fifth of each reported value.The base supervision type has no skill-generation stage.
  • Machine-readable results: Released aggregates, structured reports, task-group metrics, and generated skill artifacts are maintained in the public experiment board and report directories.These materials support construction of both tables.

D How Is the Fully Informed Oracle Ceiling Obtained?

The fully informed oracle ceiling pairs Codex with GPT-5.5 and a non-expert operator who receives complete training-task evidence while solving held-out tasks in the benchmark environment. It evaluates this condition across all 24 task groups using three attempts per held-out task and the main deterministic grader.

  • Oracle construction: The oracle pairs the Codex harness with GPT-5.5 at xhigh reasoning effort and one non-expert human operator per attempt.Multiple operators participate across the evaluation, reducing dependence on any single operator’s familiarity or workflow.
  • Information condition: The condition estimates performance when evidence for recovering task-group rules is directly available, while held-out tasks still must be completed in the benchmark environment.Training materials expose complementary hidden business logic, including output forms, validity constraints, records, and available tools.
  • Oracle construction: Each attempt uses a fresh context in which the operator and Codex receive complete packages for the five corresponding training tasks.Packages include inputs, standard answers, notes, deterministic evaluators, rubrics, supporting files, and declared dependencies, alongside the current test input and running environment access.
  • Evaluation protocol: 360 attempts cover all 24 task groups, with five held-out tasks per group and three independent attempts per task, scored by the main deterministic grader.Pairs must still find relevant records, determine applicable demonstrated rules, perform required calculations, and produce valid structured outputs.

E Breakdown and Diagnostic Views … F.2 TG014: Recombining Rules across Insurance Workflows

The paper diagnoses self-evolution through rubric- and task-group analyses, then uses case studies to show broad operational transfer and rule recombination across insurance workflows. These gains remain configuration-dependent but arise from skills that encode selective, evidence-grounded decisions rather than repeated templates.

  • E Breakdown and Diagnostic Views: Figure 4 compares base and fewshot rubric satisfaction for GPT-5.5 with Codex, exposing which individual rules become more consistently satisfied.The diagnostic view reports full-credit counts across three independent attempts for test-local rubrics and task groups.
  • E Breakdown and Diagnostic Views: Fewshot raises Opus-4.8 with Claude Code macro accuracy from 50.63% to 67.07% and outperforms base on 23 of 24 task groups.Reflect and self improve 20 and 16 groups, respectively; Opus-4.8 has the highest fewshot macro average among the four configurations.
  • F Evaluation Case Studies: The case studies examine four task groups through generated skills, traces, and deterministic rubric outputs, covering positive, negative, and limited transfer patterns.Because the cases use different model–harness configurations, they are diagnostic examples rather than controlled model comparisons.
  • F.1 TG024: Broad Transfer of Operational Decision Rules: 29.05 percentage points: GPT-5.5 fewshot improves TG024 held-out accuracy from 51.16% base to 80.21%, while self reaches 52.68% and reflect reaches 66.25%.The generated skill consistently covers 33 of 35 training rules, with the improvement spanning different workflows rather than one repeated template.
  • F.1 TG024: Broad Transfer of Operational Decision Rules: TG024 transfers conditional rules that validate records, construct and classify the population, calculate after fixing it, and act on decision-changing risks.The learned rules exclude cancelled or duplicate candidates, distinguish hard release gates from monitor-only items, and produce NO SHIP when hard gates remain.
  • F.1 TG024: Broad Transfer of Operational Decision Rules: TG024 fewshot averages approximately 15.53 tool calls per test, compared with about 30 for base, indicating more selective evidence gathering.The accuracy gain therefore accompanies reduced search rather than additional exploration.
  • F.2 TG014: Recombining Rules across Insurance Workflows: All three TG014 supervision types improve over the 51.00% base accuracy: fewshot reaches 68.00%, self 68.67%, and reflect 71.33%.The healthcare-insurance tasks require both an operational action and an audit trail distinguishing supporting, missing, and excluded records.
  • F.2 TG014: Recombining Rules across Insurance Workflows: TG014 recombines transferable rules for naming exact missing records, separating parallel programs, grounding decisions in structured factors, and recalculating payments against current benchmarks.These rules apply across changed patients, drugs, claims, and workflows, explaining why all three generated skills outperform base.

F.3 TG018: Negative Transfer from Dropped Scope Conditions … G.4 Keeping Evaluation Comparable and Auditable

The paper shows that self-evolution can fail when generated skills omit the conditions governing learned rules, while reliable benchmark evaluation requires designed transfer relationships, controlled information access, auditable grading, calibration, and execution traces.

  • F.4 TG016: Limited Transfer from Narrow Clinical Rules: TG016 demonstrates limited transfer when decisive clinical rules are already visible in the runtime protocol and the learned skill preserves only a narrow value or omits its branch condition.For hypokalemia, reusing the learned 08:00 follow-up loses the protocol’s urgent-escalation branch triggered by potassium 2.8 and arrhythmia symptoms.
  • F.5 Cross-Case Synthesis: Reliable skill transfer requires reusable training structure, preserved scope, and explicit links from triggering evidence to changed outputs and stopping conditions.Skills that record only surface values or recurring actions are more likely to overfit or interfere with current evidence.
  • G.1 Designing Train–Test Relationships: GDPevo’s automated benchmark generation treats train–test design as a learning problem, using five training and five held-out tasks per group with shared rules but changed entities, evidence, noise, and combinations.Rule hybridization distributes atomic rules across training tasks, while explicit training anchors and test-time changes distinguish transferable experience from direct procedure reuse or unrelated tasks.
  • G.4 Keeping Evaluation Comparable and Auditable: A capable, fixed evaluation agent and independent review are needed because orchestration decisions can affect measured outcomes even when semantic scoring is deterministic.GDPevo separates task-group design, environment construction, task construction, calibration, and final review to avoid shared assumptions and mistakes.
  • G.2 Controlling the Information Available during Evolution: Isolation and supervision-specific information boundaries ensure that agents receive only the files, endpoints, and training signals declared for their supervision type.The four types expose progressively different training information: none for base, inputs and answers for fewshot, inputs alone for self, and inputs plus bounded feedback for reflect.
  • G.3 Validating Grading and Calibration: Deterministic binary rubrics, distinct business outcomes, and calibration of both initial difficulty and moderate evolution gain reduce scoring artifacts and misleading benchmark results.GDPevo uses raw weights from {1, 2, 3}, awards full credit or zero per point, requires 6–10 points spanning at least four outcomes, and validates independent point behavior.
  • G.4 Keeping Evaluation Comparable and Auditable: Unique run identifiers, isolated workspaces, raw traces, protocol-preserving replacement of failed attempts, and separate skill-generation and test-time costs make evaluation comparable and auditable.These controls help distinguish transferable improvement from leakage, template reuse, scoring artifacts, and execution failures while exposing one-time skill investment versus downstream application efficiency.

H Use of Public Source Benchmarks

GDPevo uses public benchmarks only to discover seed scenarios spanning professional workflows, software and customer support, healthcare and legal operations, and data-centric tasks. It then constructs new environments, tasks, answers, and graders, preserving source-informed realism without reusing source examples as evaluation instances.

  • Seed scenario discovery: Public benchmarks serve only as sources for seed scenario discovery across professional, software, healthcare, legal, and data-centric workflows.The released task groups represent 15 public benchmarks, with each source contributing a seed type.
  • Seed scenario discovery: For each scenario, discovery abstracts the work setting, artifacts, and operational constraints from a small number of source examples.A scenario may combine examples from multiple benchmarks.
  • New evaluation instances: The source benchmarks determine scenario coverage and realism, but evaluated instances and train–test relationships are newly constructed for GDPevo.Source examples are not reused as evaluation instances.
  • New evaluation instances: Source examples motivate scenario realism, while GDPevo constructs new environments, train–test tasks, reference answers, and deterministic graders.The pipeline creates five training tasks and five held-out test tasks for each scenario.
Loading 2608.03764v1…