Source-linked AI summary
LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks
Xiao Zhang, Qumeng Sun, Jihao Li, Yiming Ren, Xiang Liu, Haoyang Zhang, Junjie Wang
TL;DR
Long-workflow tasks require models to preserve interdependent constraints and satisfy strict end-to-end verification, while successful execution experience is often lost after one run. This paper introduces EvoMap and LongWoF-Bench to externalize and evaluate verifier-confirmed experience, finding that evolved Gene outperform Skill across seven models and that their utility is associated with experience provenance.
Problem
The paper asks whether verifier-confirmed execution experience can be externalized and reused instead of forcing later models to rediscover workflow strategies and failure modes.
Method
The authors introduce LongWoF-Bench and compare No Context, Skill, and EvoMap Gene, whose construction uses verifier-guided execution, refinement, and distillation.
Results
Evolved Gene outperform Skill across all seven evaluated models by 8.7–15.5 percentage points in strict pass rate on 252 tasks with verifier-confirmed Opus trajectories.
Takeaways & Limitations
Verified execution experience can be externalized and reused across models to improve workflow reliability while reducing repeated exploration.
Takeaways & Limitations
The provenance comparison uses different task groups, so its evidence should be interpreted as provenance-associated rather than a same-task causal ablation.
Abstract
from arXiv · showhide
Large language models are increasingly expected to execute complex workflows whose success depends on maintaining interdependent constraints and producing artifacts that satisfy strict end-to-end verification. Yet successful execution experience is typically lost after a single run, forcing subsequent models to rediscover strategies and failure modes from scratch. We study whether such experience can instead be externalized and reused through EvoMap, where verifier-confirmed execution trajectories are consolidated into structured Gene. To evaluate this setting, we introduce the Long-Workflow Benchmark (LongWoF-Bench), comprising 778 machine-verifiable tasks across code generation, agent-environment synthesis, mathematical reasoning, and rule following. On the 252 tasks with verifier-confirmed Opus trajectories, evolved EvoMap Gene outperform Skill across all seven evaluated models by 8.7-15.5 percentage points, with the gains extending to consumer models from different model families. In contrast, reference-distilled Gene do not exhibit the same advantage, indicating that compact representation alone is insufficient and that Gene utility is closely associated with verified experience provenance. For Claude Opus, Gene reuse also completes 39 more tasks than Skill while reducing solve-time token consumption by 9.9%. Together, these results show that verified execution experience can be retained and shared as a reusable external resource, enabling models to improve long-workflow completion without repeatedly paying the full cost of experience discovery.
1 Introduction
The paper introduces LongWoF-Bench and EvoMap to study whether verifier-confirmed execution experience can be externalized and reused for complex, strictly verified workflows. Evolved Gene outperform Skill across models, while provenance analyses indicate that verified experience—not compact representation alone—drives the advantage.
- EvoMap stores verifier-confirmed execution trajectories as structured Gene for reuse beyond the model and run that produced them.Evolver refines execution until a verifier-confirmed trajectory is obtained, after which execution-critical knowledge is consolidated into a Gene.
- 8.7–15.5 percentage points: evolved EvoMap Gene improve strict pass rate over Skill across all seven evaluated models.The gains extend to consumer models from multiple model families, whereas reference-distilled Gene do not show the same advantage.
- For Claude Opus, Gene solves 39 more tasks than Skill while reducing solve-time token consumption by 9.9%.
- The results associate Gene utility with verified experience provenance rather than compact representation alone.The authors frame verifier-grounded execution experience as reusable external knowledge that reduces repeated exploration.
- LongWoF-Bench contains 778 machine-verifiable tasks spanning code generation, agent-environment synthesis, mathematical reasoning, and rule following.
- The benchmark compares No Context, Skill, and EvoMap Gene under shared task specifications, runtimes, and verifiers.This controls the evaluation so that auxiliary guidance is the principal varying condition.
2 LongWoF-Bench
LongWoF-Bench formalizes long-workflow tasks as dependent execution over public specifications and environments, producing deliverables judged by private machine verifiers. Its 778 tasks share objective end-to-end verification while covering four heterogeneous task families.
- 2.1 Verifiable Long-Workflow Tasks: A verifiable long-workflow task is represented as T = (S, E, Y, V), combining a public specification, environment, deliverable space, and task-specific verifier.The solver uses S and E to perform dependent operations and produce a deliverable y ∈ Y.
- 2.1 Verifiable Long-Workflow Tasks: Long workflows are defined by dependencies among later decisions, constraints, states, and intermediate artifacts, not merely by context length or action count.Locally reasonable choices can still yield invalid final results when interface, ordering, boundary, or artifact requirements are violated.
- 2.1 Verifiable Long-Workflow Tasks: Verification requires complete task fulfillment through executable tests, hidden checks, or normalized exact matching, with all mandatory conditions satisfied.The benchmark excludes tasks lacking objective and reproducible success criteria.
- 2.2 Benchmark Construction and Coverage: 778 tasks cover 341 code-generation, 127 agent-environment-synthesis, 151 mathematical-reasoning, and 159 rule-following tasks.These families emphasize implementations, multi-artifact environments, exact computation, and rule-sensitive decisions, respectively.
- 2.2 Benchmark Construction and Coverage: A representative agent-environment task requires coordinated controller implementation, mode switching, parameter tuning, simulation, and multi-artifact delivery.A private evaluator independently re-executes the submitted system and checks interface compliance and behavioral constraints.
- 2.2 Benchmark Construction and Coverage: The benchmark separates model-visible specifications and assets from evaluator-only artifacts while keeping successful-completion information recoverable from public inputs and interfaces.
3 Reusable Experience with EvoMap Genes
EvoMap turns successful, verifier-confirmed task execution into reusable Genes that can support later runs without replaying the original refinement trajectory. Gene construction preserves execution-critical strategies and can transfer experience across consumer models.
- Evolver produces reusable experience by interacting with a verifier through bounded execution, feedback, and refinement.Failed attempts receive sanitized verifier feedback and are revised until a passing trajectory is obtained or the rollout budget ends.
- Once a trajectory passes verification, GDIv2 distills its execution-critical information into a structured Gene.The Gene preserves successful strategies, prerequisite checks, boundary conditions, and applicable corrections to earlier failures.
- The primary Gene set contains 252 Genes derived from verifier-confirmed Claude Opus execution trajectories.
- Consumers receive public task information and the corresponding Gene without access to the producer trajectory or verifier feedback.This permits reuse without replaying the full trajectory or repeating verifier-guided refinement.
- EvoMap decouples experience producers from consumers, allowing Genes produced by Claude Opus to be reused by Gemini, MiniMax, and other model families.The effectiveness and efficiency of this cross-model reuse are evaluated experimentally.
4 Experimental Setup
The evaluation holds task specifications, runtimes, decoding settings, and private verifiers fixed while varying auxiliary guidance across No Context, Skill, and EvoMap Gene. Seven models and multiple provenance-defined evaluation sets support controlled comparisons.
- Evaluation protocol: Each task is evaluated under No Context, Skill, and EvoMap Gene with the public specification, runtime, decoding configuration, and private verifier held fixed.Each task–condition pair receives one recorded trial, and models receive no additional verifier feedback during inference.
- Models and evaluation sets: Seven models from four model families are evaluated as Gene consumers, with Claude Opus 4.8 serving as the primary Gene producer.The models span Claude, Gemini, MiniMax, and Qwen families.
- Models and evaluation sets: The primary comparison focuses on 252 tasks for which Opus produced verifier-confirmed evolved Gene.Other subsets support additional analyses of Gene provenance and producer overlap.
- Models and evaluation sets: The 252 Opus-evolved and 526 reference-distilled tasks represent different Gene provenance, while the common evolved set contains 180 tasks shared by Opus and Gemini.
- Metrics and statistics: Strict pass rate requires every mandatory verifier check to pass, complemented by solve-time token consumption, model calls, and tokens per passed task.Paired differences use deterministic task-level bootstrap confidence intervals and exact McNemar tests for paired significance.
5 Results
On the shared 252-task evolved subset, verifier-confirmed Gene consistently improve strict workflow completion across consumer models, with benefits associated with provenance and reuse efficiency. Gains vary by workflow family and model, while concrete cases show Gene preserving execution-critical constraints.
- 5.1 Verifier-Evolved Genes Improve Workflow Completion: 62.9% is the seven-model mean strict pass rate with EvoMap Gene, versus 51.2% with Skill and 41.0% with No Context.All conditions use the same tasks and private verifier.
- 5.1 Verifier-Evolved Genes Improve Workflow Completion: 8.7–15.5 percentage points are the Gene-over-Skill gains observed for every evaluated consumer model.The advantage extends beyond the model that produced the Opus trajectories.
- 5.2 Gene Utility Depends on Experience Provenance: Gene outperforms Skill by 8.7–15.5 points on verifier-evolved tasks but trails Skill by 3.3–11.3 points on reference-distilled tasks.The comparison is provenance-associated because the two groups contain different tasks.
- 5.2 Gene Utility Depends on Experience Provenance: Opus-authored Gene outperform Gemini-authored Gene by 4.4–11.7 points on the common 180-task set.The fixed task set controls for task selection, but does not isolate exploration quality, distillation selection, or Gene expression.
- 5.3 Gene Reuse Reduces Repeated Exploration: Gene passes 200 tasks with 723,480 solve-time tokens, compared with Skill’s 161 tasks and 803,099 tokens.This represents 39 additional tasks and a 9.9% reduction in total solve-time cost.
- 5.4 Gains Vary Across Workflow Types: Agent-environment synthesis and rule following show the most consistent gains, while coding and mathematical reasoning remain more model-dependent.Coding regressions range from 1.7 to 4.9 points; Gene is especially useful when success depends on preserving operational constraints.
- 5.5 Representative Cases: Drill-hole compositing succeeds by segmenting at every breakpoint, averaging all covering assays, and applying length-weighted aggregation.This contrasts with the initially plausible but repeatedly failing “later interval wins” convention.
- 5.5 Representative Cases: Emergency dispatch succeeds by treating equality as non-triggering and applying precedence among injury, armed-without-injury, and distance-only rules.Unguided and intermediate trajectories over-escalated by mishandling the boundary and precedence order.
6 Conclusion
LongWoF-Bench evaluates whether verifier-confirmed execution experience can be retained as EvoMap Gene and reused beyond its producing model. The results support evolved Gene as a reusable resource that improves completion and amortizes discovery cost, while reference-distilled Gene do not show the same advantage.
- 6 Conclusion: LongWoF-Bench contains 778 machine-verifiable tasks for strict end-to-end workflows with interdependent constraints.The benchmark evaluates whether Gene provides value beyond reusable procedural knowledge represented by Skill.
- 6 Conclusion: Evolved Gene outperform Skill across all seven evaluated consumer models on tasks with verifier-confirmed Opus trajectories.The advantage remains useful beyond the model that originally produced the experience.
- 6 Conclusion: Reference-distilled Gene do not exhibit the same behavior, indicating that the advantage does not arise from Gene representation alone.The conclusion associates utility with verified execution experience.
- 6 Conclusion: Gene reuse reduces solve-time cost relative to static Skill guidance and the original multi-round discovery process.Successful exploration can therefore be amortized across subsequent executions.
- 6 Conclusion: Verified agent experience can be externalized, retained, and shared through EvoMap as a reusable resource for future models and workflows.This turns successful task execution from a transient outcome into reusable experience.
A.1 Main Evolved-Gene Results
The appendix confirms the main evolved-Gene comparison with complete numerical results on the same 252 tasks. Gene is best for every consumer model, with paired effects statistically separated from zero.
- A.1 Main Evolved-Gene Results: Gene achieves the highest strict pass rate for all seven consumer models on the same 252 Opus-evolved tasks.Table 4 provides the complete numerical results underlying Figure 3.
- A.1 Main Evolved-Gene Results: All paired bootstrap confidence intervals exclude zero, and all improvements are significant under exact McNemar tests.The effects compare Opus-Gene with Skill on the evolved subset.
A.2 Gene Provenance and Producer Effects
The appendix details provenance comparisons across reference-distilled and producer-matched Gene. Reference-distilled Gene trail Skill on the complementary subset, while the common-set analysis compares Opus- and Gemini-authored Gene.
- A.2 Gene Provenance and Producer Effects: Reference-distilled Gene trail Skill across all seven consumer models on the complementary 526-task subset.Table 6 reports strict pass rates for this provenance condition.
- A.2 Gene Provenance and Producer Effects: The producer comparison uses 180 tasks for which both Opus and Gemini produced verifier-confirmed trajectories.This task-matched analysis evaluates differences between Opus-authored and Gemini-authored Gene.
A.3 Experience Discovery and Reuse Cost
Experience discovery required substantial iterative effort, while one-shot Gene reuse reduced solve-time cost and preserved broad workflow-family gains across consumer models.
- Experience Discovery: 140 of 252 verifier-confirmed trajectories were discovered on the first attempt, while 112 required at least one feedback-driven refinement.These exploration rounds quantify the effort needed to obtain reusable experience.
- Experience Discovery: 152 additional model calls beyond the one-call-per-task baseline were introduced by feedback-driven refinements.These calls represent trial-and-error costs that reusable Gene can subsequently avoid.
- Reuse Cost: Multi-round discovery reached all 252 verified solutions using 404 calls and 1,333,968 tokens, whereas one-shot Opus-Gene reuse passed 200 tasks using 252 calls and 723,480 tokens.The accounting includes failed solve attempts but excludes one-time Gene distillation and auditing costs.
- Reuse Cost: Gene reuse passed 39 additional tasks over Skill while using 9.9% fewer tokens under the same one-shot protocol.This comparison reports both effectiveness and efficiency on the same 252 tasks.
- Workflow-Family Results: Across the seven consumer models, Gene effects were consistently positive for agent-environment synthesis and rule following but more model-dependent for code generation and mathematical reasoning.The complete workflow-family breakdown covers the 252 tasks with an Opus-evolved Gene.