PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill2Env: Capability-Oriented Environment Synthesis from Skills for General Agents

Weiyi Xu, Xiaowen Yang, Wen Da, Hang Xu, Canwei Li, Hongjie You, Pusen Dong, Yucheng Zeng, Zhaokai Luo, Mu Chuan

arXiv:2609.33772v1cs.AI

TL;DR

Building executable tasks and environments at scale remains difficult, and skills still leave a gap to concrete, challenging executable tasks. Skill2Env uses capability demands, difficulty patterns, blueprints, and solver-evidence hardening; supervised fine-tuning on 1.5K generated trajectories improved performance across seven benchmarks, with Qwen3.6-35B-A3B’s unweighted average rising from 36.6 to 45.0 (+8.4 points).

  • Problem

    Creating executable tasks and environments at scale is difficult, and a substantial gap remains between skills’ information and concrete, challenging tasks with complete executable environments.

    Source: §1 · pp. 1–2

  • Method

    Skill2Env maps capability demands into difficulty patterns and task blueprints for joint environment synthesis, then hardens tasks using solver execution evidence.

    Source: §1 · p. 2

  • Results

    Across all seven downstream benchmarks, supervised fine-tuning on 1.5K Skill2Env trajectories raised Qwen3.6-35B-A3B’s unweighted average from 36.6 to 45.0 (+8.4 points).

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    The findings support using skill-based environment synthesis for agent post-training.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Executable environments are critical for post-training agents on tasks that require tool use and multi-step interaction, but constructing executable tasks together with their environments remains difficult to scale. Skills provide reusable domain knowledge, operational procedures, and tool-use instructions, but a substantial gap remains between the information contained in a skill and a concrete, challenging task with a complete executable environment. To address this gap, we introduce Skill2Env, a capability-oriented framework that starts from a skill and uses agent capability demands to guide task and environment synthesis. Skill2Env represents these demands through reusable difficulty patterns and instantiates them into task blueprints that specify objectives, challenges, environment facts, information boundaries, and acceptance criteria. These blueprints guide the joint construction of task instructions, execution substrates, workspaces, and rubric-based evaluators around source skills. We further propose Iterative Task Hardening, which uses solver execution evidence to identify insufficiently challenging task designs, strengthen or extend their difficulty-pattern instantiations, and revise the corresponding blueprints and environments. Using 1.5K high-scoring trajectories generated from Skill2Env environments for supervised fine-tuning, we observe consistent improvements across a broad range of agent benchmarks, demonstrating the effectiveness of capability-oriented environment synthesis for agent post-training.

1 Introduction

Skill2Env uses agent capability demands and source skills to synthesize executable tasks and environments, then hardens them using solver execution evidence. The paper reports that fine-tuning on 1.5K high-scoring trajectories from these environments improves performance across a broad range of agent benchmarks.

Motivation

  • Executable task synthesis is difficult to scale because it requires coordinated task instructions, usable tools, workspaces, and evaluation; skills ground these components but leave a gap to concrete, challenging tasks.Skills can include domain knowledge, operational procedures, tool-use instructions, scripts, templates, and reference resources.Source: §1 · pp. 1–2

Skill2Env framework

  • Skill2Env structures capability demands across environment understanding, planning, skill usage, long-horizon consistency, and error recovery, using difficulty patterns to translate them into environment structures.It constructs 2,963 executable tasks through this capability-oriented framework.Source: §1 · p. 2

Task blueprints

  • A task blueprint specifies objectives, challenges, environment facts, information boundaries, and acceptance criteria, guiding joint construction of instructions, execution substrates, workspaces, and rubric-based evaluators.These components are constructed and refined around the source skill.Source: §1 · p. 2

Iterative Task Hardening

  • Iterative Task Hardening uses solver rollouts to find insufficiently challenging aspects, strengthen or add compatible difficulty patterns, and revise the blueprint and environment.Reusable challenges identified during hardening can extend the difficulty-pattern pool.Source: §1 · p. 2

Empirical impact

  • 1.5K high-scoring trajectories from Skill2Env environments yield consistent improvements across a broad range of agent benchmarks.The paper selects these trajectories for supervised fine-tuning.Source: §1 · p. 2

2 Related Work

Related work distinguishes workspace-anchored task derivation from workspace-from-scratch construction; Skill2Env follows the latter and focuses on translating skill information into concrete, challenging executable tasks.

Executable Environment and Task Synthesis

  • Executable-environment synthesis divides into workspace-anchored task derivation from existing workspaces and workspace-from-scratch construction from higher-level specifications.The former restores, modifies, or perturbs workspace state; the latter builds task-specific workspaces from specifications.Source: §2 · p. 2

Skill-Based Environment and Task Synthesis

  • Skill2Env targets the skill-to-task gap by making explicit agent capability demands drive reusable difficulty patterns and execution-feedback task hardening.Its focus is constructing challenging tasks and executable environments from skills.Source: §2 · p. 3
  • SkillSynth samples skill-graph workflows [5], Terminal-World derives instructions, environments, and teacher trajectories [3], FACET grounds instructions, solutions, and verifiers in realized scenarios [24], and SKT uses feedback-guided repair [26].These methods use skills to scale construction of executable training environments and tasks.Source: §2 · p. 3

3 Preliminaries

Skill2Env defines executable tasks by coupling a reusable skill, execution substrate, and task-specific workspace with an instruction and evaluator. Agents interact through partial observations and actions, and the evaluator assigns a bounded reward after execution.

3 Preliminaries

  • A skill packages reusable procedural instructions and optional resources; the execution substrate supplies tools and runtime support, while the workspace holds task-specific resources the agent can inspect and modify.Together with an initial workspace, these components define the executable environment.Source: §3 · p. 3
  • Skill2Env represents an executable task as T = (g, E, V), combining an objective-bearing instruction and evaluator with environment E = (X, κ, W0).Environment synthesis is coupled with generating the task instruction and evaluator.Source: §3 · p. 3
  • At each step, the agent receives a partial observation and chooses an action from its interaction history; execution yields a rollout, after which the evaluator assigns reward r = V(sL) in [0, 1].The state includes the current workspace and other runtime state relevant to execution.Source: §3 · p. 3

4 Skill2Env

Skill2Env synthesizes executable tasks and environments from skills by translating capability demands into difficulty patterns and task blueprints. It uses solver execution evidence to harden challenges the solver already handles well.

4 Skill2Env

  • Skill2Env uses difficulty patterns and task blueprints to jointly construct task instructions, execution substrates, workspaces, and rubric-based evaluators around source skills.Initial synthesis establishes intended challenges; Iterative Task Hardening then uses execution evidence to strengthen challenges that solvers already handle well.Source: §4 · p. 3

4.1 Skill Curation

  • From more than 47K public skill folders downloaded from ClawHub, Skill2Env retained more than 3K after screening for practical utility, runtime compatibility, and setup feasibility.The criteria require a substantive purpose, compatibility with the Linux sandbox, and feasible provisioning of execution inputs, dependencies, data, and services.Source: §4.1 · p. 3

1 Skill Curation · 3 Difficulty Patterns

Skill2Env translates curated skills and capability demands into executable task environments through reusable difficulty patterns and blueprints, then uses solver evidence to strengthen tasks. Its pipeline produced 2,963 executable tasks.

4.2 Capability-Oriented Environment Synthesis

  • Skill2Env targets five capability demands—environment understanding, planning, skill usage, long-horizon consistency, and error recovery—through reusable task difficulty patterns.Examples include distributed evidence and implicit constraints for environment understanding, and state dependencies and resource budgets for planning.Source: §4.2 · p. 4
  • The difficulty-pattern pool starts with 100 reusable patterns organized across five capability dimensions, distilled from recurring challenges in pilot synthesis.Source: §4.2 · p. 4
  • For each skill, the synthesis agent selects compatible patterns that fit a common task objective and use operations the skill supports.Different pattern combinations let the same skill support tasks with different capability demands.Source: §4.2 · pp. 4–5
  • A task blueprint links challenges to supporting facts, acceptance criteria, execution requirements, and information boundaries, guiding construction of the instruction, substrate, workspace, and evaluator.The workspace combines synthesized and retrieved real-world materials; blueprint and environment revisions proceed iteratively.Source: §4.2 · p. 5
  • Rubric evaluators score weighted acceptance criteria using deterministic checks or LLM judgment, while a validation agent checks rollout conditions and results against the blueprint.Evidence from inconsistent task conditions or evaluator errors is excluded from capability diagnosis.Source: §4.2 · p. 5

4.3 Iterative Task Hardening

  • Iterative Task Hardening uses solver rewards to trigger diagnosis, strengthens or adds compatible patterns, and revises the blueprint and affected task components.Generalizable challenges absent from the pool can be abstracted into new reusable patterns.Source: §4.3 · pp. 5–6
  • 2,963 executable tasks result from applying the complete Skill2Env pipeline.Appendix C provides a comprehensive analysis of environment diversity.Source: §4.3 · p. 6
  • An end-to-end Docling case study illustrates the blueprint, workspace, evaluator, and successive hardening revisions.Source: §4.3 · p. 6

5 Experiments

Skill2Env supervision improves performance across seven downstream benchmarks, with gains also evident across SkillsBench domains and Terminal-Bench harnesses. Iterative Task Hardening makes tasks more demanding and improves downstream training utility.

5.1 Experimental Settings

  • The main results use a Qwen3.6-35B-A3B SFT checkpoint trained on 1.5K filtered trajectories, compared with frontier models and FACET across seven benchmarks.Kimi-K3 synthesizes environments and diagnoses execution; Qwen3.6-35B-A3B is the fixed hardening solver, and Qwen3.5-397B-A17B judges rubrics. The SFT data come from DeepSeek-V4-Flash trajectories filtered at r > 0.9; fine-tuning runs for five epochs. The benchmark suite includes Terminal-Bench 2.1, SWE-bench Multilingual, SkillsBench, Claw-Eval, τ3-Banking, AutomationBench, and VitaBench.Source: §5.1 · p. 6

5.2 Main Results

  • Skill2Env raises Qwen3.6-35B-A3B’s unweighted seven-benchmark average from 36.6 to 45.0 (+8.4 points) and improves every benchmark, with largest gains on SkillsBench (+14.34) and Terminal-Bench 2.1 (+13.5).Source: §5.2 · pp. 6–7, Table 1 · p. 7
  • Across heterogeneous benchmarks, Skill2Env beats Qwen3.5-397B-A17B on six of seven despite its smaller backbone, while frontier models remain stronger on several.Skill2Env does not optimize for any particular downstream benchmark; the results suggest capability-oriented supervision transfers beyond synthesized training environments.Source: §5.2 · p. 7

5.3 Analysis of SkillsBench Improvements

  • On SkillsBench, gains reach 14.3 percentage points with skills on and 10.4 with skills off; gains span seven of eight domains with skills on and all eight with skills off.With skills on, finance is unchanged, while media, mathematics and operations research, and software engineering lead gains (+46.7, +26.9, and +23.4 percentage points). Trajectory analysis describes better planning, long-horizon consistency, content processing, use of guidance and helper scripts, and recovery from execution failures.Source: §5.3 · pp. 7–8, Figure 3 · p. 8

5.4 Terminal-Bench 2.1 Performance Across Harnesses

  • Terminal-Bench 2.1 improvements persist across all four tested harnesses.Scores rise from 46.1 to 57.3 with pi (+11.2 percentage points), 44.9 to 58.4 with Terminus-2 (+13.5), 38.2 to 55.1 with OpenCode (+16.9), and 40.5 to 52.8 with Claude Code (+12.3).Source: §5.4 · p. 8, Figure 4 · p. 8

5.5 Effectiveness of Iterative Task Hardening

  • Across 500 paired tasks, two hardening rounds reduced DeepSeek-V4-Flash’s full-credit rate from 48.40% to 15.40% and raised mean assistant turns from 25.91 to 38.85.From iteration 0 to 2, the full-credit rate fell by 33.0 percentage points and interaction steps increased by approximately 50%.Source: §5.5 · pp. 8–9, Figure 5 · p. 9
  • With 500 qualifying trajectories and the same quality threshold at each stage, the seven-benchmark average rises from 40.58 to 42.87 after two hardening rounds.The trajectories satisfy r > 0.9; tasks not further hardened are carried forward unchanged, and the iteration-0-to-2 improvement is 2.29 points.Source: §5.5 · p. 9, Figure 5 · p. 9

6 Conclusion

Skill2Env synthesizes executable tasks and environments from skills by organizing capability demands into difficulty patterns and task blueprints. Iterative Task Hardening uses execution evidence to make designs more demanding, and supervised fine-tuning on 1.5K high-scoring trajectories consistently improves performance across a broad range of agent benchmarks.

6 Conclusion

  • Skill2Env uses reusable difficulty patterns and task blueprints to translate capability demands into challenges and coordinate instructions, execution substrates, workspaces, and evaluators.The framework synthesizes executable tasks and environments from skills.Source: §6 · p. 9
  • Iterative Task Hardening uses execution evidence to progressively revise task designs toward more demanding capability requirements.Source: §6 · p. 9
  • 1.5K high-scoring Skill2Env trajectories consistently improve supervised fine-tuning performance across a broad range of agent benchmarks.The results suggest skill-based environment synthesis can organize task construction around required agent capabilities and support increasingly demanding environments as capabilities improve.Source: §6 · p. 9

A Capability Demands … B.4 Long-Horizon Consistency: Patterns

Skill2Env translates five capability demands into concrete difficulty patterns, spanning environment understanding, planning, skill use, and long-horizon consistency. The patterns expose adjustable controls for making task challenges more demanding.

A Capability Demands

  • The five capability demands summarized in Table 2 guide environment synthesis, while Appendix B operationalizes them as difficulty patterns.The supplied passages do not enumerate the five demands themselves.Source: A Capability Demands · p. 13, Table 2 · p. 13

B.1 Environment Understanding: Patterns

  • Environment-understanding patterns test whether agents can resolve aliases, infer unstated constraints and temporal or spatial relations, and distinguish causal effects from incomplete evidence.Controls include alias multiplicity, inference depth, relation-chain length, frame ambiguity, alternative explanations, and observation coverage.Source: §B.1 · p. 14

B.2 Planning: Patterns

  • Planning patterns create tradeoffs among deadlines, resource budgets, shared capacity, locality, and reusable work, then test commitment, adaptation, and scale-sensitive strategy.Their controls include deadline slack, budget tightness, capacity contention, locality asymmetry, reusable overlap, lost options, change extent, marginal benefit, and workload scale.Source: §B.2 · p. 15
  • Planning also tests outcome contingencies, plan revision after legitimate changes, adaptive stopping, and switching strategy as workload scale changes.Controls include outcome uncertainty, change extent, marginal benefit, and workload scale.Source: §B.2 · p. 15

B.3 Skill Usage: Patterns

  • Skill-use patterns require faithful handling of operation completion, native types, schema mappings, unit conversion, precision, and defined boundary cases.Other patterns target nested syntax, exact rule semantics, recursive generality, tie-breaking, and stochastic reproducibility.Source: §B.3 · p. 16

B.4 Long-Horizon Consistency: Patterns

  • Long-horizon patterns require consistency between detailed records and derived claims, preservation of unrelated content and structures, and continuity of policies and accounting.They also test whether later edits preserve earlier obligations and whether fixed worklists are completed exactly once.Source: §B.4 · p. 17
  • Additional long-horizon patterns test deferred obligations, faithful checkpoints, invariant-preserving interleaved operations, and ownership of temporary resources through workflow completion.Their controls include deferral span, state dimensionality, interleaving count, and ownership transfers.Source: §B.4 · p. 17

B.5 Error Recovery: Patterns … C.2 Execution Substrate

Skill2Env’s error-recovery patterns specify ways to restore or safely manage failed operations, while its environment-diversity analysis characterizes the resources and execution substrates available across 2,963 synthesized tasks.

B.5 Error Recovery: Patterns

  • Error-recovery patterns cover reconstructing corrupted state, restoring interrupted operations, compensating partial effects, safely retrying uncertain commits, and resuming failed batch items.Their controls include recovery distance, mutation footprint, compensation chain, commit ambiguity, and failure dispersion.Source: §B.5 · p. 18
  • Other patterns require checkpoint reconciliation, substituting a contract-compatible fallback, resolving conflicts before retrying, and repeating the original check after repair.Controls include lost-work interval, fallback mismatch, conflict scope, and failure conditions.Source: §B.5 · p. 18
  • Further patterns check preserved behavior after repairs, qualify degraded results against acceptance criteria, diagnose cascading defects, or stop recovery safely with preserved state and accurate status.Controls include regression surface, fallback conditions, masking depth, and stop conditions.Source: §B.5 · p. 18

C Environment Diversity

  • The synthesis process yields 2,963 executable tasks, analyzed across source-skill resources, execution substrates, skill-domain coverage, and initial workspaces.Source: C Environment Diversity · p. 18

C.1 Skill Resources

  • Among 2,963 task packages, 2,613 include auxiliary skill resources beyond SKILL.md, comprising 19,649 files across varied types, formats, locations, and interfaces.The resources span 13 programming or markup languages, and static analysis finds references to 101 distinct external programs.Source: §C.1 · p. 18, Figure 6 · p. 19

C.2 Execution Substrate

  • Across 2,963 tasks, environments span 4,242 distinct dependency packages across eight ecosystems and 823 CLI names, with long-tailed usage distributions.npm has the largest number of package identities, followed by pip and Cargo.Source: §C.2 · p. 19, Figure 7 · p. 20

C.3 Skill Domains and Workspaces

Skill2Env tasks span a broad set of source-skill domains and diverse initial workspaces, with domain-specific combinations of workspace materials.

C.3 Skill Domains and Workspaces

  • The collection spans 25 source-skill domains, led by software engineering and code quality with 320 tasks (10.8%).Each task inherits one primary domain label from its source skill under a model-generated taxonomy.Source: §C.3 · p. 20
  • Initial workspaces contain 46,295 files across 113 recognized extensions, 12 asset families, and 812 distinct nonempty extension sets.Of 2,963 tasks, 2,707 (91.4%) contain at least two recognized extensions, 254 contain one, and two contain none; configured skill mounts, caches, and dependency directories are excluded.Source: §C.3 · p. 20
  • Domain-specific workspace combinations include office documents and PDFs in 52.1% of office-domain tasks, audio/video assets in 40.9% of media tasks, and geospatial/scientific assets in 24.0% of scientific-computing tasks.Figure 8 shows task counts across source-skill domains and the percentage of tasks in each domain containing each workspace asset family.Source: §C.3 · p. 20, Figure 8 · p. 21

D Per-Benchmark Results Across Iterations … E.4 Information Boundaries and Reference Facts

This section reports the seven-benchmark mean across two hardening iterations and details the Docling vendor-reconciliation case study, from its blueprint and execution setup to its information boundaries, reference calculations, evaluation strategy, and acceptance checks.

D Per-Benchmark Results Across Iterations

E Docling Case Study: From a Skill to a Reconciliation Environment

  • The Docling case study turns document-conversion and OCR guidance into a quarterly vendor-reconciliation environment, using execution evidence for two rounds of task hardening.The task materials define the solver-facing request; the blueprint and evaluation criteria set the author-side contract.Source: E Docling Case Study: From a Skill to… · pp. 22–23

E.1 The Complete Finance Brief

  • The finance brief scopes reconciliation to engagement MSA-2024-HV07 for Q3 2024 only, checking billed lines against contract terms, travel claims against receipts, invoice arithmetic, and settlement against the payments ledger, with exactly three deliverable files.Source: §E.1 · pp. 23–24

E.2 The Complete Environment Setup

  • The setup provisions the Docling CLI and CPU-only PyTorch, then warms the model cache with PDF and image conversions.The pinned versions are docling==2.123.1 and torch==2.13.0+cpu; torchvision==0.28.0+cpu.Source: §E.2 · p. 25

E.3 How the Blueprint Connects Requirements and Challenges

  • Challenges H1–H3 target multi-format extraction, rules distributed across documents, and correct task scoping; the blueprint pairs these obstacles with intended solver behaviors.Those behaviors include using Docling across file formats, extracting all applicable contract rules, and excluding the archive and inapplicable 2025 rate notice.Source: §E.3 · pp. 25–27
  • Challenges H4–H6 address numerical precision, consistency across artifacts, and exact schema fidelity, with blueprint behaviors specifying decimal arithmetic and shared corrections.The prescribed approach carries a single correction set through the CSV, JSON, and memo, using the finance brief’s target schemas.Source: §E.3 · pp. 25, 27
  • Blueprint requirements make the case evaluable, including the exact seven billed lines, three identified discrepancies, corrected totals, and a EUR 995.83 credit due.The requirements also specify as-billed totals and day counts, with an audit-ready memo explaining evidence and the credit-note recommendation.Source: §E.3 · p. 26

E.4 Information Boundaries and Reference Facts

  • The finance brief is the authoritative specification for schemas, scope, and checks, and the task requires three deliverables at specified root-relative paths.The paths are reconciliation/line_items.csv, reconciliation/summary.json, and reconciliation/findings.md.Source: §E.4 · p. 27
  • All required facts are fixed in workspace documents as of 2024-10-03, and the 2025 rate notice’s effective date makes its non-applicability resolvable from context.The blueprint requires no outside information; VAT, caps, rates, and rounding expectations are in the workspace.Source: §E.4 · p. 28
  • Evaluation is fully offline against fixed reference facts, using deterministic checks for objective criteria and an LLM judge for the subjective criterion.Source: §E.4 · p. 28
  • The reference calculations identify 240.00 rate over-billing and 13.00 taxi over-claim, applying the Q3 2024 senior rate and receipt-and-cap rules.The applicable senior rate is 420.00, not 450.00; the taxi payable amount is min(52.00, 39.00, 40.00) = 39.00.Source: §E.4 · p. 28
  • The two-day cap excess produces a 570.00 correction, and the paid invoices yield a credit due of 995.83 including VAT.Corrected totals are 9331.40 excl. VAT, 1959.59 VAT, and 11290.99 incl. VAT, against 12286.82 paid.Source: §E.4 · p. 29
  • Acceptance checks require exactly seven line-item rows, consistent quantity-times-price amounts and summary totals, and exclusion of the archive invoice and 2025 rate.The summary must satisfy billed-total, corrected-plus-corrections, and paid-minus-corrected-total relationships.Source: §E.4 · p. 29

E.5 Complete Evaluation Criteria · E.6 How Execution Evidence Informed Two Hardening Revisions · E.7 How the Task Changes Across Hardening

The sections define a deterministic, evidence-grounded reconciliation rubric, then use solver rollouts to harden the workflow around cross-record settlement challenges and robust extraction. Across versions, task complexity increases through more records, interacting discrepancies, and settlement dependencies.

E.5 Complete Evaluation Criteria

  • The rubric requires a schema-compliant line-item CSV and summary JSON that reconcile the billed invoices, ledger payment, and quarterly day cap.The CSV must use the exact 9-column schema and contain exactly seven billed Q3 2024 lines; billed quarterly totals are 10,154.40 / 2,132.42 / 12,286.82, ledger-paid total is 12,286.82, and billed days are 26 versus a 24-day cap.Source: §E.5 · p. 29
  • The rubric expects all three contract violations quantified, corrected totals and the EUR 995.83 recovery computed, and an evidence-grounded finance-manager memo.The corrections are EUR 240.00, EUR 13.00, and EUR 570.00 excl. VAT; corrected subtotal, VAT, and total are 9,331.40, 1,959.59, and 11,290.99.Source: §E.5 · pp. 29–30

E.6 How Execution Evidence Informed Two Hardening Revisions

  • After checking the workspace and evaluator against the blueprint, the original rollout earned full credit on all five rubric items.The diagnosis agent inspected execution records showing document conversion, deliverable writes, and numerical self-checks.Source: §E.6 · p. 30
  • The first hardening added challenges around signed-rate evidence, cross-invoice checks, vendor scope, a misprinted line amount, and partial payment.The diagnosis deemed the successful task insufficiently challenging; duplicate-claim and partial-payment challenges were proposed extensions.Source: §E.6 · p. 31
  • After OCR conversion failed four times, the solver recovered by reading receipts directly, prompting a revision that made scanned-document extraction essential.The hardened task requires reliable OCR/conversion, preservation of extracted structure, and use of that evidence in reconciliation; it also adds date-dependent rates, an unreceipted claim, and a day-cap deduction spanning two rate tiers.Source: §E.6 · p. 31

E.7 How the Task Changes Across Hardening

  • Across hardening stages, task complexity increases through more records, interacting discrepancies, and additional settlement dependencies.Source: §E.7 · p. 32
Loading 2609.33772v1…