Toward Scalable Terminal Task Synthesis via Skill Graphs

Introduction

• Terminal-agent training needs diverse scenarios and skills, but task synthesis remains constrained by scarce high-quality trajectories, redundant coverage, and limited control over trajectory composition.Agents act step by step through intermediate scenarios, applying skills; manually curating executable tasks is expensive and difficult to scale.Source: §1 · pp. 1–2
• SkillSynth builds a scenario-mediated skill graph, samples workflow paths through scenarios and skills, then instantiates them as executable tasks with automated verification.A multi-agent harness checks solvability through execution and specification quality through rubrics, iteratively repairing failed instances.Source: §1 · pp. 2–3, Figure 2 · p. 2
• 3,560 verified task instances were produced from 3,721 sampled paths, with a 95.7% oracle pass rate and an average cost of $27.3 per verified instance.The run was fully automated and required no human intervention.Source: §1 · pp. 2–3
• The constructed graph contains 82,073 scenarios, 57,214 filtered skills, and 185,529 LLM-verified bridges, providing a growing space for sampling workflow paths.The graph can expand as the community contributes more skills, enabling continual synthesis of diverse terminal tasks.Source: §1 · p. 3
• SkillSynth tasks are harder and more diverse than single-skill or composed multi-skill baselines, with consistent gains on Terminal-Bench 1.0 and 2.0 across model scales.Claude Opus 4.6 averaged 37 steps to solve the tasks, and 121 tasks remained unsolved after three independent rollouts.Source: §1 · pp. 2–3

Terminal agent task

• A terminal-agent task is τ = (E, s0, g, V): an agent acts from s0 in executable environment E to achieve goal g, with success determined by verifier V.Under partial observability, the agent samples actions from its policy using observations, prior actions, and the goal.Source: §2 · p. 3

Scenario and skill abstraction

• Scenarios abstract decision-relevant observations as sufficient statistics of interaction history, while skills are action subsequences that predictably transition between scenarios.Together, the induced execution trajectory characterizes the scenarios traversed and skills exercised.Source: §2 · p. 3

Learning objective

• Training maximizes expected skill-selection likelihood under the trajectory-induced distribution D, equivalent to next-token prediction, but learned capacity is limited to its scenario–skill support.Scenarios never observed and admissible skills never exercised are outside the learnable region, motivating synthetic tasks that densely cover conditional scenario–skill pairs.Source: §2 · p. 3

3.1 OVERVIEW

• The three-stage pipeline constructs a scenario-mediated skill graph, samples directed workflow paths, and instantiates each path as a verified executable task.The harness uses oracle verification and rubric-based evaluation to assess solvability and task quality.Source: §3.1 · p. 4

3.2 SKILL GRAPH CONSTRUCTION

• The skill graph represents scenarios as nodes and skills as directed transitions, retaining executable Linux workflows with deterministic, objectively verifiable outputs.An LLM infers plausible precondition scenarios from each retained skill’s specification.Source: §3.2 · p. 4
• Scenario deduplication uses hierarchical agglomerative clustering with Louvain-based coarse bucketing; cross-skill alignment retrieves top-1,000 candidates and uses an LLM to judge compatibility.Aligned pre- and postconditions are merged into scenario nodes, then filtered to retain valid transitions; manual review found each construction stage necessary for graph quality.Source: §3.2 · pp. 4–5

3.3 GRAPH-GUIDED PATH SAMPLING

• Paths interleave skills and scenarios, span lengths 1–7 in experiments, and use inverse-frequency sampling with no revisits to promote coverage and avoid duplicate skill sets.Visit and usage counts guide source-scenario and next-skill selection; the walk stops at a dead end or the maximum length, accepting only paths within range whose skill set is new.Source: §3.3 · p. 5

3.4 MULTI-AGENT HARNESS

• A planner converts each sampled path into sub-objectives before a constructor generates the task, which undergoes execution-based and rubric-based verification.The task includes an instruction, initial filesystem snapshot, containerized environment, verification scripts, and oracle solution; failed checks can trigger up to 3 repair cycles of at most 20 tool calls each.Source: §3.4 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• SkillSynth samples 3,721 graph paths; evaluation uses Terminal-Bench 1.0 and 2.0, with mean accuracy across three runs and 95% confidence intervals.Terminal-Bench 1.0 and 2.0 comprise 80 and 89 tasks, respectively; evaluation uses Terminus 2 and Harbor across 128 concurrent Docker environments.Source: §4.1 · p. 6

4.2 MULTI-AGENT HARNESS QUALITY

• The harness yields 3,560 usable instances: 95.7% pass the oracle check, 92.0% pass both quality checks, and repair recovers 721 first-round failures.Among usable tasks, 37% have 1/3 or 2/3 successes and 38% have 0/3, using three attempts per task and Hy3 Preview to assign difficulty.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• Rubric failures chiefly reflect instruction-test misalignment (77%), while unrecoverable oracle failures often stem from corrupted filesystem snapshots.Misaligned tasks are retained for supervised fine-tuning but discarded for reinforcement learning; re-running failed paths at higher sampling temperature may recover some instances.Source: §4.2 · p. 7

4.3 MAIN RESULTS

• All three trained Qwen3 sizes improve over their respective baselines, and Qwen3-32B + SS outperforms Qwen 3 Coder 480B on TB 2.0.The reported results use the Terminus 2 scaffold; Table 3 covers Terminal-Bench 1.0 and 2.0.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 ABLATION STUDY

• SkillSynth-trained models outperform single-skill-trained models by 8.4 points on TB 1.0 and 8.3 on TB 2.0, and multi-skill-trained models by 3.0 and 3.8 points, respectively.SkillSynth tasks are also harder than tasks from both baselines; random skill combinations lack sequential dependencies and yield simplified workflows requiring few execution steps.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 DIVERSITY ANALYSIS

• The graph spans long-tail domains, places 85.6% of scenarios in its largest weakly connected component, and contains 16,632,220 paths requiring at least seven skills.Algorithm 1 promotes skill- and scenario-level path diversity; SkillSynth trajectories show 31% higher unique scenario-skill coverage than single-skill baselines and 19% higher than randomly composed multi-skill baselines on average.Source: §4.5 · pp. 8–9

4.6 ERROR ANALYSIS

• Among failed trajectories, partial implementation (42.2%) and over-trust in inline self-tests (29.5%) are the dominant failure modes.In both cases, agents substitute self-narrated verification for specification-grounded testing; the analysis points to a need for stronger instruction alignment and more flexible exploration.Source: §4.6 · p. 9, Table 5 · p. 9

4.7 DISCUSSION AND FUTURE WORK

• Future work targets larger data regimes and more cost-effective harness models, as well as sampling skill-graph subgraphs rather than chains.Subgraph sampling would require parallel execution of multiple skills and increase task complexity.Source: §4.7 · p. 9

Terminal Agents

• LLM terminal-agent research covers command-line interaction, Terminal-Bench evaluation, and scaffolds for planning, execution, and tool calling.Terminal-Bench uses hand-crafted tasks in Docker environments; despite strong proprietary results, open-source models remain substantially behind, motivating scalable trajectory synthesis.Source: §5 · pp. 9–10

Synthetic Data for Terminal Agents

• Prior synthesis expands domain coverage or task counts, but offers limited trajectory control; SkillSynth uses a skill graph to control training trajectories.Prior methods use LLM-generated taxonomies, collect real-world Docker environments, or invert healthy environments into buggy states.Source: §5 · p. 10

Skill Topology

• Prior skill topologies organize or retrieve skills, whereas SkillSynth uses scenarios to model execution transitions and synthesize tasks with controllable scenario and skill diversity.AgentSkillOS uses a capability tree and DAG-based orchestration; SkillNet models skills as nodes in a relational graph with explicit connections.Source: §5 · p. 10

6 CONCLUSION

• SkillSynth models agentic trajectories as sequences of scenarios and skills, then synthesizes diverse terminal tasks by sampling compositional paths from a constructed skill graph.The framework uses these paths to guide task synthesis.Source: §6 · p. 10
• SkillSynth achieves a high synthesis pass rate at relatively low cost, enabling scalable production of verified task instances.The conclusion presents scalable synthesis as a way to turn emerging workflows into diverse task instances for continued learning from real-world experience.Source: §6 · p. 10
• Experiments show that training effectiveness depends on trajectory diversity, not merely on task volume.As communities such as ClawHub grow, SkillSynth is positioned to transform emerging workflows into diverse task instances.Source: §6 · p. 10

• Construction skills are sourced from publicly available GitHub repositories and ClawHub and used in accordance with their respective open-source licenses.Source: ETHICS STATEMENT · p. 10

A PROOF OF EQUIVALENCE

• The reduction assumes a deterministic mapping ζ 7→ξ, scenarios σt sufficient for the agent’s next decision, and autoregressive execution of each skill κt = (ait, . . . , ajt).Source: A PROOF OF EQUIVALENCE · p. 13
• The proof applies the conditional-probability chain rule, uses contiguous skill action ranges to collapse the double sum, then takes expectations and applies determinism.Source: A PROOF OF EQUIVALENCE · p. 13
• Under A1–A3, the skill-level objective reduces to the standard next-action objective, and scenario-level properties transfer to the token-level loss without additional training machinery.The transferable properties include the coverage criterion in Equation 4.Source: A PROOF OF EQUIVALENCE · p. 13

B SKILL GRAPH CONSTRUCTION DISCUSSION

• LLM-based pairwise alignment yields higher-quality graphs than embedding-based scenario alignment or single-pass subgraph generation.All LLM calls use DeepSeek Reasoner (v3.2) for extraction quality and reliability.Source: B SKILL GRAPH CONSTRUCTION DISCUSSION · p. 13
• A two-stage clustering procedure applies Louvain to a sparse semantic graph, then complete-linkage clustering within buckets to avoid global quadratic memory costs and chain drift.Complete linkage requires every pair in a cluster to be mutually close.Source: B SKILL GRAPH CONSTRUCTION DISCUSSION · p. 13
• The distance threshold is selected by inspecting held-out scenario clusters to merge paraphrases while separating negations and pre/post-condition changes.The choice balances merge quality against over-fragmentation.Source: B SKILL GRAPH CONSTRUCTION DISCUSSION · p. 13

C SUPERVISED FINE-TUNING

• SFT trains all models for 5 bfloat16 epochs with AdamW, cosine decay, 10% warmup, and a 2 × 10−5 peak learning rate.Unless otherwise noted, β1 = 0.9, β2 = 0.95, and weight decay is 1×10−4; gradient clipping uses a maximum norm of 1.0. Training uses a micro-batch size of 1 per GPU with gradient accumulation to reach the desired global batch size.Source: C SUPERVISED FINE-TUNING · p. 13

D SKILL GRAPH ANALYSIS

• Table 6 summarizes the skill graph’s structural statistics and connected-component size distribution, while Figure 6 presents its degree distribution.The supplied passages identify these analyses but do not provide their values or distribution shapes.Source: D SKILL GRAPH ANALYSIS · p. 13, Table 6 · p. 14, Figure 6 · p. 14
• The skill categories span major domains such as coding, document processing, DevOps, and security, as well as long-tail audio/speech, 3D simulation, and IoT/hardware workflows.Figure 5 presents the skill category distribution; the passage names example domains but gives no category counts.Source: D SKILL GRAPH ANALYSIS · p. 14, Figure 5 · p. 14

E PROMPTS

• SkillSynth uses DeepSeek Reasoner (v3.2) for extraction and Microsoft/Harrier-OSS-v1-27B for embedding.The embeddings are clustered to obtain unique instances.Source: E PROMPTS · p. 14
• Trajectory analysis segments an agent’s actions into scenario–skill pairs, with each scenario describing the environment state at a decision point and each skill a multi-step workflow that transitions to another scenario.Examples include a project with failing tests as a scenario and running pytest as a skill.Source: E PROMPTS · p. 15
• The extraction prompt requests a JSON array containing each pair’s step range, scenario, and skill, with both descriptions under 15 words.It instructs the model to return only the JSON array.Source: E PROMPTS · p. 15
• The scenario embedding prompt retrieves scenarios that describe the same real-world condition as the query scenario.Source: E PROMPTS · p. 15
• The skill embedding prompt retrieves skills that perform the same workflow as the query skill.Source: E PROMPTS · p. 15