Source-linked AI summary
SkillSandbox: Skill Verification via Dynamic Scenario Synthesis
Serin Kim, Kwangwook Seo, Dokyung Song, Jinyoung Yeo, Dongha Lee
TL;DR
Distilled skills can contain incorrect or non-transferable guidance, yet existing tasks may not expose the situation needed to test whether skills remain useful beyond their source experience. SkillSandbox synthesizes novel, skill-relevant scenarios and compares executions with and without each skill; across ALFWorld and WebShop, its verified libraries achieve the strongest downstream performance and improved efficiency.
Problem
Existing tasks may fail to expose the skill-specific situations needed to verify whether distilled guidance remains useful beyond its source experience.
Source: §1 · p. 1Method
SkillSandbox synthesizes a novel task and environment for each skill, then compares executions with and without the skill to assess executability, utility, and efficiency.
Source: §1 · pp. 1–2Results
Across ALFWorld and WebShop, verified libraries achieve the highest success rate and efficiency; SkillSandbox improves SR over the strongest execution-based curation baseline by up to 21.4%p.
Source: §4.2 · p. 7, §7 · p. 9Takeaways & Limitations
Scenario-specific execution evidence yields smaller gains when replaced by source-task or randomly selected task verification, with the gap widening on task types unseen in source experience.
Source: §1 · p. 2Verification selects from the original skill library rather than improving skills, so its gains are bounded by the reusable skills already present.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Self-evolving agents distill task-solving experience into skills for future reuse, but these skills can encode incorrect procedures or non-transferable knowledge. It is therefore critical to verify each skill's reusability: whether its guidance remains useful beyond the experience from which it was distilled. Such verification requires observing how a skill affects execution in new tasks, yet existing tasks may not expose the situations where the target skill can actually be exercised. To construct such situations, we propose SkillSandbox, a framework that dynamically synthesizes a task and its environment for each skill that are skill-relevant yet novel. A Proposer specifies the conditions to preserve and the source-specific details to vary, a Builder constructs an executable scenario, and a Verifier compares executions with and without the skill. The Verifier assesses executability, utility, and efficiency to assign a Keep or Reject verdict, determining whether the skill enters the library. Across ALFWorld and WebShop with three models, SkillSandbox consistently yields the strongest downstream performance and improved execution efficiency. Further analyses examine whether these gains reflect accurate assessment of skill reusability and identify which components of SkillSandbox contribute to them.
1 INTRODUCTION
Distilled skills may encode incorrect or non-transferable guidance, so their reusability must be tested beyond the source experience. SkillSandbox constructs novel, skill-relevant scenarios for verification and yields stronger downstream results across ALFWorld and WebShop.
1 INTRODUCTION
- Distilled skills can contain incorrect procedures or task-specific knowledge that repeatedly interferes when the skills are retrieved.The paper therefore frames reusability as whether guidance remains useful beyond the source experience.Source: §1 · p. 1
- Existing-task verification may lack the needed evidence: source-task replay is relevant but not novel, while other tasks may not expose the target situation.The task and environment must provide execution evidence in a new context where the skill can actually be exercised.Source: §1 · p. 1
- SkillSandbox synthesizes a task and environment per skill, with a Proposer specifying preserved conditions and variations, a Builder constructing scenarios, and a Verifier comparing runs with and without the skill.The Verifier assesses executability, task success, and efficiency to determine whether a skill is kept or rejected.Source: §1 · pp. 1–2
- Across three models, verified libraries achieve the highest success rate with the fewest steps on ALFWorld and WebShop, while analyses support the value of synthesized scenarios and verification criteria.The analyses also report prediction of individual skills’ helpful and harmful effects on held-out tasks.Source: §1 · p. 2
2 PRELIMINARY ANALYSES: SKILL-SPECIFIC EVIDENCE IS SPARSE IN EXISTING EXPERIENCE
The preliminary analyses find that existing tasks do not reliably provide skill-specific execution evidence. Semantically similar tasks often miss the target situation, and downstream task streams leave many skills insufficiently exercised.
2.1 QUESTION I: DO SEMANTICALLY SIMILAR TASKS EXPOSE THE SKILL-RELEVANT SITUATION?
- At most 6.66 out of 10 semantically similar retrieved tasks expose the skill-relevant situation, and expanding the candidate pool yields only marginal coverage gains.Task-level instruction similarity does not ensure execution reaches the states, interactions, or decisions where a skill applies.Source: §2.1 · pp. 2–3, Table 1 · p. 2
2.2 QUESTION II: HOW SPARSE ARE SKILL-RELEVANT SITUATIONS IN DOWNSTREAM EXPERIENCE?
- After 500 downstream tasks, 17–32% of skills remain unused and 45–56% lack five execution opportunities, leaving their reusability insufficiently verified.The figure tracks skills used at least once and at least five times; the reported percentages vary by access setting.Source: §2.2 · p. 3, Figure 1 · p. 3
2 PRELIMINARY ANALYSES: SKILL-SPECIFIC EVIDENCE IS SPARSE IN EXISTING EXPERIENCE
- The two analyses show that existing-task selection cannot reliably provide skill-specific evidence, motivating constructed scenarios that induce the target situation beyond its source experience.The section first frames whether existing tasks can supply that evidence as an open question.Source: §2 · p. 2, §2.2 · p. 3
3 METHODOLOGY
The method defines skill reusability in terms of executable guidance that improves success or efficiency in new situations. It synthesizes novel scenarios that preserve a skill’s condition while varying context, then compares executions with and without the skill.
3.1 PROBLEM SETUP
- The setup models an environment through state, action, and observation spaces, a transition rule, and a configuration, while a task specifies a goal and requirements.The configuration includes objects, arrangements, task-related data, and how information is presented.Source: §3.1 · p. 3
3.2 SKILL VERIFICATION
- Reusability means that a skill guides executable behavior that improves task success or execution efficiency in new situations satisfying its condition.Verification scenarios must be both skill-relevant and novel: they reproduce the condition while providing a context different from the source experience.Source: §3.2 · p. 4
3.3 SCENARIO SYNTHESIS
- The Proposer specifies how to reproduce the skill condition while modifying source details, and the Builder binds those specifications to environment data to construct executable task scenarios.A validation loop checks specification compliance and scenario novelty, discarding failures until five valid scenarios are obtained; the Verifier then compares matched trajectories with and without the skill and scores executability, utility, and efficiency.Source: §3.3 · pp. 4–5
4 EXPERIMENTS
Experiments compare verified skill libraries with unverified and execution-feedback baselines on ALFWorld and WebShop. SKILLSANDBOX achieves the strongest success-rate outcomes, with analyses showing benefits from verification and limits imposed by the original skill pool.
4.1 EXPERIMENTAL SETUP
- Evaluation covers six ALFWorld household task types and WebShop product-finding tasks, reporting SR on both, WebShop Score, and average steps.Recovery and Regression measure held-out task outcomes that flip from failure to success, or success to failure, relative to runs without a skill library.Source: §4.1 · p. 5
- Comparisons include unverified libraries and execution-based curation; SKILLSANDBOX retains only Vanilla Skill entries verified as Keep.Methods use the same 100 source tasks and libraries are frozen before held-out evaluation; the test sets contain 140 ALFWorld and 500 WebShop tasks, with top-3 skill retrieval.Source: §4.1 · pp. 5–6
4.2 MAIN RESULTS
- Unverified libraries often underperform No Skill in SR, frequently taking more steps while solving fewer tasks.The authors attribute this pattern to redundant exploration from non-reusable skills.Source: §4.2 · p. 6
- Up to 21.4%p higher SR than the strongest execution-based curation baseline: SKILLSANDBOX wins across all models and benchmarks.The comparison covers ExpeL, ACE, and SkillOS; Table 2 reports held-out ALFWorld and WebShop results.Source: Table 2 · p. 6, §4.2 · p. 7
- Approximately 37% lower Regression on ALFWorld with Gemini, while SKILLSANDBOX and MemP achieve the same Recovery.MemP lowers SR relative to No Skill, whereas SKILLSANDBOX improves it.Source: §4.2 · p. 7
- Up to 9.2%p more Recovery than Vanilla Skill, with Regression reduced or maintained and Recovery exceeding Regression in every setting.SKILLSANDBOX has the lowest Regression across settings and the highest Recovery in all but one.Source: §4.2 · p. 7
- More than 10%p with lower Regression on WebShop; gains remain bounded by reusable skills in the original library.The verified library selects from the original skills rather than improving them.Source: §4.2 · p. 7
5 ANALYSIS
The analysis tests whether synthesized scenarios improve verification, whether verdicts predict held-out skill reusability, whether a stronger proposer is needed, and what the scenarios reveal about skill reusability.
5.1 RQ1: DOES SCENARIO SYNTHESIS IMPROVE THE EFFECTIVENESS OF VERIFICATION?
- Synthesized scenarios yield the highest verdict F1 and best-performing skill libraries on both benchmarks, with held-out SR up to 15.7%p above alternatives.Source-task and random-task verification provide weaker evidence; on ALFWorld, their retained libraries fall below No Skill.Source: §5.1 · p. 7, Figure 3 · p. 8
5.2 RQ2: DOES VERIFICATION PREDICT SKILL REUSABILITY?
- On WebShop, verification retains 82.8% of helpful skills; on ALFWorld, it retains 81.8% despite only 23.9% helpful skills, raising the verified library’s helpful-skill share to 39.1%.The table reports precision, recall, F1, and specificity of verdicts on both benchmarks.Source: Table 3 · p. 8, §5.2 · p. 8
5.3 RQ3: DOES RELIABLE SKILL VERIFICATION REQUIRE A STRONGER MODEL?
- The executor’s own model matches or exceeds GPT-5.6 Luna on every metric on both benchmarks and retains more helpful skills.The comparison replaces only the proposer while holding the skills, executor, builder, and verifier fixed.Source: §5.3 · p. 8
5.4 RQ4: WHAT DO THE SYNTHESIZED SCENARIOS REVEAL ABOUT SKILL REUSABILITY?
- The analysis measures executability, utility, and efficiency; efficiency comparisons include only pairs where both runs succeed.Executability concerns whether the action implements the skill principle; utility is the success-rate gain, and efficiency compares remaining action counts.Source: §5.4 · p. 8
- On WebShop, Qwen3.5-27B and Gemini gain success but add actions; on ALFWorld, both Qwen models gain success and save actions, while Gemini’s high executability accompanies lower success and more actions.The findings show that success gains alone do not capture action costs or whether executable guidance benefits task completion.Source: §5.4 · p. 9, Figure 4 · p. 9
6 RELATED WORK
Prior work studies skill acquisition and curation as well as task and environment synthesis, but the passages identify skill-specific verification evidence as a remaining challenge.
Skill acquisition and curation
- Skill acquisition and curation range from reusable programs and textual guidance to feedback-driven updates and explicit source-task or paired-run verification, which may still lack skill-relevant novel evidence.The paper identifies reliable opportunities to evaluate each skill as a central verification problem.Source: §6 · p. 9
Task and environment synthesis
- Task and environment synthesis expands training opportunities or adapts difficulty, but these objectives do not ensure execution encounters a target skill’s situation.The paper proposes dynamically synthesizing tasks and environments for each target skill at verification time.Source: §6 · p. 9
7 CONCLUSION
The paper introduces SkillSandbox to test whether skills remain useful in new contexts and reports improved success, efficiency, and verification quality across its evaluations.
7 CONCLUSION
- SkillSandbox synthesizes tasks and environments that reproduce each skill’s applicable conditions beyond its source experience; its verified libraries achieve the highest success and efficiency across ALFWorld and WebShop.Further analyses report improved verification quality from synthesized scenarios and effective verification with the executor’s own model.Source: §7 · p. 9
AI USE STATEMENT
The authors describe AI assistance in writing and code implementation, then specify baseline verification methods and their evaluation setup.
AI USE STATEMENT
- The authors used AI assistance for manuscript polishing and experiment-code implementation, then reviewed the text and manually verified the code.They take responsibility for the final work, including AI-assisted text and code.Source: AI USE STATEMENT · p. 10
A BASELINE DETAILS
- All libraries use the same 100 source tasks and are frozen before held-out evaluation; Vanilla Skill, ReasoningBank, and MemP retain generated skills or memories without utility-based filtering.ReasoningBank appends extracted memories, while MemP appends workflow summaries regardless of task success.Source: A BASELINE DETAILS · p. 14
- Execution-based baselines use source replay in ExpeL or downstream-task curation in ACE and SkillOS to revise, add, or remove stored guidance.Their implementations use different update procedures, including retries, feedback counters, pruning, and BM25 retrieval.Source: A BASELINE DETAILS · p. 14
B IMPLEMENTATION DETAILS
The implementation distills skills into applicability conditions and actionable principles, then tests their effects through matched with-skill and without-skill executions. Verification combines executability with changes in success and action efficiency, while a separate held-out protocol assesses skill effects.
B IMPLEMENTATION DETAILS
- Executor settings cover both benchmarks; history length counts preceding observation–action pairs, with the current observation supplied separately.The executor also receives admissible actions and recent interaction history.Source: Table 4 · p. 14, B IMPLEMENTATION DETAILS · p. 14
- Skills are distilled into an applicability condition and actionable principle, with the top three full skill texts retrieved by cosine similarity to each task query.Retrieval uses text-embedding-3-small.Source: B IMPLEMENTATION DETAILS · p. 14
- The verifier aligns with-skill and without-skill trajectories at identical pre-action observations and retains the earliest skill-use pair for each observation.It compares subsequent behavior from the same observation.Source: B IMPLEMENTATION DETAILS · p. 15
- Utility compares binary terminal rewards, while efficiency discounts each reward by the remaining actions from the matched step.The reward difference is ∆u = r+γ^(T+−t+) − r−γ^(T−−t−), with 0 < γ < 1.Source: B IMPLEMENTATION DETAILS · p. 15
- Executability is marked when a skill-guided action implements its instruction, and only executable pairs contribute to the averaged reusability score and verdict.The verifier averages e∆u within trajectory pairs and then across synthesized scenarios.Source: B IMPLEMENTATION DETAILS · p. 15
- For the held-out WebShop analysis, skill effectiveness is (Help_k − Harm_k)/500 across 500 tasks, and is compared with verification scores over 50 skills.Help_k counts tasks changed from failure to success; Harm_k counts tasks changed from success to failure.Source: B IMPLEMENTATION DETAILS · p. 15
C SCENARIO SYNTHESIS
The section describes how scenario transformations and causes of rejected skills are summarized and categorized. The synthesis-type figure reports the share of scenarios using each transformation.
C SCENARIO SYNTHESIS
- Scenario operations and implementation plans are summarized, then used to induce synthesis types for WebShop and ALFWorld scenarios.The typology covers Qwen3.5-27B scenarios.Source: C SCENARIO SYNTHESIS · p. 15
- Harm categories are derived from cases where the skill run fails, the no-skill run succeeds, and the scenario score is negative.A model explains how the skill could have contributed to failure, and the explanations are grouped into types.Source: C SCENARIO SYNTHESIS · p. 15
- Figure 5 shows the percentage of scenarios using each transformation, and scenarios may use multiple transformations.The bars are reported separately for WebShop and ALFWorld.Source: C SCENARIO SYNTHESIS · p. 15, Figure 5 · p. 16
D CASE STUDY
The case study shows how synthesized scenarios test whether a skill’s source-derived guidance transfers to a changed task and environment. Both examples lead to rejection when following the principle harms execution compared with acting without it.
D CASE STUDY
- Table 5 marks the skill under verification and highlights the skill-guided action and the part of the principle associated with rejection.Its examples use paired executions of the same synthesized scenario, with and without the skill.Source: Table 5 · p. 16, D CASE STUDY · p. 17
- In ALFWorld, activating a stationary appliance before retrieving an object fails to transfer when cooling requires carrying the bread to the fridge first.The skill-guided run reaches the step limit; without the skill, the agent completes the task in six steps.Source: Table 5 · p. 16, D CASE STUDY · p. 17
- In WebShop, requiring confirmation in both product sections is unnecessary when the qualifying claim appears in Features alone.The no-skill run purchases in four steps, while the skill-guided run repeatedly checks Description and exhausts its action budget.Source: Table 5 · p. 16, D CASE STUDY · p. 17
E ADDITIONAL RESULTS
The additional results relate verification scores to held-out skill effects, examine retrieval size and rejection causes, and document prompt designs for execution, skill distillation, and scenario proposal.
E ADDITIONAL RESULTS
- Both Proposers have positive Spearman correlations between verification scores and held-out WebShop effects, while Pearson correlation does not establish calibration to improvement magnitude.Table 6 reports the score correlations with net held-out effects.Source: Table 6 · p. 17, E ADDITIONAL RESULTS · p. 17
- Retrieval size k = 3 gives the best success rate on both benchmarks, tying k = 5 on ALFWorld and outperforming k = 1 and k = 5 on WebShop.The comparison uses Gemini 3.1 Flash-Lite with the retained skill pool and other settings fixed.Source: E ADDITIONAL RESULTS · p. 17, Table 7 · p. 18
- Figure 6 groups rejected skills by primary failure cause within executability, utility, and efficiency criteria, showing the three largest types and pooling the rest as Other.A skill may fail more than one criterion, but receives one primary cause per criterion.Source: E ADDITIONAL RESULTS · pp. 17–18, Figure 6 · p. 18
F PROMPTS
- The prompt appendix presents templates for execution and skill distillation, plus structured summaries of scenario-proposal prompts and their inputs and output schemas.Runtime inputs use braced placeholders; retry and format-repair messages are omitted.Source: F PROMPTS · p. 19
F.1 EXECUTOR
- Executor prompts provide the task, current observation, admissible actions, and recent interaction history, with the agent instructed to choose an executable action.The WebShop prompt also specifies the current and total step counts.Source: §F.1 · p. 19
F.2 SKILL DISTILLATION
- Distillation prompts produce a When to apply condition and a Principle from a source trajectory, with variants for success and failure in both environments.The prompts ask for one concise reusable lesson.Source: §F.2 · p. 20
F.3 PROPOSER
- Proposer instructions preserve condition requirements while varying supported scenario details, and leave the skill-guided decision unfinished rather than guaranteeing a favorable verdict.The summarized schemas cover condition interpretation, scenario design, and operation specification for WebShop and ALFWorld.Source: §F.3 · pp. 21–22