PaperlayerView exact revision on arXiv

Source-linked AI summary

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Xinyu Che, Junqi Xiong, Yunfei Ge, Xinping Lei, Shihao Li, Hang Yan, Han Li, Yuanxing Zhang, Zhiqi Bai, Jinhua Hao, Ming Sun, Han Li, Jiaheng Liu

arXiv:2606.01993v1cs.CLcs.AIcs.LG

TL;DR

VLM agents need to turn noisy, human-oriented multimodal guides into executable skills grounded in their runtime state. MMG2Skill-Bench evaluates this problem, while MMG2Skill compiles guides into editable skills and revises them from agent-visible trajectories. Across domains and six VLM backbones, MMG2Skill consistently improves over vanilla agents, while structured construction, trajectory-driven revision, and calibrated stopping address distinct execution challenges.

  • Problem

    Human-authored multimodal guides contain useful procedural knowledge but are difficult to use directly because they are not grounded in an agent’s runtime state.

  • Method

    MMG2Skill-Bench evaluates guide-to-skill learning, and MMG2Skill compiles guides into editable skills, conditions a fixed VLM agent on them, and revises them from agent-visible trajectories without benchmark scores.

  • Results

    MMG2Skill consistently outperforms vanilla agents in every model–domain setting, with macro-average gains of +12.8 to +25.3 percentage points across six VLM backbones.

  • Takeaways & Limitations

    Raw guide prompting can hurt performance, while structured skill construction and trajectory-driven revision improve execution; calibrated analyzer-based stopping saves 25–53% of attempts on success-inferable tasks.

  • Takeaways & Limitations

    The system treats guides as static inputs selected before execution and does not study guide discovery, source filtering, or deciding when no guide is reliable enough to use.

Abstract

from arXiv · show

Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks. However, such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. To bridge the gap between human-oriented guides and agent-executable skills, we formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. To evaluate the capability of existing agents on this task, we introduce MMG2Skill-Bench, the first benchmark designed for this problem. We further propose MMG2Skill, a closed-loop framework that compiles guides into editable skills, conditions a fixed vision-language model (VLM) agent on these skills during execution, and revises the skills from trajectory-level root-cause feedback without using benchmark scores. Across GUI control, open-ended gameplay, and strategic card play with six VLM backbones, MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting, achieving macro-average gains of +12.8 to +25.3 percentage points across backbones. Ablation studies show that directly prompting agents with raw guides can degrade performance, while both structured skill construction and trajectory-driven revision are necessary for the observed improvements. On success-inferable tasks, analyzer-based early stopping further prevents late-stage performance regressions and saves 25%-53% of attempts when the success signal is properly calibrated.

1 Introduction

MMG2Skill addresses the gap between human-oriented multimodal guides and executable, state-grounded skills for long-horizon VLM agents. It introduces a benchmark and closed-loop framework that converts guides into editable skills, revises them from trajectories, and improves performance across settings.

  • Motivation: VLM agents need reusable procedures that specify applicability, progress states, recovery, and runtime grounding during long-horizon interactive work.The paper defines these executable and editable procedural objects as skills.
  • Motivation: Raw public guides are difficult to execute directly because they mix procedures, assumptions, navigation text, and recovery advice for human readers.Agents must align guide content with current observations and track which steps are already satisfied.
  • Contributions: MMG2Skill-Bench is the first benchmark for guide-to-skill learning from in-the-wild multimodal guides across GUI, game, and strategy tasks.The benchmark evaluates whether agents can convert public procedural material into execution-grounded skills.
  • Contributions: MMG2Skill compiles guides into editable skills, conditions a fixed VLM agent on them during rollout, and revises them using agent-visible trajectories without benchmark scores.The framework uses trajectory-level root-cause feedback for revision.
  • Results: +12.8 to +25.3 percentage points: MMG2Skill’s macro-average gains over vanilla agents across six VLM backbones, consistently in every model–domain setting.The evaluation covers GUI control, open-ended gameplay, and strategic card play.
  • Findings: Raw guide prompting can hurt performance, whereas structured skill construction and trajectory-driven revision provide the reported improvements.Analyzer-based early stopping saves 25–53% of attempts on success-inferable tasks when the success signal is calibrated.

2 MMG2Skill-Bench

MMG2Skill-Bench evaluates guide-to-skill learning across three interactive regimes using task-relevant public guides and domain-native scoring. Its controls are designed to test procedural grounding rather than answer copying or access to solution traces.

  • Benchmark scope: MMG2Skill-Bench pairs each task instruction with task-relevant public guide material to evaluate whether skills remain useful during environment interaction.The benchmark focuses on converting guides into procedural knowledge that supports execution.
  • Benchmark composition: The benchmark spans desktop GUI control, open-ended game play, and turn-based strategy tasks.GUI uses OSWorld, Game uses OpenHA Minecraft tasks in MineStudio, and Strategy uses Doudizhu and Mahjong from RLCard.
  • Benchmark composition: 130 success-inferable tasks comprise MMG2Skill-Bench.The main evaluation restricts tasks to those whose outcomes can be inferred from agent-visible trajectories or public final states.
  • Evaluation protocol and controls: Success-inferable evaluation keeps analyzer-based revision and stopping within the information boundary available during deployment.No-Limit Hold’em is excluded from the main Strategy evaluation because payoff can depend on unobserved opponent private cards.
  • Guide sources: Domain-specific guides come from product documentation, how-to articles, wiki pages, walkthroughs, rule descriptions, and beginner strategy material.Each task receives a task-relevant subset of public human-authored guide material.
  • Evaluation protocol and controls: All agents receive the same domain-specific system prompt before guides are added, while guides supply additional procedural knowledge to be converted into skills.The shared prompt defines interfaces, action grammar, observations, controls, and legal-action constraints where applicable.
  • Evaluation protocol and controls: Guides exclude benchmark solution traces, gold action sequences, hidden labels, and environment-specific trajectories, and tasks whose answers can be copied directly are excluded.These controls test procedural grounding rather than retrieval.
  • Evaluation protocol and controls: Each task is scored from 0 to 1 using its domain-native evaluator.Termination rules, step caps, opponents, and hyperparameters are specified separately in the benchmark protocol.

3 MMG2Skill Framework

MMG2Skill converts multimodal guides into editable skills, conditions a fixed VLM agent on them during rollout, and iteratively revises them using trajectory-grounded diagnoses. Its closed loop preserves earlier fixes, restores omitted guide details, and can stop when visible evidence suggests success.

  • Framework Overview: MMG2Skill keeps the VLM policy fixed while optimizing an editable skill set induced from the guide.The skill set, rather than model parameters, is updated across attempts.
  • Tutorial-to-Skill Stage: Guide construction normalizes multimodal instructions into editable SKILL.md procedures organized around reusable actions, applicability conditions, expected-state cues, and recovery knowledge.These fields are conceptual and can be expressed through procedural text, state descriptions, and referenced guide images.
  • Skill-Conditioned Agent Loop: During each rollout, the current skill set remains in context alongside recent observation–action history, while the domain’s action interface is unchanged.Every action is conditioned jointly on the task, bounded recent history, and current skills.
  • Skill Revision: The analyzer uses only the task instruction and agent-visible trajectory to produce evidence about failures and a self-judged outcome assessment.It cannot access benchmark scores or hidden environment state; likely_success serves as the candidate stopping signal.
  • Skill Revision: After unsuccessful attempts, the refiner edits the skill representation using the original guide, current skills, and the accumulated diagnosis chain.Accumulating diagnoses preserves earlier fixes, while retaining the guide helps recover omitted or misunderstood procedural details.

4 Experiments

Experiments across GUI, Game, and Strategy test whether structured, revisable skills improve fixed VLM agents and whether early stopping supports deployment. MMG2Skill improves broadly, while raw guides can hurt and late revisions can regress.

  • RQ1: Overall performance: MMG2Skill improves all 18 model–domain cells over vanilla, with gains spanning weak and strong backbones.The largest single-cell gain is +33.33 pp for Gemini on Game; Qwen improves by +25 pp on GUI, while GPT-5.5 improves by +6.67 pp on Game.
  • RQ1: Overall performance: MMG2Skill reduces displayed-attempt steps while improving GUI and Game scores, indicating fewer exploratory detours rather than longer rollouts.
  • RQ2: Mechanism ablation: Raw guide injection is flat on GUI but decreases Game and Strategy performance, with the largest loss of −1.67 pp on Game.The result is attributed to guide–environment grounding mismatches and the lack of an editable interface for resolving them.
  • RQ2: Mechanism ablation: Structured skill construction matches or exceeds vanilla across all domains, while revision contributes over 90% of the total gain on Game and Strategy.The largest single-stage revision contribution is +22.22 pp on Game; SKILL.md functions as an editable interface rather than merely compressed guide context.
  • RQ3: Revision dynamics: Revision repairs guide–runtime mismatches by converting trajectory evidence into missing runtime contracts, such as explicit output-file verification in the GIMP crop task.
  • RQ4: Early-stop deployment: Early stopping is safer than always taking the latest attempt when revisions are non-monotonic, reaching 66.1% versus 47.8% on Game at N=5.It saves 25.44–52.92% of attempts, while the analyzer exceeds 74% precision across domains and has lower Strategy recall of 70.8%.
  • Residual-failure analysis: Residual failures remain domain-specific: Game emphasizes grounded execution and budget exhaustion, while Strategy emphasizes wrong approaches, decision-rule conversion, and skill-following failures.

5 Related Work

Related work evaluates general agents, context learning, reusable skills, and procedural augmentation, but these lines address different combinations of environments, supplied knowledge, and skill representation.

  • Agent and context benchmarks: Agent and context benchmarks cover end-to-end environments or learning procedures from supplied textual contexts.
  • Skills and procedural augmentation: Recent systems augment language models with reusable language memories, executable programs, skill libraries, tutorials, demonstrations, reward signals, training data, or trajectory corpora.

6 Conclusion

The paper introduces a benchmark and closed-loop framework for turning in-the-wild multimodal guides into execution-grounded skills. Across three domains, skill construction, trajectory-driven revision, and calibrated early stopping address distinct deployment challenges.

  • MMG2Skill-Bench evaluates whether VLM agents can convert in-the-wild multimodal guides into execution-grounded skills across GUI control, open-world games, and strategic card play.
  • MMG2Skill compiles guides into editable skills and revises them from agent-visible trajectories, while raw guide prompting can hurt performance.
  • Analyzer-based early stopping improves deployability on success-inferable tasks by avoiding late-stage regressions when the success signal is calibrated.

Limitations

The study isolates execution-grounded skill learning after a task-relevant guide has been provided, leaving guide discovery and source filtering outside its scope. Evaluation is also constrained by the cost and sequential nature of multimodal interactive rollouts.

  • Scope boundary: The system assumes a task-relevant guide has already been selected and does not study guide discovery.Extending it with retrieval and source filtering is left as a separate problem.
  • Scope boundary: Static, low-quality, or misaligned guides can introduce errors that trajectory revision only partly repairs.The system treats guides as fixed inputs selected before execution.
  • Evaluation cost: Frontier-agent evaluation is expensive because multimodal calls are required for execution, skill revision, and analyzer-based selection.The authors therefore use fixed model versions and evaluation budgets.
  • Evaluation cost: Sequential GUI and simulator interaction, together with commercial API rate limits, constrains evaluation throughput and turnaround time.These constraints motivate fixed protocols and explicit model-version records for reproducibility.

Ethical Considerations

The study limits evaluation to sandboxed benchmark tasks and separates its artifacts from restricted third-party materials. It uses synthetic instructions and public guides without human-subject experiments or private user data.

  • Safety boundaries: Evaluation is restricted to sandboxed benchmark tasks rather than real accounts, payment systems, private documents, or production services.Credentials appearing in task instructions are synthetic benchmark artifacts.
  • Artifact handling: Third-party benchmark and guide assets are kept separate from the guide corpus and revision method, with redistribution limited by licenses or terms.OSWorld, OpenHA, MineStudio, and RLCard are used for their intended research and evaluation purposes.
  • Artifact handling: The public release includes code, metadata, prompts, scripts, and fixed RLCard opponent checkpoints, while excluding API keys, private logs, and restricted assets.Guide contents are released only when redistribution is permitted; otherwise, provenance metadata and source URLs are provided.
  • Disclosure: The authors disclose AI assistance for language polishing, LaTeX checking, and code-debugging support, while stating that authors reviewed the scientific content.
  • Human subjects: The benchmark does not involve human-subject experiments, recruited participants, or private user data.Human effort is limited to benchmark construction, guide selection, and result inspection by the authors.

B Method and Implementation Details

MMG2Skill uses chunked trajectory analysis and history-aware skill refinement within a closed-loop procedure, then applies the same core pipeline across GUI, game, and strategy settings. The implementation fixes models, budgets, interfaces, task selections, and domain-specific prompt boundaries for controlled evaluation.

  • Revision loop: Algorithm 2 constructs skills from a guide, executes repeated attempts, analyzes trajectories in chunks, and refines skills from accumulated diagnoses.The loop supports online early stopping and offline diagnostics without giving the analyzer benchmark scores.
  • Trajectory analysis: Chunked analysis keeps analyzer calls within a fixed context budget and emits a final root-cause diagnosis for stopping and refinement.Intermediate calls receive the task, rolling summary, and current trajectory chunk.
  • Skill refinement: History-aware refinement rewrites a normalized Markdown skill set using the original guide, current skills, and diagnosis history.This lets validated behavior persist while later failures are incorporated.
  • Extensibility: The released pipeline supports new environments by retaining skill construction, conditioned execution, trajectory analysis, refinement, and early-stop reporting while adding only domain boundaries.Researchers can also compare alternative guide-to-skill methods under shared budgets and protocols.
  • Evaluation settings: The benchmark uses OSWorld for GUI, MineStudio Minecraft tasks for Game, and RLCard Doudizhu and Mahjong tasks for Strategy.A No-Limit Hold’em diagnostic uses the same RLCard engine but is outside the three main benchmark domains.
  • Controlled implementation: All benchmark evaluations use an attempt budget of N=5, with model-specific interface handling and domain-specific trajectory chunk sizes.The same backbone is used for skill construction, execution, analysis, and refinement within each condition.
  • Prompt–guide boundary: Prompts define executable interfaces and legal action constraints, while guides provide domain procedures and strategic heuristics.The split covers GUI tools, Minecraft recipes and workflows, and RLCard tactics.
  • Benchmark construction: The guide corpus statistics and task selection are organized by benchmark domain, with task counts and source distributions recorded for GUI, Game, and Strategy.

C.2 Benchmark Task Selection

The benchmark task selection spans GUI application domains, Minecraft task families, and RLCard strategy hands, with statistical tests evaluating per-task improvements across models. The selection preserves upstream domain coverage while structuring comparisons by benchmark regime.

  • GUI tasks: MMG2Skill-GUI covers all 10 OSWorld application domains with proportional per-domain sampling.This preserves the relative weight of operation-heavy domains such as multi_apps, os, and vs_code.
  • Game tasks: MMG2Skill-Game samples 10 tasks each from mine_block, craft_item, and smelt_item task families.
  • Statistical analysis: Per-task improvements are averaged across six models and tested with one-sided paired Wilcoxon signed-rank and sign tests.
  • Statistical analysis: For each domain–modality column, one paired Wilcoxon test per model is combined using Stouffer’s Z-method.Strategy concatenates hands from both underlying games within a model.

D.1 Complete Result Tables

Across complete result tables and diagnostics, MMG2Skill’s gains remain statistically reliable, stable across reruns, and dependent on editable revision rather than feedback alone. Revision dynamics are heterogeneous and non-monotonic, motivating calibrated early stopping and bounded skill growth.

  • Early-stop comparison: Early-stop performance is no worse than full-run performance in nearly all success-inferable result cells.The advantage is especially pronounced on Game and Strategy-DD, where late-stage full-run regression is substantial.
  • Oracle comparison: 6.6 points on GUI, 5.6 on Game, and 4.4 on Strategy-DD are the mean oracle-minus-early-stop gaps, while Strategy-MJ has a +15.8 pp gap.Early-stop ties the oracle in 9 of 48 cells across HTML and Screen.
  • Statistical significance: All six domain–modality cells show significant MMG2Skill improvements under both paired per-task tests and model-wise p-value combination.Per-task tests reach Wilcoxon p ≤8.9 × 10^-4 and sign p ≤2.0 × 10^-3; combined tests reach Z ≥+4.27 and p ≤9.6 × 10^-6.
  • Multi-run stability: A rerun across 27 model–method–domain cells changes scores by 2.14 percentage points on average, while the MMG2Skill–Vanilla gap remains positive for all representative backbones.The rerun uses the same hyperparameters and N=5 attempt budget.
  • Revision dynamics: Per-model revision traces are uneven: some models plateau immediately, others improve through five attempts, and late full-run regressions appear in Game and Strategy.On Qwen3.6-Plus, early-stop scores plateau by N=5 on GUI and Game, while full-run scores continue oscillating through N=7.
  • Skill growth: Skill character count grows with revision budget but visibly flattens after early attempts, while the same likely_success rule bounds the deployed prompt size.The stopping mechanism both selects the deployed skill set and limits environment attempts.
  • Mechanism ablation: Root-cause memory alone remains below MMG2Skill at N=5 in all four tested model–domain pairs.The diagnostic adds trajectory feedback to raw guides but disables skill construction and refinement; persistent editable edits are more effective.

E.5.1 Analyzer Signal Calibration

Analyzer calibration supports likely_success as the preferred stopping signal on success-inferable tasks, but outcome inference degrades when completion depends on hidden information. The section also reports efficiency gains on GUI and Game, where step counts have operational meaning.

  • Signal calibration: likely_success has the best precision–recall balance across success-inferable domains, whereas no_issue usually sacrifices too much recall for small precision gains.The calibration compares the two candidate stopping signals against binarized benchmark outcomes.
  • Success-inferable tasks: GUI, Game, and Strategy-DD retain aggregate likely_success precision and recall of at least 87% and 93%, respectively.Strategy-MJ drops to 71.4% precision and 49.5% recall as hand-composition and three-outcome inference become harder.
  • Private-information boundary: 74.2% likely_success trigger rate accompanies 9.9% precision and 54.5% recall in the private-information diagnostic.The actual positive rate is 13.4%, illustrating severe miscalibration when outcome-determining cards remain unobserved.
  • Scope boundary: No-Limit Hold’em is excluded from the main Strategy evaluation because chip payoff depends on opponent hole cards that may never appear in the agent-visible trajectory.The trajectory can expose decision quality but cannot reliably reveal whether the hand was won.
  • Step efficiency: MMG2Skill reduces macro-average displayed-attempt steps from 10.08 to 9.21 on GUI and from 41.17 to 36.08 on Game.Step count remains diagnostic rather than the primary objective, since failed attempts can also terminate early.

F.2 API-Call and Token Budget Decomposition

Early stopping reduces deployment cost mainly by selecting fewer rollout attempts, while preserving the revision mechanism that turns trajectory evidence into concrete skill edits. The qualitative cases illustrate both successful runtime-contract repair and false-positive risk.

  • API-call decomposition: Early stopping lowers API-call cost mainly through fewer rollout attempts, not through the fixed one-call extractor.Refiner calls also decline from four under full-run to the average revisions before the selected stopping attempt.
  • Token decomposition: Early stopping reduces per-task token consumption in every logged domain–model cell, with larger savings on GUI and Game than Strategy.Agent-loop tokens dominate GUI and Game, while Refiner overtakes them on Strategy for Gemini-3.1-Pro-Preview.
  • API-call decomposition: 37.72% to 59.33% call savings occur on Game, compared with 22.99% to 45.06% on GUI and 10.95% to 22.15% on Strategy.The domain differences are consistent with later stopping in Strategy.
  • Runtime-contract repair: A GIMP crop attempt improves from score 0 to 1.0 after refinement adds a longer wait and an explicit output-file verification gate.The edit converts trajectory evidence about missing cropped.png into a runtime contract.
  • False-positive diagnostic: A false-positive archive rollout triggers likely_success despite an oracle score of 0 because the trajectory does not verify all submission requirements.Visible execution success is therefore insufficient when required outputs are not explicitly checked.
  • Game trajectory: The crafting case shows an initially empty inventory, followed by successful stick crafting after the agent recognizes that wood must be obtained first.The later trajectory reports a successful craft and collected stick.

G.4 MMG2Skill-Game: Craft Wheat

The craft-wheat case contrasts a vanilla agent’s incorrect impossibility judgment with MMG2Skill’s guide-derived recipe execution, showing how procedural knowledge corrects non-obvious task assumptions.

  • Interpretation: The case shows that guide-derived skills can recover non-obvious recipes, not only long action sequences.The complementary instance demonstrates reverse recipe use from hay bales to wheat.
  • Vanilla Agent: The vanilla agent repeatedly concludes that crafting wheat is impossible because it believes wheat must be harvested rather than crafted.Its trajectory treats the visible yellow item as seeds or wheat and emits a failure-oriented interpretation.
  • MMG2Skill Agent: MMG2Skill identifies the inventory item as a hay bale and places it into the crafting grid.The agent’s trajectory explicitly uses the guide-derived hay-bale recipe instead of accepting the vanilla agent’s premise.
  • Outcome: The agent successfully crafts and collects 9 wheat, with the crafting grid empty and 35 hay bales remaining.The final observation reports the wheat stack, an empty output slot, and a recipe-unlock notification.
  • Analyzer: The analyzer judges outcomes from agent-visible traces, treating terminal success or infeasibility claims as self-reported signals requiring observational verification.It receives the agent response, emitted actions, and saved observations, without a rule-based grader or environment-side done flag.
Loading 2606.01993v1…