Source-linked AI summary
MMSkills: Towards Multimodal Skills for General Visual Agents
Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu
TL;DR
Visual agents need reusable procedures that connect actions with visual state and progress evidence, but existing skill representations are mainly textual or code-level. MMSkills packages procedures with state cards and keyframes, and improves frontier and smaller multimodal agents across GUI and game-based benchmarks.
Problem
Existing visual-agent skill packages mainly specify actions textually or in code, leaving open how to represent and consult visual state evidence during reuse.
Method
MMSkills combines textual procedures, runtime state cards, and multi-view keyframes, generated from public trajectories and consulted through branch loading.
Results
MMSkills improve performance over no-skill and text-only conditions across GUI and game-based benchmarks for frontier and smaller multimodal models.
Takeaways & Limitations
External multimodal procedural knowledge complements model-internal priors for visual agents across GUI and game-based tasks.
Takeaways & Limitations
Full three-condition results for Gemini 3.1 Pro and Kimi-K2.6 are reported only on OSWorld because of higher inference cost and wall-clock time.
Abstract
from arXiv · showhide
Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.
1 Introduction
MMSkills defines reusable multimodal procedural knowledge for visual agents by coupling action procedures with state-dependent visual cues and verification evidence. It provides an agentic trajectory-to-skill Generator and branch loading, and improves visual-agent performance across GUI and game benchmarks.
- Motivation: Multimodal procedural knowledge binds action procedures to the visual evidence and state-dependent decisions needed for reuse.Reusable guidance must specify when a procedure applies, which visible cues matter, and what evidence verifies progress, failure, or completion.
- Representation: MMSkills represents reusable procedures as compact, state-conditioned packages combining textual procedures, runtime state cards, and multi-view keyframes.State cards encode applicability conditions, visible cues, verification cues, and available views; keyframes ground critical states with full-frame, focused, and optional before/after views.
- Generation: The trajectory-to-skill Generator converts public, non-evaluation trajectories into multimodal skill packages through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing.The pipeline addresses generation from interaction experience rather than hand-written examples or raw demonstration replay.
- Utilization: Branch loading selects and aligns needed state cards and keyframes in a temporary branch before returning compact structured guidance to the main agent.This avoids injecting the entire multimodal package into the main trajectory, reducing context pressure and over-anchoring to reference images.
- Evaluation: MMSkills improve performance over no-skill and text-only skill conditions across GUI and game-based visual-agent benchmarks and multiple model families.The evaluation covers OSWorld, macOSWorld, VAB-Minecraft, and Super-Mario, with gains for both frontier and smaller multimodal models.
2 Methods
MMSkills package reusable visual procedures with state-conditioned cards and multi-view keyframes, generated from public trajectories and consulted through a branch that distills evidence into guidance while preserving live-observation grounding. The method combines structured multimodal representation, staged skill construction, conservative visual grounding, and gated branch-based inference.
- Multimodal skill package: Each MMSkill couples a reusable textual procedure with runtime state cards and aligned multi-view keyframes for decision-relevant procedural states.State cards specify validity conditions, visible and verification cues, and available views; keyframes make states visually recognizable at runtime.
- Skill Generation: The Generator converts public non-evaluation trajectories into skill libraries through clustering, atomic skill planning, merging, text-first drafting, and image grounding with audit.A meta-skill supplies reusable scripts, schemas, and quality gates, while overly broad skills are rejected during merging.
- Skill Generation: Conservative grounding adds views only for state recognition, transition comparison, or completion verification, treating them as diagnostic evidence rather than demonstrations to replay.Full-frame views preserve global context, focus crops localize cues, and optional before/after views expose transitions; references are not coordinates to copy.
- Branch-loaded inference: The main agent uses branch output as an intermediate planning signal while choosing actions from the live screenshot, preventing reference images from overriding current observations.This design addresses context pressure and visual anchoring caused by directly loading state cards, keyframes, and transition examples into the main interaction context.
- Branch-loaded inference: Branch loading first selects relevant state cards and views using the live observation, history, procedure, and state descriptions, loading no images when text suffices.The branch then aligns selected evidence with the live state and returns applicability, subgoal, plan, negative constraints, and verification guidance.
3 Experiments
Experiments across GUI and game benchmarks show that MMSkills improve visual-agent performance, especially for weaker models, while ablations and behavioral analyses attribute gains to state-conditioned visual evidence and branch-loaded use.
- Overall performance: MMSkills improve OSWorld overall success across evaluated model families, including Gemini 3.1 Pro (44.08% →50.11%), Gemini 3 Flash (36.65% →47.97%), and Qwen3-VL-235B (21.34% →39.17%).Text-only skills also help but are less stable across domains, indicating that procedures alone are insufficient for visually state-dependent skill use.
- Overall performance: For Qwen3-VL-8B-Instruct, MMSkills raise OSWorld success from 10.78% to 25.40% and VAB-Minecraft success from 23.28% to 38.79%.The results indicate that explicit visual procedural knowledge can compensate for limited model-internal priors.
- Overall performance: MMSkills transfer beyond Ubuntu desktop tasks, improving completed macOSWorld and Super Mario Bros runs and increasing both success rate and average score on VAB-Minecraft.The same state-conditioned skill framework benefits realistic GUI tasks and open visual game environments.
- Ablations: State cards and multi-view visual evidence both improve skill utility, while the complete MMSkills package outperforms text-only and image- or state-card-ablated variants.The content ablation isolates complementary contributions from runtime state descriptions and visual keyframes.
- Ablations: Branch loading and view selection address distinct failure modes: direct-full loading pollutes the main context, whereas the full two-stage design performs best.Branch loading separates skill inspection from the main context, and view selection filters state cards and keyframes before guidance is distilled.
- Skill usage and interaction dynamics: MMSkills increase invocation coverage and shorten trajectories; Qwen3-VL-235B’s OSWorld invocation rises from 37.50% to 65.28%, while average steps decrease in every setting.Focus crops dominate selected visual evidence, suggesting that multimodal skills improve relevance recognition while reducing unnecessary exploration.
- Behavioral shift analysis: MMSkills reduce low-level action load and repetitive behavior, with Qwen3-VL-235B click share falling from 75.8% to 63.7% and exact repeated actions from 21.8% to 6.2%.DONE actions increase for both analyzed models, indicating stronger completion awareness alongside fewer exploratory operations.
4 Related Work
Related work spans reusable agent skills, visual-agent benchmarks and control, and multimodal skill extraction. MMSkills differs from closest approaches by organizing skills around runtime state cards and multi-view evidence, then branch-loading selected evidence before action.
- Skills for agents: Agent skill reuse extends from temporal abstraction and motor primitives to language-, code-, API-, and learned-library representations.
- Visual agents: Visual-agent research covers web, mobile, desktop, and embodied environments while improving screenshot grounding and GUI control.
- Closest approaches: Mirage-1, XSkill, and CUA-Skill represent nearby efforts using hierarchical multimodal skills, visually grounded experience, or parameterized procedures and execution graphs.
- MMSkills distinction: MMSkills organizes skills around runtime state cards and multi-view evidence, using branch loading to align selected evidence with live observations before action.
5 Conclusion and Limitations · Appendix
MMSkills represent reusable visual-agent skills as multimodal procedural knowledge and improve GUI and game-based agents across model families. Their limitations include source-trajectory coverage, generation and grounding errors, and additional inference cost from branch loading.
- 5 Conclusion and Limitations: MMSkills represent reusable skills for visual agents as multimodal procedural knowledge.The framework combines textual procedures, runtime state cards, multi-view keyframes, and branch-loaded use.
- 5 Conclusion and Limitations: MMSkills combine textual procedures, runtime state cards, multi-view keyframes, and branch-loaded use.
- 5 Conclusion and Limitations: MMSkills improve GUI and game-based visual agents across model families.
- 5 Conclusion and Limitations: Their effectiveness depends on coverage of the source trajectories used to derive skills.
- 5 Conclusion and Limitations: Skill generation or visual grounding can introduce errors.
- 5 Conclusion and Limitations: Branch loading adds inference cost during skill use.
A Benchmark Statistics
The evaluation uses four visual-agent benchmarks, with OSWorld as the primary GUI benchmark and macOSWorld as an auxiliary cross-operating-system evaluation. OSWorld contains 360 test cases, while macOSWorld contains 143, distributed across application and system domains.
- Benchmark coverage: The study evaluates agents on four visual-agent benchmarks, using OSWorld as the primary GUI benchmark and macOSWorld for auxiliary cross-operating-system evaluation.OSWorld covers Ubuntu desktop tasks spanning browsers, office software, creative tools, media, settings, code editors, email, and multiapplication workflows; macOSWorld covers file management, media, productivity, system/interface, and system-application tasks.
- Test-case distributions: 360 OSWorld test cases and 143 macOSWorld test cases are distributed across benchmark-specific domains.“Share” denotes each domain’s percentage of test cases within its corresponding benchmark.
B Skill Source Statistics
MMSkills are derived exclusively from non-test interaction trajectories, using platform-specific public datasets and preprocessing to align sources with benchmark categories. Skill-source data remain disjoint from final evaluation cases.
- Source Data: All MMSkills are extracted from non-test trajectories.This establishes the separation between skill construction and evaluation data.
- GUI Benchmarks: OSWorld and macOSWorld use Ubuntu and macOS OpenCUA trajectories as GUI skill sources.Raw macOS trajectories undergo clustering and relevance filtering before assignment to the five macOSWorld categories.
- Game Benchmarks: VAB-Minecraft uses its official training set, while Super Mario Bros uses multiple runs over four LMGame-Bench source cases.In both settings, the skill-source data are disjoint from the final evaluation cases.
C Experiment Details
The experiments evaluate visual agents on GUI and game benchmarks using screenshot observations, separated skill-construction and evaluation data, and controlled interaction budgets. Across frontier and smaller models, comparisons cover no-skill, text-only skill, and branch-loaded MMSkills conditions.
- Evaluation setup: Agents plan from desktop or game screenshots rather than privileged state, with GUI evaluations run on official benchmark images through the harness and capped at 20 interaction steps.This setup applies to OSWorld and macOSWorld, which are run primarily on Amazon Web Services.
- Data separation: VAB-Minecraft skills are generated only from training trajectories, while evaluation uses the official test set without test episodes entering skill construction.The generated artifacts include reusable procedures, state cards, and keyframes.
- Data separation: Super Mario Bros uses disjoint source and evaluation subsets, including four held-out cases, to test reusable game situations rather than memorization of measured episodes.Source cases are described in Appendix B, and the held-out cases are reserved for final evaluation.
- Model and condition comparison: The study evaluates six frontier and smaller multimodal models under no-skill, text-only skill, and MMSkills conditions, with branch loading as the default skill mechanism.Text-only branches omit state cards and images, whereas MMSkills inspect selected state cards and multi-view keyframes before returning structured guidance; direct loading is reserved for ablations.
D Branch-Loaded Runtime Algorithm · E MMSkillAgent Prompt Templates
MMSkills uses a branch-loaded runtime loop in which selected skills are consulted only when needed, while the main agent receives compact structured guidance rather than full multimodal packages. Prompt templates enforce screenshot-grounded decisions, minimal visual evidence loading, planner-only branch reasoning, and explicit fallbacks and verification.
- D Branch-Loaded Runtime Algorithm: Candidate skills are selected before execution, but branch calls occur only when the main agent chooses to consult a specific skill.The main trajectory receives structured guidance Gt instead of the full multimodal skill package.
- E MMSkillAgent Prompt Templates: The main agent may act directly or consult a skill branch, which first selects relevant state cards and keyframe views before returning compact structured guidance.The branch is temporary, and the main agent—not the branch—chooses the concrete GUI action.
- E MMSkillAgent Prompt Templates: Skill consultation is conditional: LOAD_SKILL is called only when the screenshot, recent steps, and skill hints indicate that extra procedural guidance is useful.Each skill can be consulted at most {consult_limit} times per trajectory; exhausted skills are removed from the available-skill list.
- E MMSkillAgent Prompt Templates: Both the main agent and branch prioritize the live screenshot over skill references, treating state hints, keyframes, and planner notes as fallible references rather than coordinate templates.If the skill is ineffective, the branch should say so; if progress stalls, the planner provides a concrete alternate route.
- E MMSkillAgent Prompt Templates: Stage 1 maps evidence goals to views, using full_frame or focus_crop for localization, before for pre-change recognition, after for completion verification, and before/after for transitions.The prompt explicitly discourages defaulting to the full_frame+focus_crop pair when transition evidence is sufficient.
- E MMSkillAgent Prompt Templates: Stage 1 loads visual references only for state transitions, result verification, or complex UI-state recognition, while generic textual procedures default to no images.Requests are kept minimal at most {max_states} states and {max_views} total views.
- E MMSkillAgent Prompt Templates: Stage 2 returns planner-only JSON containing skill applicability, subgoal, plan, do_not_do, fallback_if_no_progress, expected_state, and completion_scope.Its plan covers the next 2–4 actions, checks, or transitions and specifies cues for advancing versus replanning.
F Additional Behavioral Shift Analysis · G Interaction Case Studies · H Broader Impact
The paper extends behavioral-shift analysis to additional visual agents, illustrates multimodal skill consultation in representative OSWorld workflows, and discusses both the benefits and risks of externalized procedural knowledge.
- F Additional Behavioral Shift Analysis: Additional OSWorld analysis examines behavioral shifts induced by MMSkills for GLM-5V and Kimi-K2.6.The analysis complements Figure 4.
- F Additional Behavioral Shift Analysis: The analysis tracks action primitive distributions, low-level primitives per task, and repetitive behavior statistics.These panels use the same metrics as Figure 4.
- G Interaction Case Studies: A LibreOffice Calc trace shows the agent consulting different spreadsheet skills at different stages of table construction.This case is presented as a representative OSWorld interaction trace.
- G Interaction Case Studies: A terminal file-organization trace shows branch guidance moving beyond an initially brittle command and verifying the final archive structure.The case demonstrates guidance during both recovery and final verification.
- H Broader Impact: MMSkills aim to improve visual-agent reliability by externalizing reusable multimodal procedural knowledge.The stated goal is reliable reuse of procedural knowledge across visual-agent interactions.
- H Broader Impact: Potential benefits include improved desktop automation, less repeated trial-and-error interaction, stronger support for smaller models, and reusable knowledge across GUI and game-like environments.These benefits are explicitly identified as potential impacts.
- H Broader Impact: More capable visual agents may increase risks involving unwanted automation, misuse in interactive software, or accidental actions in sensitive environments.The supplied passage introduces these risks while discussing broader impact.
I Use of LLMs
LLMs support both MMSkill construction and evaluation: they help generate reusable multimodal procedures and also serve as the visual agents tested in benchmark experiments.
- Research and evaluation roles: LLM-based agents process and filter trajectories, propose reusable procedures, draft state cards, and generate multimodal skill packages under human-designed schemas and quality checks.These uses occur within the skill-generation pipeline.
- Research and evaluation roles: LLMs serve as the evaluated visual agents in the benchmark results.The work therefore uses LLMs both to construct skills and to assess their utility.
- Research and evaluation roles: LLM tools also support manuscript preparation through editing and polish.This is an additional use beyond skill generation and benchmark evaluation.
J Detailed Related Work
Related work traces agent skills from temporal abstraction and language-conditioned procedures to explicit, multimodal skill substrates, while visual-agent research advances grounded interaction across web, GUI, mobile, desktop, and embodied settings. Dedicated benchmarks also expose limitations in skill utility, GUI grounding, and long-context evidence use.
- Skills for agents: Agent skills evolved from reinforcement-learning temporal abstractions and robotic motor primitives into language-grounded, executable, and composable procedural interfaces.Recent systems ground language in affordances, emit executable programs, or interleave reasoning with actions.
- Skills for agents: Recent work treats skills as explicit improvement substrates, including reusable API-like skills, parameterized execution and composition graphs, hierarchical skill bases, and failure-driven discovery.Examples include SkillWeaver, CUA-Skill, SkillX, and EvoSkill.
- Skills for agents: SkillsBench, SkillTester, and related studies show that skill utility depends on relevance, compactness, selection, retrieval, and adaptation rather than being automatically beneficial.These works evaluate performance effects, utility, security risks, and more realistic skill-use conditions.
- Skills for agents: Multimodal and GUI-specific efforts include hierarchical search-supported skills, visually grounded rollout extraction, stateful experiences, and parameterized computer-use skills.Mirage-1, XSkill, MuSEAgent, and CUA-Skill represent the closest related line to MMSkills.
- Visual agents: Visual-agent benchmarks span realistic web navigation, mobile and desktop control, operating-system environments, embodied settings, and multimodal computer-use risks.Mind2Web, WebArena, VisualWebArena, WebVoyager, Android in the Wild, AndroidWorld, OSWorld, macOSWorld, and RiOSWorld exemplify this expansion.
- GUI grounding benchmarks: ScreenSpot-Pro evaluates GUI localization in high-resolution professional desktops, where targets can occupy less than 0.1% of the screen and leading models remain below human performance.The benchmark extends earlier ScreenSpot evaluations and highlights the difficulty of precise visual grounding.