Source-linked AI summary
Agent Skills Should Go Beyond Text: The Case for Visual Skills
Binxiao Xu, Ruichuan An, Bocheng Zou, Hang Hua
TL;DR
Visual-agent skills are often stored as text even when success depends on spatial structure, visual grounding, and localized state. The paper proposes VISUAL SKILL and AUTOVISUALSKILL to combine textual logic with reusable visual support, and reports consistent improvements over text-only skills on visual-centric benchmarks. It also identifies settings where visual skills may be unnecessary or distracting.
Problem
Text-only skills may not preserve the spatial structure, visual boundaries, and perceptual tracking protocols required by visual-centric tasks.
Method
VISUAL SKILL combines declarative textual logic, reusable visual priors or references, and a multimodal binding protocol for joint execution.
Results
Visual skills consistently improve over text-only skills across evaluated benchmarks and metrics, including GUI grounding and dense counting.
Takeaways & Limitations
Reusable visual structure can serve as a first-class skill asset when task success depends on spatial or perceptual information.
Takeaways & Limitations
Visual skills may be unnecessary or distracting when knowledge is already well represented in language or when rigid spatial schemas constrain open-ended perception.
Abstract
from arXiv · showhide
Reusable skills are a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck for visual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose \textbf{\NAME}, a multimodal skill paradigm that combines declarative textual logic with explicit visual support. We distinguish three reusable forms: static priors for stable spatial conventions, dynamic priors for in-situ visual working memory, and interleaved visual skills that bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do, visual skills also encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce \textbf{\SYSTEM}, an automatic system that converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns from task trajectories. Experiments on GUI and other visual-centric tasks show that visual skills consistently outperform text-only skills, particularly when success requires spatial correspondence, visual evidence, and state-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.
1 Introduction
Existing reusable skills mainly store experience as text, but visual-centric tasks also require spatial grounding, visual evidence, and state-aware interaction. VISUAL SKILL addresses this bottleneck with reusable multimodal components, and experiments report consistent gains over text-only skills.
- Motivation: Text-only skills leave spatial conventions and visual grounding implicit, limiting reliable reuse for visual-centric tasks.Visual tasks require knowledge of where and how to act, inspect, and verify outcomes.
- Motivation: Visual-agent benchmarks show that knowing what action to take does not ensure sufficiently precise knowledge of where to act.Web and mobile agents may need to rediscover actionable regions from each screenshot.
- VISUAL SKILL: Static, dynamic, and interleaved visual skills package reusable spatial conventions, in-situ state traces, and ordered reasoning with visual evidence.These forms target different visual bottlenecks, including spatial conventions, intermediate tracking, and source-grounded steps.
- VISUAL SKILL: VISUAL SKILL combines declarative textual logic, visual priors or references, and a multimodal binding protocol.The representation preserves visual procedural knowledge alongside textual control.
- Validation: Controlled experiments on GUI grounding and dense object counting show static references and dynamic traces consistently improve over text-only skills.The paper presents visual structure as a reusable asset for multimodal agents.
2 The Textual Bottleneck in Current Skill Paradigms
Text-only skill representations compress spatially organized interaction knowledge into language, creating ambiguity in static visual protocols and instability in dynamic perceptual bookkeeping. These failures motivate distinct forms of reusable visual support.
- Representational gap: Visual interfaces organize task-relevant knowledge through precise regions, proportions, alignment, occlusion, and connectivity that text-only skills may not preserve.Compressing these signals into a symbolic sequence can lose spatial evidence that verbal detail alone cannot recover.
- Failure modes: Static protocol ambiguity arises when visual boundaries, displacement, hit-region tolerance, whitespace, hierarchy, and proportions become underspecified in text.The resulting procedures may state what to do without preserving the visual conventions needed for reliable execution.
- Failure modes: Dynamic tracking collapse occurs when counting, maze solving, or verification requires persistent records of inspected, counted, and next regions.Coordinate lists and verbal descriptions become ambiguous as visual density increases, leading to omissions, repeated inspections, and double-counting.
- Design implication: Stable spatial conventions call for visual priors, whereas continuous perceptual bookkeeping calls for in-situ visual traces.The two failure modes share a common representational root but require different visual support.
3 VISUAL SKILL: Visual Structure as a First-Class Skill Asset
VISUAL SKILL extends text-based procedures into multimodal assets by pairing declarative logic with reusable visual support and role-specific binding. Its three support forms preserve stable structure, dynamic state, or ordered source evidence.
- Definition: VISUAL SKILL represents a reusable skill through declarative textual logic, visual priors or references, and a binding protocol for joint execution.The binding protocol governs how the modalities are grounded, retrieved, and executed together.
- Components: Declarative textual logic specifies objectives, procedures, constraints, boundary conditions, and failure-handling strategies.It retains abstraction, compositionality, interpretability, and procedural control.
- Visual support: Static priors provide reusable references for stable layout, boundary, alignment, and interaction conventions across task instances.Examples include wireframes, prototypes, annotation templates, and error-pattern examples.
- Visual support: Dynamic priors maintain anchors, trajectories, visited regions, or counting marks as continuously updated visual working memory during inference.They specify how intermediate traces are initialized, updated, and verified.
- Binding: A multimodal binding protocol selects static, dynamic, or interleaved support for each reasoning step and guides execution on the task input.Dynamic priors are updated after execution when the protocol requires maintained visual state.
- Modality division: The framework assigns text the role of specifying what to do and vision the role of preserving where and how to inspect.Examples include hit regions, layout proportions, scan order, and spatial violation patterns.
4 Paradigm Instantiation: The AUTOVISUALSKILL Framework
AUTOVISUALSKILL is a proof-of-concept pipeline that converts user goals and multimodal context into reusable VISUAL SKILL artifacts. It diagnoses visual bottlenecks, generates textual and visual components along dual tracks, and packages their execution bindings.
- Overview: AUTOVISUALSKILL synthesizes reusable visual-skill artifacts from user goals and multimodal context.Each artifact includes textual logic, visual assets, manifests, and provenance records for loading, inspection, or versioning.
- Input and normalization: The pipeline normalizes text, images, URLs, and video frames while extracting constraints and candidate reusable visual protocols.It can retrieve supplementary domain knowledge when needed.
- Visual-bottleneck gate: A visual-bottleneck gate determines whether a task needs visual support for grounding, geometry, perceptual tracking, or reusable source evidence.It distinguishes cross-instance protocols from few-shot image caches.
- Dual-track generation: Dual-track generation writes declarative logic linguistically and extracts, retrieves, renders, or generates visual support in parallel.The components are packaged with binding manifests into reusable VISUAL SKILL artifacts.
- Execution-oriented artifacts: Execution manifests record skill and prior kinds, asset roles, renderer strategy, binding rules, and usage constraints.Downstream agents can load fixed priors, maintain visual-state loops, or present source evidence beside reasoning steps.
- Open-source interface: The released system includes a command-line interface, Gradio demo, example skills, and minimal agent-integration code.Additional examples cover all three skill forms.
5 Empirical Study
The empirical study isolates static and dynamic visual mechanisms using controlled no-skill, text-only, and visual-skill settings. Across GUI grounding and dense counting, visual priors consistently improve performance over text-only skills by preserving spatial conventions or visual state.
- Experimental design: The study compares direct prompting, frozen text-only skills, and visual skills that add task-appropriate visual priors while keeping textual rules fixed.The controlled design attributes performance differences to the added visual protocol rather than different textual instructions.
- Experimental design: Visual skills yield significant and consistent improvements across all evaluated benchmarks and associated metrics, despite differing foundation-model specializations.The study uses GUI grounding with Qwen3-VL-32B-Thinking and dense object counting with Gemini-2.5-Pro.
- Static Priors: Static visual priors improve GUI grounding across three benchmarks by supplying reusable spatial conventions for target extent, hit regions, and fine-grained boundaries.The largest gains appear on Mean IoU, while text-only gains remain limited for spatially precise localization.
- Static Priors: +0.028 Point-in-Box accuracy, +0.054 Mean IoU, and a 0.0040 Mean Center Distance reduction produce normalized degradation rates of 17.1%, 8.1%, and 9.2%, respectively.These values quantify the GUI grounding gap between text-only and visual skills across the three benchmarks.
- Dynamic Priors: Dynamic visual priors improve CountBenchQA accuracy and substantially reduce MAE, whereas text-only skills do not improve over direct prompting and can reduce performance.The visual traces maintain a more consistent counting state through explicit spatial anchoring.
- Dynamic Priors: +4.12 exact-accuracy points, +2.67 Within-1 accuracy points, and a 0.1077 MAE reduction correspond to normalized degradation rates of 58.9%, 72.2%, and 66.8%.The counting results quantify the loss associated with text-only skillization relative to visual skills.
6 Discussion
VISUAL SKILL is intended as a persistent, protocol-level multimodal artifact rather than an instance-level prompt cache. Its use is most justified when reusable knowledge contains spatial or perceptual structure that text alone cannot faithfully preserve.
- Persistent multimodal skills: VISUAL SKILL encodes reusable conventions such as target granularity, spatial boundaries, layout prototypes, and inspection procedures without storing instance-specific answers.Unlike few-shot examples, visual priors are protocol-level artifacts that can be retrieved, versioned, composed, and audited across task instances.
- Why visual support matters: Visual skills preserve spatial knowledge that prose can express only ambiguously when tasks depend on dense geometry, visual boundaries, or continuous perceptual tracking.The paper presents this as a modality-matching requirement rather than merely a remedy for weaker models.
- Future forms: Future visual skills may use videos, interactive annotations, executable visual programs, or learned visual memory modules while retaining visual preservation as the core principle.The representation may change with stronger models, but inherently visual task knowledge should remain multimodal.
- Design principles: Three design principles guide effective visual priors: they should be abstract, genuinely visual, and complementary to text.They should encode shared shapes, positions, boundaries, layouts, or spatial procedures, while leaving language-clear information in text.
- Boundaries: Visual skills are most useful when the reusable task bottleneck is spatial or perceptual, but may be unnecessary or distracting when knowledge is already linguistic.The paper specifically identifies symbolic tasks and some unstructured open-ended perception settings as boundaries for use.
7 Conclusion
The paper identifies a mismatch between text-dominated skill storage and the visual structure of many agent tasks, then proposes VISUAL SKILL and AUTOVISUALSKILL to address it. Experiments report consistent advantages over text-only skills when tasks require spatial conventions, localized evidence, or grounded intermediate tracking.
- Conclusion: The paper identifies a textual bottleneck because reusable agent experience is stored mainly as text despite many tasks having inherently visual and spatial structure.This mismatch motivates multimodal skill artifacts rather than text-only reusable experience.
- Conclusion: VISUAL SKILL combines declarative textual logic, visual priors or references, and a multimodal binding protocol.The paper also defines interleaved visual skills for keeping ordered reasoning adjacent to the visual evidence grounding each step.
- Conclusion: AUTOVISUALSKILL automatically diagnoses visual bottlenecks, generates textual and visual skill components, and packages them into reusable artifacts.It is presented as a proof-of-concept authoring pipeline.
- Conclusion: Experiments on GUI grounding and dense counting consistently show that visual skills outperform text-only skills when success requires spatial conventions, localized visual evidence, and grounded intermediate tracking.These findings support extending reusable agent skills beyond text.
A.1 Complete Prompt Templates
The appendix specifies controlled prompt templates for comparing direct prompting, text-only skills, and visual skills across GUI grounding and counting tasks. The templates hold the task image, question, model, decoding parameters, and parser fixed while changing the reusable-skill intervention.
- Experimental controls: All experimental settings share the same task image, question, model, decoding parameters, and output parser; only the reusable-skill intervention changes.The compared interventions are no skill, text-only reusable logic, and the full visual-skill artifact.
- GUI Grounding: GUI grounding templates request a click point and bounding box in a 0–1000 coordinate system relative to the task screenshot.Visual-skill prompts additionally provide a reusable visual artifact and visual prior alongside the screenshot.
- GUI Grounding: The GUI comparison includes direct prompting, textual-skill prompting, and visual-skill prompting with declarative logic or a visual artifact.The task instruction and screenshot remain part of the grounding prompt in each setting.
- Counting: Counting templates request a JSON total count, while the visual-skill condition additionally returns point anchors and receives dynamic-prior feedback.The dynamic-prior prompt includes the current image with any rendered anchors.
- Counting: Dynamic counting feedback renders predicted points as numbered anchors on the task image and reuses the newly marked image as external visual working memory.The loop stops when no new points are returned or a conservative maximum-round limit is reached.
A.2 Metric Definitions
The appendix defines GUI grounding and counting metrics, including box and click representations, directional metric preferences, and an anchor-consistency diagnostic for visual-skill counting.
- GUI grounding: GUI grounding represents the ground-truth box as B⋆, the predicted box as ˆB, and the predicted click point as ˆp = (ˆx, ˆy).These quantities support the grounding metrics defined in the appendix.
- GUI grounding: Center Distance compares box centers using image dimensions W and H, while higher Point-in-Box and IoU are better and lower Center Distance is better.The center-distance normalization uses the image size.
- Counting: Counting metrics compare predicted counts ˆy_i with ground-truth counts y_i using ExactAcc, MAE, and Within1.The appendix defines these metrics over N examples.
- Counting: Visual-skill counting additionally checks whether the reported count agrees with the number of returned point anchors.This internal-consistency diagnostic does not replace count accuracy.
A.3 Failure Mode Analysis: The Limits of Visual Protocols
Visual protocols can fail when rigid spatial conventions conflict with the task’s intended semantic granularity. The section also clarifies how static, dynamic, and interleaved skills should be selected according to the underlying bottleneck.
- Failure modes: Visual protocols can introduce failures when structural spatial priors conflict with fine-grained semantic intent.The paper identifies this tension as a source of new errors despite the benefits of Visual Skills.
- GUI grounding: Static GUI priors may over-specialize target granularity by selecting a minimal glyph instead of the full functional container.This can cause the model to focus on a small icon even when the instruction refers to a broader control.
- Dense counting: Dynamic counting priors can expose ambiguity over whether composite objects, subparts, background elements, or low-contrast instances count as discrete objects.The ambiguity arises because dynamic priors require the model to commit to an object granularity during enumeration.
- Visual-skill forms: Static visual skills package reusable spatial conventions, while dynamic skills externalize intermediate state through in-situ visual anchors.The GUI example calibrates implicit hitboxes; the counting example maintains visual working memory across iterative enumeration.
- Decision boundaries: Skill modality should be selected by the task’s cognitive bottleneck: spatial convention, perceptual tracking, ordered evidence binding, or purely textual logic.Static priors address reusable spatial conventions, dynamic priors address repeated-step tracking, and interleaved skills bind reasoning steps to source visuals.
- Interleaved skills: Interleaved visual skills preserve ordered reasoning steps alongside the source frame, screenshot, page, or crop that grounds each claim.They are intended for tutorials, documentation workflows, walkthroughs, and evidence-bound explanations when a single prior or text procedure is insufficient.