PaperlayerView exact revision on arXiv

Source-linked AI summary

ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2610.12403v1cs.CVcs.CL

TL;DR

Text-centric skills can lose geometric structure, and recent visual-skill approaches typically separate skill updates from policy optimization. ViSkill uses visual skill cards in a feedback loop that guides policy learning and accumulates successful trajectories. Across its evaluated environments, it outperforms proprietary and open-source baselines, with an overall success rate of 0.89 and 0.91 with cold-start initialization.

  • Problem

    Text-centric skills can lose geometric information by linearizing spatial layouts and action–state correspondences, while visual-skill construction and policy optimization remain largely decoupled.

    Source: §1 · p. 2

  • Method

    ViSkill represents successful experience as visual skill cards, retrieves them to guide interaction and reward shaping, and distills successful trajectories back into the library.

    Source: §1 · p. 2

  • Results

    0.89 overall success rate, rising to 0.91 with cold-start initialization, while ViSkill outperforms evaluated proprietary and open-source baselines across Sokoban, FrozenLake, and PrimitiveSkill.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the evaluated settings, ablations confirm that visual representations, strategic guidance, and joint optimization each contribute to the positive feedback loop.

    Source: §5 · p. 9

    ViSkill is evaluated only in VAGEN-supported environments, and its benefits in more complex or open-ended settings remain unvalidated.

    Source: §5 · p. 9

Abstract

from arXiv · show

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.

1 INTRODUCTION

ViSkill addresses the geometric losses of text-centric skills and the separation of skill construction from policy optimization with visual skill cards and a closed learning loop. It outperforms evaluated baselines across its test environments and converges faster than standard PPO.

1 INTRODUCTION

  • Existing skill approaches often linearize spatial layouts and action–state correspondences into language, risking loss of geometric information, while visual-skill learning remains decoupled from policy optimization.Successful trajectories could instead preserve reusable procedural knowledge for future episodes.Source: §1 · pp. 1–2
  • 0.89 overall success rate, rising to 0.91 with cold-start initialization, as ViSkill outperforms evaluated proprietary and open-source baselines.The evaluation covers Sokoban, FrozenLake, and PrimitiveSkill; ViSkill also converges faster and achieves higher success rates throughout training than standard PPO.Source: §1 · p. 2
  • ViSkill stores successful experience as composite visual skill cards that preserve spatial and procedural context in a form accessible to VLM agents.The cards combine annotated trajectory frames with a policy-distilled strategy description in a unified image.Source: §1 · p. 2
  • Geometry-aware retrieval and skill-guided rewards guide policy optimization, while successful trajectories are distilled back into the library to close the feedback loop.The optional cold-start mechanism pre-populates the library with diverse solver-derived seed skills.Source: §1 · p. 2

2 RELATED WORK

Prior VLM reinforcement learning often treats interaction experience as transient rollout data, while skill-augmented agents make procedural knowledge reusable but commonly represent it textually. ViSkill combines retrievable visual-native skills with policy optimization.

2.1 REINFORCEMENT LEARNING FOR VLM AGENTS

  • ViSkill externalizes successful trajectories into a retrievable visual-native skill library, allowing accumulated experience to guide inference and provide dense rewards for policy optimization.Prior VLM reinforcement learning typically consumes interaction experience as transient rollout data absorbed implicitly into policy parameters.Source: §2.1 · pp. 2–3

2.2 SKILL-AUGMENTED AGENTS

  • Most skill-augmented agents reuse language instructions or structured workflows, while ViSkill uses composite visual skill cards and couples online distillation with policy optimization.Recent approaches incorporate visual information into skill construction and memory; AtlasVA evolves spatial atlases from interaction statistics for dense reward shaping.Source: §2.2 · p. 3

3 METHODOLOGY

ViSkill integrates visual-skill retrieval, interaction, and online distillation, preserving successful trajectories as composite cards. Retrieved skills guide policy learning, while successful experience can expand the library and the policy is optimized with PPO.

3.1 SKILL-INTEGRATED AGENT LOOP

  • Each interaction conditions the policy on the task, history, and retrieved skill; episode success updates skill utility and usage and determines eligibility for distillation.The episode outcome also supplies the policy-optimization reward.Source: §3.1 · p. 3

3.2 VISUAL-NATIVE SKILL LIBRARY

  • ViSkill represents successful experience with annotated trajectory observations and a policy-distilled strategy composed into a visual skill card for VLM context.Each card also stores a geometric descriptor, utility, and usage count.Source: §3.2 · pp. 3–4
  • At each episode’s start, ViSkill scores skills using geometric similarity and historical utility, retrieves the top skill only above a threshold, and appends its card to visual context.Geometric descriptors capture relations such as positions, distances, and obstacles.Source: §3.2 · p. 4
  • Successful trajectories are considered for library distillation only after quality and novelty gates, and the lowest utility–frequency skill is evicted when the library reaches capacity.The quality gate uses a task-specific trajectory-length budget; the novelty gate checks whether an existing skill sufficiently covers the configuration.Source: §3.2 · p. 4

3.3 SKILL-GUIDED REINFORCEMENT OPTIMIZATION

  • Optional cold-start initialization seeds the library with solver-generated trajectories selected for geometric coverage, while subsequent skills are distilled from the policy’s own successful trajectories.The cold-start ratio can be set to zero to disable external-model initialization.Source: §3.3 · p. 4
  • For failed episodes with a retrieved skill, the final reward combines the environment reward with a skill-guidance term; PPO updates the policy as the library grows through distillation.The training procedure includes skill retrieval, skill-guided reward, utility updates, gated distillation, and policy optimization.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments span grid-based 2D planning and coordinate-grounded 3D manipulation, comparing ViSkill with proprietary and open-source VLM baselines. ViSkill leads in overall success, converges faster than PPO, and shows gains from visual skill representations and jointly evolving skills and policy.

4.2 MAIN RESULTS

  • 0.89 overall success rate, ViSkill outperforms all evaluated proprietary and open-source baselines; it scores 0.82 on Sokoban and 0.84 on FrozenLake.On PrimitiveSkill, ViSkill reaches 1.00, matching AtlasVA and outperforming VAGEN by 12 percentage points.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 TRAINING DYNAMICS

  • ViSkill converges faster and reaches higher success rates than PPO across environments; cold-start initialization adds gains in early learning.On PrimitiveSkill, both ViSkill variants reach 1.00 faster than PPO, with cold-start giving an early advantage.Source: Figure 3 · p. 7, §4.3 · p. 7

4.4 ABLATION STUDY

  • Text-Skill and OCR-Skill both underperform ViSkill on Sokoban and FrozenLake, where original visual trajectories retain spatial and structural context.PrimitiveSkill is saturated by the full model, so representation ablations focus on the two 2D environments.Source: §4.4 · p. 7, Table 2 · p. 7
  • Removing Optimization causes the largest drops (−0.53 on Sokoban, −0.51 on FrozenLake); removing Strategy drops Sokoban by −0.10, while removing Skill-Guided Reward drops Sokoban by −0.05 and FrozenLake by −0.04.The results highlight co-evolution: policy training helps the model use retrieved skills, while an improving policy contributes higher-quality skills.Source: §4.4 · pp. 7–8, Table 3 · p. 7
  • Performance peaks at λ = 0.8 on both environments and at δr = 0.80, yielding 0.82 on Sokoban and 0.84 on FrozenLake.A lower retrieval threshold admits poorly matched skills, while a higher one restricts successful retrieval; λ = 1.0 over-emphasizes progress toward retrieved states.Source: §4.4 · p. 8, Figure 4 · p. 8, Table 4 · p. 8
  • δd = 0.90 achieves the best success rate on both environments, balancing library size and utility between redundant admissions at 0.85 and restricted growth at 0.95.At 0.85, redundant skills drive average utility down; at 0.95, strict diversity limits library growth and scenario coverage.Source: §4.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.5 FURTHER ANALYSIS

  • Increasing the cold-start ratio improves performance on both environments; Sokoban saturates at γ = 0.6 with 1.00 success, while FrozenLake reaches 0.89 at γ = 1.0.The cold-start ratio scales the initial library relative to the converged library size from a run without cold-start initialization.Source: §4.5 · p. 8, Figure 6 · p. 9
  • Reduced-resolution visual skills retain an advantage over Text-Skill and OCR-Skill at lower token use, while jointly optimized libraries transfer better to the unoptimized base model.At scale 0.2, visual skills outperform both baselines on Sokoban and FrozenLake; jointly optimized libraries reach 0.41 on Sokoban and 0.39 on FrozenLake.Source: §4.5 · p. 9, Figure 7 · p. 9

5 CONCLUSION

ViSkill couples visual skill cards with reinforcement optimization in a feedback loop, and experiments report gains over proprietary and open-source baselines. The authors note that its evaluation scope and skill-guided reward leave important settings unvalidated or less well guided.

5 CONCLUSION

  • ViSkill combines composite visual skill cards with skill retrieval and reinforcement optimization in a closed feedback loop.The conclusion says the approach preserves spatial and geometric structure, while strategic guidance and joint optimization contribute to the loop.Source: §5 · p. 9
  • The authors report that benefits in more complex or open-ended settings remain unvalidated because evaluation is limited to environments supported by VAGEN.The skill-guided reward may also offer limited guidance when valid solutions traverse state paths disjoint from those represented by retrieved skills.Source: §5 · p. 9
  • The authors report using generative AI for implementation, figure plotting, and manuscript drafting and editing, and say they reviewed and verified all AI-assisted content.The authors take full responsibility for the final work.Source: AI USE STATEMENT · p. 10
  • Real-world deployment would require task-specific safety constraints and human oversight; trajectory-quality and novelty gates do not replace safety validation in open-ended applications.The work was conducted exclusively in simulation, without human participants, personal data, or physical systems.Source: ETHICS STATEMENT · p. 10

A METHODOLOGY DETAILS

ViSkill retrieves skills using geometric configuration and historical utility, then renders successful experience as visual cards for inference and guidance. Trajectory-quality and novelty gates, library management, and PPO optimization govern how skills are admitted, used, and learned from.

A METHODOLOGY DETAILS

  • Geometric descriptors retrieve skills by spatial configuration rather than language-level task descriptions, since identical goals can require different strategies under different layouts.For Sokoban and FrozenLake, similarity uses weighted exact-match rates over descriptor blocks; PrimitiveSkill uses continuous similarity over corresponding object coordinates and task identity.Source: A METHODOLOGY DETAILS · p. 14
  • On three held-out Sokoban configurations, the top-ranked retrieved skill has the closest spatial arrangement, while lower-ranked candidates share progressively fewer task-relevant geometric relations.Figure 10 presents representative visual skill cards from Sokoban, FrozenLake, and PrimitiveSkill.Source: A METHODOLOGY DETAILS · p. 14, Figure 10 · p. 15
  • The trajectory-length quality gate admits successful trajectories only when they meet a task-dependent budget, instantiated as the shortest available solution length in these experiments.Breadth-first search supplies budgets for Sokoban and FrozenLake; scripted oracle plans supply fixed reference lengths for PrimitiveSkill, and solvers are not provided to the policy or used for retrieval.Source: A METHODOLOGY DETAILS · p. 15
  • Skill cards combine strategy descriptions with annotated trajectory frames; grid cards show temporal order, action labels, and path overlays, while PrimitiveSkill cards preserve third-person frames.The library stores each card with its geometric descriptor, reference trajectory, strategy, utility, and usage count; only retrieved skills have their utility and usage count updated after an episode.Source: A METHODOLOGY DETAILS · p. 15, Figure 10 · p. 15
  • Task-specific formatting and completion rewards are accumulated before adding a skill-guidance term based on progress along the retrieved reference trajectory.Progress uses relative position in Sokoban and FrozenLake and matched actions in PrimitiveSkill; the furthest matching point determines progress.Source: A METHODOLOGY DETAILS · pp. 15–16
  • PPO excludes observation tokens, including environment images and retrieved skill cards, from actor and critic losses, and optimizes policy-generated response tokens with a clipped objective.The method uses generalized advantage estimation with discount and smoothing parameters set to 1.0 and a clipping parameter of 0.2; it uses one PPO epoch per rollout batch with no entropy or KL auxiliary loss.Source: A METHODOLOGY DETAILS · p. 16

B.1 ENVIRONMENT AND TASK CONFIGURATIONS

The evaluation uses three visually grounded environments spanning 2D spatial reasoning and 3D manipulation, with RGB observations and textual instructions but no symbolic state exposed to the policy. Success is determined by task-specific predicates, and evaluation configurations are separated from training by seed.

B.1 ENVIRONMENT AND TASK CONFIGURATIONS

  • Evaluation spans Sokoban and FrozenLake grid tasks and five PrimitiveSkill manipulation tasks, with the policy receiving RGB observations and textual instructions but no symbolic state.Episodes are deterministic from their random seeds, and evaluation uses disjoint seed ranges so layouts and object configurations are unseen during training.Source: §B.1 · p. 16
  • Sokoban episodes last at most five turns and succeed when the box reaches its target; the policy may output up to three directional actions per turn.The agent must push all boxes onto target locations.Source: §B.1 · p. 16
  • FrozenLake uses randomly generated maps with frozen-cell probability 0.8, deterministic transitions, and slippery dynamics disabled.The policy may output up to five directional actions per turn over at most five turns; episodes end at the goal or a hole.Source: §B.1 · p. 16
  • PrimitiveSkill uses third-person RGB observations across Place, Stack, Drawer, Align, and Swap, with a Franka Panda arm controlled by parameterized actions using continuous xyz coordinates.The policy may issue up to two actions per turn, with a maximum of three turns for four tasks and four turns for Swap.Source: §B.1 · p. 16

B.2 TRAINING HYPERPARAMETERS

The appendix describes shared and environment-specific training configurations and provides prompt templates for strategy distillation and environment interaction. The prompts insert retrieved skill cards alongside observations, while reported figures summarize learning and library dynamics.

B.2 TRAINING HYPERPARAMETERS

  • The implementation uses a common optimization and rollout configuration across the three environments, adapting training horizon, computational resources, and sequence-length limits by task.Tables 5 and 6 summarize shared and environment-specific hyperparameters, respectively.Source: §B.2 · p. 16, Table 5 · p. 17, Table 6 · p. 17

C PROMPT TEMPLATES

  • The appendix provides full prompt templates for environment interaction and strategy distillation, with braced fields denoting runtime-supplied values.The templates are presented as part of the reproducibility materials.Source: C PROMPT TEMPLATES · p. 17

C.1 STRATEGY DISTILLATION PROMPT

  • The strategy-distillation prompt conditions a reusable strategy description on the task, initial observation, successful trajectory, and environment-specific examples.It asks for when to apply the strategy and what actions or approach to use; online self-distillation uses a forced strategy prefix and a 256-token cap, while cold-start distillation allows up to 512 tokens.Source: §C.1 · pp. 17–18

C.2 ENVIRONMENT INTERACTION PROMPTS

  • Across five PrimitiveSkill manipulation tasks, both ViSkill variants reach full success faster than PPO, and cold-start initialization accelerates early learning; retrieved composite skill cards are inserted between the system prompt and initial observation.Source: §C.2 · p. 19, Figure 11 · p. 21

D.1 ADDITIONAL TRAINING DYNAMICS

ViSkill reaches full success earlier than PPO across PrimitiveSkill tasks, while additional analyses show retrieval increasingly reuses covered skills and visual cards guide behavior in matched cases.

D.2 CASE STUDY

  • On matched Sokoban and FrozenLake configurations, retrieved visual cards guide ViSkill to successful behavior while PPO reaches failure states.Figure 13 presents the qualitative comparison.Source: Figure 13 · p. 22, §D.2 · p. 22

D.1 ADDITIONAL TRAINING DYNAMICS

  • ViSkill reaches full success earlier than PPO on all five PrimitiveSkill tasks, and both ViSkill variants achieve a success rate of 1.00.Cold-Start further accelerates early-stage learning across tasks.Source: §D.1 · p. 22
  • As the skill library gains coverage, retrieval fraction stabilizes near one while skill admissions taper after the early stage.The authors attribute this pattern to a novelty gate that limits redundant expansion.Source: §D.1 · p. 22

D.2 CASE STUDY

  • In FrozenLake, ViSkill takes a different valid route from the retrieved trajectory, using the card as procedural guidance rather than replaying its actions.The executed route remains consistent with the card’s spatial structure and high-level intent.Source: §D.2 · p. 22

D.3 QUALITY THRESHOLD ABLATION

A fixed trajectory-length budget provides an alternative to solver-derived thresholds, with task-dependent success-rate differences between the two approaches.

D.3 QUALITY THRESHOLD ABLATION

  • The ablation tests solver-derived quality thresholds by replacing them with a fixed trajectory-length budget.The comparison evaluates whether ViSkill depends critically on solver-derived thresholds.Source: §D.3 · p. 22
  • Static-10 achieves success rates of 0.84 on Sokoban and 0.78 on FrozenLake, versus 0.82 and 0.84 for solver-based filtering, respectively.Table 7 reports success rates under the different quality thresholds.Source: Table 7 · p. 22, §D.3 · p. 22
  • A fixed trajectory-length budget offers effective quality control without requiring access to task-specific solvers.It slightly outperforms solver-based filtering on Sokoban while incurring a modest decrease on FrozenLake.Source: §D.3 · pp. 22–23

D.4 FEATURE EXTRACTOR ABLATION

Replacing hand-crafted retrieval descriptors with DINOv2 features yields success rates close to the rule-based extractor on Sokoban and FrozenLake.

D.4 FEATURE EXTRACTOR ABLATION

  • The comparison replaces hand-crafted geometric descriptors with DINOv2 visual features for skill retrieval.The features receive no task-specific supervision.Source: §D.4 · p. 23
  • The narrow performance gap indicates that learned encoders are a practically viable alternative where geometric features are difficult to specify manually.The authors report that the spatial structure needed for reliable retrieval is largely recoverable from general-purpose visual representations.Source: §D.4 · p. 23
  • DINOv2 (Oquab et al., 2023) achieves success rates of 0.80 on Sokoban and 0.83 on FrozenLake, only 0.02 and 0.01 below the rule-based extractor, respectively.Table 8 compares success rates across feature extractors.Source: Table 8 · p. 23, §D.4 · p. 23
Loading 2610.12403v1…