PaperlayerView exact revision on arXiv

Source-linked AI summary

Video2Skill: From Streaming Experience to Reusable Embodied Skills

Jianshu Zhang, Ce Zhang, Xiyuan Yang, Chenwei Xu, Haoran Lu, Yijiang Li, Yaqi Xie, Katia P. Sycara, Han Liu

arXiv:2609.36691v1cs.CL

TL;DR

Models need reusable skill libraries to plan across diverse manipulation tasks, but it remains an open question whether they can build and maintain one from streaming video. Video2Skill benchmarks this ability across robot and kitchen videos. Many VLMs group events near chance; training improves grouping but does not reliably prompt new skills for unseen transformations.

  • Problem

    Vision-Language Models describe manipulation events, but it is unclear whether they can organize streaming video into a persistent library that distinguishes known skills from new ones.

    Source: §1 · p. 2

  • Method

    Video2Skill formalizes streaming embodied skill discovery and benchmarks event recovery, naming-invariant grouping, and creation–reuse decisions across robot and kitchen videos.

    Source: §1 · p. 2

  • Results

    Across 19 open-source VLMs, many group events near chance; supervision improves grouping, but transformations unseen in task-specific training are often absorbed into existing schemas.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings identify recognizing when existing skills are insufficient as a central challenge in learning from streaming visual experience.

    Source: §1 · p. 3, §7 · p. 9

    Video2Skill focuses on robot manipulation and egocentric kitchen activity; downstream planning utility and executable control require further evaluation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks. Yet an agent can only plan with skills it knows. Recovering skills from observed experience, the inverse of planning, builds this knowledge over time and yields skill data for training future agents. Vision-Language Models (VLMs) describe individual manipulation events well, but can they organize a stream of events into reusable skills? We formulate this problem as Streaming Embodied Skill Discovery (SESD): a model watches videos in sequence and maintains a persistent skill library that shapes its later decisions. To systematically measure this ability, we introduce Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: (i) locating manipulation events in time, (ii) grouping events of the same transformation, and (iii) deciding when to reuse an existing skill or create a new one. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled: joint models merge distinct transformations into one skill, while models that update the library from text descriptions duplicate recurring ones. Supervised fine-tuning, including our counterfactual library-state rebalancing (CLaRe), improves grouping but exposes a deeper bottleneck: trained models consolidate familiar skills yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill. Recognizing when existing skills are insufficient thus emerges as the central challenge.

1 INTRODUCTION

Embodied agents need reusable skill libraries to plan across diverse tasks, but building such libraries from streaming video requires consistent event recognition, grouping, and creation–reuse decisions. Video2Skill evaluates these abilities and finds that current VLMs struggle to group transformations and recognize when their libraries need to expand.

1 INTRODUCTION

  • Reusable skills let agents solve new tasks by recomposing familiar behaviors rather than learning each behavior from scratch.The paper motivates symbolic schemas as reusable, composable, and inspectable representations of transformations.Source: §1 · p. 2
  • Existing approaches often use predefined skill repertoires or signals beyond passive video, leaving online symbolic library construction from video an open question.The paper asks whether a model can use its own earlier library decisions to recognize familiar transformations and decide when a new schema is needed.Source: §1 · p. 2
  • Video2Skill benchmarks streaming skill discovery across robot tabletop manipulation and egocentric kitchen activity, measuring event recovery, naming-invariant grouping, and creation–reuse decisions.The benchmark is built from ROBOINTER and HD-EPIC.Source: §1 · pp. 2–3
  • Unified models tend to merge distinct transformations, whereas factorized models frequently duplicate schemas for recurring transformations.Unified processing jointly recognizes events and updates the library; factorized processing generates text descriptions before library updates.Source: §1 · p. 2
  • Across 19 open-source VLMs, many models group events near chance, and supervision improves grouping without reliably resolving creation–reuse errors.Models trained on task-specific data often absorb unseen transformations into existing schemas, even when they locate the events in video.Source: §1 · pp. 2–3

2 PROBLEM DEFINITION AND BENCHMARK CURATION

This section defines streaming embodied skill discovery and describes Video2Skill’s construction and evaluation. The task processes video incrementally while maintaining a library that persists across videos and informs later predictions.

2.1 TASK FORMULATION

  • SESD processes video in chunks, predicting event boundaries, symbolic skill calls, and an optional unfinished event while maintaining a persistent library.Earlier creation and reuse decisions shape later predictions; unfinished events are closed or discarded at video boundaries while the library persists.Source: §2.1 · p. 3
  • A skill schema has a stable free-form name, definition, and typed argument slots, and describes a physical transformation rather than an executable controller.Reference equivalence is based on the underlying transformation, independently of the model’s schema assignments.Source: §2.1 · p. 3

2.2 BENCHMARK CURATION

  • Video2Skill combines egocentric kitchen recordings from HD-EPIC with robot tabletop videos from ROBOINTER, whose median verified segments last 1.9 s and 4.5 s, respectively.ROBOINTER videos are drawn from DROID and RH20T.Source: §2.2 · p. 3
  • Curation verifies manipulation events against visual evidence, represents retained calls with typed argument bindings, and organizes them into 48 canonical transformation classes.The references cover verified manipulations rather than exhaustive temporal coverage; training and test splits are disjoint at the source-video level.Source: §2.2 · p. 4

2.3 EVALUATION PROTOCOL

  • The evaluation covers event recovery, naming-invariant skill grouping, and creation–reuse decisions, with temporal matches accepted at IoU ≥0.3.Unmatched references count toward the coverage denominator, while unmatched predictions incur no penalty.Source: §2.3 · p. 4
  • Grouping compares pairs of matched events assigned to the same reference class or predicted name; low precision signals overmerging and low recall signals fragmentation.Pair counts favor frequent classes, and ARI does not ensure class balance.Source: §2.3 · p. 4
  • Creation–reuse evaluation compares first occurrences of reference classes with first occurrences of predicted names, using CREATE=1 and REUSE=0.The reported recalls and accuracy measure novelty within the matched sequence, not explicit operations or complete library history, and exclude CONTINUE evaluation.Source: §2.3 · p. 4

3 HOW DO CURRENT MODELS BEHAVE?

Current models struggle to form consistent skill partitions: unified processing tends to merge distinct transformations, while factorized processing tends to duplicate recurring ones. Scaling does not consistently improve skill abstraction.

3 HOW DO CURRENT MODELS BEHAVE?

  • The evaluation uses verified test splits in balanced order and distinguishes unified joint event-and-library processing from factorized text-only library updates.Pair precision and recall are examined separately to distinguish overmerging from fragmentation.Source: §3 · pp. 4–5, Table 1 · p. 5
  • Unified models tend to overmerge distinct transformations, while factorized models tend to fragment recurring ones into duplicate schemas.On HD-EPIC, switching Qwen3.5-2B to factorized processing raises pair precision from 9.2 to 48.0 but lowers pair recall from 53.8 to 2.4.Source: §3 · p. 5
  • Scaling does not consistently improve skill partitions: Qwen3.5-27B improves HD-EPIC ARI from 8.7 to 19.9 but reduces ROBOINTER ARI from 19.4 to 4.6.Across configurations, ARI remains at or below 27.0.Source: §3 · pp. 5–6
  • Coverage and grouping quality can diverge, so event recovery, partition agreement, and creation–reuse decisions must be interpreted together.Factorized LLaVA-OneVision-2-8B reaches 71.7% ROBOINTER coverage but pair recall of 0.8 and ARI of 0.1.Source: §3 · p. 6
  • The observed biases motivate testing whether models use accumulated libraries and judge transformation equivalence across changing observations.The section identifies complementary needs: recognizing novelty despite familiar schemas and recurrence despite changes in objects, scenes, or descriptions.Source: §3 · p. 6

4 LEARNING TO MAINTAIN A SKILL LIBRARY

The paper fine-tunes models with different library histories to improve skill grouping and library maintenance. Supervision improves grouping, but gains vary by setting and do not eliminate difficulty distinguishing novelty from recurrence.

4 LEARNING TO MAINTAIN A SKILL LIBRARY

  • The shared supervised objective maps a current video chunk or proposed event description and prior library state to a target event-grounding or library-management response.The training strategies differ in how they construct the training distribution.Source: §4.1 · p. 6, §4 · p. 6

4.1 SUPERVISION STRATEGIES

  • Oracle-history SFT trains on annotated episode histories, pairing each observation with the preceding reference-produced library and unfinished-event state.Unified training also periodically resets to provide repeated supervision for initialization.Source: §4.1 · p. 6
  • CLARE edits library states while holding observations fixed, changing targets when relevant schemas change and preserving them under irrelevant edits.Its counterfactual examples branch from reference trajectories without changing subsequent states.Source: §4.1 · p. 6
  • On-policy correction trains on teacher-labeled states reached through student rollouts, recomputing targets for the library the student actually accumulated.A previously missed schema may make a recurring transformation require creation rather than the reference history's reuse decision.Source: §4.1 · p. 6

4.2 EFFECTS OF SUPERVISION

  • All three supervision strategies improve ARI across both backbones, domains, and paradigms, though unified Qwen3.5-4B pair precision gains coincide with reduced HD-EPIC coverage.Oracle-history SFT raises pair precision from 30.0 to 67.6 on ROBOINTER and from 12.7 to 31.1 on HD-EPIC; in the latter setting, coverage falls from 69.9 to 49.6.Source: §4.2 · p. 7
  • Training-state interventions provide setting-specific rather than uniform benefits over reference-history supervision.Oracle-history SFT has the highest unified ARI in all four backbone–domain settings, while CLARE and on-policy correction improve other measures in some settings.Source: §4.2 · p. 7
  • Better grouping does not ensure correct creation and reuse: unified models retain high reuse recall but substantially lower creation recall, while factorized pair recall remains limited.Unified reuse recall is 97.0–98.7 versus creation recall of 19.1–40.0; factorized pair recall is 5.5–31.4.Source: §4.2 · p. 7

5 FURTHER DISCUSSIONS

Further analyses show that video order and training frequency do not remove key library-maintenance gaps. Models remain especially challenged by recognizing when an existing skill is insufficient.

5 FURTHER DISCUSSIONS

  • Across video orders, oracle-history SFT ends with 21–22 schemas versus 46 reference classes; schema-name consistency improves to mean pairwise Jaccard similarity 0.78 from 0.59 zero-shot.In the innovation-heavy order, all 46 reference classes appear by video 20, but the model has introduced only 20 schemas.Source: §5 · p. 8, Figure 2 · p. 8
  • Training gains are uneven across skill frequencies: high-frequency classes benefit most consistently, while medium-frequency scores remain below 10 on both datasets.Performance does not vary monotonically with frequency; low-frequency groups sometimes outperform medium-frequency groups.Source: §5 · p. 8
  • For unseen HD-EPIC classes, temporal coverage reaches 57.6–66.1 after supervision, but a new schema is assigned at the first matched occurrence for only zero or one of the covered classes.Oracle-history SFT raises unseen-class macro pair F1 from 8.6 to 25.5, yet creates a new name at the first matched occurrence of only one of ten covered classes.Source: §5 · p. 8

6 RELATED WORK

Related work spans video benchmarks for predefined action understanding and approaches that learn or reuse skills. Video2Skill focuses on consolidating temporally grounded events into reusable transformation schemas and deciding when to expand their library.

Video understanding and streaming models

  • Video understanding benchmarks commonly assess predefined action categories or answers to supplied questions, while unsupervised action segmentation discovers recurring structure offline.Online detection and streaming-video models add incremental processing and temporal memory; Video2Skill instead assigns grounded events to a persistent library that conditions later predictions.Source: §6 · pp. 8–9

Skill discovery and evolving concept inventories

  • Skill-learning work ranges from interaction- or demonstration-based discovery to explicit libraries grown with feedback, while some video methods learn shared representations for downstream control.XSkill uses a fixed-size set of prototypes; other approaches build symbolic skill-composition models or video-derived repositories.Source: §6 · p. 9
  • Video2Skill studies familiar-versus-novel decisions for temporally grounded transformations with argument slots, allowing different objects and scenes to share a schema.The related framing connects library expansion with novel-category discovery, continual learning, and in-context learning.Source: §6 · p. 9

7 CONCLUSION

Video2Skill studies whether streaming visual experience can become a persistent library of reusable skills. Across robot manipulation and egocentric kitchen activity, models show complementary abstraction errors, and supervision only partly alleviates them.

7 CONCLUSION

  • Across 19 VLMs, unified models tend to merge distinct transformations, while factorized models frequently duplicate schemas for recurring transformations.Oracle-history SFT, counterfactual library-state rebalancing, and on-policy correction partially alleviate these failures but do not reliably resolve novelty-versus-reuse decisions.Source: §7 · p. 9
  • The findings support evaluating event coverage and abstraction quality jointly and establish a testbed for coherent skill libraries as experience accumulates.Source: §7 · p. 9
  • The benchmark covers robot manipulation and egocentric kitchen activity; broader settings and downstream planning or executable-control utility remain unevaluated.Outdoor activities are specifically identified as unexplored.Source: §7 · p. 9

Related papers

Loading 2609.36691v1…