Source-linked AI summary
InSight: Self-Guided Skill Acquisition via Steerable VLAs
Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, Mac Schwager
TL;DR
VLA capabilities are limited by the skills represented in their training data, making new manipulation skills costly to acquire. InSight segments demonstrations into steerable primitives and uses a VLM-guided loop to identify, generate, and integrate missing skills, achieving up to 96% success and 80% success on a 14-primitive task without target-skill human demonstrations.
Problem
VLA capabilities are bounded by training-data skills, while acquiring new manipulation demonstrations is costly in human effort, trials, and safety constraints.
Method
InSight automatically segments demonstrations into labeled primitives, then uses a VLM to identify missing primitives, generate rollouts, evaluate success, and retrain the VLA.
Results
Across five simulated and hardware tasks, InSight acquires skills without target-skill human demonstrations, reaching up to 96% success and 80% on a 14-primitive long-horizon task.
Takeaways & Limitations
Primitive-level steerability supports autonomous skill acquisition and compositional reuse for new long-horizon manipulation tasks without additional human demonstrations.
Takeaways & Limitations
Skill-gap execution is restricted to single-axis motions, limiting the complexity of primitives that can be acquired.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data. We present InSight, a framework that unlocks autonomous skill acquisition by rendering VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle"). InSight consists of two primary stages: (1) an automated segmentation pipeline that partitions demonstrations into labeled primitives via VLM plan decomposition and end-effector poses to enable VLA primitive steerability, and (2) a VLM-guided data flywheel that identifies missing primitives required to accomplish a novel task, autonomously attempts demonstrations of the missing primitives with VLM-proposed low-level control, and automatically labels, stores, and integrates successful demonstrations into the VLA training set. We evaluate InSight across simulation and real-world manipulation tasks, including block flipping, drawer closing, sweeping, twisting, and pouring, without any human demonstrations of these target skills. Once learned, these primitives can be composed to execute novel, long-horizon tasks without additional human demonstrations. Our findings demonstrate that primitive steerability provides a practical foundation for continual skill acquisition in VLA policies. Project website: https://insight-vla.github.io.
1 Introduction
INSIGHT addresses the cost and skill-coverage limits of VLA manipulation by making policies steerable through composable primitives and autonomously acquiring missing capabilities. It segments demonstrations automatically, uses VLM-guided practice to extend the training set, and achieves up to 96% success on target tasks without target-skill human demonstrations.
- Motivation: Human demonstrations and policy finetuning make teaching robots new manipulation skills expensive, while VLA capabilities remain bounded by training-data skills.Simulation-based reinforcement learning can also require thousands of trials, and a VLA trained only for scooping may fail to sweep.
- Motivation: Manipulation skills are compositional: novel tasks often reuse known primitives while adding only task-specific primitives, such as lateral pushing or rotation.Sweeping and scooping share approach and lowering; flipping reuses pick-and-lift from pick-and-place but adds rotation.
- Framework: INSIGHT makes VLAs steerable at the composable primitive level and autonomously extends them when a novel task requires a missing primitive.The framework spans primitive segmentation, VLM-guided acquisition, and modular composition of new primitives.
- Framework: Its pipeline automatically labels primitives from teleoperated demonstrations, identifies missing primitives, executes VLM-parameterized attempts, and retrains the VLA on successful autonomous demonstrations.This enables reusable acquired skills for future tasks and continual, lifelong learning.
- Results: Up to 96% success was achieved on tasks including pouring, while a complex 14-primitive long-horizon task reached 80% success without target-skill human demonstrations.Experiments covered five simulation and hardware tasks—block flipping, drawer closing, sweeping, twisting, and pouring—and retained full performance on original base skills.
2 Related Work
Prior work develops steerable VLA interfaces, hierarchical skill decomposition, foundation-model-guided robotics, autonomous skill acquisition, and continual learning. InSight differs by using primitive steerability to identify, acquire, and integrate missing skills into a VLA.
- Vision-Language-Action Models and Steerable Policies: Prior steerable VLA methods expose finer-grained language-conditioned control interfaces, but treat the resulting primitive set as fixed.Examples include dense language relabeling and related steerable-policy interfaces.
- Vision-Language-Action Models and Steerable Policies: InSight identifies missing primitives for new tasks, generates successful primitive rollouts, and adds them to the VLA’s training data.This extends steerability from a test-time interface into persistent expansion of the learned primitive library.
- Skill Decomposition and Composition: Hierarchical and bottom-up robot-learning methods decompose or discover reusable skills for sequencing and long-horizon manipulation, typically assuming an available skill or primitive structure.These approaches include clustering, representation learning, hierarchical imitation learning, and factorized skill spaces.
- LLM and VLM-Guided Robotics: Foundation-model-guided robotics provides semantic guidance, value maps, or test-time skill composition, but prior examples do not learn missing low-level skills or expand the primitive library.The passage contrasts SayCan, VoxPoser, and Code-as-Policies with InSight’s skill-acquisition focus.
- Autonomous Skill Acquisition: Real-world reinforcement-learning-based skill acquisition is constrained by dense rewards, extensive interaction data, and a narrow sim-to-real gap, motivating reduced-supervision alternatives.Related approaches use foundation-model-generated rewards, automated demonstrations, or language-labeled trajectories.
- Continual, Lifelong Skill Learning: Continual-learning methods reuse structured knowledge through evolving expert-routed spaces or semantic skill subspaces, whereas InSight retrains one VLA jointly on original and newly acquired primitives.This represents InSight’s data-centric approach to retaining and extending capabilities.
3 Skill Acquisition via Steerable Primitives
InSight enables autonomous skill acquisition by training a steerable VLA on automatically segmented primitives, then using a VLM to identify gaps, generate and evaluate rollouts, and retrain the model with successful primitives.
- Overview: InSight has two stages: primitive-segmented VLA training and VLM-orchestrated autonomous acquisition of missing primitives.The acquisition loop includes primitive gap identification, data generation, success evaluation, and retraining.
- Primitive Representation: Primitives are reusable, language-labeled action segments with defined preconditions and effects, designed around a single dominant motion mode.Examples include translation or rotation along an axis and gripper transitions.
- Primitive Segmentation and Training: Demonstrations are automatically segmented using VLM-generated primitive sequences, gripper transitions, end-effector motion features, and dominant-axis tags.Each segment becomes a separate LoRA fine-tuning episode conditioned on its primitive label, with a learned progress channel for termination.
- Autonomous Skill Acquisition: For a novel task, the VLM decomposes a plan, compares primitives against the known vocabulary, and flags absent primitives as gaps.Each gap is constrained to one single-axis motion, so distinct motions become separate gaps rather than one composite primitive.
- Autonomous Skill Acquisition: The VLM parameterizes each gap by a signed translation or rotation axis and magnitude, proposes execution parameters from the scene, and maintains consistency across chained motions.For example, paired pour-forward and tilt-back-upright motions share compatible parameters.
- Autonomous Skill Acquisition: Successful new primitives are added to the training dataset, after which the VLA is retrained so future plans can invoke them as known capabilities without additional human demonstrations.A post-plan VLM oracle judges full task success by comparing scene images before and after execution.
4 Experiments and Analyses
Experiments in simulation and on hardware test whether primitive-level steerability enables efficient acquisition of missing skills, OOD execution, compositional reuse, and retention of existing abilities. INSIGHT acquires and composes manipulation primitives without human demonstrations of the target skills.
- Simulation: block flipping: INSIGHT reaches 75% block-flip success after 246 acquired primitive rollouts over 479 attempts, while SAC never completes a flip under a comparable rollout budget.The task starts from pick-and-place demonstrations, and rotate-block is identified as the missing primitive.
- Simulation: drawer closing: Across 82 episodes, INSIGHT produces 70 successful close-drawer primitives from an OOD open-drawer state using a VLM completion check to trigger the missing push primitive.The completion check bridges imperfect termination of the known approach primitive.
- Simulation: drawer closing: The retrained unified VLA closes drawers with 100% success over 25 trials while retaining its ability to open them, despite imperfect semantic transition alignment.Incorrect axis selection is the dominant primitive-acquisition failure mode.
- Hardware: twisting and pouring: INSIGHT achieves 92% and 96% end-to-end success on twisting and pouring, versus 32% and 16% for CaP-X and 0% for π0.5.The π0.5 baseline fails entirely on twisting and pouring because those primitives are absent from its demonstration data.
- Hardware: compositional reuse: INSIGHT composes 14 primitives from separately acquired twist and pour skills into a twist-then-pour task without combined-task demonstrations, reaching 80% end-to-end success.The sequence includes top grasping, cap twisting, side regrasping, and pouring beans into a bowl.
- Hardware: efficiency and retention: INSIGHT acquires 20 successful primitives in 23 twisting trials and 31 pouring trials, while preserving 100% success on original top- and side-pick-and-place skills.The unified VLA retains existing skills after newly acquired twist and pour primitives are added.
- Hardware: contact-rich motions: INSIGHT autonomously acquires a contact-rich sweeping primitive after the VLM flags lateral pushing as missing, succeeding in all 5/5 evaluation trials.This evaluates extension beyond prehensile motions.
5 Conclusion, Limitations, and Future Work
INSIGHT enables autonomous VLA skill acquisition by discovering primitive gaps and generating training data without additional human demonstrations. Its current limitations include single-axis skill execution, reliance on successful rollouts, and the need for human environment resets.
- Conclusion: INSIGHT enables robots to acquire new skills without additional human demonstrations by segmenting primitives, discovering gaps through VLM reasoning, and generating training data with VLM-guided low-level control.The method combines autonomous primitive segmentation, VLM-guided primitive gap discovery, and execution.
- Limitations and Future Work: Skill-gap execution is restricted to single-axis motions, limiting the complexity of primitives that INSIGHT can acquire.Future work could use VLM-generated waypoints, trajectory optimization, or online RL to acquire richer primitives.
- Limitations and Future Work: INSIGHT filters for successful rollouts, while failure analysis and VLM feedback could improve sample-efficient skill acquisition.The passage identifies failure-aware learning as a future direction.
- Limitations and Future Work: Human environment resets are still necessary.
A Implementation Details · B VLM Demonstration Segmentation, Plan, Primitive Gap Proposal, and Oracle Check
The implementation uses π0.5 with LoRA fine-tuning and primitive-conditioned episodes, while InSight assigns Gemini 3 Flash four structured roles spanning segmentation, planning, gap proposal, and oracle verification.
- A Implementation Details: InSight uses the π0.5 VLA, while remaining agnostic to the underlying VLA.The experiments use π0.5, but the framework is not tied to that model.
- A Implementation Details: LoRA fine-tuning uses a Gemma-2B backbone and Gemma-300M action expert, with other weights frozen.The fine-tuning setup applies LoRA while freezing the remaining model weights.
- A Implementation Details: Each segmented primitive forms a separate training episode conditioned on its primitive label as the language prompt.Primitive segmentation therefore directly determines the training-episode structure and conditioning.
- A Implementation Details: The policy receives two 224×224 RGB views, end-effector pose, and gripper state as inputs.The views are a scene camera and a wrist camera.
- B VLM Demonstration Segmentation, Plan, Primitive Gap Proposal, and Oracle Check: Gemini 3 Flash performs offline segmentation of human demonstrations into primitive-labeled trajectories and decomposes novel tasks into primitive sequences with flagged gaps.Both VLM roles are constrained to return strict JSON.
- B VLM Demonstration Segmentation, Plan, Primitive Gap Proposal, and Oracle Check: Gemini 3 Flash maps each primitive gap to a single-axis motion for low-level execution and performs oracle checks of primitive and task completion.These are the third and fourth of InSight’s four constrained JSON VLM roles.
B.1 Demonstration Segmentation
InSight segments demonstrations offline by decomposing instructions into ordered primitives, assigning video frames to primitives using visual and end-effector-motion cues, and refining boundaries. The resulting contiguous, primitive-labeled segments become individual training episodes.
- The segmentation pipeline first decomposes each task instruction into an ordered sequence of primitives.
- It then assigns subsampled video frames to plan primitives by cross-checking images with captions of dominant end-effector translation or rotation.The pipeline returns boundary frames between consecutive primitives.
- A localized refinement reconciles end-effector-delta changepoints with the earliest visually unambiguous frames, producing contiguous primitive-labeled segments that become training episodes.
B.2 Task Planning and Primitive-Gap Flagging
InSight’s planner decomposes a goal into a complete primitive sequence, execution notes, and any novel primitive gaps. It flags only fundamentally new actions, restricting each gap to a single-axis motion so multi-motion tasks yield multiple gaps.
- Planning and gap identification: The planner returns the full primitive sequence, per-step execution notes, and the subset of steps that are novel primitive gaps.Each primitive gap is constrained to a single-axis motion.
- Planning and gap identification: Planning rules require existing primitives for covered substeps and define a gap as only the novel action, not a bundle of existing and novel actions.A gap is created only when the desired outcome is fundamentally different from every existing primitive’s result.
- Planning and gap identification: For pouring, the planner identifies two rotational gaps: “pitch bottle forward to pour” and “tilt bottle back upright.”Linear movement, grasping, and lifting are covered by existing primitives, while forward pitch and backward tilt are separate single-axis gaps.
B.3 Primitive-Gap Proposal
InSight’s primitive-gap proposal uses VLMs to select a single-axis translation or gripper-local rotation, with a signed magnitude and completion status, then executes that motion until completion. Axis selection is grounded in exterior and wrist-camera views, with geometric constraints distinguishing translation, tilt, and axial twist.
- Proposal output: For each flagged gap, the VLM predicts one motion axis, signed magnitude, and already_complete flag, while the controller drives the arm along that axis until completion.Translations use base-frame dx, dy, dz; rotations use gripper-local drx, dry, drz.
- Axis selection: Translation reasoning uses the exterior overview, whereas rotation-axis selection uses the wrist down-view because its axes move with the gripper.The wrist camera’s local axes are independent of the global room frame.
- Geometric constraints: Forward or backward tilting selects dry, lateral tilting selects drx, and spinning an object around its own centerline selects drz.drz cannot pivot, tilt, invert, or tip an object over.
- Magnitude specification: The controller represents translations in meters and rotations in degrees, with typical magnitudes of 0.05-0.20 m and 30-180 deg, respectively.The sign specifies motion along the chosen axis.
B.4 Oracle and Completion Checks
InSight uses image-based oracle checks to accept only task-completing trials as training demonstrations. Separate primitive checks rely primarily on the exterior side view, rejecting actions when a visible vertical gap remains.
- Task completion oracle: An oracle compares initial and final scene images, accepts completed trials as training demonstrations, and discards the rest.The completion response is constrained to valid JSON with a Boolean verdict and one-sentence reasoning.
- Primitive completion checks: Primitive completion checks use the exterior camera for depth and height, while the wrist camera is used only for centering and identification.The system explicitly instructs the model to reference the exterior image in its reasoning.
- Primitive completion checks: A primitive is not done if IMAGE 1 shows any vertical gap between the gripper bottom and target, even when IMAGE 2 shows apparent overlap.For move-to or touch primitives, completion means contact or proximity within ~5mm in IMAGE 1.
- Primitive completion checks: State-change primitives are complete when the target state is visibly achieved in IMAGE 1, such as a closed drawer or displaced rocks.The completion output uses JSON with a done Boolean and brief reasoning referencing IMAGE 1.
C VLM-Generated Plans
The VLM-generated plan decomposes the twist-then-pour task into a sequence of known primitives, including InSight-generated cap-twisting and bottle-pouring skills. It specifies opening the yellow bottle, regrasping it, pouring into the bowl, and returning the bottle to the table.
- Plan overview: All primitives in the twist-then-pour plan are marked known, including InSight-generated skills for twisting open the cap and pouring with the bottle.The plan identifies three InSight-generated primitives: twist open the cap, tilt bottle forward to pour, and tilt bottle back upright.
- Task decomposition: The task requires opening the yellow bottle cap, regrasping the bottle, and pouring its contents into the bowl.The plan begins with a top-down cap grasp, a 180-degree counterclockwise twist, cap detachment, and a side grasp on the uncapped bottle.
- Pouring sequence: After lifting the bottle, the plan moves it beside the bowl and tilts it forward to pour, then returns it upright.The bottle is lifted to clear the table before transport and is rotated back to a vertical orientation after pouring.
- Task completion: The final steps lower the bottle to the table and open the gripper to release it.These actions complete the plan after the bottle is returned upright.