Source-linked AI summary
EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
Wei Wang, Wenqiao Zhang, Yutong Lin, Yuqian Yuan, Tianwei Lin, Jinhao Mao, Zhenxuan Fan, Mingjian Gao, Yang Dai, Wentong Li, Zheqi Lv, Zheng Dong, Yingjie Niu, Jiaqi Zhu, Jun Xiao, Chao Li, Yueting Zhuang
TL;DR
Long-horizon VLA tasks require more than action prediction because proposed operations may be invalid or unverified as physical state evolves. EmbodiedSkills addresses this with a shared executable-skill interface and guarded agent loop, achieving strong execution results on RoboTwin 2.0 and LIBERO while exposing a limitation on memory-dependent tasks and underlying vision–language quality.
Problem
Long-horizon VLA agents must coordinate perception, planning, execution, verification, and recovery, since model decisions alone do not ensure current-state validity or verified outcomes.
Method
EmbodiedSkills separates high-level structured skill proposals from runtime validation, bounded low-level VLA execution, post-action verification, and structured trajectory recording.
Results
86.20% average success across 50 RoboTwin 2.0 tasks and 97.40% across four LIBERO suites were achieved by task-adapted low-level VLA policies, while four memory-dependent RMBench tasks reached 12.5%.
Takeaways & Limitations
The framework provides a trainable and inspectable agent layer with replaceable components for turning VLA policies into closed-loop embodied systems.
Takeaways & Limitations
Task-specific specialist policies increase training, storage, and deployment costs, and contracts cannot guarantee correct grounding or subgoal selection when progress is ambiguous or occluded.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions, but long-horizon tasks require more than action prediction. An agent must coordinate perception, planning, execution, progress verification, and recovery as the physical state evolves. An action prediction or a model-generated skill decision does not, by itself, guarantee that the proposed operation is valid in the current state or that its outcome will be verified. We propose EmbodiedSkills, a unified framework that treats each skill decision as an execution proposal: the runtime checks its prerequisites before execution and verifies the outcome afterward. A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification within a single agent loop. Because this interface remains fixed, low-level VLA policies can be replaced or adapted without changing the agent loop. The interface also records planning, execution, verification, and recovery events as structured trajectories, which provide supervision for individual components and can support optional online adaptation when interactive feedback is available. We instantiate EmbodiedSkills with Qwen3-VL and OpenPI/pi0.5 on RoboTwin 2.0 and LIBERO. Task-adapted low-level VLA policies achieve an average success rate of 86.20% across 50 RoboTwin 2.0 tasks and 97.40% across the four LIBERO suites. These results establish the execution performance of the task-adapted low-level VLA policies used in EmbodiedSkills. On four memory-dependent RMBench tasks, the same task-adapted execution approach achieves 12.5% average success. The framework provides a trainable and inspectable agent layer for turning these policies into closed-loop embodied systems.
1 Introduction
Long-horizon VLA tasks require a closed-loop agent layer that coordinates perception, planning, execution, verification, and recovery. EmbodiedSkills provides this layer through executable skills, separating policy proposals from guarded runtime validation and structured feedback.
- Motivation: Long-horizon manipulation requires coordinating perception, planning, execution, verification, and recovery as the physical state evolves.A reliable system must assess executability, verify intended outcomes, and recover from failures.
- Motivation: Existing VLA policies leave intermediate task decisions implicit, while explicit model decisions may still be invalid, stale, incomplete, or physically ineffective.These issues are difficult under partial observability, delayed progress feedback, and contact-rich failures.
- Framework: EmbodiedSkills defines executable skills with typed inputs, outputs, and prerequisites, connecting high-level selection, bounded low-level VLA execution, and post-execution verification.The shared interface supports orchestration, training, and deployment without embedding execution semantics in benchmark-specific prompts or scripts.
- Evaluation: 86.20% average success was achieved across 50 RoboTwin 2.0 tasks, versus 82.74% for the π0.5 reference, while LIBERO reached 97.40% versus 96.85% official OpenPI reference.On four memory-dependent RMBench tasks, the same task-adapted execution approach achieved 12.5% average success.
- Framework: The closed-loop AgentLoop coordinates observation, planning, readiness checks, bounded execution, progress verification, and recovery instead of following a fixed one-pass pipeline.The policy can re-observe, revise the plan, continue a subgoal, advance, or recover as task state changes.
- Framework: Policy–runtime separation makes structured decisions subject to prerequisite, freshness, validity, and state-transition checks, leaving failures explicit and trajectories diagnosable.The interfaces also record multimodal context, decisions, outcomes, errors, and state transitions for component-level training.
2 Related Work
Related work spans increasingly general VLA policies, language-guided robot-skill coordination, reinforcement learning with feedback, and broader long-horizon benchmarks. EmbodiedSkills positions its executable interfaces and AgentLoop as orthogonal to particular policies or RL objectives.
- VLA policies: VLA research has progressed from early transformer policies to open, reusable, increasingly general-purpose policies across embodiments and tasks.Examples include RT-1, RT-2, Open X-Embodiment, RT-X, OpenVLA, Octo, π0, π0.5, and Qwen-VLA.
- Robot skill coordination: Language- and vision-language-based robotic systems make skill coordination explicit through affordance scoring, environment feedback, executable programs, value maps, and embodied multimodal inputs.Recent systems also explore object-centric manipulation, visual prompting, language-grounded planning, modular routing, and hierarchical VLA execution.
- Agent reinforcement learning: Robot reinforcement learning faces costly exploration, partial observability, and sparse rewards, while deployment feedback is becoming increasingly important for robot foundation policies.EmbodiedSkills is orthogonal to a particular RL algorithm or optimization target.
- Agent reinforcement learning: EmbodiedSkills uses structured trajectories for supervised adaptation and optional online optimization, while allowing planners, verifiers, selectors, and low-level VLA policies to be adapted independently.The main contribution is the AgentLoop and executable interfaces rather than a specific RL method.
- Benchmarks: Robot benchmarks now cover long-horizon, language-conditioned, bimanual, household, and cross-embodiment settings, but terminal success alone does not localize agent failures.This motivates studying execution monitoring and more detailed agent behavior.
3 Methodology
EmbodiedSkills models an embodied VLM–VLA system as a guarded finite-stage controller over executable skills. A high-level policy selects phase-admissible structured operations, a low-level VLA executes bounded action chunks, and the runtime records outcomes.
- AgentLoop: The high-level agent policy reads the task, visual evidence, loop state, and phase-admissible skills before selecting one structured operation.The low-level VLA maps the active subgoal, current observation, and robot state to a bounded action chunk.
- Execution: The loop separates high-level scheduling from low-level action generation, allowing the low-level VLA policy to execute each active subgoal as a bounded action chunk.Figure 2 depicts decomposition, executable-subgoal selection, bounded execution, and post-action progress verification.
- Agent state: An episode begins with instruction x and environment E, while each loop step maintains a method-level state containing the current phase, available artifacts, and ordered trace.Artifacts can include observations, plans, subgoals, preflight evidence, action chunks, execution reports, verification reports, and recovery context.
- Agent state: The policy receives a deployment-consistent compact context that preserves the complete plan, active subgoal, artifact summaries, recent errors, and bounded recent decisions and results.Older history is compressed or dropped under a fixed budget, while the current plan and subgoal remain explicit.
- Skill decisions: The policy chooses from a state-dependent action set containing only choices compatible with the current artifacts.Skill execution, forward progression, and termination share one structured decision interface, while evidence-dependent rerouting remains runtime-governed.
3.2 Executable Skill Contract
The executable skill contract turns model-generated operations into runtime-validated physical transitions. It defines typed interfaces, prerequisites, execution and state updates, explicit failure evidence, and artifact invalidation rules.
- Contract structure: Each embodied skill is represented by typed input and output schemas, prerequisites, an executable operation, a post-execution state update, and explicit failure mapping.The contract applies to both model-backed skills and deterministic operations.
- Runtime validation: A model-generated result remains a proposal until its schema and prerequisites are validated, after which successful outputs become typed artifacts and failures remain explicit evidence.The runtime therefore distinguishes textual decisions from physical state transitions.
- Invalidation: Artifact provenance and freshness prevent observations, plans, and actions from being silently reused after their dependencies change.Only artifacts depending on changed evidence need to be refreshed, allowing still-valid context to be reused.
- Phases: The runtime organizes the loop into six semantic phases: Observe, Plan, Preflight, Execute, Verify, and Recover.These phases describe the loop structure rather than a rigid one-pass program, because earlier phases may be revisited when evidence changes.
- Grounding: Grounding is task- and policy-conditioned, ranging from no explicit binding to multiple objects or destinations, with geometry-based controllers potentially requiring explicit bindings.Direct language-conditioned VLA policies may instead operate from images, robot state, and subgoal text.
3.4 Guarded Runtime Transition
The runtime separates learned skill proposals from execution semantics by guarding operations, action chunks, and control transitions with state- and evidence-based checks.
- Guard design: Separate guards validate skill proposals, forward progression, and termination because these controls have different invariants.The policy proposes decisions, while the runtime determines whether they are executable under the current state.
- Guard design: The runtime records operation results, advances, terminations, and blocked decisions as distinct state transitions.This separation lets learned policies choose meaningful operations without placing execution safety or state consistency in the prompt.
- Bounded execution: A low-level VLA policy maps the active subgoal, current observation, robot state, and readiness evidence to a bounded action chunk and execution report.The execution interface explicitly associates generated actions with the current task context.
- Bounded execution: Action acceptance checks the policy-specific schema, numerical validity, horizon, and consistency with the current subgoal and observation.The runtime enforces 0 < H_t ≤ H_max and requires fresh, valid execution context.
3.6 Verification and Recovery
Verification uses post-action evidence to choose among continuation, progression, re-observation, replanning, recovery, and termination within a guarded loop.
- Verification and recovery: The verification route can advance the plan, continue the current subgoal, reobserve, replan, recover, or finish.These choices are represented by the route set V = {Advance, Continue, Reobserve, Replan, Recover, Finish}.
- Verification and recovery: Recovery reuses explicit task state and failure evidence to construct a revised execution context that is checked before another physical action.Recovery therefore returns to the same guarded execution mechanism rather than bypassing runtime validation.
- Guarded loop: Algorithm 1 initializes the loop in Observe and repeatedly builds context, computes admissible actions, queries the policy, and processes the resulting decision.The loop runs for t = 0, ..., T − 1 under a step budget T.
- Guarded loop: RunSkill decisions execute only when Gskill(st, dt) = 1; otherwise, the loop records the blocked decision and its evidence.AdvanceStage and Finish decisions are likewise accepted only when their corresponding guards pass.
- Guarded loop: Each decision and operation result is appended to the ordered trace, which becomes context for subsequent decisions.The algorithm returns the final task state together with the loop trace.
3.7 Algorithmic Summary
The algorithm summarizes EmbodiedSkills as a guarded controller whose typed interfaces connect task state, runtime validation, trajectory logging, and interchangeable policy modules.
- Algorithmic summary: Runtime guards block decisions that lack valid evidence, while every operation result is appended to the ordered trace used by later decisions.The AgentLoop therefore exposes both admissible actions and accumulated execution history to the policy.
- Algorithmic summary: EmbodiedSkills combines typed task state, phase-structured skill interfaces, runtime validation, trajectory logging, environment adapters, and action policy modules.Components may use a shared base model with stage-specific adapters or separate models if they obey the same skill and state contracts.
4 Training the Agentic Layer
EmbodiedSkills trains agent components through deployment-consistent interfaces and structured trajectories, while allowing optional interactive optimization without requiring joint training.
- Training strategy: Component-level supervision adapts planning, skill selection, execution, verification, and recovery through explicit interfaces rather than one end-to-end optimization procedure.The same trajectory interface can also support optional closed-loop policy optimization when interaction data and a reliable evaluator are available.
- Component adaptation: The planner maps instructions and visual context to ordered executable semantic subgoals, while the low-level VLA policy generates actions to reach them.Subgoal supervision avoids embedding simulator-specific control details in the high-level plan.
- Component adaptation: Subtask demonstrations condition the low-level VLA policy on observation, robot state, and the active subgoal before emitting bounded action chunks.Fresh evidence is collected after each chunk, and the verifier decides whether execution continues or the loop advances.
- Component adaptation: Qwen3-VL is supervised on deployment-consistent decision traces while the low-level VLA policy remains frozen.This isolates state-conditioned scheduling and skill use from continuous-control learning.
- Deployment consistency: Training inputs match deployment boundaries: planners receive planning context, verifiers receive fresh post-execution evidence, and schedulers receive admissible choices, artifacts, and errors.The shared history-compaction rule prevents training components with evidence unavailable at test time.
- Optional online adaptation: Interactive episodes interleave model decisions, executed skills, observed outcomes, typed runtime traces, and environment rewards.Infrastructure failures are recorded separately from agent-policy failures, and the low-level VLA policy can remain fixed during agent-level adaptation.
- Optional online adaptation: Online optimization is optional because episode-level attribution assigns identical returns to multiple decisions whose causal contributions may differ.The paper treats it as a refinement mechanism, not a replacement for component-level supervision or the headline results.
5 Experiments
Experiments evaluate task-adapted VLA execution across RoboTwin 2.0, LIBERO, and memory-dependent RMBench tasks, then isolate AgentLoop mechanisms through controlled ablations. Results show strong cross-benchmark performance but substantially lower success on memory-dependent tasks, while verification, semantic subtasks, and adaptive continuation each contribute to RoboTwin performance.
- RoboTwin 2.0: 39 of 50 RoboTwin 2.0 tasks improve over the LingBot-VA reference, one matches, and ten are lower.Gains concentrate on difficult contact-sensitive and multi-stage tasks; most regressions are one or two percentage points on references above 90%.
- LIBERO: 97.40% average success across four LIBERO suites exceeds the official OpenPI reference of 96.85% by 0.55 percentage points.The four suites are LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long.
- RMBench Memory-Dependent Tasks: 12.5% average success is achieved on four memory-dependent RMBench tasks.This evaluation tests subtask-conditioned execution when the correct action depends on prior interaction history.
- AgentLoop Ablations: Removing verification loses 38.0 points, removing semantic subtask conditioning widens the gap to 51.8 points, and one chunk per subtask is insufficient.The controlled ablations use the same 5,000 RoboTwin 2.0 episodes and vary verification, semantic conditioning, and bounded continuation.
- Execution Examples: Recorded RoboTwin executions cover object reorientation, multi-object stacking, and articulated switch interaction, with final states accepted by the environment evaluator.Each example uses visually grounded subgoals and adapted VLA execution through intermediate states.
6 Limitations
EmbodiedSkills improves modularity and explicit control, but performance and deployment remain constrained by component quality, coordination, computational cost, and task-specific policy requirements.
- Task-specific specialists improve execution support but increase training, storage, and deployment costs relative to a single generalist policy.This motivates more capable subgoal-conditioned generalist VLA policies.
- The guarded loop depends on calibrated planning, verification, and action generation, while additional observations and VLM calls introduce latency.Optional online adaptation also requires costly interactive robot experience.
- Explicit contracts prevent invalid state transitions but cannot ensure semantically correct grounding or subgoals, and verification can fail under occlusion or visual ambiguity.Robust operation therefore remains dependent on the quality and calibration of underlying vision-language components.
- Modular adaptation creates coordination trade-offs because independently specialized components must remain compatible at shared interface boundaries.Long episodes may accumulate small planning and execution errors.
- Long-horizon decomposition can increase observations, retries, and model calls before completion, making decomposition level a task- and robot-dependent design choice.These costs are described as intrinsic considerations for deployment.
7 Conclusion
EmbodiedSkills turns VLA action models into closed-loop embodied agents by separating structured skill proposals from runtime-enforced execution semantics. Its task-adapted policies achieve strong execution results on RoboTwin 2.0 and LIBERO, while RMBench exposes a harder memory-dependent setting.
- EmbodiedSkills separates structured skill proposals from runtime enforcement of prerequisites, freshness, action validity, and legal state transitions.Its AgentLoop repeatedly observes, plans, checks readiness, executes bounded chunks, verifies progress, and recovers from failures.
- The fixed interface keeps planners, verifiers, low-level VLA policies, and environment adapters independently replaceable and adaptable.This preserves the surrounding loop when components are frozen, specialized, or replaced.
- Deployment-consistent traces supervise semantic planning, phase-aware skill selection, and post-action verification, while subtask demonstrations adapt low-level VLA policies.Optional online adaptation uses the same structured trajectory interface when interactive feedback is appropriate.
- 86.20% average success was achieved across 50 RoboTwin 2.0 tasks with task-adapted π0.5 policies.The result is reported for the RoboTwin 2.0 instantiation.
- 97.40% was achieved across four LIBERO suites, while four memory-dependent RMBench tasks reached 12.5% average success.The framework also provides explicit trajectories and state interfaces for diagnosis and component training.