PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models

Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, Siheng Chen

arXiv:2605.13119v2cs.ROcs.AIcs.CV

TL;DR

Long-horizon embodied tasks require planning and diverse physical operations, which remain challenging for standalone VLAs. VLAs-as-Tools pairs a planning VLM with specialized bounded VLA tools, using TAPT and an interface for explicit calls and feedback. It improves success and invocation fidelity across the reported simulated benchmarks and real-world Franka tasks.

  • Problem

    Standalone VLAs remain limited on long-horizon tasks that demand decomposition, progress tracking, recovery, and skill composition, while embodied agents often depend on weaker physical tools.

    Source: §1 · pp. 1–2

  • Method

    VLAs-as-Tools pairs a planning VLM with specialized bounded VLA tools, using TAPT for invocation-aligned specialization and a tool-family interface for calls and feedback.

    Source: §1 · p. 2

  • Results

    Across five simulated benchmarks, VLAs-as-Tools improves π0.5 average success by 8.9 points, invocation fidelity by 15.0 points in Non-biased Rate, and real-world macro-average success from 16.7% to 58.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The formalized interface makes long-horizon robot control a sequence of bounded tool invocations that can be monitored and composed by a high-level agent.

    Source: §3 · p. 3

Abstract

from arXiv · show

Vision-language-action (VLA) models are effective robot action executors, but they remain limited on long-horizon tasks due to the dual burden of extended closed-loop planning and diverse physical operations. We therefore propose VLAs-as-Tools, a strategy that distributes this burden across a high-level vision language model (VLM) agent for temporal reasoning and a family of specialized VLA tools for diverse local physical operations. The VLM handles scene analysis, global planning, and recovery, while each VLA tool executes a bounded subtask. To tightly couple agent planning with VLA tool execution in long-horizon tasks, we introduce a VLA tool-family interface that exposes explicit tool selection and in-execution progress feedback, enabling efficient event-triggered agent replanning without continuous agent polling. To obtain diverse specialized VLA tools that faithfully follow agent invocations, we further propose Tool-Aligned Post-Training (TAPT), which constructs invocation-aligned training units for instruction following and adopts tool-family residual adapters for efficient tool specialization. Across five simulated benchmarks, VLAs-as-Tools improves the success rate of pi05 by 8.9 percentage points on average and enhances invocation fidelity by 15.0 points in Non-biased Rate. On three real-world Franka tasks, it improves success rate from 16.7% to 58.3% under matched data and compute. Code will be released.

1 Introduction

The introduction identifies a gap between VLAs’ physical execution and long-horizon reasoning, and agents’ planning and physical skills. VLAs-as-Tools pairs a planning VLM with specialized VLA tools, using TAPT and a tool-family interface; the reported evaluation shows gains across simulated and real-world tasks.

1 Introduction

  • End-to-end VLAs offer strong robot control but remain limited as standalone long-horizon agents, which must decompose tasks, track progress, recover from failures, and compose skills.The introduction attributes these limits to the demands of long-horizon tasks, limited embodied data, and model-scale constraints on planning and understanding.Source: §1 · p. 1
  • Embodied agents provide stronger planning but often rely on weaker physical tools, complementing end-to-end VLAs’ stronger execution and weaker long-horizon reasoning.Existing embodied tools are often manually specified and task-specific.Source: §1 · p. 2
  • VLAs-as-Tools distributes long-horizon reasoning across a planning VLM that tracks state and recovers, and specialized VLA tools that execute bounded subtasks.Specialized tools cover diverse subtasks and can be adapted from task-specific demonstrations.Source: §1 · p. 2
  • TAPT aligns bounded subtasks with precise instructions and uses tool-family residual adapters, while typed invocations and feedback route tools and provide intervention points for replanning.The interface supports low-frequency but responsive replanning.Source: §1 · p. 2
  • Across five simulated benchmarks, VLAs-as-Tools improves π0.5 average success by 8.9 points, invocation fidelity by 15.0 points in Non-biased Rate, and real-world macro-average success from 16.7% to 58.3%.The evaluation covers three VLA backbones, six benchmarks, and three real-world Franka tasks.Source: §1 · p. 2

2 Related Work

Related work spans VLA policies and agentic embodied systems, but the paper distinguishes its approach by making VLA tools agent-visible and aligning their behavior with invocations.

2 Related Work

  • VLA policies can follow visual priors or training-set regularities instead of language commands, making standalone execution interfaces unreliable when planners need stable, instruction-faithful behavior.The paper addresses this with subtask-centric data, invocation-aligned training, and tool-family residual adapters.Source: §2 · p. 3
  • Hierarchical embodied systems decompose goals into skills, while systems such as AtomicVLA and RoboClaw use learned skill libraries or policy primitives for long-horizon tasks.These low-level skills are often system-internal primitives rather than explicit agent-visible tool invocations.Source: §2 · p. 3
  • The paper exposes each VLA as a reusable embodied tool whose agent-visible invocation pairs a bounded language instruction with a tool-family label.This contrasts with low-level skills that remain internal to the system.Source: §2 · p. 3

3 Formalizing VLAs as Callable Tools

The formalization treats VLAs as bounded callable executors in an agentic control loop. The agent sends tool-family selections and scene-grounded instructions, tools return execution feedback, and the agent updates its state.

3 Formalizing VLAs as Callable Tools

  • A tool invocation combines a discrete tool-family label with a scene-grounded subtask instruction, selecting the VLA tool and specifying the object, relation, and desired local effect.The interface’s invocation-message set is C = G × Z.Source: §3 · p. 3
  • The selected tool executes the invocation over a bounded low-level horizon, producing a trajectory from robot observations and the instruction.The tool may use a short execution history.Source: §3 · p. 3
  • Tools return progress or completion feedback, which the agent uses with the call and trajectory to update its state and repeat the loop.The interface defines both agent-to-tool invocations and tool-to-agent feedback for monitoring.Source: §3 · p. 3, Figure 2 · p. 4

4 Methods

The methods make VLAs bounded, callable executors in an agent–tool loop and train them around the same invocation unit used at inference.

4 Methods

  • The system combines a bidirectional VLA tool-family interface with TAPT, aligning inference calls and training around bounded invocations.The interface carries agent calls and tool feedback; TAPT trains tools to execute the requested behavior and return invocation progress.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 5

4.1 Bidirectional Agent–Tool Interaction via a VLA Tool-Family Interface

  • The interface defines agent-to-tool invocation messages and tool-to-agent feedback messages for monitoring execution.The high-level agent tracks state across observations, past calls, and returned feedback.Source: §4.1 · p. 4

4.1.1 Invocation Messages C

  • Each invocation pairs a tool-family label that selects an executor with a scene-grounded instruction specifying the object, relation, and desired local effect.The data-driven taxonomy includes 11 families on DROID and 7 on each of LIBERO and RoboTwin.Source: §4.1.1 · p. 4

4.1.2 Feedback Messages R

  • A continuous 0–1 progress signal reports completion of the current bounded call, giving the agent intermediate feedback to decide whether to continue, advance, or replan.The signal is local to the invocation, not a measure of full-task success; it exposes states such as stagnation or deviation without repeated VLM inspection.Source: §4.1.2 · p. 4

4.2 Improving VLA Tool Usability with Tool-Aligned Post-Training

  • TAPT reorganizes imitation and reinforcement-learning supervision around bounded invocations, routes calls through family-specific residual adapters, and trains execution and progress feedback.Downstream SFT and RL adapt and evaluate the same invocation-aligned tool family.Source: §4.2 · p. 5

4.2.1 Invocation-Aligned Training Units

  • Imitation demonstrations are segmented into bounded windows annotated with a tool family, local scene effect, and progress target; RL uses bounded rollouts with invocation-level completion predicates.RL rollouts begin in states where the agent should make that call, and IL and RL both target reliable execution of the current invocation.Source: §4.2.1 · p. 5

4.2.2 Tool-Family Residual Parameterization

  • A shared pretrained backbone uses a low-rank residual adapter selected deterministically by tool family, giving each family a distinct execution path with small parameter overhead.A new family can be added by training only its adapter while freezing the backbone and prior adapters.Source: §4.2.2 · p. 5

4.2.3 Tool-Aligned Post-Training Objective

  • TAPT combines action cloning for the selected residual path with progress prediction; its RL stage rewards completion of the bounded invocation, not the full task.The resulting tool family accepts a family label and instruction, executes the selected tool, and returns progress feedback.Source: §4.2.3 · p. 6

5 Experiment

The experiments test supervised adaptation, reinforcement learning, long-horizon benchmarks, real-world manipulation, invocation fidelity, component ablations, and failure recovery. Results report gains across these settings and examine how the method’s components contribute.

5.2 Can the VLAs-as-tools Strategy Improve Long-Horizon Embodied Performance?

  • The full configuration exceeds Standard RL by +3.8 points for OpenVLA-OFT and +11.2 for π0.5 on LIBERO-Long; CALVIN success rises from 76.0 to 78.8 with Invocation-aligned RL and to 82.3 with full TAPT.CALVIN uses its native subtask structure, rather than the paper’s split construction.Source: Figure 3 · p. 7, §5.2 · p. 7

5.5 Does TAPT Make VLA Execution More Faithful to Agent Invocations?

  • Full TAPT raises Faithful/Non-biased Rate by +34.6/+16.2 points for OpenVLA-OFT and +30.0/+15.0 points for π0.5.Faithful Rate measures completion of the invoked counterfactual goal; Non-biased Rate measures avoidance of the source-task bias.Source: Table 3 · p. 7, §5.5 · p. 9

5.2 Can the VLAs-as-tools Strategy Improve Long-Horizon Embodied Performance?

  • TAPT with the tool-family interface beats standalone SFT by up to +5.2 points on LIBERO-Long and +31.87 points on RoboTwin-SA.Direct planner deployment changes LIBERO-Long performance by −0.2, −11.8, and −12.0 points for OpenVLA, OpenVLA-OFT, and π0.5, respectively.Source: §5.2 · p. 7

5.6 Core Design Ablation

  • On LIBERO-Long, the combined TAPT and interface configuration achieves the best results, 95.6% for OpenVLA-OFT and 97.2% for π0.5.The interface alone reaches 80.2% and 80.4%, respectively; tool-family residual adapters recover a post-training drop for π0.5.Source: Table 4 · p. 8, §5.6 · p. 9

5.3 Do the Gains Extend to Modern Long-Horizon Benchmarks?

  • Under a uniform 2,500-step budget, VLAs-as-Tools improves RoboMemArena full-task SR by 3.85 points and partial progress by 9.17, and RoboLab-120 by 6.3 and 6.1 points, respectively.RoboMemArena tests memory-dependent multi-stage execution; RoboLab-120 tests visual, relational, and procedural generalization.Source: §5.3 · p. 8

5.4 Can VLAs-as-Tools Improve Real-World Long-Horizon Manipulation?

  • Across Table 5’s real-world trials, average success rises from 16.7% to 58.3%; Conditioned shows the clearest gain, improving from 5.0% to 65.0% as identical visual input occurs at stages requiring different actions.Table 5 reports real-world success rates over 20 trials per task.Source: Table 5 · p. 8, §5.4 · p. 8

5.7 Qualitative Analysis

  • The agent detects a failed grasp through progress feedback and re-invokes the tool, turning a dropped moka pot into a local correction rather than a cascading failure.The baseline continues its nominal trajectory from the invalid state and ultimately fails.Source: Figure 5 · p. 9, §5.7 · p. 9

5.8 Additional Results in the Appendix

  • Appendix analyses report few-shot adaptation gains up to +11.5 points, planner robustness across four VLMs, progress-triggered replanning efficiency, and adapter cost of +9% parameters.The appendix also reports tool-family extensibility.Source: §5.8 · p. 9

6 Conclusion

VLAs-as-Tools separates high-level planning, progress monitoring, and recovery from bounded robot-control execution, making VLA policies more composable for long-horizon tasks. Experiments report gains in task success and instruction alignment, while the real-world study remains limited in scope.

6 Conclusion

  • The framework places callable, specialized VLA executors under a planning-capable multimodal agent, separating high-level reasoning from bounded robot control.This separation is intended to make VLA policies more composable for long-horizon embodied tasks.Source: §6 · p. 9
  • Experiments in simulation and on a real Franka robot show gains in task success, instruction alignment, few-shot adaptation, and planner robustness.The reported gains cover these four outcomes across the experiments described.Source: §6 · p. 10
  • The real-world study covers one robot, one camera, and three tabletop tasks; primitives absent from pretraining and adaptation data still require new demonstrations.Broader settings remain future work.Source: §6 · p. 10

A Progress Supervision and Feedback

The section describes invocation-local progress feedback alongside a pipeline that segments and labels bounded tool invocations. It also reports that performance is not overly sensitive to taxonomy granularity on LIBERO-Long.

A Progress Supervision and Feedback

  • Each 2 Hz VLA inference predicts an action chunk and aligned progress feedback; a runtime monitor uses completion thresholds, smoothing, and progress gains to continue, advance, or trigger replanning.Progress is local to the active invocation and is a temporal completion proxy correlated with physical execution, not a perfectly calibrated semantic completion estimate.Source: A Progress Supervision and Feedback · p. 13

B Automatic Labeling Pipeline

  • The automatic labeling pipeline segments trajectories using gripper transitions and motion cues, then assigns operational tool-family labels using local video, global context, and family definitions.It lowers confidence or sends segments for manual review when local evidence conflicts with global context or a family definition; Figure 7 depicts clips, labels, and normalized progress annotations.Source: B Automatic Labeling Pipeline · p. 14, Figure 7 · p. 14

C.1 Taxonomy Granularity

  • A modest performance gap between the tool-family and coarser task-level taxonomies indicates performance is not overly sensitive to taxonomy granularity on LIBERO-Long with π0.5.The authors describe data-derived families as usable without benchmark-specific or exhaustive manual design, with motion/contact separation providing an inductive bias.Source: Table 7 · p. 15, §C.1 · p. 15

C Tool-Family Taxonomy and Extensibility

  • The data-driven taxonomy distinguishes free-space motion from physical-contact primitives and yields 11 tool families, with 7 used in each of LIBERO and RoboTwin.The taxonomy is derived by applying the labeling pipeline to broad-coverage datasets such as DROID.Source: C Tool-Family Taxonomy and Extensibility · p. 15

C.2 Incremental Tool-Family Expansion

The section distinguishes recombinations of registered primitives from genuinely new physical primitives, then evaluates adding new families with adapters while preserving previously learned skills.

C.2 Incremental Tool-Family Expansion

  • A task expressible as a sequence of registered families can be handled by composing existing tools with scene-grounded instructions.A primitive absent from both the pretrained VLA and adaptation data is not expected to execute reliably zero-shot.Source: §C.2 · pp. 15–16
  • After Stage 3, prior-skill retention is 86.4% for end-to-end fine-tuning versus 97.6% for ours, with 95.0% final LIBERO-Long success; new primitives require training data.Table 8 reports the incremental addition of new families in two stages without data replay.Source: Table 8 · p. 16, §C.2 · p. 16
  • When demonstrations for a new primitive are available, the method adds a family-specific adapter while freezing the shared backbone and prior adapters, avoiding data replay.The three-stage experiment adds pull and push families by training only their respective adapters.Source: §C.2 · p. 16

D Additional Details for Tool-Aligned Post-Training

The appendix details invocation-level RL training, completion predicates, reset-state refresh, and the shared-backbone residual adapter parameterization.

D Additional Details for Tool-Aligned Post-Training

  • Each invocation defines a bounded subtask MDP with a reset distribution drawn from eligible start states, and the RL stage optimizes expected completion using GRPO.The invocation reward is based on whether the bounded rollout reaches its state-based completion predicate.Source: D Additional Details for Tool-Aligned… · p. 16
  • Completion predicates reuse LIBERO’s native success predicates for benchmark-defined spatial relations and add lightweight state-based predicates for intermediate calls such as reaching or grasping.Table 9 summarizes the predicate choices.Source: D Additional Details for Tool-Aligned… · p. 16, Table 9 · p. 17
  • Refreshing reset-state pools from successful preceding invocations helps match downstream training states to those induced by the current composed policy and recovers full-task gains unavailable from stale pools alone.The initial pools can become mismatched after policy improvement, sometimes reducing full-task success.Source: D Additional Details for Tool-Aligned… · p. 17
  • Family-specific LoRA residuals adapt selected linear layers while keeping the pretrained VLA backbone shared across tool families.The added parameter count scales as O(|L|Kr(din + dout)) for selected layers L and K tool families.Source: D Additional Details for Tool-Aligned… · p. 17

E.1 Tool-Family Adapter Cost

Tool-family adapters keep specialization lightweight, and TAPT improves few-shot adaptation. Planner choice has little effect on LIBERO-Long success, while progress-triggered replanning sharply reduces runtime VLM requests.

E.1 Tool-Family Adapter Cost

  • Adapters add 9% parameters and 7% inference time to OpenVLA-OFT; adapted π0.5 uses 0.50× the OpenVLA-OFT parameter baseline despite higher latency.Table 11 reports normalized model size and inference time on LIBERO-Long.Source: Table 11 · p. 18, §E.1 · p. 18
  • TAPT improved LIBERO-Long few-shot adaptation by 11.5 points for OpenVLA-OFT at 10 shots and 6.4/5.6 points for π0.5 at 10/20 shots.The largest gain occurred in the most data-limited setting.Source: §E.1 · p. 18

E.2 Agent Planning Efficiency

  • Across four VLM planners, LIBERO-Long success varied by only 0.6 points with OpenVLA-OFT tool families fixed.Only the high-level VLM planner changed.Source: §E.1 · p. 18
  • 109.5/58.2/57.2 calls per episode fell to 1.988/0.304/0.228 for OpenVLA/OpenVLA-OFT/π0.5 under progress-triggered replanning; replan success rates were 47.1%/34.4%/61.4%.The call counts exclude initial plan construction.Source: §E.2 · p. 18, Table 14 · p. 19

F Real-World Evaluation Details

The real-world evaluation compares fine-tuned π0.5 and VLAs-as-Tools on three Franka tasks under matched data, compute, hardware, and initial configurations. Conditioned tests whether the system tracks task stage when the scene returns to a visually similar state.

Real-world evaluation setup

  • Both methods use a Franka Research 3, 30 demonstrations per task, 30K training steps on eight A100s, one RTX 4090 for inference, and the same 20 initial configurations.Evaluation uses 20 rollouts per task and judges success by the final physical state.Source: F Real-World Evaluation Details · p. 19

Simple

  • Simple requires placing red and yellow cubes in designated bowls, with both cubes remaining stably inside their targets.This task evaluates multi-stage decomposition and execution stability.Source: F Real-World Evaluation Details · p. 19

Long

  • Long comprises 24 sequential tool invocations to stack cubes, reposition bowls, and place cubes in them, testing extended planning, tool composition, and recovery.Success requires both bowls in designated positions and both cubes stably inside them.Source: F Real-World Evaluation Details · p. 19

Conditioned

  • Conditioned restores the initial layout before requiring the opposite stack; after the scene recurs, the baseline repeats the wrong stage while the agent selects the opposite-stack tool.The task tests whether execution tracks completed stages rather than relying on the repeated visual state.Source: F Real-World Evaluation Details · pp. 19–20, Figure 8 · p. 19

G Evaluation Protocol Details

The evaluation protocol uses benchmark-defined success predicates and, unless specified otherwise, matches held-out tasks, initial states, horizons, seeds, and predicates across methods. It also details task-specific trial counts and metrics for six simulated benchmarks.

General protocol and LIBERO-Long

  • Unless otherwise specified, methods within a benchmark share held-out tasks, initial states, rollout horizons, seeds, and success predicates; success is the fraction of episodes meeting the predicate before timeout.LIBERO-Long uses 50 trials per task from held-out initial states and the native task predicate.Source: G Evaluation Protocol Details · p. 20

RoboTwin-SA

  • RoboTwin-SA evaluates eight single-arm-compatible tasks selected for compatibility with the demonstration-splitting pipeline, using 20 trials per task for 160 planned rollouts per configuration.Tasks requiring bimanual coordination are excluded.Source: G Evaluation Protocol Details · p. 20

CALVIN

  • CALVIN uses CALVIN_D with an 80/20 training-test split and native subtask success checks; RL experiments also use native annotations for invocation-aligned resets and rewards.Source: G Evaluation Protocol Details · p. 20

RoboMemArena

  • RoboMemArena evaluates all 26 tasks with 10 seeded rollouts each and a 2,500-step horizon, reporting full-task success and mean stage completion over 260 episodes.Its trajectories average more than 1,000 environment steps.Source: G Evaluation Protocol Details · p. 20

LIBERO-IF

  • LIBERO-IF fixes the high-level plan and varies the VLA executor across 10 counterfactual tasks, reporting Faithful Rate and Non-biased Rate over a 100-step continuation window.The counterfactual goals modify requested outcomes while keeping scenes visually close to familiar source tasks.Source: G Evaluation Protocol Details · p. 20

RoboLab-120

  • RoboLab-120 evaluates 120 tasks across visual, relational, and procedural axes and three difficulty levels, using five seeds per task and a 2,500-step horizon for 600 episodes.It reports full-task success and mean fraction of required subtasks completed.Source: G Evaluation Protocol Details · p. 21

Related papers

Loading 2605.13119v2…