Source-linked AI summary
Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-use
Yi Ding, Yanzhao Yu, Xili Dai, Xianbiao Qi, Peiwen Sun, Xueqian Wang, Xiangyu Yue, Jianan Wang
TL;DR
VLA models struggle to adapt to novel scenarios without costly post-training. ART injects modular tools into VLA models, and experiments show improved robustness and generalizability across simulation and real-world settings.
Problem
End-to-end VLA models require costly post-training for novel scenarios and tasks, while fixed tool sets mainly support simpler action outputs.
Method
ART fine-tunes VLA models to use multimodal tools for visual, affordance, and embodiment enhancement while preserving their action-generation capabilities.
Results
Experiments in LIBERO simulation and real-world settings show improved robustness and generalizability to new scenarios, objects, and action instructions.
Takeaways & Limitations
ART supports more adaptable VLA systems by integrating external tools into existing datasets and reducing the need for new data collection.
Takeaways & Limitations
ART assumes accurate embodied actions can depend on enhanced observations rather than noise and corruptions in raw observations.
Abstract
from arXiv · showhide
This paper integrates end-to-end Visual-Language-Action (VLA) models with agentic tool-use to propose Agentic Robot with Tool-use (ART). ART is a tool-injection framework that tunes any VLA model to leverage off-the-shelf tool modules for low-level vision, high-level affordance, and embodiment enhancement. Compared to vanilla VLA models with a whole continuous action solution space, ART reduces the complexity of the action solution space through tool-use, which not only improves generalizability across different tasks but also reduces data dependency. To demonstrate the advantages (high generalizability and low data dependency) of this framework, we first built a dataset of 30K tool-use trajectories and action demonstrations, which is much smaller than those used by baseline methods. We then designed a training regimen for long-trajectory tool-use reasoning in challenging environments. Experiments show that ART achieves a 20% higher success rate than mainstream baselines on simulation and real-world tasks, such as pick-and-place in the dark at novel viewpoints. Empirical results highlight the benefits of an agent-based approach: modular tool utilization enables more efficient training, lightweight deployment, and scalable integration of new tools. This design fosters robustness, adaptability, and extensibility, paving the way for the practical deployment of VLA systems in complex real-world scenarios.
1. Introduction
The introduction contrasts modular and end-to-end VLA models, then presents ART as a tool-injection framework that adds plug-and-play capabilities while constraining action generation through discrete tools. It also describes a 30K-trajectory data-generation and training strategy designed to support long-horizon tool use while preserving core VLA action generation.
- VLA paradigms: Modular VLA methods call fixed tool APIs, whereas end-to-end models rely on a single base model trained on extensive multimodal data.The modular approach decouples capabilities but limits applicability, while end-to-end training uses a single model.
- ART framework: ART injects multimodal tool use into VLA fine-tuning, enabling plug-and-play visual enhancement, spatial reasoning, and expanded action capabilities.The framework replaces continuous actions with predefined discrete tools to constrain the action solution space.
- Data generation: 30K tool-use trajectories and action demonstrations form a dataset much smaller than those used by baseline methods.The dataset is intended to demonstrate ART’s generalizability and low data dependency.
- Data generation: A three-step pipeline—task design, reasoning generation, and tool-trajectory synthesis—extends existing VLA datasets with long-chain tool-usage data.The method degrades existing VLA data by adding task complexities that explicitly necessitate tool use.
- Training strategy: Masking LoRA outputs during action generation separates tool reasoning from the core VLA model, preventing data conflicts and preserving action-generation quality.After tool use, the model can revert to its original action-output capabilities.
2. Related Works
Related work follows two main VLA paradigms—modular behavior synthesis and end-to-end action prediction—alongside multimodal agents that sequence external tools. Modularization improves flexibility, while end-to-end and affordance-guided methods face data, observation, and robustness limitations.
- Modular Robotic Behavior Synthesis: Modular robotic behavior synthesis decomposes action generation into subtasks by selecting external modules that progressively address the task.This approach bridges high-level human instructions and robotic behaviors through modular action generation.
- Modular Robotic Behavior Synthesis: Affordance modules return intermediate representations such as object positions and segmentation masks, while embodied functions translate them into robot movements.CaP sequences pick obj and place at pos to complete pick-and-place tasks.
- Vision-Language-Action Model: End-to-end VLA models map language, visual data, and proprioceptive states directly to continuous action chunks for generalist robot policies.The action chunk is defined as At = [at, at+1, · · ·, at+N], with at ∈ R^d for a robot with d degrees of freedom.
- Vision-Language-Action Model: Affordance-guided VLA methods predict intermediate signals such as object bounding boxes or manipulation keypoints to guide subsequent action generation.These methods aim to improve planning ability and action accuracy.
- Vision-Language-Action Model: Affordance-guided approaches require large-scale data with handcrafted annotations and high-quality observations, while visual perturbations can significantly degrade performance.The limitations are especially relevant in realistic robotic scenarios.
- Multi-modal Agents with Tool-use: Multimodal agents use language models to generate pseudocode, scripts, or API lists that invoke external tools sequentially for complex tasks.External tools enhance generalist reasoning in applications including web search and image editing.
3. ART: Agentic Robot with Tool-use
ART augments VLA models with language reasoning and external tool-use actions, separating tool-assisted observation enhancement from embodied action generation. Its architecture and training procedure preserve the backbone while enabling discrete tool decisions and autoregressive action prediction.
- Optimization Objective: ART expands the action space to include embodied actions, language reasoning, and tool-use actions that activate external tools to enhance observations.The framework represents task solutions as trajectories of augmented actions.
- System Architecture: The architecture uses a fine-tuned LoRA module to reason over raw observations, activate vision, affordance, and embodiment tools, and generate final embodied actions from enhanced observations.LoRA-based tool injection provides the mechanism for integrating external tools into the VLA pipeline.
- Optimization Objective: ART assumes accurate embodied actions should depend on enhanced observations rather than noise and corruption in raw observations.This factorization decomposes training into vanilla VLA learning and a separate reasoning-and-tool-use objective.
- System Architecture: To avoid backbone degradation and catastrophic forgetting, ART freezes the pretrained VLA backbone and dynamically activates LoRA for tool-use reasoning while the backbone generates embodied actions without LoRA.Tool activations are represented as discrete tokens corresponding to binary tool states.
- Training Procedure: Training optimizes reasoning-token prediction from raw observations and autoregressive embodied-action-token prediction from enhanced observations using the pretrained 3B π-FAST model.FAST discretizes continuous actions with a discrete cosine transform, while the embodied-action formulation uses the VLA parameters without LoRA modules.
4. Data Collection
ART extends existing VLA datasets with long-trajectory tool-use reasoning to address the high cost of robotic data collection. The dataset defines visual, affordance, and embodiment tools and generates trajectories that connect environmental challenges, tool chains, and step-by-step robot decisions.
- Dataset Contribution: ART extends existing VLA datasets with reasoning and tool use, introducing a VLA dataset designed for long-trajectory tool-use reasoning.The authors describe this as the first VLA dataset with long-trajectory tool-use reasoning.
- Tool Definition: Tools modify or enhance observations across visual, affordance, and embodiment modalities corresponding to visual data, language instructions, and proprioceptive robot states.The visual category includes 10 enhancement types, such as low-light enhancement, denoising, jitter correction, and deblurring.
- Long-Trajectory Generation: Task generation creates challenges requiring environmental reasoning and tool selection across visual degradation, affordance identification, and embodiment changes.Examples include simulated noise or lighting transitions, object or property identification, viewpoint shifts, and altered initial states.
- Long-Trajectory Generation: Tool-chain generation pairs each generated task with the sequence of tools required to resolve its disturbances and complete the task.A task may require a visual enhancement tool followed by an embodiment adjustment tool.
- Long-Trajectory Generation: Trajectory generation prompts a model such as GPT to produce a consolidated, step-by-step reasoning process explaining each tool’s purpose and guiding every action and decision.The task-tool pair is converted into a long trajectory that guides the robot through completion of the task.
5. Experiment
ART is evaluated for tool-use reasoning across simulated, closed-loop real-world, and open-loop real-world settings. Results indicate that on-the-fly tools improve robustness to disturbances, generalization across environments, and performance relative to conventional VLA approaches.
- Experimental setup: ART is evaluated in three benchmark settings: simulation, closed-loop real-world interaction, and open-loop real-world testing.The experiments assess external-tool utilization across varied environments and tasks.
- Benchmark environments: LIBERO introduces lighting, noise, instruction-precision, camera-position, and robot-body disturbances to test performance in complex simulated scenarios.These controlled modifications compare ART with current state-of-the-art baselines under visual, semantic, and embodiment variability.
- On-the-fly tool use: ART integrates external tools on-the-fly, addressing disturbances that hinder traditional VLA models and improving generalization to complex, dynamic tasks.ART-FAST outperforms other models in both LIBERO and Astribot S1 tasks, according to Table 1.
- Generalization: ART generalizes tool-use reasoning across LIBERO simulation and real-world robotic tasks, using fine-tuning data from LIBERO, Bridge v2, and DROID.This cross-dataset training enables tools to be applied across multiple environments beyond the original training domains.
- Comparisons with reasoning models: ART performs slightly better than ECoT without vision corruption, while ECoT degrades significantly when visual disturbances are introduced.The comparison tests success rates on affordance tasks with and without vision corruptions.
6. Conclusion
ART is a fine-tuning framework that bridges modular and end-to-end VLA models by enabling external-tool use for novel tasks while preserving precision and continuous action capabilities. It integrates multimodal tool use into existing datasets and uses two-stage LoRA fine-tuning to separate tool reasoning from core VLA tasks.
- Framework: ART bridges modular and end-to-end VLA models through fine-tuning.The framework is named Agentic Robot with Tool-use.
- Tool-use: ART enables VLA models to leverage external tools for novel tasks without compromising precision or continuous action capabilities.
- Data efficiency: ART integrates multimodal tool use into existing VLA datasets, reducing the need for new data collection.
- Fine-tuning: A two-stage LoRA fine-tuning process decouples tool reasoning from core VLA tasks.