PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA^2: Empowering Vision-Language-Action Models with an Agentic Framework for Unseen Concept Manipulation

Han Zhao, Jiaxuan Zhang, Wenxuan Song, Pengxiang Ding, Donglin Wang

arXiv:2510.14902v1cs.RO

TL;DR

VLA models struggle with object concepts outside their training data, including unseen descriptions and textures. VLA2 combines an OpenVLA execution backbone with external retrieval and object-detection tools, and achieves the best overall average success rate across the custom environments, with its largest reported advantage on Hard. The authors note that the framework remains procedurally rigid and has not yet been tested in the real world.

  • Problem

    VLA models’ success rates drop on object concepts outside training data, including unseen descriptions and textures.

    Source: §I · p. 1

  • Method

    VLA2 combines task planning, web and memory retrieval, VLA-based execution, and verification to handle unseen concepts.

    Source: §I · p. 1

  • Results

    VLA2 attains the best overall average SR at 81.5%; on Hard, it reaches 76.2%, exceeding π0 by 16.2% and OpenVLA-OFT by 28.8%.

    Source: B. Main Results · p. 6

  • Takeaways & Limitations

    Experiments show that the module design improves generalization in grasping objects from unseen concept categories.

    Source: §V · p. 7

    The framework remains confined to relatively rigid procedural structures, and the authors have not conducted real-world experiments.

    Source: §V · p. 7

Abstract

from arXiv · show

Current vision-language-action (VLA) models, pre-trained on large-scale robotic data, exhibit strong multi-task capabilities and generalize well to variations in visual and language instructions for manipulation. However, their success rate drops significantly when faced with object concepts outside the training data, such as unseen object descriptions and textures in the dataset. To address this, we propose a novel agentic framework, VLA^2, which leverages OpenVLA as the execution backbone and effectively leverages external modules such as web retrieval and object detection to provide visual and textual knowledge about target objects to the VLA. This approach mitigates generalization failure when handling out-of-distribution objects. Based on the LIBERO simulation environment, we introduced novel objects and object descriptions to construct a new evaluation benchmark with three difficulty levels to test the effectiveness of our method. Our framework successfully outperformed the current state-of-the-art models on our designed hard-level generalization benchmark. Compared to the standalone OpenVLA baseline, VLA^2 achieves a 44.2% improvement in the success rate in the hard-level benchmark and an average improvement of 20.2% in all customized environments without any performance degradation on in-domain tasks. Project website: https://vla-2.github.io.

I. INTRODUCTION

VLA models struggle with entirely unseen object concepts, while training with web-scale multimodal data is resource-intensive and difficult to update. VLA2 adds external tools and memory to support unseen-concept manipulation, alongside a three-level LIBERO generalization benchmark.

I. INTRODUCTION

  • VLA models can handle some unseen instructions but fail on entirely unseen concepts and cannot transfer learned manipulation experience to those scenarios.Joint training with web-scale multimodal data is resource-intensive and makes updates for emerging concepts impractical.Source: §I · p. 1
  • VLA2 is proposed as a system-level framework that invokes diverse tools, including task planning, web search, and object detection, to extend VLA capabilities.The introduction situates VLA models as integrating visual perception, language understanding, and action execution through large-scale robotic-data training.Source: §I · p. 1
  • VLA2 integrates task planning, web and memory retrieval, VLA-based execution, and verification to address unseen-concept manipulation.The framework also fine-tunes OpenVLA to accept masked images as input conditions.Source: §I · pp. 1–2
  • The custom LIBERO benchmark tests object generalization at Easy, Medium, and Hard levels, from color variation to unseen object concepts.Medium tasks involve generalized target objects.Source: §I · p. 2

A. Vision-Language-Action Models

The paper positions external tools as a way to augment VLA execution systems rather than directly optimize the VLA for generalization. Its preliminary processing decomposes instructions and localizes relevant objects and locations for downstream modules.

A. Vision-Language-Action Models

  • VLA models integrate perception and decision-making end to end, but open-vocabulary manipulation and long-horizon execution remain generalization challenges.This work adds external modules to existing models rather than primarily optimizing the VLA itself.Source: A. Vision-Language-Action Models · p. 2

B. Embodied Agents

  • The framework combines web search, object detection, and other tools with observations and instructions to identify objects beyond a single VLA’s cognitive scope.Prior embodied-agent work incorporated planning, situational memory, and skill libraries.Source: B. Embodied Agents · p. 2

III. METHOD

  • The method has three parts: preliminary information processing, cognition and memory, and judgment and execution.The framework is evaluated in the LIBERO simulation environment.Source: §III · p. 2

1) Planner:

  • The planner decomposes compound natural-language instructions into structured, single-action subtasks with explicit objects and locations.A three-layer post-processing module extracts information, detects and regenerates errors, and falls back to task-specific parsing.Source: 1) Planner · p. 3

2) Vision Pre-processing:

  • MMGroundingDINO generates bounding boxes for the objects and locations specified for visual processing, though some boxes may be empty after recognition or post-processing failures.Such cases are passed to cognition and memory for further handling.Source: 2) Vision Pre-processing · p. 3
  • The vision detector is fine-tuned on 500 annotated LIBERO images drawn from four environments to recognize objects appearing in those environments.The images were augmented, and bounding boxes and object names were manually annotated.Source: 2) Vision Pre-processing · p. 3

B. Cognition & Memory

The cognition and memory pipeline adds web-based retrieval to the text–image processing system. In the vision component, web-derived descriptions help localize unfamiliar objects, which are segmented and represented with color-coded masks.

B. Cognition & Memory

  • Web-based information retrieval is integrated into the higher-level text–image pipeline to enhance the underlying VLA’s out-of-distribution performance.The section introduces separate logic for visual and linguistic information enhancement.Source: B. Cognition & Memory · p. 3

1) Vision:

  • For an object or location without a bounding box or auxiliary keywords, visual search retrieves web images, forms a collage, and asks a vision-language module for descriptive keywords.The resulting keywords, collage, and images are cached in vision memory; if re-detection still fails, no mask is applied to that item.Source: 1) Vision · p. 3
  • Given the first image, a web collage, and the current word, the vision-language module produces five keywords describing attributes such as color, shape, function, or size.Those keywords support re-detection and are stored for subsequent tasks.Source: 1) Vision · p. 3
  • SAM2.1-L turns validated boxes into pixel-accurate masks, which are packaged with vision memory and routed to language processing and the VOS pipeline.VOS uses term–color mappings to guide temporally consistent, color-coded masked image flows.Source: 1) Vision · pp. 3–4
  • The authors call the process “instant learning”: decomposing unfamiliar concepts into attributes and anchoring them to known ones using prior knowledge.They also state that explicit color–mask alignment improves visual–text overlap.Source: 1) Vision · p. 4

2) Language:

The language component aligns object terms with a controlled vocabulary and attempts replacements for terms outside it. Judgment and execution use a verifier and recovery procedure, while OpenVLA is fine-tuned for masked images and sequential subtask prompts.

2) Language:

  • The language component aligns object-related prompt tokens with a controlled vocabulary derived from training and fine-tuning.For vocabulary-absent terms, a replacement is generated when possible and recorded in text memory for reuse; otherwise, no substitution or memory is created.Source: 2) Language · p. 4
  • After substitution mappings are determined, target terms are replaced and the task list is repaired to address errors from long-chain information propagation.The text-processing step applies the mappings within the current task.Source: 2) Language · p. 4

C. Judgment & Execution

  • A fine-tuned Qwen2.5-VL-3B-Instruct verifier judges subtask completion and whether execution can proceed, while a recovery detector can trigger a temporary gripper-lift action when it flags an anomaly.After a fixed number of steps, the system resumes the interrupted subtask and restores its progress.Source: C. Judgment & Execution · p. 4
  • OpenVLA is fine-tuned on LIBERO videos augmented with transparent colored masks and temporally segmented, plan-based prompts specifying the current subtask and whole task.OOD-related memories are re-initialized before each validation run; the model uses a KnownList of object terms at inference.Source: C. Judgment & Execution · pp. 4–5

IV. EXPERIMENTS

Experiments compare VLA2 with VLA baselines on original LIBERO tasks and new Easy, Medium, and Hard object-generalization variants. VLA2 achieves the strongest reported custom-environment results, while its in-domain average is highest within the OpenVLA family.

IV. EXPERIMENTS

  • The experiments target zero-shot OOD adaptability, comparing original LIBERO results and testing custom OOD performance and module contributions.The new variants are based on LIBERO Spatial and Goal environments.Source: §IV · p. 5, A. Experimental Setup · p. 5

A. Experimental Setup

  • The custom LIBERO variants change object appearances only; each task runs 50 times, with overall and task-level success rates reported.Easy changes bowl color; Medium changes a bottle's appearance and replaces objects; Hard introduces Moutai and a patterned porcelain bowl.Source: A. Experimental Setup · p. 5
  • Comparisons include OpenVLA, OpenVLA-OFT, π0, π0-FAST, and Agentic Robot, all fine-tuned on the same LIBERO training dataset.Evaluation covers the four original simulation suites and three newly crafted OOD environments.Source: A. Experimental Setup · p. 5

B. Main Results

  • 80.1% is VLA2's highest Class 2 average SR, though its Object SR (86.2%) remains below Agentic Robot and finetuned OpenVLA.The authors attribute the Object result to perception errors that can leave the first subtask unresolved and prevent verifier progression.Source: Table I · p. 6, B. Main Results · p. 6
  • 81.5% overall average SR and 76.2% on Hard make VLA2 the custom-environment leader, exceeding π0 by 16.2% and OpenVLA-OFT by 28.8% on Hard.All methods' SR declines as OOD difficulty increases.Source: Table II · p. 6, B. Main Results · p. 6
  • On moutai–rack, VLA2 scores 72 versus π0's 44; on bowl–saucer, it scores 88 versus π0's 16.These task-level results illustrate robustness on large semantic shifts.Source: B. Main Results · p. 6

C. Ablation Study

The LIBERO-Hard ablations isolate contributions from masks, lexical replacement, and web retrieval, while the conclusion identifies procedural rigidity and absent real-world testing as limitations.

C. Ablation Study

  • Disabling transparent masks reduces average SR from 76.2 to 64.8 (−11.4), with the largest drops in interaction-heavy and cluttered scenes.The reported pattern links masks most strongly to localization around containers or occlusions and discrimination among similar instances; simple placements show little effect.Source: C. Ablation Study · p. 6, Table III · p. 7
  • Removing semantic substitution drops average SR from 76.2 to 51.2 (−25.0), the largest average degradation among the reported ablations.Failures are especially large when novel or compositional nouns must be grounded, while well-known nouns can show neutral or positive shifts.Source: C. Ablation Study · p. 7, Table III · p. 7
  • Removing web search and retrieved memory lowers average SR to 65.2 (−11.0), with disproportionate losses on novel-brand targets.Retrieval yields slight gains on some familiar scenes, but butter-bowl remains difficult across settings because the low-resolution target is visually ambiguous.Source: C. Ablation Study · p. 7, Table III · p. 7
  • Removing all modules and subtask augmentation reduces average SR to 26.2 (−50.0), with several hard tasks falling to zero.The task-list prompt format can increase OOD content after decomposition and cause the verifier to fail the first subtask repeatedly.Source: C. Ablation Study · p. 7, Table III · p. 7

V. CONCLUSIONS

  • VLA2 integrates task planning, web search, scene memory, and process verification, and experiments report improved generalization to unseen object concepts.The conclusion describes the framework as integrating arbitrary VLAs into an embodied agent system.Source: §V · p. 7
  • The framework remains procedurally rigid, and the authors have not yet evaluated it in real-world grasping.They identify greater autonomy, broader tool invocation, and open-world real-world grasping evaluations as future directions.Source: §V · p. 7

APPENDIX

The appendix traces how task and image inputs are transformed into aligned representations for downstream execution, then reports module-runtime behavior and added latency.

DETAILED DESCRIPTION OF THE PROJECT FRAMEWORK

  • The pipeline starts from the task prompt and first image, decomposes the task into subtasks, and identifies object and location terms for visual processing.For the example prompt, the planner separates picking up the porcelain bowl from placing it on the stove.Source: DETAILED DESCRIPTION OF THE PROJECT… · pp. 9–10
  • Object and location names plus the image produce bounding boxes, distinguishing known objects from unknown ones before subsequent Vision processing.For the running example, the stove is known while the blue-and-white porcelain bowl is unknown.Source: DETAILED DESCRIPTION OF THE PROJECT… · p. 10
  • Web-search images and visual processing support cognition for unknown objects; the resulting memory can make a previously unknown category identifiable and support masks.Masks are color-coded into object and location palettes; language processing uses an analogous framework with a JSON replace map.Source: DETAILED DESCRIPTION OF THE PROJECT… · p. 10
  • The final representation aligns mask overlays with the task list, while VOS retains mask memory to infer masks in subsequent frames.The task list is finalized after cognitive interpretation; the appendix says VOS follows the Cutie project’s design principles.Source: DETAILED DESCRIPTION OF THE PROJECT… · p. 10

COMPUTATIONAL EFFICIENCY ANALYSIS

  • Vision, vision preprocessing, Language, and VOS add an average 10.035 seconds per task over Agentic Robot across 50 validation runs.The authors characterize this as modest overhead for the OOD-to-ID conversion pipeline.Source: Table IV · p. 11, COMPUTATIONAL EFFICIENCY ANALYSIS · p. 11
  • Planner latency is roughly 20 seconds per task; Vision and Language first-time inference is comparable, but repeated invocations can reuse memory and run much faster.Recognition-centric modules take longer on custom environments due to unknown-object cognition and GLM memory generation.Source: COMPUTATIONAL EFFICIENCY ANALYSIS · pp. 11–12

LIBERO-HARD TASK EXPLANATION

The task explanation expands abbreviated LIBERO-Hard labels and notes that custom object names preserve the original LIBERO-LONG task schema.

LIBERO-HARD TASK EXPLANATION

  • The table expands abbreviated labels such as “drawer-bowl” and “moutai-rack” into their full human-readable task names.For example, “drawer-bowl” means opening the top drawer and putting the blue-white porcelain bowl inside.Source: LIBERO-HARD TASK EXPLANATION · p. 12
  • The custom tasks preserve the LIBERO-LONG schema and differ by substituting object terms with custom names.The passage gives “bowl” and “moutai” as examples of substituted terms.Source: LIBERO-HARD TASK EXPLANATION · p. 12

PLANNER PROMPT

The planner prompt turns a high-level task into a minimal sequence of essential commands for a VLA. It constrains command format and object annotations, while correction prompts address invalid planner output.

PLANNER PROMPT

  • When the planner output is invalid, the prompt provides corrective instructions to regenerate subtasks or object annotations in the required format.Unsuccessful situations are mainly attributed to unmatchable or unreadable model output.Source: PLANNER PROMPT · p. 13
  • Plans use numbered commands for significant sub-goals, with object and location names added in /( )/ after each command.The examples show pick-up and placement steps, with additional commands for actions such as opening or turning on a device.Source: PLANNER PROMPT · pp. 13–14
  • The /( )/ annotations exclude positional and ordering details, allowing color and shape descriptors while retaining those details in the subtask text.The instructions say to annotate the largest specific item, such as a cabinet rather than its drawer.Source: PLANNER PROMPT · p. 14
  • The allowed commands cover picking up, placing, opening, closing, and switching devices; low-level motions are assumed to be handled internally by the VLA.The prompt says to omit separate locate, move, grasp, lift, release, push, pull, rotate, and adjustment steps.Source: PLANNER PROMPT · p. 14
  • The planner is instructed to decompose a task into the minimal sequence of essential high-level commands needed to fulfill it.The commands are intended for direct execution by a capable VLA model.Source: PLANNER PROMPT · pp. 13, 15

VERIFIER PROMPT

The verifier prompt tailors completion checks to each subtask’s main verb. It asks for a yes-or-no judgment based only on the provided media.

VERIFIER PROMPT

  • The verifier distinguishes subtask situations according to the main verb and constructs a corresponding completion question.For a pick-up task, it checks whether the named object or anything else was grasped and lifted from a surface.Source: VERIFIER PROMPT · p. 15
  • For placement tasks, the check asks whether the object or anything else was placed at the specified location and the gripper is away.The answer is constrained to “Yes” or “No,” based only on the provided media.Source: VERIFIER PROMPT · p. 15
  • For turn-on, turn-off, open, and close tasks, the verifier checks whether the target reached the corresponding specified state.The prompt requires a yes-or-no answer based only on the provided media.Source: VERIFIER PROMPT · p. 15

GLM UNDERSTANDING (VISION) PROMPT

The vision prompt uses images to describe a queried object or person with five keywords. It emphasizes specific visual attributes that can help another model identify the same or a similar subject in a real-world image.

GLM UNDERSTANDING (VISION) PROMPT

  • When the same object appears across the combined images, the prompt prioritizes its physical characteristics as descriptors for matching similar subjects.It says the keywords should be specific and descriptive, and should help another model find the same or a similar subject in a real-world image.Source: GLM UNDERSTANDING (VISION) PROMPT · p. 16
  • If the model cannot locate the queried object’s bounding box in the current image, it is instructed to compare that image with the combined web image and describe the same thing in both.The prompt supplies the combined web image and current camera image as inputs.Source: GLM UNDERSTANDING (VISION) PROMPT · pp. 16–17
  • The vision prompt requests five specific keywords describing the queried person or object, returned as a JSON list.The query is defined as the object or location the model is meant to understand.Source: GLM UNDERSTANDING (VISION) PROMPT · pp. 15–16

GLM UNDERSTANDING (TEXT) PROMPT

The GLM text component assembles available visual, language, vocabulary, and external information into inference messages. Its prompt directs the model to map new object mentions to the known vocabulary or return NONE.

GLM UNDERSTANDING (TEXT) PROMPT

  • The message builder adapts which image evidence it sends according to the availability of a composite web image, keywords, detector crop, boxes, and scores.Cases include a crop or context image when composite imagery and keywords are absent, and separate turns for composite imagery, crop, and keywords when all are available.Source: GLM UNDERSTANDING (TEXT) PROMPT · pp. 17–18
  • The prompt instructs the model to normalize an open-world object mention to one label from the allowed vocabulary, or return NONE.The allowed vocabulary is supplied alongside the new object mention.Source: GLM UNDERSTANDING (TEXT) PROMPT · p. 17
  • When present, inference messages can include web or Wikipedia snippets fetched from the normalized prompt and keywords, with a limit of four snippets.The snippets are added as a separate user turn.Source: GLM UNDERSTANDING (TEXT) PROMPT · p. 18
  • The system prompt requires exactly one allowed label or NONE, with no explanation or additional text.The specified response format wraps the label in an answer tag.Source: GLM UNDERSTANDING (TEXT) PROMPT · p. 18

Related papers

Loading 2510.14902v1…