PaperlayerView exact revision on arXiv

Source-linked AI summary

DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA

Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa, Takato Horii

arXiv:2608.28108v1cs.ROcs.CV

TL;DR

VLAs can struggle to identify among similar objects from language alone, motivating a unified interface for language and deictic gestures. DeicticVLA canonicalizes LI, VLI, and VI into text prompts and masks for one pretrained VLA, and real-world tests show stronger VLI and VI performance under unseen conditions, including 100% versus 16.7% for unseen categories.

  • Problem

    Language-only manipulation instructions require detailed referring expressions for similar objects, which VLAs may not reliably use.

  • Method

    DeicticVLA canonicalizes LI, VLI, and VI into a common text-prompt-and-deictic-mask representation and evaluates prompting and training strategies.

  • Results

    In three real-world tasks, VLI and VI outperformed LI under unseen expressions, appearance changes, and novel objects; unseen categories achieved 100% success versus 16.7% for jointly trained LI.

  • Takeaways & Limitations

    Two-stage training improves deictic-mask use in unseen layouts, while retaining second-stage LI data mitigates language-following forgetting without reducing VLI and VI performance.

Abstract

from arXiv · show

Vision-Language-Action models (VLAs) allow users to specify manipulation tasks in natural language, but distinguishing a target or placement goal among objects of the same category or similar appearance requires detailed expressions that VLAs may not use reliably. We propose DeicticVLA, which canonicalizes Language Instruction (LI), Vision-Language Instruction (VLI), and Visual Instruction (VI) into a text prompt and deictic masks through text-prompt completion and deictic gesture grounding, enabling a single pretrained VLA to handle all three instruction modes. With a shared backbone, demonstrations, and matched training steps, we compare two RGB visual prompting methods, two separate-channel mask prompting methods, and three training strategies in simulation. Under two-stage training, the four prompting methods achieve high in-distribution success but differ in their ability to use deictic masks in unseen layouts. Across methods, training-strategy ablations show that two-stage training improves such use, while retaining second-stage LI data mitigates forgetting without reducing VLI and VI performance. In three real-world tasks, one policy supports all modes. VLI and VI outperform LI under unseen expressions, appearance changes, and novel objects. For unseen categories, both achieve 100% success, compared with 16.7% for jointly trained LI. These results demonstrate the unified three-mode interface and guide DeicticVLA design.

I. INTRODUCTION

DeicticVLA addresses ambiguous target specification by unifying language-only, language-plus-gesture, and gesture-only instructions in one pretrained VLA. The study compares prompting and training choices, then validates the unified interface in simulation and real-world manipulation.

  • Language alone can require detailed appearance, spatial, or ordinal descriptions that VLAs may not reliably use when similar objects coexist.
  • Deictic gestures complement language by directly indicating targets or regions in the shared perceptual space.
  • No established single policy previously handled LI, VLI, and VI together, including gesture-only Visual Instruction.
  • Real-world validation shows VLI and VI outperform LI under unseen expressions, visual changes, and unseen objects across three manipulation tasks.
  • DeicticVLA canonicalizes LI, VLI, and VI into a common text-prompt-and-deictic-mask representation for one pretrained VLA.
  • The study systematically compares RGB visual prompting, separate-channel mask prompting, and training strategies using shared backbones, demonstrations, and matched training steps.

C. Visual and Mask Prompting for Manipulation Policies

The paper situates DeicticVLA among RGB visual-prompting and separate-channel mask-prompting approaches, then defines a shared interface that supplies canonicalized instructions and masks to one VLA.

  • C. Visual and Mask Prompting for Manipulation Policies: Visual prompting overlays task-relevant markers or regions on RGB observations, while separate-channel prompting supplies masks independently and fuses them at the feature level.
  • C. Visual and Mask Prompting for Manipulation Policies: Prior studies have not systematically isolated prompting effects with the same pretrained VLA, demonstrations, and matched total training steps.
  • III. PROPOSED METHOD: DeicticVLA accepts LI, VLI, and VI, differing in whether users provide language, a deictic gesture, or both.
  • III. PROPOSED METHOD: Canonicalization transforms these user inputs into a text prompt and deictic masks before prompting the VLA.
  • A. Preliminaries: A standard VLA conditions an action chunk on RGB observations, robot state, and a text prompt.
  • A. Preliminaries: DeicticVLA adds deictic masks as conditioning without changing the base VLA’s output action space or training objective.
  • 1) User-Facing Instruction Modes:: Visual Instruction uses only a deictic gesture, enabling concise specification of frequently performed routine manipulation tasks but limiting it to simple tasks such as pick-and-place.

2) Instruction Canonicalization:

Instruction canonicalization standardizes each user mode into a text prompt and deictic masks, separating the shared representation from the prompting method used by the VLA.

  • Text-prompt completion supplies user language unchanged for LI and VLI, while VI receives the default prompt “follow visual instruction”.
  • The VI default prompt contains no task content, so deictic masks specify the manipulated object, placement goal, or spatial relation.
  • Users click task-relevant objects in the robot-view image, and SAM 2 grounds those clicks into deictic masks.
  • Generated masks are retained separately as the user-provided mask set for downstream prompting.
  • Mask-T identifies the object to grasp or move, Mask-G identifies its placement region, and Mask-R identifies a spatial-reference object.
  • The canonical representation is upstream of prompting and can route the same text-and-mask representation to either visual or separate mask prompts.

C. RGB Visual Prompting and Separate-Channel Mask Prompting

DeicticVLA presents deictic masks either by rendering them into RGB observations or by injecting separate mask embeddings into the visual pipeline. All methods share the same canonical text-and-mask representation, while fusion location distinguishes MP-Early from MP-Late.

  • All four prompting methods use the same canonical text prompts and deictic masks, differing only in how masks are presented to the VLA.
  • RGB visual prompting transforms observations using mask-dependent rendering functions, including VP-Fade and VP-BBox.VP-Fade fades non-mask and non-robot regions gray, whereas VP-BBox draws red mask bounding boxes.
  • Separate-channel mask prompting leaves the RGB observation unchanged and supplies masks through an additional mask-prompt fusion module.RGB images and mask prompts are independently divided into patches and projected into the visual feature dimension.
  • MP-Early adds mask embeddings before the visual encoder, whereas MP-Late adds them after feature extraction and before the LLM.
  • For LI, the zero mask produces zero mask embeddings, so both mask-prompting variants leave image embeddings unchanged.RGB prompting likewise uses the original RGB observation without a rendered prompt for LI.

D. Instruction-Mode Datasets and Two-Stage Training

The simulation constructs matched LI, VLI, and VI datasets from shared demonstrations and compares two-stage training with single-stage and NoLI ablations. It evaluates four prompting methods across LIBERO suites while controlling segmentation effects and training-step totals.

  • D. Instruction-Mode Datasets and Two-Stage Training: Each LIBERO demonstration yields LI, VLI, and VI variants, forming Dall from all three mode-specific datasets and DVLI+VI from the two deictic modes.
  • D. Instruction-Mode Datasets and Two-Stage Training: Two-stage training fine-tunes first on DLI and then on Dall, while ablations either train once on Dall or omit LI from the second stage.All conditions have matched cumulative training steps.
  • IV. SIMULATION EXPERIMENTS: The simulation compares prompting methods on in-distribution success, deictic-mask use in unseen layouts, training-stage effects, and possible language-instruction forgetting.
  • 1) Tasks and Dataset Construction:: Each task has 50 expert demonstrations; LI uses original text, VLI uses role-specific masks, and VI uses a fixed prompt with Mask-T+G.
  • 1) Tasks and Dataset Construction:: LIBERO-Object selects one target among six visually distinct objects, while LIBERO-Spatial selects one of two identical bowls using a reference-object relation.
  • 1) Tasks and Dataset Construction:: Table I reports shared training hyperparameters for the compared simulation conditions.
  • 1) Tasks and Dataset Construction:: LIBERO-Goal moves a specified object to a specified goal location, using six tasks that contain both a manipulated object and placement goal.
  • 1) Tasks and Dataset Construction:: Ground-truth simulator segmentation generates deictic masks, eliminating segmentation accuracy as a factor in prompting and training-strategy comparisons.

2) Zero-Shot Evaluation Suites:

The zero-shot evaluation tests deictic-mask use in perturbed object arrangements and spatial layouts, using four prompting methods and three matched-training configurations. Performance is summarized by task success and the success contribution of valid deictic masks.

  • 2) Zero-Shot Evaluation Suites:: LIBERO-Object-ZS progressively perturbs initial object arrangements so tasks cannot be identified from the initial observation alone.
  • 3) Compared Configurations and Evaluation Metrics:: The comparison includes VP-Fade, VP-BBox, MP-Early, and MP-Late under two-stage training, plus single-stage and NoLI ablations, for 12 conditions.
  • 3) Compared Configurations and Evaluation Metrics:: Task success rate is evaluated over 25 trials, while ΔSR measures the contribution of deictic masks by comparing valid-mask and no-deictic inputs.
  • 3) Compared Configurations and Evaluation Metrics:: For VI, Fig. 3 uses the fixed text prompt “follow visual instruction” with separate Mask-T and Mask-G inputs shown together.
  • 3) Compared Configurations and Evaluation Metrics:: Training fixes the cumulative number of steps at 60k across 2S, 2S-NoLI, and 1S conditions.
  • 1) Prompting Method Comparison under Two-Stage Training (Q1):: All prompting methods reach mean in-distribution SRs of 94.1–95.6%, with VP-BBox highest at 95.6%.The maximum difference among methods is 1.5 points.
  • 2) Zero-Shot Evaluation Suites:: On Object-ZS, VP-BBox reaches +29.2 points for VLI and +27.8 points for VI, while MP-Late reaches +29.2 and +25.2 points.
  • 2) Zero-Shot Evaluation Suites:: On Spatial-ZS, VP-BBox reaches +16.0 points for VLI and +11.1 points for VI, while the other methods remain in the single digits.

2) Training-Stage Ablation (Q2):

Two-stage training primarily improves deictic-mask use in unseen layouts rather than in-distribution success. Retaining LI data in the second stage preserves language following and generally maintains zero-shot performance for promising methods.

  • Training stages: Under 1S, Object-ZS deictic gains decrease for all methods, with VP-Fade and VP-BBox falling to nearly zero.MP-Late retains +7.2 points for VLI and +8.0 points for VI under 1S, the highest performance in that setting.
  • Training stages: 2S is effective for acquiring deictic-mask use in unseen layouts rather than improving in-distribution task success.Its effect depends on prompting method: RGB visual prompting depends strongly on 2S, whereas MP-Late retains some zero-shot performance under 1S.
  • Second-stage LI data: Under 2S-NoLI, VLI and VI retain 95.4–97.0% success, while LI falls to 63.6–72.1%.Excluding LI data therefore permits deictic-mask learning but selectively degrades language-instruction following.
  • Second-stage LI data: Excluding LI data from stage two provides no consistent zero-shot improvement and degrades methods that perform promisingly under 2S.VP-BBox’s Spatial-ZS improvement is substantially reduced, while Object-ZS effects vary by prompting method.
  • Second-stage LI data: Jointly training LI, VLI, and VI in stage two preserves language following without impairing deictic-mask use in VLI and VI.It also maintains the zero-shot performance of VP-BBox and MP-Late.

A. Experiment Setup

The real-world study uses three manipulation tasks to test unified LI, VLI, and VI operation across unseen expressions, workspace appearance changes, and novel objects. Conditions distinguish training-distribution tasks from increasingly novel instructions and object settings.

  • Tasks: The study evaluates PickBlock, PutBlock, and OrganizeToy as three real-world manipulation tasks.PickBlock and PutBlock test unseen instruction expressions, while OrganizeToy tests workspace-appearance changes and novel objects.
  • Tasks: PickBlock selects one specified block from six colored blocks, while PutBlock places a block into one of two to five visually identical bowls.LI uses color, spatial, or ordinal descriptions; VLI and VI use deictic masks for target or goal specification.
  • Tasks: OrganizeToy requires grasping a specified stuffed toy and placing it into a specified basket.The task uses category or super-category labels for LI and target-plus-goal masks for VLI and VI.
  • Experiment conditions: TL1 is in-distribution, TL2 contains unseen combinations of learned concepts, and TL3 requires extrapolation to unseen concepts.For PutBlock, TL2 reverses the training counting direction, while TL3 adds unseen ordinal positions and spatial expressions.
  • Experiment conditions: VC-Surface changes the workspace appearance with a tablecloth absent during training, while NO-Instance and NO-Category test novel object instances and categories.NO-Instance uses new instances from trained categories; NO-Category uses categories excluded from training.

4) Data Collection and Training:

The real-world policy is trained from small-scale demonstrations and evaluated with controlled initial scenes across multiple instruction modes and generalization conditions. Deictic modes outperform language instruction, especially for unseen expressions, while PickBlock exposes a multiview targeting limitation.

  • Data collection: Demonstrations were collected at 30 Hz, with 100 PickBlock, 110 PutBlock, and 150 OrganizeToy episodes.Each episode records camera images, deictic masks, proprioception, gripper state, and actions.
  • Training and policy selection: MP-Late was selected for real-world evaluation because it retained zero-shot instruction following relatively well and leaves RGB observations unchanged.This avoids potential visual interference from VP-BBox and contextual-information loss from VP-Fade.
  • Evaluation procedure: The evaluation GUI aligns live object poses with a captured reference image before each trial to reduce operator-dependent initial-condition variation.The evaluator manually adjusts positions and orientations using the overhead-camera overlay.
  • Results: VLI achieved 50.0–77.5% and VI 45.8–75.0% success across tasks, compared with 29.2–62.5% for LI.A single policy supporting all three modes was learned from 100–150 episodes per task.
  • Limitations: PickBlock had lower success across all instruction modes, with failures linked to grasping wrist-camera-large blocks instead of the instructed target.The authors motivate improved multiview fusion because overhead-only trials suffered inaccurate reaching.
  • Generalization: At unseen instruction levels, VLI and VI maintained high success while LI decreased substantially from TL1.For PickBlock TL3, VI achieved 93.8% versus 18.8% for LI; for PutBlock TL2, VLI and VI each achieved 86.7% versus 26.7% for LI baseline.

3) Robustness to Visual Perturbations (Q3):

Deictic masks preserved strong performance under appearance changes and novel objects, while language-only instruction degraded more substantially, especially for unseen categories. The three modes provide complementary target-specification options for different environments and tasks.

  • Appearance changes: 87.5% and 90.0% success for VLI and VI, respectively, substantially exceeded LI under the VC-Surface appearance change.LI (baseline) and LI achieved 65.0% and 60.0%.
  • Novel instances: 95.8% success for both VLI and VI exceeded LI (baseline) at 66.7% and LI at 62.5% on novel instances.VLI and VI with MP-Late maintained high success for novel instances in known categories.
  • Unseen categories: 100% success for both VLI and VI exceeded LI (baseline) at 33.3% and LI at 16.7% under unseen categories.The result was achieved by specifying manipulation targets and placement goals with deictic masks without category names in language.
  • Complementary modes: LI supports hands-free operation and abstract or quantified expressions that deictic masks alone cannot naturally express.Examples include speech-based language and expressions such as “all the red blocks” or “any vegetable.”
  • Complementary modes: VLI and VI compensate for LI failures under unseen spatial and ordinal expressions and novel objects, so the modes are not substitutes.Users can select among the modes according to the task, context, and environment.
  • Unified deployment: A single policy lets users use LI ordinarily and fall back to VLI or VI without switching the underlying policy.The same-task fallback is presented as one practical use of the unified interface.

B. Modularity and Extensibility of DeicticVLA

DeicticVLA separates instruction-mode choice from mask-prompting choice, supporting a modular interface across prompting methods and pretrained VLAs. Simulation and real-world results identify training benefits and practical evaluation boundaries for this modular design.

  • Modularity: LI, VLI, and VI share a text-prompt-and-mask representation, after which RGB or separate-channel mask prompting can be applied.The interface is therefore not tied to one prompting method or base VLA.
  • Prompting-method choice: No prompting method consistently dominated in simulation; the four methods performed similarly in-distribution, with VP-BBox best on Spatial-ZS and MP-Late retaining some Object-ZS ability under single-stage training.These results support selecting prompting methods according to generalization and operating conditions.
  • Real-world deployment: MP-Late was selected for the physical robot because it retained zero-shot instruction following across training strategies, preserved RGB observations, and used SAM 2 masks without extra rendering.Only MP-Late was evaluated in the real-world system.
  • Extensibility: DeicticVLA can extend to pretrained VLAs beyond π0 without changing the base action space or training objective.RGB prompting requires no base architecture modification, while separate-channel prompting adds a mask input pathway.
  • Evaluation boundaries: Ground-truth masks isolated prompting effects in simulation, whereas real-world masks were generated and tracked from clicks using SAM 2.Segmentation and tracking errors were not separately evaluated for their effects on policy performance.
  • Evaluation boundaries: Only MP-Late was evaluated on the physical robot, so RGB and separate-channel prompting were not compared in real-world tasks.The proposed structural benefits of separate-channel prompting were also not verified as performance improvements.
  • Human-centered evaluation: The study did not directly test whether deictic interaction reduces descriptive burden or cognitive load.The proposed validation requires user studies comparing completion time, instruction length, corrections, and subjective workload.
  • Future configurations: An upstream VLM could propose candidate manipulation regions for user confirmation before DeicticVLA generates actions.This configuration separates deciding what to manipulate from deciding how to act.

APPENDIX A EXPLORATORY EVALUATION OF MP-INTER-k

The appendix explores MP-Inter-3, which injects mask-prompt embeddings at multiple intermediate visual-encoder layers. In the tested configuration, it performed worse and appeared to hinder optimization, without ruling out all intermediate-layer fusion designs.

  • Configuration: MP-Inter-3 injects mask-prompt embeddings after self-attention and before the feed-forward network at selected intermediate visual-encoder layers.The evaluated layer set was K = {0, 9, 18} in a 27-layer visual encoder.
  • Configuration: For LI, MP-Inter-3 leaves image embeddings unchanged because Z_M = 0.The appendix evaluates MP-Inter-3 under 2S, 2S-NoLI, and 1S.
  • Results: MP-Inter-3 achieved substantially lower mean SR than the four main-text methods across all three training strategies.Its final training loss was three to seven times as high, and deictic masks contributed little or negatively.
  • Interpretation: The results suggest that repeated intermediate-layer injection hindered optimization in the tested setting.Because only K = {0, 9, 18} was evaluated, the result does not rule out intermediate-layer fusion generally.
Loading 2608.28108v1…