PaperlayerView exact revision on arXiv

Source-linked AI summary

LexiconVLA: Learning Reusable Atomic Action Codebooks for Unseen Tasks

Zeming Wei, Jianheng Ye, Xinshuai Song, Sirui Chen, Yang Liu, Liang Lin

arXiv:2609.36774v1cs.RO

TL;DR

Reliable completion of manipulation tasks does not ensure that their constituent atomic actions can be executed consistently or reused in new task contexts. LexiconVLA learns retrievable action codes and composes code-conditioned subtasks for a shared policy. Across five backbones, it improves unseen-task success while largely preserving seen-task performance.

  • Problem

    Task-level success can mask inconsistent execution of constituent atomic actions, while shared manipulation primitives need to be organized for reuse across unseen tasks.

    Source: §1 · p. 2

  • Method

    LexiconVLA learns global and detail codebooks grounded in trajectories and visual effects, then uses a planner and scene-aware adapter to retrieve codes for subtasks and condition a shared policy.

    Source: §1 · pp. 2–3

  • Results

    Across five policy backbones, LexiconVLA improves unseen-task success while largely preserving seen-task performance; with BridgeVLA, unseen-task success rises from 16.67% to 34.17% and overall success from 67.28% to 71.08%.

    Source: §1 · p. 3, §5.1 · p. 7

  • Takeaways & Limitations

    The authors report that the learned lexicon supports cross-task reuse without skill-specific experts or deployment-time parameter updates, with real-robot rollouts demonstrating stepwise execution and recovery from errors.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

Vision-language-action (VLA) models struggle to reuse recurring interactions in unseen tasks. Our diagnostic study reveals that reliable task completion does not imply consistent execution of constituent atomic actions across task contexts. We present LexiconVLA, a retrievable atomic-action lexicon for cross-task reuse. Global and detail codebooks capture shared interaction structure and fine-grained execution variation, respectively, preserving both reusable patterns and execution details. Visual-Atomic Action Alignment couples trajectory reconstruction from visual state changes with visual outcome prediction from action codes, grounding the lexicon in motion and its effects. We learn these codebooks with trajectory reconstruction and visual alignment on our AtomAction Dataset of 57,803 segments from 69 tasks. A planner and scene-aware adapter translate new goals into code-conditioned subtasks for a shared policy, without skill-specific experts or deployment-time parameter updates. Across five policy backbones on 26 RLBench tasks, LexiconVLA largely maintains performance on 18 seen tasks while improving success on 8 tasks held out from policy training. With BridgeVLA, unseen-task success rises from 16.67% to 34.17% (+17.50 percentage points), and overall success reaches 71.08%, the highest among methods with reported results. Real-robot experiments demonstrate stepwise execution and failure recovery.

1 INTRODUCTION

Task-level success can mask unreliable execution of constituent atomic actions, limiting reuse across unseen tasks. LexiconVLA learns retrievable action codes and uses them to compose subtasks, improving unseen-task performance while largely preserving seen-task performance.

1 INTRODUCTION

  • Full-task success does not guarantee reliable atomic execution: task-averaged success falls from 65.7–95.7% on complete tasks to 28.5–38.9% on isolated actions.The diagnostic probes five pretrained policies across seven tasks and finds that even the approach primitive varies across task contexts.Source: §1 · p. 2
  • LexiconVLA quantizes interaction-phase trajectories into shared action codes, which a scene-aware adapter retrieves to condition a shared policy while a planner sequences subtasks.The approach avoids skill-specific execution experts and deployment-time parameter updates.Source: §1 · pp. 2–3
  • Global and detail codebooks capture shared interaction structure and execution detail, while Visual-Atomic Action Alignment couples trajectory reconstruction with prediction of visual effects.The objectives ground the lexicon in motion and its consequences.Source: §1 · p. 2
  • AtomAction provides 57,803 annotated atomic-action segments across 69 manipulation tasks, pairing segmented trajectories with boundary observations and textual descriptions.The dataset is motivated by the gap between full-task completion and reliable standalone execution of constituent actions.Source: §1 · p. 2
  • Across five policy backbones, LexiconVLA improves unseen-task success while largely preserving seen-task performance; with BridgeVLA, unseen-task success rises from 16.67% to 34.17% and overall success from 67.28% to 71.08%.Real-robot rollouts demonstrate sequential atomic-action execution and recovery from execution errors.Source: §1 · p. 3

2 RELATED WORKS

Prior VLA approaches generally leave behaviors shared across tasks implicit, while action representations encode motions, frame windows, or specialized skills. LexiconVLA instead represents interaction phases with codes a shared policy can retrieve.

2.1 VISION-LANGUAGE-ACTION MODELS

  • VLA generalization relies largely on pretrained backbones and training data, while task-level instruction-to-motion learning leaves shared behaviors implicit and difficult to invoke independently.LexiconVLA makes these behaviors explicit as retrievable atomic-action codes that condition a shared policy.Source: §2.1 · p. 3

2.2 LEARNING REUSABLE ACTION REPRESENTATIONS

  • Action tokenizers encode motions or fixed-length segments, latent-action models encode visual changes between frames, and skill methods use longer horizons but often rely on specialized experts or goal-level labels.LexiconVLA represents interaction phases, jointly supervised by trajectory reconstruction and observed state changes, to distinguish mechanisms that share a goal label.Source: §2.2 · p. 3

3 ATOMACTION DATASET

AtomAction addresses the lack of segment-level supervision in manipulation corpora by collecting temporally localized segments that each realize one mechanism-level atomic action. Its samples retain varied observations and language descriptions.

3 ATOMACTION DATASET

  • AtomAction comprises 57,803 segments from 69 tasks, with each sample realizing exactly one mechanism-level atomic action.Existing corpora often pair a complete demonstration with one instruction, while coarse labels such as “open” can combine mechanically distinct motions.Source: §3 · p. 3
  • Figure 2 shows atomic-action categories and contributing tasks, segment counts by category and task, and an example segment with boundary frames and language descriptions.Colored block widths show individual task contributions to each category’s segment count.Source: Figure 2 · p. 4
  • Each sample retains a low-dimensional state trajectory, five-view RGB, depth and mask observations at segment boundaries, an action label, and eight phase-level language descriptions.The examples include the flip-open phase of an open-box task.Source: §3 · p. 4

4 METHOD

LexiconVLA combines a frozen atomic-action lexicon with a planner and scene-aware adapter to condition a shared policy. Its codebooks are learned through trajectory reconstruction and visual alignment to represent motion and its observed effects.

4.2 LEXICONVLA

  • A VLM planner decomposes goals into atomic subtasks, while a scene-aware adapter retrieves frozen global and detail codes to condition a shared VLA policy.Execution feedback selects NEXT, CONTINUE, RETRY, or REPLAN; retrieved codes refresh at subtask transitions and replanning.Source: Figure 3 · p. 4, §4.2 · pp. 4–5

4.3 VECTOR-QUANTIZED ATOMIC ACTION MODEL

  • The VQAA combines trajectory reconstruction with two visual objectives to ground atomic-action codes in motion and observed state changes.The visual objectives reconstruct actions from paired observations and predict final-frame features conditioned on the initial observation and global code.Source: §4.1 · p. 4, Figure 4 · p. 5, §4.3.2 · pp. 6–7

4.3.1 ATOMIC ACTION-NSVQ

  • Trajectory encoding supplements end-effector pose with joint and gripper measurements to supervise codebooks with richer motion and contact information.The segment is represented as a sequence of 39-dimensional measurements.Source: §4.3.1 · p. 5
  • Atomic Action-NSVQ encodes each trajectory as one global and multiple detail codes, using noise-substituted quantization and conditional flow-matching reconstruction for codebook learning.The decoder is used only during codebook training.Source: §4.3.1 · p. 6

4.3.2 VISUAL-ATOMIC ACTION ALIGNMENT

  • V3A reconstructs actions from visual changes between initial and final observations and predicts final-frame features from the initial observation and global code.The visual-change encoder combines cross-attention with projected feature differences; its decoder reconstructs the action segment.Source: §4.3.2 · pp. 6–7

5 EXPERIMENTS

Across simulation and real-robot experiments, LexiconVLA improves transfer to policy-unseen tasks while largely preserving seen-task performance. Ablations and real-world trials examine the components and settings behind these results.

5.1 SIMULATION EXPERIMENTS

  • The simulation evaluation uses 18 seen and eight policy-unseen RLBench tasks, with policies trained on seen-task demonstrations and no deployment-time parameter updates.Real-robot evaluation compares methods fine-tuned on the same 30 demonstrations per task and includes perturbations absent from those demonstrations.Source: §5.1 · p. 7, §5.3 · pp. 8–9
  • With BridgeVLA, unseen-task success rises from 16.67% to 34.17% (+17.50 points) and overall success from 67.28% to 71.08%, the highest among evaluated methods.Across five backbones, LexiconVLA improves unseen-task and overall success while largely preserving seen-task performance.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2 ABLATION AND ANALYSIS

  • Removing the planner, codebooks, global codebook, detail codebook, or V3A lowers unseen-task success, while seen-task success changes little.Without the planner, success falls from 34.17% to 28.50%; without codebooks it falls to 26.00%, and removing the global codebook, detail codebook, or V3A yields 26.50%, 29.50%, and 26.50%, respectively.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 REAL-WORLD EXPERIMENTS

  • On real robots, LexiconVLA averages 62.0% success in the Standard setting versus 26.0% for BridgeVLA and 28.0% for π0.5, and reaches 56.0% under unseen perturbations.BridgeVLA and π0.5 reach 16.0% and 12.0% under perturbations; LexiconVLA is best or tied-best across all ten conditions.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

LexiconVLA makes recurring manipulation interactions explicit and reusable across tasks. Its codebooks and subtask-conditioned execution improve unseen-task performance in simulation and remain effective on physical-robot tasks and unseen conditions.

6 CONCLUSION

  • LexiconVLA combines global and detail codebooks grounded in motion and visual effects with a planner and scene-aware adapter that condition a shared policy without deployment-time updates.Across five RLBench policy backbones, it improves unseen-task success while largely preserving seen-task performance; the simulation-learned lexicon also supports physical-robot execution and failure recovery.Source: §6 · p. 10

A.1 DIAGNOSTIC PROBE OF ATOMIC-ACTION INVOCATION

The diagnostic probe compares complete-task success with isolated execution of constituent atomic actions. It uses controlled handoffs and strict endpoint-based scoring to assess the isolated segments.

A.1 DIAGNOSTIC PROBE OF ATOMIC-ACTION INVOCATION

  • The probe covers five pretrained policies across seven tasks, using 70 episodes, 290 segments, and 13 atomic labels.Ten validation demonstrations per task are selected with complete keyframe coverage.Source: §A.1 · p. 14
  • For isolated atoms, the protocol replays each demonstration's ground-truth prefix and excludes handoffs outside 2 cm position or 5° orientation error, or with mismatched gripper state.Segments are evaluated independently from verified handoffs.Source: §A.1 · p. 14
  • Table 4 reports each task's full-task success rate and pooled valid-segment atomic success rate.Full-task success uses RLBench success detection.Source: Table 4 · p. 14

A.2 ADAPTER ARCHITECTURE AND TRAINING

The adapter predicts global and detail code indices from visual observations and a subtask instruction. At deployment, it retrieves the corresponding frozen code vectors and refreshes them at subtask transitions, retries, or replanning.

A.2 ADAPTER ARCHITECTURE AND TRAINING

  • Frozen DINOv2 and CLIP encoders tokenize front- and wrist-camera images and the instruction before a four-layer Transformer processes a 215-token sequence.The sequence includes 64 visual tokens per camera, 77 text positions, one global query, and nine detail queries.Source: §A.2 · p. 14
  • The adapter produces global and detail code logits using codebooks of 36 global entries and 192 detail entries.The two-layer GELU heads use a shared detail head across detail slots; argmax selects the predicted indices.Source: §A.2 · p. 14
  • Training uses nearest-neighbor code indices from demonstration trajectories as fixed labels, which the adapter predicts from initial observations and instructions.The adapter trains for 100 epochs with AdamW, batch size 64, learning rate 3×10^-4, and label smoothing; encoders and codebooks remain frozen.Source: §A.2 · pp. 14–15
  • At deployment, the frozen adapter retrieves code vectors by predicted indices; CONTINUE retains codes, while transitions, retries, and replanning refresh them from current observations.Source: §A.2 · p. 15

A.3 ADDITIONAL ANALYSES

The ablations identify a codebook-size trade-off and show that planner capability tracks unseen-task success. The learned global and detail codes exhibit distinct structure and retain atomic-action information across held-out tasks.

Codebook size

  • 34.17% unseen-task success at the default size; doubling the codebooks drops it to 24.50%, while ×1/4 and ×1/2 reach 32.00% and 28.50%.Seen-task success varies by less than 3 points across sizes; all entries remain active, while overly small codebooks must merge distinct actions.Source: Figure 6 · p. 15, §A.3 · p. 15

Choice of planner

  • Unseen-task success rises from 27.50% with gpt-5.6-luna to 35.50% with gemini-3.8-flash as planner MMMU-Pro scores rise; seen-task success stays between 86.67% and 88.89%.Even the weakest planner improves unseen-task success over BridgeVLA from 16.67% to 27.50%.Source: §A.3 · p. 15, Figure 7 · p. 16

Learned codebook structure

  • All 36 global and 192 detail codewords are active; global features cluster around recurring patterns, while detail codes cover finer execution variation.On tasks held out from probe training, global and detail codes predict annotated atomic actions at 41.38% and 31.08% accuracy, versus 23.58% for the majority class.Source: §A.3 · pp. 15–16, Figure 8 · p. 16

A.4.1 CLOSED-LOOP PLANNING AND EXECUTION

The online procedure plans atomic subtasks, retrieves codes for the active subtask, and conditions policy execution on those codes and current observations. Visual monitoring can schedule progression or recovery decisions within the closed loop.

A.4.1 CLOSED-LOOP PLANNING AND EXECUTION

  • PLAN creates atomic subtask instructions from the current images and task instruction; the adapter retrieves and caches codes when the active subtask key changes, and the VLA executes with observations, instructions, and code context.The local state machine tracks the plan, active index, completion records, execution counters, and cached lexicon entries.Source: §A.4.1 · pp. 16–17
  • After keyframe actions, event or heartbeat triggers can invoke MONITOR with images, plan and execution state; it returns a scheduling decision, optional target index, and rationale.Selective VLM assessment occurs while the policy continues to respond to observations.Source: §A.4.1 · pp. 16–17
  • NEXT records completion and advances, CONTINUE retains the active subtask and cached codes, RETRY retries a step, and REPLAN replaces the plan and resets progress records.Recovery budgets and terminal plan-extension rules also govern the state-machine procedure.Source: §A.4.1 · p. 17

A.4.2 MONITORING AND RECOVERY RULES

This appendix defines the monitoring, preprocessing, dataset-construction, evaluation, and action-labeling procedures, including atomic-action labels, phase-generation and online PLAN/MONITOR prompts, and per-task simulation success-rate reporting. It also reports real-world results and discusses the limits of recovery and generalization.

A.4.2 MONITORING AND RECOVERY RULES

  • Monitoring runs at keyframe steps, while NEXT requires visible completion, CONTINUE addresses incomplete motion, REPLAN responds to invalidated plans, and RETRY can revoke later completion records.Only the first eligible trigger invokes MONITOR; at the final planned subtask, only prolonged execution and inactivity triggers remain active. Extensions, VLM calls, replans, and retries are bounded, and excess replans become CONTINUE.Source: §A.4.2 · pp. 17–18

A.5 ACTION PREPROCESSING AND NORMALIZATION

  • The encoder uses 39 scalar components spanning end-effector pose, joints, gripper state, and contact forces, with field-specific normalization and padding masks for valid timesteps.Per-dimension dataset statistics are computed before padding; position inputs and reconstruction targets use the same transformation, while rotation and gripper representations receive their specified distinct treatment.Source: §A.5 · pp. 18–19

A.6 DATASET CONSTRUCTION AND ANNOTATION PIPELINE

  • AtomAction segments RLBench demonstrations through multi-signal boundary detection, merging, interaction-aware refinement, and static-segment pruning, with expert phase-count priors and limited manual refinement.Experts assign atomic-action types, and GPT-4o generates eight phase-specific English descriptions per phase for the assembled dataset samples.Source: §A.6 · pp. 19–20

A.7 COMPARISON WITH ATOMICVLA DATA

  • On selected shared motion subsets, AtomAction scores better than AtomicVLA on Calinski–Harabasz (266 vs. 78), inter-/intra-class ratio (1.09 vs. 0.779), and silhouette (0.116 vs. 0.0994).It also has a lower Davies–Bouldin index (2.31 vs. 2.84) and modestly lower intra-class distance (4.50 vs. 4.62); its 18 phase-level labels distinguish mechanisms and stages that AtomicVLA's coarse labels combine.Source: §A.7 · pp. 20–21, Figure 10 · p. 20

A.8 SIMULATION SETUP AND EVALUATION PROTOCOL; A.9 REAL-WORLD SETUP AND EVALUATION PROTOCOL

  • Simulation evaluation separates 18 policy-training tasks from eight policy-unseen tasks, freezes pretrained codebooks during adaptation, and uses fixed parameters and shared initial scenes at test time.Real-world experiments use a 7-DoF Franka Research 3, 30 demonstrations per task, and 10 trials per task and setting; partial completion and execution faults count as failures.Source: §A.8 · pp. 21–22, §A.9 · pp. 23–24

A.10 DETAILED REAL-WORLD ANALYSIS

  • LexiconVLA succeeds on every open-drawer trial, versus 50.0% for BridgeVLA and 60.0% for π0.5; in generalization settings, it retains about 90% of its standard success rate (56.0% vs. 62.0%).It also leads on tasks with and without simulation counterparts, though failures remain in chained manipulation. Under perturbations, instruction paraphrases leave LexiconVLA at 80.0% on open drawer, while distractors reduce its success on push buttons from 80.0% to 60.0% and on place fruits from 50.0% to 20.0%; RETRY can recover from failed grasps, but does not guarantee completion.Source: §A.10 · pp. 24–25

A.11 ATOMIC-ACTION STANDARD; A.11.1 APPROACH

  • Each released action label pairs a normalized definition with reference-example start and terminal frames; Approach ends at a preparatory pose without physical contact.Annotation is determined by the label definition rather than scene appearance alone.Source: §A.11 · p. 25, §A.11.1 · p. 26

A.11.2 GRASP; A.11.3 LIFT; A.11.4 TRANSFER

  • GRASP establishes a stable hold by closing on contact; LIFT raises the grasped object clear of support; TRANSFER carries it toward a target-region pre-placement pose.The reference examples are pick and lift, meat off grill, and put item in drawer.Source: §A.11.2 · p. 26, §A.11.3 · p. 26, §A.11.4 · p. 26

A.11.5 ROTATE; A.11.6 REVOLVE-OUT; A.11.7 REVOLVE-IN

  • ROTATE changes a held object’s orientation while keeping position approximately fixed; REVOLVE-OUT and REVOLVE-IN move fixed-axis components outward or inward, respectively.REVOLVE-IN may use pushing without a grasp when contact is sufficient.Source: §A.11.5 · p. 27, §A.11.6 · p. 27, §A.11.7 · p. 27

A.11.8 PUSH; A.11.9 FLIP-CLOSE; A.11.10 PLACE

A.11.11 PRESS; A.11.12 HANG; A.11.13 INSERT

A.11.14 FLIP-OPEN; A.11.15 PULL; A.11.16 SLIDE

A.11.17 POSE-ADJUST; A.11.18 WIPE

A.12 PROMPT TEMPLATES

  • The phase-level prompt generates eight English segment descriptions; PLAN decomposes remaining goals into atomic subtasks, while MONITOR selects a scheduling skill for the active subtask.The templates are Tables 12–14; scheduling skills and call, retry, and extension limits follow Appendix A.4.Source: §A.12 · p. 30, Table 12 · p. 31, Table 13 · p. 32

A.13 PER-TASK SIMULATION RESULTS

  • The excerpt reports 58.7±8.3% for LexiconVLA (PerAct) versus 21.3±2.3% for PerAct among the unseen-task rates, but does not identify the task for these values.The reported rates are means±std over three random seeds.Source: Table 15 · p. 33
Loading 2609.36774v1…