PaperlayerView exact revision on arXiv

Source-linked AI summary

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

Kechun Xu, Zhenjie Zhu, Anzhe Chen, Rong Xiong, Yue Wang

arXiv:2606.12366v1cs.RO

TL;DR

Continuous-action VLA policies struggle to generalize from seen tasks to unseen and compositional language instructions amid imbalanced training data. APT pretrains an action expert on vision-action pairs before injecting language, and consistently improves instruction generalization across architectures and experiments.

  • Problem

    Continuous-action VLA policies struggle with unseen and compositional instructions because VLA data contains far less language diversity than visual-action content.

  • Method

    APT factorizes the policy into a language-agnostic vision-action prior and language-conditioned likelihood, then pretrains the action expert before gated language fusion.

  • Results

    APT consistently improves OOD instruction generalization across diverse VLA architectures in simulation and real-world experiments.

  • Takeaways & Limitations

    Action expert pretraining provides a broadly applicable training strategy for improving instruction generalization in continuous-action VLA policies.

  • Takeaways & Limitations

    APT does not explicitly model long-horizon memory and is evaluated only on tabletop manipulation.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the $π$ and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/

1 Introduction

APT addresses poor out-of-distribution instruction generalization in continuous-action VLA models by pretraining an action expert on balanced vision-action pairs, then injecting language through gated fusion while preserving the visuomotor prior.

  • Motivation: Structural imbalance pairs many vision-action frames with one instruction, encouraging visual shortcuts that bypass language and hinder instruction generalization.Discrete-action methods mitigate this through vision-language co-training, whereas continuous-action methods use generative action experts but lack this protection.
  • Method: APT factorizes the policy as π(a|v, ℓ) ∝πp(a|v)·L(ℓ|v, a), separating a language-agnostic Vision-Action prior from a language-conditioned VLA likelihood.Vision-action pairs are well-balanced and avoid the shortcut incentives present in full VLA triplets.
  • Method: In Stage 1, a diffusion-based action expert learns the VA prior from visual tokens produced by a frozen VLM backbone.This pretraining uses vision-action pairs while bypassing the language imbalance.
  • Method: In Stage 2, newly introduced attention layers inject language tokens to form the VLA likelihood and align the pretrained action distribution with task instructions.APT uses a layer-wise gated fusion mechanism for effective VLM feature fusion while preserving the learned visuomotor prior.
  • Contributions: APT generalizes across architectures and delivers consistent gains on OOD language generalization in extensive simulation and real-world experiments.The method is designed for existing VLA datasets and applies to mainstream VLA architectures, including π- and GR00T-style architectures.

2 Related Work

Related work establishes VLAs as a dominant paradigm, progressing from training policies from scratch to adapting pretrained VLMs for continuous motor control. Despite strong in-distribution performance, these models often rely on visual shortcuts and generalize poorly to unseen instructions, objects, and compositional tasks.

  • Vision-Language-Action Models: VLA models evolved from transformer policies trained from scratch on large robot datasets to architectures that finetune pretrained VLMs.Later approaches combine VLMs with continuous motor control and extend the paradigm through reasoning co-training and world-model objectives.
  • Vision-Language-Action Models: Action discretization can retain language capability through vision-language co-training, but discrete representations struggle with continuous dexterous control.
  • Vision-Language-Action Models: Despite strong in-distribution performance, continuous-action VLAs struggle to generalize to unseen instructions, objects, and compositional task variations.The related work attributes this difficulty partly to randomly initialized action experts trained jointly with the VLM.
  • Generalization on Task Instructions: Benchmarking shows that many VLA models treat language instructions as task identifiers rather than grounded semantic descriptions, causing seen-instruction success to collapse under novel variations.The identified shortcut exploits visual correlations while discarding language semantics.

3 Method

APT addresses instruction generalization by separating language-agnostic visuomotor learning from language-conditioned action alignment. It pretrains an action expert on balanced vision-action pairs, then injects VLM features through gated fusion while preserving the learned prior.

  • Bayesian Factorization: APT factorizes π(a | v, ℓ) into a language-agnostic Vision-Action prior πp(a | v) and a Vision-Language-Action likelihood L(ℓ| v, a).The prior models multimodal actions from visual observations, while the likelihood aligns that pretrained action distribution with task-specific instructions.
  • Instruction Generalization Bottleneck: Continuous-action VLA policies jointly train a randomly initialized action expert with the VLM, allowing structurally imbalanced data to encourage visual shortcuts that bypass language.Each trajectory contains T vision-action pairs but only one language instruction, producing richer visual-action than language diversity and noisy gradients that can harm VLM language representations.
  • Two-Stage Pretraining: The VA prior is trained without language on vision-action pairs, whose frame-level action annotations provide balanced data for learning diverse manipulation behaviors without shortcut incentives.All VLM parameters remain fixed during this stage, and the action expert learns πp(a | v).
  • Two-Stage Pretraining: After prior pretraining, Stage 2 conditions the action expert on language and trains the full model to align the learned action distribution with instructions.This decouples action generation from language grounding, making alignment easier while preserving the action expert’s learned priors.
  • Layer-wise VLM Feature Gated Fusion: APT injects intermediate Qwen3-VL features into every action-expert self-attention layer through learnable sigmoid gates.Uniformly sampled features provide shallow spatial and deep semantic information while the action expert retains its own self-attention pathway.

4 Experiments

Experiments evaluate APT on simulated and real-world benchmarks targeting instruction and compositional generalization. APT consistently improves OOD performance through action-expert pretraining and language-injection designs that preserve the learned visuomotor prior.

  • Simulation Benchmarks: APT is evaluated on LIBERO-PRO and a language-conditioned rigid-object pick-place benchmark designed to test positional, object, environment, and instruction generalization.LIBERO-PRO perturbs object positions or instructed objects, while Pick-Place randomizes layouts, viewpoints, and instructions across four suites.
  • LIBERO-PRO: On LIBERO-PRO, OpenVLA and π0 achieve 0% success under both perturbations, while π0.5 recovers Pos performance but remains near zero on Task.LangForce improves Task over π0.5 but sharply degrades on Pos, whereas APT is evaluated against these baselines for instruction generalization.
  • Rigid Object Pick-Place: On Pick-Place, π0 has the weakest generalization, π0.5 improves with Knowledge Insulation and large-scale vision-language co-training, and APT variants isolate the contribution of two-stage pretraining.The ablation varies KI, 2-Stage action-expert pretraining, and joint VLM-action-expert training.
  • Action Pretraining on More Architectures: Two-stage action-expert pretraining improves generalization across almost all π-style and GR00T-style settings, with the largest gains for APT and GR00T-style architectures.Gated fusion in APT and final-layer conditioning in GR00T-style better preserve the Stage 1 VA prior than π-style fusion at every attention layer.
  • Real-World Single-Task Generalization: In real-world pick-place and clutter pick-place tasks, APT maintains robust OOD performance and outperforms π0.5 across clutter settings after finetuning with 30 demonstrations per task.The evaluations use 110 pick-place trials and 80 clutter pick-place trials across increasingly difficult seen and unseen settings.
  • Real-World Compositional Generalization: For compositional generalization, APT remains strong under task chaining while π0.5 nearly collapses, despite comparable seen-language performance under sequential task coaching.Unseen coaching perturbs verbs, nouns, and cross-task object assignments; chaining instead concatenates multiple task instructions into one prompt.

5 Conclusion

APT pretrains action experts on balanced vision-action data by decomposing continuous-action VLA policies into a VA prior and VLA likelihood, improving OOD instruction generalization across architectures. Its current evaluation is limited by the absence of explicit long-horizon memory modeling and focus on tabletop manipulation.

  • Conclusion: APT uses balanced vision-action action-expert pretraining and a VA-prior/VLA-likelihood decomposition to improve OOD language generalization in continuous-action VLA policies.The method induces a two-stage pretraining procedure.
  • Conclusion: APT consistently improves instruction generalization and applies across diverse VLA architectures.
  • Limitations: APT does not explicitly model long-horizon memory and has been evaluated only on tabletop manipulation, leaving locomotion and mobile manipulation unexplored.These limitations affect tasks requiring multi-step progress tracking.

A Implementation Details

APT uses a camera-frame, embodiment-equivariant action representation and a two-stage architecture combining PRoPE and mRoPE positional encodings. Training uses specified optimizer settings, diffusion schedules, large-scale multi-embodiment pretraining datasets, and balanced sampling weights.

  • Action Representation: Actions form a 10-dimensional vector of 3D translation, 6D continuous rotation, and normalized gripper width in the camera coordinate frame.This representation provides embodiment equivariance across heterogeneous pretraining platforms.
  • Network Architecture: The Qwen3-VL-2B-Instruct backbone feeds separately projected visual and language tokens into a 20-layer action expert with 2-layer, 768-dimensional action encoder and decoder MLPs.The model uses an action chunk length of Ta = 32, history action length 1, and 256 × 256 wrist and third-view images.
  • Positional Encodings: Stage 1 applies PRoPE to visual and action tokens, while Stage 2 retains PRoPE in inherited layers and inserts mRoPE layers attending jointly over vision, language, and action tokens.The two-stage attention-mask design is shown in Figure 9.
  • Training Hyperparameters: Training uses AdamW with action-expert learning rate 10^-4 and weight decay 10^-2, plus VLM learning rate 10^-5 and weight decay 10^-10 during joint finetuning.Models use batch size 256; each stage runs for 100k iterations, with DDPM training for 100 diffusion steps and DDIM inference for 20 denoising steps.
  • Pretraining Datasets: Pretraining uses DROID, AgiBotWorld-Alpha, InternData-A1, and InternVLA-M1, covering diverse embodiments, scenes, and skills.The datasets are sampled with weights 5:5:4:1 respectively to balance real-robot diversity with synthetic skill coverage.

B Visual Shortcut Analysis … D Simulation Experiment Details

The analysis attributes weak OOD instruction following to a visual shortcut caused by imbalanced VLA data and shows that two-stage prior training encourages language grounding. The paper also specifies the main VLA and complementary baselines used for comparison.

  • B Visual Shortcut Analysis: VLA training can converge toward a visual shortcut that largely ignores language, degrading OOD instruction generalization.Information-theoretic analysis validates that jointly training a randomly initialized action expert with the VLM admits this shortcut.
  • B Visual Shortcut Analysis: Language diversity is much smaller than vision-action diversity, so each visual observation is often associated with an almost deterministic instruction.A trajectory typically contains many visual frames linked to one language prompt, while overlaps across differently instructed trajectories are rare.
  • B.1 Shortcut Learning: The shortcut arises because language contributes only marginally to action prediction, making the simpler vision-only solution attractive to gradient descent.The vision-only policy has conditional-entropy loss only marginally larger than the full VLA policy and does not depend on the language instruction.
  • B.2 Two-stage Conditioning: APT avoids the shortcut by first learning a language-agnostic vision-action prior and then adding language conditioning on its pretrained visual embedding.Stage 1 learns πp(a|v); Stage 2 initializes θVA from Stage 1 while randomly initializing the added language-conditioning parameters.
  • B.2 Two-stage Conditioning: Because the Stage 2 loss can improve beyond the language-ignored bound, optimization incentivizes the policy to use language for action generation.The pretrained prior preserves the desired visuomotor policy while the added conditioning pathway can reduce loss by grounding the prompt.
  • C Baseline Details: The primary continuous-action baselines include OpenVLA, π0, and π0.5, spanning Prismatic and PaliGemma backbones, diffusion experts, and hybrid internet-robot pretraining.π0.5 is described as the current state of the art in generalist VLA policies.
  • C Baseline Details: Additional comparisons cover language-grounding, discrete-action, latent-action, world-model, reinforcement-learning, flow-matching, and code-as-policies approaches.These include LangForce, CaP-X, π0-FAST, UniVLA, WorldVLA, RIPT-VLA, and X-VLA, each providing a distinct alternative for language following, action representation, pretraining, or post-training.

D.1 Benchmark Details

APT is evaluated on simulation benchmarks spanning increasing language-generalization difficulty, including LIBERO, LIBERO-Plus, LIBERO-PRO, and a diverse IsaacSim rigid-object pick-place benchmark.

  • LIBERO: LIBERO includes four suites—Spatial, Object, Goal, and Long—each with 10 tasks and 50 demonstrations, yielding 500 trajectories per suite.The suites assess spatial relations, object-instance discrimination, goal-directed instruction following, and long-horizon sequential behavior.
  • LIBERO-Plus: LIBERO-Plus extends LIBERO with seven controlled perturbation axes covering object layout, viewpoint, robot state, language paraphrase, lighting, textures, and sensor noise.Five axes test visual and configuration robustness while keeping task and language fixed; the language axis paraphrases the original instruction.
  • LIBERO-PRO: LIBERO-PRO evaluates language generalization through Pos perturbations that swap manipulated-object positions while holding the task instruction fixed.This tests whether policies use language to locate targets rather than relying on positional priors.
  • LIBERO-PRO: Task perturbations in LIBERO-PRO replace the manipulated object in the instruction with a different scene object, creating an unseen task for OOD language generalization.Replicating training trajectories fails under both Pos and Task perturbations, preventing dataset-level shortcuts.
  • Rigid Object Pick-Place: The IsaacSim rigid-object pick-place benchmark uses a UR5 arm, randomly samples 4 of 25 objects per scene, and randomly selects pick and place targets.It contains 10k pick-place trajectories across 500 scenarios for multi-task training, with randomized camera settings mentioned in the benchmark description.

D.2 Results on Original LIBERO … E.1 Real-world Setup

APT preserves in-distribution task-solving performance on original LIBERO, leads robustness results on LIBERO-Plus, and qualitatively improves grounding of unseen objects. Real-world evaluation uses an Agilex Cobot setup, diverse tele-operated demonstrations, and comparison with π0.5 under specified action representations.

  • D.2 Results on Original LIBERO: APT achieves a 96.1% average success rate on original LIBERO, including 98.4% on Spatial, 99.4% on Object, 96.4% on Goal, and 90.2% on Long.These results confirm that APT does not sacrifice task-solving capability on this primarily in-distribution benchmark.
  • D.2 Results on Original LIBERO: APT remains competitive with π0.5 and exceeds π0 and UniVLA on training-distribution tasks, while improving language generalization on more challenging benchmarks.The original LIBERO results are not intended to claim superiority because the benchmark primarily measures in-distribution completion.
  • D.3 Results on LIBERO-Plus: APT achieves the highest average success rate on LIBERO-Plus, slightly above X-VLA and clearly ahead of π0 and UniVLA.LIBERO-Plus evaluates robustness across seven perturbation axes.
  • D.3 Results on LIBERO-Plus: APT performs best on object layout at 80.1% and lighting at 93.6%, while ranking second on language at 77.6%, background at 92.3%, and robot initial state at 63.1%.The layout-and-language combination indicates reliable instruction following under perturbations.
  • D.4 Case Studies on Rigid Object Pick-Place: In the UO split, π0.5 grasps a distractor with similar color, whereas APT correctly grounds the unseen object.In the SO split, both methods reliably follow seen instructions and grasp the correct object.
  • E.1 Real-world Setup: The real-world platform uses an Agilex Cobot with a Piper arm and two ORBBEC DaBai cameras capturing RGB-D images at 640 × 480 resolution.The cameras provide third-view and wrist-view observations, and evaluations vary objects, containers, and backgrounds.
  • E.1 Real-world Setup: Each real-world task uses 30 tele-operated demonstrations spanning diverse language instructions and object layouts, comparing APT with π0.5.π0.5 is finetuned with joint-space actions, while APT uses the camera-frame representation from Section A.
  • E.1 Real-world Setup: Real-world evaluation covers single pick-place, clutter pick-place, and compositional storage-plus-pick-place tasks across seen, unseen, and chained settings.The overview includes SO, UO, UOUC, UOUCUE, UC, UOUE, and task-specific language-coaching splits.

E.2 Single Task Detailed Settings

This section details the pick-place and clutter pick-place tasks, including their instructions, evaluation setup, and the planning challenges required for successful execution.

  • Clutter Pick-place Task: Clutter pick-place requires pushing aside distractors before grasping and placing the target, demanding long-horizon planning and correct object and container identification.The task combines push, pick, and place sub-tasks because the object is tightly surrounded by distractors.

E.3 Compositional Task Detailed Settings

The compositional evaluation covers two real-world tasks and tests increasingly difficult instruction-following settings through sequential prompts, OOD language variations, and concatenated task execution.

  • Tasks: The evaluation uses Table Storage (T1) and Pick-Place (T2), with seen objects and varied instruction structure, wording, or layouts.T1 stores specified table items sequentially, then closes the box; T2 performs a single pick-and-place action.
  • Evaluation Protocols: Seen Language Coaching issues the two tasks sequentially with training-distribution wording across 20 trials, balancing task order.Ten trials issue T1 before T2, and ten issue T2 before T1.
  • Evaluation Protocols: Unseen Language Coaching preserves the two-prompt protocol while rewriting instructions through verb, noun, or task substitution OOD axes.These substitutions retain or alter task semantics through changed verbs, synonymous nouns, or swapped manipulated objects.
  • Evaluation Protocols: Task Chaining concatenates both tasks into one prompt, requiring the policy to parse and execute them without an explicit segmentation signal.The combined instruction first performs storage and box closure, then executes pick-place.

E.4 Detailed Results · E.5 Case Studies · E.6 Failure Studies

Detailed results show that APT improves robustness on unseen objects, clutter, and compositional instructions, while case studies and failure analyses reveal π0.5’s grounding and transition failures and APT’s remaining errors.

  • E.4 Detailed Results: In UO, π0.5 achieves only 5/10 and 6/10 on grape and bottle, whereas APT reaches 9/10 and 8/10.Under UOUCUE, π0.5’s bottle performance further degrades to 6/20, while APT maintains a stronger result.
  • E.4 Detailed Results: APT outperforms π0.5 on almost every clutter pick-place object and split, including pepper in SO at 9/10 vs. 4/10 and grape in UOUE at 6/10 vs. 3/10.π0.5 often hesitates during pepper’s push-to-grasp transition and misgrounds targets sharing colors with distractors, whereas APT’s pretrained VA prior largely avoids these behaviors.
  • E.5 Case Studies: On unseen-grape pick-place cases, π0.5 hesitates or slips during grasping, whereas APT grasps the target on the first attempt.When a non-target object is already inside the container, π0.5 often remains completely still, likely misinterpreting the task as completed.
  • E.5 Case Studies: In clutter transitions, π0.5 either keeps pushing without creating grasping space or continues pushing after the target is fully exposed.These cases indicate failures to switch from decluttering to grasping at the required sub-task boundary.
  • E.5 Case Studies: For chained T1-storage and T2-pick-place instructions, π0.5 places the wrong object or places both target objects before stopping.The observed behavior suggests that concatenated prompts affect language grounding and may be treated as a single task judged by T1 completion.
  • E.6 Failure Studies: Starting from 10 trials, π0.5 retains 10 →8 →5 →3 →3 rollouts across the checkpoints, while APT retains 10 →9 →8 →6 →6.The largest gap appears at the push-to-grasp transition in UOUE clutter pick-place.
  • E.6 Failure Studies: APT occasionally continues pushing after grasping instead of placing and over-attends to the pick-place sub-task during compositional chaining.These failures occur much less often than for π0.5 but have not been fully eliminated.
Loading 2606.12366v1…