PaperlayerView exact revision on arXiv

Source-linked AI summary

MAST-VLA: Skill-Conditioned Adaptation with Trajectory-Affordance Priors for Multi-Skill Robot Manipulation

Jiachi Yang, Zhiyi Xu, Hongquan Zhang, Xin Tan, Zhizhong Zhang, Jingyu Gong, Yuan Xie

arXiv:2610.32312v1cs.RO

TL;DR

Multi-skill VLA policies can leave primitive-skill choice and object-interaction grounding implicit, although both matter for long-horizon manipulation. MAST-VLA combines skill-conditioned LoRA-MoE adaptation with trajectory-affordance action guidance. It reports state-of-the-art CALVIN ABC→D results and real-robot sequence experiments.

  • Problem

    Multi-skill policies may leave primitive-skill choice and interaction grounding implicit, despite their relevance to stable execution across consecutive subtasks and skill transitions.

    Source: §1 · p. 1

  • Method

    MAST-VLA combines skill-conditioned LoRA-MoE specialization in the LLM backbone with trajectory-affordance spatial guidance in the action head.

    Source: §1 · p. 2

  • Results

    MAST-VLA achieves state-of-the-art CALVIN ABC→D performance with an Avg. Len of 4.65 and improves Success@5 from 80.0% to 84.7%.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Experiments on CALVIN ABC→D and real-robot sequence tasks show improved long-horizon execution stability and complementary gains from both conditioning paths.

    Source: §6 · p. 9

    The spatial prior assumes reasonably reliable RGB-D geometry; extreme occlusion, reflective or transparent surfaces, and large calibration errors can degrade it.

    Source: §5 · p. 9

Abstract

from arXiv · show

Generalist robot manipulation requires a single vision-language-action (VLA) policy to execute diverse skills across scenes and instructions. However, current multi-skill VLA policies often leave two aspects of an instruction insufficiently modeled: which primitive skill to execute and where/how to contact and move objects. We propose MAST-VLA (Multi-skill Adaptation with Skill-conditioned experts and Trajectory-affordance priors), a dual-conditioning framework for robot manipulation. MAST-VLA introduces skill-conditioned LoRA Mixture-of-Experts (LoRA-MoE) adapters in the LLM backbone, where skill-query tokens and routing biases guide low-rank expert specialization, and converts a predicted affordance heatmap into a patch-level trajectory-affordance spatial prior for the action head. On the CALVIN ABC$\rightarrow$D long-horizon multi-skill cross-scene generalization setting, MAST-VLA achieves state-of-the-art performance, reaching an Avg. Len of 4.65 over five-subtask evaluation sequences. Real-robot multi-skill sequence experiments further validate its deployability under real perception and control noise.

1 Introduction

MAST-VLA addresses two grounding gaps in multi-skill manipulation by conditioning a shared VLA policy on primitive skills and interaction geometry. It combines skill-conditioned expert adaptation with spatial action guidance and reports strong CALVIN ABC→D results.

1 Introduction

  • Long-horizon manipulation requires inferring both the primitive operation and the contact regions, motion paths, and object interactions needed for execution.Insufficient modeling of either factor may lead to errors accumulating across consecutive subtasks and unstable skill transitions.Source: §1 · p. 1, Figure 1 · p. 2
  • MAST-VLA combines skill-conditioned LoRA-MoE specialization in the LLM backbone with a trajectory-affordance spatial prior in the action head.Skill-query tokens and routing biases condition low-rank experts; predicted affordance heatmaps become patch-level attention biases for action decoding.Source: §1 · p. 2
  • The framework jointly models primitive-skill structure and interaction-relevant spatial grounding within a shared policy.The paper presents these as complementary conditioning paths for multi-skill manipulation.Source: §1 · p. 2
  • MAST-VLA achieves state-of-the-art CALVIN ABC→D performance with an Avg. Len of 4.65 and improves Success@5 from 80.0% to 84.7%.The paper also reports real-robot experiments, ablations, and visualizations as validation of sequential robustness and the complementarity of the two paths.Source: §1 · p. 2

2 Related Work

Prior work develops generalist VLA policies, parameter-efficient adaptation, and spatial or affordance grounding for manipulation. MAST-VLA frames multi-skill adaptation as dual grounding: aligning adaptation with primitive skills and action decoding with trajectory-derived geometry.

Generalist Adaptation

  • Generalist VLA policies transfer pretrained visual-language knowledge to manipulation, but multi-skill adaptation often leaves primitive-skill separation and interaction grounding implicit.Action modeling has progressed from autoregressive tokens to action chunking, diffusion policies, and fast decoding.Source: §2 · p. 3

Modular Adaptation and Skill Specialization

  • Unlike task-level or domain-level expert allocation, MAST-VLA uses instruction-derived primitive skills to modulate skill queries and router biases.Prior adapters, prompt tuning, LoRA, and sparse expert routing provide parameter-efficient specialization approaches.Source: §2 · p. 3

Spatial and Affordance Grounding for Manipulation

  • Prior methods add spatial representations or affordance cues, whereas MAST-VLA converts demonstration trajectories into a dense action-decoding prior injected into action-head attention.The cited approaches include transport-based, voxel, multi-view, 3D-aware, and trajectory-conditioned methods.Source: §2 · p. 3

3 Methodology

MAST-VLA conditions a shared policy through skill-aware expert adaptation and trajectory-affordance spatial guidance, linking primitive-skill structure with interaction-aware action decoding.

3 Methodology

  • The policy uses the current observation, language instruction, and a coarse primitive-skill condition to predict a future action sequence.Instructions are grouped into a compact skill taxonomy, and the condition indexes skill-query and routing-bias banks without changing the instruction or action target.Source: §3 · p. 3

3.1 Skill-Conditioned LoRA-MoE Adaptation

  • Skill-query tokens and routing biases guide top-k LoRA expert selection while keeping adaptation context-dependent rather than hard-bound to a skill.The condition retrieves skill queries and a layer-wise routing bias for the LoRA-MoE adapters.Source: §3.1 · p. 4

3.2 Trajectory-Affordance Spatial Prior

  • The affordance target is derived from distances between visible scene surfaces and the expert demonstration’s future gripper trajectory, providing dense geometric supervision.The target projects the demonstrated motion tendency into the current observation space without additional manual annotation.Source: §3.2 · pp. 4–5
  • The predicted affordance heatmap becomes a patch-aligned attention bias, increasing relative attention to relevant interaction regions without masking other visual context.The heatmap is normalized and downsampled into a spatial prior for visual tokens in the action head.Source: §3.2 · p. 5

3.3 Training Objective

  • Training jointly optimizes action-chunk regression, affordance-heatmap supervision, and MoE routing regularization.The action loss is ℓ1 regression, the heatmap term is weighted binary cross-entropy, and the router auxiliary loss discourages routing collapse.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate MAST-VLA on CALVIN ABC→D long-horizon generalization and real-robot multi-skill sequences. The full model leads on CALVIN, while real-robot results and analyses support sequence robustness and complementary conditioning paths.

4.1 Experimental Setup

  • The evaluation tests five consecutive subtasks without resetting the scene, training on scenes A–C and testing on visually and spatially different scene D.Success@k measures sequences completing at least k subtasks; Avg. Len is the mean completed sequence length.Source: §4.1 · p. 5

4.2 Main Results on CALVIN ABC→D

  • 4.65 Avg. Len is best on CALVIN ABC→D; Success@4 and Success@5 rise from 85.3% and 80.0% to 89.2% and 84.7%.The gains are more pronounced in later stages of the five-subtask sequences.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Real-World Robot Experiments

  • On real-robot episodes, MAST-VLA achieves higher per-sequence Avg. Len and stronger overall sequence completion than the baselines.The benchmark uses length-3 sequences in two tabletop scenes, with instructions issued one primitive at a time after completion.Source: §4.3 · pp. 6–7, §4.1 · p. 6, Figure 3 · p. 7

4.4 Ablation and Analysis

  • Skill-query tokens alone provide limited gains, while adding LoRA-MoE yields a larger gain and the trajectory-affordance prior further improves long-horizon completion.The full model performs best among ablated variants; replacing the matched skill condition with a random one reduces Avg. Len from 4.65 to 4.55.Source: Table 2 · p. 7, §4.4 · pp. 7–8
  • Routing signatures show distinct primary and secondary expert preferences across skills, but usage shifts rather than assigning fixed experts to skills.Routing remains coherent across token groups, consistent with soft skill-conditioned and content-dependent expert selection.Source: Figure 4 · p. 8, §4.4 · p. 8
  • Predicted affordance heatmaps concentrate on future interaction regions and extend along likely object-motion directions for motion-centric operations.Visualized regions include contacts, target containers, insertion areas, and stacking interfaces.Source: Figure 5 · p. 8, §4.4 · p. 8

5 Limitations

MAST-VLA does not explicitly correct deviations between predicted actions and realized end-effector motion, and its spatial prior depends on reasonably reliable RGB-D geometry. Longer or less controlled deployments may need feedback and replanning, while severe depth or calibration failures can degrade the prior.

5 Limitations

  • The policy does not explicitly correct low-level deviations between predicted actions and realized end-effector motion.Longer or less controlled deployments may need closed-loop feedback, temporal memory, and replanning to compensate.Source: §5 · p. 8
  • The trajectory-affordance prior assumes reasonably reliable RGB-D geometry, and extreme occlusion, reflective or transparent surfaces, or large calibration errors can degrade it.Routine depth noise and missing pixels are handled in preprocessing.Source: §5 · p. 9

6 Conclusion

MAST-VLA combines skill-conditioned LoRA-MoE specialization with trajectory-affordance action guidance in a shared policy. CALVIN ABC→D and real-robot sequence experiments show improved long-horizon stability, state-of-the-art simulation performance, and complementary gains from both conditioning paths.

6 Conclusion

  • Experiments on CALVIN ABC→D and real-robot sequence tasks show improved long-horizon execution stability and state-of-the-art simulation performance.The conclusion also reports complementary gains from the skill-conditioning and trajectory-affordance paths.Source: §6 · p. 9

A Additional Method Details

The additional method details explain how primitive-skill conditions are obtained and applied, how skill biases guide sparse expert routing, and how predicted affordance heatmaps guide action attention.

A.1 Primitive Skill Conditioning

  • Primitive-skill conditions retrieve learnable skill-query tokens and routing-bias vectors, rather than becoming instruction text or action targets.For CALVIN, instruction matching and taxonomy grouping provide the condition; real-robot conditions update at each primitive step.Source: §A.1 · p. 15

A.2 Skill-Conditioned LoRA-MoE Configuration

  • Skill routing adds learned biases to content-dependent logits before top-k selection, shifting expert preference without hard-assigning skills to fixed experts.The original dense backbone stays frozen while sparse, skill-conditioned low-rank residual updates are learned.Source: §A.2 · p. 15

A.3 Affordance Heatmap Head Architecture

  • The action head turns predicted affordance heatmaps into patch-level soft attention biases for interaction-relevant static-view patches.A lightweight geometric head combines VLA context with a point map; at inference, the heatmap is predicted from the current observation and instruction, while the neutral secondary-view prior preserves access to full visual context.Source: §A.3 · p. 16

B Heatmap Construction and Training Details

The heatmap target is built from RGB-D scene geometry and future gripper positions, with loss weighting that reduces the influence of inpainted real-robot depth regions. Training hyperparameters are listed separately.

B.1 Trajectory-Affordance Target Construction

  • CALVIN targets use a static RGB-D frame converted into a 200 × 200 world-frame xyz-rgb point map; real-robot targets use calibrated static-camera RGB-D geometry.Real-robot preprocessing fills and smooths invalid depth regions, tracks the original validity mask, and down-weights inpainted pixels in the heatmap loss.Source: §B.1 · pp. 16–17
  • The target assigns a Gaussian response to visible scene surfaces near future end-effector positions, using σ = 0.05m and truncation radius rmax = 0.20m.This is the only heatmap target used for supervision.Source: §B.1 · p. 17
  • Heatmap supervision uses weighted binary cross-entropy, with positive pixels defined by H∗ > 0.1 and positive-class weights clipped to [20].When available, the point-validity mask reduces the influence of inpainted regions; the heatmap loss weight is λhm = 0.005.Source: §B.1 · p. 17

B.2 Optimization Hyperparameters

  • Table 5 lists training hyperparameters for the main MAST-VLA run.Source: Table 5 · p. 17

C.1 CALVIN ABC→D Evaluation

CALVIN ABC→D evaluation trains on scenes A–C and tests on unseen scene D using five-subtask sequences. Each sequence advances instruction-by-instruction after success, with success determined by the official task oracle or a 360-step subtask limit.

C.1 CALVIN ABC→D Evaluation

  • 1,000 predefined initial-state and subtask-sequence pairs are evaluated, each containing five consecutive subtasks.Training uses scenes A, B, and C; evaluation uses the unseen scene D.Source: §C.1 · p. 17
  • Rollouts stop when the CALVIN task oracle detects success or the 360-step per-subtask horizon is reached; the next instruction follows completion.Source: §C.1 · p. 18
  • The sequence-level metric counts consecutive subtasks completed in each five-subtask sequence, and the main paper reports this official metric.The Table 2 ablation baseline uses the same implementation setting as the variants.Source: §C.1 · p. 18

C.2 Real-Robot Evaluation Protocol

The real-robot protocol evaluates sequential primitive manipulation on two tabletop setups, with calibrated two-view RGB-D input and changing object positions. Trials proceed without resetting the scene between subtasks and use task-specific success criteria.

C.2 Real-Robot Evaluation Protocol

  • The Franka Research 3 uses a gripper and two calibrated RealSense D435i cameras, with static global and wrist/secondary local views.Object positions vary within an approximately 10 cm × 10 cm workspace; evaluation includes illumination variation and RGB-D sensing noise, but no distractor objects.Source: §C.2 · p. 18
  • Eight length-3 sequences across two scenes receive 20 trials each; instructions arrive one at a time, without scene resets between completed primitives.Human intervention is limited to resetting between trials; Table 6 lists the real-robot length-3 task sequences.Source: §C.2 · p. 18, Table 6 · p. 18
  • Success is judged by primitive-specific outcomes, including stable grasping, release within a target, insertion, displacement, stacking, or nesting.Source: §C.2 · p. 18
  • For each trial, c_i counts consecutive primitive subtasks completed before the first failure, with values from zero to three.Source: §C.2 · p. 19

D.1 Per-Sequence Real-Robot Results

Across real-robot trials, MAST-VLA achieves an Avg. Len of 2.62 and improves over four baselines. Reported failure modes include perception noise, ambiguous primitive boundaries, and accumulating pose errors.

D.1 Per-Sequence Real-Robot Results

  • 98.8% Success@1, 92.5% Success@2, and 70.6% Success@3 are reported across all 160 real-robot trials.Source: §D.1 · p. 19

D.2 Failure Cases

  • Reported failures include occlusion or noisy depth yielding incomplete affordance targets, ambiguous primitive boundaries destabilizing skill conditioning, and accumulated pose errors complicating later subtasks.Source: §D.2 · p. 19

D.1 Per-Sequence Real-Robot Results

  • 2.62 Avg. Len across 160 real-robot trials; MAST-VLA exceeded OpenVLA-OFT, π0, π0.5, and VLA-Adapter-Pro by 0.77, 0.44, 0.21, and 0.60, respectively.Table 7 reports per-sequence results, with 20 trials per method per sequence and 160 trials total.Source: §D.1 · p. 19, Table 7 · p. 20

Related papers

Loading 2610.32312v1…