Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAs

• A skill is an operation paired with its specified entity, so demonstrations can contain all constituent skills while omitting their task combination.Source: §1 · p. 2
• Fine-tuned VLAs can perform well on demonstrated combinations yet struggle with new combinations, even when every constituent skill was demonstrated.Source: §1 · p. 2
• A vision shortcut can make a policy use observations as an instruction proxy, prompting a demonstrated combination instead of the instructed new one.Source: §1 · p. 2
• CRAFT forms counterfactual pairs by changing the instruction while holding an observation fixed, then transfers supervision from a demonstrated execution of the required skill.The demonstrated action is not a direct target for the counterfactual pair because the same skill may require different actions across observations.Source: §1 · p. 2
• CRAFT improves success on undemonstrated skill combinations while maintaining high performance on demonstrated combinations across π-family and GR00T models.The evaluation covers two simulation benchmarks, with additional real-robot evaluation of π0.5.Source: §1 · p. 2

2.1 COMPOSITIONAL GENERALIZATION IN VLAS

• Robot-learning work has studied new combinations of environmental factors and recombination of demonstrated behaviors into new tasks [Gao et al., 2024; Mandlekar et al., 2020].Compositional generalization has also been studied in computer vision and large language models [Uselis et al., 2025; Sakai et al., 2025].Source: §2.1 · p. 2
• Approaches to new skill combinations include composing task representations or reusable skills at deployment and generating successful synthetic demonstrations [Li, 2025; Zhai et al., 2026; Peng et al., 2026].This paper instead uses constituent-skill coverage in existing demonstrations to supervise fine-tuning without generating new execution trajectories.Source: §2.1 · p. 3

2.2 SHORTCUT LEARNING AND COUNTERFACTUAL SUPERVISION

• A vision shortcut uses visual observations as a proxy for instructions, potentially favoring a demonstrated combination associated with similar observations [Fang et al., 2026; Xu et al., 2025; Lian et al., 2026; Xing et al., 2025].Source: §2.2 · p. 3
• Counterfactual training changes a task-relevant factor while holding other inputs fixed; prior action-supervision methods synthesize actions or connect trajectories [Kaushik et al., 2020; Glossop et al., 2025; Chen et al., 2026].Here, the changed instruction specifies an undemonstrated combination of demonstrated skills, so executions of the required skill provide supervision without constructing a counterfactual action target.Source: §2.2 · p. 3

• The VLA policy predicts a chunk of future actions from a visual observation and a language instruction.Source: §3 · p. 3
• A skill is an operation–entity pair, and an entity tuple specifies a task when the operation sequence is fixed.The formulation assumes that changing future-operation entity assignments does not change the current operation's required action for a fixed observation and current skill.Source: §3 · p. 3
• The target tasks have entity tuples absent from the demonstration set, while every constituent skill appears in the demonstrations.Source: §3 · p. 3
• Because the same skill may require different actions across observations, executions in other combinations do not directly specify the action for a new combination.The goal is to reuse demonstrated skill supervision while accounting for observation-dependent actions and preserving performance on demonstrated combinations.Source: §3 · p. 3
• Figure 2 depicts reusable skill representations and counterfactual skill representations supervised using the state and target velocity from a demonstrated required-skill execution.Source: Figure 2 · p. 4

4 METHOD: CRAFT

• CRAFT trains counterfactual skill representations using action prediction for demonstrated executions of the required skill, avoiding the need for action targets for undemonstrated combinations.The same skill can require different actions across observations, so actions from other executions cannot generally serve as direct targets.Source: §4 · p. 4, §4.3 · p. 5

4.1 SKILL AND STATE REPRESENTATIONS

• The policy combines a skill representation for the currently required skill with a state representation derived from visual observations to condition action prediction.Skill queries attend to image and text, while state queries attend only to image tokens.Source: §4.1 · p. 4

4.2 LEARNING SKILL REPRESENTATIONS FROM DEMONSTRATIONS

• A contrastive objective trains skill representations to be reusable across same-skill executions while preserving distinctions between skills in action prediction.It compares prediction errors using representations from same- and different-skill executions; a VLM segments demonstrations to assign skill labels for sampling positives and negatives.Source: §4.2 · p. 5

4.3 COUNTERFACTUAL SUPERVISION FROM DEMONSTRATED SKILLS

• For counterfactual supervision, CRAFT holds an observation fixed, changes the instruction, and predicts actions using the counterfactual skill representation with a demonstrated reference’s state and target velocity.If only future-operation entities change, the original sample is the reference; if the current-operation entity changes, the reference is a demonstrated execution of the newly required skill.Source: §4.3 · p. 5

4.4 TRAINING OBJECTIVE

• The training objective combines flow-matching, skill-representation, and counterfactual losses, with gradients routed differently for each objective.Flow matching updates all trainable components; the skill loss stops gradients through the state representation, while the counterfactual loss freezes the action expert and reference state representation.Source: §4.4 · p. 6

5.1 EXPERIMENTAL SETUP

• PICK-PLACE evaluates 12 unseen combinations out of 16, while PICK-PLACE-PRESS evaluates six out of eight; each demonstrated combination has 50 demonstrations.Both datasets cover every constituent skill but only a subset of combinations; the evaluation reports success rates over shared initial states and three evaluation seeds.Source: §5.1 · pp. 6–7
• The evaluation compares CRAFT with Full, LoRA, Frozen, CAG-VA, and Skill Composition across π0, π0.5, and GR00T N1.7.The π family uses layer-wise attention to VLM features, whereas GR00T conditions its action expert on the VLM output.Source: §5.1 · pp. 6–7

5.2 COMPOSITIONAL GENERALIZATION

• Across all three VLA models and both benchmarks, CRAFT achieves the highest success on undemonstrated combinations while maintaining high demonstrated-combination performance, though a substantial gap remains, especially on PICK-PLACE-PRESS.Standard fine-tuning often performs well on demonstrated combinations but substantially less well on the remaining combinations.Source: Table 1 · p. 7, §5.2 · p. 7
• The gap persists across Full, LoRA, and Frozen adaptation, while CAG-VA gains vary and Skill Composition gains are smaller on PICK-PLACE-PRESS.Full has low success on undemonstrated combinations despite updating the entire VLM.Source: §5.2 · p. 7

5.3 BEHAVIOR AND REPRESENTATION ANALYSIS

• Policies maintain the current skill when an instruction changes only a future target, but switch behavior when the currently required skill changes.The observed switching occurs from a state reached while executing a different skill.Source: §5.3 · p. 8, Figure 4 · p. 8
• With Lskill, representations form clearer entity-specific groups within pick and place operations than without it.Figure 5 shows demonstration frames projected with t-SNE, colored by the entity of the currently required skill.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 TRAINING OBJECTIVE ABLATIONS

• On PICK-PLACE, Lskill alone leaves undemonstrated-combination success low, Lchange improves it, and combining Lchange with Lpreserve yields substantially higher success than either term alone.The combined objective also maintains high success on demonstrated combinations; the comparison keeps architecture and flow-matching loss unchanged.Source: Table 2 · p. 8, §5.4 · p. 8

5.5 REAL-ROBOT EVALUATION

• On the real robot, CRAFT achieves higher success than Full on undemonstrated combinations while maintaining high success on demonstrated combinations.The comparison uses π0.5 and the same demonstration dataset for both methods.Source: §5.5 · p. 9

• CRAFT draws supervision for an unseen combination from demonstrated executions of its currently required skill, rather than transferring actions directly across observations.It also avoids synthesizing trajectories for new combinations.Source: §6 · p. 9
• The reported gains suggest existing demonstrations can provide supervision beyond the combinations they contain.The paper notes that high success on demonstrated combinations did not consistently translate into high success on undemonstrated ones.Source: §6 · p. 9
• The formulation assumes a fixed operation sequence and demonstrated constituent skills; new skills and instruction-dependent operation sequences are out of scope.Source: §6 · p. 9

7 CONCLUSION

• Across three VLA models and two simulation benchmarks, CRAFT improves success on undemonstrated combinations while maintaining high performance on demonstrated ones.It also improves compositional generalization on a real robot.Source: §7 · p. 9
• The results support improving compositional generalization by making better use of constituent-skill supervision already present in demonstrations.Source: §7 · p. 9
• The study asks whether fine-tuned VLAs can recombine demonstrated skills into combinations absent from their demonstrations.Source: §7 · p. 9

ETHICS STATEMENT

• The study does not involve human or animal subjects, personally identifiable information, or sensitive personal data.Source: ETHICS STATEMENT · p. 10
• The work studies compositional generalization for VLA policies in simulation and on a real robot.Source: ETHICS STATEMENT · p. 10
• The real-robot experiments use a controlled laboratory setting and evaluate compositional generalization, not safety for unconstrained real-world deployment.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The datasets and evaluation environments developed for the work, together with the training code, will be publicly released.Source: REPRODUCIBILITY STATEMENT · p. 10
• Implementation details appear in the main paper, with additional architecture details, training configurations, and hyperparameters in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 10
• The paper specifies the evaluation protocols used for its reported experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 COMPOSITIONAL GENERALIZATION GAP IN FINE-TUNED VLA MODELS

• The six pick-and-place demonstrations define six combinations; five of nine feasible undemonstrated combinations are selected, reusing demonstrated object and target phrases.Source: §A.1 · p. 13
• Across four LIBERO-fine-tuned models, success ranges from 88.8–97.4% on demonstrated combinations versus 2.0–50.9% on undemonstrated combinations, a gap of 46.5–94.1 percentage points.The five evaluated undemonstrated combinations retain every constituent skill in the demonstrated tasks.Source: §A.1 · p. 14

A.2 FAILURE BEHAVIOR CONSISTENT WITH A VISION SHORTCUT

• Failed rollouts that complete an evaluated task favor demonstrated combinations sharing an instructed object or target, while empty-instruction trials also produce demonstrated-task behaviors.The object-versus-target pattern varies across models, and the authors note that other contributing factors are not excluded.Source: §A.2 · p. 15

B.1 ARCHITECTURE

• CRAFT uses skill and state queries to condition action prediction; the π family uses query-position layer-wise K/V, while GR00T uses final hidden states.Skill queries attend to image and instruction tokens; state queries attend to visual tokens without directly attending to instruction tokens.Source: §B.1 · p. 16

B.2 SKILL ANNOTATION AND COUNTERFACTUAL INSTRUCTIONS

• Training annotations assign operation–entity labels by locating transitions between known operations; labels are not supplied to the policy at inference.Counterfactual instructions change current or future entity assignments while keeping the observation and operation order fixed, excluding demonstrated combinations.Source: §B.2 · pp. 16–18, Table B.1 · p. 18

B.3 TRAINING

• The training procedure combines demonstrated-skill comparisons, counterfactual supervision, and joint optimization.Source: §B.3 · p. 18

B.3.1 LEARNING SKILL REPRESENTATIONS FROM DEMONSTRATIONS

• Skill comparisons use a same-label positive from another execution and negatives with the same operation but different entities, varying only the skill representation.Prediction error is normalized by 1 + mean(a2_i), accounting for unit Gaussian-noise variance and mean squared action magnitude.Source: §B.3.1 · p. 18

B.3.2 COUNTERFACTUAL SKILL-REPRESENTATION ALIGNMENT

• For skill-changing counterfactuals, the counterfactual supplies the skill representation and a demonstrated reference supplies state, noisy action, flow time, and target; preserving cases use the original sample.Counterfactual loss is restricted to reference-chunk timesteps within the matched operation, while demonstrated flow matching covers the complete valid chunk.Source: §B.3.2 · p. 18

B.4 IMPLEMENTATION DETAILS

• CRAFT trains for 20k steps with skill-balanced batches, three negative samples per anchor, τ = 0.1, β = 0.03, γ = 1.0, and 64 total queries.Batch sizes are 32 for PICK-PLACE and 18 for PICK-PLACE-PRESS; auxiliary-loss weights ramp up over the first 3k steps. Query splits are 48/16 for π0.5 on PICK-PLACE and 32/32 otherwise.Source: §B.4 · p. 19
• Optimization uses AdamW with warmup and cosine decay; learning rates, gradient clipping, EMA, and LoRA settings vary by model, while Skill Composition uses annotated segments and gripper-state transitions.Checkpoints are evaluated every 2k steps from 10k to 20k and selected by mean undemonstrated-combination success across three seeds; rollouts execute 10 of each predicted 20-action chunk before replanning.Source: §B.4 · p. 19

C BENCHMARK CONSTRUCTION

• Both LIBERO benchmarks fix the operation sequence and vary the instructed entities; demonstrations cover same-color combinations, while evaluation tests remaining combinations on separate initial scenes.Each benchmark uses 50 shared initial scenes for demonstrations, with one teleoperated demonstration per demonstrated combination and scene; evaluation scenes are separately sampled from the same initialization distribution.Source: C BENCHMARK CONSTRUCTION · p. 20, Figure C.1 · p. 20

C.1 PICK-PLACE

• PICK-PLACE has four cube and plate colors: four same-color demonstrations cover all eight pick/place skills, leaving 12 of 16 combinations undemonstrated.Cube positions vary within the quadrant of their same-color plate; success requires the specified cube to be on the specified plate.Source: §C.1 · p. 20

C.2 PICK-PLACE-PRESS

• PICK-PLACE-PRESS adds a button-press operation with red and blue objects; all-red and all-blue demonstrations cover six skills, leaving six of eight combinations undemonstrated.Success requires the specified cube on its plate and the end-effector within the button-center tolerance and below a height threshold.Source: §C.2 · p. 20

D.1 PERFORMANCE ACROSS TASK COMBINATIONS

• On PICK-PLACE, the blue-cube/green-plate combination is among the lowest-success cases for all three models, while red-cube/green-plate success is 90.7% for π0.5 and 30.7% for GR00T N1.7; PICK-PLACE-PRESS success is concentrated in two combinations for GR00T N1.7 but broader for π0.5.For GR00T N1.7, the two stronger combinations are red–blue–blue and blue–red–red, with colors specifying cube, plate, and button; the other four undemonstrated combinations are substantially lower.Source: §D.1 · pp. 21–22

D.2 ACTION PREDICTIONS UNDER INSTRUCTION CHANGES

• With a fixed observation, CRAFT’s predicted trajectories separate by instructed cube or plate, whereas FT (Full) predictions largely overlap across instruction changes.The comparison varies the instructed entity while holding the observation fixed in picking and placing examples.Source: §D.2 · p. 22, Figure D.3 · p. 22

D.3 REUSING SKILL REPRESENTATIONS ACROSS EXECUTIONS

• Same-skill representation substitutions change predicted action chunks by 4.0 ± 3.1% on average, while different-skill substitutions produce substantially larger changes.The comparison keeps the recipient state representation and sampling noise fixed across substitutions.Source: §D.3 · p. 23, Figure D.4 · p. 23

D.4 QUERY COUNT AND ALLOCATION

• For π0.5 and GR00T N1.7, undemonstrated-combination success rises from 32 to 64 queries and falls at 128, while demonstrated success remains high across the sweep.Thus, adding queries does not monotonically improve compositional generalization.Source: §D.4 · p. 23, Figure D.5 · p. 24
• With 64 queries, π0.5 has lower undemonstrated success at a 16/48 skill/state split than at 32/32 or 48/16; demonstrated success remains high across all three allocations.The balanced and skill-heavy allocations perform similarly well.Source: §D.4 · p. 23, Figure D.5 · p. 24

D.5 GUIDANCE-STRENGTH SELECTION FOR CAG-VA

• Guidance effects vary by model and adaptation: stronger guidance improves π0.5 LoRA’s undemonstrated success on both benchmarks but can reduce demonstrated success; π0 gains are smaller and may lack corresponding undemonstrated improvement.These differences motivate selecting guidance strength separately for each configuration.Source: §D.5 · p. 24, Figure D.6 · p. 25

D.6 ADDITIONAL REAL-ROBOT ROLLOUTS

• Additional real-robot rollouts show CRAFT picking the instructed cube and placing it on the instructed plate for combinations absent from demonstrations.The figure presents different instructions and their executions over time.Source: Figure D.7 · p. 26