PaperlayerView exact revision on arXiv

Source-linked AI summary

NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models

Guodong Du, Wanyu Lin

arXiv:2610.23271v3cs.ROcs.LG

TL;DR

VLA skills are often fine-tuned independently, but consolidating them without pooled training data is difficult and direct weight merging can cause interference. NestedVLA stores compressed task-specific VLM updates and uses a task-conditioned hypernetwork to generate action-expert updates. Across three policies, it improves on the strongest dynamic merging baseline while using 1.5× storage.

  • Problem

    Post-hoc consolidation must combine independently fine-tuned VLA policies without pooling their original training data, while direct merging can cause task interference.

    Source: §1 · p. 1

  • Method

    NestedVLA retains compressed task-specific VLM updates and uses a task-conditioned hypernetwork to generate sparse action-expert updates for composition with a frozen base.

    Source: §1 · pp. 1–2

  • Results

    66.7% average success at 1.5× storage, improving by 41.1 pp over the strongest dynamic baseline across three skills.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The remaining MetaWorld gap is bounded by the top-K ideal itself, making adaptive sparsity a natural next direction.

    Source: §5 · p. 10

    All experiments consolidate M=3 skills; scaling quality, training time, and hypernetwork capacity as the skill set grows remains unmeasured.

    Source: H Limitations · p. 23

Abstract

from arXiv · show

Vision-language-action (VLA) models can learn a wide range of robotic skills, but combining separately trained skills remains challenging. The original training data may be unavailable for joint retraining, while directly merging model weights can cause interference. We propose NestedVLA, which consolidates independently fine-tuned VLA policies without access to their original training data. Instead of merging all updates into one checkpoint, a hypernetwork embedded in the VLA generates action-expert weight updates for a selected task. Each task's compressed vision-language model update is stored separately, and both updates are added to a frozen base VLA to instantiate the corresponding policy. We evaluate NestedVLA by consolidating three policies fine-tuned with reinforcement learning on MetaWorld, ManiSkill, and CALVIN. It achieves 66.7% average success, improving on the strongest model-merging baseline by 41.1 percentage points. It performs within 1 percentage point of per-task fine-tuning on ManiSkill and substantially narrows the gap on CALVIN. In this three-policy setting, total storage is 1.5x that of a single full checkpoint. Code is available at https://github.com/WanyuGroup/NeurIPS2026-NestedVLA.

1 Introduction

The paper studies post-hoc consolidation of independently fine-tuned VLA policies when training data cannot be pooled, a setting where directly combining updates can cause interference. NestedVLA generates task-conditioned action-expert updates while retaining compressed task-specific VLM updates, and improves on model-merging baselines across three benchmarks.

1 Introduction

  • Post-hoc consolidation targets independently fine-tuned policies when original training data cannot be pooled, avoiding duplicated full checkpoints for each benchmark.Joint multi-task training can avoid duplication when data from all benchmarks can be pooled.Source: §1 · p. 1
  • Directly merging independently learned VLA updates can impair source-task performance; weak alignment across parameter groups is especially consequential in the action expert that governs control.The paper notes that this finding is consistent with MergeVLA [16].Source: §1 · p. 1
  • NestedVLA replaces parameter aggregation with a task-conditioned hypernetwork that generates sparse action-expert updates while retaining compressed task-specific VLM residuals.Both updates compose with a shared base to instantiate the selected VLA.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 41 pp average improvement over the strongest model-merging baseline, with performance within 1 pp of per-task fine-tuning on ManiSkill and substantially closing the gap on CALVIN.The evaluation uses the open-source π0.5 backbone on MetaWorld, ManiSkill, and CALVIN.Source: §1 · p. 2

2 Related Work

Prior VLA multi-task methods learn skills from pooled data, while NestedVLA addresses policies acquired independently and consolidated afterward. Related work spans multimodal adaptation, hypernetworks, reusable task updates, and model merging; NestedVLA applies task-conditioned generation to sparse VLA updates.

Vision-Language-Action models

  • VLA multi-task ability is typically learned from pooled data with co-located datasets and predefined skills, whereas NestedVLA targets independently acquired skills consolidated after the fact.The related-work discussion contrasts this setting with instruction-following VLA backbones including OpenVLA [26], π0 [1], and π0.5 [40].Source: §2 · p. 2

Parameter-efficient visual adaptation

  • Prompt, adapter, and structured low-rank methods adapt multimodal representations; NestedVLA instead consolidates parameter updates from independently trained VLA policies.The discussion also describes visual alignment methods [60] and AHRA [38].Source: §2 · p. 2

Hypernetworks

  • Hypernetwork work generates model parameters or task updates, while NestedVLA targets sparse top-K deltas on a frozen billion-scale VLA backbone from independently fine-tuned RL policies.The cited precedents include HyperNetworks, latent-diffusion weight synthesis [48], and prompt-to-LoRA generation [31].Source: §2 · p. 2

Reusable task updates

  • Language-model task-update methods retain or select reusable modules, whereas NestedVLA generates closed-loop VLA action-expert updates with an embedded hypernetwork.The passage contrasts text-only methods including SkillPacks [13] [30] and HoE [29].Source: §2 · pp. 2–3

Model merging

  • Model merging selects or combines existing checkpoint values, while NestedVLA generates a per-skill delta; embodied merging remains underexplored, with ReVLA [6] addressing single-skill domain forgetting instead.The passage groups prior merging as static or dynamic methods and distinguishes multi-skill consolidation from single-skill domain adaptation.Source: §2 · p. 3

3 NestedVLA

NestedVLA consolidates task updates from a shared frozen VLA base by compressing each skill and training one task-conditioned hypernetwork to generate action-expert deltas. At inference, the generated action update and retained VLM update compose with the base to form a deployable task-specific VLA.

3.1 Problem Formulation

  • Task consolidation seeks to use independently fine-tuned skills across downstream scenarios without storing full per-task checkpoints or using expert routing.The policy updates can conflict when aggregated into a single static delta, motivating generation of task-specific parameters instead.Source: §3.1 · p. 3

3.2 Overview

  • The three-stage pipeline independently fine-tunes and compresses task vectors, trains one hypernetwork on action-expert targets, then generates the selected task update for composition with the base.The corresponding compressed VLM update is retrieved at inference.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Skill Acquisition

  • Each task vector is compressed into a sparse skill by retaining module-level top-K magnitudes and applying a global rescale factor to compensate for sparsification energy loss.The deterministic mask is used in training and inference, and the hypernetwork learns dense values only at retained positions.Source: §3.3 · p. 4

3.4.1 Patch-based Parameter Tokenization

  • The hypernetwork regresses over action-expert targets using parameter patches, keeping each token's output dimension bounded while exposing the backbone's structured layout.A single generator absorbs the target set into shared parameters.Source: §3.4 · p. 5, §3.4.1 · p. 5

3.4.2 Hypernetwork Architecture

  • Task and patch features meet after task encoding and broadcast; a shared FC trunk, per-task LoRA path, and dedicated per-task output heads produce patch predictions.The task encoder uses 2D convolutions and average pooling; per-task heads mitigate multi-task interference at additional output-module parameter cost.Source: §3.4.2 · pp. 5–6, Figure 3 · p. 5

3.4.3 Training Objective

  • Training uses a multi-task masked MSE over retained action-expert positions, freezes the base VLA, and trains only hypernetwork parameters.Task-balancing weights compensate for heterogeneous benchmark learnability.Source: Figure 2 · p. 4, §3.4.3 · p. 6

3.5 Skill Generation at Inference

  • At inference, one hypernetwork query generates the task's action-expert update, which is combined with its retrieved compressed VLM update and the frozen base.The resulting model requires no architectural modification, expert routing, or runtime mixture computation.Source: §3.5 · p. 6

4 Experiments

NestedVLA consolidates three independently fine-tuned policies, evaluated on MetaWorld, ManiSkill, and CALVIN, using a shared hypernetwork and task-specific compressed updates. It substantially outperforms model-merging baselines, with ablations examining design choices and analysis locating the remaining performance gap.

4.1 Setup

  • MetaWorld and ManiSkill use success_once, while CALVIN measures success_at_end across five-task chains; evaluations average three seeds.The setup uses π0.5 and independently fine-tunes skills with FlowSDE; MetaWorld has 32 tasks, ManiSkill 25, and CALVIN uses 96 rollouts.Source: §4 · p. 6, §4.1 · p. 7

4.2 Main Results

  • 66.7% average success outperforms every static and dynamic merging baseline by 41–54 pp; ManiSkill reaches 87.2% versus 88.1% for per-task FlowSDE.CALVIN reaches 75.0% success_at_end; the average gap to per-task FlowSDE is −13.8 pp and concentrates almost entirely on MetaWorld.Source: Table 1 · p. 7, §4.2 · p. 8
  • 14.8% average success for parameter averaging; all four static merging baselines reach 0% CALVIN chain completion.TIES, WUDI, and TSV-Merging each underperform averaging.Source: §4.2 · p. 8
  • 25.6% average success is Free-Merging’s best result among dynamic baselines, which cluster within 3 pp despite using 2.09×–2.89× storage.NestedVLA’s total storage is 1.5× for three skills; compressed VLM residuals, masks, and task-specific generator components mean storage still grows with skill count.Source: §4.2 · p. 8

4.3 Ablations

  • Removing Multi-head Output degrades in-mask standardized loss by 0.2–0.5, while removing shared-FC LoRA yields the lowest loss on all three benchmarks.Per-task patch embeddings give less than 0.1 loss difference; removing shared-FC LoRA uses 97% of default parameter cost.Source: Table 2 · p. 8, §4.3 · p. 9
  • Drop-and-rescale at K=10% yields reconstruction errors of 0.30 on the VLM half and 0.19 on the action half, and requires no calibration data.One-bit quantization degrades task success by −71 to −81 pp; adding an SVD residual improves fidelity by at most +0.005 while increasing token budget by an order of magnitude.Source: §4.3 · p. 9
  • Disabling the out-of-mask penalty adds +2.4 pp average success; magnitude weighting and absolute loss each harm performance alone but combine constructively with that change.The standardized objective with the out-of-mask penalty disabled is the default recipe.Source: §4.3 · p. 9

4.4 Where Does the Remaining Gap Come From?

  • MetaWorld reaches 37.9% versus a 72.3% FlowSDE reference, but its K=10% ideal is only 48.4%; raising K to 0.20 lifts that ideal to 60.2%.NestedVLA reaches 99% and 96% of the K=10% ideal on ManiSkill and CALVIN, respectively; the analysis attributes the MetaWorld gap primarily to sparsity budget at skill acquisition.Source: §4.4 · p. 9, Table 3 · p. 9

5 Conclusion

NestedVLA consolidates independently fine-tuned VLA policies, reaching 66.7% average success at 1.5× storage. The conclusion identifies MetaWorld as the remaining gap and adaptive sparsity as a next direction.

5 Conclusion

  • 66.7% average success at 1.5× storage, 41.1 pp above the strongest dynamic baseline, with ManiSkill within 1 pp of per-task fine-tuning.The conclusion says the method approaches the K=10% sparsity-budget ceiling on CALVIN; the MetaWorld gap is bounded by the top-K ideal.Source: §5 · p. 10

Appendix Overview

  • The appendices provide implementation and evaluation details, baseline configurations, storage derivations, compression comparisons, loss ablations, and study limitations.Appendix F gives the VLM-merging sanity test, and Appendix I covers broader-impact considerations.Source: Appendix Overview · p. 16

A Implementation Details

A.1 Backbone and Skill Collection

  • The shared base is π0.5, and three skills are independently fine-tuned on MetaWorld, ManiSkill, and CALVIN.The VLM half has about 3.21 B parameters and the action expert about 0.40 B; static VLM merging sharply degrades performance in the sanity test.Source: §A.1 · p. 16

A.2 Hypernetwork Architecture

  • The action-expert hypernetwork covers 162 weight tensors, tiled into 256×256 patches, with 6,480 patches per task.The tensors span 18 transformer blocks and nine weight types.Source: §A.2 · p. 16

A.3 Training Details

Training uses balanced examples from the three tasks and a 6,000-step AdamW schedule. Experiments fit on one A6000 GPU, while inference adds negligible hypernetwork overhead per skill switch.

A.3 Training Details

  • Training uses AdamW for 6,000 steps, with batch size 3 and one example per task per step.The schedule uses 600-step linear warmup, a 1,500-step cosine decay tail, and task weights of (1.0, 0.4, 1.0) for MetaWorld, ManiSkill, and CALVIN.Source: §A.3 · p. 16
  • Each run takes about 10 hours on one 48 GB NVIDIA A6000 GPU, with peak training memory of 42.6 GB.The paper enables expandable CUDA memory segments to reduce fragmentation under bf16.Source: §A.3 · p. 16
  • The hypernetwork query adds negligible inference overhead and runs once per skill switch.The forward pass through π0.5 dominates inference cost.Source: §A.3 · p. 17

A.4 Evaluation Protocol

Evaluation measures success across three benchmark suites, including full-chain completion on CALVIN. The section describes static and dynamic merging baselines and per-task FlowSDE as a reference ceiling.

A.4 Evaluation Protocol

  • CALVIN ABC-D evaluates 96 five-subtask rollouts using success_at_end, while MetaWorld and ManiSkill use success_once over 256 and 320 trajectories, respectively.MetaWorld covers 32 tasks and ManiSkill 25; results average three evaluation seeds with typical standard deviation of ±2.7%.Source: §A.4 · p. 17

B Baselines Details

  • Static baselines produce one fixed delta, including element-wise averaging, conflict-handling merges, and singular-vector merging.The described methods include Averaging, TIES-Merging, WUDI-Merging, and TSV-Merging.Source: B Baselines Details · p. 18
  • The per-task FlowSDE reference consists of three independently fine-tuned full checkpoints at 3.0× storage.It is described as the performance ceiling for approaches that do not change the underlying RL pipeline.Source: B Baselines Details · p. 18
  • Dynamic baselines retain task-specific masks, signs, scalars, low-rank residuals, or frequency-domain corrections at inference.TALL-Mask selects positions from an averaged delta, EMR adds sign and scale modulators, Twin uses low-rank residuals, and Free-Merging filters task deltas.Source: B Baselines Details · p. 18

C Storage Accounting

NestedVLA’s storage totals 1.5× a full checkpoint, using per-task compressed VLM deltas and generated action-expert updates. At matched budgets, drop-and-rescale is favored over one-bit quantization or added SVD residuals.

C Storage Accounting

  • Static merging methods use 1.00× total storage, while dynamic methods range from 2.09× for TALL-Mask to 2.89× for EMR-Merging.Twin-Merging totals 2.25× and Free-Merging 2.16×; the comparison accounts for per-task auxiliary state.Source: C Storage Accounting · p. 19
  • 1.5× total storage is reported for NestedVLA, about 0.7× smaller than Free-Merging and 1.5× smaller than per-task FlowSDE.The VLM is stored per task as a top-K compressed delta; the action half is generated at inference, with only its precomputed mask stored per task.Source: C Storage Accounting · p. 19

D.1 Operator Comparison at Matched Storage

  • One-bit quantization yields 0.60–0.96 reconstruction error and drops task success by 71–81 pp relative to per-task FlowSDE.The authors attribute the failure to near-isotropic, approximately zero-mean VLA deltas that a single per-tensor scalar cannot capture.Source: §D.1 · p. 20, Table 7 · p. 20
  • Drop-and-rescale at K=10% and α=1.1 has reconstruction errors of 0.30 on the VLM half and 0.19 on the action half, retaining at least 96% of per-task FlowSDE on ManiSkill and CALVIN.It is the only compared operator that requires no calibration data.Source: §D.1 · p. 20, Table 7 · p. 20
  • Adding an SVD residual improves absolute reconstruction error by at most 0.005 at 10% residual storage, while increasing the token budget by an order of magnitude.The authors therefore adopt drop-and-rescale alone throughout NestedVLA.Source: §D.1 · p. 20

D.2 Keep-Ratio Sweep

The keep-ratio sweep finds task-dependent optima and a persistent MetaWorld gap to per-task fine-tuning. Supplementary diagnostics report weak alignment between action-expert and VLMbody deltas, while a separate test examines static VLM consolidation with ground-truth action experts.

D.2 Keep-Ratio Sweep

  • ManiSkill peaks at K=0.15 (91.6% success_once, +3.4 pp over FlowSDE), while CALVIN peaks at K=0.20 (85.4% success_at_end, +4.2 pp over FlowSDE).MetaWorld improves monotonically through K=0.20 but reaches only 60.2%, below FlowSDE’s 72.3%.Source: §D.2 · pp. 20–21, Table 8 · p. 21
  • ManiSkill’s non-monotonic K pattern is interpreted as a signal-to-noise sweet spot: smaller K drops important parameters, while larger K reintroduces low-magnitude noise.The same sweet-spot behavior is consistent with the rescale sweep, which is not shown.Source: §D.2 · p. 21
  • Across all three benchmarks, the embedding layer is fully dropped at every tested keep-ratio, while action input/output projections retain 44–64%.This pattern motivates module-level rather than per-tensor top-K selection.Source: §D.2 · p. 21

E Supplementary Weight-space Diagnostics

  • Action-expert and VLMbody deltas have low pairwise alignment, although these source-checkpoint statistics alone do not establish a causal explanation of rollout failure or objective superiority.The diagnostics use concatenated-delta cosine similarity and sign agreement on valid coordinates.Source: E Supplementary Weight-space Diagnostics · p. 21

F.1 Setup

  • The VLM static-merging test replaces only the VLM in each per-task FlowSDE checkpoint, retains its ground-truth action expert, and measures only its corresponding benchmark.The setup does not measure cross-task transfer; it tests TIES and WUDI without mask or rescale corrections.Source: F VLM Static Merging Sanity Test · p. 21, §F.1 · pp. 21–22

F.2 Results

Static VLM consolidation performs poorly even when each benchmark retains its ground-truth action expert. The authors therefore keep per-task VLM storage and leave generating VLM deltas with a hypernetwork for future work; the proposed full-model extension is unevaluated.

F.2 Results

  • CALVIN success_at_end falls to 0% with both VLM merge methods, and MetaWorld penalties exceed −60 pp for both methods relative to FlowSDE.Only WUDI on ManiSkill stays within 10 pp of the FlowSDE reference; it is the sole positive cell among six.Source: §F.2 · p. 22
  • The authors conclude that static VLM consolidation is not viable as a standalone design, even with a ground-truth per-task action expert, and retain per-task VLM storage.They attribute the diagnostic to sensitivity of cross-task language-instruction grounding and leave hypernetwork-generated VLM deltas to future work.Source: §F.2 · p. 22
  • The unevaluated v2 extension would generate both VLM and action-expert updates, removing stored VLM values but substantially enlarging the output space and potentially requiring more generator capacity and metadata.All reported results and storage measurements concern v1.Source: §F.2 · p. 22

G Loss-Side Ablation

The loss-side ablation identifies the out-of-mask penalty as the most effective single objective adjustment. Other loss-space and magnitude-weighting choices interact non-monotonically, and the standardized loss with no out-of-mask penalty is the final default.

G Loss-Side Ablation

  • Setting wout = 0 improves average success by +2.4 pp, the largest single-knob effect measured.Positions outside the mask are removed at inference, so penalizing them during training allocates capacity to values that do not contribute to the deployed delta.Source: G Loss-Side Ablation · p. 22
  • Absolute loss space alone costs −2.9 pp and magnitude weighting at p=1 costs −5.4 pp, but combining them recovers most of the loss at −0.6 pp.The standardized loss with wout = 0 beats every other tested configuration and is the final default; Table 11 uses a single-seed evaluation.Source: G Loss-Side Ablation · p. 23, Table 11 · p. 23

H Limitations

The evaluation covers three simulated skills and does not test physical-robot transfer. The paper also discusses potential benefits and inherited risks of reducing the deployment cost of multi-task robotic policies.

H Limitations

  • The evaluation consolidates M=3 skills; scaling training time, hypernetwork capacity, and output quality as M grows by an order of magnitude remains unmeasured.Adding a skill currently requires extending the per-task patch-embedding table and multi-head output, then retraining the hypernetwork.Source: H Limitations · p. 23
  • All reported benchmarks are simulated, so transfer to physical hardware is unevaluated.The broader-impact discussion states that no physical robot was operated in the study.Source: H Limitations · p. 23, I Broader Impact · p. 23

I Broader Impact

  • Lowering multi-task policy storage and serving costs may amplify beneficial uses as well as risks already present in the underlying VLA stack.The deployed behavior inherits biases and failure modes of the source FlowSDE checkpoints.Source: I Broader Impact · p. 23

2. Limitations

The paper identifies limits in the scale, update process, storage, and evaluation setting of its consolidation experiments. The released code also omits materials needed to reproduce every reported result.

2. Limitations

  • Consolidation is evaluated on only three skills, with scaling left unmeasured.The authors also report that all evaluations are simulated.Source: §2 · p. 24
  • Adding a new skill requires retraining the hypernetwork.Source: §2 · p. 24
  • The VLM update remains stored per task, contributing approximately 0.35× of the 1.5× storage budget.Source: §2 · p. 24
  • All evaluations are simulated, so the reported evaluation setting does not include real-robot tests.Source: §2 · p. 24
  • The code release lacks benchmark-specific training and evaluation pipelines, simulator configurations, and checkpoints needed to reproduce every reported result.The core algorithm is publicly available with installation instructions and a minimal example.Source: §2 · p. 26
  • NestedVLA inherits biases from its source checkpoints and introduces no new training data.Source: §2 · p. 28

Related papers

Loading 2610.23271v3…