PaperlayerView exact revision on arXiv

Source-linked AI summary

Robotic Policy Adaptation via Weight-Space Meta-Learning

Christian Bianchi, Siamak Yousefi, Alessio Sampieri, Andrea Roberti, Luca Rigazio, Fabio Galasso, Luca Franco

arXiv:2606.07217v1cs.ROcs.CVcs.LG

TL;DR

VLA adaptation commonly requires task-specific action-labeled demonstrations and fine-tuning, limiting scalable deployment. WIZARD generates LoRA adapters for a frozen VLA from language and video evidence in one forward pass, without action labels or test-time optimization. It improves unseen-task performance, data efficiency, and warm-start adaptation on LIBERO, while remaining limited by ambiguous evidence and single-task generation for compositional tasks.

  • Problem

    VLA models often experience severe performance drops on unseen tasks, while standard adaptation requires task-specific demonstrations with action labels and additional fine-tuning.

  • Method

    WIZARD uses weight-space meta-learning to generate task-specific LoRA updates for a frozen VLA from a language instruction and short demonstration video.

  • Results

    WIZARD reaches 90% success without task-specific gradient updates, and generated warm-start weights reach the 96% expert success rate in 70 rather than 90 steps.

  • Takeaways & Limitations

    WIZARD enables zero-shot adaptation without action labels or test-time optimization, with reported generalization, data efficiency, and effective warm-starts on LIBERO.

  • Takeaways & Limitations

    WIZARD depends on task-evidence quality and targets single-task expert generation, limiting performance on long-horizon compositional tasks such as LIBERO-10.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are emerging as a promising paradigm for robotic manipulation, enabling general-purpose policies trained from large corpora of demonstrations and action labels. However, adapting these models to new tasks still typically requires task-specific demonstrations, action annotations, and additional fine-tuning, making deployment costly and difficult to scale. We propose WIZARD, a weight-space meta-learning framework that sidesteps task-specific fine-tuning by generating task-specific LoRA parameters for a frozen VLA policy. Given only a language instruction and a short demonstration video, WIZARD predicts the corresponding adaptation weights in a single forward pass, without target-task action labels or test-time optimization. During meta-training, WIZARD learns to map task evidence directly to expert LoRA updates, capturing relationships between tasks in weight space. Experiments on LIBERO show that WIZARD improves performance by up to ~2x on unseen dataset collections and up to ~14x on unseen tasks. On a Franka Emika Panda, WIZARD consistently improves over a real-domain adapted baseline, showing that generated adapters provide task-level specialization beyond simulation.

1 Introduction

VLA policies often require action-labeled demonstrations and task-specific fine-tuning to recover performance on unseen tasks. WIZARD instead generates task-specific LoRA adapters directly from language and video evidence, avoiding test-time optimization.

  • Standard adaptation requires task-specific demonstrations with action labels and additional fine-tuning for each new task.LoRA reduces storage costs but does not remove the need for labeled demonstrations or optimization.
  • Unseen-task deployment can cause severe VLA performance drops, motivating task-specific adaptation.π0.5 achieves 0% on LIBERO-Spatial without adaptation, versus 19% after related-dataset fine-tuning and 94% with direct test-time fine-tuning.
  • WIZARD predicts task-specific LoRA updates from a language instruction and short demonstration video by learning task relationships in weight space.Its meta-network maps task evidence to expert adapter weights rather than optimizing the policy in action space.
  • WIZARD generates adapters for frozen VLA policies in a single forward pass, without action labels or test-time optimization.The generated adapter is applied to the frozen backbone to produce a task-specialized policy.
  • WIZARD is presented as the first weight-space meta-learning framework that generates task-specific LoRA adapters for frozen VLA policies.

2 Related works

Related work spans large-scale VLA generalization, conditioning-based adaptation, meta-learning, and weight generation. WIZARD addresses the underexplored combination of large VLA policies, weight-space inference, and adaptation without action labels or online optimization.

  • Large pretrained VLA policies remain vulnerable to failures on genuinely unseen tasks and environments outside their training distributions.Multi-task training can improve robustness, while LoRA enables parameter-efficient specialization.
  • In-context prompting and few-shot conditioning adapt behavior from small inference-time contexts without updating model parameters.VIMA uses language and visual demonstrations, while related methods infer tasks from minimal examples.
  • Meta-learning leverages relationships across tasks and datasets for fast adaptation but remains underexplored for large robotic VLA policies.The stated gap particularly concerns settings without action labels or online optimization.
  • Parameter-generation methods synthesize model weights directly, supporting fast model instantiation, architecture search, and parameter prediction.HyperNetworks are identified as a foundational deep-learning approach in this line of work.
  • Robotic parameter-generation methods have used full policy parameters, privileged goal inputs, episode-level diffusion, or action-space supervision.WIZARD is contrasted with these approaches in its use of compact LoRA updates and non-privileged task evidence.

3 Method

WIZARD formulates robotic adaptation as weight generation: task evidence is encoded into a representation, a meta-network predicts expert LoRA updates, and the updates are applied to a frozen VLA backbone. Its training design models multimodal structure, parameter scale, and directional alignment.

  • 3.1 Meta-learning LoRA for VLA policies: WIZARD treats adaptation as generating task-specific parameter updates for a frozen VLA backbone without test-time gradient optimization.
  • 3.1 Meta-learning LoRA for VLA policies: The meta-training set pairs task embeddings z with relative expert LoRA parameters ΔW.Expert policies are obtained by adapting the frozen backbone with LoRA.
  • 3.1 Meta-learning LoRA for VLA policies: Task evidence uses language prompts and visual observations, while state-action trajectories train expert policies rather than serving as meta-inference inputs.
  • 3.1 Meta-learning LoRA for VLA policies: The meta-network maps task representations to predicted LoRA updates and is trained with a weight-space reconstruction objective.
  • 3.1 Meta-learning LoRA for VLA policies: At inference, an encoder forms a task embedding from a prompt and short visual demonstration, and the meta-network generates an adapter in one forward pass.The update is injected into the frozen backbone for zero-shot execution without state-action annotations or test-time optimization.
  • 3.2 Design principles for meta-network training in robotics: WIZARD tensorizes LoRA updates across layers and vision, language, and action components to preserve architectural boundaries while modeling modalities jointly.
  • 3.2 Design principles for meta-network training in robotics: The meta-network jointly predicts normalized updates and per-layer statistics because heterogeneous parameter magnitudes can destabilize training.Without per-layer normalization, training diverges.
  • 3.2 Design principles for meta-network training in robotics: Training combines MSE reconstruction, scale matching, and cosine alignment losses to enforce magnitude, structural, and directional consistency.

4 Experiments

Experiments evaluate WIZARD under held-out LIBERO shifts, adaptation-efficiency tests, and real-world deployment, showing gains in zero-shot success, warm-start speed, and task-level performance.

  • Experimental protocol: WIZARD is evaluated on LIBERO under a held-out distribution protocol spanning Spatial, Object, Goal, and long-horizon 10 datasets.The meta-network is trained on three datasets and evaluated on the remaining unseen dataset without gradient updates.
  • Zero-shot task adaptation: WIZARD reaches 0.40 average success on LIBERO-Spatial versus 0.19 for MT-VLA, while improving LIBERO-Goal from 0.14 to 0.22 and LIBERO-Object from 0.01 to 0.03.These are zero-shot results under held-out distribution shifts.
  • Task-level adaptation behavior: WIZARD recovers task-specific behaviors across shifts, including 0.90 on LIBERO-Spatial Task 1, 0.86 on LIBERO-Goal Tasks 5 and 9, and gains on difficult tasks where MT-VLA remains near-zero.The reported behaviors include spatial priors, visual-shift recovery, and task-specific affordances.
  • Task-level adaptation behavior: LIBERO-10 full-task zero-shot completion remains 0.00, although subtask metrics improve over MT-VLA, indicating a compositional-sequencing boundary.The results suggest isolated skill execution is stronger than sequential composition.
  • Real-world evaluation: On real Franka Emika Panda tasks, WIZARD raises average success from 0.22 to 0.41 over a real-domain adapted π0.5 baseline.Both methods share the same real-domain initialization, and gains are reported across all five tasks.
  • Adaptation efficiency: WIZARD reaches 90% success without task-specific gradient updates, matching MT-VLA after about 25 demonstrations, while generated weights reach 96% expert success in 70 rather than 90 steps when fine-tuning.The data-efficiency result is reported on LIBERO-Spatial Task 1; the warm-start result is reported on LIBERO-Spatial Task 10.

5 Conclusions

The conclusion presents WIZARD as a weight-space framework for zero-shot VLA adaptation and identifies evidence quality and long-horizon composition as key boundaries.

  • Contributions: WIZARD generates LoRA adapters for frozen VLA policies from language and video evidence, enabling zero-shot adaptation without action labels or test-time optimization.The conclusion summarizes strong generalization, data efficiency, and warm-starts on LIBERO.
  • Limitations: Ambiguous or incomplete demonstrations may produce unreliable adapters, and single-task expert generation limits performance on long-horizon compositional tasks such as LIBERO-10.Future work is proposed around richer task representations and expert composition.

Appendix

The appendix supplies supplementary analyses, qualitative rollouts, task definitions, implementation details, and accompanying videos.

  • Supplementary analyses: The appendix reports analyses and ablations covering embedding structure, conditioning granularity, support size, modality, loss components, generalization, and scaling costs.It also includes implementation details such as expert LoRA training.
  • Qualitative and implementation material: Additional qualitative rollouts cover LIBERO and real-world deployment, while semantic prompts and task definitions support evaluation documentation.Supplementary videos are provided through the accompanying index.html file.

A Additional analysis

Additional analyses show that task-level multimodal conditioning produces structured weight representations, while scale-aware objectives and compact LoRA generation support performance and feasible scaling.

  • Topological structure: Task embeddings form distinct clusters across tasks and datasets, then transform into structured representations in weight space.The analysis examines both the conditioning space and the meta-network’s final latent representation.
  • Conditioning granularity: 0.40 average success from task-level conditioning exceeds 0.11 from dataset-level conditioning on LIBERO-Spatial.Dataset-level conditioning reaches 0.11, below the π0.5 baseline of 0.19; task-level conditioning improves by 29 percentage points over dataset-level performance.
  • Support size: A single support episode provides sufficiently informative task evidence, with performance remaining largely stable as support size varies.Support embeddings are formed by averaging randomly sampled episodes from the same task.
  • Modality ablation: 0% success without video conditioning contrasts with 18% for video-only and 33% for full multimodal input.The results assign text the high-level task intent and video the geometric information needed for adaptation.
  • Training loss: 0.00 average success with reconstruction-only training rises to 0.27 after adding the scale-aware objective.The full objective combines reconstruction, cosine alignment, and scale-aware losses; the cosine term further improves the combined objective.
  • Speed and memory: 23.5GB at rank r = 512, while full-model generation reaches out-of-memory, shows why compact LoRA updates remain necessary for scaling.Generation remains fast across ranks, but memory—not runtime—is the main bottleneck.

B Additional qualitative results

Qualitative rollouts illustrate task-specific behavior across simulated LIBERO tasks and physical Franka experiments, while LIBERO-10 exposes an unresolved challenge in composing sequential skills.

  • LIBERO-Spatial: Generated policies recover precise kinematic priors and collision-aware trajectories for atypical spatial configurations in LIBERO-Spatial.Reported rollouts retrieve bowls from different scene positions and place them on plates.
  • LIBERO-Object: Generated policies identify novel objects among distractors and execute appropriate pick-and-place actions under appearance variation.Examples include alphabet soup, cream cheese, and ketchup in LIBERO-Object.
  • LIBERO-Goal: Language instructions drive specialized affordances such as placing objects, inserting items into containers, and activating the stove.The LIBERO-Goal rollouts demonstrate synthesis of manipulation primitives from task descriptions.
  • LIBERO-10: LIBERO-10 rollouts separate each two-skill task into constituent stages, leaving successful completion of the composed behavior for future work.This visualization isolates intermediate sub-actions rather than demonstrating the full sequential objective.
  • Real-world experiment: WIZARD grounds target objects from language and camera observations and executes grasping behavior on a physical Franka Emika setup.The real-world examples involve picking up an apple, banana, and cup.

C.1 Teleoperated Data Acquisition

Teleoperated demonstrations were collected through a handheld interface that commands Cartesian motion, while impedance control and ROS-based streaming support compliant execution and policy deployment.

  • Demonstration collection: Human operators guided the end effector through a handheld input device using the Franka Cartesian impedance controller.The controller regulates position and orientation in task space and enables compliant motion during demonstrations.
  • Teleoperation interface: Cartesian velocity and pose increments were tracked in real time, producing smooth motion and contact stability.The resulting demonstrations contain synchronized robot-state streams including joint positions, velocities, and applied torques.
  • System architecture: ROS1 separates workstation inference from low-level robot control while synchronizing robot states and sensory observations.The structured state representation is passed to the inference pipeline.
  • Policy execution: Remote inference returns short-horizon chunks of joint and gripper actions, which are executed open-loop before requesting an update.Joint commands are published through ROS and gripper actions use the standard Franka interface.

D Semantic prompts and task definitions

The semantic prompt suite defines task families and their intended generalization challenges, spanning object grounding, spatial relations, goal-directed interactions, and long-horizon composition.

  • Prompt conditioning: The prompt strings are explicitly specified because π0.5 uses them to project visual observations into the task latent space.This defines the semantic input used for zero-shot conditioning and evaluation.
  • LIBERO-Spatial: LIBERO-Spatial evaluates relative positional geometry, with ‘wooden cabinet’ identified as an unseen concept associated with performance collapse.The dataset includes object-placement tasks phrased with spatial relations and scene references.
  • LIBERO-Object: LIBERO-Object isolates visual object grounding while keeping the kinematic intent invariant across target objects.The tasks use pick-and-place instructions involving objects such as alphabet soup, cream cheese, and ketchup.
  • LIBERO-Goal: LIBERO-Goal tests goal-directed interactions, including opening drawers, placing objects, and turning on the stove.Tasks containing the ‘open’ primitive are described as semantic boundaries for zero-shot synthesis when held out.
  • LIBERO-10: LIBERO-10 introduces sequential compositions of multiple primitives to test temporal consistency in generated LoRA parameters.Examples combine actions such as placing two objects, turning on the stove, closing drawers, and manipulating multiple mugs.

E Extended implementation details

WIZARD’s extended implementation combines structured LoRA tokenization, a multimodal convolutional meta-network, and masked multi-objective supervision to generate adaptation parameters. Training uses distributed, memory-efficient procedures, including two-stage expert construction, modality slicing, streaming, and mixed-precision optimization.

  • LoRA fine-tuning and variance generation: A two-stage LoRA pipeline first trains domain-specific π0.5 experts, then generates diverse high-frequency weight variance around the expert solution for meta-training.Phase 1 freezes the base model and trains LoRA adapters; Phase 2 harvests capable configurations near the optimization minimum.
  • Data processing and LoRA tokenization: Pi05LoRATokenizer2D maps heterogeneous one-dimensional LoRA parameters into standardized 16 × 512 token grids using chunking, padding, normalization, and scale tracking.The tokenizer accommodates weight matrices with substantially different dimensions while preserving continuous parameter information.
  • Meta-network architecture: The meta-network aligns 2048-dimensional VLA embeddings into a 168 × 16 × 512 representation and processes them with decomposed 3D convolutional blocks.The decoder uses parallel width-first and height-first convolutions, followed by averaging, bias shifting, SiLU activation, and dropout with p = 0.15.
  • Meta-network architecture: A parallel scale predictor maps flattened 16 × 512 embeddings through a 1024-dimensional bottleneck to output four-parameter scale vectors for 5,929 tokens.The predicted vectors track μ and σ for both up and down projections and use SiLU activation with dropout p = 0.15.
  • Objective formulation: WIZARD uses a composite objective with masked MSE over valid token positions, dataset-specific criterion weighting, scale loss, and cosine loss.The valid-token losses exclude padded regions, while weighting balances heterogeneous architectural modules.
  • Optimization and training dynamics: Distributed implementation uses streaming data, modality-specific temporal slicing, bf16 mixed precision, 8-bit AdamW, and Gaussian condition noise with σ = 0.05.These procedures address parameter I/O, memory overhead, and controlled conditioning during training.

Topological parameter sorting and deterministic rulebook

WIZARD imposes a deterministic, priority-based topological ordering on LoRA parameters before converting them into unified token grids. The rulebook groups parameters by architectural modality and functional depth, while chunking and padding handle irregular matrix sizes.

  • Motivation: Strict topological ordering preserves functional relationships among parameters because random or lexicographical sorting can separate dependent layers and cause mode collapse.The ordering is therefore treated as a structural requirement for convolutional generation in weight space.
  • Tokenization rulebook: Asymmetric chunking and zero-padding accommodate LoRA matrices with irregular dimensions after topological sorting.The rulebook allocates separate up and down chunks for the A and B LoRA matrices, respectively.
  • Tokenization rulebook: The complete tokenization rulebook determines the 5,929 discrete tokens generated by the meta-network.Table 8 specifies the tokenization hyperparameters, including the up-chunk and down-chunk allocations.
Loading 2606.07217v1…