PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts

Taewook Kang, Taeheon Kim, Donghyun Shin, Jonghyun Choi

arXiv:2607.00666v1cs.ROcs.CVcs.LG

TL;DR

VLA models often need costly target-domain demonstrations to preserve learned task performance under environmental shifts. DART adapts them by adding a filtered domain vector extracted through weight arithmetic from one-shot updates, and outperforms existing methods across diverse simulated and real-world visual and embodiment shifts.

  • Problem

    VLA models degrade under environmental shifts, while existing adaptation approaches often require extensive expert demonstrations for each target-domain task.

  • Method

    DART extracts and adds filtered domain-specific directions from one-shot weight updates, removing task-specific directions through analogy-based weight arithmetic and subspace filtering.

  • Results

    DART consistently outperforms existing VLA adaptation baselines in one-shot simulated and real-world experiments across diverse visual and embodiment shifts.

  • Takeaways & Limitations

    DART provides a one-demonstration approach for adapting VLA models across diverse visual and embodiment shifts without additional architectural changes.

  • Takeaways & Limitations

    DART struggles under severe shifts, and its scalar coefficient requires a small hyperparameter search.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models often fail to perform the same learned tasks under environmental shifts, such as changes in camera pose and shifts to a different but similar robot (e.g., from Panda to UR5e). Adapting these models to the shifted environment (i.e., target domain) often requires training on multiple demonstrations for each task, which are costly to collect. To reduce the burden of data curation and training, we propose an analogy-based method that adapts VLA models under environmental shifts through weight vector arithmetic with domain-specific information addition, named Domain ARiThmetic (DART). Unlike prior approaches, DART requires collecting only a single demonstration, enabling efficient adaptation. To accurately isolate domain-specific information for addition, DART performs subspace alignment between singular components in weight vectors to filter out noisy components. In both simulated and real-world experiments, DART outperforms existing VLA adaptation methods in one-shot scenarios across diverse visual and embodiment shifts. Code is available at https://github.com/snumprlab/dart.

1 Introduction

VLA models degrade under environmental shifts, motivating one-shot adaptation from a single demonstration. DART extracts and adds domain-specific weight directions, using subspace alignment to suppress task-specific and noisy components, and outperforms prior adaptation methods across simulated and real-world shifts.

  • Motivation: Environmental shifts in camera pose, sensor calibration, or robot embodiment substantially degrade VLA performance on learned tasks.The shifts occur when deploying policies outside their trained source domains.
  • Motivation: One-shot adaptation aims to adapt a policy under environmental shifts using only one demonstration from a single task.This targets practical settings where collecting task-wise demonstrations at scale is infeasible.
  • Analysis: One-shot fine-tuned update-vectors are predominantly task-specific, while smaller domain-specific directions remain present and appear additively composable.Subspace alignment analysis supports decomposing updates into shareable task and domain directions.
  • Method: DART extracts a reusable domain vector by subtracting source-domain updates from target-domain updates and adds it to the base model without architectural changes.The method transfers multi-task capabilities to the target domain through analogy-based weight arithmetic.
  • Method: Subspace filtering removes misaligned basis vectors, while subspace scaling down-weights noisy domain vectors according to source-target subspace alignment.These operations address source-domain artifacts and fine-tuning noise introduced by direct subtraction.
  • Results: DART outperforms existing VLA adaptation baselines in one-shot experiments across diverse visual and embodiment shifts.Experiments use π0.5 and π0-FAST in both simulated and real-world settings, with fast, hyperparameter-robust adaptation and multi-domain merging.

2 Related Work

VLA models achieve strong performance across many tasks but often require adaptation in novel environments to avoid degradation. Existing weight-space methods emphasize merging, while analogy remains limited and cannot selectively transfer domain knowledge while preserving other capabilities.

  • VLA adaptation: VLA models integrate pretrained vision-language backbones with large-scale robotic datasets and perform strongly across diverse tasks.Examples include RT-2, OpenVLA, and the π series.
  • VLA adaptation: VLA models often require adaptation in novel environments to avoid performance degradation, commonly using diverse augmentations with additional fine-tuning.
  • Task Arithmetic: Task Arithmetic manipulates models through weight-space merging to compose capabilities and analogy to estimate parameter changes that transfer target properties.
  • Task Arithmetic: Analogy remains limited to direct subtraction, whereas VLA research has focused on merging for generalization or skill composition.
  • VLA adaptation: Merging cannot selectively transfer specific capabilities such as domain knowledge while preserving others, motivating analogy for efficient VLA adaptation.

3 Preliminaries

The preliminaries define VLA policies, source and shifted target domains, and a one-shot adaptation setting. They formalize adaptation as behavior-cloning fine-tuning from a base policy and represent the resulting change through target-domain weight update vectors.

  • Problem Setup: A VLA policy πθ maps observations and task instructions to an action distribution at each time step.Observations may include third-person and wrist-camera images, while instructions may be language prompts.
  • Problem Setup: The target domain Etgt differs from the source domain Esrc through shifts such as camera viewpoint or robot embodiment.The setting considers a single or small number of environmental shifts.
  • Problem Setup: A base policy θ0 trained on source-domain tasks is adapted using one target-domain demonstration for a single task per environment.The demonstration dataset is Dm,tgt for task Tm ∈ T.
  • Adaptation Objective: The objective is to obtain θ∗ that performs well across all tasks in the shifted target domain despite single-task supervision, using behavior-cloning fine-tuning initialized from θ0.The resulting θm,tgt is evaluated across all tasks in T.
  • Weight Updates: For each layer l, the target-domain adaptation is represented as a weight update vector defined by Δ^(l)m,tgt = θ^(l)m,tgt − θ^(l)0.This formulation builds on Task Arithmetic and characterizes how fine-tuning changes model weights.

4 Analysis of One-shot Fine-tuning Failures

One-shot fine-tuning performs well on the adaptation task but generalizes poorly to held-out tasks because its updates are dominated by task-specific directions. Analysis reveals reusable domain-specific directions that combine additively with task directions in weight space.

  • 4.1 One-shot Fine-tuning Behavior: One-shot fine-tuning preserves high performance on the adaptation task but generalizes poorly to other held-out tasks.This failure is observed on LIBERO with π0.5 under a target camera viewpoint.
  • 4.1 One-shot Fine-tuning Behavior: Update-vectors for the same adaptation task strongly align across domains, showing that one-shot updates are dominated by task-specific directions.Vectors targeting the same domain across different tasks also overlap slightly more than vectors targeting different domains.
  • 4.1 One-shot Fine-tuning Behavior: Domain-shared components remain present in one-shot updates, indicating that domain knowledge can be learned through one-shot fine-tuning.The observed domain component is consistent across tasks despite the stronger task-specific alignment.
  • 4.2 Additive Task-Domain Directions: The composed estimate b∆m,tgt = ¯∆m + ¯∆tgt − ¯∆ achieves the highest alignment with each update-vector, supporting linearly combinable task and domain directions.Alignment scores across 16 update-vectors have low standard deviation, indicating consistent prototype directions.
  • 4.2 Additive Task-Domain Directions: Similar environmental changes produce similar update directions, while combined shifts partially reuse directions learned for each shift alone.This indicates that different environment changes correspond to distinct, reusable directions in weight space.

5 DART: Domain ARiThmetic

DART adapts a base VLA policy to a shifted target domain from one target demonstration by isolating domain-specific directions from source and target update-vectors. It improves extraction through subspace filtering and alignment-based scaling before adding the refined domain vector to the base policy.

  • Domain vector extraction: DART extracts a target-domain direction from one target update-vector by subtracting task directions estimated with a same-task source demonstration.The source task can be selected from the source dataset, and its corresponding target-domain expert demonstration collected to guarantee task matching.
  • Domain vector extraction: Subtracting source and target update-vectors neutralizes shared task-specific directions, leaving domain-specific directions that encode the environmental shift.The source and target updates are learned from the same adaptation task and therefore primarily share common task components.
  • Subspace filtering: Subspace filtering retains source basis vectors aligned with the target subspace, preventing source-specific noise from corrupting the extracted domain vector.The method uses left-singular-vector column spaces and overlap energy, with a dynamic cutoff based on the subspace alignment score γ.
  • Alignment-based scaling: When updates are fundamentally misaligned, DART scales the filtered difference by alignment score γ to down-weight noise-dominant or irrelevant domain vectors.Filtering alone may be insufficient as γ → 0, so the refined vector is γ · (∆m,tgt − filtered ∆m,src).
  • Policy adaptation: DART adapts the base policy by adding α · refined domain vector, where α controls adaptation strength while preserving the base policy’s multi-task capabilities.The resulting policy transfers the base policy to the target domain while retaining its inherent multi-task capabilities.

6 Experiments

Experiments evaluate DART under simulated visual shifts, real-world viewpoint shifts, and cross-embodiment transfer, primarily using π0.5 and additionally π0-FAST. Across these settings, DART consistently outperforms one-shot adaptation baselines while supporting domain transfer from a single demonstration.

  • Experimental Settings: Experiments cover LIBERO visual shifts, MimicGen Panda-to-UR5e transfer, and five real-world UR10e tasks under viewpoint shifts.LIBERO includes novel viewpoints and combined noise and illumination perturbations; MimicGen reports progress and success rates.
  • Architectural Generality: DART consistently outperforms all baselines on π0-FAST, indicating that update-vector arithmetic generalizes beyond π0.5’s flow-matching architecture.π0-FAST uses autoregressive action-token generation.
  • Visual-Shift Results: DART outperforms all baselines across diverse novel viewpoints and combined visual perturbations, with applying the target domain vector to θ0 yielding a 24.6 percentage-point gain.Baselines include Zero-shot, One-shot FT, FLA, and RETAIN; the results support preserving multi-task capabilities while adapting the entire model.
  • Transfer Results: DART also bridges the Panda-to-UR5e physical domain gap and achieves high success rates across all five real-world tasks from one Stack Cube demonstration.Real-world evaluation uses a UR10e robot under third-person viewpoint shifts, while MimicGen evaluates Stack and Stack Three transfer.
  • Additional Analyses: Ablations show that analogy-based arithmetic improves over One-shot FT even without subspace alignment, while subspace alignment improves stability across scaling coefficients and fine-tuning steps.One-shot FT degrades over time from catastrophic forgetting, whereas DART maintains strong performance at small fine-tuning steps.

7 Limitation

DART’s performance degrades under severe environmental shifts, a challenge shared by one-shot methods. Its domain-vector extraction and scalar scaling remain areas for improvement despite stability across a wide range of coefficient values.

  • Severe shifts: DART’s performance degrades under severe shifts such as Large viewpoint, a challenge shared by all one-shot methods.Future work may improve reliable domain-vector extraction or base-model training and fine-tuning.
  • Scaling: The scalar coefficient α requires a small hyperparameter search, although DART remains stable across a wide range of values and generalizes across environments.Hyperparameter-free per-layer adaptive scaling is left for future practical real-world applications.

8 Conclusion

DART adapts VLA models to environmental shifts using only a single demonstration by adding filtered domain-specific directions through weight arithmetic. Simulation and real-world evaluations show consistent improvement across diverse visual and embodiment shifts.

  • DART adapts VLA models to environmental shifts with only a single demonstration collection.
  • DART adds filtered domain-specific directions isolated through weight arithmetic, motivated by an approximately additive decomposition of one-shot fine-tuned parameters.
  • Extensive simulation and real-world evaluations show DART consistently improves adaptation across diverse visual and embodiment shifts.

Supplementary Material … A.3 Accelerating DART with Randomized SVD

The supplementary material details DART’s algorithm, source-demonstration assumptions, task-matching robustness, and implementation protocols. It also presents randomized SVD as an acceleration that preserves comparable performance while reducing computational cost.

  • Supplementary Material: The supplementary material covers DART’s algorithm, source-demonstration justification, implementation details, experimental setups, and additional analyses.It includes Sections A–D, spanning algorithm design, baselines, model configurations, simulation and real-world protocols, and extended results.
  • A.1 Algorithm: DART computes source- and target-domain one-shot update vectors, extracts domain information in aligned subspaces, and adapts the base policy by adding a scaled target domain vector.Algorithm 1 fine-tunes source and target policies, computes update differences, applies layer-wise SVD-based extraction, and returns θ∗←θ0 + α · ˜δtgt.
  • A.2 Justification for Using Source-domain Demonstrations: DART assumes access to the source training dataset and at least one source-domain demonstration for the adaptation task.This assumption is motivated by large-scale open-source robotic datasets and task-wise fine-tuning demonstrations commonly used for multi-task VLA models.
  • Obtaining the same adaptation task across domains.: When exact task lookup is difficult, DART can select the adaptation task from the source dataset first and then use the corresponding target-domain demonstration.This provides a practical alternative for large or weakly organized source datasets where direct task matching is unavailable.
  • Robustness to imperfect task matching: Similarity-based retrieval is a viable fallback when only approximate task matches are available.Using a different task degrades performance, whereas selecting a similar task by feature cosine similarity consistently outperforms random selection.
  • Takeaway.: Source-domain demonstrations are a mild practical requirement that can be met through source-data access or by selecting the task from the source side and collecting a target demonstration.Approximate matching remains usable through similarity-based retrieval, with potential for further improvement from stronger task representations.
  • A.3 Accelerating DART with Randomized SVD: Randomized SVD replaces full per-layer decompositions because DART’s SVD computations can become expensive for larger models.The approximation estimates the top-r singular subspace instead of performing a full decomposition.
  • A.3 Accelerating DART with Randomized SVD: O(mn min(m, n)) to approximately O(mnr) reduces the dominant per-layer cost when r ≪ min(m, n), while maintaining comparable performance to full SVD.Table 8 reports success rate and runtime for novel viewpoints using randomized SVD with target rank r=256.

A.4 Why Do One-Shot Update-Vectors Decompose into Task and Domain-Specific Directions

One-shot fine-tuning can produce partially disentangled weight directions because VLA tokens carry overlapping but distinct task and domain information. Prototype-induced feature shifts show that Task directions affect text and task-relevant objects, whereas Domain directions affect backgrounds.

  • Motivation: VLA inputs contain overlapping token subsets primarily associated with task information or domain information, enabling one-shot fine-tuning to induce partially disentangled weight directions.Task information includes language instructions and task-relevant objects; domain information includes background appearance, camera viewpoint, and robot embodiment.
  • Empirical inspection: Prototype updates are evaluated by measuring last-layer token feature shifts after addition to the base model θ0.The shift is the L2 distance between features before and after adding each prototype.
  • Empirical findings: The Task prototype predominantly shifts text and task-relevant object tokens, while the Domain prototype predominantly shifts background tokens.These patterns are reported in Table 9 and Figure 7.
  • Interpretation: The observed prototype-specific shifts align with findings that distinct tasks activate separable column subspaces and weight interpolation changes features and outputs approximately linearly.This supports interpreting task and domain prototypes as functionally meaningful directions.

B Details on Baseline Methods … C Experiment Setup Details

The paper details RETAIN and FLA as baseline adaptation methods, adapting their protocols to the scene-wise one-shot setting used for comparison. RETAIN merges source and one-shot fine-tuned parameters, while FLA inserts LoRA layers into the vision encoder and freezes the remainder.

  • B.1 RETAIN [67]: RETAIN interpolates original and fine-tuned VLA parameters to learn a target-domain task while retaining previously learned multi-task capabilities.The target-domain task is treated as the new task in this baseline.
  • B.2 FLA [32]: Table 10 verifies the FLA implementation by comparing success rates reported in [32] with reproduced results under the same experimental setup.The table is used as an implementation-correctness check before the main comparison.
  • B.1 RETAIN [67]: RETAIN merges a one-shot fine-tuned target-domain model with the original source model using the same training setting as DART.This implements the target-domain task as RETAIN’s new task.
  • B.1 RETAIN [67]: RETAIN’s best module-specific scaling coefficients are 0.6 for the vision encoder, 0.4 for the LLM, and 0.2 for the action expert.The coefficients are selected by independently sweeping values for the three modules.
  • B.2 FLA [32]: FLA adapts VLA models by inserting LoRA layers into the vision encoder while freezing the remaining parameters.FLA was originally designed for new-environment adaptation with task-wise one-shot demonstrations.
  • B.2 FLA [32]: For fair comparison, FLA uses one demonstration per scene rather than one per task, matching DART’s restrictive scene-wise one-shot data budget.The total demonstrations therefore equal the number of scenes.
  • B.2 FLA [32]: The reproduced FLA model follows the original paper’s implementation details and achieves performance consistent with its reported results under task-wise one-shot evaluation.This reproduction is summarized in Table 10.

C.1 VLA Model and Training Hyperparameter Details … D Additional Experiment Results

The supplementary sections specify the VLA architectures, training and evaluation protocols, and domain-shift implementations used across LIBERO, MimicGen, and real-world experiments. They also detail one-shot adaptation data, task setups, and evaluation procedures for viewpoint and cross-embodiment shifts.

  • C.1 VLA Model and Training Hyperparameter Details: π0.5 and π0-FAST use the default openpi JAX architectures, taking third-person and wrist RGB images plus language instructions to output 7D action chunks.All training runs use AdamW with batch size 64; setup-specific resolutions, horizons, and learning rates are summarized in Table 11.
  • C.2 LIBERO Setup Details: LIBERO contains four 10-task suites with 50 demonstrations per task, distributed across five scenes, and uses filtered data with fixed checkpoint action-normalization statistics.The π0.5 source checkpoint comes from openpi, while π0-FAST is trained from pi0_fast_base because no official LIBERO checkpoint exists.
  • C.2 LIBERO Setup Details: One-shot LIBERO fine-tuning uses the first trajectory from one task in each of five scenes, with regenerated target-domain data and success rates averaged across three adaptation-task combinations.Evaluation uses 50 rollouts per task, suite-specific horizons, and Success Rate averaged across tasks and suites.
  • Implementation of Visual Domain Shifts.: LIBERO visual shifts modify camera translation and scene-dependent orientation, while Light changes illumination and Noise applies 27 × 27 Gaussian blur with σ ≈4.4 at every timestep.Viewpoint and visual perturbation strengths use α = 0.8 and α = 0.6, respectively.
  • C.3 MimicGen Setup Details: MimicGen cross-embodiment experiments pretrain π0.5 on Square and Threading, then train a Panda source model on Stack and Stack Three before one-shot adaptation using first trajectories from Panda and UR5e.Pretraining uses 3,800 demonstrations, while source-model training uses 1,900 demonstrations under D0 object initialization.
  • Evaluation.: MimicGen evaluation reports Progress Rate and Success Rate under fixed action normalization, with 200-step and 400-step limits for Stack and Stack Three and α = 0.4.Progress Rate is the maximum milestone reached within an episode, averaged across evaluation rollouts.
  • C.4 Real-world Setup Details: Real-world experiments use a UR10e with a Robotiq 2F-85 gripper and three RealSense D455 cameras, evaluating three pick-and-place and two fine-grained manipulation tasks.The source policy is trained from 120 teleoperated demonstrations, and one-shot fine-tuning uses a single target-domain demonstration for Stack Cube.
  • D Additional Experiment Results: The real-world base π0.5 policy achieves near-perfect source-domain success rates, while evaluation reuses 12 object positions and executes only the first 15 of each predicted 20-action sequence.Source and Target viewpoints are distinguished in the reported pre-adaptation performance table.

D.1 Detailed LIBERO Results by Task Suite … Per-layer overlap energy e(l)

Across LIBERO task suites and environmental shifts, DART generally improves adaptation performance while preserving source-domain behavior and requiring only one-shot data. Additional analyses show that task matching, robust scaling, layer selection, and subspace filtering help isolate transferable domain information efficiently.

  • D.1 Detailed LIBERO Results by Task Suite: DART achieves consistent gains across most LIBERO suites and viewpoint or combined visual shifts, with especially strong performance under larger viewpoint changes.The comparisons cover Spatial, Object, Goal, and Long suites using both π0.5 and π0-FAST.
  • D.2 Upper Bound Performance of Adaptation: Full-data fine-tuning provides an empirical upper bound but requires substantially more target-domain demonstrations and costly collection and training.The upper-bound regime fine-tunes for 10,000 steps using target data at the base-training scale, whereas DART uses one-shot adaptation.
  • D.3 Comparison with Model Merging Methods: Recent model-merging methods underperform DART because interference-mitigation strategies do not directly isolate the transfer signal needed for model analogy.DART aims to cancel shared components between source and target updates rather than merely combine capabilities.
  • D.4 Comparison with test-time-adaptation method.: DART outperforms SCALE across all novel viewpoints, demonstrating the benefit of explicit VLA policy adaptation under environmental shifts.The comparison uses π0-FAST on LIBERO.
  • Source domain performance after adaptation.: DART remains comparable to the zero-shot base policy on the source domain after target-domain adaptation, indicating limited forgetting and cross-environment usability.This source-domain evaluation uses LIBERO with π0.5.
  • Merging multiple domain vectors with detailed results.: Merging domain vectors from Small, Medium, and Large viewpoint shifts yields moderate, consistent success across all three domains while reducing storage to one consolidated vector.The result supports approximately additive domain directions in weight space.
  • Different adaptation task from source and target domains.: Matching source and target adaptation tasks gives the highest performance, while cosine-similarity retrieval consistently beats random task selection when exact matching is unavailable.The analysis evaluates 40 LIBERO tasks with π0.5; performance remains stable across scaling coefficients, with standard deviations of 0.8% for Medium and 2.1% for Large.
  • Impact of scaling coefficient and training time: DART consistently outperforms FLA and RETAIN under comparable training time, while adapting Vis+LLM performs nearly as well as all-layer adaptation and LLM layers show lower cross-domain alignment.Action adaptation provides only marginal benefit; MLP/Up_proj and MLP/Gate_proj layers exhibit low overlap and therefore receive more subspace filtering.
Loading 2607.00666v1…