PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-ZO: Fast Zeroth-Order Adaptation for Vision-Language-Action Models

Jaemin Kim, Jiahn Kim, Taesik Gong

arXiv:2610.06271v1cs.ROcs.AI

TL;DR

Deployment-time shifts can challenge fixed VLA policies, while first-order adaptation may exceed inference memory and naive ZO can be slow. VLA-ZO reuses frozen-prefix conditioning states during action-side adaptation; on LIBERO camera-viewpoint shifts, it cuts adaptation time and improves average success at larger query budgets.

  • Problem

    VLA policies can degrade under deployment-time distribution shifts, while first-order adaptation may exceed inference-oriented memory budgets and naive ZO requires costly repeated queries.

    Source: §1 · pp. 1–2

  • Method

    VLA-ZO confines adaptation to the action side and reuses frozen vision-language prefix states across ZO probes and optimizer steps, with prefetching to hide transfer overhead.

    Source: §3.2 · p. 4, §4 · p. 5

  • Results

    Adaptation time falls from 38,487 to 1,504 seconds at q=16 and from 153,220 to 4,709 seconds at q=64; average success rises from 48.27% without adaptation to 58.17% at q=16 and 63.58% at q=64.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    The results indicate that faster ZO can make larger query budgets practical and provide a promising path toward resource-efficient VLA adaptation on deployment platforms.

    Source: §6 · p. 9

    Reuse is equivalent to recomputation under exact arithmetic only when perturbed parameters lie after the reuse boundary, cached states carry all prefix information, and the frozen prefix is deterministic.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Adapting vision-language-action (VLA) models to deployment-time distribution shifts is important for reliable robotic operation, but conventional first-order adaptation can exceed the memory budget of inference-oriented deployment platforms. Zeroth-order (ZO) optimization offers a forward-only alternative with inference-level memory, but accurate gradient estimation requires many perturbation queries, making naive ZO prohibitively slow for large VLA models. We present VLA-ZO, a framework for fast ZO adaptation that exploits the structure of VLA computation. By confining adaptation to the action side, VLA-ZO keeps the expensive vision-language prefix frozen and reuses its conditioning states across perturbation queries and optimizer steps, while schedule-aware prefetching hides state-transfer overhead. On LIBERO camera-viewpoint shifts, VLA-ZO reduces end-to-end adaptation time by 25.59$\times$ at $q=16$ and 32.54$\times$ at $q=64$ relative to baseline ZO, while improving average task success from 48.27% without adaptation to 58.17% and 63.58%, respectively. These results show that making ZO faster can make larger query budgets practical, providing a promising path toward resource-efficient VLA adaptation on deployment platforms.

1 INTRODUCTION

VLA policies can degrade under deployment-time distribution shifts, while first-order adaptation can exceed inference-oriented memory budgets. ZO avoids backpropagation but repeated queries make adaptation slow, motivating a faster approach.

1 INTRODUCTION

  • Camera-viewpoint and initial-state perturbations can reduce state-of-the-art VLA success from 95% to below 30%, motivating post-deployment adaptation.One-shot adaptation uses a single target-environment demonstration.Source: §1 · p. 1
  • At batch sizes 16 and 64, π0.5 first-order peak memory is 20.27 and 54.69 GiB versus forward-only ZO's 12.74 and 27.71 GiB.The relative memory gap grows from 1.59× to 1.97× despite gradient checkpointing.Source: §1 · p. 1
  • ZO estimates updates from perturbed forward evaluations, avoiding backpropagation and reducing memory needs, but its potential for VLA adaptation remains largely unexplored.The method is attractive where backward execution is unavailable or undesirable.Source: §1 · p. 2
  • Each additional query direction requires two full-model forward passes, so larger q improves adaptation quality toward first-order performance but increases runtime and can make naive VLA adaptation take hours.The figure summarizes the quality–time trade-off as q increases.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Related work covers data-efficient VLA adaptation, zeroth-order optimization, and techniques for making perturbation-based execution more efficient. This paper focuses on reducing the computational cost of ZO VLA adaptation under deployment constraints.

2 RELATED WORK

  • One-shot VLA methods adapt from a single target-domain demonstration; FTM and FLA address novel camera viewpoints, while DART considers visual and embodiment shifts and domain-level adaptation.This work is complementary, focusing on computational cost rather than developing a new adaptation method.Source: §2 · p. 2
  • ZO methods estimate gradients from perturbed forward evaluations; MeZO applies this approach to LLM fine-tuning, while later work studies task alignment, optimizer design, and low-rank perturbations.The cited low-rank method is SubZero.Source: §2 · p. 2
  • Execution-efficiency work includes MobiZO's parallel perturbation evaluations and DeepZero's sparse coordinate-wise estimation, feature reuse, and forward parallelization.These methods address ZO or deep-network execution efficiency.Source: §2 · p. 3

3 ZEROTH-ORDER VLA ADAPTATION

The section formulates one-shot ZO adaptation and identifies repeated vision-language prefix computation as a bottleneck. It motivates action-side adaptation with a frozen prefix, allowing conditioning states to be reused across probes and updates.

3.1 PROBLEM SETUP AND ZO WORKLOAD

  • The setup evaluates a policy adapted from one target-domain demonstration on both the demonstrated task and unseen tasks in that domain.The demonstration consists of observation–action pairs with camera images, proprioceptive state, and a shared language instruction.Source: §3.1 · p. 3
  • The policy separates a vision-language prefix that forms each sample's conditioning state from an action expert that uses it to generate action chunks.Across the described VLA designs, the prefix accounts for most parameters and floating-point operations in one objective evaluation.Source: §3.1 · p. 3
  • ZO replaces the objective gradient with randomized finite differences over q perturbation directions; each update uses 2q objective evaluations without constructing a backward graph.The paired evaluations share a stochastic draw, and the minibatch stays fixed across the update's directions.Source: §3.1 · p. 3

3.2 ACTION-SIDE ADAPTATION WITH A FROZEN PREFIX

  • Under matched setup, action-side ZO raises average success across the three camera-viewpoint shifts from 48.3% to 58.2%, matching vision-side ZO's 56.6% with approximately 3.8× fewer trainable parameters.The action-side LoRA placement uses query and value projections in each action-expert self-attention block, while the vision-side placement targets vision-encoder feed-forward projections.Source: §3.2 · p. 4

3.3 REPEATED PREFIX COMPUTATION

  • Because the prefix is frozen, each sample's conditioning state is independent of adaptation parameters and can be computed once, reducing prefix executions from 2qBT to at most N.The action-expert evaluations remain necessary; supplying cached states efficiently motivates within-step reuse, across-step reuse, and prefetching.Source: §3.3 · p. 4

4 VLA-ZO: REUSING FROZEN CONDITIONING ACROSS ZO PROBES

VLA-ZO accelerates zeroth-order adaptation by reusing frozen vision-language conditioning states within and across updates, then overlapping cached-state transfers with probe computation. Its cost model separates action-expert probe work from prefix execution and cache loading.

4 VLA-ZO: REUSING FROZEN CONDITIONING ACROSS ZO PROBES

  • VLA-ZO leaves the objective, perturbation directions, and trainable parameter space unchanged while progressively reducing prefix-related overhead through reuse and prefetching.Its cost model distinguishes frozen-prefix execution, action-expert execution, and cached-state loading.Source: §4 · p. 5

4.1 WITHIN-STEP STATE REUSE

  • Within each update, VLA-ZO computes each sample’s frozen-prefix state once and reuses it across 2q action-expert probes, reducing prefix evaluations from 2qB to B.Reuse preserves the objective and finite-difference estimator under exact arithmetic when adaptation occurs after the reuse boundary, the cached state is complete, and the frozen prefix is deterministic.Source: §4.1 · pp. 5–6

4.2 ACROSS-STEP STATE REUSE

  • Across-step reuse replaces repeated prefix executions with cached-state loads, with savings growing as repeated visits accumulate and the one-time state-materialization cost is amortized.The cache-loading cost is substantially lower than recomputing the vision-language prefix.Source: §4.2 · p. 6

4.3 SCHEDULE-AWARE STATE PREFETCH

  • VLA-ZO stores persistent conditioning states on disk and stages only minibatch states, overlapping next-batch preparation with the current update’s 2q probes.One-update lookahead hides loading fully when cld ≤ 2q csuf; otherwise, exposed loading latency is B max(0, cld − 2q csuf).Source: §4.3 · pp. 6–7

5 EXPERIMENTS

The experiments evaluate one-shot VLA adaptation and execution optimizations under viewpoint and compound visual shifts, then test transfer to OpenVLA-OFT. VLA-ZO improves adaptation success while reducing runtime through reuse mechanisms.

5.1 SETUP

  • The evaluation covers 40 LIBERO tasks, five expert demonstrations across scenes, three camera shifts, and compound blur or lighting shifts.Each condition uses 40 initial states per task; baselines include the unadapted policy, FLA, and DART.Source: §5.1 · p. 7

5.2 CAMERA-VIEWPOINT ADAPTATION

  • At q=16, average success is 58.17% versus 48.27% unadapted, rising to 63.58% at q=64.The larger query budget helps most under Medium and Large shifts; Small does not improve consistently with q.Source: §5.2 · p. 8, Table 1 · p. 8
  • Adaptation time falls 25.59× at q=16 and 32.54× at q=64 relative to baseline ZO.The reported times include cold-cache population.Source: §5.2 · p. 8, Table 1 · p. 8
  • Within-step reuse provides the largest time reduction; across-step reuse and schedule-aware prefetch then reduce runtime further.The execution ablation progressively enables these mechanisms at fixed query budgets and adaptation settings.Source: §5.2 · pp. 7–8
  • Execution variants show similar observed success ranges: 57.63–58.17% at q=16 and 62.83–63.77% at q=64.The authors caution that small differences should not be interpreted as quality changes caused by caching.Source: §5.2 · p. 8

5.3 COMPOUND VISUAL SHIFTS

  • Under View+Noise, average success is 51.1% at q=16 and 55.8% at q=64 versus 49.2% unadapted; under View+Light, it is 58.4% and 60.8% versus 49.0%.The gains are larger for lighting; at q=64, VLA-ZO is close to DART on View+Noise and exceeds it on View+Light.Source: Table 2 · p. 8, §5.3 · p. 9

5.4 GENERALIZATION TO OPENVLA-OFT

  • On OpenVLA-OFT, action-head adaptation raises average viewpoint-shift success from 49.48% to 52.67% at q=4 and improves results under compound shifts.For View+Noise, average success rises from 23.9% unadapted to 33.5% at q=64; for View+Light, it rises from 40.9% to 46.3%.Source: §5.4 · p. 9, Table 3 · p. 9, Table 4 · p. 9

6 CONCLUSION

VLA-ZO is a fast forward-only framework for one-shot VLA adaptation that eliminates redundant computation across zeroth-order queries. The authors argue that faster execution can support more query directions within a fixed time budget, making fast ZO promising for resource-constrained deployment.

6 CONCLUSION

  • VLA-ZO provides a fast, forward-only framework for one-shot VLA adaptation by eliminating redundant computation across ZO queries.It is presented as a path toward practical adaptation on resource-constrained deployment platforms.Source: §6 · p. 9
  • The authors argue that faster ZO execution can make more query directions practical within a fixed time budget, improving gradient estimates.Source: §6 · p. 9
  • The paper identifies fast ZO as a promising path toward practical VLA adaptation on resource-constrained deployment platforms.Source: §6 · p. 9

A.1 DART ADAPTATION DATA PROTOCOL

The adaptation protocol uses one demonstration trajectory per scene and evaluates the adapted policy across both demonstrated and unseen tasks. Visual perturbations combine camera changes with blur or lighting changes, with condition-matched adaptation data.

A.1 DART ADAPTATION DATA PROTOCOL

  • The protocol uses five expert demonstrations, one per scene, and evaluates all 40 tasks, including tasks outside the five demonstration tasks.Demonstrations retain their task instructions and actions across camera views.Source: §A.1 · p. 13
  • The experiments use combination 1 and seed 0; its five trajectories are pooled into one dataset for each visual condition.The three DART task combinations contain 1,085, 1,044, and 1,025 retained frames per camera view, respectively.Source: §A.1 · p. 13
  • The selected first demonstration is the lowest original per-task demonstration ID remaining in the filtered release, identified by matching action sequences after no-op removal.The selection differs from simply taking the first serialized TFRecord because shard order is shuffled and filtering removes some demonstrations.Source: §A.1 · p. 13

A.2 VISUAL PERTURBATIONS

  • The viewpoint transformation moves external cameras and reorients them toward the scene origin, while leaving the wrist camera unmoved.Source: §A.2 · p. 13
  • Noise applies Gaussian blur to the external image; Light changes scene illumination affecting both views, and compound conditions pair either change with the Medium camera pose.Adapters use demonstrations matched to each target condition rather than applying a clean-view adapter to an additional corruption.Source: §A.2 · p. 13

A.3 MODELS, ACTION WINDOWS, AND EVALUATION

The appendix specifies model-specific LIBERO rollout protocols, adaptation configurations, and procedures for learning-rate selection and timing.

A.3 MODELS, ACTION WINDOWS, AND EVALUATION

  • π0.5 retains 50-action chunks and ten flow-integration steps, but replans every ten actions; OpenVLA-OFT uses an eight-action replanning interval.Both protocols use ten settling steps; π0.5 uses its native flow-matching loss, while OpenVLA-OFT uses a continuous L1-regression head.Source: §A.3 · p. 14
  • Evaluation uses 50 initial states per task, with indices 0–9 for learning-rate selection and 10–49 for final evaluation, totaling 1,600 episodes per condition.Success rates weight tasks equally; suite and condition averages follow the groups specified in each result table.Source: §A.3 · p. 14

A.4 HYPERPARAMETERS AND LEARNING-RATE SWEEPS

  • Action-side ZO, FLA, and DART are compared with the unadapted policy, while execution ablations progressively add reuse and schedule-aware prefetching.The daggered FLA and DART baselines use forward-only two-sided MeZO estimates followed by SGD; these methods change the prefix and cannot use the frozen-prefix cache.Source: §A.4 · p. 15
  • All listed optimization stages use task combination 1, seed 0, batch and microbatch size 64, Gaussian two-sided perturbations, and ϵ = 10^-3.The ZO update averages q directional estimates and applies SGD without momentum, weight decay, or gradient clipping; LoRA uses rank and scaling 16 with exact-SVD PiSSA initialization.Source: §A.4 · p. 15

A.5 TIMING AND SOFTWARE

  • OpenVLA-OFT learning-rate sweeps use a half-decade grid from 10^-5 to 10^-1, and each completed candidate receives a 400-episode pilot evaluation per view.Profiling uses one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB memory per training process; DART full adaptation timing includes both training windows and CPU merging.Source: §A.4 · p. 15, §A.5 · p. 15

B IMPLEMENTATION OF FROZEN-STATE REUSE AND PREFETCHING

The implementation caches frozen-prefix conditioning states, reconstructs upcoming minibatches, and stages cached data for transfer while preserving the configured sample order.

B.1 CACHE CONTENTS

  • The cache stores the frozen prefix information needed by each action head: π0.5 attention keys, values, and padding mask, or OpenVLA-OFT action-token hidden states.Action targets, sampled action noise, flow time, and trainable action projections remain outside the π0.5 cache; OpenVLA-OFT processes images, proprioception, and prompt before its cache boundary.Source: §B.1 · p. 16
  • Cache construction is lazy; fully cached microbatches skip prefix computation, while a cache miss recomputes the frozen prefix at the original full microbatch shape.A mixed microbatch may recompute already cached observations during population; missing files fall back to computation, while corrupt or mismatched records raise an error.Source: §B.1 · p. 16

B.2 RECONSTRUCTING THE NEXT MINIBATCH

  • For update t, the training thread schedules the next update’s microbatches before evaluating the current update’s 2q probes, with the queue bounded to one future update.Only fully cached future microbatches are eligible; an unready future delays its own batch without changing sample order, and speculative futures are rebuilt after resume.Source: §B.2 · p. 16

B.3 HOST STAGING AND CUDA STREAMS

  • A background worker assembles disk-cached records in pinned host buffers and transfers them on a dedicated CUDA stream, with an event establishing the copy-to-compute dependency.The worker handles file reads, tensor assembly, and transfers; model calls, cache-miss computation, and objective randomness remain in the training thread.Source: §B.3 · pp. 16–17

B.4 WHAT REUSE COSTS AND WHAT IT PRESERVES

Reuse trades additional state storage for fewer repeated prefix computations, while leaving the ZO objective and update design unchanged.

B.4 WHAT REUSE COSTS

  • Accelerator staging is bounded at 2BS for current and prefetched batches, while host storage grows to N records regardless of q or T.A π0.5 conditioning state occupies approximately 17.02 MiB per sample in BF16; OpenVLA-OFT’s 56×4096 action-token hidden state occupies 0.4375 MiB per sample in BF16.Source: §B.4 · p. 17

B.4 WHAT REUSE PRESERVES

  • Reuse does not alter the objective, perturbation directions, trainable parameters, or update rule, but bitwise-identical trajectories are not guaranteed.The changed GPU execution path can produce small loss differences that may amplify across updates; the cache-mode ablation reports success-rate differences comparable to repeated runs of an identical configuration.Source: §B.4 · p. 17
Loading 2610.06271v1…