PaperlayerView exact revision on arXiv

Source-linked AI summary

DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack

Hoseong Tae, Jong-Seok Lee

arXiv:2608.03207v1cs.CVcs.LG

TL;DR

Flow-matching VLAs appear robust to adversarial attacks that ignore their multi-step denoising ODE, leaving open whether deployed frozen policies can be attacked through that trajectory. DRIFT uses a universal gripper patch to target the denoising velocity field and breaks essentially all originally solvable tasks on π0 and π0.5, with first-step attacks stronger and cheaper than wider windows.

  • Problem

    Existing attacks target action outputs or visual embeddings while ignoring the multi-step denoising ODE that generates flow-matching VLA actions.

  • Method

    DRIFT optimizes a universal small gripper patch offline to directly perturb the denoising velocity field of a frozen, unmodified policy.

  • Results

    DRIFT breaks essentially all originally solvable tasks across π0 and π0.5 on four LIBERO suites, while first-step attacks outperform wider denoising-step windows.

  • Takeaways & Limitations

    Flow-matching VLA robustness depends on whether attacks account for the denoising trajectory, and the first denoising step is the most effective test-time target.

  • Takeaways & Limitations

    Effective patches require white-box gradient access to each deployed policy because optimized perturbations transfer poorly across model versions.

Abstract

from arXiv · show

Flow-matching vision-language-action (VLA) models such as pi0 generate robot actions by integrating a learned denoising velocity field, and have been reported to resist adversarial perturbations that readily fool autoregressive VLAs. We show that this robustness is largely illusory: it stems from prior attacks ignoring the multi-step denoising ODE. We introduce DRIFT (Denoising Redirection via Input perturbation of the Flow-matching Trajectory), a test-time universal adversarial patch placed on the robot's gripper that attacks the denoising velocity field of an off-the-shelf policy. Our central finding is counterintuitive: attacking only the first denoising step is both stronger and cheaper than attacking a wider window of steps, which we explain through a gradient conflict unique to input-space optimization and which is exactly opposite to the training-time backdoor regime. On pi0 and pi0.5 across four LIBERO suites, DRIFT breaks essentially all originally-solvable tasks with a small single patch, far exceeding action- and embedding-space attack baselines.

1 INTRODUCTION

DRIFT shows that flow-matching VLA robustness is largely an artifact of attacks ignoring the multi-step denoising ODE. A universal test-time patch attacking only the first denoising step derails unmodified policies more effectively and efficiently than broader or action-/embedding-space attacks.

  • DRIFT is a universal test-time adversarial patch that perturbs the denoising velocity field of an unmodified flow-matching VLA.The patch is optimized offline with white-box gradients, placed on the robot’s gripper, and requires no policy modification or deployment-time model-weight access.
  • Across π0 and π0.5 on four LIBERO suites, DRIFT breaks essentially all originally-solvable tasks and exceeds UADA and EDPA under the same patch-size budget.UADA is the action-space baseline, while EDPA is the embedding-space baseline; both ignore the denoising ODE.
  • DRIFT attacks only the first denoising step, outperforming wider windows of early steps in the test-time setting.This directly contrasts with the training-time backdoor regime, where a narrow early-step window is insufficient.
  • A gradient conflict explains why adding denoising steps hurts: early- and late-step patch gradients are misaligned and cancel during accumulation.The conflict is specific to input-space optimization and explains both the first-step sufficiency and the less-is-more effect.
  • DRIFT reduces optimization cost by a factor of K relative to full-trajectory attacks while producing a stronger patch.The forward and backward passes require only a single denoising step.

2 RELATED WORK

Related work spans autoregressive and continuous VLAs, visual-observation attacks, universal physical patches, and training-time backdoors. DRIFT revisits reported flow-matching robustness by targeting the denoising trajectory directly.

  • VLA architectures: VLAs map visual observations and natural-language instructions to robot actions, with autoregressive models decoding discretized action tokens and continuous VLAs generating real-valued actions directly.RT-1, RT-2, and OpenVLA represent the autoregressive family described in the passage.
  • Robustness and attacks: Flow-matching models, notably π0, appear substantially more robust to naive perturbations than autoregressive VLAs, motivating attacks on their denoising trajectories.Earlier attacks perturb visual observations, while DP-Attacker demonstrates vulnerability in diffusion-based visuomotor policies.
  • Test-time adversarial attacks: UADA introduces a universal adversarial patch optimized for task failure across manipulation scenarios, creating a practical test-time physical threat model without training-procedure access.The patch is treated as a physical sticker placed in the robot’s camera view.
  • Backdoor attacks: Training-time backdoor attacks implant triggers that elicit attacker-chosen behavior, while FlowHijack conditions a flow-matching denoising trajectory on a trigger to steer target behavior.These backdoor attacks are largely agnostic to how actions are generated, except that FlowHijack specifically targets flow-matching VLAs.

3 PRELIMINARY

Flow-matching VLAs condition future action chunks on images, language, and proprioception, then generate actions by integrating a learned velocity field from Gaussian noise. DRIFT targets this denoising process with a conservative, universal test-time patch attack that maximizes failure without modifying the model.

  • Flow-matching VLA formulation: Flow-matching VLAs model future action chunks conditioned on multi-view images, language instructions, and robot proprioceptive state.The action chunk is represented as A_t = [a_t, a_t+1, …, a_t+H−1].
  • Flow-matching VLA formulation: At inference, the policy starts from Gaussian noise and iteratively integrates its learned velocity field through K discrete Euler steps before returning the action chunk.The denoising trajectory uses step size Δτ = 1/K and reaches A^1_t ≈ A_t after K steps.
  • Adversarial objective: The untargeted adversary maximizes task failure by driving the denoising velocity field under perturbed observations away from its clean behavior.This operationalizes action deviation through velocity-field divergence rather than direct final-action optimization.
  • Threat model: The attack uses white-box, test-time input-space optimization against a frozen policy, without weight modification, fine-tuning, or training-data poisoning.This distinguishes the threat model from training-time backdoor attacks requiring a compromised model.
  • Threat model: A single 32 px patch covering roughly 2% of the wrist image is placed over the gripper, leaving proprioception, language, and other pixels untouched.The patch is physically approximately 2–3 cm and occupies a small contiguous image region.
  • Threat model: The optimized universal patch is fixed across tasks, scenes, and episodes, requiring neither per-instance optimization nor deployment-time model access.The capability profile assumes no control over the model pipeline and no online feedback.

4 METHODOLOGY

DRIFT analyzes which denoising steps are most vulnerable to a universal wrist-image patch, then defines a differentiable attack that maximizes clean-versus-perturbed velocity discrepancy. The resulting attack targets only the first denoising step because early errors cascade through the full trajectory, while wider windows suffer gradient conflict.

  • Universal patch: A single universal patch δ is pasted onto a fixed wrist-image region R, changing o_t to o⁺ while leaving the remaining observation untouched.The patch is shared across observations rather than optimized per instance.
  • Step-wise analysis: The analysis varies the attacked denoising-step set S to measure how input-space perturbations affect the velocity field across the K-step Euler trajectory.The first step k=0 departs from pure noise, whereas k=K−1 is adjacent to the final action.
  • Gradient conflict: M=1 is strongest on average, while adding attacked steps degrades performance because the summed per-step patch gradients increasingly cancel.The gradient survival ratio falls from 1.00 to 0.45 as the first-M window widens from M=1 to M=10; cos(g0, g9)≈−0.35.
  • Step-wise analysis: Early-step attacks achieve markedly higher velocity divergence, and errors introduced at k=0 propagate through every subsequent Euler update to produce large final-action deviations.The first step also maximizes the cascade horizon and requires no preceding ODE rollout.
  • DRIFT formulation: DRIFT is the single-step attack S={0}, optimizing a differentiable clean-versus-perturbed velocity discrepancy with projected gradient ascent and optional ℓ∞ projection.Its objective is L_DRIFT(δ)=L_0(δ), evaluated at the first denoising step.

5 EXPERIMENTS

Experiments on frozen π0 and π0.5 across all four LIBERO suites show that a universal 32 × 32 DRIFT patch nearly eliminates originally solvable tasks, with failures caused by early gripper closure and trajectory diversion. The attack exhibits a sharp perturbation threshold, while cross-model transfer remains weak and model-specific.

  • Experimental setup: DRIFT attacks frozen π0 and π0.5 using publicly released LIBERO-finetuned checkpoints with default K=10-step denoising.The attacker uses gradients to optimize the patch but never updates model weights.
  • Experimental setup: A 32 × 32 patch trained only on libero spatial observations is evaluated unchanged across Spatial, Goal, Object, and Long suites.Each suite contains 10 tasks and 100 episodes; Spatial is in-distribution, while the other suites test generalization.
  • Main results: On π0, DRIFT reaches nearly 100% ASR on every LIBERO suite, exceeding UADA at 13.2% and EDPA at 25.3% under the same patch-size budget.EDPA is reported with a larger 64 px patch because it is ineffective at 32 px.
  • Failure mechanism: Patched policies close the gripper within the first 7 control steps in all five illustrated episodes, versus approximately step 46 for clean policies.The patched arm continues moving but is diverted before completing the grasp.
  • Perturbation budget: 97% ASR at ϵ=0.1 follows ASR values of ≤8.2% for ϵ≤0.05, demonstrating a sharp all-or-nothing perturbation threshold.The passage attributes the jump to corruption of the denoising trajectory through the first step and a resulting cascade effect.
  • Transferability: Cross-model transfer is asymmetric: patches transfer to π0 at 24.7% but to π0.5 at 8.9%, indicating shared vulnerability but model-specific perturbations.Increasing patch size from 32→64 px raises π0-to-π0.5 transfer only from 0.4→8.9%.

6 CONCLUSION · APPENDIX

DRIFT is a test-time universal adversarial gripper patch that corrupts the denoising velocity field of a frozen flow-matching VLA. Its central finding is that perturbing only the first denoising step is stronger and cheaper than attacking a wider window, due to gradient conflict opposite to the training-time backdoor regime.

  • 6 CONCLUSION: DRIFT introduces a test-time universal adversarial patch for flow-matching VLAs.The patch is applied as a single sticker on the robot’s gripper.
  • 6 CONCLUSION: The patch corrupts the denoising velocity field of a frozen flow-matching VLA.The attack operates through a single sticker placed on the robot’s gripper.
  • 6 CONCLUSION: Only the first denoising step must be perturbed for the attack.The conclusion identifies first-step perturbation as sufficient.
  • 6 CONCLUSION: Attacking only the first step is stronger than attacking a wider denoising window.The paper characterizes this as a “less-is-more” effect.
  • 6 CONCLUSION: Attacking only the first step is cheaper than attacking a wider denoising window.The efficiency advantage is part of the reported “less-is-more” effect.
  • 6 CONCLUSION: The first-step advantage is traced to gradient conflict in input-space optimization.This conflict runs opposite to the training-time backdoor regime, where a wide early-step window is preferred.

A EXPERIMENTAL SETTING DETAILS · A.1 LIBERO EVALUATION PROTOCOL

The LIBERO evaluation follows the policies’ native preprocessing and standard episode initialization, with both π0 and π0.5 replanning every five executed actions. Each episode begins with ten warm-up steps before policy queries.

  • A.1 LIBERO EVALUATION PROTOCOL: LIBERO evaluation resizes 256 × 256 RGB observations with zero-padding to 224×224, executes 10 warm-up steps, and replans every 5 actions for both π0 and π0.5.Each policy predicts an action chunk, but only its first five actions are executed open-loop before querying again.

A.2 PATCH PLACEMENT · A.3 BASELINE REIMPLEMENTATION · B ATTENTION MAP ON THE TARGET OBJECT

The appendix specifies exact wrist-image patch placement, reimplements baselines under a shared PGD budget, and investigates whether DRIFT redirects attention away from the target object. These controls distinguish occlusion, action-space, and embedding-space effects while probing the phantom-grasp mechanism.

  • A.2 PATCH PLACEMENT: A.2 places the patch at the wrist image’s bottom-center, with top-left corner (y, x) = (224 − p_h, (224 − p_w)/2).This formula applies to a patch of height h and width w.
  • A.2 PATCH PLACEMENT: Table 4 reports the exact top-left pixel coordinates for the two patch sizes used in the paper.
  • A.3 BASELINE REIMPLEMENTATION: All baselines use PGD for 500 iterations with step size α=0.01 and the same universal observation pool, isolating attack-objective effects on ASR.
  • A.3 BASELINE REIMPLEMENTATION: The random-patch baseline samples one uniform-random pixel patch per seed and leaves it unoptimized, isolating occlusion from gradient-based optimization.
  • A.3 BASELINE REIMPLEMENTATION: UADA maximizes squared deviation between clean and patched final denoised action chunks while backpropagating through the full K-step rollout.Its objective operates in action space without an explicit per-step velocity term.
  • A.3 BASELINE REIMPLEMENTATION: EDPA disrupts the frozen PaliGemma vision/language tower using InfoNCE repulsion and image–language alignment terms combined by EMA-balanced weighting.The InfoNCE temperature is 0.07; EMA decay is 0.9 and mixing weight is 0.5.
  • B ATTENTION MAP ON THE TARGET OBJECT: The phantom-grasp failure mode makes the patched policy close the gripper almost immediately without approaching the object.This motivates testing whether the patch redirects internal attention away from the object to be grasped.
  • B ATTENTION MAP ON THE TARGET OBJECT: The attention analysis extracts the policy’s actual softmax self-attention weights to directly evaluate the target-object redirection hypothesis.

B.1 METHOD. · B.2 PATCH REDIRECTS ATTENTION

The method measures how the language token for “bowl” attends to visual tokens and tests whether a 32×32 patch redirects that attention. In the failed patched rollout, the patch captures 28–33% of this attention across the first three steps, versus 1.7% under clean observation and 3.5% chance.

  • B.1 METHOD.: B.1 METHOD.: The analysis uses the LIBERO-Spatial task of placing a black bowl on a plate, identified as task 0 and shown in Fig. 5a.The task is the same task referenced in the main paper’s Fig. 5a.
  • B.2 PATCH REDIRECTS ATTENTION: B.2 PATCH REDIRECTS ATTENTION: The patched rollout fails while the clean rollout of the same task succeeds.Figure 7 contrasts the clean and DRIFT-patched rollouts.
  • B.1 METHOD.: B.1 METHOD.: The analysis extracts real softmax attention weights for the language token corresponding to the object noun “bowl.”Weights are averaged over layers and heads.
  • B.1 METHOD.: B.1 METHOD.: Attention is restricted to the 16×16=256 visual tokens from a selected camera.The extracted weights are averaged over layers and heads before restriction to the camera’s visual tokens.
  • B.2 PATCH REDIRECTS ATTENTION: B.2 PATCH REDIRECTS ATTENTION: The test compares patch-region attention against a 3.5% chance level determined solely by the region’s area fraction.The 32×32 patch region corresponds to 3.5% of the 256 visual tokens.
  • B.2 PATCH REDIRECTS ATTENTION: B.2 PATCH REDIRECTS ATTENTION: 1.7% of the “bowl” token’s attention falls on the patch region under the clean observation, below chance.The clean unpatched rollout succeeds.
  • B.2 PATCH REDIRECTS ATTENTION: B.2 PATCH REDIRECTS ATTENTION: 28–33% of the “bowl” token’s attention falls inside the patch region across the first three steps of the failed patched rollout.This represents an 8–9× increase over chance and is concentrated in the patch region.

C A SIMPLE DEFENSE BASELINE: JPEG COMPRESSION

JPEG compression is a cheap, training-free defense whose effectiveness against DRIFT increases as quality decreases. Strong compression largely blocks the attack on Spatial, Goal, and Object, but Long retains substantially higher ASR.

  • Defense evaluation: JPEG quality sweeps show that DRIFT’s ASR decreases progressively as compression strengthens, while clean TSR remains between 71–100%.The evaluation spans Q ∈{95, 75, 50, 30, 10} across all four LIBERO suites, using 100 episodes per setting.
  • Defense evaluation: At Q≥75, DRIFT remains effective on Spatial and Object with 75–100% ASR, but is weaker and more variable on Goal with 38–94% ASR.These mild-compression results indicate that high JPEG quality does not reliably mitigate the attack.
  • Strong compression: At roughly Q≲50–60, ASR falls to 0–12% on Spatial, Goal, and Object, largely blocking DRIFT.This supports JPEG compression as a cheap first mitigation for simpler tasks when sufficiently strong compression is applied.
  • Strong compression: Long is an exception: ASR remains 16–21% and plateaus even at Q=10, roughly 4× higher than the other suites at the same setting.Thus, JPEG compression is less reliable on the Long suite despite the strongest tested compression.
Loading 2608.03207v1…