PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuning

Di Zhu, Ziheng Yan, Fang Wan

arXiv:2609.34982v1cs.CVcs.RO

TL;DR

VLA models struggle to align coarse multimodal semantics with fine-grained temporal execution, limiting their generalization and robustness in cluttered environments. ActionUNet combines a temporal U-Net in action-feature space with a conditional SIREN continuous decoder and explicit smoothness constraints. It reports improved π0.5 success rates across benchmark and real-world hard or perturbed evaluations.

  • Problem

    VLA models have difficulty aligning coarse-grained multimodal semantics with fine-grained temporal execution, and exhibit limited generalization and robustness in cluttered environments.

    Source: §1 · p. 1

  • Method

    ActionUNet refines temporal-aligned action features with a lightweight temporal U-Net and decodes them with a conditional SIREN using second-order smoothness constraints.

    Source: §1 · p. 2

  • Results

    ActionUNet improves π0.5 success rates by 9.8%, 6.1%, and 11.4% on RoboTwin 2.0, LIBERO-Plus, and real-world dual-arm manipulation with hard setting, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ActionUNet consistently improves task success and robustness under environmental distractions, supporting multi-scale temporal mapping as an efficient VLA adaptation strategy.

    Source: §5 · p. 10

    The method uses fixed temporal scales, models action chunks independently, and introduces scale-aware modeling only at the downstream action stage.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown great promise for robotic manipulation by mapping multi-modal semantics to physical actions. However, this mapping inherently struggles to align these coarse-grained semantics with fine-grained temporal execution. It leaves VLA models with limited generalization and insufficient robustness in cluttered environments. To overcome this issue, we propose ActionUNet, an efficient multi-scale fine-tuning framework that enhances pre-trained VLA models with minimal computational cost. ActionUNet first constructs a lightweight temporal U-Net within the temporal-aligned action feature space to fuse hierarchical structural priors, effectively bridging the scale gap between semantics and temporal executions. Recognizing that multi-scale modeling can disrupt microscopic temporal continuity and cause mechanical oscillations, ActionUNet then employs a conditional SIREN as a continuous action decoder. Equipped with explicit second-order smoothness constraints, this decoder guarantees temporal continuity and reduces high-frequency motion jitter. By smoothing temporal discontinuities from multi-scale fusion, this continuous formulation reduces mechanical execution failures while preserving the base VLA model's generalization and manipulation robustness. Extensive experiments on RoboTwin 2.0 and LIBERO-Plus benchmarks, together with real-world hard evaluations, demonstrate that ActionUNet significantly improves {\pi}0.5 success rates by absolute 9.8%, 6.1%, and 11.4%, respectively, while also generalizing to the regression-based OpenVLA-OFT backbone, highlighting its effectiveness and efficiency as a fine-tuning strategy. Code and implementation details are available at https://github.com/Di-Zhu123/ActionUNet.

1 Introduction

VLA models struggle to align coarse semantic representations with fine-grained temporal execution, limiting generalization and robustness in cluttered or distracting environments. ActionUNet adds multi-scale action features and continuous smooth decoding, with reported success-rate gains in hard or perturbed evaluations.

1 Introduction

  • VLA models struggle to map coarse-grained multimodal semantics to fine-grained action sequences, limiting generalization and robustness in cluttered or distracting environments.Per-step end-effector displacement varies across tasks and across stages within a task, indicating different temporal action scales.Source: §1 · pp. 1–2
  • Multi-scale action modeling risks discontinuities at scale boundaries, which can produce non-smooth trajectories and mechanical oscillations.The introduction identifies balancing multi-scale structural priors with action smoothness as a core challenge.Source: §1 · p. 2
  • ActionUNet adds a lightweight temporal U-Net in the temporal-aligned action feature space and a conditional SIREN decoder with second-order smoothness constraints.The framework combines hierarchical structural priors with continuous action decoding to preserve temporal continuity and mitigate high-frequency jitter.Source: §1 · p. 2
  • ActionUNet improves π0.5 success rates by 9.8%, 6.1%, and 11.4% on RoboTwin 2.0, LIBERO-Plus, and real-world dual-arm manipulation with hard setting, respectively.The reported gains are in hard/perturbed settings.Source: §1 · p. 3

2 Related Work

Prior VLA and visuomotor approaches address robustness and multi-scale action modeling through varied representations and policy designs. ActionUNet instead injects multi-scale structure into pretrained VLA action features, aiming to reuse those priors with low fine-tuning overhead.

2 Related Work

  • VLA models remain vulnerable to visual corruptions, while many use single-scale action heads without explicit multi-scale or smoothness priors.Existing robustification strategies often rely on external large models or focus only on visual perturbations [21].Source: §2 · p. 3
  • Discrete multi-scale action representations can hinder transfer of rich visual features, while specialized heads and retraining can limit reuse of pretrained VLA priors [23] [24] [30].Continuous approaches also introduce multi-scale structure through action chunks, frequency components, or hierarchical perception [25] [26] [27] [29].Source: §2 · p. 3
  • ActionUNet injects multi-scale structure directly into action features, bypassing discretization and specialized heads while retaining base-model generalization with low fine-tuning overhead.The related-work discussion contrasts this design with approaches requiring full-model retraining [30].Source: §2 · p. 3

3 Method

ActionUNet addresses the scale mismatch between semantic features and physical execution by refining temporal-aligned action features with a lightweight U-Net, then decoding them as continuous trajectories. The method retains the base policy’s training objective and uses correlated flow noise to align flow targets with its smoothness prior.

3.1 Preliminary

  • Standard VLA fine-tuning maps high-level multimodal features to individual action steps, often failing to capture the multi-scale requirements of physical execution.ActionUNet responds with a temporal U-Net and continuous decoder that restore multi-scale structure and impose smoothness constraints.Source: §3.1 · p. 3

3.2 Temporal U-Net in Action Feature Space

  • The temporal U-Net downsamples temporal-aligned action features to encode multi-scale priors, then upsamples them and uses skip connections to recover fine local details.Its convolutional stages halve temporal resolution; fusion with encoder skip features restores the original resolution.Source: §3.2 · p. 4

3.3 Conditional SIREN Decoder for Continuous Action Generation

  • The conditional SIREN uses refined action features to define time-dependent local fields, whose smooth fusion produces continuous action trajectories.Time-dependent biases make the local fields infinitely differentiable with respect to time, and Matérn-kernel aggregation encourages second-order smoothness across neighboring feature positions.Source: §3.3 · pp. 4–5

3.4 Training Objective

  • ActionUNet supports both regression and flow-matching policies while retaining the base policy’s training objective; flow training uses temporally correlated Matérn-5/2 noise.The correlated noise reduces high-frequency jitter in the transport target and better matches the decoder’s C2 continuity.Source: §3.4 · pp. 5–6

4 Experiments

Experiments evaluate ActionUNet across simulation benchmarks, real-world manipulation, and component analyses. It improves success rates in perturbed settings and across action-scale compositions, with ablations examining how multi-scale modeling and continuous decoding contribute.

4.1 Experimental Setup

  • The evaluation spans RoboTwin 2.0, LIBERO, LIBERO-Plus, and real-world dual-arm tasks, using success rate as the metric.LIBERO-Plus evaluates models trained on LIBERO under seven perturbation types; real-world hard settings add distractors.Source: §4.1 · p. 6, §4.4 · p. 8

4.2 Results on RoboTwin 2.0

  • 8.8% easy and 9.8% hard success-rate improvements averaged over 12 RoboTwin 2.0 tasks show gains across scale compositions.The hard setting adds stronger environmental perturbations and distractors.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Results on LIBERO and LIBERO-Plus

  • On LIBERO, ActionUNet reaches 97.7% versus 95.6%; on LIBERO-Plus, it reaches 71.9% versus 65.8%, while OpenVLA-OFT improves from 60.9% to 64.6%.LIBERO-Plus tests seven perturbation types; gains are consistent across them, particularly for camera, robot-state, and sensor-noise perturbations.Source: Table 2 · p. 7, §4.3 · pp. 7–8, Table 3 · p. 7

4.4 Real-World Experiments

  • Real-world average success rates rise from 76.0% to 88.7% in easy settings and from 57.3% to 68.7% in hard settings.The evaluation covers three dual-arm tasks on a Realman Robot.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 Ablation Study

  • On RoboTwin 2.0, temporal U-Net alone raises hard-setting success from 34.5% to 37.3%, while SIREN alone raises easy-setting success from 69.0% to 75.5%.The U-Net-only variant slightly drops in the easy setting, while SIREN-only drops in the hard setting; combining them without Matérn reaches 75.0% easy and 45.3% hard, and adding Matérn reaches 77.8% and 46.5%.Source: §4.5 · pp. 8–9, Table 5 · p. 8
  • Grad-CAM responses are strongest when temporal U-Net features match the corresponding ground-truth action scale, indicating scale-aware temporal representations.The compared layers represent fine, mid, and coarse scales; downsampling rate r = 2 performs best in easy and hard settings.Source: Figure 4 · p. 9, §4.5 · p. 9, Figure 5 · p. 9
  • Under the same initial settings, ActionUNet’s predicted end-effector chunks have smaller spatial deviation and smoother trajectories than π0.5 in the visualized examples.The ablation study also analyzes the downsampling-rate effect on Move Can Pot, Stack Blocks Two, and Beat Block Hammer.Source: Figure 6 · p. 9, §4.5 · p. 9, Figure 5 · p. 9

5 Conclusion

ActionUNet adapts pre-trained VLA policies with temporal multi-scale features and a continuous decoder, reporting improved task success and robustness with negligible computational overhead. The appendix details its decoder, action representation, inference measurements, and action-scale analysis; the conclusion notes that scales are fixed, chunks are independent, and scale-aware modeling is confined to the downstream action stage.

5 Conclusion

  • ActionUNet is a lightweight temporal-mapping module that consistently improves task success and robustness under environmental distractions, with negligible computational overhead.The conclusion identifies fixed temporal scales, independent action chunks, and downstream-only scale-aware modeling as current limits.Source: §5 · pp. 9–10

A Additional Method Details

  • The continuous decoder represents an action chunk over normalized time using local SIREN fields and Matérn-weighted aggregation, with second-order smoothness verified for the decoded function.In flow mode, the output is an estimated velocity field; in regression mode, it is the predicted action chunk.Source: A Additional Method Details · p. 14
  • For flow-based training, ActionUNet uses temporally correlated Gaussian noise and a linear path from that noise to the clean action chunk.The velocity target is aligned with the decoder’s second-order temporal prior; the noise covariance includes diagonal jitter for numerical stability.Source: A Additional Method Details · p. 15

Action Representation and Normalization

  • ActionUNet inherits each backbone’s action representation, while preprocessing normalizes actions and maps the discrete horizon to normalized continuous time.π0.5 uses continuous end-effector poses in a flow-matching framework; OpenVLA-OFT uses continuous joint/task-space commands in a regression framework.Source: Action Representation and Normalization · p. 16

B Inference Efficiency

  • Rollout-level policy-computation latency increases by 0.934 seconds in Easy and 0.301 seconds in Hard with ActionUNet.These estimates use average taken steps and compiled policy-call latency; they exclude physical robot execution time.Source: B Inference Efficiency · p. 16, Table 7 · p. 16

C.1 End-effector Displacement

  • Action scale is measured as Cartesian end-effector displacement between consecutive recorded steps, reported in m/step as a dataset-level proxy rather than physical velocity.Task-level scale averages valid consecutive steps from successful episodes.Source: §C.1 · p. 17

C.2 Stage-level Scale Regimes

  • Within-task stages are segmented at stable gripper-state transitions; stages with zero displacement are treated as idle or no-op.Only stable binary gripper states, 0 and 1, define stage boundaries.Source: §C.2 · p. 17

C.3 Scale Thresholds

  • Empirical tertiles divide task- and stage-level displacements into low, medium, and high regimes, and tasks are categorized by their distinct active regimes.A task is single-, dual-, or three-scale when it has one, two, or three distinct active regimes.Source: §C.3 · pp. 17–18

C.4 Scale Analysis on 50 Tasks

The 50-task scale analysis reports overall task scale, the temporal sequence of non-idle stage scales, and each task’s scale composition and number of active regimes.

C.4 Scale Analysis on 50 Tasks

  • Table 8 reports task-level overall scale, stage sequence, scale composition, and number of regimes for 50 tasks, with displacements in m/step.L, M, and H denote low-, medium-, and high-scale stages; stage sequence gives their temporal order, while scale composition gives the active regime set.Source: Table 8 · p. 18, §C.4 · p. 18
  • Put Object Cabinet is listed as a single-regime task with M composition, while Rotate QRcode is listed with L+M composition and two regimes..Source: §C.4 · p. 19
  • Stack Blocks Two and Stack Bowls Two are each listed as three-regime tasks with L+M+H scale composition.Their reported stage sequences include M, L, and H.Source: §C.4 · p. 19

D Additional Experimental Results in Real World

Real-world evaluations visualize π0.5+ActionUNet completing three manipulation tasks in easy and hard settings, including settings with distractor objects.

D Additional Experimental Results in Real World

  • The real-world evaluation covers Place Object Basket, Stack Blocks Two, and Stack Bowls Two, with task-specific criteria for successful placement or stacking.The tasks use 50 keyboard-teleoperated demonstrations each; hard-setting distractor layouts vary across episodes but are held identical across compared models.Source: D Additional Experimental Results in… · p. 19
  • Across all three tasks, π0.5+ActionUNet successfully completes the manipulation process in both easy and hard settings.The visualizations are presented in Figures 7–9; a blinded human evaluator judges success using each task’s stated criteria.Source: D Additional Experimental Results in… · p. 19

E Architecture Exploration: Where Should Multi-scale Features Enter the Continuous Decoder?

The additional experiments compare where multi-scale features enter the decoder, action smoothing against ActionUNet, Matérn bandwidth sensitivity, and a parameter-matched single-scale control. They favor temporal feature fusion before continuous decoding and report stronger hard-setting gains from multi-scale modeling than from added capacity alone.

E Architecture Exploration: Where Should Multi-scale Features Enter the Continuous Decoder?

  • Injecting U-Net features from different temporal resolutions into separate SIREN layers performs consistently worse than the default design.The proposed design first reconciles multi-scale features in the temporal action feature space, then supplies one temporally aligned feature to the continuous decoder.Source: E Architecture Exploration: Where… · pp. 19–21, Table 9 · p. 21

F Comparison with Action Smoothing Baselines

  • ActionUNet achieves higher average success than RTC in both settings, with especially clear gaps on tasks with scale transitions.RTC replans frequently with soft-mask continuation but does not change the learned action representation; its gains are limited and inconsistent.Source: F Comparison with Action Smoothing… · pp. 21–22, Table 10 · p. 22

G Ablation on the Matérn Length-Scale Parameter ρ

  • Changing Matérn length-scale ρ within the tested range has only a minor performance effect, so the experiments fix ρ = 0.2.The passage attributes continuous action-field representation primarily to the conditional SIREN, with the frozen Matérn kernel providing lightweight aggregation.Source: G Ablation on the Matérn Length-Scale… · p. 22, Figure 10 · p. 22

H Parameter-Matched Single-Scale Control

I.1 Repeated Evaluations with Standard Errors

Repeated evaluations show stable improvements across RoboTwin 2.0, LIBERO, and LIBERO-Plus, while sequential tests find statistically significant gains in most RoboTwin comparisons.

I.1 Repeated Evaluations with Standard Errors

  • RoboTwin 2.0 average success rates improve by 9.3 and 10.2 percentage points under Easy and Hard, respectively; LIBERO and LIBERO-Plus gains average 2.1 and 6.0 points.Across repeated evaluations, standard errors are approximately 0.2–0.4 points on RoboTwin 2.0 and small relative to the observed LIBERO gains.Source: §I.1 · p. 23

I.2 Sequential A/B Hypothesis Testing

  • 20 of 24 RoboTwin task-setting comparisons establish statistically significant improvements over π0.5, including 10 of 12 Easy and 10 of 12 Hard comparisons.The remaining four cases are FailToDecide within the available budget; this does not imply policy equivalence or acceptance of H0.Source: §I.2 · p. 24, Table 15 · p. 24

J Additional RoboTwin 2.0 Evaluation

Additional RoboTwin 2.0 evaluations extend beyond the original 12-task subset and show gains across five further tasks.

J Additional RoboTwin 2.0 Evaluation

  • The original RoboTwin 2.0 evaluation covers 12 tasks selected for temporal-scale diversity, with three single-scale, six dual-scale, and three three-scale tasks.The subset also includes five low-, four medium-, and three high-overall-scale tasks.Source: J Additional RoboTwin 2.0 Evaluation · pp. 24–25
  • ActionUNet increases average success rates on five additional RoboTwin 2.0 tasks by 8.0 and 8.2 percentage points under Easy and Hard, respectively.All five additional tasks improve, and their selection broadens scale-composition and overall-scale coverage beyond the original subset.Source: J Additional RoboTwin 2.0 Evaluation · p. 25, Table 16 · p. 25

K Comparison with Additional Multi-Scale and Hierarchical Methods

ActionUNet is competitive with multi-scale and hierarchical policies while retaining the backbone’s native action space. Comparisons and ablations favor feature-space multi-scale modeling with a U-Net and a continuous SIREN decoder.

K Comparison with Additional Multi-Scale and Hierarchical Methods

  • ActionUNet is competitive with recent multi-scale and hierarchical policies while preserving the backbone’s native action space through lightweight action-head fine-tuning.The comparison is reported for LIBERO-Plus in Table 19; external-method results are taken from their corresponding papers.Source: K Comparison with Additional… · pp. 25–26, Table 19 · p. 26

L Comparison with Action-Space Multi-Scale Decomposition

  • ActionUNet consistently improves Easy and Hard performance across evaluated tasks, whereas DCT gains are modest in Easy and slightly below baseline in Hard, and the Laplacian pyramid improves only two of eight comparisons.The alternatives decompose actions directly in output space; ActionUNet instead adds multi-scale temporal structure in temporally aligned action features while preserving the pretrained action mapping.Source: L Comparison with Action-Space… · p. 26, Table 20 · p. 26

M Quantitative Analysis of Trajectory Accuracy and Smoothness

  • 46.5% is ActionUNet’s highest Hard-setting success rate, alongside the lowest Manipulation Drift at 62.28 mm and relatively low Jerk RMS.SIREN-only has the lowest Jerk RMS but lowers Hard success from 34.5% to 33.3% and raises drift from 81.64 mm to 83.03 mm; U-Net-only raises success to 37.3% and lowers drift to 76.05 mm while increasing jerk.Source: M Quantitative Analysis of Trajectory… · p. 27, Table 21 · p. 27

N Module-Wise Robustness Ablation on LIBERO-Plus

On LIBERO-Plus perturbations, the U-Net alone provides broader standalone robustness gains than SIREN alone, while the full model performs best across perturbations. ActionUNet also improves success and partial progress on two precision-demanding RoboDojo tasks.

N Module-Wise Robustness Ablation on LIBERO-Plus

  • Under LIBERO-Plus perturbations, U-Net-only improves average success by 2.4 percentage points and improves all seven categories, while SIREN-only improves the average by only 0.1 points.SIREN-only improves Robot perturbation performance, matches the baseline under Language, and decreases performance under the other perturbations; full ActionUNet performs best under every perturbation and improves the average by 6.0 points.Source: N Module-Wise Robustness Ablation on… · pp. 27–28, Table 23 · p. 28
  • The module-wise results support that multi-scale modeling improves downstream action-feature prediction robustness rather than making the VLM encoder itself more robust.The evaluation trains on clean LIBERO data and tests on seven controlled LIBERO-Plus perturbations.Source: N Module-Wise Robustness Ablation on… · pp. 27–28

O Precision-Demanding Manipulation on RoboDojo

  • On Insert Tubes, ActionUNet improves success by 9.3 percentage points and Progress Score by 8.8 points; on Build Tower, gains are 8.7 and 5.2 points, respectively.Progress Score measures partial completion on the two precision-demanding tasks, in addition to complete success.Source: O Precision-Demanding Manipulation on… · p. 28
  • ActionUNet lowers Jerk RMS from 127.81 to 122.99 m/s3 on Insert Tubes and from 119.75 to 109.53 m/s3 on Build Tower, while remaining less smooth than ground-truth trajectories.Together with the success and progress gains, these results support effectiveness without over-smoothing motions needed for precise manipulation.Source: O Precision-Demanding Manipulation on… · p. 28

Related papers

Loading 2609.34982v1…