PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyStep-WAM: Budget-Aligned Distillation and Adaptive Inference for World Action Models

Rui Wang, Xiangyu Wang, Donglin Yang, Yibo Li, Canyang Chen, Zhongrui Wang, Xiaojuan Qi

arXiv:2609.33748v1cs.RO

TL;DR

World-action models use repeated denoising, but action chunks vary in how much precision they need. AnyStep combines frozen-teacher interval distillation with fidelity-aware scheduling from one preview. Across three WAMs, it reduces denoising steps while maintaining average success near full-budget baselines, with real-world evaluation across six tasks.

  • Problem

    Fixed denoising budgets do not accommodate varying action precision needs, while flexible-step WAM training and selection must address accurate transitions and budget-dependent student fidelity.

    Source: §1 · p. 1

  • Method

    AnyStep distills finite-interval transitions from frozen teacher trajectories and uses a single-preview risk-benefit scheduler to select the smallest budget predicted to meet fidelity requirements.

    Source: §1 · p. 2

  • Results

    On RoboTwin, adaptive inference cuts mean denoising steps by 60.2% on Motus, 49.8% on FastWAM, and 85.28% on LingBotVA, with average success within 0.24 percentage points of full-budget baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across three WAM backbones and six real-world tasks, results support coupling tunable-budget generation with fidelity-aware scheduling for efficient inference at comparable or higher average success.

    Source: §6 · p. 10

Abstract

from arXiv · show

World-action models (WAMs) couple predictive visual modeling with action generation, typically relying on iterative denoising with a fixed denoising steps. However, manipulation tasks contain actions chunks with varying sensitivity to generation errors: critical actions require precision, while less sensitive actions allow faster generation with fewer denoising steps. Here we introduce AnyStep World Action Model, a general framework for tunable-budget prediction and scene-dependent computation allocation. Our budget-aligned teacher-trajectory distillation trains interval-conditioned flow maps using explicit frozen-teacher transitions and shared low-rank adapters, supporting action generation from one-step prediction to multi-step refinement. Building on this capability, a lightweight risk-benefit scheduler predicts teacher-curvature-based difficulty and budget-specific student-teacher fidelity from a single one-step preview, selecting the smallest budget predicted to satisfy risk-adaptive fidelity requirements. We evaluate our framework on three widely used WAMs Motus, FastWAM, and LingBotVA using RoboTwin 2.0. Our method reduces average denoising steps by 60.2%, 49.8%, and 85.28%, respectively, while maintaining baseline task success rates. In particular, our AnyStep training substantially improves model performance under a one-step denoising budget, increasing task success rates by 7.07%, 12.08%, and 8.94% on Motus, FastWAM, and LingBotVA, respectively. Experiments on six real-world manipulation tasks further validate its effectiveness.

1 INTRODUCTION

AnyStep WAM addresses the cost of repeated denoising and the mismatch between fixed budgets and action chunks with differing precision needs. It combines frozen-teacher interval distillation with a risk-benefit scheduler and reports lower denoising budgets while maintaining task success.

1 INTRODUCTION

  • Fixed budgets can miss varying precision demands, while adapting AnyFlow in WAM experiments produced variable targets, frequent gradient spikes, and low success at one or two steps.Contact-sensitive actions may require more refinement than less sensitive motions.Source: §1 · p. 1
  • Frozen-teacher trajectory displacements supervise finite transitions across budgets, while continuation training from student preview states and shared low-rank adapters support refinement within one model.The targets avoid temporal-derivative estimates from the evolving student.Source: §1 · p. 2
  • A single one-step preview predicts teacher-trajectory difficulty and budget-specific student–teacher fidelity, then selects the smallest budget predicted to meet a difficulty-dependent fidelity threshold.Reusing the preview avoids extra denoising passes, online teacher evaluation, and candidate rollouts.Source: §1 · p. 2
  • 60.2%, 49.8%, and 85.28% fewer mean denoising steps on Motus, FastWAM, and LingBotVA, respectively, with average success within 0.24 percentage points of full-budget baselines.On six real-world tasks, per-call inference speedups are 1.67–6.14× with comparable or higher average success.Source: §1 · p. 2
  • 7.07, 8.94, and 12.08 percentage-point one-step success improvements on Motus, LingBotVA, and FastWAM, respectively, show gains under the one-step budget.The evaluations use RoboTwin and six real-world tasks.Source: §1 · p. 2

2 RELATED WORK

Prior work compresses iterative generation and adapts computation using denoising or input statistics. AnyStep instead aligns finite-interval supervision to frozen teacher trajectories and models student fidelity separately from trajectory difficulty.

2 RELATED WORK

  • Consistency and distribution-matching methods compress generation to one or a few steps, while MeanFlow and AnyFlow learn finite-interval flow maps for flexible-step generation.Related work includes consistency learning (Luo et al., 2023), distribution matching distillation (Yin et al., 2024b;a), MeanFlow, and AnyFlow.Source: §2 · p. 3
  • Adaptive computation uses trajectory statistics, input-conditioned gating, or learned policies; AnyStep uses teacher-trajectory variation for risk and separately models budget-dependent student fidelity.Trajectory curvature is associated with larger integration errors under coarse discretization.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe WAM action generation and flow-based denoising, then distinguish finite-interval flow maps from derivative-based supervision. These concepts set up prediction across inference budgets.

3.1 WORLD ACTION MODELS

  • WAMs condition action-chunk generation on an observation and instruction, combining predictive visual modeling with action generation.The paper studies flow-based inference branches that denoise actions and, where applicable, future visuals.Source: §3.1 · p. 3

3.2 FLOW MATCHING AND FLOW MAPS

  • Flow matching transports noise x_1 to data x_0 by solving a context-conditioned velocity-field ODE from t = 1 to 0.The velocity field is conditioned on context c.Source: §3.2 · p. 3
  • A flow map represents a finite-time transition along an ODE trajectory, with endpoint conditioning supporting different transition intervals and inference budgets.Its interval-average velocity is defined over the interval from r to t.Source: §3.2 · p. 3
  • MeanFlow and AnyFlow construct derivative- or finite-difference-based targets that depend on the evolving student and can be sensitive to prediction and optimization noise.MeanFlow uses a Jacobian-vector product, whereas AnyFlow uses central finite differences.Source: §3.2 · p. 3

4 METHOD

AnyStep WAM combines integral-supervised flow maps for tunable-budget generation with a lightweight scheduler trained on offline teacher–student evaluations. At inference, a one-step preview informs budget selection and is reused in the chosen trajectory.

4.1 ENABLING ANYSTEP WAMS WITH INTEGRAL FLOW-MAP DISTILLATION

  • Frozen-teacher finite-interval targets are sampled from candidate schedules; local, endpoint, and recovery losses additionally train continuation from preview-recycled student states.Teacher endpoints make the interval targets independent of student parameters, and the recovery loss uses detached student-generated states.Source: §4.1 · pp. 4–5

4.2 SELF-AWARE RISK-BENEFIT PREDICTION

  • A lightweight predictor uses one-step-preview features and offline supervision to estimate teacher-trajectory difficulty and budget-specific student fidelity.Teacher velocity variation serves as a budget-independent difficulty proxy; student fidelity is evaluated along the preview-recycled path for each candidate budget.Source: §4.2 · pp. 5–6
  • Risk is relative denoising difficulty, not task-failure probability, while preview agreement and subsequent recovery fidelity remain proxies for terminal action error or task success.The fidelity score combines directional and normalized velocity agreement, with earlier denoising intervals emphasized.Source: §4.2 · pp. 5–6

4.3 RISK-CONDITIONED ADAPTIVE INFERENCE

  • The scheduler selects the smallest budget meeting risk-conditioned fidelity requirements, or the budget with the largest summed predicted benefit if none qualifies.Higher predicted difficulty imposes a stricter fidelity requirement.Source: §4.3 · p. 6
  • The selected trajectory reuses the one-step preview, requiring exactly K⋆ denoising evaluations including the preview, without online teacher access or candidate rollouts.For K⋆=1, the preview prediction is returned directly; otherwise, it is re-noised to the first intermediate time.Source: §4.3 · p. 6

5 EXPERIMENT

Experiments evaluate AnyStep on RoboTwin 2.0 and six real-world tasks across three WAM backbones. Adaptive inference reduces denoising while maintaining RoboTwin success and achieves higher average success on the real robot.

5 EXPERIMENT

  • Evaluation covers 50 bimanual RoboTwin tasks under clean and randomized settings, plus six named manipulation tasks on a Unitree G1D dual-arm robot.Source: §5 · p. 6

5.1 SIMULATION EXPERIMENTS

  • AnyStep keeps average RoboTwin success within 0.24 percentage points of baseline while reducing denoising steps by 60.2% for Motus, 49.8% for FastWAM, and 85.28%/92.64% for LingBotVA video/action branches.Mean budgets are 3.98, 5.03, and 3.68 steps, respectively.Source: Table 1 · p. 7, §5.1 · p. 7
  • Against AnyStep+Gate, the scheduler improves average success by 0.70, 1.43, and 1.88 percentage points while reducing mean steps by 21.8%, 24.5%, and 23.5% on Motus, FastWAM, and LingBotVA.Source: §5.1 · pp. 7–8
  • At one denoising step, AnyStep improves average success over Base by 7.07, 12.08, and 8.94 percentage points on Motus, FastWAM, and LingBotVA, respectively.It also exceeds Flash-WAM by 3.19, 7.58, and 1.30 percentage points, respectively.Source: §5.1 · p. 8
  • Frozen-teacher targets remain concentrated and gradients stable, unlike finite-difference targets, where 45.9% of steps exceed the 1.0 clipping threshold.AnyStep also remains best at 2 and 10 denoising steps.Source: Figure 4 · p. 8, §5.1 · p. 8

5.2 REAL-WORLD EXPERIMENTS

  • Across six real-world tasks, average success reaches 69.17%, 71.67%, and 70.00% while denoising steps fall by 59.4%, 63.9%, and 85.84% for Motus, FastWAM, and LingBotVA.Per-call speedups are 1.67×, 2.30×, and 6.14×, respectively.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 ABLATION EXPERIMENTS

  • On Motus, removing interval supervision lowers two-step success from 86.21% to 78.94% and adaptive success from 87.96% to 78.09%; endpoint and continuation ablations also lower performance.Source: Table 3 · p. 9, §5.3 · p. 9
  • With randomized conditions, the clean-only scheduler reaches 87.56% success at 4.05 steps versus Full’s 87.80% at 3.94; threshold shifts of ±5 points keep success within 0.36 points of Full.Source: Table 3 · p. 9, §5.3 · p. 9

6 CONCLUSION

AnyStep WAM combines tunable-budget generation with fidelity-aware scheduling, and its formulation accommodates different video-and-action generation arrangements. Flow maps express finite-time transitions that can be conditioned on different inference budgets.

6 CONCLUSION

  • AnyStep WAM combines frozen-teacher integral flow-map distillation with a lightweight scheduler that selects the smallest predicted-feasible budget and reuses a one-step preview.Across Motus, FastWAM, and LingBotVA, it improves one-step success and reduces denoising steps and latency while maintaining comparable or higher average success rates.Source: §6 · pp. 9–10

A WORLD ACTION MODEL FORMULATIONS

  • The WAM formulation expresses a visuomotor policy as a marginal over future visual states and actions, without prescribing their generation order.The framework applies budget allocation to video and action denoising when both are used at inference, or to action denoising alone otherwise.Source: A WORLD ACTION MODEL FORMULATIONS · p. 12

B FLOW MATCHING AND FLOW MAPS

  • A flow map represents a finite-time transition along an ODE trajectory, and conditioning on both time endpoints lets one model support different inference budgets.The model parameterizes the transition using a predicted average velocity.Source: B FLOW MATCHING AND FLOW MAPS · p. 12
  • Derivative-based flow-map supervision uses local derivative information from the learned average-velocity field, through a Jacobian-vector product or a finite-difference approximation.The passage notes that such derivative-dependent objectives can be sensitive to optimization and prediction noise.Source: B FLOW MATCHING AND FLOW MAPS · p. 12

C IMPLEMENTATION DETAILS

Implementation spans RoboTwin 2.0 and real-world demonstrations, with frozen pretrained backbones adapted through shared LoRA modules and an offline-trained scheduler.

C IMPLEMENTATION DETAILS

  • RoboTwin 2.0 provides clean and randomized demonstrations for 50 tasks, and each backbone is trained as one multi-task AnyStep model across all tasks.The two demonstration settings are sampled with equal probability during distillation.Source: C IMPLEMENTATION DETAILS · p. 12
  • Distillation freezes the original backbone and vision-language parameters and trains shared LoRA adapters across tasks and denoising budgets.The adapters use rank 128, and the objective combines instantaneous-velocity, endpoint, interval, and recycled-state recovery supervision; Table 4 summarizes hyperparameters and trainable parameter counts.Source: C IMPLEMENTATION DETAILS · p. 13, Table 4 · p. 13
  • The real-world dataset contains 300 demonstrations across six manipulation tasks, and each backbone’s base model is frozen before AnyStep LoRA adaptation.The base model and its AnyStep variant use the same demonstration dataset.Source: C IMPLEMENTATION DETAILS · p. 13
  • All three backbones use candidate budgets from 1 to 10, but their inference paths differ: Motus denoises video and actions jointly, FastWAM denoises actions only, and LingBotVA generates video before actions.Motus uses synchronized video and action schedules; LingBotVA conditions action generation on the denoised video.Source: C IMPLEMENTATION DETAILS · p. 14
  • The scheduler is trained offline on frozen teacher-student evaluations, using preview features to predict teacher-derived risk and budget-dependent fidelity.A shared Transformer processes preview features with risk and budget queries; Figure 6 depicts the architecture, and scheduler training requires no additional environment interaction or demonstrations.Source: Table 5 · p. 13, C IMPLEMENTATION DETAILS · p. 14, Figure 6 · p. 14

D TRAINING-FREE GATING BASELINE

The training-free gating baseline probes a one-step action proposal, combines action-coordination and visual-irregularity measures, then selects a denoising budget.

D TRAINING-FREE GATING BASELINE

  • The baseline first executes complete one-step action denoising and measures how strongly the dominant moving joints change within that proposal.The action-coordination metric is based on each joint’s relative contribution to predicted motion.Source: D TRAINING-FREE GATING BASELINE · pp. 14–15
  • Visual irregularity uses changes in patch-wise RGB differences between the latest three observations, with each difference based on a 4×4 patch grid.The patch-wise difference is the 90th percentile of mean absolute differences.Source: D TRAINING-FREE GATING BASELINE · p. 15
  • The action and visual metrics are normalized using frozen ECDFs before they are combined for budget selection.The passage gives the combination as 2C + 1/2V.Source: D TRAINING-FREE GATING BASELINE · p. 15
  • With K = 10 and γ = 1.6965, the selected budget determines whether the one-step probe is reused or a full trajectory is rerun from the same initial noise.The probe is reused when N = 1.Source: D TRAINING-FREE GATING BASELINE · p. 15

E FIXED DENOISE STEP RESULTS ON ROBOTWIN

Fixed-budget RoboTwin 2.0 results report a pronounced small-budget performance drop for AnyFlow, with larger budgets partially mitigating errors through smaller updates.

E FIXED DENOISE STEP RESULTS ON ROBOTWIN

  • AnyFlow exhibits a pronounced performance drop at small denoising budgets in the fixed-step RoboTwin 2.0 success-rate results.Table 6 reports complete fixed-step success rates for Base, AnyFlow, and AnyStep, averaging clean and randomized settings.Source: Table 6 · p. 15, E FIXED DENOISE STEP RESULTS ON ROBOTWIN · p. 15
  • Small grasping or contact errors can cause task failure, and the reported training instability may compromise large denoising updates.Increasing the budget enables smaller updates and partially mitigates these errors, consistent with AnyFlow’s improving success rates as steps increase.Source: E FIXED DENOISE STEP RESULTS ON ROBOTWIN · p. 16

F ADDITIONAL ABLATIONS ON FASTWAM AND LINGBOTVA

FastWAM and LingBotVA ablations extend the Motus variant comparisons. The section defines threshold shifts and reports fixed-step and adaptive measures for both models.

F ADDITIONAL ABLATIONS ON FASTWAM AND LINGBOTVA

G ROBOTWIN DETAILED RESULTS

The detailed RoboTwin results compare base models with AnyStep variants across tasks. They report clean and randomized success rates alongside AnyStep's average denoising steps.

G ROBOTWIN DETAILED RESULTS

  • AnyStep adaptively selects the denoising budget for each prediction chunk, while base schedules use 10 steps for Motus and FastWAM and 25 video plus 50 action steps for LingBotVA.Source: G ROBOTWIN DETAILED RESULTS · p. 17
  • Tables 9–11 compare task-wise success rates under clean and randomized settings and average AnyStep denoising steps, with aggregate results in the final row.Source: Table 9 · p. 18, Table 10 · p. 19, Table 11 · p. 20
  • Table 9's final row reports the unweighted mean across Motus tasks.Source: Table 9 · p. 18

Related papers

Loading 2609.33748v1…