SANTS: A State-Adaptive Scheduler for World Action Models

Motivation

• State-dependent denoising scans show shallow refinement can suffice for coarse states, while contact-rich states may benefit from deeper evolution and late predictions can reduce action usefulness.The best action condition is therefore neither the noisiest intermediate state nor necessarily the fully denoised video.Source: §1 · p. 2

Contribution

• SANTS converts fixed video denoising into state-adaptive noise-trajectory control by selecting when to stop and how far to progress.It is designed for pixel-space WAMs because fixed schedules choose terminal depth and noise levels before observing whether the current state contains sufficient action cues.Source: §1 · p. 2
• The scheduler jointly predicts a cumulative stopping hazard and relative noise-progression ratio from the current video state and noise level.Stopping can skip low-value regions, while progression remains finer for contact-rich or alignment-sensitive states.Source: §1 · p. 2

Training

• SANTS is post-trained with a path-level reward after frozen action generation, optimizing downstream action quality while penalizing unnecessary video updates.Only the lightweight scheduler is updated, avoiding manually annotated stopping labels and preserving compatibility as a plug-in module for compatible WAMs.Source: §1 · p. 2

Contribution

• The paper reports a better success–latency tradeoff on RoboTwin 2.0 and real robots through adaptive scheduling.Trajectory analyses and ablations support the proposed mechanism.Source: §1 · p. 2

Evolution of World Action Models

• WAMs couple future visual prediction with action generation to transfer spatiotemporal priors about object motion, contacts, and scene dynamics to manipulation policies.Research spans action-conditioned rollouts, future prediction followed by inverse dynamics, and intermediate predictive representations used directly by action branches.Source: §2 · pp. 2–3

Efficient and Action-Useful WAM Inference

• Diffusion acceleration methods reduce inference cost, but they are generally optimized for image quality, reconstruction error, or generative consistency rather than action utility.This leaves a gap between efficient visual denoising and selecting representations that best condition robot actions.Source: §2 · p. 3
• Recent WAMs show that intermediate denoising features and compact future-aware interfaces can contain action-relevant information without waiting for full video denoising.Existing approaches typically still assume a fixed denoising depth or feature extraction point.Source: §2 · p. 3

3.1 Frozen Video–Action Policy Interface

• SANTS reuses a frozen video–action Transformer, extracting a video latent and pooled video-token representation before conditioning the action branch on the selected terminal state.The scheduler observes intermediate video features without changing the video backbone, heads, or action denoising process.Source: §3.1 · p. 3
• At each decision point, SANTS either terminates video integration or advances the noise level using a learned relative progression ratio.The selected terminal representation is passed to the action branch, while continuation applies a nonuniform noise update.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Stopping Hazard and Relative Noise Progression

• The stopping head accumulates nonnegative hazard increments into termination evidence and uses the cumulative probability for deterministic deployment stopping.Training uses conditional stopping probabilities for stochastic trajectory decisions, whereas deployment stops when cumulative evidence exceeds a threshold.Source: §3.2 · p. 3
• The progression policy chooses continuous noise-space movement: larger retained noise gives conservative progress, while smaller retained noise gives more aggressive progress.This avoids restricting continuation to a preset discrete grid.Source: §3.2 · pp. 3–4

3.3 Path-Level Reward and Scheduler

• SANTS learns from a path-level reward computed after the frozen action branch generates an action chunk, combining action-quality gains with a penalty for executed video-state updates.The reward compares shallow and full-denoising references for the same input, suppresses positive gain on plateaued cases, and emphasizes action-critical frames.Source: §3.3 · pp. 4–5
• Post-training updates only the scheduler and uses PPO with reference regularization while keeping the video–action backbone and action denoising process frozen.This isolates scheduler learning from policy learning.Source: §3.3 · p. 5

4.1 Offline Diagnostic and Scheduling Behavior

• Offline scans show rapid action-error reduction at low-to-moderate depths followed by diminishing returns, with fine-phase error reaching 0.67 versus 0.89 for coarse phases.Errors are normalized to each segment’s 4%-depth baseline across seven scanned depths and phase groups.Source: §4.1 · p. 5, Figure 3 · p. 5
• Full denoising is not the best scanned depth for 55.65% of coarse and 41.33% of fine segments, while adjacent error increases occur in 87.90% and 77.49%.Selecting each segment’s best scanned depth reduces mean error from 0.8943 to 0.8433 for coarse phases and from 0.6571 to 0.6084 for fine phases, although this oracle ignores computation.Source: §4.1 · pp. 5–6, Table 1 · p. 6
• SANTS selects different terminal denoising depths across tasks and successive control segments, including shallow stops and depths near full denoising.These closed-loop trajectories show state-dependent scheduling but do not establish that every selected depth is optimal because unselected depths are not evaluated.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Experimental Setup

• The setup freezes the video–action policy during scheduler training and reports task success and end-to-end latency as the primary evaluation metrics.Simulation compares SANTS with LingBot-VA, Motus, Fast-WAM, and π0.5; real-robot evaluation includes π0.5 and a matched full-denoising WAM.Source: §4.2 · p. 6

4.3 Simulation and Real-Robot Results

• 94.4% overall success at 523.7 ms on RoboTwin 2.0 exceeds full-denoising LingBot-VA’s 92.2% while reducing latency from 2868.4 ms.SANTS is evaluated against multiple baselines under the same evaluation pipeline and comparable fine-tuning budgets.Source: §4.3 · pp. 6–7, Table 2 · p. 6, §4.2 · p. 6
• 73.1% mean success across seven real-robot tasks exceeds π0.5 and Full-Denoising WAM by 18.6 and 7.4 percentage points, while reducing latency from 2769.3 to 581.3 ms.SANTS matches or exceeds full denoising on every task, with 50 trials per task.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Ablation Study

• 94.4% success at 523.7 ms beats the latency-matched Fixed-5-step baseline’s 87.2% at 553.9 ms, while Stop-only and Jump-only remain below full SANTS.The ablation indicates that adaptive terminal-state selection and relative progression are complementary rather than replaceable by a fixed update budget.Source: §4.4 · p. 7, Table 4 · p. 7

• Evaluation covers only RoboTwin 2.0 and two real-robot platforms, leaving generality across embodiments, cameras, task families, and video–action backbones unestablished.The frozen video–action backbone also leaves joint backbone–scheduler optimization unexplored.Source: §5 · p. 7
• Deployment still depends on stopping and cost hyperparameters and mainly reduces video-denoising cost.Extending adaptive budget allocation to action denoising, perception, communication, and control frequency remains future work.Source: §5 · p. 7

• 94.4% RoboTwin 2.0 success and 73.1% mean success on seven real-robot tasks accompany latency reductions of 81.7% and 79.0% versus full-denoising WAMs.Reported latencies are 523.7 ms in simulation and 581.3 ms averaged across real-robot tasks.Source: §6 · p. 7
• SANTS stops when an intermediate video representation suffices for action generation and otherwise predicts progress along the remaining noise axis.Stopping and progression ablations support the adaptive scheduling mechanism.Source: §6 · p. 7
• The results suggest that WAM future reasoning can preserve control benefits without requiring full pixel-level video reconstruction.Selecting intermediate video states removes redundant video inference according to the reported conclusion.Source: §6 · p. 7

A Real-Robot Evaluation Protocol

• Seven tasks are evaluated across four AgileX bimanual tasks and three UR10 kitchen tasks, covering diverse manipulation settings.The AgileX tasks include cloth folding, backpack organization, sock placement, and charger insertion; UR10 tasks include plate transfer, refrigerator placement, and fruit sorting.Source: A Real-Robot Evaluation Protocol · p. 10
• Each method runs 50 independent trials per task under shared task definitions, camera inputs, action interfaces, deployment stacks, termination rules, data, and adaptation budgets.Methods use equivalent initial-state distributions and no additional demonstrations or privileged state information.Source: A Real-Robot Evaluation Protocol · p. 10
• Trials begin from manually reset valid states and end at success, time limits, safety stops, or unrecoverable failures.Human intervention is restricted to pre-trial reset and post-trial safety recovery, with no online correction or mid-trial teleoperation.Source: A Real-Robot Evaluation Protocol · p. 10
• Object positions, orientations, colors, and lighting are varied while background scenes remain partly fixed to test closed-loop robustness under visual variation.Randomization protocols differ by platform but consistently target replaceable objects and lighting conditions.Source: A Real-Robot Evaluation Protocol · p. 10
• Success requires the target object to reach the specified region or pose without safety stops, drops, incorrect placement, or irrecoverable collisions.All trials failing these task-level conditions count as failures, and failure-type labels are not used for training or aggregation.Source: A Real-Robot Evaluation Protocol · p. 10

B Scheduler Implementation and Hyperparameters

• SANTS reads the intermediate video state and noise level, then predicts a stopping hazard and relative noise-progression ratio without updating the WAM backbone or action process.The scheduler can attach to compatible pipelines exposing intermediate video features and nonuniform noise updates.Source: B Scheduler Implementation and… · p. 10
• 73.1% average success follows target-domain scheduler tuning, improving on 70.6% from direct transfer across seven real-robot tasks.Only the scheduler is tuned for 3000 optimization steps while all WAM parameters remain frozen.Source: B Scheduler Implementation and… · p. 10, Table S1 · p. 11
• The scheduler uses a two-layer MLP trunk with two output heads for stopping and relative noise progression.Its input concatenates the pooled video feature with the current noise level.Source: B Scheduler Implementation and… · pp. 10–11
• Figure S1 depicts temporal task progress for real-robot sequences on the AgileX bimanual and UR10 kitchen platforms.Source: Figure S1 · p. 11
• Noise thresholds and stopping thresholds are selected on a held-out validation split and then fixed across tasks.Table S2 summarizes the scheduler network size and fixed hyperparameters.Source: B Scheduler Implementation and… · p. 11, Table S2 · p. 11

Reward Definition

• Normalized sequence and temporal-difference action errors measure physical action quality against demonstrations, including end-effector motion, orientation, gripper state, and motion trends.Errors emphasize timesteps near large end-effector motions and gripper events.Source: C Reinforcement-Learning Post-Training… · p. 11
• Two shallow- and full-denoising reference paths normalize the recoverable action-error gap, reducing entanglement between sample difficulty and scheduling quality.The references use terminal noise levels σlo ≈0.963 and σhi = 0.Source: C Reinforcement-Learning Post-Training… · pp. 11–12
• The difficulty coefficient suppresses gains when shallow and full denoising differ marginally, because such samples contain little recoverable action information.This prevents low-information states from dominating scheduler training.Source: C Reinforcement-Learning Post-Training… · p. 12
• The final quality term rewards positive recovery over shallow denoising while avoiding a blanket incentive to continue every trajectory to σterm = 0.The objective targets action-relevant information rather than video fidelity alone.Source: C Reinforcement-Learning Post-Training… · p. 12

Path Return

• The path return combines weighted action-quality gain with a monotonic cost based on the number of executed video-state updates.This creates an explicit quality–compute trade-off for scheduled trajectories.Source: C Reinforcement-Learning Post-Training… · p. 12

PPO Post-Training

• PPO post-training samples scheduler trajectories, evaluates them after the frozen action branch produces the final chunk, and updates only the scheduler head.The WAM backbone, video head, action head, and action-denoising process remain frozen; fixed-terminal anchors normalize rewards.Source: C Reinforcement-Learning Post-Training… · p. 12

Training and Deployment

• Validation selects the deployed checkpoint using task success rate, path return, and average video updates, while globally fixed hyperparameters are not tuned per task.An overly large compute-cost weight can cause premature termination during contact and alignment phases.Source: C Reinforcement-Learning Post-Training… · p. 12

Path-Reward Ablations

• Removing the update-cost penalty raises success to 94.9% but increases latency to 2197.8 ms, or 4.2× the full reward.The full reward cuts latency by 76.2% relative to this no-cost variant while losing only 0.5 percentage points in success.Source: D Reward-Function Ablations · p. 13, Table S4 · p. 13
• Replacing two-anchor normalized gain with raw action MSE lowers success from 94.4% to 89.7%, despite reducing latency to 436.3 ms.The cheaper policy sacrifices control quality when action error is optimized without normalization.Source: D Reward-Function Ablations · p. 13
• Removing the difficulty gate decreases success to 90.6%, indicating that low-information samples can miscalibrate reward gains when full denoising provides little improvement.The ablation keeps the frozen backbone, scheduler, training budget, and evaluation protocol fixed.Source: D Reward-Function Ablations · pp. 12–13
• Removing temporal-difference error lowers success to 93.6% and increases latency to 576.4 ms, suggesting that motion trends and contact transitions matter to reward quality.The effect is attributed to capturing temporal structure rather than merely reducing computation.Source: D Reward-Function Ablations · p. 13