PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Throw Away the Tail: Action Upcycling for Policy Acceleration

Taesung Kwon, Jangho Park, Sunwoo Park, Youngmin Kim, Seonghyun Jin, Youngjun Jun, Kyumin Choi, Jong Chul Ye

arXiv:2609.34911v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Chunked robot policies trade frequent, reactive replanning against the cost of repeated policy calls, while existing adaptive-horizon methods use model internals or extra samples. Action Upcycling reuses discarded actions when motion remains smooth, reducing calls by 1.2–1.7× with no loss in success rate across diverse policies and tasks.

  • Problem

    Choosing how many actions to execute before replanning trades environmental reactivity against the cost of repeated backbone calls, while adaptive methods rely on model internals or extra samples.

    Source: §1 · p. 1, §4.1.1 · p. 6

  • Method

    Action Upcycling uses the predicted action chunk to reuse its trustworthy tail until velocity begins to fluctuate, without training, model internals, or extra samples.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    Across four policies, three benchmarks, and real-world manipulation, Action Upcycling reduces policy calls by 1.2–1.7× with no loss in success rate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Action Upcycling applies to chunked policies at negligible cost and composes with other acceleration methods, including few-step sampling and streaming action decoding.

    Source: §5 · p. 9

Abstract

from arXiv · show

Modern robot policies predict a chunk of future actions from a single observation, execute only a prefix, and discard the rest before replanning. Choosing the length of this prefix, the execution horizon, poses a trade-off between reactivity and efficiency. A short horizon keeps the policy reactive to the environment, but requires frequent policy calls. Recent test-time methods adaptively select the horizon for each chunk, but they either read model internals, where the signal must be chosen for each architecture, or draw extra samples, which adds cost. We propose Action Upcycling, a training-free algorithm that reuses actions the policy would otherwise discard, without accessing model internals or drawing extra samples. We find that discarded actions stay close to their replanned versions as long as the action velocity remains smooth. Action Upcycling therefore extends the execution horizon up to the point where the velocity begins to fluctuate. Extensive experiments on simulated and real-world manipulation tasks show that Action Upcycling reduces policy calls by 1.2-1.7x with no loss in success rate, across multiple Vision-Language-Action Models (VLAs) and even a World Action Model (WAM). It applies to any chunked policy at negligible cost and is orthogonal to other policy acceleration methods such as few-step sampling and streaming action decoding, opening a new axis for policy acceleration.

1 INTRODUCTION

Chunked policies balance reactive replanning against the cost of repeated backbone calls. Action Upcycling reuses trustworthy discarded actions based on motion smoothness, reducing calls without sacrificing success across diverse policies and tasks.

1 INTRODUCTION

  • Action chunking predicts a sequence because demonstrated motions can carry intent across steps, while reducing compounding error and amortizing backbone latency.Examples include slowing before contact or pausing at decision boundaries.Source: §1 · p. 1
  • Short execution horizons improve reactivity through frequent re-observation, but each new observation requires a costly backbone forward pass.The optimal horizon varies across tasks and within a task.Source: §1 · p. 1
  • Existing adaptive-horizon methods either inspect architecture-specific model internals or draw multiple samples, adding design complexity or computational cost.The figure compares discarded actions with replanned versions across four policies and relates their RMSE to velocity fluctuation.Source: §1 · p. 2, Figure 1 · p. 2
  • Action Upcycling is training-free and reuses discarded actions without accessing model internals or drawing extra samples, extending execution while velocity remains smooth.Its signal comes from the already-sampled action chunk, and it is orthogonal to few-step sampling and streaming action decoding.Source: §1 · p. 2
  • Across four policies, three benchmarks, and real-world manipulation, Action Upcycling reduces policy calls by 1.2–1.7× with no loss in success rate.It also combines with other acceleration methods and works on a World Action Model.Source: §1 · p. 2

2 RELATED WORK

Prior work explains the benefits and trade-offs of action chunking, adapts execution horizons using internal signals or extra samples, and accelerates inference through per-call cost reductions. Action Upcycling instead reduces the number of policy calls and complements per-call acceleration.

Action chunking and the execution horizon

  • Action chunking predicts action sequences but leaves the environment unobserved during open-loop execution, creating a trade-off between long-term consistency and short-term reactivity.Most policies therefore execute only a prefix and discard the remaining actions.Source: §2 · p. 3

Adaptive execution horizons

  • Adaptive-horizon methods use model attention or multiple sampled chunks, requiring model internals or extra forward passes; Action Upcycling uses only one action chunk.Architecture-specific choices are needed for attention-based signals.Source: §2 · p. 3

Policy acceleration

  • Most acceleration methods reduce the cost of each policy call, whereas Action Upcycling reduces the number of calls, making the approaches complementary.The paper reports combining it with few-step sampling and FlashVLA.Source: §2 · p. 3

3 ACTION UPCYCLING

Action Upcycling reuses trustworthy actions from a predicted chunk’s discarded tail, using velocity fluctuation to decide how far to extend execution before replanning. A threshold selected from pooled signals targets a desired mean execution length.

3 ACTION UPCYCLING

  • Action Upcycling reuses trustworthy tail actions while their accumulated velocity fluctuation stays below a threshold, extending execution before replanning.It uses the predicted action chunk without accessing model internals or drawing extra samples.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

  • The policy predicts H actions, executes a prefix of h, and may reuse up to Δh_i additional tail actions; mean execution length h̄ defines the upcycling ratio r.For a fixed episode length, larger r proportionally reduces policy calls; r = 2 halves them, while r = 1 recovers the default.Source: §3.1 · pp. 3–4

3.2 VELOCITY FLUCTUATION AS A SIGNAL

  • Accumulated changes between consecutive action velocities stay small during steady predicted motion and rise when velocity fluctuates, providing a signal for tail trustworthiness.The signal sums velocity changes beyond the original execution horizon h; its nonnegative terms make it non-decreasing along the tail.Source: §3.2 · p. 4

3.3 ADAPTIVE HORIZON SELECTION

  • A threshold τ sets the longest execution length whose accumulated fluctuation remains at or below τ, determining how many tail actions are reused.The method pools fluctuation signals from predicted chunks and chooses the smallest threshold that achieves the target mean execution length; pool construction requires no additional rollouts.Source: §3.3 · p. 5, §3.2 · p. 5

4 EXPERIMENTS

Experiments evaluate four policies on three simulation benchmarks and deploy two policies on physical manipulation tasks. Across these settings, Action Upcycling reduces policy calls while matching or improving success rates.

4.1.1 QUANTITATIVE RESULTS

  • Across four policies and three benchmarks, Action Upcycling reduces policy calls by 1.2–1.7× while matching or improving success rates.For π0.5, success improves on LIBERO (96.9% to 97.9%), LIBERO-Plus (83.7% to 86.5%), and RoboTwin 2.0 (59.7% to 60.5%). FastWAM also improves success while reducing calls by 1.2–1.5×.Source: §4.1.1 · pp. 5–6, Table 1 · p. 6
  • On LIBERO and LIBERO-Plus, Action Upcycling leads adaptive-horizon comparisons in success, call reduction, and inference time; on RoboTwin 2.0, it improves success while reducing calls.AutoHorizon reduces calls slightly more on RoboTwin 2.0 but degrades success, while AAC increases calls there and incurs higher latency and inference time from multiple samples.Source: Table 2 · p. 6, §4.1.1 · p. 6
  • Action Upcycling combines with few-step sampling and streaming decoding, yielding total speed-ups of 2.8× and 6.7×, respectively.With N=2 denoising steps, it improves success over the same-N baseline; with FlashVLA, it reduces calls by 1.51× while improving success.Source: §4.1.1 · p. 6, Table 3 · p. 7, Table 4 · p. 7

4.1.2 ABLATION STUDY

  • Adaptive execution improves success over fixed-length execution by 0.9–1.3 points across four policies, while fixed execution falls 0.7–1.1 points below baseline for three.Selected execution lengths spread over a range, reflecting adaptation to each chunk’s velocity fluctuation rather than use of one fixed length.Source: Table 5 · p. 7, §4.1.2 · p. 7, Figure 3 · p. 7

4.2 REAL-WORLD EXPERIMENTS

  • Real-world evaluation uses π0.5 and GR00T N1.6 on a YAM arm, with 424 collected episodes spanning eight pick-and-place and drawer tasks.π0.5 is evaluated on all eight tasks and GR00T N1.6 on four pick-and-place tasks, with 10 episodes per task.Source: §4.2 · p. 8

4.2.1 QUANTITATIVE RESULTS

  • π0.5’s overall success rises from 72/80 to 77/80 as calls fall from 48.3 to 34.0 per episode; GR00T N1.6 rises from 34/40 to 35/40 as calls fall from 90.6 to 71.0.Time per episode falls from 24.9 s to 21.5 s for π0.5 and from 28.0 s to 24.9 s for GR00T N1.6; across tasks, success matches or improves while calls decline.Source: Table 6 · p. 8, §4.2.1 · p. 8, Table 7 · p. 9

4.2.2 QUALITATIVE RESULTS

  • In Figure 4, baselines time out at 60 s, while Action Upcycling completes the drawer and pick-and-place tasks in 27 s and 46 s.The rollouts use π0.5 for the drawer task and GR00T N1.6 for pick-and-place.Source: Figure 4 · p. 9, §4.2.2 · p. 9

5 CONCLUSION

Action Upcycling reuses discarded action-chunk tails and uses accumulated velocity fluctuation to decide how far they can be trusted. It requires no model internals, extra samples, or additional training, and is evaluated across VLAs, a World Action Model, benchmarks, and real-world experiments.

5 CONCLUSION

  • Across three VLAs, a World Action Model, three benchmarks, and real-world experiments, Action Upcycling demonstrates policy acceleration and composes with orthogonal acceleration methods.The conclusion describes this as opening a new axis for faster policy inference.Source: §5 · p. 9
  • Accumulated velocity fluctuation signals how far the discarded tail can be trusted while motion remains smooth.Discarded actions stay close to their replanned versions as long as motion remains smooth.Source: §5 · p. 9
  • Action Upcycling reuses the tail of each action chunk without model internals, extra samples, or additional training.The signal is computed from a single predicted chunk.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The method is training-free, and the paper describes what can be reproduced from public materials as well as what will not be released.

REPRODUCIBILITY STATEMENT

A FURTHER IMPLEMENTATION DETAILS

Simulation experiments apply Action Upcycling to released policy checkpoints without fine-tuning; the appendix describes four policies and lists their horizon settings in Table 8.

A.1 POLICIES

  • The simulation experiments use released policy checkpoints without fine-tuning.The execution and prediction horizons for each policy are listed in Table 8.Source: §A.1 · p. 13, Table 8 · p. 13
  • π0.5 and SmolVLA use flow-matching action experts with 10 denoising steps; their attention arrangements differ.π0.5 uses shared attention to vision-language tokens at every layer; SmolVLA alternates cross-attention and self-attention blocks and attends to a frozen vision-language model.Source: §A.1 · p. 13
  • GR00T N1.7 uses a flow-matching action expert with 4 denoising steps, while FastWAM jointly generates actions and future video latents with 10 steps.GR00T N1.7 receives features from a frozen vision-language model through cross-attention.Source: §A.1 · p. 13

A.2 BENCHMARKS

The appendix describes the evaluated benchmarks and states that official protocols are followed unless specified otherwise; it also identifies the baseline methods and implementation settings used for comparisons or combinations.

A.2 BENCHMARKS

  • The experiments follow each benchmark’s official evaluation protocol unless otherwise specified.Source: §A.2 · p. 13
  • LIBERO is a simulated single-arm benchmark with four ten-task suites, while LIBERO-Plus tests robustness across seven perturbation dimensions.LIBERO-Plus shares LIBERO’s action space; LIBERO actions are relative end-effector displacements.Source: §A.2 · p. 13
  • RoboTwin 2.0 covers 50 bimanual manipulation tasks in clean and randomized settings, using absolute joint-position actions.Source: §A.2 · p. 13

A.3 BASELINES

  • AAC uses 20 sampled chunks to select its execution point, AutoHorizon uses action-expert attention, and FlashVLA advances a four-chunk rolling buffer by one denoising step per call.The methods are compared or combined using official implementations and default hyperparameters unless otherwise specified.Source: §A.3 · p. 13

A.4 REAL-WORLD EXPERIMENTAL SETUP

The real-world setup uses a single YAM follower arm with overhead and wrist cameras; π0.5 and GR00T N1.6 are fine-tuned with their respective official recipes. Experiments use specified GPU configurations for training, deployment, and measurement.

A.4 REAL-WORLD EXPERIMENTAL SETUP

  • All real-world experiments use only the right YAM follower arm, with overhead and wrist RGB cameras and seven-dimensional joint-state and action interfaces.The setup includes two leader arms for teleoperation, and limits joint velocity to 0.75 rad/s.Source: §A.4 · p. 14, Figure 5 · p. 14
  • π0.5 is fine-tuned for 20,000 steps with all parameters updated, using zero-padded 32-dimensional state and action inputs.The underlying YAM state and action are seven-dimensional; padded dimensions are ignored at deployment.Source: §A.4 · p. 14
  • GR00T N1.6 is fine-tuned for 10,000 steps, updating embodiment-specific projectors and the diffusion transformer while freezing the vision-language backbone.The YAM is registered as a new embodiment with two camera views and six arm joints plus the gripper.Source: §A.4 · p. 14

A.5 COMPUTING RESOURCES

  • Simulation experiments run one policy per RTX 3090 GPU, while policy-call latency and episode inference time are measured on an RTX 4090 GPU.Real-world π0.5 and GR00T N1.6 are fine-tuned on an NVIDIA B200 and deployed on an RTX 4090.Source: §A.5 · p. 14

B.1 POOL CONSTRUCTION

A small rollout-derived signal pool is sufficient to select the upcycling threshold, and the pool can be built online as well as offline. Increasing the upcycling ratio can reduce policy calls while preserving or slightly improving success within a range, but larger ratios are associated with declining success.

B.1 POOL CONSTRUCTION

  • Using signals from only a random 2% of chunks yields nearly the same selected τ and mean execution length as using all chunks across all four policies.The signal pool can be built from policy rollouts without demonstrations or success labels.Source: §B.1 · p. 15, Table 9 · p. 15
  • An online signal pool achieves comparable success and call reduction to an offline pool, while the offline pool gives slightly higher success and shorter episode time in the reported experiments.The comparison is for π0.5 on LIBERO.Source: §B.1 · p. 15, Table 10 · p. 15

B.2 EFFECT OF THE UPCYCLING RATIO

  • For every policy on LIBERO, an upcycling ratio between 1.2 and 1.7 slightly improves success while reducing policy calls; success gradually drops beyond that range.The authors suggest larger ratios admit tail actions with greater velocity fluctuation.Source: §B.2 · p. 15, Figure 6 · p. 15

C MORE QUALITATIVE REAL-WORLD RESULTS

Additional real-world examples show faster completion with Action Upcycling across drawer tasks using π0.5 and pick-and-place tasks using GR00T N1.6. In each set, two baseline timeouts become successes, while both methods succeed on the other two tasks.

C MORE QUALITATIVE REAL-WORLD RESULTS

  • In two π0.5 drawer tasks, Action Upcycling succeeds where the baseline times out; in the other two, both succeed, with upcycling completing faster.The figure shows baseline results above Action Upcycling results, with red and green boxes marking failure and success.Source: Figure 7 · p. 16
  • In two GR00T N1.6 pick-and-place tasks, Action Upcycling succeeds where the baseline times out; in the other two, both succeed, with upcycling completing faster.The figure shows baseline results above Action Upcycling results, with red and green boxes marking failure and success.Source: Figure 8 · p. 17

Related papers

Loading 2609.34911v2…