PaperlayerView exact revision on arXiv

Source-linked AI summary

PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models

Xianghui Wang, Feng Chen, Wenbo Zhang, Hua Yan, Zixuan Wang, Changsheng Li, Yinjie Lei

arXiv:2606.22540v3cs.CV

TL;DR

VLA deployment efficiency is limited by unreliable action-chunk tails and redundant physical steps, while policy efficiency remains underexplored. PolicyTrim uses two-stage reinforcement-learning post-training to extend reliable action horizons and reduce redundant steps, tripling action-chunk utilization and delivering up to 5.83× end-to-end speedup without sacrificing success rate.

  • Problem

    Policy efficiency in VLA models remains largely unexplored despite being governed by executable action-chunk length and required physical steps.

  • Method

    PolicyTrim applies two-stage reinforcement-learning post-training with dynamic horizon exploration and redundancy-aware rewards to extend reliable chunks and reduce redundant steps.

  • Results

    3× action-chunk utilization and up to 5.83× end-to-end deployment speedup are achieved while physical execution steps are reduced without sacrificing task success rates.

  • Takeaways & Limitations

    PolicyTrim targets total inference frequency as an orthogonal complement to compute-centric acceleration for more efficient VLA deployment.

  • Takeaways & Limitations

    The horizon reward must be activated only when a sampled group contains at least one successful trajectory to avoid encouraging unreliable long-chunk behavior.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models provide a unified paradigm for robotic manipulation, yet their real-world deployment is often bottlenecked by execution efficiency. While existing efforts predominantly focus on compute-centric efficiency to reduce per-step inference latency, the intrinsic \textbf{policy efficiency} of these models remains largely unexplored. Policy efficiency is fundamentally affected by two factors, namely the effective executable length of predicted action chunks and the total physical steps required to complete a task. These two factors jointly determine the total number of forward inference calls during execution. We observe that current VLA policies struggle with planning unreliability and action redundancy, suffering from severe prediction degradation at the tail of action chunks and tending to generate unnecessarily redundant physical steps. To address this, we propose \textbf{PolicyTrim}, a reinforcement learning-based post-training framework that extends the reliable action chunk length and reduces redundant physical steps. For reliable chunk extension, we employ a dynamic exploration strategy that explicitly rewards the successful completion of longer executable lengths, progressively pushing the trustworthy prediction horizon to its empirical limit. For step efficiency, we design a redundancy-aware reward that directly favors successful task completions with fewer steps while penalizing unreproducible shortcuts, effectively eliminating redundant physical actions. Extensive experiments across three benchmarks and three VLA models demonstrate that PolicyTrim improves action chunk utilization by 3$\times$ and reduces physical execution steps by 51.4\%. Ultimately, our framework delivers up to a 5.83$\times$ end-to-end deployment speedup without compromising task success rates.

1 Introduction

PolicyTrim addresses an overlooked VLA deployment bottleneck: policy efficiency, which depends on reliable action-chunk execution length and required physical steps. Its RL-based post-training extends the reliable planning horizon and reduces redundant execution, yielding higher chunk utilization, fewer steps, and faster deployment without sacrificing success rate.

  • Motivation: Repeated rollouts show variable step counts, while longer chunk execution degrades success and increases physical steps because of unreliable tail predictions.The figure identifies tail prediction errors as a source of misalignment and grasp failures.
  • Method: PolicyTrim is a two-stage RL-based post-training framework for reliable chunk extension and redundant step reduction without architectural changes or extra demonstrations.Its execution-window diversification performs a progressive reliability sweep over tail predictions at different chunk positions.
  • Motivation: Policy efficiency is distinct from computational efficiency and is governed by executable action-chunk length and total physical execution steps.These factors jointly determine the number of forward inference calls during task execution.

2 Related Work

VLA models unify perception, language, and action for robotic manipulation, with architectures including autoregressive and diffusion-based policies that commonly use action chunking. Existing efficiency research primarily reduces per-inference computation, while policy efficiency and reliable chunk-horizon extension remain comparatively underexplored.

  • VLA architectures: VLA models unify visual perception, language understanding, and action generation, spanning autoregressive and diffusion-based architectures that widely adopt action chunking.Representative models include OpenVLA, RT-2, π0, π0.5, and GR00T.
  • Efficiency perspectives: Existing deployment-efficiency efforts focus almost exclusively on computational efficiency, leaving intrinsic policy efficiency largely unexplored.The distinction is between per-inference computational cost and the deployed model’s intrinsic policy efficiency.
  • Computational efficiency: Current methods reduce per-inference overhead through token pruning, action-token compression, decoding acceleration, KV-cache reuse, quantization, and lightweight model designs.These approaches generally treat the learned policy as fixed.
  • RL post-training: RL post-training has progressed from PPO toward GRPO, which removes the value model and makes scaling to large VLAs more practical.PPO’s actor-critic architecture introduces prohibitive memory overhead for large VLA backbones.
  • PolicyTrim: PolicyTrim addresses the gap with two RL stages that extend reliable action chunks and eliminate redundant physical steps to reduce forward inference calls.Its first stage rewards successful longer chunks, while the second combines step-saving rewards with group-anchored stability regularization.

3 Method

PolicyTrim is a two-stage reinforcement-learning post-training framework that separately extends reliable action chunks and reduces redundant physical steps. It progressively probes trustworthy horizons and rewards concise, reproducible successful executions while updating the policy with group-relative comparisons.

  • Framework overview: PolicyTrim decouples policy-efficiency optimization into sequential stages for reliable action-chunk extension and redundancy-aware physical-step reduction.The method first widens the trustworthy prediction horizon, then constrains total execution steps.
  • Reliable chunk extension: The chunk-extension stage dynamically assigns varied execution windows across sampled trajectories and rewards successful trajectories for sustaining longer accurate predictions.This progressive reliability sweep avoids forcing maximum-length execution immediately and targets degradation near chunk tails.
  • Reliable chunk extension: Horizon rewards activate only when a sampled group contains at least one successful trajectory, preventing unreliable long-chunk behavior when all trajectories fail.Without this condition, constant task-completion signals can make the objective horizon-biased.
  • Policy optimization: Both stages optimize group-normalized advantages with GRPO and update the policy using a clipped surrogate objective plus KL regularization against the reference policy.The KL penalty is computed per token across generated action chunks to preserve pretrained manipulation priors.
  • Redundancy-aware step reduction: The step-reduction stage gives successful shorter trajectories higher rewards while applying group-anchored penalties to discourage fragile, non-reproducible shortcuts.Failed rollouts receive zero reward, so step-saving incentives cannot reinforce unsuccessful behaviors.

4 Experiment

PolicyTrim is evaluated across diverse simulation benchmarks, VLA architectures, and real-world manipulation tasks, consistently improving policy efficiency while preserving task success. Ablations show that reliable chunk extension and redundancy-aware step reduction are complementary, and PolicyTrim can further combine with compute-centric acceleration.

  • Evaluation Setup: PolicyTrim is evaluated on LIBERO, ManiSkill, Meta-World, and a physical robot platform using average success rate, physical steps, chunk length, speedup, and wall-clock time.The evaluation spans four LIBERO subsets, physics-rich ManiSkill tasks, Meta-World tasks, and real-world tabletop manipulation.
  • Benchmark Results: Across LIBERO models and subsets, PolicyTrim reduces physical steps while maintaining comparable success rates, with π0.5 on Object reaching 51.4% of baseline steps.Diffusion-based π0.5 and GR00T support reliable chunk extension, whereas OpenVLA-OFT’s parallel decoding is highly sensitive to longer horizons.
  • Benchmark Results: Up to 2.52× Meta-World and 2.36× ManiSkill end-to-end speedups with π0.5 demonstrate generalization across simulators, action spaces, and architectures.PolicyTrim improves both success rates and step efficiency across these benchmarks.
  • Architectural Generality: The two-stage pipeline achieves 2.97× speedup on re-pretrained OpenVLA-OFT with h = 16 while preserving task success.The original h = 8 parallel decoder leaves no room for Stage 1 extension, motivating the larger-capacity backbone.
  • Real-World Deployment: 1.86× average real-world speedup is achieved by π0.5 across FlipMug, HangMug, and TapeBox while maintaining or improving success under standard and dynamically perturbed targets.Qualitatively, PolicyTrim produces smoother, more direct trajectories than the baseline’s redundant corrective motions and target-area jitter.
  • Ablation Studies: Chunk Extension alone increases hchunk from 5 to 15 and yields 2.86× speedup but raises physical steps from 108.3 to 113.8, requiring explicit step reduction.The Step-Saving Reward alone cuts Stotal by 24.6% from 108.3 to 81.7 but lowers success from 97.8% to 93.7%; Group-Anchored Regularization restores success to 97.5% and reduces Stotal to 61.6.

5 Conclusion

PolicyTrim frames policy efficiency as an overlooked deployment bottleneck and addresses it with a two-stage reinforcement-learning post-training framework. It reduces inference frequency while preserving strong task competence and complements compute-centric acceleration methods.

  • Framework: PolicyTrim is a two-stage RL post-training framework that improves policy efficiency without architectural modifications or additional demonstrations.Its first stage uses dynamic horizon exploration to push the trustworthy prediction frontier toward its empirical limit.
  • Framework: Dynamic horizon exploration progressively extends the trustworthy prediction frontier toward its empirical limit.
  • Framework: A redundancy-aware reward forms the second stage of PolicyTrim’s execution-efficiency framework.The supplied passage indicates that this reward is coupled with group-anchored stabilization, but the sentence is truncated.
  • Deployment implications: PolicyTrim significantly reduces inference frequency while maintaining strong task competence.
  • Deployment implications: PolicyTrim complements compute-centric acceleration and can be combined with existing inference optimization techniques for compounded efficiency gains.The passage presents this combination as a path toward more holistic, practical, and scalable robotic deployments.

A PolicyTrim Training Algorithm · A.1 PolicyTrim Training Algorithm

PolicyTrim is a two-stage GRPO-based post-training procedure designed to improve intrinsic policy efficiency. It first extends reliable action chunks, then reduces redundant physical steps using group-relative, KL-regularized optimization.

  • A.1 PolicyTrim Training Algorithm: PolicyTrim Post-Training is organized as two sequential optimization stages.The procedure is explicitly summarized as a two-stage optimization process.
  • A.1 PolicyTrim Training Algorithm: The algorithm initializes policy πθ, reference policy πref, task distribution T, acceptance ratios Γ, maximum chunk length H, group size G, and step budget Sbase.It also specifies reward coefficients and constants for optimization.
  • A.1 PolicyTrim Training Algorithm: Stage 1 samples acceptance ratios γi ∼ Γ, converts them into chunk horizons hi = ⌊γiH⌋, and rolls out trajectories for group-based evaluation.The stage targets reliable action chunk extension.
  • A.1 PolicyTrim Training Algorithm: After Stage 1, the current policy becomes the reference policy before Stage 2 begins with the maximum sampled horizon h = ⌊max(Γ)H⌋.Stage 2 rolls out trajectories using this fixed window and records task success and step counts.
  • A.1 PolicyTrim Training Algorithm: Stage 2 computes group statistics over successful step counts and uses them to form the redundancy-aware reward for step reduction.Successful trajectories provide the step-count subset used for µgroup and σgroup.
  • A.1 PolicyTrim Training Algorithm: Across stages, rewards are rescaled to the range of rsucc, while Stage 2 renormalizes the aggregated reward to preserve that range.This keeps reward magnitudes aligned with the success reward.
  • A.1 PolicyTrim Training Algorithm: Rewards are standardized within each sampled group to compute group-relative advantages for policy optimization.The grouped normalization is applied after the stage-specific reward construction.
  • A.1 PolicyTrim Training Algorithm: Policy updates use a clipped KL-regularized GRPO objective.The objective follows reward standardization and group-relative advantage computation.

B Implementation Details … C Additional Results

PolicyTrim was implemented with critic-free GRPO and a two-stage training design for chunk extension and step reduction. Experiments used backbone-specific configurations across Pi0.5, OpenVLA-OFT, and GR00T, with the supplied details covering training and backbone settings.

  • B Implementation Details: Critic-free GRPO used group-relative reward normalization and direct policy updates from rollout returns, without a critic or separate reward model.Training used a GRPO group size of 8 on a single node unless otherwise noted.
  • B Implementation Details: PolicyTrim sequentially trains Stage 1 for reliable action-chunk extension and Stage 2 for redundant-step reduction.Stage 2 uses a step-saving reward and group-anchored stability regularization; default coefficients are 0.8 and 0.2, respectively.
  • B Implementation Details: Pi0.5 and GR00T used both stages, whereas OpenVLA-OFT used only Stage 2 because of backbone constraints.Each stage was trained for up to 500 epochs.
  • B.1 Training Hyperparameters: Pi0.5 used global batch sizes of 2048 on LIBERO and 5120 on ManiSkill, with prediction horizons of 15 and 10, respectively.On LIBERO, environment horizons were 160 for Spatial and Object, 240 for Goal, and 400 for LIBERO-10; action horizons varied by task.
  • B.2 Backbone-Specific Settings: Pi0.5 used the OpenPI backbone with FSDP in no-shard mode, without gradient checkpointing, and with three denoising steps.Its action head used Flow-SDE and the learning rate was 5 × 10−6.
  • B.2 Backbone-Specific Settings: OpenVLA-OFT used bfloat16 full-parameter fine-tuning, enabled gradient checkpointing in checked-in GRPO configurations, and used a learning rate of 2 × 10−5.The supplied backbone-specific passage does not provide additional complete settings for OpenVLA-OFT.
  • B.2 Backbone-Specific Settings: GR00T used bfloat16 precision and four denoising steps.The supplied passage truncates before completing GR00T’s remaining backbone-specific settings.

C.1 Qualitative Results

PolicyTrim qualitatively improves intrinsic policy efficiency across GR00T and OpenVLA-OFT by producing more compact, reliable trajectories with fewer redundant physical steps. The examples support simultaneous enlargement of the reliable action horizon and reduction of unnecessary motion.

  • Cross-backbone qualitative comparison: Across GR00T and OpenVLA-OFT, PolicyTrim reaches target states with more compact execution trajectories and fewer redundant physical steps.Figure 5 compares baseline and PolicyTrim execution snapshots, with green boxes marking salient late-stage completion frames.
  • GR00T: On GR00T, PolicyTrim preserves reliable long-horizon behavior while reducing unnecessary motion before successful placement.
  • OpenVLA-OFT: On OpenVLA-OFT, PolicyTrim produces shorter, cleaner trajectories, particularly in tasks involving hesitation or redundant adjustments.

C.2 Ablation on Group Size

PolicyTrim is largely insensitive to group size: varying G from 8 to 16 causes only minor, non-systematic changes across all four LIBERO subsets. The results support G = 8 as sufficient in practice.

  • C.2 Ablation on Group Size: Varying G from 8 to 16 produces only minor fluctuations in success rate, total execution steps, and end-to-end speedup across all four LIBERO subsets.No consistent trend emerges across the tested group sizes.
  • C.2 Ablation on Group Size: G = 8 is already sufficient in practice for PolicyTrim.This is supported by the observed insensitivity to group size.

C.3 Ablation on Components · C.4 Real-World Deployment

Component ablations show that combining reliable chunk extension, step-saving rewards, and group-anchored regularization provides the strongest balance between execution efficiency and task performance. PolicyTrim is also evaluated on a real robot across three tabletop manipulation tasks, including FlipMug visualization.

  • C.3 Ablation on Components: Reliable Action Chunk Extension mainly improves execution speed by enlarging the action horizon.It can slightly increase physical steps and cause a small success-rate drop.
  • C.3 Ablation on Components: Step-Saving Reward alone substantially reduces Stotal but degrades task success.Shorter trajectories alone are therefore not necessarily reliable.
  • C.3 Ablation on Components: The component ablation study uses the π0.5 model across all four LIBERO subsets.The results show a consistent pattern across Spatial, Object, Goal, and Long.
  • C.3 Ablation on Components: Adding Group-Anchored Regularization largely recovers task competence while further improving execution efficiency.The component stabilizes concise behaviors.
  • C.3 Ablation on Components: Enabling all three components together achieves the best overall trade-off across the four LIBERO subsets.The combined configuration consistently provides the highest end-to-end speedup while preserving strong task performance across Spatial, Object, Goal, and Long.
  • C.4 Real-World Deployment: PolicyTrim is deployed on a real robot platform for FlipMug, HangMug, and TapeBox tabletop manipulation tasks.The deployment evaluation covers three real-world tasks.
  • C.4 Real-World Deployment: Figure 6 visualizes real-world execution on the FlipMug task.Additional real-world visualizations are available on the project GitHub page.

C.5 Robustness under Visual Perturbations … C.8 Hyperparameter Sensitivity

Across visual perturbations, training cost, horizon controls, and hyperparameter settings, PolicyTrim preserves robust and efficient execution. Its gains arise from reliable RL-based chunk extension and step reduction rather than simply forcing longer horizons, while default settings remain stable without per-task tuning.

  • C.5 Robustness under Visual Perturbations: PolicyTrim remains robust under Gaussian blur and 50% random occlusion on LIBERO-Spatial.The perturbations use blur kernel size k = 13 and 50% random occlusion to test degraded or partially missing visual observations.
  • C.5 Robustness under Visual Perturbations: PolicyTrim outperforms the baseline under both visual perturbations while reducing physical execution steps.It improves success rate and learns a more robust, concise execution strategy rather than overfitting to clean observations.
  • C.6 Post-Training Cost: 68 hours on 8×H100 GPUs is the approximate post-training cost, versus 78 hours for the corresponding RLinf setting.The reduced cost is attributed mainly to the extended reliable action-chunk horizon lowering rollout inference calls.
  • C.6 Post-Training Cost: PolicyTrim’s one-time post-training cost is justified by deployment-time speedups and cross-task reuse of the learned efficiency improvement.The policy-efficiency improvement generalizes across tasks after post-training.
  • C.7 Horizon-Sweep Baseline: 97.8% to 93.1%: naively increasing π0.5’s fixed execution horizon lowers success rate.The horizon-sweep experiment tests whether PolicyTrim’s gains come merely from executing longer action chunks.
  • C.7 Horizon-Sweep Baseline: Stage 1 maintains high success under an extended horizon, while full PolicyTrim reduces physical steps from 111.8 to 59.8.These results support RL-based reliable chunk extension and Stage 2 step reduction rather than forced long-horizon execution.
  • C.8 Hyperparameter Sensitivity: PolicyTrim is evaluated for sensitivity to α, G, and λstab on LIBERO-Spatial using π0.5.The tested hyperparameters are the step-budget multiplier, GRPO group size, and stability regularization coefficient.
  • C.8 Hyperparameter Sensitivity: Across tested settings, success rate varies narrowly and physical step count remains stable, so defaults require no careful per-task tuning.Table 12 reports SR / Step, with default values shown in bold.

D Failure Case Analysis

Failure cases show that overly aggressive step-saving can sacrifice execution safety: without group-anchored stability regularization, the policy takes a shorter unsafe trajectory, collides with the bowl, and fails the task.

  • Failure Case Analysis: Representative failures include incomplete task execution and cases requiring additional corrections during execution.These examples provide qualitative evidence of limitations in task completion and execution reliability.
  • Failure Case Analysis: Without group-anchored stability regularization, the policy overemphasizes step-saving and execution speed, approaching the bowl with insufficient clearance and causing collision-related task failure.The robot follows a shorter trajectory without lifting the end-effector high enough, knocking the bowl out of the reachable workspace.
Loading 2606.22540v3…