PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Plans, Faithful Actions: Closing the Planning-Execution Gap in Hierarchical Vision-Language-Action Models

Chuanliang Xie, Boyu Ma, Gen Li, Yizhou Liu, Houwang Chen, Xinyu Zhou, Jianfei Yang

arXiv:2609.30833v1cs.RO

TL;DR

Hierarchical VLAs need plans that arrive in time and influence action, but a π0.5-derived waypoint baseline falls short on both. The paper introduces waypoint-aligned Block-AR and normalized goal modulation (NGM); NGM improves Block-AR by 2.6 points on average across four LIBERO suites.

  • Problem

    Hierarchical VLA interfaces must generate plans quickly enough for real-time control and ensure those plans genuinely contribute to behavior.

    Source: §1 · p. 1

  • Method

    The paper aligns autoregressive decoding blocks with complete waypoints and uses phase-gated, anti-shortcut normalized goal modulation to inject goals through the action expert.

    Source: §1 · pp. 2–3

  • Results

    2.6 points on average: NGM improves Block-AR across all four LIBERO suites, while endpoint sensitivity rises from 0.6% to 42.6%.

    Source: §7 · p. 11

  • Takeaways & Limitations

    End-to-end task success alone establishes neither planning efficiency nor effective use of the planner–executor interface.

    Source: §7 · p. 11

    Transfer beyond a π0.5-derived hierarchy is untested, and each model variant was trained only once, preventing estimation of training-run variability.

    Source: §7 · p. 11

Abstract

from arXiv · show

Hierarchical vision-language-action (VLA) systems consist of a high-level vision-language planner and a low-level action expert that generates continuous actions. This hierarchical design has practical value only if the planner can generate plans fast enough to meet real-time control requirements, and the resulting plans actually contribute to the generation of action. We study one such system, a waypoint hierarchy pipeline adapted from $\pi_{0.5}$, and find that neither requirement is satisfied. This baseline relies on token-level autoregressive decoding (Token-AR) to generate a waypoint plan, requiring 57 very expensive vision-language model (VLM) forward passes. However, we find that erasing the waypoint endpoints has little effect on task success. Two findings reveal the misalignment of planner-executor: the planner generates outputs at an excessively fine granularity, and the executor underuses plans as a control condition. We address the latency issue with waypoint-aligned block-autoregressive decoding (Block-AR), and plan underuse issue with normalized goal modulation (NGM), a layer-wise goal path constrained by phase gating and anti-shortcut training so that the waypoint influences action generation maintaining other signals. Our method reduces the maximum number of VLM forward passes from 57 to 8 on LIBERO, including one prefix prefill, and achieves an $8.7\times$ reduction in planning latency on a Rokae dual-arm robot. With normalized goal modulation and anti-shortcut training, Block-AR's success rate on LIBERO-Long increases from 91.0% to 96.2%, while its average success rate across the four suites increases from 95.85% to 98.45%. On three bimanual tasks with this robot, success rates remain comparable across methods.

1 Introduction

Hierarchical VLAs require plans that are generated quickly and meaningfully shape actions, yet the studied waypoint baseline falls short on both. The paper aligns planner decoding to waypoint blocks and introduces normalized goal modulation (NGM) to strengthen waypoint influence on action generation.

1 Introduction

  • 57 to 8 VLM passes and 1094 ms to 125 ms per plan; on the robot, motion time rises from 29% to 66%.Block-AR decodes a complete waypoint per VLM forward pass; the figure shows these results for a seven-waypoint LIBERO plan and a Rokae dual-arm robot.Source: Figure 1 · p. 2, §1 · p. 3
  • The studied π0.5-derived hierarchy uses PaliGemma to predict waypoints containing target configuration, gripper state, and execution duration, which the flow-matching expert receives as suffix tokens.The authors built this baseline for controlled experiments; it is not the original π0.5 architecture.Source: §1 · p. 2
  • With suffix-token conditioning, endpoint erasure changes actions by only 0.6% of the change from resampling flow noise and has little effect on task success.The intervention holds images, language, and flow-matching noise fixed; the expert does respond measurably to execution duration.Source: §1 · pp. 2–3
  • Block-AR generates complete waypoints per VLM pass while preserving causal dependencies between waypoints, aligning planner output granularity with the executor’s waypoint-by-waypoint use.The motivating interface requirements are timely plan generation and genuine contribution of the plan to behavior.Source: §1 · pp. 1–2
  • NGM adds a layer-wise goal path constrained by phase gating and anti-shortcut training, improving Block-AR on all four LIBERO suites while retaining responses to other inputs.The method is designed to make waypoint information contribute to action generation without erasing the effects of other signals.Source: §1 · p. 3

2 Related Work

Related work covers sparse waypoint and keypose interfaces, faster VLA decoding, and policies that underuse relevant conditions. This paper situates its approach among those lines while emphasizing waypoint-aligned blocks and controlled assessment of planner influence.

2.1 Waypoint and keypose interfaces

  • Waypoint interfaces range from offline AWE extraction to VLM-generated sketches, pixel-depth sequences, and 3-D or image-space waypoints executed by different mechanisms.AWE selects a minimal waypoint set under reconstruction-error, gripper-transition, and fixed-horizon constraints; the paper uses it to extract waypoints offline.Source: §2.1 · p. 3

2.2 Decoding efficiency in VLAs

  • Block decoding is established, but this work aligns each block with a semantic waypoint and jointly decodes its configuration, gripper state, duration, and termination flag.The paper contrasts this with other efficiency approaches, including action compression, parallel or consistency decoding, asynchronous reasoning, and joint discrete/continuous training.Source: §2.2 · p. 3

2.3 Conditions ignored by the policy

  • Prior work describes shortcut reliance and weak condition use; here, waypoint changes are tested while other inputs are held fixed to assess the planner–executor interface.The cited examples include causal confusion and copycat behavior in behavior cloning, weak language conditioning, and guidance methods for condition adherence.Source: §2.3 · pp. 3–4

3 Problem Formulation

The planner produces a bounded sequence of waypoint blocks, and the executor turns each waypoint into a segment of continuous actions. The formulation also defines horizon compatibility and measures whether waypoint changes affect execution and task success.

Plan representation

  • The executable plan is the sequence of K waypoint blocks decoded before an end code or the decoding limit, with K ≤ M.Each waypoint contains a discretized target configuration, gripper command, and action duration.Source: §3 · p. 4

Execution

  • For each waypoint, the executor uses the current state and waypoint to generate the requested number of continuous actions.The next block starts from the measured state; the action expert denoises D = 32 actions and executes the first d_i.Source: §3 · p. 4, Figure 2 · p. 5

Horizon compatibility

  • Horizon compatibility means each requested action duration fits the action expert’s output horizon, but does not guarantee collision-free or dynamically feasible motion.The paper states that Section 4.1 guarantees this property by construction.Source: §3 · p. 4

Faithfulness

  • Faithfulness is assessed by comparing action sensitivity to waypoint interventions with task-success change after endpoint erasure, alongside intact-plan success and visual-input sensitivity.The endpoint is erased by setting the target configuration to the current state; neither measure alone proves effective conditioning.Source: §3 · pp. 4–5
  • NGM adds a normalized-goal residual to every layer’s AdaRMS condition, gated on during later denoising phases.The gate is open for t ≥ 0.5 and closed for t ≤ 0.2; training also noises or nulls the endpoint inputs.Source: §3 · p. 5

4 Method

The method adapts π0.5 into a waypoint hierarchy, with a VLM planner emitting plans and a flow-matching action expert generating continuous actions. It uses waypoint-aligned block decoding and a constrained layer-wise goal route.

4.1 Waypoint representation and execution

  • Waypoints are extracted with gripper-transition keyframes and a maximum segment length, then a flow-matching expert generates actions between targets using the segment duration.On the real robot, replanning starts after the first segment; in simulation, a new plan is requested after the current plan is fully executed.Source: §4.1 · pp. 5–6

4.2 Waypoint-aligned block-autoregressive planning

  • Block-AR predicts one complete waypoint per forward pass, reducing maximum planning passes from 57 to 8 on LIBERO and from 120 to 8 on the robot.Attention is causal across blocks and bidirectional within each block; decoding ends when a block predicts d = 0.Source: §4.2 · p. 7

4.3 Normalized goal modulation

  • NGM scales the relative waypoint displacement, adds a zero-initialized residual to every expert layer, and uses phase gating, endpoint noise, and dropout to discourage the displacement-integrator shortcut.During training, endpoint noise is σc = 0.7 and the goal is replaced with a learnable null embedding with probability 0.15.Source: §4.3 · pp. 7–8

4.4 Training

  • Training adapts the backbone, vision encoder, and action expert with LoRA, retaining the action-expert gradient path into the backbone rather than using knowledge insulation.The three waypoint variants share data, initialization, and training steps; the retained gradient path has weight λ = 1.7.Source: §4.4 · p. 8

5 Experimental Setup

The experiments evaluate the shared waypoint hierarchy on four LIBERO suites and on three bimanual tasks with a dual-arm robot. The setup also measures waypoint dependence and visual responsiveness under controlled input changes.

Experimental setup

  • LIBERO evaluation covers four suites, with 500 episodes per suite across 10 tasks and 50 official initial states per task.A single model is trained on the union of the suites, using 1,693 episodes and suite-specific step budgets.Source: §5 · p. 8
  • Waypoint dependence is measured by action change after endpoint erasure or a duration increase, relative to change from resampling flow noise; visual responsiveness is measured under a main-camera swap.The compared interventions hold the other inputs and flow noise fixed and use the same 64 frozen segments.Source: §5 · p. 9, Table 4 · p. 9
  • The robot study uses three bimanual tasks and 20 trials per task and method, with success judged by an operator.The 120 training demonstrations are drawn from 154 collected episodes after validation and test holdouts; the representative shelf-placement sequence stops before placement.Source: §5 · p. 9, Figure 5 · p. 9

6 Results

Block-AR sharply reduces planning latency while keeping LIBERO success near Token-AR, and NGM improves Block-AR results while increasing the executor’s use of waypoints. Real-robot results also report latency, motion duty cycle, and success across tasks.

6.1 End-to-end performance

  • NGM improves Block-AR on every LIBERO suite by 2.6 points on average and 5.2 points on LIBERO-Long; Block-AR alone changes average success by −0.85 points versus Token-AR.The number of VLM planner forward passes per plan drops from 57 to 8 with Block-AR.Source: §6.1 · p. 9

6.4 Real robot

  • Table 5 reports real-robot results, with operator-judged success over 20 trials per task and method and latency and duty-cycle averages from telemetry.The All row averages over requests and episodes, not over the three task rows.Source: Table 5 · p. 10, §6.2 · p. 10

6.2 Fast: the cost of a plan

  • Planning time falls from 1094 ms to 125 ms (8.7×), and motion duty cycle rises from 28.7% to 65.6% with Block-AR.Round-trip latency is 293 ms; the remaining latency comes mainly from the action expert at 160 ms per segment.Source: §6.2 · pp. 9–10

6.3 Faithful: does the executor follow the plan?

  • With suffix-token conditioning, erasing planned endpoints changes actions by only 0.6% of the noise-resampling change and causes no drop in closed-loop success.Camera swaps change actions by 281% of the noise baseline, while language swaps change them by 158%.Source: §6.3 · p. 10
  • Naive goal injection raises endpoint sensitivity to 348% but drops task success to 15%, with failures reported as timeouts.The model’s response to visual changes falls to 27% of the suffix-token-conditioned baseline, alongside the displacement-integrator shortcut.Source: §6.3 · p. 10
  • NGM raises LIBERO-Long success from 91.0% to 96.2%; endpoint sensitivity rises from 0.6% to 42.6%, while image-change sensitivity remains 82%.Erasing the endpoint under NGM reduces success by 7.4 points.Source: §6.3 · p. 10

7 Conclusion

The conclusion identifies high planning latency and weak use of planned endpoints as two forms of planner–executor misalignment. It reports that Block-AR reduces planner passes while keeping success close to Token-AR, and NGM improves results and endpoint dependence.

7 Conclusion

  • Block-AR keeps LIBERO success within 1.4 points of Token-AR on every suite while reducing planner passes from 57 to 8; NGM improves all four suites by 2.6 points on average.With NGM, endpoint sensitivity rises from 0.6% to 42.6%, and endpoint erasure reduces success by 7.4 points.Source: §7 · p. 11
  • Both methods are validated on one π0.5-derived hierarchy; Block-AR assumes fixed-format plan token groups, NGM relies on AdaRMS conditioning, and transfer to other VLA backbones is untested.Each model variant was trained only once, so run-to-run variability cannot be estimated.Source: §7 · p. 11
  • The real-robot study covers three tasks with 20 trials per task and method, limiting precision of success-rate estimates and conclusions about small between-method differences.Source: §7 · p. 11

Related papers

Loading 2609.30833v1…