PaperlayerView exact revision on arXiv

Source-linked AI summary

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

Ziyi Yin, Sangmin Woo, Kang Zhou, Sungyeon Kim, Aosong Feng, Haibo Ding, Jun Huan

arXiv:2609.36352v1cs.CVcs.LGcs.RO

TL;DR

Long-horizon VLA tasks require dependent manipulations from one instruction, but terminal-only online RL rewards are sparse and do not distinguish partial progress. StructRL rewards verified subtasks according to prerequisites and completion pace, and consistently outperforms evaluated online RL baselines across RoboCasa365 and LIBERO-Long with both reported backbones.

  • Problem

    Terminal-only online RL rewards are sparse on long-horizon tasks and treat rollouts with substantial partial progress like those failing near the beginning.

    Source: §1 · p. 2

  • Method

    StructRL decomposes commands into verifiable subtasks, gates intermediate rewards on prerequisites, and scales eligible rewards by completion pace.

    Source: §1 · p. 2

  • Results

    Across RoboCasa365 and LIBERO-Long, StructRL improves GR00T-N1.5 and π0.5 over evaluated online RL baselines, including gains of 7.6 and 4.2 percentage points on GR00T-N1.5, respectively.

    Source: §4.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    Verifiable subtask rewards organized by task structure and calibrated to demonstration pace can improve long-horizon VLA post-training.

    Source: §6 · p. 9

    Policies sometimes collect intermediate rewards and then idle until timeout; separate safety validation is required before physical-robot deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.

1 INTRODUCTION

VLA research has emphasized single-skill manipulation, while long-horizon tasks require dependent actions from one instruction and remain difficult for policies trained on demonstrations. StructRL addresses sparse terminal-only RL rewards with prerequisite-aware, pace-scaled subtask rewards and outperforms evaluated online RL baselines across the reported benchmarks and backbones.

1 INTRODUCTION

  • Long-horizon tasks require composing dependent skills from one instruction, while demonstration-trained policies can enter poorly covered states after small rollout errors.Existing VLA evaluation primarily emphasizes single self-contained skills.Source: §1 · pp. 1–2
  • Terminal-only online RL rewards become sparse as task horizons grow and give equal return to near-complete and early-failure rollouts.The paper identifies a need for dense supervision that recognizes meaningful progress before final success.Source: §1 · p. 2
  • StructRL decomposes instructions into verifiable subtasks and grants intermediate credit only when prerequisite subtasks are complete.A locally detected completion such as closing a box is not credited before required objects have been placed inside.Source: §1 · p. 2
  • Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and π0.5, StructRL consistently outperforms evaluated online RL baselines.On RoboCasa365 with GR00T-N1.5, success rate is 49.1%, versus 41.5% for the strongest evaluated online RL baseline and 38.6% for SFT.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe VLA interaction and PPO optimization at the action-chunk level. They contrast chunk-level training with terminal-only reward assignment and note a likelihood-estimation adjustment for flow-matching VLAs.

2 PRELIMINARIES

  • VLA interaction is modeled as a sequence of action chunks, with the environment returning the next observation after executing each complete chunk.A rollout contains M observation-action-chunk pairs and is used for RL training.Source: §2 · pp. 2–3
  • PPO optimizes each action chunk using a clipped surrogate objective and chunk-level advantages computed from rewards by a learned critic.Under terminal-only rewards, intermediate chunks receive zero reward and the final chunk is rewarded only upon task success.Source: §2 · p. 3
  • For flow-matching VLAs, Gaussian noise converts deterministic ODE sampling into an SDE so PPO can evaluate the chunk likelihood ratio.Figure 2 previews StructRL's subtask decomposition and reward construction pipeline.Source: §2 · p. 3, Figure 2 · p. 3

3 METHODOLOGY

StructRL turns a long-horizon command into verifiable subtasks with dependencies, then converts detected completions into chunk-level rewards. It uses prerequisite gating and demonstration-calibrated pacing, and the structured rewards can also be used with GRPO.

3 METHODOLOGY

  • StructRL uses an LLM to organize verifiable, goal-aligned subtasks into dependency groups, then constructs chunk-level rewards and optimizes the VLA with PPO.The methodology has a decomposition stage followed by reward construction.Source: §3 · p. 3

3.1 SUBTASK DECOMPOSITION

  • Candidate subtasks must have binary environment-state completion criteria and represent progress toward the final goal.The LLM retains candidates that pass both a verifiability check and a progress check; Approach Box and Open Gripper are examples that fail one check each.Source: §3.1 · pp. 3–4
  • Prerequisite sets encode which earlier subtasks must be complete, while subtasks within a dependency group may occur in any order.In the Pack Lunch example, adding chicken and apple may occur in either order, but closing the box follows both.Source: §3.1 · p. 4

3.2 STRUCTURED REWARD CONSTRUCTION

  • Chunk-level reward construction makes two decisions: whether a detected completion is eligible for credit and the magnitude of that credit.Eligibility follows task dependencies, and magnitude follows completion pace.Source: §3.2 · p. 4

3.2.1 STRUCTURE-AWARE REWARD GATING

  • A subtask earns intermediate reward only when detected after all its prerequisites and before it has already been rewarded.The complete task also contributes a terminal reward upon first success.Source: §3.2.1 · p. 4

3.2.2 DYNAMIC REWARD PACING

  • Eligible subtask rewards are scaled by completion pace, with elapsed chunks measured from the previous rewarded completion and demonstration intervals setting the reference pace.The scale parameter β upper-bounds the reward for one subtask.Source: §3.2.2 · pp. 4–5

3.2.3 EXTENSION TO GRPO

  • The structured rewards can be used with GRPO, which normalizes summed chunk-level returns within each rollout group to compute group-relative advantages.The described setup collects groups of eight rollouts per task instruction and fixed initial state.Source: §3.2.3 · p. 5

4 EXPERIMENTS

Across RoboCasa365 and LIBERO-Long, STRUCTRL outperforms evaluated online RL baselines across both evaluated backbones. Experiments also compare reward sources and components, test decomposition density and optimizer compatibility, and assess transfer and shorter-horizon tasks.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers two manipulation benchmarks, SFT and online RL baselines, and matched interaction budgets; task decompositions are generated before training and reused during rollouts.RoboCasa365 Composite-Seen has 16 tasks across 10 held-out scenarios, while LIBERO-Long has 10 tasks evaluated on 50 official initial scenarios each.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • STRUCTRL improves overall SR over the strongest online RL baseline by 7.6 points on RoboCasa365 and 4.2 points on LIBERO-Long with GR00T-N1.5, and exceeds that baseline in every horizon bucket.For π0.5, gains are 3.9 points on RoboCasa365 and 2.2 points on LIBERO-Long; the largest GR00T-N1.5 bucket gains are 7.0 points on RoboCasa365 and 6.6 points on LIBERO-Long.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 REWARD SOURCE: STRUCTURED EVENTS vs. LEARNED PROGRESS MODEL

  • STRUCTRL outperforms Robometer by 2.4 points with GR00T-N1.5 and 3.2 points with π0.5 under matched PPO training on LIBERO-Long.The comparison contrasts learned visual progress estimates with simulator-verified, dependency-aware completion rewards; STRUCTRL does not require reward-model inference during rollouts.Source: Table 2 · p. 6, §4.3 · pp. 6–7

4.4 REWARD-COMPONENT ABLATION

  • Adding verified subtask rewards raises SR from 41.3% to 47.4% on RoboCasa365 and from 91.2% to 94.7% on LIBERO-Long; pacing and gating add further gains.The full reward reaches 49.2% and 96.4%, respectively, and increases SR by 10.3 points on the longest RoboCasa365 tasks, from 26.2% to 36.5%.Source: Figure 3 · p. 7, §4.4 · p. 7

4.5 EFFECT OF SUBTASK DECOMPOSITION DENSITY

  • On RoboCasa365, SR rises from 40.2% at N̄ = 0 to 50.4% at N̄ = 5, then falls to 37.6% at N̄ = 52.The controlled sweep permits fine motion checkpoints normally excluded by the filtering criteria; analyses attribute the decline to less task-aligned signals and increasing total intermediate reward.Source: Figure 4 · p. 7, §4.5 · pp. 7–8

4.6 COMPATIBILITY WITH GRPO

  • On RoboCasa365, STRUCTRL-GRPO exceeds SimpleVLA-RL by 3.4 points with GR00T-N1.5 and 1.3 points with π0.5, while PPO remains stronger overall.The PPO advantage over GRPO is 4.2 points with GR00T-N1.5 and 2.6 points with π0.5.Source: Table 3 · p. 8, §4.6 · p. 8

4.7 ZERO-SHOT EVALUATION ON ADDITIONAL ROBOCASA365 TASKS

  • Zero-shot transfer to 16 Composite-Unseen tasks is limited, with 4.8% SR versus 4.3% for the strongest comparison; on Atomic-Seen, STRUCTRL reaches 20.5%.The 65 Atomic-Seen tasks were encountered during pretraining but excluded from composite-task RL; STRUCTRL exceeds the SFT policy's 17.0% SR and SimpleVLA-RL's 16.9%.Source: Table 4 · p. 8, §4.7 · p. 8

4.8 APPLICABILITY TO SHORTER-HORIZON TASKS

  • On shorter-horizon LIBERO suites, STRUCTRL improves over SimpleVLA-RL from 91.2% to 92.1% on Spatial, 98.5% to 99.4% on Object, and 94.8% to 95.6% on Goal.The same decomposition and optimization procedures are used without short-task-specific modifications; gains are modest alongside high baseline success rates.Source: §4.8 · pp. 8–9, Table 5 · p. 9

5 RELATED WORK

Related work spans long-horizon VLA benchmarks and offline or online RL post-training. Existing online methods often use sparse terminal rewards, while learned progress estimators can require substantial rollout-time or task-specific resources.

Long-horizon VLA tasks

  • Long-horizon benchmarks compose multiple skills and test manipulation, instruction following, navigation, and extended execution beyond individual pick-and-place tasks.SFT policies remain sensitive to demonstration coverage and compounding execution errors.Source: §5 · p. 9

Reinforcement learning for VLAs

  • Many online VLA RL methods reward only complete success, while learned progress estimators may require a large rollout model or task-specific reward-model training.Terminal rewards become sparser as execution horizons grow.Source: §5 · p. 9
  • STRUCTRL derives intermediate rewards from simulator-verifiable subtasks, prerequisite dependencies, and demonstration-derived pacing without a learned reward model during rollout collection.The related work contrasts offline optimization on pre-collected rollouts with online methods that collect new interactions and optimize using PPO or GRPO.Source: §5 · p. 9

6 CONCLUSION

STRUCTRL replaces terminal-only supervision with structured rewards for long-horizon VLA tasks and improves both evaluated backbones across two benchmarks. The conclusion also notes reward-induced idling, the need for physical-robot safety validation, and resources for reproducing the experiments.

6 CONCLUSION

  • Across RoboCasa365 and LIBERO-Long, STRUCTRL improves GR00T-N1.5 and π0.5 over evaluated online RL baselines, with the largest gains on longer-horizon tasks for GR00T-N1.5.The paper concludes that verifiable subtask rewards organized by task structure and demonstration pace can improve long-horizon VLA post-training.Source: §6 · p. 9
  • Generative AI tools assisted with experimental infrastructure, log analysis, and manuscript drafting and revision; the authors independently verified reported results and claims.The authors take full responsibility for the paper.Source: AI USE STATEMENT · p. 10
  • A typical online VLA RL run uses 32 NVIDIA A100 GPUs for approximately 48 hours on RoboCasa365 or 24 hours on LIBERO-Long.These runs correspond to about 1,536 and 768 GPU-hours, respectively.Source: ETHICS STATEMENT · p. 10
  • Pacing discourages delaying future subtask completions but does not eliminate idling after intermediate rewards, and physical-robot deployment requires separate safety validation.The paper notes that other reward-induced failure modes may remain.Source: ETHICS STATEMENT · p. 10
  • The released materials include training and evaluation code, generated subtask decompositions, and reward implementation.Static subtask graphs support reproduction without querying an LLM or depending on a particular model version.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Evaluation uses 16 RoboCasa365 composite tasks with 100 episodes per task and 10 LIBERO-Long tasks with 50 episodes per task.Macro SR is the mean of per-task success rates.Source: REPRODUCIBILITY STATEMENT · p. 10

A TRAINING PROCEDURE

The training procedure collects policy rollouts, computes structured rewards, and updates the VLA with PPO. The appendix describes task-command decomposition and deterministic grounding of subtasks to simulator predicates.

A TRAINING PROCEDURE

  • STRUCTRL’s training loop collects rollouts, computes gated and paced subtask rewards, adds terminal completion credit, and updates the policy with PPO.The algorithm presents rollout collection, structured reward computation, and policy optimization as its three steps.Source: A TRAINING PROCEDURE · p. 14

B.1 LLM DECOMPOSITION PROMPT

  • The LLM receives the benchmark command and decomposition requirements only, producing ordered stages of atomic, physically checkable manipulation events.Subtasks within a stage may occur in any order; retreat steps and final “task finished” subtasks are excluded, and decoding is greedy for reproducibility.Source: §B.1 · pp. 14–15

B.2 GROUNDING SUBTASKS TO SIMULATOR PREDICATES

  • Subtasks are grounded once to fixed binary predicates; RoboCasa365 uses simulator-state predicates, while LIBERO uses goal conjuncts and adds grasp predicates for specified suites.Filtering removes unsupported, already-satisfied, unreliable, untimable, or duplicate predicates while preserving dependency order.Source: §B.2 · p. 15
  • Task object and fixture mentions bind to simulator handles, with repeated mentions assigned to corresponding task instances.RoboCasa365 predicate templates bind their arguments to task objects or fixtures.Source: §B.2 · p. 15

B.3 SENSITIVITY TO THE DECOMPOSER

The decomposer sensitivity study finds lower overall success with Qwen3.5-9B than with the default Claude Opus 4.8. The reported decompositions also compare coarser and finer subtask signal settings.

B.3 SENSITIVITY TO THE DECOMPOSER

  • Overall SR falls from 49.1% with Claude Opus 4.8 to 45.7% with Qwen3.5-9B in the decomposition-author ablation.The table reports SR (%) grouped by horizon bucket under identical training hyperparameters.Source: §B.3 · p. 15, Table 8 · p. 16
  • At N̄=5.0, decompositions add grasp milestones, intermediate fixture states, and timed activity checkpoints compared with N̄=2.375.The listed N̄ values are average subtask completion signals per task; retreat signals are excluded.Source: Table 7 · p. 16
  • The pipeline can be instantiated with Qwen3.5-9B, while the reported gap indicates that decomposition quality affects downstream RL performance.Claude Opus 4.8 is the default decomposer used for Table 1.Source: §B.3 · pp. 15–16, Table 8 · p. 16

C FURTHER ANALYSIS OF SUBTASK DECOMPOSITION GRANULARITY

The appendix finds that performance first improves with denser subtask decomposition and then declines when decomposition becomes excessive. Controlled analyses associate this decline with both reward timing and total intermediate reward, while demonstration-derived completion durations outperform uniform horizon allocation.

C.1 CONSTRUCTING THE DENSITY LADDER

  • Each density level refines existing milestones into finer child subtasks while retaining the previously introduced milestones.The density ladder relaxes the default verifiability and progress constraints to let the LLM decompose tasks more finely.Source: §C.1 · p. 17, Figure 5 · p. 17, Table 9 · p. 17

C.2 EFFECT OF DECOMPOSITION DENSITY ACROSS HORIZON BUCKETS

  • Success rate peaks at moderate decomposition density and declines at the densest setting, with larger proportional losses in the longer-horizon buckets.From the best density (N̄ = 5.0) to the densest (N̄ = 52.0), SR falls 11.5, 17.0, and 10.4 points for the 800–1000, 1000–1400, and 1400–2900 buckets, respectively.Source: §C.2 · pp. 17–18, Figure 6 · p. 18, Table 10 · p. 18

C.3 UNDERSTANDING PERFORMANCE DEGRADATION UNDER DENSE DECOMPOSITION

  • Shifting 78 of 80 completion signals earlier by an average of 3.1 action chunks lowers SR from 50.4%±1.0 to 47.4%±0.8, while capping total intermediate reward at N̄=14 raises SR from 44.1% to 47.7%.The two controls isolate timing and total reward as comparable contributors to degradation under dense decomposition.Source: §C.3 · p. 18

C.4 DOES REWARD PACING REQUIRE DEMONSTRATION DURATIONS?

  • Replacing demonstration-average completion durations with uniform H/N durations reduces SR from 49.2%±1.1 to 43.1%±0.4 over three evaluation runs.The authors leave more adaptive demonstration-free duration estimates for future work.Source: §C.4 · p. 19, Table 12 · p. 19

D.1 MULTI-SEED EVALUATION

  • Across three evaluation runs, STRUCTRL achieves 49.2% overall SR on GR00T-N1.5 and 44.8% on π0.5, exceeding the strongest baselines by 7.1 and 2.9 points, respectively.The paper reports these multi-seed gains as robust across evaluation runs.Source: §D.1 · p. 19

D.2 TRAINING DYNAMICS

Training curves show stable STRUCTRL performance that separates from baselines after roughly 50 steps and remains higher through training. The section also reports sensitivity to the subtask-reward weight and qualitative examples of progress through long-horizon tasks.

D.2 TRAINING DYNAMICS

  • STRUCTRL separates from baselines after roughly 50 training steps and remains consistently higher without performance collapse, with gains evident by around step 100.The authors report main experiments at step 100 given the additional computational cost of longer training.Source: §D.2 · pp. 19–20, Figure 7 · p. 20

D.3 SENSITIVITY TO THE SUBTASK-COMPLETION REWARD

  • On RoboCasa365 with GR00T-N1.5, SR rises from 40.0% at β=0.2 to 49.1% at β=0.6, stays at 49.0% at β=1.0, and falls to 46.5% at β=2.0.Performance is relatively insensitive to β in a moderate range; too little or too much subtask-reward weight can degrade it.Source: §D.3 · pp. 20–21, Figure 8 · p. 20

E CASE STUDY

  • In two case studies, STRUCTRL completes more intermediate steps than SFT and reaches later task stages, while SFT becomes stuck earlier.The examples track placing a bowl in a refrigerator and moving a bowl into a microwave.Source: E CASE STUDY · p. 21, Figure 9 · p. 21

F ROBOMETER BASELINE IMPLEMENTATION

The Robometer baseline predicts scalar task progress from causal visual history and converts it to chunk-level rewards. Its LIBERO-Long reward model is evaluated in-distribution because its training corpus includes demonstrations and failure trajectories from the same tasks.

F ROBOMETER BASELINE IMPLEMENTATION

  • Robometer-4B predicts scalar progress from the task instruction and causal visual history, and its training corpus includes LIBERO-Long demonstrations and task failure trajectories.The authors use the released checkpoint without additional fine-tuning, making its LIBERO-Long evaluation in-distribution.Source: F ROBOMETER BASELINE IMPLEMENTATION · p. 21
  • Before detected success, each chunk receives βR(pk−1); at the first successful chunk, the reward becomes βRpk + λc, with no further reward after success.The negative base is applied once per chunk, and the terminal reward is λc=2.0.Source: F ROBOMETER BASELINE IMPLEMENTATION · pp. 21–22
  • Robometer uses eight uniformly sampled frames from the causal history and is queried once per action chunk, every 16 simulator steps for GR00T-N1.5 or every 10 for π0.5.The history ends at the current boundary frame and uses no future observations.Source: F ROBOMETER BASELINE IMPLEMENTATION · p. 22
  • The reward-scale coefficients are βR=0.07041 for GR00T-N1.5 and βR=0.04401 for π0.5 on LIBERO-Long, selected before RL training to match STRUCTRL’s intermediate-reward scale.Other PPO hyperparameters, interaction budgets, terminal rewards, and evaluation settings are shared with STRUCTRL.Source: F ROBOMETER BASELINE IMPLEMENTATION · p. 22

G LIMITATIONS

The paper introduces STRUCTRL as online RL combining verifiable subtask supervision, structure-aware reward gating, and dynamic reward pacing. Its evaluation and subtask signals have stated boundaries: experiments are in simulation, and completion predicates rely on simulator signals.

G LIMITATIONS

  • STRUCTRL combines verifiable subtask supervision with structure-aware reward gating and dynamic reward pacing for long-horizon VLA training.The paper identifies strong empirical performance while noting that further study remains.Source: G LIMITATIONS · p. 22
  • The main experiments use simulator predicates for precise, reproducible subtask-completion detection; real-world use would require signals from sensory observations or other task-specific feedback.The authors identify suitable progress-detection mechanisms as a future-work direction.Source: G LIMITATIONS · p. 22
  • Evaluation is limited to simulation; physical-robot evaluation remains a next step for studying real-world perception, dynamics, and execution variability.Source: G LIMITATIONS · p. 22
Loading 2609.36352v1…