PaperlayerView exact revision on arXiv

Source-linked AI summary

ARM: Advantage Reward Modeling for Long-Horizon Manipulation

Yiming Mao, Zixi Yu, Weixin Mao, Yinhao Li, Qirui Hu, Zihan Lan, Minzhao Zhu, Hua Chen

arXiv:2604.03037v2cs.ROcs.AIcs.CV

TL;DR

Long-horizon manipulation requires richer reward signals because sparse rewards provide limited credit-assignment guidance, while dense progress supervision is costly and poorly suited to regressions. ARM estimates relative advantage using tri-state labels, reconstructs progress, and reweights behavior-cloning data; it achieves 99.4% success on towel folding.

  • Problem

    Sparse rewards provide limited credit-assignment guidance, while dense progress reward models are costly and unreliable for non-monotonic long-horizon behavior.

  • Method

    ARM uses tri-state relative-advantage labels, multimodal reward modeling, automated progress reconstruction, and advantage-weighted behavior cloning for fragmented data.

  • Results

    99.4% success rate was achieved on a challenging long-horizon towel-folding task.

  • Takeaways & Limitations

    ARM provides a scalable approach for generating dense supervision and training policies from heterogeneous or fragmented demonstrations.

Abstract

from arXiv · show

Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.

1. Introduction

Long-horizon manipulation challenges RL because sparse rewards provide weak credit-assignment signals, while dense supervision is costly and difficult to design reliably. Existing approaches therefore face a reward-engineering bottleneck in scaling stable VLA learning.

  • Human demonstrations can contain suboptimality and noise, especially in complex long-horizon tasks, impeding policy convergence.
  • Sparse success rewards often provide insufficient learning signals for effective credit assignment in long-horizon manipulation.
  • Dense rewards or informative value functions are needed to provide continuous supervision for policy improvement.
  • Existing dense-signal methods depend on high-precision progress reward models, creating a costly and difficult reward-engineering bottleneck.

A. Advantage Reward Model(ARM) B. Global Progress Reconstruction

ARM replaces absolute-progress modeling with relative-advantage estimation and combines tri-state supervision, global progress reconstruction, and advantage-weighted behavior cloning. The framework is designed to handle regressions and fragmented data while filtering suboptimal samples.

  • Advantage Reward Model(ARM): ARM models relative advantage instead of absolute progress, allowing progress rewards to accommodate regressive behaviors and error recovery.
  • Advantage Reward Model(ARM): The tri-state labeling strategy categorizes segments as Progressive, Regressive, or Stagnant to provide lightweight, task-agnostic supervision.
  • Global Progress Reconstruction: ARM combines multimodal temporal predictions with a task-completion head to reconstruct globally consistent dense progress trajectories from discrete labels.
  • Global Progress Reconstruction: AW-BC uses adaptive scaling and predicted interval gains to support fragmented DAgger data, filter suboptimal samples, and prioritize recovery trajectories.
  • Global Progress Reconstruction: 99.4% success rate was achieved on a challenging long-horizon towel-folding task.

2. Related Work

Prior reward-learning and behavior-cloning methods address reward specification or demonstration quality through distance signals, temporal heuristics, and reweighting. Their limitations include weak fine-grained precision, monotonicity assumptions, inaccessible environment rewards, and costly manual annotations.

  • IRL and RLHF infer reward functions, but the literature identifies identifiability and scalability issues, respectively.
  • Self-supervised VLM distance signals lack the precision needed for fine-grained, contact-rich manipulation and intermediate long-horizon progress.
  • Several progress methods assume monotonic alignment between task progress and chronological order, which mishandles mistakes, retries, and temporary regressions.
  • Advantage-based reweighted behavior cloning suppresses suboptimal trajectories, but traditional methods require explicit environment rewards to fit global value functions.
  • RA-BC replaces environment rewards with a stage-aware reward model but relies heavily on costly manual language annotations.

3. Method

ARM shifts long-horizon manipulation reward modeling from absolute progress to relative advantage estimation, using tri-state labels, multimodal temporal modeling, global progress reconstruction, and advantage-weighted behavior cloning. This pipeline produces dense supervision and reweights offline demonstrations to prioritize useful behaviors.

  • Global Progress Reconstruction: ARM reconstructs globally consistent dense progress trajectories by aggregating discrete interval predictions and anchoring them with task-completion signals.The reconstruction pipeline converts local relative predictions into full-sequence progress curves for downstream policy learning.
  • Advantage Reward Model: ARM replaces absolute progress modeling with relative advantage estimation using a MIMO temporal Transformer over multimodal observations.The model processes historical visual, proprioceptive, and task-instruction inputs within a causal window to estimate local state transitions.
  • 3. Method: The method addresses dense-reward engineering challenges by reducing annotation burden and supporting automated labeling of unannotated trajectories.Simplified human labels provide a cold start for model inference over larger trajectory collections.
  • Tri-state Auto Labeling: Tri-state labels classify state transitions as progressing, regressing, or stagnant, providing a lightweight and task-agnostic supervision signal.The labels encode movement toward the goal, deviation or failure, and the absence of substantial progress.
  • Policy Optimization via AW-BC: AW-BC uses reconstructed progress signals to reweight demonstrations, prioritizing high-advantage recovery behaviors and filtering suboptimal segments.This enables offline policy improvement from suboptimal demonstrations without explicit online interaction.
  • Policy Optimization via AW-BC: Length-adaptive gains normalize progress changes by episode length so action chunks receive comparable advantage signals across heterogeneous demonstrations.The gain uses the current episode length and the dataset-average episode length to reduce instability from inconsistent progress gradients.

4. Experiments

Experiments evaluate ARM on long-horizon bimanual towel folding, covering reward reconstruction, annotation efficiency, inference speed, and downstream policy performance. ARM produces more faithful progress signals and substantially improves policy success over baselines.

  • Experimental Setup: The evaluation uses a challenging 8-stage bimanual towel-folding task, with data comprising 972 episodes: 809 expert demonstrations and 163 DAgger error-correction episodes.The task includes extraction, placement, flattening, four folding operations, and transport into a target box.
  • Reward Model Performance: ARM achieves lower progress-reconstruction MSE than SARM, reaching 0.0014 versus 0.0059, while perfectly identifying Standard and Failure episodes.The reward-model evaluation uses 50 validation trajectories and measures both numerical fidelity and terminal-state classification.
  • Reward Model Performance: ARM reconstructs smooth, dense progress curves that track ground truth during non-monotonic adjustments, unlike SARM’s stepped curves and missed regressions.The tri-state approach also produces smoother signals than manual segmentation and VLM methods.
  • Efficiency and Quality of Reward Labeling: Tri-state annotation reaches 250 samples per 8-hour shift, a 2.5× speedup over the 100-sample baseline, while Auto Tri-state exceeds 400,000 samples per shift.The automated pipeline achieves a greater-than-133× speedup over human baselines.
  • Inference Efficiency: ARM’s MIMO inference runs at 14.1 iterations per second, delivering a 13.7× speedup over VLM labeling and a 3.6× improvement over SARM.The architecture processes multiple advantage predictions simultaneously in one forward pass.
  • Policy Performance Analysis: The full framework achieves a 99.4% success rate, while tri-state labeling raises success from 78.5% to 92.3% and AW-BC raises it further to 99.4%.The framework also reports 32 episodes per hour and folding precision of 3.6.

5. Conclusion

The paper presents ARM as a scalable framework for reward engineering in long-horizon robotic manipulation. Its tri-state supervision and advantage-weighted behavior cloning achieve a 99.4% success rate on towel folding.

  • Conclusion: ARM models relative advantages to reduce the inconsistency and cost of traditional dense labeling in long-horizon manipulation.The framework uses tri-state supervision and automated labeling to provide high-fidelity training signals.
  • Conclusion: ARM with Advantage-Weighted Behavior Cloning achieves a 99.4% success rate on the challenging long-horizon towel-folding task.The conclusion describes the framework as scalable and robust for training high-performance policies.

A. VLM Prompting Details

The VLM prompting procedure converts a complete towel-folding video into ordered, timestamped subtask segments. It requires visual boundary detection, full-video coverage, and strict JSON output using a closed label vocabulary.

  • Label Set: The prompting procedure uses an eight-stage towel-folding vocabulary covering extraction, placement, flattening, four folds, and final transport.The labels must be used exactly without creating or modifying categories.
  • Ground-Truth Semantics: The timeline is defined by visual state changes: an action starts when motion clearly initiates and ends when the object reaches a completed, stable configuration.Short pauses and ambiguous micro-motions remain assigned to the current action.
  • Output Procedure: The full video must be covered from 00:00 through the final timestamp without gaps, and the output must contain only valid JSON matching the timeline.The system parses stage names and timestamps into dense subtask annotation fields.
  • Segmentation Constraints: Stages must appear once in logical order, with adjacent timestamps meeting exactly and the first stage beginning at 00:00.Uniform segmentation is disallowed unless supported by the video.

B. Implementation Details

The implementation uses separate reward and policy models with distinct objectives. ARM is trained for multimodal temporal reward estimation, while AW-BC uses its predicted gains to weight policy-learning samples.

  • The framework comprises an Advantage Reward Model and a Policy Model optimized for distinct objectives.
  • ARM combines a CLIP ViT-B/32 vision-text encoder with a causal five-frame Transformer aggregator sampled at 1 Hz.
  • ARM jointly trains tri-state interval classification with task-completion prediction using cross-entropy and focal losses.
  • AW-BC derives sample weights from ARM-predicted gains within a GR00T-N1.5-based policy.

C. ARM Inference Results

ARM reconstructs progress on a held-out, non-monotonic towel-folding trajectory and detects temporary regressions. Its predicted progress remains temporally consistent while capturing the ground-truth downward dip.

  • ARM reconstructs dense progress for a long-horizon towel-folding sequence containing non-monotonic behavior.
  • A necessary towel-edge adjustment temporarily moves the state farther from the target rectangular configuration.
  • At t = 69s to t = 70s, predicted progress falls from 86.15% to 84.62% with ∆pred = −1.
  • The predicted curve captures the regression as a downward dip aligned with ground truth.
  • ARM maintains a smooth, temporally consistent signal without cumulative drift or stepped artifacts during inference.

D. Real-World Implementation Details

Real-world experiments use a 6-DoF bimanual robot operated through AgileX master-slave teleoperation. The system combines global and wrist-mounted cameras with 14-dimensional proprioceptive observations and actions.

  • The hardware platform uses a 6-DoF bimanual robot with an AgileX master-slave teleoperation interface.
  • Visual observations come from one global base camera and two wrist-mounted cameras.
  • The system provides three RGB views: a global high view and left and right wrist views.
  • Proprioceptive state and action spaces are both 14-dimensional, covering joint positions and gripper states.

E. Folding Precision Evaluation Protocol

Folding quality is evaluated with a 0–5 precision score that distinguishes complete folds by their final geometric accuracy and records partial completion separately.

  • A score of 5 denotes a completed fold with precision within 1 cm, while scores 4 and 3 correspond to 1–2 cm and 2–3 cm.
  • Scores 2 and 1 represent successful flattening with partial folding or no valid folding steps, respectively.
  • A score of 0 indicates that no task steps were successfully completed.
Loading 2604.03037v2…