PaperlayerView exact revision on arXiv

Source-linked AI summary

ARM: Advantage Reward Modeling for Long-Horizon Manipulation

Yiming Mao, Zixi Yu, Weixin Mao, Yinhao Li, Qirui Hu, Zihan Lan, Minzhao Zhu, Hua Chen

arXiv:2604.03037v2cs.ROcs.AIcs.CV

TL;DR

Long-horizon manipulation requires richer reward signals because sparse rewards provide limited credit-assignment guidance, while dense progress supervision is costly and poorly suited to regressions. ARM estimates relative advantage using tri-state labels, reconstructs progress, and reweights behavior-cloning data; it achieves 99.4% success on towel folding.

Abstract

from arXiv · show

Long-horizon robotic manipulation remains challenging for reinforcement learning (RL) because sparse rewards provide limited guidance for credit assignment. Practical policy improvement thus relies on richer intermediate supervision, such as dense progress rewards, which are costly to obtain and ill-suited to non-monotonic behaviors such as backtracking and recovery. To address this, we propose Advantage Reward Modeling (ARM), a framework that shifts from hard-to-quantify absolute progress to estimating relative advantage. We introduce a cost-effective tri-state labeling strategy -- Progressive, Regressive, and Stagnant -- that reduces human cognitive overhead while ensuring high cross-annotator consistency. By training on these intuitive signals, ARM enables automated progress annotation for both complete demonstrations and fragmented DAgger-style data. Integrating ARM into an offline RL pipeline allows for adaptive action-reward reweighting, effectively filtering suboptimal samples. Our approach achieves a 99.4% success rate on a challenging long-horizon towel-folding task, demonstrating improved stability and data efficiency over current VLA baselines with near-zero human intervention during policy training.

1. Introduction

Long-horizon manipulation challenges RL because sparse rewards provide weak credit-assignment signals, while dense supervision is costly and difficult to design reliably. Existing approaches therefore face a reward-engineering bottleneck in scaling stable VLA learning.

  • Human demonstrations can contain suboptimality and noise, especially in complex long-horizon tasks, impeding policy convergence.Source: §1 · p. 1
  • Sparse success rewards often provide insufficient learning signals for effective credit assignment in long-horizon manipulation.Source: §1 · p. 1
  • Dense rewards or informative value functions are needed to provide continuous supervision for policy improvement.Source: §1 · p. 1
  • Existing dense-signal methods depend on high-precision progress reward models, creating a costly and difficult reward-engineering bottleneck.Source: §1 · p. 1

A. Advantage Reward Model(ARM) B. Global Progress Reconstruction

ARM replaces absolute-progress modeling with relative-advantage estimation and combines tri-state supervision, global progress reconstruction, and advantage-weighted behavior cloning. The framework is designed to handle regressions and fragmented data while filtering suboptimal samples.

Advantage Reward Model(ARM)

Global Progress Reconstruction

2. Related Work

Prior reward-learning and behavior-cloning methods address reward specification or demonstration quality through distance signals, temporal heuristics, and reweighting. Their limitations include weak fine-grained precision, monotonicity assumptions, inaccessible environment rewards, and costly manual annotations.

  • IRL and RLHF infer reward functions, but the literature identifies identifiability and scalability issues, respectively.Source: §2.1 · p. 2
  • Self-supervised VLM distance signals lack the precision needed for fine-grained, contact-rich manipulation and intermediate long-horizon progress.Source: §2.1 · p. 2
  • Several progress methods assume monotonic alignment between task progress and chronological order, which mishandles mistakes, retries, and temporary regressions.Source: §2.1 · p. 3
  • Advantage-based reweighted behavior cloning suppresses suboptimal trajectories, but traditional methods require explicit environment rewards to fit global value functions.Source: §2.2 · p. 3
  • RA-BC replaces environment rewards with a stage-aware reward model but relies heavily on costly manual language annotations.Source: §2.2 · p. 3

3. Method

ARM shifts long-horizon manipulation reward modeling from absolute progress to relative advantage estimation, using tri-state labels, multimodal temporal modeling, global progress reconstruction, and advantage-weighted behavior cloning. This pipeline produces dense supervision and reweights offline demonstrations to prioritize useful behaviors.

Global Progress Reconstruction

  • ARM reconstructs globally consistent dense progress trajectories by aggregating discrete interval predictions and anchoring them with task-completion signals.The reconstruction pipeline converts local relative predictions into full-sequence progress curves for downstream policy learning.Source: §3.1 · p. 3, §3.3 · pp. 4–5, Figure 3 · p. 5

Advantage Reward Model

  • ARM replaces absolute progress modeling with relative advantage estimation using a MIMO temporal Transformer over multimodal observations.The model processes historical visual, proprioceptive, and task-instruction inputs within a causal window to estimate local state transitions.Source: §3.2 · p. 3, §3.2.1 · pp. 3–4

3. Method

  • The method addresses dense-reward engineering challenges by reducing annotation burden and supporting automated labeling of unannotated trajectories.Simplified human labels provide a cold start for model inference over larger trajectory collections.Source: §3.2.2 · p. 4

Tri-state Auto Labeling

  • Tri-state labels classify state transitions as progressing, regressing, or stagnant, providing a lightweight and task-agnostic supervision signal.The labels encode movement toward the goal, deviation or failure, and the absence of substantial progress.Source: §3.1 · p. 3, §3.2.2 · p. 4

Policy Optimization via AW-BC

  • AW-BC uses reconstructed progress signals to reweight demonstrations, prioritizing high-advantage recovery behaviors and filtering suboptimal segments.This enables offline policy improvement from suboptimal demonstrations without explicit online interaction.Source: §3.1 · p. 3, §3.4 · p. 5, §3.4.3 · p. 6
  • Length-adaptive gains normalize progress changes by episode length so action chunks receive comparable advantage signals across heterogeneous demonstrations.The gain uses the current episode length and the dataset-average episode length to reduce instability from inconsistent progress gradients.Source: §3.4.1 · p. 5

4. Experiments

Experiments evaluate ARM on long-horizon bimanual towel folding, covering reward reconstruction, annotation efficiency, inference speed, and downstream policy performance. ARM produces more faithful progress signals and substantially improves policy success over baselines.

Experimental Setup

  • The evaluation uses a challenging 8-stage bimanual towel-folding task, with data comprising 972 episodes: 809 expert demonstrations and 163 DAgger error-correction episodes.The task includes extraction, placement, flattening, four folding operations, and transport into a target box.Source: §4.1 · p. 6

Reward Model Performance

  • ARM achieves lower progress-reconstruction MSE than SARM, reaching 0.0014 versus 0.0059, while perfectly identifying Standard and Failure episodes.The reward-model evaluation uses 50 validation trajectories and measures both numerical fidelity and terminal-state classification.Source: §4.2 · p. 6, Table 1 · p. 6
  • ARM reconstructs smooth, dense progress curves that track ground truth during non-monotonic adjustments, unlike SARM’s stepped curves and missed regressions.The tri-state approach also produces smoother signals than manual segmentation and VLM methods.Source: Figure 5 · p. 6, §4.2 · p. 7, §4.3 · p. 7

Efficiency and Quality of Reward Labeling

  • Tri-state annotation reaches 250 samples per 8-hour shift, a 2.5× speedup over the 100-sample baseline, while Auto Tri-state exceeds 400,000 samples per shift.The automated pipeline achieves a greater-than-133× speedup over human baselines.Source: §4.3 · p. 7

Inference Efficiency

  • ARM’s MIMO inference runs at 14.1 iterations per second, delivering a 13.7× speedup over VLM labeling and a 3.6× improvement over SARM.The architecture processes multiple advantage predictions simultaneously in one forward pass.Source: §4.4 · p. 7

Policy Performance Analysis

  • The full framework achieves a 99.4% success rate, while tri-state labeling raises success from 78.5% to 92.3% and AW-BC raises it further to 99.4%.The framework also reports 32 episodes per hour and folding precision of 3.6.Source: §4.5 · pp. 7–8

5. Conclusion

The paper presents ARM as a scalable framework for reward engineering in long-horizon robotic manipulation. Its tri-state supervision and advantage-weighted behavior cloning achieve a 99.4% success rate on towel folding.

Conclusion

  • ARM models relative advantages to reduce the inconsistency and cost of traditional dense labeling in long-horizon manipulation.The framework uses tri-state supervision and automated labeling to provide high-fidelity training signals.Source: §5 · p. 8
  • ARM with Advantage-Weighted Behavior Cloning achieves a 99.4% success rate on the challenging long-horizon towel-folding task.The conclusion describes the framework as scalable and robust for training high-performance policies.Source: §5 · p. 8

A. VLM Prompting Details

The VLM prompting procedure converts a complete towel-folding video into ordered, timestamped subtask segments. It requires visual boundary detection, full-video coverage, and strict JSON output using a closed label vocabulary.

Label Set

  • The prompting procedure uses an eight-stage towel-folding vocabulary covering extraction, placement, flattening, four folds, and final transport.The labels must be used exactly without creating or modifying categories.Source: A. VLM Prompting Details · p. 11

Ground-Truth Semantics

  • The timeline is defined by visual state changes: an action starts when motion clearly initiates and ends when the object reaches a completed, stable configuration.Short pauses and ambiguous micro-motions remain assigned to the current action.Source: A. VLM Prompting Details · p. 11

Output Procedure

  • The full video must be covered from 00:00 through the final timestamp without gaps, and the output must contain only valid JSON matching the timeline.The system parses stage names and timestamps into dense subtask annotation fields.Source: A. VLM Prompting Details · p. 11

Segmentation Constraints

  • Stages must appear once in logical order, with adjacent timestamps meeting exactly and the first stage beginning at 00:00.Uniform segmentation is disallowed unless supported by the video.Source: A. VLM Prompting Details · p. 11

B. Implementation Details

The implementation uses separate reward and policy models with distinct objectives. ARM is trained for multimodal temporal reward estimation, while AW-BC uses its predicted gains to weight policy-learning samples.

C. ARM Inference Results

ARM reconstructs progress on a held-out, non-monotonic towel-folding trajectory and detects temporary regressions. Its predicted progress remains temporally consistent while capturing the ground-truth downward dip.

D. Real-World Implementation Details

Real-world experiments use a 6-DoF bimanual robot operated through AgileX master-slave teleoperation. The system combines global and wrist-mounted cameras with 14-dimensional proprioceptive observations and actions.

E. Folding Precision Evaluation Protocol

Folding quality is evaluated with a 0–5 precision score that distinguishes complete folds by their final geometric accuracy and records partial completion separately.

Related papers

Loading 2604.03037v2…