PaperlayerView exact revision on arXiv

Source-linked AI summary

Value as Memory: Online Value-Guided Post-training for General Robot Policies

Wenye Yu, Yijin Chen, Kaiyi Xiong, Wenbo Han, Wendi Chen, Yang Jin, Han Xue, Jun Lv, Cewu Lu, Chuan Wen

arXiv:2610.22368v1cs.RO

TL;DR

Real-world robot post-training must address ambiguous action selection under partial observability and delayed feedback from deployment experience. VaM combines recent value predictions as task-progress memory with online learning from offline demonstrations, off-policy interventions, and on-policy rollouts; across real-world tasks, it reports a 29.4% mean absolute success-rate gain with value caching.

  • Problem

    Partial observability can make visually similar states require different actions, while offline post-training delays feedback from newly encountered deployment states.

    Source: §1 · pp. 1–2

  • Method

    VaM conditions actions on cached recent value predictions and performs online RL using offline demonstrations, off-policy interventions, and on-policy rollouts.

    Source: §1 · p. 2

  • Results

    29.4% mean absolute success-rate gain with value caching over its variant without value caching across real-world tasks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The evaluated online schedule performs 8× as many policy iterations under the same gradient-step budget, with more than 2× lower recorded time per gradient step during concurrent data collection and training.

    Source: §1 · p. 2

    The real-world RL framework is currently limited to a single robot; the experiments do not establish scaling with data from multiple robots.

    Source: §6 · p. 10

Abstract

from arXiv · show

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.

1 INTRODUCTION

VaM addresses action ambiguity under partial observability and delayed feedback by caching recent value predictions and learning online from a hybrid of offline and deployment data. Across real-world manipulation tasks, it reports higher success and more frequent policy updates under matched training budgets.

1 INTRODUCTION

  • Partial observability can make visually similar states require different actions, while existing real-world RL approaches limit historical-data reuse or delay feedback from newly encountered states.The passage attributes limited reuse to on-policy updates and delayed feedback to iterated offline RL.Source: §1 · p. 1
  • 29.4% higher mean success rate over the ablation without value caching, and 32.8% higher mean absolute success rate over Offline RL under matched training budgets.The evaluated online schedule also performs 8× as many policy iterations under the same gradient-step budget.Source: §1 · p. 2
  • With concurrent data collection and training, the evaluated online schedule records more than 2× lower time per gradient step.The comparison is against the offline schedule under the same gradient-step budget.Source: §1 · p. 2
  • Value caching supplies recent task-progress estimates as policy inputs, while the online RL framework jointly adapts policy and value models using offline demonstrations, off-policy trajectories, and on-policy rollouts.Training progresses from offline RL warmup through human-assisted collection to autonomous learning with sparse rewards.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries formulate iterative robot post-training as cumulative-reward optimization, describe distributional value learning for task progress, and specify an advantage-conditioned action-learning objective.

2.1 PROBLEM FORMULATION

  • Post-training deploys successive policies to collect rollout and human-intervention trajectories, then uses accumulated data to update the policy toward maximizing expected discounted cumulative reward.The setup begins from a policy initialized with successful expert demonstrations.Source: §2.1 · p. 2

2.2 DISTRIBUTIONAL VALUE LEARNING

  • The value expert predicts a categorical return distribution, initialized from normalized demonstration progress and adapted by TD learning with rewards only at successful terminal transitions.Successful terminal transitions receive +1; other transitions receive zero reward.Source: §2.2 · pp. 2–3

2.3 ADVANTAGE-CONDITIONED POLICY LEARNING

  • The action expert learns by flow matching with an optional binary advantage condition; at inference, condition 1 requests high-advantage actions.The training condition is formed by thresholding the estimated advantage at a percentile.Source: §2.3 · p. 3

3 METHOD

VaM combines a distributional value expert with an action expert, using recent progress predictions as memory and updating both experts through offline-to-online training. Its value cache can distinguish visually similar states with different action requirements, while the training stages reuse demonstrations, interventions, and rollouts.

3.1 METHOD OVERVIEW

  • VaM uses a distributional value expert to provide recent progress predictions to the action expert through a value cache, alongside advantage labels for policy post-training.The four stages are offline demonstration initialization, offline RL warmup, Online DAgger, and Learn from Experience.Source: §3.1 · p. 3

3.3 CONTINUAL OFFLINE-TO-ONLINE RL SYSTEM

  • The four-stage system moves from demonstration initialization and offline RL warmup to Online DAgger and autonomous Learn from Experience.Value and action experts are updated in parallel during online RL.Source: §3.1 · p. 3, §3.3 · p. 5

3.2.1 VALUE CACHING

  • The action expert uses a fixed prior-round value snapshot to condition current actions on recent task progress without delaying action generation.With at least two cached predictions, the sequence can indicate whether progress is improving, stalled, or declining.Source: §3.2.1 · pp. 3–4

3.2.2 WHY VALUE CACHING EASES SOFT ALIASING

  • When Lεobs < Δa ≤ Lεπ, an observation-only policy cannot match both target action distributions under the local Lipschitz condition, while value-cache inputs can weaken the lower bound.The lower bound is strictly positive when Δa > Lεr; because επ ≥ εobs, the cache can make the two states easier to separate.Source: §3.2.2 · p. 5

3.3.1 TRAINING OBJECTIVES

  • Demonstration initialization uses progress learning for the value expert and behavior cloning for the action expert; thereafter, the value expert uses TD learning and the action expert uses advantage-conditioned flow matching.The algorithm denotes these operations Init, BC, TD, and FM.Source: §3.3.1 · p. 5

3.3.2 TRAINING STAGES

  • Offline warmup adapts both experts to deployment states using policy rollouts and human interventions, while Online DAgger co-adapts them using current rollouts, replay data, and continuing human corrections.Learn from Experience then uses sparse terminal rewards from autonomous rollouts without new human corrections, retaining the historical intervention buffer fixed.Source: §3.3.2 · p. 5

3.4 IMPLEMENTATION DETAILS

  • The action expert retains π0.5’s pretrained PaliGemma-2B visual-language backbone and 300M-parameter action transformer, while the value expert repurposes the action transformer as its backbone.Both experts use temporal-spatial attention; the action expert encodes value predictions and proprioception with a GRU, and the value expert predicts categorical progress.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate VaM on three long-horizon manipulation tasks and compare value caching and offline-to-online learning against ablations and matched Offline RL. They examine success rates, progress-aware recovery, and training efficiency.

4.1 TASKS AND EVALUATION PROTOCOL

  • The evaluation covers Box Assembly, Laundry Folding, and Magnetic Bus Assembly, three long-horizon workflows involving contact-rich assembly and deformable-object manipulation.Success rate is the fraction of episodes completed within each task’s time limit; Offline RL and Online RL comparisons also report wall-clock time per gradient step.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 HOW MUCH DOES VALUE CACHING IMPROVE POLICY PERFORMANCE IN RL POST-TRAINING?

  • 29.4% mean success-rate gain for VaM over the ablation without value caching across all three tasks, measured over the final three checkpoints.The gap is most pronounced during LfE and persists after on-policy DAgger data stops; VaM also exceeds the ablation without offline RL warmup on every task.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 WHY DOES VALUE CACHING IMPROVE POLICY PERFORMANCE?

  • On Box Assembly, cached rollouts show smoother value-expert embedding trajectories ordered by progress, with deviations at stuck states preceding corrective actions.Without caching, stages are more intermingled, with low or fluctuating values during failed execution.Source: §4.3 · p. 7, Figure 5 · p. 8
  • VaM exceeds its caching ablation in observed recovery rate on all three tasks during LfE.Recovery Rate is the ratio of successful recoveries to recovery-requiring events.Source: §4.3 · pp. 7–8, Table 2 · p. 7

4.4 HOW MUCH DOES VaM’S OFFLINE-TO-ONLINE RL SYSTEM ENHANCE SAMPLE EFFICIENCY?

  • 32.8% average absolute success-rate gain over matched Offline RL baselines after LfE across all tasks.Online RL reduces experimental time per gradient step by 57.8% on average through concurrent data collection and training.Source: §4.4 · p. 8, Figure 6 · p. 9

5 RELATED WORK

Related work covers policy post-training, intervention-based adaptation, and offline-to-online RL. VaM is distinguished by continually updating value from deployment experience and using value predictions both to guide policy learning and as policy inputs.

5.1 POST-TRAINING OF GENERALIST ROBOT POLICIES

  • Prior post-training work adapts generalist robot policies with tailored residual or RL objectives, generative-policy formulations, and world models; value methods also filter data or estimate trajectory progress.VaM instead continually updates value from online rollouts and human interventions, using predictions both as policy-learning supervision and as explicit policy inputs.Source: §5.1 · p. 9

5.2 HUMAN-IN-THE-LOOP ONLINE ADAPTATION

  • Human-in-the-loop methods use interventions as corrective demonstrations and scale collection through selective assistance or fleet-scale systems.VaM also uses intervention trajectories to update a value model that guides policy learning and conditions policy input.Source: §5.2 · p. 9

5.3 OFFLINE-TO-ONLINE REINFORCEMENT LEARNING

  • Offline-to-online RL methods retain or replay offline data during online training; VaM combines offline demonstrations, historical off-policy intervention data, and current on-policy experience.Source: §5.3 · pp. 9–10

6 DISCUSSIONS AND CONCLUSIONS

VaM uses recent value predictions as task-progress memory and combines demonstrations, off-policy trajectories, and on-policy rollouts for online policy and value adaptation. The paper identifies dependence on accurate value predictions and single-robot evaluation as limitations.

6 DISCUSSIONS AND CONCLUSIONS

  • Value caching depends on a value function that closely tracks task progress, and the framework has been evaluated on a single robot.The experiments do not establish how the framework scales when multiple robots contribute data.Source: §6 · p. 10
  • The paper describes VaM as combining recent value predictions with offline demonstrations, off-policy trajectories, and on-policy rollouts for online adaptation.It presents the combination as a practical approach to post-training generalist robot policies.Source: §6 · p. 10

AI USE STATEMENT

  • The authors report using generative AI to polish writing, assist with research execution, and draft sections of the paper.They state that they did not use it for literature retrieval or discovery, research ideation, synthetic dataset generation, or proving mathematical claims.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

A EXPERIMENTAL SETUP

  • Figure 8 summarizes the real-robot platforms and task workspaces, while Table 3 reports episode counts for demonstrations, offline RL warmup, and Online RL collection rounds.Source: A EXPERIMENTAL SETUP · p. 15

A.1 TASK PROCEDURES

The appendix describes three real-robot tasks and supplementary evaluations of value-expert replacement, checkpoint aggregation, and recovery events. The reported sensitivity analysis finds that VaM’s success-rate advantage over the caching ablation persists across checkpoint aggregation windows.

A.1 TASK PROCEDURES

  • Box Assembly requires folding and clamping both side flaps until the box lies horizontally, within a 120 s limit.The task is evaluated on four tablecloths with different textures and appearances.Source: §A.1 · p. 15
  • Laundry Folding requires spreading, folding, and moving a garment to a table corner while keeping it folded, within 240 s.Source: §A.1 · p. 15
  • Magnetic Bus Assembly requires joining six blocks into a bus with a connected horizontal base and adjoining body blocks spelling “VAM,” within 300 s.Source: §A.1 · p. 15
  • Figure 8 summarizes the robot hardware, sensing configuration, human-intervention interface, and task workspaces.Source: Figure 8 · p. 15
  • The Offline RL and Online RL comparisons use matched total gradient-step and data-collection budgets within each stage.Offline RL batches updates, whereas Online RL distributes them across repeated collection-and-training iterations.Source: §A.1 · p. 15

B.1 VALUE-EXPERT REPLACEMENT AT INFERENCE

  • The value-expert replacement evaluation fixes action experts at the final three LfE checkpoints and compares demonstration-initialized and post-warmup value experts with the original LfE value reference.The replacement conditions preserve value caching; the original reference reuses main-experiment results.Source: §B.1 · p. 16

B.2 SENSITIVITY TO CHECKPOINT AGGREGATION

  • 31.3% mean success-rate gap over the caching ablation across all available LfE checkpoints, compared with 29.4% over the final three checkpoints.The authors report gains on each task and describe the consistent gaps as supporting robustness to the aggregation window.Source: §B.2 · p. 16, Table 4 · p. 16

B.3 RECOVERY-EVENT COUNTS

  • The blind recovery evaluation reports successful recoveries, recovery events, and reviewed episodes, with rates calculated at the event level.Multiple events may occur in one episode, so event and episode counts are different units.Source: §B.3 · p. 16, Table 5 · p. 16

B.4 TOTAL EXPERIMENTAL TIME

The appendix describes how experimental time is measured and identifies hyperparameter information reported for value and action learning. Online RL runs collection and both training stages concurrently, unlike the sequential Offline RL schedule.

B.4 TOTAL EXPERIMENTAL TIME

  • Online RL runs data collection, value training, and action training concurrently, while Offline RL collects data before training the value expert and then the action expert.Table 6 reports time per gradient step and total experimental time; time per step is given as mean ± sample standard deviation.Source: §B.4 · p. 17, Table 6 · p. 17

B.5 HYPERPARAMETERS

  • Table 7 summarizes value- and action-learning hyperparameters, including task-specific settings.The observation-history stride is the interval in environment steps between sampled observations, and update counts are measured in gradient steps.Source: §B.5 · p. 17, Table 7 · p. 17

C ADDITIONAL QUALITATIVE COMPARISONS OF VALUE CACHING

Qualitative comparisons show VaM’s value caching alongside more progress-ordered value trajectories and recovery from stuck states in autonomous rollouts. The section also describes the offline-to-online training stages and how distributional TD targets are constructed.

C ADDITIONAL QUALITATIVE COMPARISONS OF VALUE CACHING

  • Across Laundry Folding and Magnetic Bus Assembly, VaM shows more progress-ordered PCA trajectories and recovery from stuck states, while the ablation shows failed execution with low or misleading value predictions.Figure 9 presents camera images, value-expert PCA embeddings, and scalar value predictions; its dark-blue trace is a 15-frame mean.Source: C ADDITIONAL QUALITATIVE COMPARISONS… · p. 18, Figure 9 · p. 18
  • In two Laundry Folding cases with fixed action expert and observation inputs, replacing the original LfE values with demo-trained or post-warmup estimates changes predicted trajectories and gripper commands.Figures 10 and 11 show predictions after losing contact and after re-grasping seams; the original cache maintains right-gripper closure after re-grasping, while the replacement values predict opening commands near the action-chunk end.Source: C ADDITIONAL QUALITATIVE COMPARISONS… · p. 19, Figure 11 · p. 19, Figure 10 · p. 19

D TRAINING PIPELINE

  • The training pipeline proceeds from demonstration initialization and offline RL warmup to Online DAgger and autonomous Learn from Experience rounds.The algorithm fragments describe warmup data collection with interventions, online rounds with intervention enabled, and LfE rollouts with sparse terminal outcomes.Source: D TRAINING PIPELINE · p. 20

E.1 DISTRIBUTIONAL TD TARGETS

  • For each retained H-step tuple, the TD target combines accumulated reward with a discounted target-network atom unless the segment is terminal, then clips the atom location to the support.The linear categorical projection distributes mass between neighboring support atoms; at termination, bootstrap atoms map to accumulated reward.Source: §E.1 · p. 21

E.2 WARMUP VALUE-DATA CONSTRUCTION

The warmup constructs value-learning data from policy rollouts and human interventions, while online rounds combine demonstrations, past experience, and current rollouts. The formal setup also extends the soft-aliasing bound to states with unequal relative weights.

E.2 WARMUP VALUE-DATA CONSTRUCTION

  • Human-takeover prefixes receive zero-reward failure supervision, while complete rollouts retain outcome rewards and bootstrap across takeovers.Temporal downsampling preserves each H-step reward and terminal-masked bootstrap when forming transitions from the prefixes and rollouts.Source: §E.2 · p. 21

E.3 ONLINE DATA SOURCES

  • Online rounds draw from fixed demonstrations, current on-policy rollouts, and earlier off-policy rollouts, with human-controlled segments extracted for correction data.The resulting samples receive value-derived advantage conditions for policy post-training.Source: §E.3 · pp. 21–22

F FORMAL SETUP FOR SOFT-ALIASING ANALYSIS

  • The soft-aliasing analysis defines target action-chunk distributions for two states and extends its bound to states with relative weights α and 1 − α.The setup assumes action-sequence distributions have finite second moments so the Wasserstein distance is finite.Source: F FORMAL SETUP FOR SOFT-ALIASING… · p. 22

G PROOF OF THE SOFT-ALIASING BOUND

The proof bounds the combined prediction error for two target action distributions using their Wasserstein separation and the policy’s local Lipschitz constraint. It then derives a frequency-weighted extension.

G PROOF OF THE SOFT-ALIASING BOUND

  • The target-distribution separation Δa is at most the sum of both prediction errors and Lεr, yielding a lower bound of (Δa − Lεr)+ on their combined error.The proof uses the Wasserstein triangle inequality and the local Lipschitz assumption.Source: G PROOF OF THE SOFT-ALIASING BOUND · p. 22
  • When Lεobs < Δa, exact observation-only matching is ruled out, while Δa ≤ Lεπ makes exact matching compatible with the local Lipschitz constraint.Compatibility does not establish that a particular parameterized policy attains the target distributions after optimization.Source: G PROOF OF THE SOFT-ALIASING BOUND · p. 22
  • A frequency-weighted lower bound follows by combining the two-state error inequality with the relative weights α and 1 − α.The proof establishes the inequality by expanding a nonnegative square.Source: G PROOF OF THE SOFT-ALIASING BOUND · p. 22

Related papers

Loading 2610.22368v1…