HiRE: Hindsight Reward Editing for Policy Finetuning

Reward-modeling challenge

• Sparse rewards lack process feedback for long-horizon tasks, human-designed dense rewards can be biased and task-specific, and current general-purpose reward models struggle with OOD tasks.The introduction also notes that high-quality labels for large-scale reward models are a bottleneck and that such models often need calibration for new scenarios.Source: §1 · p. 1

Motivation

• HiRE targets rewards that generalize across visual scenarios while adapting to physical task outcomes, using foundation representations and interactive feedback without costly human labels or reward-model training.Foundation vision(-language) representations offer similarity-based rewards but are not control-aware because they were not trained on robot behavior.Source: §1 · pp. 1–2

Method

• HiRE contrasts successful and failed trajectories to penalize states near failure traps and boost states leading to success, calibrating foundation-representation rewards without gradient-based optimization.It combines refined dense rewards with sparse success rewards through potential-based reward shaping, which the paper says leaves the optimal task-success policy unchanged.Source: §1 · p. 2

Empirical results

• HiRE raises base success rates below 10% to near 100% on manipulation benchmarks and achieves at least 3× improvement on a YAM robot arm, sometimes with 30 online rollouts.The reported tasks span prehensile, non-prehensile, and long-horizon manipulation on MimicGen and RoboMimic, alongside a single real-world YAM robot arm.Source: §1 · p. 2

Analyses and compatibility

• The paper studies HiRE’s effects by visualizing reward landscapes during finetuning and ablating its key components.The proposed recipe is described as compatible with foundation representations such as DINOv2 and SigLIP and RL finetuning algorithms summarized in DICE-RL.Source: §1 · p. 2

2.1 Reward Learning for Robotics

• Task-specific reward models use human annotations, video rankings, trajectory metrics, or curated robotic data, limiting generalization to novel scenes and task semantics.Examples include DrS [18], SARM [19], Rank2Reward [20], RoboReward [3], RoboMeter [4], and Robo-Dopamine [6].Source: §2.1 · p. 2
• Passive-behavior embeddings provide zero-shot progress rewards, but can fail in unseen scenarios; GCR [12] finetunes VIP embeddings on task-specific successful and failed trajectories.The passage names VIP [8], LIV [9], R3M [21], and DecisionNCE [22] as pretrained representations.Source: §2.1 · p. 2
• Foundation-model rewards generalize broadly but are control-unaware and may overestimate deceptive states; HiRE edits them training-free by contrasting successful and failed rollouts.HiRE adds no auxiliary network or manual dense labels and is agnostic to representation models and RL algorithms.Source: §2.1 · p. 3

2.2 RL Policy Finetuning for Robotics

• HiRE is orthogonal to policy-improvement recipes and can integrate with them, providing dynamic, training-free reward shaping toward physically correct trajectories.The passage discusses πRL [26], DSRL [28], ResFiT [29], Policy Decorator [30], RL-Token [2], and DICE-RL [15].Source: §2.2 · p. 3

• The RL problem is a finite-horizon MDP defined by state and action spaces, rewards, transition dynamics, discount factor, and task horizon.Source: §3 · p. 3
• The objective is to find a policy that maximizes expected cumulative discounted reward from the initial state distribution.Source: §3 · p. 3
• Approximate dynamic-programming RL methods typically learn action-value Q(s, a), and may also learn state-value V(s), to estimate cumulative discounted reward for policy optimization.Source: §3 · p. 3

4 HiRE: A Training-free Reward Recipe for Policy Finetuning

• Inspired by adversarial imitation [33] and maximum-entropy IRL, HiRE treats the success-versus-failure discriminator’s log-odds as the ideal reward.This reframes reward construction as estimating the success-to-failure density ratio rather than training a parameterized discriminator.Source: §4 · pp. 3–4

4.1 Non-parametric Reward Estimation in Foundation Representation Space

• HiRE estimates success and failure densities non-parametrically with KDE and a von Mises–Fisher kernel suited to normalized representations on a hypersphere.The ratio λ = κ−/κ+ aligns the success and failure landscapes; for precision-critical assembly, λ < 1 yields sharp attraction toward narrow successful states and gentler repulsion from broad failure modes.Source: §4.1 · p. 4
• Potential-based shaping combines HiRE’s control-aware potential with sparse task rewards to provide dense guidance without adding unintended optima.The discount factor γ is used in the shaping term, preserving the task objective.Source: §4.1 · p. 4
• HiRE raises rewards for visually distant but dynamically correct states, penalizes goal-like states that have physically failed, and cancels shaping gradients in equilibrium regions.In under-covered or out-of-distribution regions, the gradient can vanish, letting the method degrade to sparse-reward learning rather than issue conflicting signals.Source: §4.1 · p. 5

4.2 RL Policy Finetuning with HiRE

• During finetuning, HiRE estimates densities from expert and recent-success states in a positive buffer and terminal failure states in a negative FIFO buffer.Its shaped reward enters a generalized actor-critic objective, leaving the underlying RL algorithm interchangeable.Source: §4.2 · p. 5
• HiRE supports off-policy and on-policy backbones, behavior-regularized exploration, and safe finetuning; the main implementation combines it with DICE-RL.Examples include PPO [27], TD3 [37], SAC [38], and DICE-RL [15].Source: §4.2 · p. 5

5.1 Experimental Setup

• The evaluation combines RoboMimic and MimicGen simulation tasks with two I2RT YAM manipulation tasks, using DICE-RL to finetune pretrained policies and comparing HiRE with four reward baselines.DINOv2 and SigLIP serve as foundation encoders; baselines are sparse reward, RoboMeter, TOPReward, and GCR. Offline BC base-policy success rates range from 1% to 20% across the reported tasks.Source: §5 · p. 5, §5.1 · p. 6

5.2 Benchmark Comparative Results

• On Stack_Three, HiRE reaches ~50% versus sparse ~30%; on Three_Piece_Assembly it exceeds 80% versus ~70%, and on Tool_Hang reaches ~95% while matching sparse performance on Threading with fewer interactions.The reported advantages arise from dense intermediate progress feedback on long-horizon and contact-rich tasks where sparse rewards provide limited credit assignment or task successes are rare.Source: Figure 1 · p. 6, §5.2 · pp. 6–7

5.4 Visualizations and Analyses

• Raw DINO goal similarity under-rewards visually distinct but useful target approaches and over-rewards deceptive empty-background states, while HiRE’s edited potential reflects progress and corrects these errors.The comparison uses Threading and Stack_Three trajectories; in Threading, raw potential falls during correct early approach and rises during target-loss and overshoot phases.Source: Figure 2 · p. 7, §5.4 · p. 8

5.2 Benchmark Comparative Results

• RoboMeter and TOPReward achieve substantially lower final success rates than HiRE; GCR needs roughly 3 times as many interactions to reach 30% success on multi-stage tasks and makes little progress on precision tasks.GCR uses gradient-based reward-model updates, whereas HiRE’s dense reward recipe is training-free.Source: §5.2 · p. 7

5.3 Real-Robot Comparative Results

• On Bottle_Pick-and-Place, HiRE improves success from 20% to 67%; on Towers_of_Hanoi, it reaches 53.3% after 20 episodes and peaks at 73.3% (3.67× base policy).Sparse peaks at 53.3% after 30 episodes and RoboMeter at 46.7% after 40; both baselines later degrade, while HiRE sustains gains on both hardware tasks.Source: §5.3 · pp. 7–8, Figure 3 · p. 8

5.4 Visualizations and Analyses

• As finetuning improves, HiRE’s negative buffer shifts from initial-state failures to localized later-stage failure clusters, avoiding global over-penalization; its positive samples remain concentrated while negatives are more dispersed.This observed distributional difference supports assigning a larger concentration factor κ+ than κ−.Source: §5.4 · p. 9, Figure 5 · p. 9

6 Conclusion and Limitations

• HiRE consistently improves final success rate and sample efficiency over other reward methods across RoboMimic, MimicGen, and real-robot platforms.The supplied passage gives no specific benchmark values.Source: §6 · p. 9
• HiRE edits rewards online from successful and failed rollouts, suppressing deceptive trap states while preserving genuine task progress without reward-model training or human labels.Source: §6 · p. 9
• The lightweight calibration remains agnostic to the foundation encoder and finetuning algorithm, offering a practical, scalable recipe for existing RL finetuning pipelines.Source: §6 · p. 9
• Success and failure signals provide accessible, clean feedback but overlook intermediate, control-centric information.Source: §6 · p. 9
• Future work could incorporate human guidance into interaction data to make rewards more control-aware and nuanced.Source: §6 · p. 9
• Future buffer construction could account for suboptimal segments within successful trajectories, including retries and recovery behaviors of varying degrees.Source: §6 · p. 9

• It describes the common setup, then details HiRE and baseline reward implementations for the main comparison in Fig. 1.Source: A Implementation Details · p. 13

A.1 Benchmark and Policy Setup

• Experiments use Tool_Hang, Threading, Stack_Three, and Three_Piece_Assembly with a 20 Hz OSC_POSE controller, 7-D actions, and binary task-success evaluation.Episodes count as failed only after a timeout without task completion; the failure buffer stores the final frame, avoiding penalties on valid intermediate states shared with successful trajectories.Source: §A.1 · p. 13
• All baselines share a ResNet-18 visual encoder with spatial softmax and a 1D UNet action head, using diffusion policies for MimicGen and flow matching for Tool_Hang.MimicGen uses ϵ-prediction with 50 DDPM steps; Tool_Hang uses velocity prediction with 20-step Euler integration.Source: §A.1 · p. 13

Hardware setup

• Experiments use a 6-DoF I2RT YAM arm with two RealSense cameras, one providing a global base view and the other a close-up wrist view.A teleoperation workstation supports expert demonstration collection, and human operators judge real-hardware success and failure.Source: §A.2 · pp. 13–14

Task descriptions

• Bottle_Pick-and-Place tests compliant coarse-to-fine grasping and placement across six initial bottle positions spanning a wide tabletop workspace.Initial positions are randomized within approximately ±10 cm; the six fixed evaluation positions are shown in Figure 6.Source: §A.2 · p. 14, Figure 6 · p. 14
• Towers_of_Hanoi requires moving a green disk between rods, while unanchored base and sliding-cover shifts create perturbations and potentially unrecoverable alignment errors.The initial setup is standardized across trials, but platform movement demands precise execution and robust feedback.Source: §A.2 · p. 14

Policy training and evaluation

• Base diffusion policies are initialized from 117 teleoperated Bottle_Pick-and-Place demonstrations and 50 Hanoi demonstrations, following DICE-RL [15].Finetuning begins with 20 warmup episodes, followed by 2,000 optimization steps at batch size 256; 10 new episodes are collected after each 1,000 steps.Source: §A.2 · p. 14
• HiRE, sparse-reward, and RoboMeter finetuning share the same pretrained policy and 20 warmup episodes, with success rates measured over 30 bottle-task and 15 Hanoi trials per checkpoint.HiRE uses positive and negative buffer sizes of 4096 and 320, respectively, and κ+ = 10, κ− = 1, and λ = 0.1; a human annotator labels episodes as success or failure.Source: §A.2 · p. 14

A.3 Method Implementations

• All reward baselines share the DICE-RL backbone and fixed optimization settings, differing only in transition rewards; the sparse baseline gives 1 only for successful terminal transitions.DICE-RL freezes the pretrained policy as an action proposal model and learns a residual actor with an ensemble critic.Source: §A.3 · pp. 14–16
• HiRE augments sparse success reward with an online-edited control-aware potential built from frozen DINOv2 patch-token similarities averaged across two camera views.Images are resized to 224×224 and ImageNet-normalized before encoding.Source: §A.3 · pp. 15–16
• HiRE’s positive buffer mixes offline expert frames sampled at stride 5 with online successful terminal frames, while its negative buffer stores failed terminal frames in capacity-128 FIFOs.The positive sources are sampled in a 0.5 ratio; terminal-only online references avoid reinforcing exploratory intermediate behavior or penalizing valid pre-failure behavior.Source: §A.3 · pp. 15–16
• HiRE applies potential-based shaping with γ = 0.99, normalizes by action-chunk length 8, and decays shaping weight as recent success rises.Replayed expert demonstrations retain only the sparse environment reward and are not relabeled with the online contrastive reward.Source: §A.3 · p. 16
• RoboMeter [4] supplies clipped progress-head rewards from subsampled task-conditioned episode prefixes, queried every four action chunks with cached progress estimates between queries.The reward measures progress increments rather than repeatedly rewarding the same absolute progress level.Source: §A.3 · p. 16
• TOPReward [5] uses frozen Qwen3-VL-8B-Instruct to score task-completion log-probabilities from ordered base-view frames and rewards clipped score differences.The first query of each episode receives zero reward; this signal replaces sparse environment reward for online rollouts.Source: §A.3 · p. 17
• GCR(SC) [12] finetunes a LIV [9] visual backbone using successful-goal contrast and online failure negatives, then applies a goal-similarity difference reward.The reward model is first finetuned on offline demonstrations and then updated online as new policy experience arrives.Source: §A.3 · p. 17

B Additional Mathematical Interpretations

• As κ →∞, HiRE’s potential becomes the maximum positive-buffer similarity minus λ times the maximum negative-buffer similarity; finite κ retains smooth, denser gradients over the interaction history.κ acts as the inverse temperature of the LogSumExp operator; the final implementation uses finite κ.Source: B Additional Mathematical… · p. 17

C.1 Ablation Results

• The ablations compare encoder choice, contrastive weight and negative-buffer capacity, full HiRE against two alternative reward forms, and full HiRE against a positive-buffer-only variant.The reward alternatives directly add the edited potential without PBRS or add raw goal similarity without hindsight editing or PBRS; buffer design removes the negative buffer while retaining attraction toward success states.Source: §C.1 · pp. 17, 19
• Across four tasks, frozen DINOv2 and SigLIP both provide useful dense guidance, but DINOv2 consistently achieves higher success; SigLIP approaches it on Tool_Hang and Three_Piece_Assembly.SigLIP improves more slowly on Stack_Three and especially Threading, where its across-seed variability is larger.Source: Figure 8 · p. 18, §C.1 · p. 18
• On Stack_Three, hyperparameter variants perform similarly; on Three_Piece_Assembly, all configurations improve substantially, with some differences in learning speed and asymptotic performance.The sweep varies contrastive weight λ over {0.1, 0.3, 0.7, 0.9} and tests a negative-buffer capacity of 512.Source: Figure 9 · p. 18, §C.1 · p. 18
• Threading is more sensitive: reducing λ or using buffer capacity 512 slows improvement, while Tool_Hang improves more with λ = 0.1 or 0.3 than with λ = 0.9.The passage attributes these patterns to outdated failures weakening focus on current traps, insufficient repulsion when failures are broadly distributed, and a concentrated positive versus scattered negative distribution on Tool_Hang.Source: §C.1 · p. 18, Figure 9 · p. 18
• On Stack_Three, full HiRE improves faster and achieves higher success than edited-potential addition without PBRS or raw similarity; on Tool_Hang, both alternatives stay near zero while HiRE improves.The comparisons vary reward scaling and replay settings as well as reward form, so they compare resulting configurations rather than isolate a single implementation change.Source: Figure 10 · p. 19, §C.1 · p. 19
• Above 90%, full HiRE reaches a higher final success rate on Tool_Hang than positive-buffer-only, while the variants closely track on Stack_Three.The negative buffer can suppress visually successful near-goal failures that positive-buffer attraction cannot distinguish; its benefit is clearer on the harder, contact-rich Tool_Hang task.Source: Figure 11 · p. 20, §C.1 · p. 20