PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Sinjae Kang, Dong Kyu Shin, Jaehong Kim, Kamin Lee, Hyoeun Kim, Kaixin Wang, Li Zhao, Seonghyeon Ye, Kimin Lee

arXiv:2610.37393v1cs.ROcs.AIcs.LG

TL;DR

Robot policies trained on successful demonstrations do not distinguish how tasks are executed, despite deployment needs for controllable qualities such as speed and smoothness. QCP conditions behavior on quantized quality labels and expands their range through iterative collection and hindsight relabeling of successful rollouts. Across benchmarks, quality and task success improve over iterations, with real-robot tests showing task-specific gains.

  • Problem

    Successful demonstrations are often treated equally, leaving policies without a way to steer execution toward deployment-specific quality attributes such as speed or smoothness.

    Source: §1 · p. 1

  • Method

    QCP conditions actions on quantized quality labels and iteratively adds successful rollouts relabeled by achieved quality before retraining.

    Source: §1 · pp. 1–2

  • Results

    QCP with iterative hindsight relabeling improves target quality and task success across iterations on Robomimic and RoboEval, and real-robot tests report gains in conveyor throughput and wine-carrying success.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Real-robot experiments demonstrate that QCP can be directed toward the execution quality desired for a given task.

    Source: §6 · p. 10

    The possible benefit of reinitialization for avoiding loss of plasticity and primacy bias was not studied directly.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.

1 INTRODUCTION

Binary success alone does not distinguish execution behaviors such as speed and smoothness, although deployment tasks may require different quality attributes. QCP conditions actions on quality labels, and iterative hindsight relabeling expands the controllable range using successful online rollouts.

1 INTRODUCTION

  • BC treats successful trajectories equally, so it reproduces mixed execution behaviors without letting users steer toward task-specific needs such as speed or stability.The paper contrasts conveyor operations, which prioritize speed, with carrying wine glasses, which prioritizes transport stability.Source: §1 · p. 1
  • Static demonstrations typically cover a bounded quality range, motivating online data collection and policy updates to extend controllability beyond that range.The target regimes may include maximum speed or ideal smoothness.Source: §1 · p. 1
  • QCP makes execution quality an explicit conditioning input, using measured quality attributes and quantized labels to guide behavior at inference time.Users specify a target label to steer execution, such as the highest speed label for conveyor operations.Source: §1 · pp. 1–2
  • Iterative hindsight relabeling collects rollouts under the highest quality label, retains successful trajectories, relabels them by achieved quality, and retrains the policy.The loop recomputes labels for accumulated trajectories, progressively shifting the highest label toward higher quality.Source: §1 · p. 2
  • QCP with iterative hindsight relabeling improves target quality and task success across iterations on Robomimic and RoboEval, outperforming quality-filtered BC and RL baselines.The reported improvement covers both individual and joint quality control; real-world tests also report increased conveyor throughput and improved wine-carrying success under task-specific quality control.Source: §1 · p. 2

2 RELATED WORK

Prior work represents task-execution styles through learned trajectory representations or explicit labels, while some methods balance style alignment with task performance. QCP pairs quantized quality conditioning with iterative hindsight relabeling to improve execution quality and task success together.

2 RELATED WORK

  • QCP combines quantized quality conditioning with iterative hindsight relabeling, and experiments report simultaneous gains in execution quality and task success.The related-work discussion frames these gains as overcoming the trade-off between task performance and requested style.Source: §2 · p. 2
  • Freeform Preference Learning uses human trajectory comparisons for online quality improvement, whereas QCP uses automatically computed quality labels and successful rollouts without additional human annotation.Quark uses a related highest-reward sampling and relabeling loop in text generation; QCP applies a similar loop to robot execution quality.Source: §2 · p. 3

3 METHOD

QCP conditions a policy on quantized execution-quality labels, then iteratively collects successful rollouts and relabels accumulated trajectories to extend the controllable range. The method uses these labels in quality-conditioned policy training and updates the dataset's quality ranges as higher-scoring rollouts accumulate.

3.1 QUALITY-CONTROLLABLE POLICY

  • Trajectory scores are quantized into B equal-frequency bins, and each trajectory receives the bin index corresponding to its empirical score quantile.The score function measures a quality axis; examples include execution speed, smoothness, and efficiency.Source: §3.1 · p. 3
  • Training samples pair observation histories and future action chunks with each trajectory's quality label, which conditions the behavior-cloning policy.For vision-language-action models, the quality label can be inserted into the language prompt as text, allowing quality conditioning without architectural changes.Source: §3.1 · pp. 3–4
  • At deployment, users choose a target quality bin, and the policy generates actions conditioned on that requested bin.The reported evaluation generally requests the highest bin, while testing all bins finds behavior shifts toward the requested quality range.Source: §3.1 · p. 4

3 METHOD

  • The method alternates between rollout collection, hindsight relabeling, and policy updates to push controllable quality beyond the range present in demonstrations.Rollouts conditioned on the highest bin can exceed the best demonstrations, enabling the loop to expand the range over iterations.Source: §3.2 · p. 4

3.2 ITERATIVE HINDSIGHT RELABELING

  • Each iteration collects rollouts at the highest bin, keeps successful ones, relabels them by achieved score, recomputes labels for all trajectories, and retrains.Accumulating higher-scoring rollouts shifts bin boundaries upward, while original demonstrations preserve the lower-quality range; policies are reinitialized for retraining because relabeling changes the behavior associated with labels.Source: §3.2 · pp. 4–5
  • Including failed rollouts as additional low-quality data degraded performance in the authors' experiments.The iterative dataset therefore adds successful rollouts rather than retaining failed ones.Source: §3.2 · p. 4

4 EXPERIMENTS

The experiments test whether QCP can improve execution quality while retaining task success, expand controllable quality beyond demonstrations, jointly control multiple attributes, and deliver gains on real robots.

4.1 EXPERIMENTAL SETUP

  • Experiments span four simulation tasks and two real-robot tasks, comparing QCP with BC, style-learning, and online-RL baselines under fixed quality-bin rollout collection.The possible benefit of reinitialization against plasticity loss and primacy bias is not directly studied.Source: §4.1 · p. 5

4.2 ITERATIVE IMPROVEMENT OVER QUALITY AND SUCCESS RATE

  • Across tasks, QCP improves target quality and task success together; at the final iteration it also achieves higher quality than DSRL and EXPO.Figure 2 compares success against quality across iterations; Table 1 covers final-iteration comparisons with RL baselines.Source: §4.2 · p. 6, Figure 2 · p. 6, Table 1 · p. 7

4.3 QUALITY CONTROL AND RANGE EXPANSION

  • Across iterations, requested-bin ordering persists while quality shifts upward; successful-rollout distributions extend beyond demonstrations, with normalized Wasserstein distance above 1.5 for smoothness and joint path length.The distance is normalized by the initial demonstrations’ interquartile range.Source: §4.3 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8

4.4 MULTI-ATTRIBUTE QUALITY CONTROL AND IMPROVEMENT

  • On Square, QCP shifts speed and lift amplitude in all four requested combinations; across four RoboEval tasks, it improves most of six attributes over initial BC and BCacc.The two-attribute policies reach quality levels close to policies trained for each attribute alone.Source: §4.4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.5 QUALITY CONTROL FOR REAL ROBOT TASKS

  • Wine-serving tilt falls 27.0% (0.0660 to 0.0482 rad) as success rises from 30.7% to 42.0%; conveyor speed rises 36.6% and throughput 59.3% versus BC.The conveyor measurements are end-effector speed (188.6 to 257.6 mm/s) and throughput (8.2 to 13.0 cans per minute).Source: §4.5 · p. 9, Figure 7 · p. 9

5 ANALYSIS

The analysis finds that iterative policy updates on relabeled data—not additional sampling from a frozen policy—shift rollouts toward quality levels the earlier policy could not produce. It also finds that quantized labels outperform continuous score conditioning on Square, while finer quantization trades task success for a small speed gain.

5.1 EFFECT OF QUALITY CONDITIONING ON ROLLOUT DISTRIBUTIONS

  • None of 1,800 successful-or-unsuccessful rollouts from the frozen iteration-0 policy reached the final iteration’s median smoothness, unlike iterative retraining with 300 rollouts at each of iterations 0–5.The comparison uses equal rollout budgets under the best quality bin; the authors attribute the shift to updates on relabeled data.Source: §5.1 · pp. 9–10, Figure 8 · p. 10

5.2 CONTINUOUS QUALITY SCORES VERSUS QUANTIZED QUALITY LABELS

  • With four quantile bins, roughly the best quarter of training trajectories share each best label, giving the policy multiple examples of requested behavior.The authors suggest this can provide a more stable training signal than a continuous score represented by only a few examples.Source: §5.2 · p. 10
  • On Square, quantized labels execute faster than a fixed continuous target at every iteration, and four bins achieve higher task success from iteration 1 onward.Increasing bins beyond four brings a small speed gain at a growing cost in task success.Source: §5.2 · p. 10

6 CONCLUSION

QCP uses quantized quality labels to control execution attributes individually or jointly, and iterative hindsight relabeling extends its controllable range beyond the demonstrations. Across simulation and real-world experiments, it improves execution quality and task success while retaining quality control across iterations.

6 CONCLUSION

  • QCP uses quantized quality labels to control individual execution attributes and their combinations at inference time.Iterative hindsight relabeling extends the controllable range beyond demonstrations through online rollouts and policy updates, without additional human annotation.Source: §6 · p. 10
  • Experiments on Robomimic and RoboEval show improved execution quality and task success while retaining quality control across iterations.Source: §6 · p. 10
  • Real-robot experiments demonstrate that QCP can be directed to pursue the execution quality desired for a given task.Source: §6 · p. 10

A QUALITY SCORE FUNCTIONS

Quality scores are measured over complete trajectories and transformed so the highest bin always denotes the preferred direction. The appendix specifies the attribute measures used in simulation and real-robot experiments.

A QUALITY SCORE FUNCTIONS

  • Each attribute is measured over a complete trajectory; scores are inverted or reciprocated when needed so the highest bin denotes the preferred direction.Bin edges are dataset quantiles of successful trajectories, recomputed each iteration; joint-conditioning attributes are quantized independently.Source: A QUALITY SCORE FUNCTIONS · p. 14
  • Robomimic measures speed as steps to success, action and end-effector smoothness through normalized jerk, and translation path length.Speed is preferred lower in the main experiments, while the supplied definitions mark smoothness as lower-is-better and end-effector displacement per step as higher-is-better.Source: A QUALITY SCORE FUNCTIONS · p. 14
  • RoboEval measures completion time, joint and Cartesian path lengths, orientation path length, jerk, and bimanual differences.The listed lower-is-better measures include completion time, path lengths, orientation path length, jerk, and bimanual gripper vertical difference.Source: A QUALITY SCORE FUNCTIONS · p. 14
  • Real-robot quality measures include end-effector speed over successful grasp segments and wine-serving tilt angle over successful episodes.End-effector speed is higher-is-better; tilt is lower-is-better and reflects the uprightness of glasses when the tray is rigidly held.Source: A QUALITY SCORE FUNCTIONS · p. 15

B TRAINING AND EVALUATION DETAILS

The experiments use a shared quality-conditioned policy and retrain it from scratch at each iteration, with task-specific real-robot data and evaluation settings. Baselines vary in how they form training sets or optimize task and quality objectives.

B TRAINING AND EVALUATION DETAILS

  • Policies use a shared flow architecture, adaptive quartile labels, and best-bin rollout collection, with policies retrained from scratch at every iteration.The quality embedding is concatenated with encoded observation history; training uses AdamW, and actions are sampled with nine Euler steps.Source: B TRAINING AND EVALUATION DETAILS · p. 15, Table 2 · p. 15

C BASELINE DETAILS

B TRAINING AND EVALUATION DETAILS

  • Real-robot tests use a Franka arm; wine-serving success requires reaching the target table without a fallen glass, while conveyor throughput is success fraction times arrival rate.Wine serving starts with 50 successful episodes from 382 attempts; conveyor data comprise 400 successful grasp segments from 936, with 20 rollouts of 20 cans collected per iteration.Source: B TRAINING AND EVALUATION DETAILS · pp. 15–16

C BASELINE DETAILS

  • BCacc appends successful unconditioned-policy rollouts, whereas BCfilter repeatedly selects high-scoring trajectories from the accumulated dataset.BCfilter keeps the best 10% in simulation and 20% in real-robot experiments; it reapplies the filter to the full accumulated set.Source: C BASELINE DETAILS · p. 16
  • SORL and SCIQL first learn value and Q functions, then train a quality-conditioned actor using different advantage-based sample weights.SORL uses task advantage; SCIQL makes style advantage primary and adds task advantage only for actions matching the requested style.Source: C BASELINE DETAILS · p. 16
  • DSRL optimizes a frozen policy’s initial noise, while EXPO learns an edit policy and critic ensemble on top of a frozen behavior-cloning policy.Both start from a flow-matching policy trained on demonstrations and act on chunks of actions.Source: C BASELINE DETAILS · p. 16
  • The RL reward combines task success with a weighted per-step quality signal; speed rewards shorter episodes, while motion penalties measure deviations from demonstration-typical movement.For speed, the episode quality return is negative episode length divided by the demonstrations’ average length; motion penalties are clipped, and task reference constants are computed from successful demonstrations and kept fixed.Source: C BASELINE DETAILS · p. 17

D EFFECT OF THE QUALITY COEFFICIENT

The coefficient sweep selects RL-baseline settings that balance success and quality on Robomimic Square, then uses those settings across tasks. Table 3 reports the sweep and identifies the settings used for the main results.

D EFFECT OF THE QUALITY COEFFICIENT

  • β = 3 for SORL and SCIQL and λ = 4 for DSRL and EXPO were selected for a good balance of success rate and quality, then used across all tasks.The sweep used three seeds per setting and the same endpoint as Table 1.Source: D EFFECT OF THE QUALITY COEFFICIENT · p. 17, Table 3 · p. 18
  • Table 3 reports success rate and successful-episode quality across advantage temperatures β and quality-reward weights λ on Robomimic Square.Quality is measured as steps to success for Speed and action jerk for Smoothness; lower is better.Source: Table 3 · p. 18

E MULTI-QUALITY CONDITIONING EXPERIMENT DETAILS

The multi-quality experiments test joint control of two attributes on Square and six attributes across RoboEval tasks. They compare requested combinations with single-attribute references or unconditioned policies.

E MULTI-QUALITY CONDITIONING EXPERIMENT DETAILS

  • Joint speed and lift-amplitude commands on Square reach the corresponding single-label reference means on both coordinates within a few percent, with lower success for slow commands.Four corner combinations are tested, each with its own policy lineage and rollout command.Source: E MULTI-QUALITY CONDITIONING… · p. 18
  • The RoboEval experiment conditions on six task-selected attributes across Lift Pot, Stack Two Blocks, Handover, and Valve, comparing iteration-6 QCP with initial BC and BCacc.Two attributes are dropped after a preliminary run because they were unaffected by conditioning or conflicted with the task; Table 4 lists the attributes and reasons, and Table 5 gives Figure 6’s numerical values.Source: E MULTI-QUALITY CONDITIONING… · p. 18

F ANALYSIS EXPERIMENT DETAILS

The analyses separate gains from iterative relabeling from those of quality conditioning alone, examine quality-label representations, and compare shorter-segment labeling approaches. Results indicate that relabeling extends the reachable smoothness range, while label granularity trades success against speed.

F ANALYSIS EXPERIMENT DETAILS

  • Figure 10 compares complete-trajectory, segment, hybrid, and 10%-filtered BC labels by success rate and mean EEF displacement per step over iterations on Square and Transport.Transport 10% filtered BC has no successful rollouts, so its quality values are omitted.Source: Figure 10 · p. 19
  • 73.9% success and median action jerk 0.0098 at iteration 5 contrast with the frozen policy’s 54.5% and 0.0335, showing the loop reaches smoother behavior absent from the iteration-0 policy.The smoothest frozen rollout’s jerk, 0.0171, is above the 75th percentile of the iteration-5 distribution; the comparison uses equal rollout budgets.Source: F ANALYSIS EXPERIMENT DETAILS · pp. 18, 20
  • At iteration 6, four-, eight-, sixteen-, and thirty-two-bin variants reach 70.3%, 63.2%, 57.7%, and 28.1% success with 139, 120, 116, and 112 steps, respectively.More bins make executions slightly faster but cost success, sharply beyond sixteen bins; the continuous-target comparison does not rule out continuous conditioning generally because it also changes target selection.Source: F ANALYSIS EXPERIMENT DETAILS · p. 20

G QUALITY LABELS FOR SHORTER SEGMENTS

The comparison favors one quality label per trajectory over segment-level or hybrid labels. Segment scores remain entangled with task stage and can improve locally without advancing task completion.

G QUALITY LABELS FOR SHORTER SEGMENTS

  • The comparison tests trajectory labels, one label per decision step from a ±8-step score window, and hybrid conditioning on both labels.For segment labels, quartile boundaries are computed separately across ten normalized trajectory-time intervals, and the best quartile is requested at every step.Source: G QUALITY LABELS FOR SHORTER SEGMENTS · p. 20
  • At iteration 9 on Square, trajectory labels reach 84.3% success at 6.08 mm per step, versus 48.0% at 4.80 for segment and 71.0% at 5.07 for hybrid.Figure 10 compares end-effector speed over ten iterations, using 10% filtered BC as a reference.Source: G QUALITY LABELS FOR SHORTER SEGMENTS · p. 20
  • On Transport at iteration 9, trajectory labels reach 13.3% success, versus 3.7% for segment and 4.0% for hybrid, at similar speeds of 5.2 to 5.7 mm per step.The comparison uses one seed, 100k training steps, and 300 rollouts per iteration.Source: G QUALITY LABELS FOR SHORTER SEGMENTS · p. 20
  • Segment scores reflect task stage as well as execution quality, and a policy can improve a local score without progressing toward completion.The authors suggest that finer quality signals may require separating quality from task progress.Source: G QUALITY LABELS FOR SHORTER SEGMENTS · p. 20

Related papers

Loading 2610.37393v1…