PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph-Reweighted Behavior Cloning with Human Interventions for Precise Robotic Manipulation

Xiaoman Li, Zhiqiang Yang, Jiayu Hu, Jipeng Kong, Yang Zhang, Jingwei Zhang, Fangzheng Yan, Chunhua Shen, Chenjia Bai

arXiv:2610.20504v1cs.RO

TL;DR

Long-horizon precise manipulation data can mix useful actions with detours, errors, and recoveries, making it difficult to identify which action chunks advance the task. GR-BC uses a task-conditioned reachability graph to estimate progress and weight executed chunks, reaching over 90% success across multiple tasks and VLA backbones after iterative offline training.

  • Problem

    Mixed-quality demonstrations and intervention-assisted rollouts make it difficult to identify task-advancing action chunks, while standard SFT treats every chunk as a positive target.

    Source: §1 · p. 1

  • Method

    GR-BC uses reverse search on a task-conditioned goal-reachability graph to estimate minimal remaining steps and reweight executed action chunks for iterative offline VLA training.

    Source: §1 · pp. 2–3

  • Results

    GR-BC achieves over 90% success on multiple long-horizon precise manipulation tasks with multiple VLA backbones, and has the highest final success rate on all four evaluated tasks.

    Source: §1 · p. 3, §5.2 · p. 8

  • Takeaways & Limitations

    GR-BC prioritizes task-advancing segments and reuses recovery behaviors from failures with a critic-free progress signal for VLA refinement.

    Source: §7 · p. 8

    GR-BC depends on dataset-supported graph reachability, which may be distorted by incorrect neighbors or overly conservative filters and has limited generalization outside collected-data support.

    Source: §6 · p. 8

Abstract

from arXiv · show

Long-horizon precise manipulation produces mixed-quality data for post-training vision-language-action (VLA) models: even successful demonstrations and intervention-assisted rollouts may contain detours, hesitations, failed attempts, and imperfect recoveries. The key challenge is to identify which action chunks actually advance the task, rather than uniformly cloning or discarding imperfect trajectories. We propose Graph-Reweighted Behavior Cloning (GR-BC), an offline iterative post-training framework that builds a task-conditioned goal-reachability graph over demonstrations and iteratively collected rollouts, including both successful and failed episodes with human interventions. Reverse graph search computes minimal remaining steps to task success within the dataset support, and each action chunk is weighted by how much its execution reduces this potential. This consequence-based weighting reuses recoverable segments from failed trajectories, leverages interventions as recovery paths, and down-weights locally regressive chunks even inside successful trajectories. New rollouts are incorporated by appending them to the graph and rerunning search, without training a reward model, action-conditioned critic, or progress predictor. Across four precise manipulation tasks and three VLA backbones, GR-BC improves single-round supervised fine-tuning and reaches 91% success after a few offline iterations, without trajectory comparisons or progress labels. More details and videos are available at: https://gr-bc.github.io

1 Introduction

Mixed-quality demonstrations and intervention-assisted rollouts make action-chunk credit assignment a central challenge: useful recovery segments and locally regressive actions can occur in trajectories with different overall outcomes. GR-BC uses graph-retrieved progress to reweight executed chunks and iteratively incorporates new rollouts.

1 Introduction

  • Mixed-quality demonstrations and intervention-assisted rollouts can contain hesitations, detours, failures, and recoveries, making chunk-level credit assignment important.A trajectory’s final outcome does not establish whether each action chunk advances the task.Source: §1 · p. 1
  • Standard SFT clones every chunk, while progress methods may assume monotonic progress or rely on dense annotations and reward models.Prior graph-search imitation retrieves neighboring actions rather than quantifying the executed chunk’s effect on progress.Source: §1 · p. 2
  • GR-BC estimates minimal remaining steps to success through a task-conditioned reachability graph, then uses the resulting progress to reweight action chunks.It avoids reward models, critics, progress labels, and VLM annotations.Source: §1 · pp. 2–3
  • Its consequence-based weighting reuses progress-making chunks from failed rollouts while suppressing regressive chunks in successful trajectories.The quality signal reflects whether executing a chunk reduces the minimal remaining steps to task completion.Source: §1 · pp. 2–3
  • GR-BC achieves over 90% success on multiple long-horizon precise manipulation tasks with multiple VLA backbones through iterative offline training.Intervention-assisted rollouts update the reachability graph for subsequent rounds.Source: §1 · p. 3

2 Related Work

Prior progress-estimation work uses temporal, goal-proximity, ranking, stage-aware, or model-based signals, while weighted behavior cloning commonly relies on values, rewards, or advantages. GRP instead derives progress from graph reachability and reverse search.

Progress Rewards for Robotic Manipulation

  • Progress methods use temporal order, goal proximity, video ranking, stage or subtask models, and foundation-model or process-reward estimates, often relying on learned signals or temporal assumptions.ARM [13] uses relative advantage labels for progressive, regressive, and stagnant behavior.Source: §2 · p. 3
  • GRP retrieves progress from graph reachability and reverse search rather than learning a reward or progress function.This contrasts with approaches that use progress labels, stage annotations, VLM scoring, or temporal-order assumptions.Source: §2 · p. 3

Weighted Behavior Cloning for VLA Post-training

  • Weighted behavior cloning methods commonly derive action weights from estimated values, advantages, rewards, interventions, or off-policy estimates.GR-BC instead derives weights from graph-based changes in minimal remaining steps, without estimating action values, rewards, or advantages.Source: §2 · p. 3

3 Preliminaries

The paper formulates iterative offline VLA post-training over a mixed-quality pool of trajectories, with trajectory-level outcomes and intervention metadata but chunk-level policy targets. Weighted supervised training retains the standard imitation objective while deriving sample weights from graph-retrieved progress.

Problem Formulation

  • At iteration round r, the mixed-quality data pool contains trajectories labeled by final task outcome and may include human interventions at arbitrary moments.Trajectory metadata includes a language instruction, outcome, and intervention timestamps.Source: §3 · p. 3
  • The state aggregates three RGB views, proprioception, and a binary gripper state, while the policy predicts future action chunks conditioned on state and language instruction.The policy’s observed chunks receive fine-grained quality weights because trajectory-level outcomes do not guarantee localized action optimality.Source: §3 · p. 3

Offline Weighted VLA Post-training

  • GR-BC derives sample-level weights from graph-retrieved progress while retaining the standard supervised VLA imitation objective on observed chunks.The interface recovers standard SFT when the weight equals 1; the underlying loss may be behavioral cloning, denoising, or flow matching.Source: §3 · p. 4

4 Methodology

GR-BC builds a task-conditioned graph over mixed-quality trajectories, estimates remaining steps to success through reverse search, and uses progress consequences to reweight action chunks for iterative offline VLA post-training.

4.1 Goal-reachability Graph Construction

  • GR-BC builds a task-conditioned graph from sparsely sampled trajectory states, retaining terminal and successful states and linking temporally adjacent nodes with frame-based costs.Source: §4.1 · p. 4
  • Cross-trajectory links require multi-view neighbor consensus, gripper-state consistency, and locally normalized similarity, with adaptive costs preserving penalties for bridges.Source: §4.1 · pp. 4–5

4.2 Remaining Steps and Interpolation

  • Reverse Dijkstra search from a virtual success sink retrieves each node’s minimum task-completion cost; unreachable nodes receive infinite potential and zero training weight.Source: §4.2 · p. 5
  • Linear interpolation extends sparse graph-node potentials to dense trajectory states and future-horizon states for chunk-level supervision.Source: §4.2 · p. 5

4.3 Weighted SFT and Iterative Offline Update

  • Chunk weights use normalized progress change and within-chunk regression depth, suppressing unstable chunks even when their endpoint shows net progress.Source: §4.3 · p. 6
  • Each round adds new rollouts to the graph and reruns reverse search to refresh weights for the weighted VLA objective.Source: §4.3 · p. 6

5 Experiments

The experiments assess progress estimation, iterative policy improvement, and graph construction for precise manipulation. GR-BC improves more consistently across post-training rounds, while the ablation indicates that gripper-consistent edges retain recovery reachability while reducing mismatches.

5 Experiments

  • The experiments evaluate GR-BC on four real-world manipulation tasks and compare policy improvement against DAgger, AWR, and RA-BC.The policy-improvement comparison is shown across offline post-training rounds.Source: §5 · p. 7, Figure 6 · p. 7, §5.2 · p. 8

5.1 Progress Estimation Evaluation

  • The progress evaluation compares methods under their intended supervision and data assumptions, using fixed evaluation trajectories without partial-success or per-frame progress labels.It evaluates 30 successful and 30 failed episodes per task, trajectory-level classification, the success–failure progress gap, and 100 manually localized deviation windows.Source: §5.1 · p. 7

5.2 Policy Improvement Evaluation

  • GR-BC achieves the highest final success rate on all four tasks and improves more consistently across offline post-training rounds than DAgger, AWR, and RA-BC.The comparison uses a shared warm-start checkpoint; GR-BC weights action chunks by graph-retrieved progress change.Source: Figure 6 · p. 7, §5.2 · p. 8

5.3 Ablation Study

  • Temporal edges alone cannot connect failed-rollout nodes to the success sink, while gripper-consistent similarity edges preserve similar reachability and remove gripper-inconsistent links.Cross-trajectory similarity edges improve failure reachability but may introduce incompatible shortcuts.Source: §5.3 · p. 8, Table 2 · p. 8

6 Limitations

GR-BC depends on the quality and support of its dataset-derived reachability graph; its path estimates do not explicitly represent physical risk.

6 Limitations

  • Incorrect visual neighbors can create shortcut paths that distort remaining-step estimates, while overly conservative edge filters may miss valid recovery bridges.Gripper-consistency and multi-view checks reduce common mismatches but do not fully certify physical reachability.Source: §6 · p. 8
  • Shorter graph paths are not necessarily safer or more robust because current edge costs do not explicitly account for force or collision risk.Source: §6 · p. 8
  • GR-BC has limited generalization to states outside the support of the collected data.Source: §6 · p. 8

7 Conclusion

GR-BC uses a task-conditioned goal-reachability graph to derive progress estimates for consequence-based action-chunk weighting. The appendix clarifies that the graph uses observed states while the policy objective remains supervised action-chunk prediction.

7 Conclusion

  • GR-BC derives Graph-Retrieved Progress from a task-conditioned goal-reachability graph to estimate minimal remaining steps and weight action chunks by their task-advancing consequence.The approach prioritizes task-advancing segments while reusing recovery behaviors from failures.Source: §7 · p. 8

A Implementation Details and Notation Clarifications

A.1 Real-World Setup and Task Details

The appendix describes four real-robot manipulation tasks, how their mixed-quality demonstrations and intervention-assisted rollouts are collected, and how graphs and progress baselines are constructed and evaluated.

A.1 Real-World Setup and Task Details

  • The four tasks cover assembly, graphics-card insertion, sequential screw tightening, and memory-stick insertion, with task-specific physical success criteria.The evaluations use terminal physical success; episodes can fail despite containing recoverable segments, while successful demonstrations can include hesitations or local regressions.Source: §A.1 · pp. 13–14

A.2 Dataset Composition

  • Each offline iteration adds current-policy real-world rollouts to the expert-demonstration dataset, retaining episodes regardless of outcome.SpaceMouse shared control triggers human takeover on nonzero input and returns control after a fixed quiet timeout, producing interleaved autonomous and corrective segments.Source: §A.2 · p. 14, §A.1 · p. 14

A.3 Graph Construction Details

  • The task-conditioned graph links temporal transitions and visually compatible cross-trajectory states, then reverse Dijkstra search estimates minimum remaining graph cost to success.Similarity links are not executable transitions; only finite node-pair progress is interpolated to dense frames, with other progress set to 0.Source: §A.3 · pp. 14–15

B Progress Evaluation Protocol and Additional Progress Results

  • Progress baselines are evaluated under their intended supervision and data interfaces rather than forced into an identical training protocol.GRP uses terminal success and graph connectivity without dense progress labels, subtask labels, or trajectory comparisons; the baseline evaluations use fixed sets of 30 successful and 30 failed episodes per task.Source: §B.1 · p. 16

B.1 Baseline Implementation and Supervision

  • The baseline implementations use distinct inputs and supervision, including language-guided rewards, stage-aware progress, before/after comparisons, and trajectory-preference signals.ReWiND, SARM, Robo-Dopamine, and RoboMeter are implemented per task; Robo-Dopamine adaptation uses successful trajectories, while RoboMeter uses terminal-success and preference signals.Source: §B.1 · pp. 16–17

B.2 Metric Definitions

  • The evaluation metrics assess terminal success–failure progress separation, sensitivity to annotated local deviations, and episode classification of successful and failed rollouts.Deviation windows are annotated from raw video before scoring, and the same frozen annotations and threshold are used across methods and tasks.Source: §B.2 · pp. 17–18

B.3 Demo-Only Zero-Shot Progress Evaluation

This evaluation tests whether GRP can score unseen rollouts using a graph built from successful demonstrations alone. It reports a large terminal-progress gap between held-out successful and failed rollouts, while the broader policy study uses X-VLA.

B.3 Demo-Only Zero-Shot Progress Evaluation

  • The demo-only diagnostic reports a large terminal-progress gap between held-out successful and failed rollouts.All failed trajectories receive zero terminal progress, while most held-out successful trajectories reach high terminal progress.Source: §B.3 · p. 18, Table 9 · p. 19
  • Held-out rollouts are inserted one at a time as queries and removed before the next evaluation, against a graph built from successful demonstrations only.The protocol tests query-time progress before unseen rollouts are added to the training pool.Source: §B.3 · p. 18, Table 9 · p. 19

C.1 VLA Training Details

  • The main iterative policy-learning study uses X-VLA, while the separate compatibility study also evaluates SmolVLA and π0.5.The reported demo-only progress result is a diagnostic split, not the policy-learning comparison.Source: §C.1 · p. 19, C Policy Improvement Evaluation… · p. 19

C.2 Single-task Backbone Compatibility Study on Graphics Card

A controlled Graphics Card study compares GR-BC with a DAgger-style uniform aggregation baseline across three VLA backbones, finding improvement across all three.

C.2 Single-task Backbone Compatibility Study on Graphics Card

  • The comparison holds graph construction, GRP computation, transition weights, data split, and evaluation poses fixed while changing the VLA backbone and its supervised objective.For SmolVLA's 50-frame action chunk, the study applies weights computed with a fixed 30-frame GRP consequence horizon to the full chunk.Source: §C.2 · p. 19
  • GR-BC improves over the DAgger-style baseline across X-VLA, SmolVLA, and π0.5 on the Graphics Card task.The improvement is especially clear after the second and third post-training rounds; the study supports backbone compatibility in this controlled single-task setting.Source: Figure 11 · p. 20, §C.2 · p. 20
  • The evidence is limited to a controlled single-task compatibility study, with broader architecture and task coverage left for future evaluation.The method changes only the sample-level multiplier on the original supervised VLA loss in this setting.Source: §C.2 · p. 20

C.3 Policy Baseline Training Details

The appendix specifies policy baselines and implementation details for comparing uniform aggregation, learned-reward weighting, and critic-derived weighting with graph-reweighted cloning.

C.3 Policy Baseline Training Details

  • The DAgger-style baseline uniformly aggregates executed policy chunks and human intervention chunks for unweighted supervised training.It does not retroactively query an expert for every learner-visited autonomous state, and it treats all aggregated transitions as equally positive supervision.Source: §C.3 · p. 20
  • RA-BC trains a single-stage progress model on successful demonstrations and uses progress changes to weight policy data.Its continuous reward signal lacks graph-based topological granularity and can miss local errors such as slight misalignments or transient intra-chunk regressions.Source: §C.3 · p. 21
  • AWR first trains an offline action-chunk critic from the shared supervised checkpoint, then uses critic-derived chunk advantages for weighted behavior cloning.The critic is frozen for the subsequent weighted cloning update.Source: §C.3 · p. 21

D Graph Diagnostics, Representation Ablations, and Hyperparameter Analysis

D.1 Hyperparameter Analysis

The appendix examines graph connectivity and chunk-weight sensitivity, compares visual encoders, and illustrates multi-mode success and paths from failed states. These are graph-level diagnostics, not an exhaustive policy-retraining sweep.

D.1 Hyperparameter Analysis

  • With k = 15 as the default cost–coverage tradeoff, increasing δmin raises the zero-weight ratio, while larger βw and γ sharpen bounded weight separation.The diagnostics also report high failure-rollout reachability for k = 10–15.Source: §D.1 · p. 22

D.2 Encoder Comparison: R3M, MCR, and V-JEPA2.1

  • MCR yields a more conservative graph with the lowest total variation; R3M is competitive on mean regression and encoding speed, while both V-JEPA2.1 encoders are slower and produce denser, less smooth graphs.MCR is selected for its conservative graph and acceptable encoding cost; the comparison uses the same graph-construction checks.Source: §D.2 · p. 23, Table 14 · p. 23, Figure 12 · p. 23

D.3 Multiple Successful Terminal Clusters

  • Assembly Model has two visually and physically distinct successful terminal groups, and the graph connects them to a virtual sink so progress targets the nearest reachable success mode.The feature-space visualization and representative camera frames show the two groups.Source: §D.3 · p. 23, Figure 13 · p. 24

D.4 Shortest-Path Visualization

  • A recovered path routes a failed state through temporal transitions and cross-episode similarity bridges to a successful terminal, reusing recovery segments across episodes and success modes.Figure 14 labels the queried state and recovered goal as Start and Goal; T marks temporal transitions and S similarity bridges.Source: §D.4 · p. 24, Figure 14 · p. 24

E Additional Progress Visualization

Related papers

Loading 2610.20504v1…