PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Absorb Corrections: Critic-Weighted Residual Flow for Vision-Based Insertion

Yiqing Yuan, Zhi Li, Jinrong Yang, Mingshan Sun, Mei Yu Zhi, Yong Zhao, Liangliang Ren, Hui Cheng

arXiv:2610.29117v1cs.RO

TL;DR

Vision-based insertion requires correcting alignment errors while retaining useful behavior from demonstrations, but limited demonstrations may not cover deployment states. ACT–Flow adapts an ACT prior alongside a residual flow policy, absorbing successful actions and combining critic-weighted candidate fitting with successful-action supervision. In a matched-checkpoint diagnostic, ACT’s mean success rises from 16% to 76% when the residual is disabled.

  • Problem

    Limited demonstrations may not cover deployment states, while successful corrections must remain useful as the action prior changes.

    Source: §I · p. 1

  • Method

    ACT–Flow updates ACT from successful executions, recomputes residual targets against the current prior, and combines critic-weighted candidate fitting with successful-action flow-matching supervision.

    Source: §I · pp. 1–2

  • Results

    ACT within Full improves from 16% to 76% mean success when the residual is disabled at the same combined-policy checkpoint without retraining.

    Source: C. Ablation Studies · p. 7

  • Takeaways & Limitations

    Across five real-world tasks, the framework demonstrates early adaptation and behavior absorption, while residuals retain additional benefits.

    Source: §VI · p. 8

Abstract

from arXiv · show

Vision-based insertion requires correcting alignment errors while retaining useful behavior learned from demonstrations. We present ACT--Flow, an online learning framework that couples an adaptive Action Chunking Transformer (ACT) with a conditional residual flow policy. Following anchor-guided warmup, ACT absorbs successful executed actions through behavioral cloning, while residual targets are recomputed relative to the updated prior. A chunk-level critic learns from successful and failed interactions and evaluates final composed actions. Its estimates weight generated residual candidates for flow matching, complemented by supervision from successful real executions. Candidate scoring is confined to training; deployment uses RGB observations and samples one residual per action chunk. We evaluate the framework on power-plug, USB, and key insertion, together with cable routing and needle threading. Results show a five-task mean success rate of 66% at 20,000 post-initialization interaction steps, exceeding the strongest baseline by 18 percentage points at this checkpoint. Evaluations at matched checkpoints with the residual disabled show ACT's mean success increasing from 16% to 76%. Component ablations further support the complementary roles of behavior absorption, value weighting, and successful-action supervision.

I. INTRODUCTION

Vision-based insertion needs policies that can correct errors beyond their demonstrations while retaining successful corrections in the learned behavior. ACT–Flow pairs an adaptive ACT prior with a residual flow policy, updating the prior from successful actions and using critic guidance alongside successful-action supervision for residual learning.

I. INTRODUCTION

  • Limited demonstrations may not cover deployment states, and deviations from the training distribution can compound during insertion.Small position or orientation errors, or target movement after approach, can cause insertion failure.Source: §I · p. 1
  • Existing corrective approaches draw on demonstrations or human intervention, while residual methods can leave successful corrections outside the prior itself.Demonstration guidance depends on reference behavior, and human corrections require operator effort.Source: §I · p. 1
  • The framework initializes ACT and residual flow through anchor-guided demonstrations and correction warmup, then uses successful and failed experience to guide residual learning.Successful executed actions also update ACT to absorb corrected behavior.Source: Figure 1 · p. 1, §I · p. 2
  • The paper evaluates three insertion tasks plus cable routing and needle threading, with comparisons and ablations examining adaptation, behavior absorption, and residual supervision.It also reports diagnostics of base-policy improvement, residual contribution, and critic ranking quality.Source: §I · pp. 1–2
  • ACT–Flow updates ACT from successful executed actions and recomputes residual targets against the updated prior, keeping correction supervision consistent as the base policy changes.Residual learning combines critic-weighted fitting of generated corrections with flow-matching supervision from successful executions.Source: §I · pp. 1–2

A. Demonstrations and Corrective Interaction

The paper situates ACT–Flow among demonstration- and human-guided correction, residual adaptation, and value-guided generative learning. Its formulation treats progress and terminal success as feedback within a finite interaction budget.

A. Demonstrations and Corrective Interaction

B. Residual Adaptation and Prior Updating

  • Updating ACT can absorb successful behavior, but changes the residual needed to reproduce earlier actions, so residual targets are recomputed against the current prior.Executed successful actions remain targets even as their decomposition changes.Source: B. Residual Adaptation and Prior… · p. 2

C. Value-Guided Generative Policy Learning

  • Critic rankings guide generated-candidate fitting but cannot replace successful-action targets, so ACT–Flow combines critic-weighted flow matching with flow matching on successful executions.The objectives both reference the current ACT prior.Source: C. Value-Guided Generative Policy… · p. 2

III. PROBLEM FORMULATION

  • The formulation considers RGB-based insertion in which a policy outputs normalized action chunks, and interaction is evaluated by expected discounted return under a finite budget.Feedback combines progress differences with terminal success; success adds 100 once, while progress increases or decreases provide positive or negative feedback.Source: §III · p. 2

IV. METHOD

ACT–Flow composes an ACT prior with a conditional residual flow policy to produce legal action chunks. The same composition supports collection, scoring, and value targets, while training may score multiple residual candidates and deployment samples one residual per chunk.

IV. METHOD

  • The executed action combines the prior and residual, with fixed scaling and clipping producing the legal composed chunk.The composition is shared by warmup, collection, scoring, and value targets; a zero residual recovers ACT.Source: §IV · p. 3
  • ACT–Flow uses an ACT prior and conditional residual flow whose base, residual, and noise are action chunks, with raw residuals unrestricted.Multiple residual candidates are generated for critic-weighted flow updates, but each interaction step uses a single residual sample.Source: §IV · p. 3

A. Anchor Warmup and Behavior Absorption

Anchor warmup initializes the policy before online updates alternate behavior absorption and residual relabeling.

A. Anchor Warmup and Behavior Absorption

  • After anchor warmup initializes the policy, online updates alternate behavior absorption with residual relabeling.The passage states the sequence but does not specify the update mechanics.Source: A. Anchor Warmup and Behavior Absorption · p. 3

1) Anchor-guided warmup:

Anchor-guided warmup initializes ACT and the residual policy; online updates then absorb successful executed actions into ACT and recompute residual targets against the changing prior.

1) Anchor-guided warmup:

  • Anchors specify stage-relative end-effector poses, and varied stage-local demonstrations initialize a frozen ACT prior whose inputs exclude privileged anchor geometry and target information.The warmup residual is derived from successful correction chunks; anchor guidance ends after initialization, but demonstrations remain available for rehearsal.Source: 1) Anchor-guided warmup · p. 3

2) Online behavior absorption:

  • Each ACT update mixes anchor rehearsal with successful online executions, so executed combined-policy actions become behavior-cloning targets while ACT receives only BC gradients.ACT retains reconstruction and latent regularization, and masks unexecuted suffixes.Source: 2) Online behavior absorption · p. 3

3) Residual relabeling and replay:

  • After ACT changes, successful residual targets are recomputed so composing the updated base with the new residual still reconstructs the fixed executed action.Historical residual labels are outdated relative to the new base, so batches hold updated ACT fixed for both Flow objectives.Source: 3) Residual relabeling and replay · p. 4
  • Replay separates critic transitions, including failures and executed prefixes, from fully executed successful chunks used as Flow targets.A successful chunk truncated at termination can train ACT on its prefix and the critic on the observed transition, but does not provide an incomplete Flow target.Source: 3) Residual relabeling and replay · p. 4

B. Critic-Weighted Residual Flow Learning

Residual Flow learning combines critic-weighted generated corrections with supervision from successful executions, while evaluating composed actions and fitting candidates separately.

B. Critic-Weighted Residual Flow Learning

  • Flow matching interpolates Gaussian noise and target residuals, then fits a conditional vector field to their velocity using the observation and current base chunk.At sampling time, integration over flow time generates a residual; K controls numerical integration cost.Source: B. Critic-Weighted Residual Flow… · p. 4
  • A frozen chunk critic scores composed, clipped candidate actions, and group-normalized weights give higher-value corrections more regression weight.Candidates are generated from a frozen snapshot; scoring adds no robot executions or replay transitions.Source: B. Critic-Weighted Residual Flow… · p. 4
  • Successful-action Flow matching anchors residual learning to observed task-completing actions, while critic-weighted matching emphasizes generated candidate corrections at successful and failed states.Failed attempts provide value evidence and candidate contexts, not successful-action imitation targets; both data views are needed for joint updates.Source: B. Critic-Weighted Residual Flow… · p. 4
  • Candidate losses are averaged rather than their actions, avoiding an intermediate target that may match neither distinct corrective motion.Only the vector field receives gradients during the Flow phase; candidate snapshots refresh after each step while ACT and Qscore remain fixed.Source: B. Critic-Weighted Residual Flow… · p. 4

C. Progress Estimation and Chunk Value Learning

The framework estimates progress from RGB images and task descriptions, then trains a chunk critic to value composed actions using observed rewards and bootstrapped returns.

C. Progress Estimation and Chunk Value Learning

  • A ReWiND-based progress model predicts progress from one RGB image and task description, uses terminal success labels, and is frozen during online learning.Qwen3-VL-8B supplies terminal labels with 97% classification accuracy against human annotations.Source: C. Progress Estimation and Chunk Value… · p. 4
  • The chunk critic evaluates the complete composed action, and its estimates represent discounted task return.The critic trains on valid successful and failed transitions, using observations and planned legal chunks.Source: C. Progress Estimation and Chunk Value… · pp. 4–5
  • Chunk returns account for the observed execution duration, including the terminal bonus once at its control-step discount when execution ends early.The bootstrap mask distinguishes true termination from permitted bootstrapping; target values use the mean of two target heads.Source: C. Progress Estimation and Chunk Value… · p. 5
  • Each learning round updates ACT, trains the critic with updated ACT and pre-update Flow, then optimizes Flow using fixed critic scores and current-base residual targets.The update order aligns residual targets and critic evaluation with the current prior while keeping scoring fixed during each Flow update.Source: C. Progress Estimation and Chunk Value… · p. 5

V. EXPERIMENTS

The experiments evaluate ACT–Flow on five real-world tasks, including three insertion tasks, through comparisons and component-focused analyses.

V. EXPERIMENTS

  • The five-task real-world evaluation examines online learning efficiency, successful-behavior absorption, and critic-guided residual learning.Comparisons assess overall performance, while ablations and paired diagnostics examine the contributions of the components.Source: §V · p. 5

A. Environmental Setup

Across five tasks, ACT–Flow reaches a 66% mean success rate at 20k online steps, 18 percentage points above the strongest baseline at that checkpoint. By 60k steps, its mean reaches 87%, close to its 88% endpoint at 120k.

A. Environmental Setup

  • The protocol uses 30 initialization episodes, then counts online steps from zero; learning curves extend to 120k real environment steps.Five baselines are evaluated on the same tasks, with online steps counted after initialization.Source: A. Environmental Setup · p. 5
  • Success is judged by humans across 100 episodes per task per seed at each checkpoint; learning curves report means and standard deviations over three seeds.Final success rate averages the last three predetermined checkpoints, while NAUC measures performance throughout adaptation using all evaluations.Source: A. Environmental Setup · p. 5
  • Figure 3 shows a six-axis arm, gripper, task-specific flanges, and close-view camera across power-plug, USB, key, cable-routing, and needle-threading tasks.The setup also includes work and ring lights; baseline data collection uses controllers.Source: A. Environmental Setup · p. 5, Figure 3 · p. 6

B. Comparative Experiments

  • At 20k online steps, ACT–Flow’s five-task mean is 66%, versus 48% for FTC, the strongest baseline, an 18-percentage-point difference.ACT–Flow has the highest mean on each task at this checkpoint.Source: B. Comparative Experiments · pp. 5–6
  • By 60k steps, ACT–Flow reaches a mean SR of 87%, close to its 88% endpoint at 120k, while its gap to FTC narrows from 18 points at 20k to 9 at 40k and 4 at 60k.Final task rates are 86%, 99%, 99%, 87%, and 70%, respectively.Source: B. Comparative Experiments · p. 6
  • Figure 4 depicts key insertion and cable placement, plus power-plug re-localization after board displacement and pose following as the board is lowered.The plug sequences show redirection toward the moved socket and tracking of its changing position and orientation.Source: Figure 4 · p. 6, B. Comparative Experiments · p. 6
  • Needle threading reaches 70% mean success, exceeding FTC’s 60% by ten points, with standard deviations of 12 and 13 points, respectively.Several baselines approach similar late performance on insertion tasks, while needle threading retains a larger separation.Source: B. Comparative Experiments · p. 6

C. Ablation Studies

The ablations support complementary contributions from value-guided candidate fitting, successful-action supervision, and updating ACT with successful corrections. Diagnostics show that ACT gains capability as the residual is disabled, while residual corrections remain useful and all-buffer critic weighting better distinguishes and favors higher-return candidates.

Component effects

  • 88 mean SR and 79 NAUC for Full; Uniform W. gives 82 SR and 67 NAUC, while w/o SFM gives 78 SR and 67 NAUC.Frozen ACT gives 84 SR and 73 NAUC; Succ-only Q gives 78 SR and 64 NAUC, the largest component-wise NAUC drop.Source: C. Ablation Studies · p. 6, Table I · p. 7
  • ACT-only reaches 69 mean SR and 55 NAUC, while Flow-only reaches 85 SR and 65 NAUC, fourteen points below Full on NAUC.Because the comparison also changes action parameterization, it assesses the overall prior–residual structure rather than a single loss term.Source: C. Ablation Studies · pp. 6–7

Behavior absorption and residual contribution

  • 16% to 76% mean success for ACT within Full when the residual is disabled at matched checkpoints without retraining; the combined policy reaches approximately 88%.ACT within Full improves by 60 points, demonstrating absorption of successful corrections into ACT.Source: C. Ablation Studies · p. 7, Figure 6 · p. 7
  • 66% versus 32% at 20k steps for ACT–Flow and independently trained Flow-only, respectively; their diagnostic endpoints are 88% and 87%.The early lead highlights the action prior’s contribution to efficient online adaptation.Source: C. Ablation Studies · p. 7, Figure 6 · p. 7
  • Approximately 12 points of final mean gain remain from the residual, ranging from 7 on key insertion to 25 on threading.ACT improves most on USB and key insertion (78 and 74 points), while threading has the smallest ACT improvement (35 points) and largest remaining residual contribution.Source: C. Ablation Studies · p. 7, Figure 6 · p. 7

Critic ranking and weighting

  • All-buffer critic groups have mean returns of 28/55/83 across low/middle/high candidates, versus 47/56/64 for the success-only critic; weighted means are 56, 68, and 75 for uniform, success-only, and all-buffer weights.The high-minus-low gaps are 55 versus 17 return units, and all-buffer weighting leads on every task.Source: C. Ablation Studies · p. 7, Figure 7 · p. 8

VI. CONCLUSION

ACT–Flow combines successful-action updates to ACT with residual relabeling and critic-weighted and successful-execution flow matching. Across five real-world tasks, it demonstrates early adaptation and behavior absorption while residuals retain additional benefits; the framework requires feasible anchor demonstrations and correction warmup.

VI. CONCLUSION

  • ACT–Flow combines successful-execution updates to ACT, residual relabeling against the evolving prior, critic-weighted candidate fitting, and successful-action flow matching.Across five real-world tasks, the policy demonstrates efficient early adaptation and behavior absorption, while residuals retain additional benefits.Source: §VI · p. 8
  • The resulting policy both learns corrective behavior and incorporates successful experience into its action prior.Source: §VI · p. 8
  • The framework requires tasks that admit subtask decomposition, feasible anchor demonstrations, and correction warmup.Future work will learn task decompositions and initialize corrections with less reliance on predefined anchors.Source: §VI · p. 8

Related papers

Loading 2610.29117v1…