RESETTLE: Robotic Recovery through Disagreement-Triggered Retrieval and Efficient Corrective Control

1 INTRODUCTION

• Timely interventions can address both emerging problems and the consequences of earlier mistakes, helping robots restore task progress.Environmental changes and imperfect actions can leave policies in unfamiliar states where they struggle to make progress.Source: §1 · p. 1
• Prior recovery strategies require additional data and policy updates, repeated verification or recovery computation, or online gradient computation.For generalized VLA/WAM policies, feature-distance distributions can overlap more between successful and failed executions, reducing LPB’s effectiveness for error detection.Source: §1 · pp. 1–2
• RESETTLE triggers on persistent disagreement between independently sampled action proposals, then retrieves a same-task reference and combines state-servo control with a guarded visual residual.It generates one corrective action without online trajectory optimization or additional vision-language reasoning, then returns control to the base policy.Source: §1 · pp. 2–3
• Across six base policies, RESETTLE improves aggregate task success, with further improvements on four real-world tasks and lower QwenPI monitoring-and-recovery latency than VoLoAgent’s monitoring-and-planning latency.The measured latency reduction is 74.04%–93.57% for grasp and place tool calls; Harness VLA integration also shows compatibility with high-level agentic planning.Source: §1 · p. 2

Generative VLA and World-Action Models

• Generative VLA and WAM methods advance policy representations and action generation, while RESETTLE adds execution recovery at the action interface without changing either.The base-policy weights and action-generation mechanisms remain unchanged.Source: §2 · p. 3

Latent World Models and JEPA-Based Control

• Latent-world-model methods use online optimization for goal-directed control, whereas RESETTLE uses demonstration states to anchor servo recovery and keeps its dynamics predictor inactive at deployment.A guarded visual residual refines the reference-derived state-servo prior before control returns to the frozen policy.Source: §2 · p. 3

Failure Recovery in Visuomotor Policies

• Unlike recovery approaches relying on gradient-guided refinement or additional VLM reasoning, RESETTLE monitors persistent disagreement and constructs reference-based corrections directly.The cited approaches preserve policy weights but add computational burden through gradient-guided action refinement.Source: §2 · p. 3

Diffusion and Flow Matching in VLAs and WAMs

• Different noise samples can yield different action chunks under identical observation and task conditioning, enabling action disagreement to be measured without changing base-policy weights.Diffusion models denoise action sequences, while flow-matching models learn a velocity field that transforms noise into actions.Source: §3 · p. 3

Action-Conditioned Prediction with V-JEPA 2

• V-JEPA 2-AC predicts future visual representations from visual context, robot states, and actions, motivating action-conditioned representations for recovery.The adapted encoder supports reference retrieval and residual action prediction at deployment.Source: §3 · p. 3

RESETTLE overview

• The RESETTLE overview shows disagreement-triggered retrieval followed by reference-guided recovery with a state-servo prior and guarded visual residual, then one native control tick before re-observation.The diagram labels independently seeded action chunks and current/reference observations and measured robot states.Source: Figure 2 · p. 4

Disagreement-Based Intervention

• A trigger requires disagreement to exceed a training-derived P95 threshold at two consecutive decisions, filtering isolated sampling fluctuations.The threshold is shared across LIBERO-Plus perturbation variants and tasks, and is benchmark-wide across tasks on the other benchmarks.Source: §4 · p. 4

V-JEPA-Based Retrieval

• After triggering, the adapted V-JEPA-2 encoder retrieves the nearest successful same-task demonstration by cosine distance and returns its observation and robot state.Matching seeks a reference relevant to the current execution phase.Source: §4 · p. 4

Reference-Guided Recovery

• The retrieved robot-state displacement defines a native-coordinate servo prior, which a learned visual residual adjusts using current and reference scene context.A guard checks prior alignment and limits residual magnitude; accepted residuals are added before command clipping, while rejection preserves the bounded servo action.Source: §4 · pp. 4–5

Training and deployment

• The recovery encoder and residual head are trained offline on expert demonstrations while the base policy remains frozen; the dynamics predictor is inactive at deployment.Each intervention discards pending base-policy actions, executes one native control action, and returns control with a fresh observation, without online trajectory optimization or additional LLM/VLM reasoning calls.Source: §4 · p. 5

5.1 EXPERIMENTAL SETUP

• Evaluation spans LIBERO-Plus, Meta-World, RoboCasa Tabletop, and LIBERO-Pro Swap, with matched conditions and frozen base-policy weights within each comparison.Recovery modules are shared across policies within settings, while thresholds are separately calibrated per base policy using training-data P95 inference.Source: §5.1 · pp. 5–6

5.2 EXPERIMENTS ON SINGLE-ARM MANIPULATION

• FastWAM rises from 49.3% to 58.0%, π0.5 from 70.0% to 73.7%, and QwenPI from 80.2% to 84.6% on LIBERO-Plus; Meta-World macro-averages also rise for both policies.On Meta-World, QwenPI rises from 60.66% to 65.99% and SmolVLA from 59.20% to 65.48%.Source: Table 1 · p. 6, §5.2 · pp. 6–7, Table 2 · p. 7

5.3 EXPERIMENTS ON ROBOCASA TABLETOP

• On RoboCasa Tabletop, LDA rises from 50.33% to 52.42% and QwenGR00T from 53.67% to 60.50%, with base-policy weights retained.Source: Table 3 · p. 7, §5.3 · p. 7

5.4 HIERARCHICAL EXECUTION ON LIBERO-PRO SWAP

• On the same 100 LIBERO-Pro Swap trials, success rises from 15.00% to 24.00% for direct π0.5 and from 42.00% to 50.00% with the GPT-5.5 planner.The planner and execution framework are retained while RESETTLE is added at the low-level policy interface.Source: Table 4 · p. 7, §5.4 · p. 8

5.5 REAL-WORLD MANIPULATION EXPERIMENTS

• Across four real-world tasks, average metrics rise from 66.67% to 78.75% for QwenPI and from 86.25% to 92.92% for VLAct.Each policy uses frozen weights; the evaluation includes binary success on three short-horizon tasks and stage completion on Use Spoon.Source: Table 5 · p. 8, §5.5 · p. 8

Analysis of when to intervene

• With two consecutive P95 exceedances, triggers cover 77.89–100.00% of failed trajectories and 9.22–42.81% of successful trajectories across seven LIBERO-Plus perturbation categories.Training-data calibration distributions remain similar with 5 trajectories, 100 trajectories, or the full training set.Source: §6 · pp. 8–9

Monitoring and recovery latency

• RESETTLE’s 83.6 ms recovery latency is 74.04–93.57% lower than VoLoAgent’s place and grasp branches in QwenPI-based comparisons.The comparison measures computation, excluding nominal inference, polling delays, and physical execution; response remains bounded by decision rate and the two-exceedance trigger.Source: Table 6 · p. 9, §6 · p. 9

Analysis of how to recover

• Adding the guarded visual residual raises aggregate success from 49.3% to 58.0% for FastWAM, 70.0% to 73.7% for π0.5, and 80.2% to 84.6% for QwenPI.The residual improves or matches servo-only recovery across perturbation dimensions at reported precision, but the results do not establish collision avoidance.Source: §6 · p. 9

Failure cases

• Consistent action proposals may leave spatial or instruction-following errors untriggered, and recovery may fail with unresolved physical configurations or mismatched goals.Progress and contact signals or selective planning are proposed as possible mitigations, but are not explored because they add sensing and computation costs.Source: §6 · p. 9

• RESETTLE combines disagreement-based intervention, successful same-task reference retrieval, and corrective control using a state-servo prior with a guarded visual residual.Source: §7 · p. 9
• Interventions execute one corrective action before returning control to the base policy, without fine-tuning, online trajectory optimization, or additional VLM reasoning calls for recovery.Source: §7 · pp. 9–10
• Experiments across six policies and three manipulation benchmarks show improved task completion, while integration with Harness VLA supports compatibility with high-level planning.Source: §7 · p. 10

A QUALITATIVE ANALYSIS OF BASE-POLICY FAILURES AND RECOVERY BEHAVIORS

• Gripper-pose adjustments enable renewed grasping after a drop or obstruction and correct contact during stove-knob operation.The examples include a moka pot, a dropped bowl, an obstructed creamcheese grasp, and knob rotation.Source: A QUALITATIVE ANALYSIS OF BASE-POLICY… · p. 13, Figure 4 · p. 14
• Placement adjustment lets an obstructed cup enter the microwave, while corrective motion and gripper realignment recover from an unintended grasp interaction.The latter example illustrates local recovery from a grasp error, not general correction of semantic target-selection errors.Source: A QUALITATIVE ANALYSIS OF BASE-POLICY… · p. 13, Figure 5 · p. 15
• Two interventions in one rollout preserve a usable grasp configuration and support subsequent lifting and transfer toward the basket.Control returns to the base policy between the interventions.Source: A QUALITATIVE ANALYSIS OF BASE-POLICY… · p. 13, Figure 5 · p. 15
• Across seven paired examples, RESETTLE corrects local grasp, contact, and placement problems, with the base policy resuming from fresh observations between interventions.The selected sequences are temporally ordered; paired columns do not imply synchronized execution after intervention.Source: A QUALITATIVE ANALYSIS OF BASE-POLICY… · p. 13, Figure 4 · p. 14, Figure 5 · p. 15
• The examples show pose realignment, disengagement from obstruction, and renewed grasping or placement, but do not isolate retrieval, servo-prior, or visual-residual contributions.Whether local correction succeeds depends on physical configuration and intervention timing; contact or object loss alone does not determine success.Source: A QUALITATIVE ANALYSIS OF BASE-POLICY… · p. 13

B ADDITIONAL METHOD DETAILS

• The method-details section specifies the intervention criterion, reference retrieval, recovery controller, and training objectives; its notation distinguishes observation, measured robot state, task, and embodiment.A proposal contains H actions, while an intervention executes one native control tick.Source: B ADDITIONAL METHOD DETAILS · p. 16

B.1 DISAGREEMENT-BASED INTERVENTION

• Two independently sampled proposals under identical conditioning are compared with an embodiment-specific, temporally weighted action metric; two consecutive threshold exceedances trigger recovery.Calibration and deployment use the same metric and action normalization, and disagreement is a proxy for decision uncertainty rather than a direct test of feasibility or failure.Source: §B.1 · pp. 16–17
• Five training trajectories yield a LIBERO-Plus disagreement distribution similar to those from 100 trajectories and the full training set, supporting lightweight offline threshold calibration.Thresholds use training-data inference P95 except in the percentile ablation; the five-trajectory subset is for calibration only, not recovery-model training or reference-bank construction.Source: §B.1 · pp. 16–17, Figure 6 · p. 17

B.2 V-JEPA-BASED RETRIEVAL

• The adapted V-JEPA encoder retrieves a successful, same-task observation and its associated robot state as visual conditioning and a corrective-control reference.The reference is an empirical anchor of successful behavior, not a certificate of physical reachability or the base policy’s full operating range.Source: §B.2 · pp. 17–18

B.3 REFERENCE-GUIDED RECOVERY

• A state-servo prior maps the embodiment-specific reference displacement into a bounded native command, and a learned visual residual adjusts that prior rather than the base-policy proposal.The guard checks directional similarity and residual-to-prior magnitude; rejecting a residual preserves the bounded servo command, but does not guarantee collision avoidance.Source: §B.3 · pp. 18–19

B.4 EXECUTION-ALIGNED JEPA TRAINING

• Execution-aligned JEPA training combines single-step latent dynamics prediction with residual-action supervision, native-action loss, and penalties for gate and action-limit violations.Training clips contain eight observations and seven transitions; the dynamics predictor and target encoder are not required online.Source: §B.4 · pp. 19–20

C.1 INITIALIZATION AND NETWORK ARCHITECTURE

• The LIBERO recovery model uses a ViT-g/16 context encoder, while deployment requires the adapted context encoder and residual head rather than the dynamics predictor or EMA target encoder.The LIBERO configuration has approximately 2.336B instantiated training parameters and approximately 1.019B deployed parameters; “lightweight” refers to measured online computation and brief, selective interventions, not a small training model.Source: §C.1 · p. 20

C.2 TRAINING DATA AND TEMPORAL SAMPLING

• Training data include LIBERO expert demonstrations, 24 GR1 task datasets, and 2,500 Meta-World episodes; the Meta-World temporal loader produces 17,060 valid clips under its sampling rule.LIBERO uses eight-frame clips sampled at 4 FPS, with frames 0 and 4 supplying current and reference observations.Source: §C.2 · p. 21

C.3 EMBODIMENT-SPECIFIC ACTION INTERFACES

• Embodiment adapters handle seven LIBERO coordinates, four Meta-World coordinates, and 29 selected GR1 coordinates, with learned corrections limited by each interface.LIBERO learned corrections act on six motion coordinates while the prior controls the gripper; Meta-World uses no rotational residual dimensions.Source: §C.3 · p. 21

C.5 SIMULATION MODEL SETTINGS AND EVALUATION PROTOCOLS

• Simulation evaluation covers seven LIBERO-Plus perturbation categories, all 50 Meta-World MT50 tasks, 24 RoboCasa Tabletop tasks, and 10 LIBERO-Pro Swap tasks.The specified budgets are 10,030 LIBERO-Plus cases, 500 episodes per Meta-World configuration, 1,200 episodes per RoboCasa configuration, and 100 episodes per LIBERO-Pro Swap configuration.Source: §C.5 · p. 22
• Harness VLA uses GPT-5.5 as a high-level planner, π0.5 for low-level action chunks, SAM3 for perception, and RESETTLE for recovery when enabled.GPT-5.5 does not generate the low-level action chunks.Source: §C.5 · p. 22

C.6 OPTIMIZATION AND TRAINING BUDGETS

• Training budgets are expressed in optimizer updates, with LIBERO’s 20 virtual epochs and GR1’s 15 virtual epochs defined by fixed update counts rather than full dataset traversals.Meta-World epochs follow the distributed DataLoader, which drops incomplete batches.Source: §C.6 · p. 22, Table 8 · p. 23

C.7 REFERENCE BANK CONSTRUCTION

• Reference banks store 1408-dimensional normalized visual features alongside aligned robot states and source observations, with banks partitioned by semantic task.Individual demonstration frames can enter the reference bank even when they do not belong to a valid temporal training clip; online recovery needs the adapted encoder and residual head, not the dynamics predictor or EMA target encoder.Source: §C.7 · p. 23

D ADDITIONAL ANALYSIS OF INTERVENTION AND RECOVERY

• Across six perturbations, feature distances overlap substantially between outcomes, while disagreement aligns training inference and success at low scores and shows heavier failure tails.Figure 7 compares same-task reference distances with QwenPI disagreement; its counts are frames or policy decisions, not rollouts.Source: D ADDITIONAL ANALYSIS OF INTERVENTION… · p. 23, Figure 7 · p. 24
• 56.24–90.18% is the failure-minus-success trigger-rate gap for action disagreement, versus −2.41% to 23.25% for feature distance.Action disagreement triggers on 77.89–100.00% of failed trajectories and 9.22–42.81% of successful trajectories.Source: Table 9 · p. 24, D ADDITIONAL ANALYSIS OF INTERVENTION… · p. 25
• 42.81% of successful trajectories triggered under noise, showing disagreement is an intervention cue rather than a definitive failure label.RESETTLE follows brief corrective interventions with re-observation and a return to the base policy.Source: D ADDITIONAL ANALYSIS OF INTERVENTION… · pp. 25–26
• Across FastWAM, LDA, and SmolVLA, failed trajectories trigger more often and repeatedly than successful ones, although separation varies by policy and benchmark.FastWAM trigger rates are 42.01–74.75% for failures versus 7.95–12.99% for successes.Source: D ADDITIONAL ANALYSIS OF INTERVENTION… · p. 25, Table 10 · p. 25
• 2.67% of 1,855,166 executed actions were recovery, and 72.69% of trajectories had no takeover; Robot supplied 79.24% of recovery actions.Recovery share is computed from executed actions, excluding initialization waiting actions.Source: D ADDITIONAL ANALYSIS OF INTERVENTION… · p. 26, Table 11 · p. 26
• 84.6% at P95 is the best aggregate success, versus the 80.2% baseline, but no percentile leads every perturbation category.Robot success declines from 84.2% at P85 to 76.2% at P99, while Noise success increases from 79.5% to 89.6%.Source: D ADDITIONAL ANALYSIS OF INTERVENTION… · p. 27, Table 12 · p. 27

D.1 VISUAL RESIDUAL ABLATION

• Adding the guarded visual residual raises aggregate success by 2.1%, 0.8%, and 1.5% for FastWAM, π0.5, and QwenPI, respectively.The residual adds scene-conditioned adjustment to the state-servo prior; the gains do not directly establish fewer collisions.Source: §D.1 · p. 27

D.2 MONITORING AND RECOVERY LATENCY DETAILS

• RESETTLE recovery takes 83.6 ms, dominated by V-JEPA encoding and nearest-neighbor retrieval; corrective command generation takes 0.8 ms.These are computation measurements, excluding nominal policy inference, polling delays, and physical action execution; responsiveness still depends on policy decision rate and the two-exceedance criterion.Source: §D.2 · p. 28

D.3 FAILURE CASES AND RECOVERY LIMITATIONS

• Unsuccessful rollouts include both missed triggers and triggered interventions that do not complete the task; selected cases illustrate failure modes, not their evaluation-wide frequency.Consistent proposals can accompany contact, spatial, or semantic errors, while triggering does not establish that a brief intervention can recover the physical state.Source: §D.3 · pp. 28, 30

E.1 HARDWARE AND OBSERVATIONS

• The real-world setup uses a stationary seven-degree-of-freedom Franka Research 3 with external and wrist-mounted RealSense cameras.All real-world evaluations use NVIDIA RTX 4090D GPUs.Source: §E.1 · p. 32

E.2 TASKS AND DATA COLLECTION

• Evaluation covers three short-horizon tasks and the long-horizon Use Spoon task, using 50 teleoperated demonstrations per single-arm task with randomized placements and arm configurations.Successful training demonstrations also form the task-conditioned reference bank.Source: §E.2 · p. 32

E.3 MODEL TRAINING

• The recovery encoder and dynamics predictor are initialized from VJEPA-2-AC and trained with a visual residual head on expert demonstrations.The dynamics predictor is used only during training; intervention thresholds use each frozen policy's training-data P95 scores and require two consecutive exceedances.Source: §E.3 · pp. 32, 34

E.4 EVALUATION PROTOCOL

• Each comparison uses identical frozen base-policy weights, settings, and execution budgets; a triggered recovery discards pending policy actions, executes one native control action, then returns control to the policy.Recovery adds no high-level planner or additional LLM/VLM reasoning calls.Source: §E.4 · p. 34

E.5 EVALUATION METRICS

• Short-horizon tasks receive binary success scores, while each of Use Spoon's three stages contributes one third of its trial score.The average equally weights the four task metrics and is not the fraction of trials completing every stage.Source: §E.5 · p. 34

E.6 DETAILED TASK-LEVEL RESULTS

• QwenPI gains 10%, 15%, and 10% on Press-button, Cube-up, and Carrot-in-pot, plus 13.3% on Use Spoon; VLAct gains 10%, 5%, 5%, and 6.7%, respectively.Both augmented policies succeed in all 20 Press-button trials and 19 of 20 Carrot-in-pot trials.Source: §E.6 · p. 34
• 66.67% to 78.75% for QwenPI and 86.25% to 92.92% for VLAct in average task metric; both policies improve on all four tasks.The base-policy checkpoints remain fixed, so the gains do not come from additional base-policy fine-tuning.Source: §E.6 · p. 34

E.7 QUALITATIVE ANALYSIS OF RECOVERY BEHAVIOR

• The Carrot-in-pot and Press-button sequences show object acquisition and placement alongside repeated approach positioning near a contact target.Placement calls for a usable grasp and receptacle alignment, whereas pressing calls for alignment with the contact target.Source: §E.7 · p. 35
• In Cube-up, renewed approaches precede cube acquisition and stacking, revisiting an unfinished interaction within the same rollout.The sequence preserves the physical prerequisite of acquiring the cube before completing placement.Source: §E.7 · p. 36
• Use Spoon progresses from grasping the spoon to source-bowl interaction and movement toward the target bowl, but frames alone do not establish successful bean transfer.Bean transfer is assessed by the stage-completion metric, not inferred from the depicted sequence.Source: §E.7 · pp. 35–36
• The behaviors are compatible with reference-guided local correction followed by resumed policy execution, without an explicit task-stage planner.The figure shows only RESETTLE rollouts and is not a paired baseline comparison.Source: §E.7 · pp. 35–36

F PER-TASK SIMULATION RESULTS

• The per-task tables report ASR for Meta-World, RoboCasa Tabletop with GR1, and LIBERO-Pro Swap, with episode budgets and evaluation settings following the stated protocols.Bold identifies the better result within each Base./Ours pair, including ties.Source: F PER-TASK SIMULATION RESULTS · p. 36, Table 13 · p. 37, Table 14 · p. 38

F.1 META-WORLD MT50

• Task-level effects vary across Meta-World policies, with gains and regressions despite improved difficulty-group aggregates.The results do not imply uniform benefit on every task.Source: §F.1 · p. 36, Table 13 · p. 37, Table 14 · p. 38

F.2 ROBOCASA TABLETOP / GR1

• On RoboCasa, averages reproduce Table 3: 53.67% to 60.50% for QwenGR00T and 50.33% to 52.42% for LDA, while task-level gains coexist with regressions and ties.Source: §F.2 · p. 36

F.3 LIBERO-PRO SWAP

• 15% to 24% for direct π0.5 execution and 42% to 50% for Harness VLA on LIBERO-Pro Swap; RESETTLE improves or matches baseline on every task in both routes.Some tasks remain unsolved.Source: §F.3 · p. 40, Table 16 · p. 40