Source-linked AI summary
Diagnosing and Recovering from Observation-Space Shift at Long-Horizon Skill Seams
Pranav Wagh, Yu Fang, Yue Yang, Mingyu Ding
TL;DR
At skill seams, downstream skills inherit states their training distributions do not cover, and long-horizon success can collapse despite reliable isolated skills. The paper uses privileged simulator interventions to diagnose seam shift, then tests a learned detect–restore–resume system with seam-robust fine-tuning. The fully learned, non-privileged detect–restore–resume system improves full-chain success 3.5× over the frozen base (p=0.047), reaching 51% of a privileged oracle from cameras and proprioception alone.
Problem
At skill seams, downstream skills inherit states their training distributions do not cover, and long-horizon success can collapse despite reliable isolated skills.
Source: §I · p. 1Method
The paper uses privileged simulator interventions to diagnose seam shift, then tests a learned detect–restore–resume system with seam-robust fine-tuning.
Source: §I · pp. 1–2Results
The fully learned, non-privileged detect–restore–resume system improves full-chain success 3.5× over the frozen base (p=0.047), reaching 51% of a privileged oracle from cameras and proprioception alone.
Source: §VII · p. 8Takeaways & Limitations
The tested controls do not recover the evaluated seam, whereas restoration does, supporting restoration as a targeted response to these off-support failures.
Source: §VII · pp. 7–8The quantitative core is in simulation; hardware monitoring is limited by exterior-camera observability, and restoration is defined only for drawer seams.
Source: §VII · p. 8
Abstract
from arXiv · showhide
Long-horizon robotic manipulation is often built by chaining independently trained skills. Although each skill can be reliable in isolation, performance degrades sharply when skills are chained: each downstream skill must start from the state its predecessor leaves behind rather than from its training distribution. We study this failure mode, Observation-Space Shift (OSS), and ask what causes these skill-seam failures. Using privileged simulator resets, we find that the dominant shift comes from displaced scene state (e.g., an open drawer or secondary objects left behind by earlier skills), not from the robot's joint configuration or the object the downstream skill manipulates. To test this diagnosis, we build a fully learned detect-restore-resume system: a task-progress monitor detects the stall, a learned policy restores the displaced scene components, and seam-robust fine-tuning lets the skill resume. It recovers the seam where every tested alternative fails, which we treat as evidence for the diagnosis rather than as a general-purpose method. On the BOSS-44 benchmark, the system improves full-chain success from 7.6% to 26.5%, a 3.5x improvement over the base policy and 51% of a privileged restoration oracle, whereas best-of-K resampling, a Diffusion Policy, and world-model baselines fail to recover from the evaluated seam states. On a real Franka arm running a fine-tuned $\pi_{0.5}$ policy, the same monitor is limited by exterior-camera observability, yet closing the loop still recovers some otherwise-terminal failures, motivating wrist and gripper sensing. These results suggest that some long-horizon composition failures are better addressed by restoring the scene before resuming the policy than by retrying from an off-support state.
I. INTRODUCTION
Independently reliable skills can stall at seams because they inherit off-support scene states. The paper diagnoses the shift and tests a learned detect–restore–resume system that improves full-chain success.
I. INTRODUCTION
- A downstream skill inherits its predecessor’s scene state rather than its canonical training state, creating Observation-Space Shift at the seam.On the studied three-skill chains, success falls from 88.5% after one skill to 7.6% after all three, despite 79.0% average isolated-skill success.Source: §I · p. 1
- Privileged simulator interventions identify displaced articulation and secondary objects—not arm configuration or the manipulated object—as the dominant sources of seam shift.Restoring articulation or secondary objects alone yields partial gains; restoring them jointly recovers substantially more oracle performance.Source: §I · p. 1
- The learned, non-privileged system detects stalls, restores displaced scene state, and resumes the downstream skill using deployable observations.It also uses seam-robust fine-tuning to improve re-entry after approximate restoration.Source: §I · p. 2
- 3.5× improvement in BOSS-44 full-chain success over the base policy is reported for the proposed framework.The paper also reports that the real-arm monitor is limited by exterior-camera observability.Source: §I · p. 2
II. RELATED WORK
The paper situates skill-seam failures within skill composition and recovery research, then defines the benchmark setting and the scope of restoration. Its three-skill chains expose compounded seam shift despite competent component skills.
II. RELATED WORK
- Prior work addresses composition through training-time regularization, recovery data, retrieval, backtracking, correction, or learned resets; this paper localizes the shift to specific state components.The related-work discussion also contrasts seam-triggered restoration with planning and sampling approaches evaluated from OSS states.Source: §II · p. 2
III. PROBLEM FORMULATION
- The benchmark uses continuous three-task episodes in which objects and articulation carry over between skills, while only the arm is re-homed.Three skills are the shortest chain length containing two seams, allowing shift to compound across an upstream seam.Source: §III · pp. 2–3
- Observation-Space Shift is the mismatch between a skill’s canonical initial-state distribution and the predecessor’s terminal-state distribution at a seam.Seam collapse occurs when the predecessor’s terminal states fall off the downstream skill’s training support.Source: §III · p. 3
- Restoration is defined for reversible side effects with known canonical targets, not inherited changes that the next skill requires as preconditions.For example, restoring a drawer to closed would undo a change needed before placing an object inside it.Source: §III · p. 3
- Chain success falls from 0.885 to 0.076 across two seams, while frozen skills succeed at 0.79 on average from canonical initial states.The evaluation focuses on 10 catastrophic chains drawn from 13 skills; across BOSS-44, mean RPD is 0.45 over one seam and 0.51 over two.Source: §III · p. 3
IV. METHOD: DETECT, RESTORE, RESUME
The method adds learned detection, scene restoration, and skill resumption while leaving the base skills frozen. Each component uses deployable camera and proprioceptive inputs.
IV. METHOD: DETECT, RESTORE, RESUME
- A task-progress monitor detects when the current skill has stalled.The method assigns a learned component to each of the detect, restore, and resume stages.Source: §IV · p. 3
- A learned restoration component returns displaced scene state toward the downstream skill’s support.The system leaves the base skills frozen and uses deployable observations throughout.Source: §IV · p. 3
- After restoration, the downstream skill resumes execution using RGB cameras and proprioception.These are the method’s stated deployable inputs.Source: §IV · p. 3
A. Detect: a task-progress monitor with a calibrated trigger
Detection uses stalled task progress as its signal, while restoration learns to return displaced scene components toward downstream support. The learned loop is distilled for camera-based deployment.
A. Detect: a task-progress monitor with a calibrated trigger
- The detector triggers when a task-progress score stops rising, rather than relying on prediction surprise at a stall.The trigger uses progress-slope stagnation over an eight-sample window and a threshold calibrated on success demonstrations; its operating point is set empirically on cached rollouts.Source: A. Detect: a task-progress monitor… · p. 3
B. Restore: a learned scene-restoration skill
- The restoration policy is learned with reinforcement learning because scripted control, replay, backtracking, and library-inverse correction do not restore displaced scene state.The restoration target includes displaced articulation and secondary objects identified by the diagnosis.Source: B. Restore: a learned… · p. 3
- A camera-only student uses dual RGB and 16-D proprioception, and the controller runs unchanged in simulation and on hardware.At an alarm, the actor runs until the drawer returns to canonical or a 150-step cap fires; the arm then re-homes and the skill resumes with cleared observation history.Source: B. Restore: a learned… · p. 3
C. Resume: seam-robust prevention fine-tuning
Seam-robust fine-tuning helps downstream skills re-enter after approximate restoration, while the implementation uses learned progress detection and scene restoration. The section describes the data and training choices for those components.
C. Resume: seam-robust prevention fine-tuning
- Post-seam skills are fine-tuned on deep hand-off states reached by recovery, making them more tolerant of approximate restoration.On its own, this fine-tuning adds only marginally; the authors treat it as complementary to restoration.Source: C. Resume: seam-robust prevention… · p. 3
D. Implementation Details
- The progress detector uses a frozen DINOv3 encoder and a task-conditioned head trained with three self-supervised losses.The losses anchor absolute progress, rank later demonstration frames higher, and push failed-window tails below success frames; evaluation is cross-validated by chain.Source: D. Implementation Details · p. 4
- A reset library pairs about 14,000 realistic failure states with canonical restoration targets, without reward engineering or labels.The per-scene libraries contain about 5.7k, 4.6k, and 3.6k states for S1, S2, and S4, respectively.Source: D. Implementation Details · p. 4
- Scene-specific restoration agents use SAC for S1 and S2, and demonstration-augmented RLPD for S4, where SAC exploration stalls.Drawer restoration reaches skill-level success of 1.00/0.93/0.78 for S1/S2/S4.Source: D. Implementation Details · p. 4
- Prevention fine-tuning uses 581 deep hand-off demonstrations mixed with original demonstrations; scaling this set to 6,073 does not improve it.The selected checkpoint minimizes seam-window loss subject to a base-policy validation-loss constraint.Source: D. Implementation Details · p. 4
V. EXPERIMENTS
The experiments diagnose seam collapse through privileged component resets and evaluate the broader recovery system. The component analysis identifies displaced scene state, rather than the arm, as the main restoration target.
V. EXPERIMENTS
- The evaluation localizes seam collapse, tests simpler alternatives, presents the recovery system, and examines detector quality and real-robot transfer.Detection is scored by alarm-statistic AUROC over per-seam windows.Source: §V · p. 4
A. What causes seam collapse, and what must be restored
- The diagnosis uses a privileged teleport oracle to restore one scene component at a time and measure recovery gains.The coarse component-decomposition run uses n=100 catastrophic chains.Source: A. What causes seam collapse, and what… · p. 4
- Jointly restoring drawer articulation and secondary objects yields a 0.52 gain, exceeding the 0.35 predicted from their individual increments.The excess is 0.170 ± 0.052; arm-only resetting is near base, and the manipulated object is essentially canonical. The component-decomposition base differs from the five-seed headline base, so it is not the headline result.Source: A. What causes seam collapse, and what… · p. 4
B. No tested alternative recovers the seam
The tested alternatives do not recover the evaluated seam states: more sampling, a stronger policy architecture, world models, and end-to-end recovery all fall short. The authors limit this conclusion to the fixes and data distributions they tested.
B. No tested alternative recovers the seam
- At the mid-chain seam, all eight base-policy rollouts fail, while Diffusion Policy matches behavior cloning despite succeeding from canonical starts.From a clean initialization, at least one of K=8 rollouts succeeds, leaving no rollout to select at the seam.Source: B. No tested alternative recovers the… · pp. 4–5
- World models fail as generators, monitors, and planners; even simulator-scored shooting with a perfect model does not recover the seam.Prediction surprise becomes anti-predictive on stalls, and model-free RLPD exceeds the tested model-based planners at 3× sample efficiency.Source: B. No tested alternative recovers the… · p. 5
- End-to-end RecoveryChaining remains below the demonstration baseline, while a specialist restorer with robust skills outperforms it.Source: B. No tested alternative recovers the… · p. 5
- Within the sampled source distribution, scaling seam demonstrations 10× leaves prevention unchanged; this does not test every conceivable fix.The reported L2 remains 0.117 after scaling.Source: B. No tested alternative recovers the… · p. 5
C. The detect–restore–resume system recovers the collapse
The full detect–restore–resume system improves recovery beyond a same-budget re-home/retry baseline and reaches part of the privileged oracle’s performance. Gains also appear on held-out seam orderings, alongside a reported false-alarm cost in one scene.
C. The detect–restore–resume system recovers the collapse
- The full system adds +0.136 L2 (2.05×) over the same-budget re-home/retry base (0.129), reaching 51% of the privileged oracle.Neither recovery nor prevention alone is significant; the authors make the claim at the system level.Source: C. The detect–restore–resume system… · p. 6
- False alarms can erase gains locally: S4 reaches L2 0.16 versus base 0.19, and chain ch3 8 regresses from 0.27 to 0.17.Source: C. The detect–restore–resume system… · p. 6
- On seven held-out reorderings, the learned system improves L2 from 0.066 to 0.244 (3.7×), though the subset result is not significant at p=0.125.These are novel seam transitions among the studied skills; the oracle reaches 0.499 on the subset.Source: C. The detect–restore–resume system… · p. 6
D. When restoration is the right fix
Restoration helps when a displaced drawer blocks a downstream skill that does not operate it, but can regress drawer tasks and does not yet recover free-object-blocked seams. The detector and camera-only restoration loop are deployable within the stated control-loop budget, while recovery quality remains the bottleneck.
D. When restoration is the right fix
- 0 → 0.28 when an off-canonical drawer blocks a non-drawer skill: learned restoration clears the obstruction, while retry cannot move the drawer.The comparison isolates the recovery action per chain, with matched re-home and retry versus learned restoration without prevention.Source: Table V · p. 6, D. When restoration is the right fix · p. 6
- 0.32 → 0.21 when the downstream skill operates the drawer: restoration undoes its effect, while a displaced free object remains unrecovered by the drawer restorer.The tested object restorer reaches 0.65 at skill level but only L2 ≈ 0.10 at chain level, versus 0.265 for drawer restoration.Source: D. When restoration is the right fix · p. 6
E. Detector quality and deployability
- L2 0.226 with ground-truth seam timing is below the selective monitor’s 0.265 and the 0.518 oracle teleport, leaving recovery quality—not detection—as the stated bottleneck.The better offline detector reaches end-to-end parity with the video-model monitor.Source: E. Detector quality and deployability · p. 6
- 0.903 for the final DINOv3-B/16 progress head versus 0.782 for the earlier Wan-VAE head, with a 6× latency reduction.The roughly 1M-parameter head uses a frozen encoder; in-loop, the improved offline detector reaches end-to-end parity with the video-model monitor.Source: E. Detector quality and deployability · pp. 6–7
- The camera-only student matches the privileged teacher at 0.85/0.90, and the RGB-and-proprioception loop runs within the 50 ms budget of the 20 Hz control loop.The monitor costs 6.1 ms per sampled frame and the restoration student 6 ms per step on CPU; restoration is still trained per scene in simulation.Source: E. Detector quality and deployability · p. 7
F. The monitor transfers to a real robot
On the Franka, restoration recovered some otherwise-terminal place attempts when compared at matched detector flags, but the study reports no overall success-rate improvement. Exterior-camera observability limits detection, and the measured transfer results are weak.
F. The monitor transfers to a real robot
- The unchanged simulation monitor was evaluated on a Franka with a fine-tuned π0.5 policy across three rearrangement chains, using place-only restoration and a matched step budget.Each condition receives 200 steps per skill, with restoration counted inside that budget; the hardware restorer is a language-prompted π0.5 behavior.Source: F. The monitor transfers to a real robot · p. 7
- At matched detector flags, restoration recovers otherwise-terminal place attempts that the baseline never recovers, but per-skill success is within noise and no success-rate improvement is claimed.Both conditions use the same live flags, so the reported recovery effect is attributed to the restore action rather than detector quality.Source: Table VII · p. 7, F. The monitor transfers to a real robot · p. 7
- Detection caught 50% of failures at a 26% false-alarm rate; at n=45, neither AUROC separated from chance (Fisher p≈0.12).Source: F. The monitor transfers to a real robot · p. 7
- Exterior-camera vision can miss failed grasps or dropped objects, limiting transfer detection; wrist and gripper sensing are the stated next step.The in-sample estimate is leaked because the monitor was fit on rollouts of the same objects it scores.Source: F. The monitor transfers to a real robot · p. 7
VI. DISCUSSION
The discussion argues that seam failures stem from off-support states and that restoring displaced scene state succeeds where the tested alternatives do not. The conclusion reports a 3.5× full-chain gain, while emphasizing unresolved restoration quality, object-restoration, and evaluation limits.
VI. DISCUSSION
- The tested test-time-compute, stronger-base-policy, and world-model interventions fail from the same seam states where restoration succeeds.The discussion frames the seam as an off-support state rather than a miscalibrated policy.Source: §VI · p. 7
- 0.13–0.27 of the 0.52 oracle: any single-component fix reaches only this range, because articulation and object restoration are complementary.Source: §VI · p. 7
- Re-entry timing is a hypothesized restoration-quality bottleneck: the monitor can re-fire mid-restoration, and fixed hand-back tolerance is a crude proxy for readiness.The discussion proposes a learned re-entry value function and enabling restoration by skill type as next steps.Source: §VI · p. 7
VII. CONCLUSION AND FUTURE WORK
- 3.5× over the frozen base, the learned detect–restore–resume system reaches 51% of the privileged oracle using cameras and proprioception alone.The reported full-chain result has p=0.047; absolute L2 is 0.265.Source: §VII · p. 8
- Restoration is defined only for drawer seams; chains whose second seam displaces a free object remain unrecovered and need an object-restoration skill.The tried object restorer does not yet transfer.Source: §VI · p. 7, §VII · p. 8
- The evaluation uses one base-policy seed and an uncorrected n=10 test; scaling same-distribution seam data 10× does not help (0.117).The paper leaves broader training-distribution randomization untested.Source: §VII · p. 8