Source-linked AI summary
Which Hand Was Late? Diagnosis for Robot Failure Recovery in Bimanual Tool Use
Baptiste Bonin, Caro Strickland, Audrey Durand
TL;DR
Bimanual tool use creates failures that single-arm diagnosis settings do not cover, including coordination failures between hands. This paper probes sensing channels and a VLM on a simulated task with trace-read causes, finding that passive sensing has limits but escalation can match true-cause repair, while VLM answer readout depends on question wording.
Problem
Single-arm failure diagnosis does not cover bimanual failures such as tool slip, mistimed correct gestures, and failures with multiple causes, where identifying the cause is necessary for recovery.
Source: §1 · p. 2Method
The authors inject faults in a simulated bimanual lock task, read causes from mechanism traces, and probe individual sensing channels and a VLM's state and answers.
Source: §1 · p. 2, §3.3 · p. 5Results
When the tool hand is late, no passive channel exceeds AUC 0.554 for distinguishing timing from two-cause failures, while an escalation policy repairs 297 of 386 attempts within two tries, matching the true-cause oracle.
Source: §4.3 · p. 7, §5 · p. 7Takeaways & Limitations
Abstract
from arXiv · showhide
Failure diagnosers for robot manipulation are increasingly vision--language models (VLMs), trained and evaluated on single-arm failures produced by perturbing a successful demonstration. Bimanual tool use adds failures of other kinds: a tool that slips out of its socket while still held, two correct gestures made at different times, and attempts that fail for two reasons at once. On a simulated bimanual tool-use task whose failure causes are read from the recorded trace, we measure what each sensing channel lets a diagnoser tell. The tool's slip is read on the tool hand; which hand was late is read on both arms, while a sensor on one hand reads its own hand's delay and at most part of the other's; and whether a late attempt was also pushed too weakly is not identifiable from passive sensing when the tool hand is late. Within two attempts, a repair policy that escalates when the cause is not identifiable repairs as many failures as one told the true cause. A VLM's internal state carries the late hand almost perfectly, along a direction no closer to the one its answer reads than a random one; whether the answer reports it depends on the wording of the question.
1 Introduction
Bimanual tool use adds failures that depend on interaction between the arms, which single-arm visual diagnosers do not address. The paper measures what sensing channels reveal about these causes and whether a VLM reports what its internal state represents.
1 Introduction
- Existing visual VLM diagnosers address single-arm, single-object failures produced by perturbing successful demonstrations, with causes visible in the scene.The cited systems address failure detection, diagnosis, visual correction, or joint detection and reasoning.Source: §1 · p. 2
- Bimanual failures can arise from a slipped tool, insufficient force, or two correct gestures occurring at different times, making the cause a relation between the hands.The paper asks what a diagnoser must observe to identify which hand caused such a failure and whether VLMs report it when observed.Source: §1 · p. 2
- The study builds a simulated bimanual tool-use task and reads failure causes from recorded mechanism traces rather than video annotation.It probes one sensing channel at a time and examines both a VLM's internal representation and language output, without proposing a new diagnoser.Source: §1 · p. 2
- Tool slip is read on the tool hand and which hand was late on both arms, but when the tool hand is late no passive sensor reveals whether the push was also too weak.Sensor placement determines which signals are available.Source: §1 · p. 2
- An escalation policy that applies a matching correction and then both repairs as many failures within two attempts as a policy told the true cause.The policy first asks whether the hands overlapped; two-cause failures require correcting both causes.Source: §1 · p. 2
- The VLM's internal state encodes which hand was late even though its answer does not read out that information.Whether the answer reports the late hand depends on question wording.Source: §1 · p. 2
2 Related work
Related work centers on VLM-based failure reasoning, commonly evaluated through textual explanations, alongside bimanual platforms and representation probes. The paper distinguishes its analysis by comparing model answers with information in the model's own state.
2 Related work
- Prior VLM failure-diagnosis work is almost entirely focused on single-arm tasks and judges explanations by their text.The review also covers how answers are scored, bimanual platforms, and representation probes.Source: §2 · p. 3
- AHA, RoboFAC, ViFailback, and ARMOR use visual reasoning for failure explanation, detection and localization, policy correction, or joint detection and reasoning.I-FailSense fine-tunes a VLM for failure detection; REFLECT uses non-visual modalities as evidence for scene-level explanations rather than diagnosing which modality carries each cause.Source: §2 · p. 3
- Explanation scoring commonly uses a classifier head or reference similarity metrics, while the paper compares answers with what the model's own state contains.The authors say they know of no prior work making that comparison.Source: §2 · p. 3
- Linear probes on frozen features separate information represented by a model from what it reports, and the study applies the same probe across sensing channels and VLM states.The review notes no known diagnosis benchmark whose taxonomy includes a relation between two arms.Source: §2 · p. 3
3 A tool-use task with known failure causes
The paper introduces a simulated bimanual lock task with trace-derived failure causes, including failures with simultaneous causes, and applies a common probe across sensing channels and the VLM.
3.1 Scene and mechanism
- Two Franka Panda arms operate a lock: one seats and turns a square key-tool while the other pushes the door.The key springs back when released, so opening requires it to remain turned while the push exceeds a hidden threshold τ that is absent from observations.Source: §3.1 · p. 4
3.2 Causes read from the trace
- A scripted controller injects one fault per attempt, and failure classes are assigned from recorded traces using whether gestures were delivered, overlapped, and reached the force threshold.Wrong denotes an undelivered gesture; timing and force distinguish whether threshold force and key-held overlap occurred.Source: §3.2 · p. 4, Figure 3 · p. 4
- The 119 two-cause failures require both corrections: re-timing alone opens 3, lowering the threshold alone opens none, and both open all 119.These attempts are kept as a separate class; 75 have the tool hand late and 44 the push hand late.Source: §3.2 · p. 4
- The campaign comprises 758 attempts, including successes, wrong gestures, timing failures, force failures, and two-cause failures.The same injected fault can produce several trace-read classes.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3 Probing protocol
- The probing protocol fits an L2-regularized logistic regression to one input channel at a time, using door-grouped folds and within-door AUC.Inputs include hand sensors, both-arm poses and commands, camera images, and mechanism readings; the latch state is an oracle ceiling rather than evidence.Source: §3.3 · p. 5
- The protocol compares trial-window signals with an approach-window negative control, using paired door resampling for channel comparisons and bootstrap intervals.The trial window runs from when the key should turn through the end of the nominal push hold.Source: §3.3 · p. 5
4 What can be diagnosed, and from where
The section maps diagnosis to sensor location: tool failures appear on the tool hand, timing is best read bilaterally, and some joint causes remain unidentifiable passively.
4 What can be diagnosed, and from where
- Both-arm channels distinguish blocked attempts from wrong gestures: arm poses reach 0.876, commands 0.862, and images 0.916.The section measures within-door AUC, where 0.5 is chance and 1 is perfect separation.Source: §4 · p. 5
4.1 On one arm: the tool’s own failure
- Cam-out is localized on the tool hand: socket contact reaches 0.986 [0.959, 1.000], while push-hand force and contact are near chance (0.519, 0.512).The front camera alone, which is given to the VLM, identifies cam-out at 0.768 [0.559, 0.908].Source: §4.1 · p. 6
4.2 Between the arms: which hand was late
- One-hand sensors primarily capture their own hand’s delay, while two-arm channels capture both; lock angle shows the sharpest interaction at +1.224 [+0.954, +1.487].Figure 4 compares within-door AUC by sensor location and hand-late class, with interaction effects above.Source: §4.2 · p. 6, Figure 4 · p. 6
- Arm poses distinguish tool-hand-late and push-hand-late attempts at 1.000 and 0.970, and identify out-of-sync hands at 0.986.Arm poses use 108 numbers, compared with 3,056 GFLOPs for one Qwen3-VL 8B call.Source: §4.2 · p. 6
- The bilateral pattern persists with full-window signals and sensor noise, but not with random start times; relative onset still separates the halves at 1.000 and 0.990.Before contact, every channel is at chance or barely above it (≤ 0.54).Source: §4.2 · p. 6
4.3 On no sensor: failures with two causes
- When the tool hand is late, the passive-sensing ceiling is exactly 0.500; no channel exceeds 0.554, while the latch-state oracle reaches 1.000.The ceiling reflects a hidden threshold: within a door, the push hand delivers the same force on these attempts, and threshold alone determines whether it is enough.Source: §4.3 · p. 7
5 Diagnosis decides the repair
The repair results show that diagnosis should allow for uncertainty: escalating from a timing check to both corrections matches the true-cause oracle within two attempts.
5 Diagnosis decides the repair
- 297 of 386 failures are repaired within two tries by checking whether the hands overlapped, applying the matching correction, then escalating to both; this matches the true-cause oracle.The oracle repairs more on the first attempt: 289 versus 200.Source: §5 · p. 7
6 The model knows the late hand but does not say it
The study tests whether Qwen3-VL 8B represents which hand was late and whether its answer reports that information. The model’s state reads the late hand reliably, but answer accuracy varies with question wording and a reference can improve it without aligning the answer’s readout with the state’s probe direction.
6 The model knows the late hand but does not say it
- The study scores yes/no answers using log P(Yes)−log P(No) at the first answer token, without parsing generated text.The model receives the first and last front-camera frames and questions in three phrasings per failure half.Source: §6 · p. 7
- Answer accuracy depends on phrasing: Qwen3-VL 8B’s median AUCs are 0.497 for tool-hand-late and 0.604 for push-hand-late, with phrasings ranging from near chance to near-perfect.The model scores 0.865 on whether the door is open in the second frame.Source: §6 · p. 7
- A linear probe reads both late-hand labels at 0.956 or higher from layer 4 onward, reaching 0.987 and 0.984 at the final-normalisation input.The answer reads the same state along a direction whose cosine with the probe direction is −0.010 and +0.020, comparable to random directions.Source: §6 · p. 8, Figure 5 · p. 8
- Providing a successful same-door reference raises median answer AUC to 0.971 for tool-hand-late and 0.793 for push-hand-late without aligning the answer direction with the probe direction.Removing the probe direction changes answer AUC by only +0.002 for tool-hand-late and +0.009 for push-hand-late.Source: §6 · p. 8
7 Conclusion
The conclusion presents the findings as evidence from a simulated task with a scripted controller and a particular delay schedule, and identifies learned policies and real robots as next settings to test.
7 Conclusion
- The bilateral timing pattern is limited to the scripted controller’s delay schedule; future work will test learned policies, real robots, more tasks and tools, and active diagnosis.The authors also propose checking VLM answers against the model’s state.Source: §7 · p. 8
A Task details
The simulated task uses two scripted arms to operate a tool-mediated lock, with failures labeled from the recorded trace. It also replays blocked attempts under alternative corrections.
Simulation
- The task runs in ManiSkill3 with two Franka Panda arms, joint-position control, and 262 control steps per attempt.The cabinets come from PartNet-Mobility.Source: A Task details · p. 11
Lock and tool
- A shallowly seated tool can cam out because the chamfer’s inclined faces push it out under torque.The lock has a 16 mm square socket 20 mm deep; the tool’s drive is 14 mm square and nominal insertion is 16 mm.Source: A Task details · p. 11
Simulation and controller
- Two scripted arms operate a spring-returned lock that opens only when the turned key is held while the push meets the attempt-specific threshold τ.The latch checks each control step; τ is drawn log-uniformly from 20 to 700 N.Source: A Task details · p. 11
Classes
- Trace rules distinguish wrong, timing, force, and two-cause failures, and identify the delayed arm from the trace intervals.A timing failure is one where the push reaches τ without the key held; 26 timing failures also meet the force definition but retain their timing label.Source: A Task details · p. 11
Campaign and replay
- The campaign draws 1,000 seeds across 23 doors; 758 attempts qualify, and replay reproduces their recorded signals exactly.Seeds on six ineligible doors are skipped rather than replaced, preserving each seed’s door and threshold.Source: A Task details · p. 11
Repair replays
- Blocked attempts are replayed with re-timing, a deeper push, both corrections, or a lower-threshold counterfactual.The deeper push commands 0.066 m, three times the nominal depth; ten unmodified replays reproduce the recorded attempts exactly.Source: A Task details · p. 11
B Protocol details
The protocol compares sensing channels across defined trial and approach windows, with bootstrap intervals for pooled AUCs. It also tests robustness under noise and randomized onsets.
Windows
- The trial window spans steps 106–152 of 262, while the approach window spans steps 0–26; the VLM receives front-camera frames at steps 106 and 151.Source: B Protocol details · p. 12
Intervals
- AUCs are computed within each door and pooled by pairs; intervals use 10,000 door resamples, with paired differences and interactions reported as bootstrap means.Source: B Protocol details · p. 12
Ceiling for failures with two causes
- For failures with two causes, the passive-reading ceiling scores each attempt by the prior probability that its largest push exceeds τ, then computes within-door AUC.The ceiling was computed and committed before probing that population.Source: B Protocol details · p. 12
Robustness of the bilateral pattern
- Full time series show positive interactions for lock angle against push-hand force (+1.305 [+1.062, +1.533]) and tool-hand torque against push-hand force (+0.736 [+0.510, +1.021]).With the full series, tool-hand torque reads push-hand lateness at 0.846 [0.649, 0.981], while push-hand force reads tool-hand lateness at 0.384 (asymmetry −0.457 [−0.637, −0.257]).Source: B Protocol details · p. 12
- Under 0.2×-spread sensor noise, the reported interaction intervals remain above zero; in the approach window, every channel is 0.500 for tool-hand lateness.For push-hand lateness in the approach window, arm poses are 0.526 [0.506, 0.548] and the front camera is 0.539 [0.507, 0.576].Source: B Protocol details · p. 12
Second campaign (randomised onsets)
- In the randomized-onset campaign, the pre-registered check was not triggered: the largest single-arm clock AUC was 0.636 [0.514, 0.717].The campaign had 400 attempts on the same 17 doors and was run as a recorded deviation after the pilot gate failed.Source: B Protocol details · p. 12