Source-linked AI summary
Audit Before You Commit: Locating Belief Failures in Active Identification for One-Shot Manipulation
Mohamed Abouagour, Byung-Cheol Min
TL;DR
A robot must decide whether a few probes provide enough evidence before an irreversible action, but belief coverage and failure-score accuracy are distinct conditions. The paper audits both offline on a deployed probe-then-commit pipeline. Across task families and engines, probing reduces miss or failure, while audit-guided correction improves insertion outcomes.
Problem
A robot needs to decide when a few probes provide enough evidence to commit safely to one irreversible action, with both decision-coordinate coverage and failure-score accuracy at issue.
Source: §I · p. 1Method
The paper separately audits decision-coordinate coverage and mean optimism of the failure score on ground-truth instances using a deployed particle-belief, scenario-score, probe-then-commit pipeline.
Source: §I · p. 1, §IV · p. 3Results
Across seven task families and three engines, a few probes at a fixed executor reduce miss or failure; correcting the insertion tap boundary cuts failure from 0.354 to 0.112 on untouched instances.
Source: §I · p. 1, C. Audit-guided correction and… · pp. 5–6Takeaways & Limitations
Audit signatures can guide model investigation, but they do not uniquely identify the cause.
Source: §VII · p. 8The protocol assumes task state can be restored between probes, and its conformal guarantee covers the decision-coordinate interval marginally under exchangeability, not failure risk conditional on commitment.
Source: §VII · p. 8
Abstract
from arXiv · showhide
A robot that probes a few times before one irreversible action, such as tapping a surface before inserting a peg, must decide when the evidence is enough to commit. We argue that this decision rests on two conditions that existing methods do not separate: the belief must still cover the truth in the coordinate that decides the action, and the failure model that scores actions must track realized failure. We audit both conditions separately, offline and with ground truth, on a deployed probe-then-commit pipeline: a particle belief, a scenario failure score, and one commit. On simulated insertion, more taps sharpen the belief while the truth leaves its support on 16.9% of episodes and the failure score turns optimistic by 0.31. Conformal calibration restores coverage but not the decision: confidently wrong instances still pass a confidence gate. The audit's signatures instead point at the observation model, where a hand scan finds a 2.1 mm error in the tap boundary. Correcting that one number cuts failure from 0.354 to 0.112 on untouched instances and transfers unrefitted to a second engine, while in a third engine the same audit suggests an execution-model mismatch instead. Across seven task families in three engines, a few probes at a fixed executor reduce miss or failure. On a physical arm inserting a tool into a rigid pocket by touch, the gain and the audit's two conditions reproduce, and replaying the recorded taps under an injected model error shows the audit's signature on real data. Additional materials are available at https://sites.google.com/view/auditbeforeyoucommit.
I. INTRODUCTION
The paper studies when a few probes provide enough evidence for a robot to commit to one irreversible action. It separates the audit into whether the belief covers the action-deciding truth and whether the executor’s failure score tracks realized failure.
I. INTRODUCTION
- The motivating setting is a few exploratory interactions before an irreversible action, where the prior can span positions that call for incompatible aims.Source: §I · p. 1
- One-shot commitment depends on both decision-coordinate coverage and a failure score that tracks realized failure; calibrating either condition does not repair the other.Source: §I · p. 1
- A hand scan found a 2.1 mm tap-boundary error; correcting it cut failure from 0.354 to 0.112 on untouched instances and transferred unrefitted to PyBullet.In PhysX, the audit instead suggested a remaining capture-model mismatch.Source: §I · p. 1
- The paper audits both conditions separately and offline on a deployed pipeline using a particle belief, scenario failure score, and single commit.Source: §I · p. 1
- The audit-guided model correction is independently confirmed on untouched instances, tested across engines, and accompanied by a physical-arm touch-probe run.Source: §I · pp. 1–2
II. RELATED WORK
Related work covers policies trained over parameter distributions, informative interaction, calibration, and model criticism. This paper focuses on auditing decision-coordinate coverage and executor-score optimism separately.
II. RELATED WORK
- Domain randomization trains policies over physical-parameter distributions, while history-conditioned variants infer dynamics from interaction history.Source: §II · p. 2
- Active-identification methods select informative interactions, and Phys2Real conditions on probed estimates with their uncertainty.Source: §II · p. 2
- Split conformal prediction provides marginal coverage under exchangeability; prior robotics work calibrates future configurations, whereas this paper calibrates a latent instance’s decision coordinate [23].Source: §II · p. 2
- Filter consistency and Bayesian model criticism test whether a model explains observed data, a condition that can pass while the paper’s two audit conditions fail.Source: §II · p. 2
III. THE PROBE-THEN-COMMIT PIPELINE
The pipeline draws a latent instance, probes and updates a particle belief, then commits once using a retained-particle scenario objective. It compares active probing with no-probe baselines under matched risk objectives.
III. THE PROBE-THEN-COMMIT PIPELINE
- An episode draws latent parameters from an environment distribution, updates a belief from noisy probe outcomes, and commits after k probes to one action with task loss.Source: §III · p. 2
- Blind-Bayes and Blind-Robust take no probes under expected-loss and set-based objectives, while Probe-Robust(k) probes before using the set-based objective.Source: §III · p. 2
- The particle filter weights particles by observation likelihood, resampling and jittering them when effective sample size falls below half the particle count.Source: §III · p. 2
- The insertion model uses hole position and clearance as latent parameters, with a tap boundary and capture half-width defining modeled observations and failure.Source: §III · pp. 2–3
- For binary insertion and grasp losses, the executor minimizes the fraction of retained particles that fail; this scenario score is not a posterior failure probability or certified bound.Source: §III · p. 3
- Probe duration is excluded from task loss, and frozen miss curves price budgets of zero, two, or five probes to identify when probing is worthwhile.Source: §III · p. 3
IV. OFFLINE AUDIT AND CALIBRATION
The offline audit separates coverage of the action-relevant truth from agreement between modeled and realized failure, then calibrates coverage and adjusts action scoring. Its history-based gate analysis is limited to the evaluated gates and probe policy.
IV. OFFLINE AUDIT AND CALIBRATION
- Coverage is measured in the action coordinate, including across repeated episodes; support escape occurs when the truth lies outside the range of all current particles.For insertion, the decision coordinate is hole position.Source: §IV · p. 3
- The episode score is the shortest retained particle mass whose coordinate interval covers the truth, and it exceeds 1 when the truth lies beyond the particle hull.Clipping the score at 1 would hide support escapes.Source: §IV · p. 3
- Calibration uses each instance’s worst-repeat score, or worst score across prespecified budgets, to set thresholds with marginal coverage under exchangeable calibration and test clusters.The stated guarantee covers all repeats, and the simultaneous construction covers all selected budgets.Source: §IV · p. 3
- A validation-tuned margin shrinks the capture window to address score optimism, but it changes action selection rather than the conformal score or retained set.The margin is not distribution-free.Source: §IV · p. 3
- For identical history laws and disjoint success sets, any history-based gate with positive commitment has a committed-episode failure lower bound; the experiments do not prove impossibility for every gate.Changing the probe policy can change history laws, unlike correcting the observation model for a fixed history.Source: §IV · p. 4
A. Testbeds and comparisons
The evaluation spans seven task families and a cross-engine pushing transfer, with comparisons designed to separate changes in risk functional from the effect of probing. The reported paired reductions exclude zero, while grasp is descriptive and the matched GRU comparison is within a prespecified equivalence margin.
A. Testbeds and comparisons
- The setup covers seven task families across MuJoCo, SAPIEN (PhysX), and PyBullet, with PyBullet used for the 2-D pushing transfer.Table I lists engines, latents, probes, actions, noise, budgets, instance counts, and metrics.Source: Table I · p. 4, A. Testbeds and comparisons · p. 4
- The insertion study uses nonoverlapping draws from one prior box, with four repeats; Split B separates calibration from test, and Test C evaluates after correction freeze.Split B has 96 calibration and 96 test instances per protocol; Test C has 96 per protocol.Source: A. Testbeds and comparisons · p. 4
- Original-model cover-schedule failure is 0.234, 0.255, and 0.326 on instance sets A, B, and C, respectively—not three procedures.Set C was drawn after the correction froze.Source: A. Testbeds and comparisons · p. 4
- The frozen cover schedule was selected from 126 five-position subsets on 24 development instances; Split A evaluates it on held-out instances.Its sequence is [−12, −6, 0, 6, 9] mm.Source: A. Testbeds and comparisons · p. 4
- Table II compares blind and probed methods across families; changing Blind-Bayes to Blind-Robust changes only the risk functional, while adding probes changes only information.The table reports uncalibrated methods, with five probes except two for shuffleboard and eight for grasp.Source: Table II · p. 4
- The three paired reductions exclude zero; grasp improves for three of six feasible truths but is descriptive, and a matched GRU is equivalent within the prespecified ±0.020 m margin.The GRU comparison is 0.153 against 0.157 m, with difference −0.004 m and 95% CI [−0.008, +0.000].Source: A. Testbeds and comparisons · pp. 4–5
B. The insertion audit
In simulated insertion, the original belief loses decision-coordinate coverage as probes accumulate, while its failure score becomes increasingly optimistic. Calibration brings cluster coverage near its target and reduces mean optimism, but calibration has a measurable failure cost.
B. The insertion audit
- Support escape rises from 5.5% at k = 1 to 16.9% at k = 8, while mean optimism e8 reaches +0.31 in the original model.Calibration brings cluster coverage within a few points of the 0.80 target and lowers mean optimism from 0.216 to 0.048.Source: B. The insertion audit · p. 5
- Episode coverage falls from 0.94 at k = 1 to 0.77 at k = 8, below the nominal 0.80 target.Cluster coverage requires all four repeats of an instance to be covered.Source: B. The insertion audit · p. 5, Table III · p. 5
- On Split B, failure is 25.5% with five probes, 26.0% with set calibration, and 30.2% with set calibration and margin.The full adjustment increases paired failure by +0.047 [+0.005, +0.091].Source: B. The insertion audit · p. 5, Table IV · p. 5
C. Audit-guided correction and cross-engine transfer
A hand scan found the analytic tap boundary 2.1 mm inside the simulator’s, causing resampling to remove the true hole from the belief. Correcting that boundary reduced failure on untouched instances; cross-engine audits reproduced the correction’s benefit in PyBullet but pointed to a different likely mismatch in PhysX.
C. Audit-guided correction and cross-engine transfer
- The analytic boundary is 2.1 mm inside the simulator’s; after a +4.5 mm reading, its hypotheses exclude the true hole while the corrected model retains it.The correction changes only this boundary.Source: Figure 4 · p. 5, C. Audit-guided correction and… · p. 5
- On untouched instances, correcting the boundary reduces failure by 0.242 at random k = 8 and by 0.167 on the cover schedule.The frozen correction eliminates final-budget escapes, but random k = 8 failure remains 0.112, an unconditional rate.Source: C. Audit-guided correction and… · p. 5
- In PyBullet, the audit reproduces the escape-and-optimism signature, and the unrefitted MuJoCo offset cuts failure from 0.307 to 0.109.The PyBullet-specific scan yields no further reduction.Source: C. Audit-guided correction and… · pp. 5–6
- In PhysX, coverage remains above the 0.80 nominal while optimism is +0.369 and failure is 0.474, consistent with a capture-model mismatch that was not separately verified.Neither the MuJoCo offset nor PhysX’s own boundary offset resolves the optimism or failure.Source: C. Audit-guided correction and… · p. 6, Table VI · p. 6
D. Physical-arm run
On a physical arm, touch-based probing improved insertion performance while retained-set coverage and predicted-versus-realized failure agreement held. Replaying recorded taps with injected faults reproduced the audit’s signatures on real data.
D. Physical-arm run
- The tool entered on 19 of 26 one-tap commits and 24 of 24 five-tap commits, while mean aim error fell from 12.6 to 3.6 mm.Both changes were resolved by a cluster bootstrap over positions.Source: D. Physical-arm run · p. 6, Table VII · p. 6
- Across 118 episodes, the truth stayed in the retained set on every episode, and predicted failure tracked realized failure at every budget.Optimism stayed within 0.12 of zero, with every interval covering zero.Source: D. Physical-arm run · p. 6, Table VII · p. 6
- On recorded taps, an inward boundary error resolves the optimism condition from 8 mm upward, while coverage holds to 14 mm under the wide prior.With the prior narrowed to ±12 mm, coverage falls to 0.61, support escape is 0.39, and optimism is +0.30.Source: D. Physical-arm run · p. 6, Figure 5 · p. 6
VI. ANALYSIS
The analysis compares deployable commit signals by failure among committed episodes and commit rate, then tests how different model mismatches appear in audit metrics. The resulting signatures distinguish failure modes, but tested robust filters repaired support without repairing the decision.
VI. ANALYSIS
- On the cover schedule, area under the failure-versus-commit-rate curve is 0.293 for the raw signal, 0.306 for posterior variance, and 0.407 for the adjusted signal.Under random probing, posterior variance is lowest at 0.325; these curves compare combined scoring and execution procedures.Source: §VI · p. 7, Figure 6 · p. 7
- At 60% commitment on the cover schedule, the raw signal claims 0.017 failure but realizes 0.313, while the adjusted signal claims 0.075 and realizes 0.359.The hindsight oracle reaches zero failure at an 89% commit rate; the evaluated gates retain confidently wrong instances at four times the rate of the rejected pool.Source: §VI · p. 7, Figure 6 · p. 7
- Boundary error escapes rise from 0.07 to 0.17 across budgets as optimism rises from +0.07 to +0.30; a truncated prior instead escapes on 0.25 of episodes from the first tap.Particle impoverishment raises support escape while score residual improves, whereas a frame offset leaves coverage at 1.00 while executor optimism grows and failure reaches 0.26.Source: §VI · p. 7, Figure 7 · p. 7
- Known-fault correction reduces failure by −0.281 for boundary error and −0.161 for frame offset, while particle impoverishment’s −0.026 change does not establish a reduction.The widened-prior correction’s 0.18 failure rate is measured on the wider task, not as evidence of an uncorrected prior fault.Source: §VI · p. 7, Table VIII · p. 7
- Under the boundary error, the mixture and rejuvenation filters reduce support escape but do not establish failure reductions, with optimism remaining above +0.29.When the model is right, the mixture increases failure from 0.102 to 0.167; among alternatives tested, only known-fault correction repairs both conditions.Source: §VI · pp. 7–8
VII. LIMITATIONS AND CONCLUSION
The conclusion argues that decision-coordinate coverage and mean executor optimism must be audited separately, with calibration and model inspection both needed. Its limitations bound the evidence by the hardware setup, restoration protocol, conformal guarantee, and engine-transfer results.
VII. LIMITATIONS AND CONCLUSION
- Separate audits of decision-coordinate coverage and mean executor optimism reveal distinct failures; calibration restores coverage, but evaluated commit gates still select confidently wrong episodes.The conclusion says calibration and inspection are needed together.Source: §VII · p. 8
- On hardware, the slender tool, shared tap sensor, and wide prior make coverage a weaker test than in simulation.Source: §VII · p. 8
- The protocol assumes task-state restoration between probes, as in a fixtured cell; cross-engine tests assess solver portability, not the reality gap.Source: §VII · p. 8
- The conformal guarantee is marginal coverage of the decision-coordinate interval under exchangeability, and it certifies neither failure-score accuracy nor commitment-conditional risk.Source: §VII · p. 8
- A frozen correction held on new instances and transferred to one of two engines, while audit signatures guide investigation without identifying a cause.The physical-arm run inserts a tool into a rigid pocket, and recorded taps show the audit signatures under injected faults.Source: §VII · p. 8