Source-linked AI summary
Kinematic Constraint-Referenced Evaluation for Articulated Object Manipulation
Baptiste Bonin, Caro Strickland, Audrey Durand
TL;DR
Binary task success cannot show whether a commanded gesture matches an articulated object's kinematics, and richer evaluation measures can require additional task-specific information. CRS projects the commanded twist onto the joint axis to separate progress from off-constraint waste. In simulation, its two channels distinguish three transfer failures that success registers but does not attribute, and its verdicts remain stable under axis errors up to 30° when compared with an oracle scored on the same axis.
Problem
Binary success collapses perception, grasping, execution, and strategy into one score and can pass a completed task even when its commanded gesture violates the object's kinematics.
Source: §1 · p. 1Method
CRS probes commanded end-effector twists at oracle-selected reference states and projects them onto the joint axis to measure progress and off-constraint waste.
Source: §1 · pp. 1–2, §2 · p. 2Results
CRS's signed-progress and waste channels place three transfer failures in distinct cells that binary success registers but does not attribute.
Source: §3 · p. 4Takeaways & Limitations
CRS can complement per-stage success and trajectory stability in behavior evaluation when annotated joint axes are available.
Source: §4 · p. 4CRS assumes a rigid grasp and a single 1-DOF joint, is validated in simulation on three mechanisms, and scores command-side angular waste whose physical cost was not measured.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Manipulation policies are evaluated predominantly by binary task success, which cannot tell whether the commanded gesture is the one the object's kinematics admit. We introduce Constraint-Referenced Strategy (CRS), which projects the commanded end-effector twist onto the object's joint axis, separating progress along the joint from off-constraint waste. CRS requires a joint model and a joint-following scripted controller derived from it, but no recorded demonstration or learned judge. On 74 simulated articulated objects, success registers three transfer failures as lower rates but attributes none, while the two CRS channels give each a distinct signature. A door-trained behavior-cloning policy opens drawers in 59 of 110 episodes, and in these passes it commands about $40\times$ the off-axis rotation of a drawer-trained policy's passes ($1.27$ vs.\ $0.032$~rad/s). Relative to a scripted controller scored with the same axis, CRS verdicts are unchanged under joint-axis errors up to $30^\circ$; absolute thresholds tolerate about $8^\circ$.
1 Introduction
Binary success can reward a completed task even when the commanded gesture violates the object's kinematics, while richer evaluators often require demonstrations, task-specific stages, or a learned judge. CRS uses joint kinematics to measure progress and off-constraint waste in the command.
1 Introduction
- A door-trained policy can open held-out drawers with an inadmissible arcing gesture, yet binary success scores those episodes as passes.Success also compresses perception, grasping, execution, and strategy into one bit, so different failures can receive the same score.Source: §1 · p. 1
- Existing richer metrics may require per-object demonstrations, task-specific subgoals, or a trained judge, and do not provide a graded, signed reading of joint progress versus waste.Geometry checks can be pass/fail at a skill-specific tolerance, while task-independent command stability does not model the object.Source: §1 · p. 1
- CRS evaluates the commanded gesture against task kinematics, complementing existing outcome-focused evaluation.Under a rigid grasp, the joint defines which end-effector motions advance it; one screw axis encodes every valid opening mode.Source: §1 · p. 1
- Projecting the commanded twist onto the joint axis yields two channels: commanded progress along the joint and off-constraint waste.CRS requires neither a recorded demonstration nor a learned action model.Source: §1 · p. 2
2 Constraint-Referenced Strategy (CRS)
CRS probes a policy at reference states and evaluates its commanded twist against the target joint's geometry. It separates admissible joint progress from motion the constraint does not admit, with a one-step physical check on verdicts.
2 Constraint-Referenced Strategy (CRS)
- Teacher-forced probing evaluates the policy's commanded gesture at oracle-placed reference states, assuming the handle is already grasped at a valid pose.Reference states determine where the policy is evaluated; joint geometry determines what counts as a valid command.Source: §2 · p. 2
- For revolute joints, signed progress is angular velocity projected onto the joint axis, while waste is the magnitude of the perpendicular angular component.For prismatic joints, progress is translation along the axis and waste equals the angular-velocity magnitude; verdict-bearing channels depend only on the axis direction.Source: §2 · p. 2
- Each progress verdict is gated by executing the command for one simulator step and comparing its reading with realized joint displacement.Ordering uses R2, direction uses sign agreement above a noise threshold, and null verdicts use an equivalence band of 5% of the oracle rate.Source: §2 · p. 2
3 Experiments and Results
Across transfer failures, binary success can register lower rates without attributing the errors, while CRS's signed-progress and waste channels distinguish reversed, off-axis, and false-pass behavior. CRS verdicts remain stable under substantial joint-axis perturbation when compared with an oracle scored using the same axis.
3 Experiments and Results
- The evaluation tests OpenVLA-7B and from-scratch point-cloud BC in leave-one-mechanism-out generalization across 74 GAPartNet doors, drawers, and faucets in simulation.The categories comprise 39 doors, 22 drawers, and 13 faucets; comparison measures include success, realized displacement, command-side measures, demonstration DTW, and MetaFine stability.Source: §3 · pp. 2–3
- Signed progress and waste place three transfer failures in distinct cells, while success registers lower rates but attributes none.Alignment is unsigned, so it cannot distinguish a drawer pushed shut from one pulled open.Source: Table 1 · p. 3, §3 · p. 4
- Commanded rates track realized displacement with R2 = 0.69/0.89/0.65 for door/drawer/faucet, but calibration slopes below one make rates directional and relative.Oracle command and realized directions agree at 1.000 above the noise floor; the drawer-trained policy commands no revolute motion on doors, with |q̇ω| 12.9× smaller than ε.Source: §3 · p. 3
- Detection is defined against the target-mechanism scripted oracle: catch means beyond its 95% CI in the failure direction, while mislead means within that interval despite a wrong gesture.A miss lies beyond the interval in the other direction; CRS pairs signed-progress sign or null with whether waste is flagged.Source: §3 · p. 3
- A door-trained BC policy succeeds on drawers in 59 of 110 episodes, commanding 1.27 rad/s off-axis rotation versus 0.032 rad/s for the drawer-trained policy's successful episodes.The door-trained policy's off-axis waste is similar in successful and failed episodes: 1.27 vs. 1.26 rad/s.Source: §3 · p. 3
- On held-out drawers, OpenVLA commands closing on all three seeds, while on faucets it reaches 68% of oracle realized displacement despite waste 5.4× the oracle's.The drawer does not move within one step, so realized displacement shows a stall rather than the command-side reversal.Source: §3 · p. 3
- Verdicts for the three failures and null control remain stable through 30° axis perturbations relative to an oracle scored with the same axis.As an absolute threshold, the misaimed verdict tolerates about 8°; the lever-based channel's door R2 falls from 0.90 to 0.34.Source: §3 · p. 4
4 Conclusion and Discussion
CRS reads commanded gestures against object joints, attributing failures and exposing passes with inadmissible gestures. Its current assumptions, validation scope, and unmeasured physical cost constrain interpretation; the authors identify extensions to other joints and hardware.
- CRS can complement per-stage success and trajectory stability in MetaFine’s behavior dimension when joint axes are annotated.Source: §4 · p. 4
- CRS assumes a rigid grasp and a single 1-DOF joint; the faucet case suggests rigid grasp is not always met.Source: §4 · p. 4
- CRS scores angular waste only, is validated in simulation on three mechanisms, and has not measured waste’s physical cost.Source: §4 · p. 4
- Future work includes other lower pairs and real hardware, with proprioceptive progress, RGB-D axis estimation, and same-axis in-domain references proposed.Source: §4 · p. 4