PaperlayerView exact revision on arXiv

Source-linked AI summary

A Physics-Consistent Benchmark for Contact-Rich Human-Robot Interaction in Assistive Care

Chengxiao He, Shanghai Yuan, Liuqun Fan, Shenzhen Zhu

arXiv:2609.02402v1cs.RO

TL;DR

Task-level evaluation can miss contact failures in assistive care, where reaching a task goal does not establish physically valid interaction. This paper introduces a physics-consistent, manikin-calibrated bathing benchmark and finds that task completion can diverge from safety-gated success.

  • Problem

    Conventional task-level metrics do not capture contact-rich interaction challenges such as deformation, contact force, passive motion, and joint-limit constraints.

    Source: §I · p. 1

  • Method

    The benchmark combines a deformable, passively responding human, physics-aware scores alongside task success, a frozen vision-only/scorer-only protocol, and manikin-calibrated region-wise simulated responses.

    Source: §I · p. 1

  • Results

    Across methods, task completion did not imply physically valid contact; State Machine success fell from 102/140 to 79/140 under a safety gate.

    Source: §I · p. 1, §V · p. 8

  • Takeaways & Limitations

    The benchmark can serve as a pre-deployment physical/safety screen for contact-rich care policies.

    Source: §V · p. 8

    With 20 trials per task, the study characterizes the benchmark rather than providing population-level statistical estimation.

    Source: §V · p. 7

Abstract

from arXiv · show

Conventional task-level evaluation asks whether a robot policy completes a specified action, but can miss failures that emerge only during physical human contact. This limitation is critical in contact-rich assistive tasks, where meaningful evaluation requires a physically responsive human, interaction-quality assessment beyond task success, and a leak-free observer-scorer protocol. We introduce a physics-consistent benchmark for contact-rich human-robot interaction, instantiated in robot-assisted bathing. The benchmark combines a deformable, passively responding human, physics-aware scores alongside task-level success, and a frozen vision-only / scorer-only evaluation protocol. To establish physical validity, region-wise simulated responses are calibrated against force-indentation measurements from Franka impedance pushes on a medical-care manikin. Under a frozen T1-T7 protocol with 140 runs per method, an LLM-augmented state machine (State Machine) achieves 72.9% task success but drops to 56.4% after correct-region and force-safety screening; VoxPoser produces lighter and more stable contact but completes only 27.9% of trials; and zero-shot pi0.5 achieves 0.7% task success with no correct-region or safety-gated successes. These results show that task completion alone does not imply physically valid contact and motivate physics-aware screening before deployment of contact-rich assistive robot policies.

I. INTRODUCTION

The paper introduces a physics-consistent benchmark for contact-rich assistive-care interaction, instantiated in robotic bathing, to assess physical interaction quality alongside task success. Its results show that nominal task success can conceal failures revealed by post-hoc physical screening.

I. INTRODUCTION

  • Task-level metrics can miss excessive force, unstable contact, deformation, passive joint motion, and joint-limit risks, so task completion alone does not establish physically valid contact.These concerns arise because physical interaction with a human body differs from geometrically evaluated rigid environments.Source: §I · p. 1
  • Existing benchmarks may use motion-driven humans, emphasize task completion or collision measures –[8], or expose privileged simulator states that violate the intended observation contract.The introduction identifies these as gaps in human physical response, interaction-quality evaluation, and separation of policy-visible observations from scorer-only physical states.Source: §I · p. 1
  • The benchmark combines a deformable, passively responding human, manikin-calibrated contact responses, physics-aware scores alongside task success, and a frozen vision-only / scorer-only protocol.Simulated region-wise responses are calibrated against force–indentation measurements on a medical-care manikin; the paper frames this as physical grounding, not a standalone real-to-sim method or a new deformable solver.Source: §I · pp. 1–2
  • State Machine success falls from 102/140 to 79/140 under a safety gate, while π0.5 falls from 1/140 original success to 0/140 gated success.The screening distinguishes nominal task success from deployment-usable success.Source: §I · p. 1
  • The diagnostic evaluation covers State Machine, VoxPoser, and zero-shot π0.5 transfer, with method-specific failure modes in force, joint-limit margin, contact stability, and region grounding.The benchmark is instantiated in robotic bathing across T1–T7 Touch / Scrub / Push tasks.Source: §I · p. 2

II. RELATED WORK · A. Care Systems and Policy Evaluation · B. Why Task-Level Benchmarks Miss Contact

Prior work provides separate ingredients for care-task execution, simulation, safety criteria, and policy evaluation, but not a unified protocol that combines a calibrated, passively responding human with leak-free observations and physics-aware scoring.

A. Care Systems and Policy Evaluation

  • I-SUPPORT, Manip4Care, and VTTB demonstrate local bathing subtasks, but do not provide a reusable protocol for scoring physical contact under a frozen observation contract,,,.Source: A. Care Systems and Policy Evaluation · p. 2
  • VoxPoser and ReKep ground language in spatial value maps or keypoint constraints, while π0, π0.5, and OpenVLA target continuous control and open-set generalization,, [8], [15].General manipulation benchmark performance does not establish safe sustained contact with a deformable human body.Source: A. Care Systems and Policy Evaluation · p. 2

B. Why Task-Level Benchmarks Miss Contact

  • Human-Robot Gym offers a safety-aware HRC benchmark but uses a primarily motion-capture-driven human; the paper instead evaluates physical validity through real-to-sim agreement.XPBD, VBD, SoftMAC, skeleton–soft coupling, and Newton are treated as simulation infrastructure, not as engines to beat on timestep or convergence –,.Source: B. Why Task-Level Benchmarks Miss… · p. 2
  • ISO 15066 gives fixed collaborative force limits by body region, while Behrens et al. report regional pain-onset forces and PrioriTouch discusses conservative scaling for contact preference,,.These sources do not supply a reusable apparatus for scoring contact quality under a frozen observation contract.Source: B. Why Task-Level Benchmarks Miss… · p. 2
  • HRI workload reviews identify robot motion, rhythm, and predictability as relevant factors, while LIBERO and Open X-Embodiment are not designed around passive human joints or calibrated contact force,,.Source: B. Why Task-Level Benchmarks Miss… · p. 3
  • Existing systems do not jointly provide a deformable, passively responding human, calibrated contact response, leak-free policy observations, and physics-aware scoring in one evaluation protocol.The benchmark integrates these capabilities to assess whether successful contact-rich interaction is physically valid, rather than introducing another task or simulation engine.Source: B. Why Task-Level Benchmarks Miss… · p. 3

III. BENCHMARK DESIGN · A. Benchmark Environment

The benchmark defines physics consistency at the benchmark level: human responses must correspond measurably to real contact, contact-induced states must emerge dynamically, and policies must not receive privileged simulator information. Its environment uses deformable, passively responding human mechanics and vision-facing observations separated from physics and scoring state.

III. BENCHMARK DESIGN

  • Physics consistency requires measurable correspondence to real contact, dynamically generated force, deformation, and passive motion, and evaluation without privileged simulator information reaching the policy.The paper frames physical realism, physical validity, evaluation validity, and diagnostic capability as benchmark-level properties.Source: §III · p. 3

A. Benchmark Environment

  • The environment requires a physically responsive human rather than a kinematic target, with an observation interface that avoids simulator-state leakage.Source: A. Benchmark Environment · p. 3
  • The care recipient combines an articulated rigid skeleton with region-wise deformable soft-tissue shells, while passive joints allow contact forces to induce posture changes without an active human controller.Low-compliance parts such as fingers and the head may be rigid; load-bearing regions use flexible shells anchored to bone segments.Source: A. Benchmark Environment · p. 3
  • Contact deforms soft regions, transfers force to the skeleton through regional anchors, and can move passive joints, so policies encounter tissue-mediated forces rather than a rigid mannequin.Source: A. Benchmark Environment · p. 3
  • Physics state is separated from planner-visible perception: bones and shells drive mechanics, while an SMPL-X visual skin is driven from joint coordinates for rendering.A virtual camera and robot TCP pose provide the snapshot-query interface.Source: A. Benchmark Environment · p. 3
  • Policies receive RGB and depth, point clouds, zero-shot semantic masks [24], and vision-estimated skeletal landmarks [25], while ground-truth joint angles, anchor states, and scorer-only contact metadata remain closed.Subjects must infer semantic and spatial information from the vision-facing modalities.Source: A. Benchmark Environment · p. 3

B. Benchmark Tasks and Evaluation Dimensions

The benchmark evaluates continuous contact with deformable body regions across seven micro-tasks and three regions, pairing task-level assessment with physics-aware scoring and conservative force-safety gates.

B. Benchmark Tasks and Evaluation Dimensions

  • Seven micro-tasks span three soft-relevant body regions; upper-arm push is omitted because its short moment arm yields limited displacement diversity.Trials generally start with a supine care recipient on a nursing bed and randomized comfort-arm orientation; scrub means sustained sliding cleaning contact.Source: B. Benchmark Tasks and Evaluation… · p. 3, Table II · p. 4
  • Tasks use natural-language prompts naming region and intent, while excluding rigid-landmark, language-ambiguity, and conventional arm-repositioning tasks.Examples include gently touching or pushing the forearm and gently scrubbing the abdomen.Source: B. Benchmark Tasks and Evaluation… · p. 4
  • Scorer-only ground truth provides a task-level baseline and physics-aware scores, which describe physical contact but are not clinically validated care scores.The score dimensions are presented in Table III.Source: B. Benchmark Tasks and Evaluation… · p. 4
  • Pain-onset limits are 100 N for forearm muscle, 100 N for deltoid muscle, and 60 N for abdominal muscle; γ′=0.35 yields safety gates of 35 N, 35 N, and 21 N [19].Fpain and Fmax are two presentations of the same published limits, with Fmax applying a uniform conservative discount rather than representing an independent measurement.Source: B. Benchmark Tasks and Evaluation… · p. 4

C. Evaluation Protocol

The evaluation protocol gives each method a planner-visible snapshot and freezes the scorer–logger contract. It withholds ground-truth coordinates and scorer-only state from vision-facing methods, and future contact labels from scorer-only streams.

C. Evaluation Protocol

  • Each method consumes the planner-visible snapshot interface defined in Section III-A0c.The protocol freezes the scorer–logger contract for all trials in Section IV (Table IV).Source: C. Evaluation Protocol · p. 4
  • State Machine derives contact geometry from the initial RGB snapshot using conventional CV segmentation and scripted fitting of candidate keypoints and approach directions.It receives neither ground-truth coordinates nor scorer-only state.Source: C. Evaluation Protocol · p. 4
  • Scorer-only streams exclude future contact labels, while vision-facing methods receive neither ground-truth joint angles nor soft-anchor states.Source: C. Evaluation Protocol · p. 4

IV. EXPERIMENTS

The experiments define a frozen evaluation contract that pairs task-level scoring with physics-aware measures of contact, motion, and human-joint limits. Passive-joint response is required, not treated as an optional ablation.

IV. EXPERIMENTS

  • C3 is the task-level baseline, while C1, C2, C4, and C5 are physics-aware evaluation dimensions, with all quantities hidden from evaluated policies.The symbols match the scorer implementation.Source: Table III · p. 4
  • The safety-gate specification uses Fpain(b) and a uniform γ′ = 0.35 gate [19].Source: Table 8 · p. 4
  • The care recipient’s joint-limit measure is Dlimit = min(q −qmin, qmax −q), and closed-loop completion uses geometric/contact gates with failures attributed to perception, planning, execution, or safety truncation.These are human kinematic limits, not robot joint limits.Source: §IV · p. 4
  • The active-motion fraction ηact = Tact/Ttotal is measured after human settle to capture pacing and temporal demand.Source: §IV · p. 4
  • Contact stability is measured as within-run population SD of ∥Fc∥ over the contact window, with lower values indicating more stable contact.Force–indentation coupling is reported in Section IV-B rather than as a separate deformation score.Source: §IV · p. 4
  • The unified protocol freeze requires all methods and selected tasks to share the evaluation contract before metric aggregation.The supplied passage identifies this contract but does not enumerate its full contents.Source: Table IV · p. 4
  • Passive-joint response remains a required care-recipient property rather than an optional ablation switch.Source: §IV · p. 4

A. Benchmark Setup · B. Physical Validity of the Benchmark

The benchmark evaluates contact-rich bathing policies under a shared vision-only observation and logging contract, while checking simulated contact responses against manikin measurements. Its physical-validity claim is limited to calibrated forearm and abdomen responses; upper-arm scores are comparative rather than independently biomechanically validated.

A. Benchmark Setup

  • Scripted probes confirm that push depth raises C1, joint-envelope violations lower C2, stop–go execution lowers C4, and greater within-window force fluctuation raises C5.These probes validate scorer–logger responses, not the human-model calibration; policy runs share a seed and logging contract. Valid ratio denotes scoreable contact or trajectory data, not C3; binary rates use Wilson 95% confidence intervals and continuous metrics use medians.Source: A. Benchmark Setup · pp. 4–5
  • The full T1–T7 suite runs 20 initializations across seven tasks, yielding 140 runs per method for State Machine and VoxPoser policy paradigms.State Machine represents finite-state coordination; VoxPoser represents spatial reasoning.Source: A. Benchmark Setup · p. 5
  • The State Machine selects among bounded Cartesian motion primitives fitted from an initial RGB snapshot, while VoxPoser generates zero-shot 3D value-map trajectories.State Machine uses conventional CV segmentation and does not update target geometry during execution or access ground-truth simulator state or scorer-only signals; it is a structured snapshot-conditioned reference, not an end-to-end learned vision policy. Its finite-state coordination is consistent with assistive bathing systems such as I-SUPPORT.Source: A. Benchmark Setup · p. 5
  • The publicly released π0.5 DROID checkpoint [8] is evaluated zero-shot on contact-rich assistive care without teleoperation or fine-tuning, as a transfer stress test rather than a peer paradigm.The checkpoint is pretrained for Franka Panda generalization.Source: A. Benchmark Setup · p. 5

B. Physical Validity of the Benchmark

  • Real contact trials use a Franka Panda under Cartesian impedance control to push a compliant, movable-joint care manikin and record pose, force, and displacement for force–indentation curves.Physical recovery after unloading is not instrumented, so unload recovery and energy dissipation are excluded from the real-to-simulation claim.Source: B. Physical Validity of the Benchmark · p. 5
  • Matched forearm and abdomen push schedules show agreement between measured and calibrated force–indentation curves, supporting calibration of Flex parameters to those contact responses.The curves also show force–deformation coupling; deformation is not presented as a separate metric.Source: B. Physical Validity of the Benchmark · p. 5
  • Figure 2 records Touch, Scrub, and Push scenes on the forearm, upper arm, and abdomen alongside contact force, wrist displacement, and local deformation traces.The traces illustrate the scorer–logger contract, not a separate deformation metric.Source: Figure 2 · p. 5
  • The upper-arm simulation reuses forearm parameters, so upper-arm C1/C2/C5 scores are comparative benchmark results, not independent region-specific biomechanical validation.Manikin upper-arm compliance may differ from human tissue.Source: B. Physical Validity of the Benchmark · p. 5

C. Benchmark Evaluation

The benchmark separates task-level completion from correct-region, force-gated deployment success. State Machine’s high completion rate does not imply stable contact, and its gated success varies with the force threshold.

C. Benchmark Evaluation

  • Across γ′ from 0.20 to 1.00, State Machine’s gated success rises monotonically from 35.0% to 72.1%, while VoxPoser and π0.5 remain at 27.9% and 0.0%.The method ranking and gap between State Machine’s nominal and gated success persist across the sweep; γ′ = 0.35 is reported as a representative conservative point, not a precisely calibrated value.Source: C. Benchmark Evaluation · p. 5
  • 19.48 N median peak force and the second-worst C5 show that State Machine’s task completion did not establish stable contact or an architecture-neutral policy ranking.The comparison characterizes distinct failure modes rather than establishing an architecture-neutral policy leaderboard.Source: C. Benchmark Evaluation · p. 5
  • C4 is an operational stop–go continuity proxy: a higher active-motion fraction indicates fewer execution interruptions.Source: C. Benchmark Evaluation · p. 5
  • 72.9% [65.0, 79.5] task-level completion makes State Machine highest, but safety-gated success is 56.4% at γ′ = 0.35; VoxPoser reaches 27.9% and π0.5 0.0%.Safety-gated deployment success requires evaluator success, correct-region contact, and peak force within the γ′ = 0.35 gate.Source: C. Benchmark Evaluation · pp. 5–6, Table V · p. 6

INITIAL RGB SNAPSHOT AND PERFORMS NO EXECUTION-TIME VISUAL

VoxPoser’s light, stable contact often fails to complete force-demanding actions, while π0.5 achieves near-zero success and contacts only non-target regions. The evaluation also notes that State Machine uses an initial RGB snapshot without execution-time visual replanning, and that any anxiety effect of stop–go motion remains unmeasured.

Implementation

Limitations

  • The proposed link between stop–go motion and recipient anxiety is not measured; C4 quantifies motion continuity only, leaving the experiential hypothesis for future work.Robot-motion rhythm and predictability are workload-relevant in HRI.Source: INITIAL RGB SNAPSHOT AND PERFORMS NO… · p. 6

Results

  • VoxPoser has C1 1.46 N and C5 0.21 N, but achieves C3 only 27.9% of the time, mostly on Touch, because its motion is often too weak for Push or Scrub.State Machine and VoxPoser also differ in correct-region success and contact-force statistics; the specified contrasts are not a full pairwise screen and omit π0.5.Source: INITIAL RGB SNAPSHOT AND PERFORMS NO… · p. 6, Table VII · p. 6
  • π0.5 achieves 0.7% success, with 30 force runs and 6 Push-margin runs scoreable; every recorded contact is on a non-target body region.The passage says this near-zero success is consistent with sensitivity to zero-shot transfer from the DROID / Franka pretraining distribution into this setting. Its C1/C5 statistics come entirely from off-target-region contact, and its TCP trajectories fail to lock onto the instructed region.Source: INITIAL RGB SNAPSHOT AND PERFORMS NO… · p. 6

D. What Conventional Evaluation Misses

The diagnostics show that reaching the instructed region, completing the task, and sustaining effective contact are distinct outcomes. They also show that safety-oriented behavior can coexist with insufficient progress or force and contact failures.

D. What Conventional Evaluation Misses

  • VoxPoser’s safety-proxy objective can leave contact safe but ineffective; the passage proposes switching after contact to an explicit progress objective such as effective stroke or target indentation.It suggests separate control regimes for reaching and sustained contact rather than one value-map objective throughout.Source: D. What Conventional Evaluation Misses · pp. 6–7
  • 95.0% of forearm runs by both State Machine and VoxPoser reach the target region, yet VoxPoser succeeds on only 33.3% of that forearm task set; π0.5 records 0 target-region contacts at all three sites.Region contact is distinct from task completion; valid ratio measures scoreable-data coverage, not closed-loop success.Source: D. What Conventional Evaluation Misses · p. 6, Figure 4 · p. 7
  • 68/101 of VoxPoser’s region-correctness failures are attributed to insufficient progress, despite Touch contact comparable to State Machine and C4 motion activity of 0.883.The failure denominator is trials failing region correctness; high motion activity does not establish task progress.Source: D. What Conventional Evaluation Misses · p. 6, Figure 4 · p. 7
  • 102/140 State Machine trials fall to 79/140 under the safety gate, with 36/140 force exceedances despite its structured candidate generator and geometric scaffolding.Only a subset of the exceedances were originally successful and region-correct.Source: D. What Conventional Evaluation Misses · p. 7
  • 1/140 π0.5 trials succeed under the original evaluator, but 0/140 are correct-region and 0/140 are safety-gated successes.Its failures comprise 106/140 trials with no target contact and 30/140 with off-target contact, consistent with sensitivity to zero-shot transfer from the DROID / Franka pretraining distribution; the benchmark can screen care policies before deployment.Source: D. What Conventional Evaluation Misses · p. 7, Figure 4 · p. 7

V. CONCLUSION

Task completion can conceal physically invalid contact in contact-rich care, and the benchmark makes these dissociations measurable as a pre-deployment physical and safety screen.

V. CONCLUSION

  • The evaluated policies are limited to a LLM-augmented state machine, a spatial-reasoning planner, and one zero-shot Franka-DROID checkpoint in a single robotic-bathing scenario.State Machine generates bounded scripted Cartesian candidates from an initial RGB snapshot for LLM selection, without execution-time visual replanning or privileged simulator state.Source: §V · pp. 7–8
  • The study uses 20 trials per task for benchmark characterization, not population-level statistical estimation, and its manikin contact partner is not human tissue.The upper-arm simulation reuses forearm calibration, Kc/Dc sensitivity is not systematically studied, and the evaluated policy set is limited.Source: §V · p. 7
  • All established π0.5 contacts are off-target; regional outcome rankings use 60 forearm runs and 40 each for upper arm and abdomen, excluding process failures.Omitted process failures were State Machine 3/3/1, VoxPoser 1/3/1, and π0.5 1/2/1 for forearm/upper arm/abdomen, respectively.Source: §V · p. 8
  • Task completion does not imply physically valid contact: State Machine can finish with an unstable force tail, VoxPoser can stay light without sustained contact, and π0.5 can move continuously while off-target.The benchmark is intended to screen contact-rich care policies before deployment.Source: §V · p. 8

Related papers

Loading 2609.02402v1…