PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

Chengyue Huang, Khang Vo Huynh, Mellon M. Zhang, Sebastian Elbaum, Zsolt Kira, Lu Feng

arXiv:2605.12386v3cs.RO

TL;DR

Task success alone does not show whether manipulation was safe, because unsafe behavior can occur along the way and temporal requirements can span an execution. SafeManip evaluates low-level rollouts with reusable LTLf properties, ManipVerse semantics, and per-trigger metrics. Across seven policies and two suites, the results show that task capability and temporal safety can diverge, including successful rollouts with violations.

  • Problem

    Task completion and isolated unsafe-state checks do not capture safety requirements that depend on how behavior unfolds over time.

    Source: §1 · p. 1

  • Method

    SafeManip monitors low-level manipulation rollouts with reusable LTLf safety properties grounded in execution predicates and ManipVerse object semantics, and scores violations per triggered event.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Results

    Across seven policies on RoboCasa365 and LIBERO, task capability and temporal safety dissociate: higher success does not imply safer execution, and some successful rollouts contain unsafe behavior.

    Source: §1 · p. 2, §5.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    SafeManip provides a reusable way to evaluate how safely manipulation is executed, rather than only whether tasks are completed.

    Source: §1 · p. 2, §6 · p. 9

    Current predicate grounding uses privileged object poses and contact lists, so hardware deployment requires onboard-sensor grounding and evaluation of monitor reliability under perceptual error.

    Source: §6 · p. 9

Abstract

from arXiv · show

Robotic manipulation is typically evaluated by task success, but reaching the correct final state does not guarantee safe execution. A robot may succeed despite contamination, premature release, or incorrect sequencing. These failures are poorly captured by task-completion metrics or isolated state checks because safety often depends on how behavior unfolds over time. We introduce SafeManip, a benchmark for evaluating temporal safety directly from low-level manipulation rollouts across eight physical and semantic categories. SafeManip grounds symbolic manipulation predicates into reusable temporal properties over finite executions, formalized with LTLf. We further introduce ManipVerse, a property ontology enabling semantic safety rules generalize across objects, tasks, and environments. Finally, SafeManip introduces per-trigger safety metrics that normalize violations by the safety-relevant opportunities that activate each obligation, accounting for differences in how often those opportunities arise across policies and tasks. We instantiate SafeManip on RoboCasa365 and LIBERO to audit seven state-of-the-art robot foundation-model policies. Our results reveal a gap between task capability and temporal safety: higher task success does not necessarily imply safer execution, successful rollouts can still contain violations, and failure patterns vary systematically across safety categories, task horizons, and manipulation suites. SafeManip provides a reusable layer for evaluating how safely manipulation tasks are executed, not only whether they are completed.

1 INTRODUCTION

Task success alone cannot establish safe manipulation: a robot may reach the goal through unsafe contact or incorrect sequencing, while isolated state checks miss requirements that unfold over time. SafeManip evaluates realized rollouts with reusable temporal properties and per-trigger metrics, then audits seven policies across RoboCasa365 and LIBERO.

1 INTRODUCTION

  • Task-completion metrics miss unsafe execution paths, including contamination, unstable manipulation, or incorrect sequencing that still reach the correct final state.Safety can depend on how behavior unfolds, not just isolated unsafe events.Source: §1 · p. 1
  • SafeManip combines direct monitoring of realized low-level execution with expressive temporal safety specifications.Its LTLf formulation evaluates how behavior unfolds over finite executions.Source: §1 · p. 1, Figure 1 · p. 2
  • The framework spans eight physical and semantic safety categories and uses ManipVerse object semantics to make safety specifications reusable across tasks and environments.Per-trigger metrics normalize violations by safety-relevant opportunities, accounting for differences in how often they arise across policies and tasks.Source: §1 · p. 2
  • Higher task success does not necessarily imply safer execution; SafeManip’s seven-policy audit finds distinct category-specific failure profiles and safety variation by task horizon and suite.Successful rollouts can still contain violations.Source: §1 · p. 2

2 RELATED WORK

Prior safety benchmarks evaluate either realized physical execution or semantic and procedural requirements, leaving a gap between low-level rollout grounding and temporal expressiveness. SafeManip applies temporal safety specifications directly to realized manipulation behavior.

2 RELATED WORK

  • Physical-execution benchmarks ground safety in robot behavior but commonly use local predicates or accumulated costs, limiting expression of dependencies spanning execution stages.Table 1 compares benchmarks by evaluation target, safety assessment, scope, and capabilities.Source: §2 · p. 2, Table 1 · p. 3
  • Semantic and procedural approaches express prerequisites, action order, and thresholds, but generally evaluate plans, reasoning traces, or controller-mediated behavior rather than continuous low-level rollouts.Table 1 distinguishes local and temporal physical and semantic safety.Source: §2 · p. 2, Table 1 · p. 3
  • SafeManip bridges these directions by applying expressive temporal specifications to realized low-level manipulation behavior over finite execution traces.The comparison identifies both the representation evaluated and the expressiveness of each benchmark’s safety specification.Source: §2 · p. 2, Table 1 · p. 3

3 METHOD

SafeManip connects object semantics and per-frame execution predicates to reusable LTLf requirements over complete traces, then monitors those requirements online. Its design includes an ontology for cross-object generalization and recovery-aware monitors for repeated violations.

3.1 BACKGROUND: LTLf SPECIFICATION LANGUAGE

  • LTLf extends propositional logic with temporal operators that specify properties over finite traces, such as conditions holding throughout or until another condition becomes true.A trace satisfies a formula when the formula holds over that finite sequence of proposition sets.Source: §3.1 · p. 3

3.2 MANIPVERSE: ONTOLOGY OF OBJECT PROPERTIES FOR MANIPULATION

  • ManipVerse assigns objects properties across kind, hazard, affordance, and role dimensions, allowing semantic rules to generalize beyond individually enumerated objects.It covers 262 entities, 60 properties, and 1,569 entity-property associations; relational rules are composed over property combinations.Source: §3.2 · p. 3

3.3 SAFEMANIP FRAMEWORK

  • At each timestep, SafeManip grounds robot, contact, motion, gripper, fixture, and action signals into predicates, binds them to task-relevant entities, and composes them into reusable LTLf requirements over complete executions.The eight-category suite includes persistent constraints and multi-stage temporal requirements; persistence, grace, and timeout windows address transient simulator noise.Source: §3.3 · pp. 4–5, Table 2 · p. 5

3.4 MONITORING PROCEDURE

  • Each LTLf requirement is compiled into a DFA that monitors the grounded predicate trace; after a violation, a recovery formula lets the monitor reset and detect further events.This supports recording repeated violations, recovery events, and violation duration within a rollout.Source: §3.4 · pp. 5–6, Figure 4 · p. 5

4 EVALUATION IMPLEMENTATION AND PROTOCOL

The evaluation audits foundation-model policies on kitchen-scale and tabletop manipulation tasks in two simulators. It reports task completion separately from safety and normalizes safety outcomes by triggered events.

4 EVALUATION IMPLEMENTATION AND PROTOCOL

  • The implementation releases property templates, task bindings, temporal monitors, rollout processing code, and evaluation scripts.The evaluation uses simulator state to ground safety predicates.Source: §4 · p. 6, §4.1 · p. 6

4.1 MANIPULATION TASK SUITE AND POLICIES

  • The evaluation covers RoboCasa365’s 50 kitchen-scale tasks and 40 LIBERO tabletop tasks across four suites, with seven policies tested across the two simulators.RoboCasa exercises the full safety-specification set, while LIBERO invokes a subset.Source: §4.1 · p. 6

4.2 EVALUATION METRICS

  • Safety metrics are normalized by triggered events, where a trigger occurs when a property’s antecedent becomes active and is then fulfilled or violated.The evaluation reports violations and unsafe-state exposure per triggered event, alongside task success rate.Source: §4.2 · p. 6

5 EXPERIMENTAL RESULTS

Across RoboCasa365 and LIBERO, task capability and temporal safety can diverge, while failure profiles vary by safety category and task structure. The benchmark-specific horizon results separate per-opportunity risk from violation duration.

5 EXPERIMENTAL RESULTS

  • The evaluation covers ten robotics policies and training variants across 50 RoboCasa365 tasks and four LIBERO suites (40 tasks), with 50 rollouts per task.Every rollout is monitored using the instantiated temporal safety properties and monitoring procedure.Source: §5 · p. 6

5.1 RQ1: RELATIONSHIP BETWEEN MODEL CAPABILITY AND TEMPORAL SAFETY

  • In RoboCasa365, the capability–safety gap persists after exposure normalization; failed episodes have 1.7–7× higher per-trigger risk, yet successful rollouts still violate temporal properties.Task success reflects the final goal, not whether the trajectory reaching it was safe.Source: Figure 5 · p. 7, §5.1 · p. 7

5.2 RQ2: TEMPORAL SAFETY FAILURES BY CATEGORY

  • Collision/contact leads by raw frequency, but per-trigger normalization puts its RoboCasa365 risk comparable to release stability and action-onset; cross-contamination instead dominates duration (0.48–1.01% of the rollout).Cross-contamination’s per-trigger rate exceeds one because the obligation accrues across an interval; the benchmark scripts it into tasks, so it is excluded from the color scale.Source: §5.2 · pp. 7–8, Figure 6 · p. 8
  • Policy profiles differ: OpenVLA is worst on collision risk (0.128) and duration (1.98%), while π0 and π0.5 concentrate risk on containment.OpenVLA also spikes on enclosure/access, with unsafe contacts more frequent and sustained than other policies’ contacts.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 RQ3: TEMPORAL SAFETY FAILURES BY TASK HORIZON AND MANIPULATION SUITE

  • On RoboCasa365, atomic-to-long per-trigger risk rises from ∼0.03 to ∼0.075 while exposure falls; LIBERO has no long-horizon tasks and shows no such degradation.The passage characterizes the horizon effect as specific to the task distribution, not a general law of VLA policies.Source: §5.3 · p. 8, Figure 7 · p. 9
  • Risk reaches 0.065 on RoboCasa365 in contact-rich suites, while successful plating/serving rollouts show higher risk (∼0.10) than overall (∼0.06) and higher exposure.Most suites’ risk drops when failed rollouts are removed; plating/serving is a marked inversion among successful rollouts.Source: §5.3 · pp. 8–9, Figure 7 · p. 9

6 DISCUSSION & CONCLUSION

SafeManip finds that task capability and temporal safety can diverge: unsafe behavior varies by category, policy, horizon, and manipulation suite. The evaluation remains limited to fixed checkpoints on two simulated benchmarks, while related work and policy audits are positioned around trajectory safety and policy-agnostic monitoring.

6 DISCUSSION & CONCLUSION

  • Task capability and temporal safety dissociate: the most capable policies are often least safe per opportunity, and unsafe behavior concentrates by category, policy, task horizon, and contact-rich manipulation.Successful rollouts can still reach the goal through unsafe behavior that a success metric scores as a clean win.Source: §6 · p. 9
  • Results describe fixed checkpoints on RoboCasa365 and LIBERO, not transfer of violation rates across robots or domains; hardware deployment also requires onboard predicate grounding and monitor-reliability assessment under perceptual error.The current predicates use privileged object poses and contact lists; a sim-to-real study would ground specifications from onboard sensors and recalibrate temporal windows.Source: §6 · p. 9

A EXTENDED RELATED WORK

  • Simulation and embodied-agent benchmarks commonly emphasize task completion rather than the safety of the trajectory that reaches the goal.The related-work survey frames trajectory safety as the gap SafeManip targets.Source: A EXTENDED RELATED WORK · p. 14
  • SafeManip audits realized rollouts without access to policy internals, and other models with the same interface can be evaluated under the same specifications.The covered policy families include vision-language-action, humanoid/generalist, world-model/video-based, and systems with memory, reasoning, or steering.Source: A EXTENDED RELATED WORK · p. 14

B POLICY CHECKPOINTS AND TRAINING VARIANTS

The evaluation uses externally provided policy checkpoints and distinguishes benchmark tracks, data, and checkpoint granularity when interpreting comparisons. The appendix also tests safety-prompt variants and provides detailed safety guidance for those prompts.

B POLICY CHECKPOINTS AND TRAINING VARIANTS

  • All evaluated checkpoints are externally provided; policies are not trained, fine-tuned, or adapted to SafeManip, and each rollout uses 50 episodes per task.That yields 2,500 episodes per policy on RoboCasa365 and 2,000 on LIBERO.Source: B POLICY CHECKPOINTS AND TRAINING… · p. 14
  • RoboCasa365's Multitask Learning track evaluates in pretraining scenes, whereas its Foundation Model Learning track evaluates in held-out target scenes with an unseen object set.The two rollout versions of the same multitask checkpoint are not interchangeable because the scenes and objects differ.Source: B POLICY CHECKPOINTS AND TRAINING… · p. 14
  • GR00T-pt, GR00T-to, and GR00T-tpt isolate pretraining, target-only fine-tuning, and target post-training over pretraining, respectively.These settings are target-split ablations, not direct comparators to the Multitask Learning policies; each aggregate spans three independently fine-tuned models.Source: B POLICY CHECKPOINTS AND TRAINING… · p. 15
  • On LIBERO, some policies use one mixture-trained checkpoint across 40 tasks, while GR00T N1.5 and OpenVLA use four per-suite checkpoints; no evaluated policy uses LIBERO-90.The checkpoints are evaluated as released, and GR00T N1.6's LIBERO checkpoint is a community release while GR00T N1.5's are documented in its repository.Source: B POLICY CHECKPOINTS AND TRAINING… · pp. 15–16

C PROMPT-BASED SAFETY GUIDANCE

  • Against GR00T-tpt's 43.9% success and 71.8% violation rate, the short safety prompt reached 26.4% success and 69.4% violations, while the long prompt reached 6.9% success and 65.1% violations.The authors treat prompt guidance as exploratory and note that its effect may depend on interaction with the policy's training distribution.Source: C PROMPT-BASED SAFETY GUIDANCE · p. 16
  • The safety prompts prioritize safe progress over completion, prescribing conservative action and responses such as pausing, retracting, or stopping when contact, grasp, contamination, containment, or access is uncertain.They specify checks for stable grasp and release, clear target regions, fixture mechanisms, containment, and enclosure access.Source: C PROMPT-BASED SAFETY GUIDANCE · pp. 16–17

D.1 ADDITIONAL METRICS

The appendix supplements per-trigger safety rates with rollout-level, count-based, duration, and coverage measures. These metrics distinguish whether a rollout is unsafe, how many violations occur, how long they persist, and how much of the task's safety structure was encountered.

D.1 ADDITIONAL METRICS

  • The rollout safety violation rate measures the fraction of episodes that violate at least one monitored property, while its per-property version measures the fraction violating a given property.This binary view does not distinguish a brief lapse from a sustained or repeated violation.Source: §D.1 · p. 17
  • Raw violation totals Vφ and triggered-event totals Nφ expose absolute failure volume and show how the per-trigger rate Vφ/Nφ is formed.These counts are aggregated over rollouts for each property.Source: §D.1 · p. 17
  • Unsafe-state exposure is the fraction of rollout timesteps spent violating safety, while violation duration is the mean number of violating timesteps per episode.Exposure is not opportunity-normalized and can be worse for a policy encountering more safety-critical situations; duration is comparable across policies only when rollout lengths are similar.Source: §D.1 · p. 17
  • Triggered-event coverage measures the share of a task's reference-demonstration properties that a rollout actually triggers, flagging apparently safe but unengaged execution.A policy that stalls early triggers few obligations, so its per-trigger scores may reflect a small, unrepresentative slice of the task.Source: §D.1 · p. 17

D.2 TRIGGERED EVENTS AS THE EVALUATION INSTRUMENT

Triggered-event counts track task complexity but also distinguish flailing from progress, supporting their use as the denominator for per-opportunity safety rates. The results show that successful rollouts can remain unsafe, while failure risk, category exposure, and repeated violations vary across outcomes and categories.

D.2 TRIGGERED EVENTS AS THE EVALUATION INSTRUMENT

  • On RoboCasa365, task-level triggered events rise with subtask count (ρ = +0.79, p < 0.001 for successful episodes), while distinct-property coverage also rises (ρ = +0.68, p < 0.001).At matched complexity, failures trigger more events overall but cover fewer distinct properties.Source: §D.2 · p. 18, Figure 8 · p. 18
  • Across both benchmarks, policy-level triggered events per episode anti-correlate with success (pooled r = −0.97, p < 0.001), while property coverage correlates positively (pooled r = +0.98, p < 0.001).Figure 9 plots both trends; the authors interpret the event-count decline with capability as less flailing, supporting its use as the per-opportunity denominator.Source: §D.2 · p. 18, Figure 9 · p. 18

D.3 RQ1: ROLLOUT OUTCOMES, PER EPISODE AND PER OPPORTUNITY

  • Unsafe rollouts comprise 29–42% of RoboCasa365 episodes and 3–27% of LIBERO episodes, with unsafe episodes present on both benchmarks.RLDX-1 has the largest safe-and-successful share on RoboCasa365; Figure 10 decomposes episodes by safety and task outcome.Source: §D.3 · p. 18, Figure 10 · p. 19
  • Failed episodes carry ∼1.7× to ∼7× the per-trigger risk of successful episodes, yet successful episodes retain nonzero per-trigger risk of 0.007–0.041.The outcome split appears across every policy and benchmark; the figure reports violations per triggered event by outcome.Source: §D.3 · p. 18, Figure 10 · p. 19

D.4 RQ2: EXPOSURE AND VIOLATION MULTIPLICITY BY CATEGORY

  • Collision/contact dominates RoboCasa365 exposure at 8.3–18.8 triggered events per episode, while release stability occurs 1.5–2.2 times per episode.Figure 11 reports category exposure per rollout; the exposure imbalance helps explain why per-episode rates and per-trigger risks rank categories differently.Source: §D.4 · p. 18, Figure 11 · p. 19
  • Among violating instances on RoboCasa365, collision/contact averages 3.1–4.5 violations and cross-contamination 2.6–3.2, whereas most other categories have a multiplicity of 1.0.Figure 12 measures repeat violations of the same obligation within an episode.Source: §D.4 · p. 19, Figure 12 · p. 19

D.5 RQ3: TASK-STRUCTURE EFFECTS UNDER THE PER-EPISODE METRICS

Per-episode metrics show safety violations rising with task horizon on RoboCasa365, while LIBERO remains below 11% across its available horizons. The plating/serving inversion also persists: successful rollouts can spend more time in violation than all rollouts.

D.5 RQ3: TASK-STRUCTURE EFFECTS UNDER THE PER-EPISODE METRICS

  • RoboCasa365’s per-episode safety violation rate rises from 12% on atomic tasks to 60% on long-horizon tasks; LIBERO remains below 11% at every available horizon.The authors attribute the RoboCasa365 gradient to longer executions accumulating more safety-relevant actions, and note that LIBERO has no long-horizon tier.Source: §D.5 · pp. 19–20
  • Successful plating/serving rollouts have higher unsafe-state exposure than all rollouts, indicating that some reach the goal through unsafe contact.This inversion is also visible in the per-episode view.Source: §D.5 · p. 20
  • Figure 13 groups safety violation rate and unsafe-state exposure rate by task horizon and category, separating all rollouts from successful rollouts.Group labels give RoboCasa365 and LIBERO counts; a dash marks a group absent from that benchmark.Source: Figure 13 · p. 20
  • Per-episode exposure can diverge from per-trigger exposure because longer horizons increase the denominator of rollout timesteps.Both views are reported to reflect these different denominators.Source: §D.5 · p. 20

D.6 ROBUSTNESS: PER-EPISODE METRICS VS. PER-TRIGGER NORMALIZATION

The robustness checks show that per-episode and per-trigger metrics preserve policy rankings while framing safety differently. The benchmark-dependent capability–safety pattern and category profile also remain visible under per-episode metrics.

D.6 ROBUSTNESS: PER-EPISODE METRICS VS. PER-TRIGGER NORMALIZATION

  • Both benchmarks yield the same policy ranking under per-episode safety violation rate and violations per triggered event.Per-trigger normalization changes the question to reliability per opportunity, not the leaderboard ordering.Source: §D.6 · p. 20, Figure 14 · p. 20
  • Under per-episode metrics, LIBERO policies are more successful and safer than RoboCasa365 policies, but within-suite correlations have opposite signs: r = +0.40 (n.s., n = 5) and r = −0.99 (p < 0.001), respectively.The analyses are reported separately by benchmark because pooling would average over opposite trends.Source: §D.6 · p. 20, Figure 15 · p. 21
  • The positive RoboCasa365 capability–safety trend strengthens under per-trigger normalization (r = +0.66, n.s.), indicating that exposure alone does not account for the gap.The passage describes the per-trigger scatter as testing whether progress merely creates more safety-relevant opportunities.Source: §D.6 · pp. 20–21
  • Collision/contact and cross-contamination dominate RoboCasa365 under per-episode metrics, while OpenVLA retains the LIBERO collision spike in both rate and exposure.Low containment and enclosure/access rates partly reflect when those monitors activate; per-trigger normalization removes this triggering asymmetry.Source: §D.6 · p. 21, Figure 16 · p. 21

D.7 HOW THE TRAINING PARADIGM AFFECTS SAFETY

With the model, tasks, and monitor held fixed, training paradigm changes success and violation composition but not the pooled per-opportunity violation rate. Human-demonstration checks support monitor calibration, with audited firings confirmed and most thresholds insensitive on these corpora.

D.7 HOW THE TRAINING PARADIGM AFFECTS SAFETY

  • Success triples from 17.8% to 54.7% across training paradigms, while violations per triggered event remain between 0.049 and 0.055 with overlapping intervals.The most capable paradigm still violates at least one property in 35.5% of episodes.Source: §D.7 · pp. 21–22, Figure 17 · p. 22
  • Adding target demonstrations shifts category composition: manipulation-quality violations decrease while task-semantic violations increase, offsetting in the pooled rate.Collision/contact and action-onset preconditions account for 88% of triggers and are flat across the three settings.Source: §D.7 · p. 21

E MONITOR FAITHFULNESS: HUMAN AGREEMENT AND SENSITIVITY

  • The monitor has low but nonzero violation rates on human demonstrations, and all 100 audited instances were human-confirmed.The appendix frames the demonstrations as a control for detecting a monitor that flags constantly.Source: E MONITOR FAITHFULNESS: HUMAN… · p. 22

E.1 CALIBRATION ON HUMAN DEMONSTRATIONS

The appendix reports low monitor firing on human demonstrations, human confirmation of audited violations, threshold sensitivity concentrated in forbidden-contact tolerance, and reviewable annotation and grounding procedures.

E.1 CALIBRATION ON HUMAN DEMONSTRATIONS

  • Human demonstrations show lower instance-level violation rates than evaluated policies: 0.73% vs. 6.6% on RoboCasa365 and 0.10% vs. 0.8% on LIBERO.The residual demonstration positives concentrate in the contact family, and all 100 audited violated or spot-checked satisfied instances were confirmed.Source: §E.1 · pp. 22–23, Figure 19 · p. 23

E.2 THRESHOLD SENSITIVITY

  • Six of seven predicate thresholds are flat across their sweeps; forbidden-contact tolerance agreement is 100% through 20 frames and falls to 75% at 40 frames and beyond.The default tolerance sits at the edge of the human-agreement plateau.Source: §E.2 · p. 23, Figure 20 · p. 24

E.3 THE ANNOTATION VIEWER

  • The annotation viewer pairs demonstration video with synchronized ground-truth state playback, filters episodes by corpus and task, and lists property instances with formulas and first violated frames.Reviewers can inspect verdict and predicate timelines with frame-stamped edges and per-object activation windows; annotations are stored for later agreement and sensitivity scoring.Source: Figure 18 · p. 22, §E.3 · pp. 23–24, Figure 21 · p. 24

F PREDICATE AND MONITOR IMPLEMENTATION DETAILS

  • The monitor uses three layers: object semantics, simulator-specific predicate grounding, and shared LTLf specifications compiled with recovery formulas; only predicate grounding is simulator-specific.It runs over recorded traces using privileged simulator state, enabling the same monitor to evaluate demonstrations and policy rollouts.Source: F PREDICATE AND MONITOR IMPLEMENTATION… · pp. 24–25

F.1 ENFORCED SPECIFICATIONS AND THEIR GROUNDING

This section describes SafeManip’s grounded temporal specifications, which separate obligations by trigger and recovery behavior and use bounded, smoothed rollout signals to avoid finite-trace and measurement-related misfires. It covers threshold choices and LIBERO’s coarser contact grounding and six specifications that never trigger, distinguishing active safety signal from vacuous satisfaction.

F.1 ENFORCED SPECIFICATIONS AND THEIR GROUNDING

  • Eventuality obligations use 100-frame timeouts, while safety-oriented until formulas use weak until and grasp-ending obligations allow a geometric re-grasp escape.These choices bound verdict timing, avoid treating a rollout ending mid-obligation as a violation, and excuse a transient contact dropout only when the gripper never separated from the object.Source: §F.1 · pp. 25–26
  • Grasp stability is enforced as separate synchrony and drop-must-be-a-release checks, preserving distinct triggers, violation counts, and recovery conditions.The first excuses a rollout that ends mid-hold and closes on the undebounced grasp-ending signal; the companion demands evidence of intent and uses a re-grasp escape to suppress contact flicker.Source: §F.1 · pp. 26–27
  • The enforced rules also adjust target conditions: φ6 requires retraction to begin, φ8 checks that the microwave is empty when an object enters, and hygiene is excluded from action preconditions.The paper keeps contamination in φ4 because scripted scene contamination could otherwise be counted both as a contamination violation and an unsafe action onset, while φ5 measures unsafe local action states.Source: §F.1 · pp. 27, 29, 30

F.2 DEBOUNCING, EDGES, AND LATCHES

  • The monitor uses exactly four smoothing devices, and each proposition combines same-frame signals with at most one device rather than requiring unbounded history.Table 5 gives window lengths in monitor frames, with each frame lasting 50 ms in both simulators.Source: §F.2 · p. 31, Table 5 · p. 31

F.3 THRESHOLD PROVENANCE

  • Global constants were set in three distinct ways, and only the first is described as evidence-derived.The supplied passage does not specify the other two ways.Source: §F.3 · p. 31
  • Thresholds use measured gaps where available and explicit tolerances or simulator-scale choices otherwise.The 5-frame persistence window falls between 1–2-frame noise runs and genuine stability periods of at least 7 frames; the 10-frame approach window filters brief near-passes. Contact tolerance is 20 frames (the 90th percentile), while the 100-frame settle timeout is not extended because 22% of the long tail never settles within recording. Load-bearing constants are assessed through sensitivity analysis and human-demonstration calibration.Source: §F.3 · pp. 31–32

F.4 CROSS-SIMULATOR GROUNDING: LIBERO

  • All 20 specifications, recovery formulas, and metric definitions transfer verbatim to LIBERO, with grounding modules translating its scene representation into shared predicates.Both suites record at 20 Hz at stride 1, so frame-count constants transfer unchanged; stability thresholds are converted from velocities to per-frame pose deltas. The gripper-closure constants cannot transfer because they use different parameterizations.Source: §F.4 · p. 32, Table 6 · p. 32
  • Six specifications never trigger on LIBERO because the corpus lacks the required faucet, push button, raw-attribute objects, and relevant fixture outputs or tasks.These are vacuous satisfactions, not passes: a specification that never triggers contributes no denominator under per-triggered-event normalization. The remaining 14 specifications show active signal; LIBERO also grounds allowed contact more coarsely as arm-link contact because its scene description lacks RoboCasa’s per-task entity roles.Source: §F.4 · pp. 32–33

Related papers

Loading 2605.12386v3…