PaperlayerView exact revision on arXiv

Source-linked AI summary

Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems

Barak Or

arXiv:2606.00090v1cs.ROcs.AI

TL;DR

Physical AI systems can produce confident actions despite invalid state evidence or unmet physical constraints, while existing safety methods do not provide a complete authorization boundary for black-box systems. This review formalizes runtime authorization, distinguishes capability milestones from authorization evidence, and concludes that deployment requires independent checks for state validity, action feasibility, constraints, fallback, and auditability.

  • Problem

    Physical AI models may confidently propose physically consequential actions from invalid representations, while existing safety methods generally assume inspectable states, interfaces, safe sets, or policy boundaries.

  • Method

    The review develops a bounded runtime-authorization formulation, connects it to existing safety theory, and organizes guardrail functions and evaluation requirements around authorization decisions and runtime evidence.

  • Results

    Reported task success, hallucination reduction, transfer, and latency improvements demonstrate measurable capability progress but do not define a unified probability that an authorization decision is correct for a proposed action, state estimate, and constraint set.

  • Takeaways & Limitations

    Before physical commitment, the autonomy stack should independently establish valid state evidence, action feasibility, satisfied operational constraints, available fallback, and an auditable decision.

  • Takeaways & Limitations

    The review abstracts away many hardware-specific, perception-stack, and organizational details, derives evaluation requirements without introducing a benchmark or empirical guardrail evaluation, and treats runtime guardrails as one assurance layer rather than a complete solution.

Abstract

from arXiv · show

Physical AI systems increasingly map multimodal observations, language instructions, and learned world representations into physically consequential actions. Robotics foundation models, vision-language-action models, and world-model-based autonomous systems can condition decisions that move vehicles, robots, drones, and industrial machines. This transition exposes a safety problem that is not fully captured by conventional AI content moderation or by classical robot safety alone: a black-box model may issue a physically consequential action while appearing confident, plausible, and semantically aligned. The resulting failure can be silent, arising from sensor drift, occlusion, state-estimation error, distribution shift, hallucinated affordances, or invalid physical assumptions before downstream hardware controllers detect a violation. Across embodied foundation models, world models, robotics simulation, embodied safety benchmarks, safe control, runtime assurance, uncertainty estimation, verification, and guardrail evaluation, model capability and safety mechanisms have advanced along largely separate technical tracks. A recurring gap synthesized here is that no single stream surveyed in this review supplies a complete runtime authorization boundary between black-box Physical AI models and physical execution. The resulting analysis develops a bounded problem formulation, a definition of silent physical-action failure, a taxonomy of runtime guardrail functions, and evaluation requirements for comparing guardrails as Physical AI assurance mechanisms.

1 Introduction

Physical AI systems increasingly transform observations and instructions into actions that can commit heterogeneous hardware, creating a need to authorize proposed actions against current state and constraints before execution. The review frames an independent boundary between black-box model outputs and physical execution, while defining silent failures and synthesizing guardrail functions and evaluation requirements.

  • Problem setting: Physical AI models increasingly convert multimodal observations, language instructions, and learned representations into trajectories, policies, navigation decisions, and controller inputs.These systems can infer affordances, predict future states, choose trajectories, or command manipulators rather than merely describe scenes.
  • Authorization gap: The review motivates an independent authorization boundary that composes model proposals, state evidence, physical constraints, fallback behavior, and audit traces.It scopes the analysis to black-box VLA, world-model, and foundation-model-based autonomous systems whose outputs can become physical actions.
  • Silent failures: Silent physical-action failures occur when systems act with high apparent confidence on incorrect, incomplete, or physically invalid world representations.They can arise from corrupted or partial observations, distribution shift, sensor drift, occlusion, or hallucinated affordances without an explicit software crash.
  • Authorization gap: Existing confidence signals, semantic refusals, offline benchmarks, and hardware controllers each address part of safety but do not define the complete action-authorization boundary.The recurring deployment question is whether state evidence is reliable, the action is feasible, constraints are satisfied, and the decision can be reconstructed.
  • Problem setting: Runtime authorization asks whether a proposed physical action should be permitted in the current state under active physical, operational, and temporal constraints.Relevant evidence includes payload, contact, clearance, routing, terrain, balance, airspace, and energy limits across different platforms.
  • Review contributions: The paper defines a linked vocabulary, formalizes runtime authorization, synthesizes eleven research streams into a guardrail taxonomy, and derives evaluation requirements and a minimal authorization-event schema.The review proceeds from formalization and taxonomy through literature mapping, capability trends, silent failures, runtime authority, evaluation, synthesis, implications, limitations, and conclusion.

2 Problem Formalization and Theoretical Anchors

This section formalizes runtime action authorization as a decision interface between black-box Physical AI policies and physical execution. It connects uncertainty-aware checks, safety-control concepts, and authorization-event records into a common unit for evaluating proposed actions.

  • 2 Problem Formalization and Theoretical Anchors: The autonomy stack maps observation history and task goals through a learned policy into a proposed physical action.The policy maintains an estimated world state that may differ from the true state because of noise, occlusion, drift, latency, or distribution shift.
  • 2 Problem Formalization and Theoretical Anchors: Runtime authorization evaluates the proposed action against the estimated state, active constraints, runtime evidence, and fallback behavior.The authorization event records the observation history, action, state, constraints, evidence, decision, and fallback or recovery action.
  • 2 Problem Formalization and Theoretical Anchors: Model likelihood is not a safety certificate: fluent, high-probability, or semantically aligned actions still require state-validity, feasibility, operational, timing, fallback, and audit checks.The decision may authorize, modify, block, invoke fallback, or escalate rather than commit the original action.
  • 2 Problem Formalization and Theoretical Anchors: A silent physical-action failure occurs when the system accepts an invalid state-action pair while remaining operational rather than producing an explicit software crash.The formal definition concerns failures under physical, state, temporal, or operational constraints.
  • 2.1 Connection to Existing Safety Theory: Conservative uncertainty handling authorizes only actions whose constraints hold throughout an uncertainty set around the estimated state.As stale, occluded, or unreliable state estimates enlarge the uncertainty set, the runtime decision should move toward modification, fallback, escalation, or blocking.
  • 2.1 Connection to Existing Safety Theory: Safety filters can project a proposed continuous control input onto the set of actions satisfying active constraints, but this requires evaluable action, state, and constraint representations.This projection view aligns with safety-filter and predictive-safety-filter work while retaining an explicit interface assumption.
  • 2.1 Connection to Existing Safety Theory: Control barrier functions provide a strong physical-feasibility tool but do not alone validate state estimates, semantic acceptability, site policy, fallback, or audit records.The authorization interface composes barrier methods and related safety mechanisms with semantic, state, temporal, operational, fallback, and audit evidence.
  • 2.2 The Authorization Event as a Unit of Analysis: The authorization event is an implementation-independent unit for comparing systems by which parts of the action, state, constraints, evidence, decision, and fallback record they inform.A collision-clearance example shows that uncertainty-adjusted clearance below stopping distance should trigger blocking, modification, or fallback even for a plausible high-confidence motion.

3 Guardrail Taxonomy

The taxonomy organizes runtime guardrails as an interface for evaluating black-box Physical AI proposals before they become hardware commitments, without replacing standards, controller design, or certification.

  • The taxonomy specifies what a runtime authority should evaluate before a black-box Physical AI proposal becomes a hardware commitment.It organizes the authorization interface rather than replacing standards, controller design, or certification.

4 Literature Map and Interface Assumptions

The review maps literature to the action-authorization pathway and shows that surveyed fields rely on different interface assumptions that do not automatically compose into a complete runtime boundary.

  • The review selects work addressing action generation, uncertainty and invalid-state estimation, runtime safety mechanisms, and embodied safety evaluation.
  • Sources are coded by the authorization-event component they inform, including model proposal, state evidence, feasibility, constraints, decisions, fallback, evaluation, and audit evidence.
  • The synthesis covers eleven research bodies, including cross-embodiment learning, hallucination diagnostics, trajectory guardrails, and embodied safety benchmarks.
  • Across the surveyed literatures, no single family provides a complete unifying runtime layer linking model outputs, state evidence, physical constraints, and authorization.
  • VLA and embodied foundation-model works improve action generation, but task success, imitation quality, or fluency do not by themselves establish deployment-specific action admissibility.
  • CBFs and safety filters are strong when dynamics, state variables, and safe sets are explicit, but black-box action interfaces and unverified state estimates create an interface mismatch.
  • Semantic guardrails assess content, intent, or policy compliance, whereas physical authorization also requires geometry, timing, dynamics, observability, spatial permission, and fallback.
  • The review treats authorization as a composition problem because model proposals, perception evidence, constraints, policy, fallback, and auditability use different assumptions.

5 Capability Trends: From Prediction to Action

Physical AI is progressing toward more general, multimodal, predictive, and action-oriented systems, with reported gains in task success and hallucination-related measures. These capability milestones sharpen, rather than resolve, the need for independent action authorization.

  • Foundation-model systems now span robot policies, autonomous vehicles, drones, industrial automation, humanoids, and embodied agents.
  • VLA surveys describe policies mapping visual observations and natural-language goals into increasingly opaque action sequences.
  • Fluent action generation is not equivalent to feasible execution because learned action distributions can be structurally mismatched with feasible robot behavior.
  • OpenVLA reports a 7B-parameter model trained on 970k robot demonstrations, with absolute task-success gains of 16.5 percentage points over RT-2-X and 20.4 percentage points over Diffusion Policy.
  • WoVR reports average LIBERO success improving from 39.95% to 69.2% and real-robot success from 61.7% to 91.7% when hallucination in imagined rollouts is controlled.
  • EvoVLA reduces stage hallucination from 38.5% to 14.8% while reporting 54.6% average real-world success across four manipulation tasks.
  • These results mix task completion, hallucination reduction, prediction fidelity, sample efficiency, and latency rather than defining a unified probability of authorization correctness.
  • Reported capability and evaluation milestones do not by themselves establish valid state evidence, constraint satisfaction, fallback, or audit records for deployment.

6 Silent Failures in Closed-Loop Autonomy

Silent failures occur when closed-loop systems remain operational while acting from invalid internal world representations. The review therefore places authorization before hardware commitment and distinguishes model confidence from physical safety.

  • Silent failures arise when perception, prediction, planning, and control remain computationally active while the assumptions flowing through the loop are wrong.
  • Sensor drift, occlusion, distribution shift, hallucinated affordances, and semantic-physical mismatch can produce invalid action premises.
  • The safety-critical moment is often when an invalid world state is accepted as a basis for action, before the final actuator command.
  • Embodied safety studies report harmful physical actions, contextual-rule failures, and multi-step failures that static final-state evaluation may miss.
  • Confidence scores, probabilities, logits, and related internal signals are properties of model computation, not guarantees that a physical action is authorized.
  • Uncertainty estimates and OOD detectors can indicate unreliability or distribution mismatch, but they do not determine whether a specific action is authorized or which fallback is correct.
  • Runtime authorization may authorize, modify, defer, request human input, degrade, switch to fallback, block, or log a proposed action.

7 Runtime Authority as an Independent Layer

Runtime authority is framed as an independent interface that composes semantic, state, physical, spatial, operational, fallback, and audit evidence before physical execution. This layer addresses assumptions that semantic guardrails, controller-level filters, and runtime assurance do not cover alone.

  • Existing Guardrails and Their Limits: Safe reinforcement learning, barrier functions, reachability, predictive filters, and runtime assurance constrain actions but often assume dynamics, safe sets, or limited autonomy stacks.Deployed Physical AI may instead combine black-box models, heterogeneous hardware, learned planners, vendor controllers, and site-specific constraints.
  • Independent Runtime Authority: Physical action authorization asks whether a learned model’s proposed action should execute in the current state, under current constraints, with a reconstructable decision.It therefore sits between semantic policy, low-level safety control, runtime assurance, and deployment evidence.
  • Composition Across Evidence Types: Existing methods require explicit composition because semantic validity does not guarantee physical validity, and monitors must assess state-estimate reliability as well as controller safety.The relevant interface connects semantic policy, state validity, physical feasibility, spatial constraints, fallback behavior, and audit evidence around one proposed action.
  • Architectural Separation: A runtime guardrail should remain sufficiently independent of the evaluated black-box model to support independent checks, deterministic rules, reproducible logs, and cross-model comparison.Hybrid designs may combine learned anomaly detection, formal constraints, geometric reasoning, predictive checks, policy engines, temporal logic, and control monitors.
  • Adjacent Guardrail Research: Recent guardrail research adds contextual rules, trajectory-level diagnosis, and multimodal hallucination tests, but physical execution introduces state, feasibility, hardware, fallback, and spatial evidence.These contributions extend guardrails while leaving the full physical authorization pathway as the review’s focus.

8 Evaluation Under Dynamic Edge Cases

Evaluation under dynamic edge cases treats authorization as a continuing, record-based process rather than a static task-success benchmark. It combines closed-loop scenarios, intervention metrics, residual violations, audit checks, simulation, and runtime logs.

  • Dynamic Edge Cases: Embodied safety evaluation must cover multi-step scenarios, hazardous instructions, adversarial prompts, object interactions, context-dependent constraints, and physical consequences.Static benchmarks can miss failures that emerge only through closed-loop interaction among uncertainty, environment change, human behavior, and hardware constraints.
  • Failure Classes: Illustrative failure classes include occluded obstacles, stale perception, hallucinated affordances, restricted-zone violations, payload or velocity-limit violations, late intervention, and unsafe fallback.The examples span state, physical, spatial, operational, timing, and fallback evidence rather than defining an exhaustive benchmark suite.
  • Authorization Records: The authorization record ξ_t provides a platform-independent unit for comparing models and guardrail layers without requiring shared internal architectures.Records can encode valid or invalid proposed actions, intervention decisions, timing, and post-intervention constraint outcomes.
  • Metric Families: The minimum quantitative core includes unsafe-action intervention rate, false block rate, pre-commit intervention rate, and residual violation severity.These metrics distinguish safety prevention, usability cost, intervention timing, and remaining constraint violations.
  • Metric Interpretation: UAIR = 1 means every invalid action was intercepted, FBR = 0 means valid actions were not unnecessarily interrupted, and RVS = 0 means all measured constraints were satisfied afterward.PCIR below UAIR indicates that some interventions arrived too late; empty valid or invalid sets require a not-applicable report.
  • Simulation and Continuous Evaluation: Simulation can generate repeatable edge cases and software- or hardware-in-the-loop tests, but it does not by itself determine runtime authorization.Offline benchmarks and runtime logs can instead feed a continuous guardrail-evaluation workflow through authorization records.

9 Synthesis: Action-Authorization Gap and Open Questions

The review identifies a remaining assurance category: a model-independent runtime boundary that translates heterogeneous Physical AI outputs into validated, constrained, and auditable actions. It frames silent failure as continued confident operation grounded in an invalid state and derives open questions for authorization research.

  • Action-Authorization Gap: Model-centric, control-centric, and safety-centric literatures address capabilities, dynamics, formal methods, uncertainty, monitoring, and evaluation, but the review finds an action-authorization gap between them.The gap concerns how these technical tracks connect around a proposed physical action.
  • Model-Independent Boundary: The remaining assurance category is model-independent runtime authorization for translating heterogeneous Physical AI outputs into validated, constrained, auditable physical actions.The review characterizes the issue as limited availability of a common boundary that composes technical evidence types.
  • Silent Physical-Action Failure: Silent failure describes a system that remains operational, confident, and acting while its action is grounded in an invalid state.This deployment-oriented framing connects hallucination, distribution shift, perception error, and adversarial attack to physical consequences.
  • Auditability: Auditability makes it possible to determine whether failures arose from interacting perception, planning, control, or organizational factors and supports reproducibility, incident analysis, accountability, and regulatory readiness.Structured records explain why actions were allowed, modified, blocked, or escalated.
  • Open Questions: Open questions concern action interfaces, state reliability, constraint composition, guardrail evaluation, cross-platform learning, and runtime governance.The questions ask how to quantify authorization evidence, compare guardrails beyond task completion, preserve model independence, and explain interventions.
  • Scope of the Synthesis: Runtime guardrails address the intermediate event in which a black-box system proposes an action and the surrounding system evaluates whether it should proceed.The framing remains falsifiable if future systems expose certified action representations, reliable state evidence, composable constraints, and auditable decisions, or if existing mechanisms alone consistently prevent silent failures.

10 Assurance Implications and Minimal Event Schema

The assurance implications center on intervention quality and a minimal authorization-event schema independent of model, simulator, controller, or hardware implementation. This schema supports comparison across platforms without claiming to be a standard or complete guardrail design.

  • Assurance Implications: Evaluation should measure whether unsafe, invalid, or poorly grounded actions are detected, modified, blocked, or escalated before execution, not only task success.The authorization event becomes the concrete unit for assessing intervention quality.
  • Contribution Boundary: The paper maps the gap between model capability and physical-action assurance, including failure modes, formal interfaces, evaluation requirements, metric families, and comparison evidence.It explicitly does not propose a new standard or complete guardrail design.
  • Minimal Event Schema: Each proposed physical action can be represented by an authorization event record independent of the policy, world model, simulator, or controller’s internal representation.The record captures whether the action was authorized, modified, blocked, escalated, or routed to fallback without prescribing an API or implementation standard.
  • Cross-Platform Instantiation: Platforms can instantiate common schema fields differently, emphasizing geofences and GNSS integrity for aerial systems, clearance and human proximity for mobile robots, or contact and payload constraints for manipulators.The comparison unit is the authorization event rather than the model architecture or controller alone.

11 Limitations and Threats to Validity

The review is bounded to runtime guardrails and action authorization for black-box Physical AI systems, with limitations spanning scope, evidence validity, construct validity, external validity, and temporal validity.

  • Scope: The review does not survey the entire Physical AI field, robotics standards, simulation infrastructure, certification, or general AI ethics.Its source selection centers on the action-authorization pathway.
  • Scope and evidence validity: The formalization abstracts away many hardware-specific dynamics, perception-stack details, and organizational deployment processes.Autonomous-vehicle incidents are operational analogues rather than direct evidence about VLA or world-model foundation systems, and no new guardrail benchmark or empirical evaluation is introduced.
  • Technical boundary conditions: Runtime authorization cannot guarantee safety when observability is poor, constraints are incomplete, fallback behavior is underspecified, or the environment is adversarial.Formal guarantees can weaken with stale state estimates, incorrect dynamics models, or mismatched action representations, and authorization latency can itself become hazardous.
  • Construct validity: The term “silent failure” groups hallucination, out-of-distribution behavior, sensor drift, state-estimation error, and invalid affordance inference.Although useful for deployment analysis, these mechanisms may require different instrumentation and mitigation.
  • External and temporal validity: A cross-domain guardrail taxonomy still requires domain-specific constraints, fallback policies, and evidence thresholds, while rapidly changing literature may narrow parts of the identified gap.Physical AI deployments vary across robots, vehicles, drones, factories, warehouses, hospitals, roads, and homes.

12 Conclusion

Physical AI safety is also an authorization problem: before learned actions become physical commitments, systems need independent evidence that state, feasibility, constraints, fallback, and auditability are adequate. The review therefore frames runtime guardrails as assurance mechanisms for AI that acts in the physical world.

  • Conclusion: Before a learned action becomes a physical commitment, the autonomy stack should independently verify state validity, feasibility, operational constraints, fallback availability, and auditability.The conclusion positions authorization alongside, rather than instead of, model training, control, and simulation.
  • Conclusion: Silent failures arise when systems act confidently on corrupted or incomplete world states, motivating integration of existing safety ingredients into runtime authority.Relevant ingredients include safe control, runtime assurance, shielding, uncertainty estimation, out-of-distribution detection, embodied benchmarks, verification, simulation, and robot-specific guardrails.
  • Implications: Researchers should evaluate authorization events rather than only task success, while builders should separate model proposal from runtime authority.Reconstructable action records can help safety teams and regulators examine why actions were authorized, modified, blocked, or escalated under uncertainty.
  • Conclusion: Runtime guardrails can be studied as assurance mechanisms for the transition from AI that predicts the world to AI that acts in it.
Loading 2606.00090v1…