Source-linked AI summary
Bounded Autonomy and Verifiable Safety for Agentic AI Enabled Automation
Srini Ramaswamy, Deveeshree Nayak
TL;DR
DNRS combines SME-derived invariant anchors, candidate depth-aware access, governance states, and a finite shielded framework to bound autonomy as epistemic risk changes. In simulation, the finite governance process converged across tested groupings with no configured guard violations, under fixed abstraction and feasible-shield assumptions.
Problem
High-stakes agentic systems may lose visibility of SME constraints as reasoning deepens, while current architectures do not reliably preserve domain constraints through that process.
Source: pp. 1, 4Method
DNRS encodes SME constraints as invariant anchors, proposes MoDA-style access to keep them available, and maps monitored risk into governance states and shielded transitions in a finite LBCF abstraction.
Source: pp. 3–4, A. Finite Governance Abstraction · p. 18Results
Across 120,000 simulated runs, the LBCF process achieved 100.0% convergence and zero configured guard violations across all three grouping strategies.
Source: A. Finite Governance Abstraction · p. 19Takeaways & Limitations
Under a fixed finite abstraction and feasible-shield assumptions, contested actions must descend toward lower-risk governance states or route to Assisted or Regulated intervention.
Source: A. Finite Governance Abstraction · p. 20DNRS does not prove intrinsic neural-model safety, and MoDA-style invariant access remains proposed rather than a validated production transformer implementation.
Source: pp. 2, 5
Abstract
from arXiv · showhide
Agentic AI-enabled automation cannot be safely deployed in high-stakes environments on probabilistic reasoning alone. A recurring risk is epistemic drift: as reasoning deepens, system behavior may move away from subject-matter-expert constraints for safe operation. This paper presents BRaVeS, a bounded reasoning and safety-governance framework termed the Defensible Next-Gen Reasoning System (DNRS). BRaVeS encodes SME-defined constraints as invariant anchors, proposes MoDA-Style (Mixture of Depths Attention) depth-aware access as a candidate mechanism for keeping these anchors visible during inference, and uses a state hierarchy (SMARtAutonomy) to reduce autonomy as epistemic risk increases. To formalize bounded recovery, we introduce the Lyapunov-Bounded Consensus Framework (LBCF), which maps continuous epistemic-risk signals into a finite K-bag abstraction and applies shielded state transitions that enforce Lyapunov-style energy descent or route the system to a human-mediated terminal state. The formal convergence result applies to the finite LBCF abstraction under fixed thresholds and feasible-shield assumptions; it does not prove safety of the full continuous neural activation space. We evaluate the framework through a discrete event Monte Carlo simulation using HAI 22.04 industrial-control-system time-series data with synthetic noise and sensor-degradation regimes. Across the tested parameter-grouping strategies and thresholds, the LBCF process achieved finite-step convergence and no safety-guard violations. These results provide simulation-based evidence that bounded governance behavior can be enforced under the stated abstraction, while motivating future work on deployed transformer implementations, live human-in-the-loop validation, and broader adversarial settings.
E. Reasoning Boundary
DNRS treats invariant monitoring as a monitored epistemic state, not an autonomous decision, and couples it to explicit autonomy reduction, shielded transitions, and human-mediated terminal states. SMARtAutonomy scales permissions as drift rises, moving from internal recovery toward SME support when grounding cannot be restored.
E. Reasoning Boundary
- The monitored epistemic state summarizes divergence from the SME-derived anchor and uncertainty about that movement; it is not itself an autonomous decision.The anchor remains fixed during the governed episode unless explicitly updated through the V&V process.Source: E. Reasoning Boundary · p. 5
- As the distance between the current reasoning state and the invariant baseline increases, SMARtAutonomy reduces autonomy through Stable, Meta-Cognitive, Assisted, and Regulated states.Meta-Cognitive mode pauses external execution for internal recovery; if grounding cannot be restored, Assisted mode triggers SME or verifier support.Source: E. Reasoning Boundary · p. 6
- SMARt-compliant operation permits externally committed actions only in Stable, bounds recovery in Meta-Cognitive and Assisted states, and requires unresolved ambiguity to reach Regulated.Return to Stable requires measured risk reduction and satisfaction of active predicates.Source: E. Reasoning Boundary · p. 6
- The V&V loop identifies the stressed invariant dimension, surfaces a potentially conflicting parameter or signal, and routes the system to a governance state.Routine ambiguity may be handled internally, while novel or unresolved ambiguity may be routed to Assisted mode and hard violations to Regulated intervention.Source: E. Reasoning Boundary · pp. 6–7
A. Executing the V&V Loop: Fluid Governance
The section describes DNRS governance that uses invariant violations and epistemic risk to guide V&V, validator feedback, and graduated autonomy responses, restricting contested actions and escalating unresolved or prohibited conditions. It maps these responses to LBCF’s finite abstraction and shielded transitions; the convergence claim applies to that abstraction, not neural activations or intrinsic model safety.
A. Executing the V&V Loop: Fluid Governance
- The V&V loop triggers on elevated composite risk, critical divergence from an SME invariant, or a threatened or violated SME predicate, with response determined by the source of resolution.Internal knowledge and checks support Meta-Cognitive recovery; external authority supports Assisted mode; hard-boundary violations or unauthorized operation require Regulated mode.Source: A. Executing the V&V Loop: Fluid… · p. 7
B. V&V Dossier Construction
- When the system enters Assisted or Regulated mode, its V&V Dossier identifies the triggering context, threatened invariant, contested parameter, counterfactual effect, current governance state, risk tuple, and audit trace.This narrows validation to what is contested, what would change if constrained, and what authority is needed, without requiring full mechanistic interpretability.Source: A. Executing the V&V Loop: Fluid… · p. 8
C. Validator Response and Invariant Update
- Validator responses can constrain an unsafe parameter, integrate a legitimate contextual exception into the active invariant reference, or route the system to Regulated mode.Authorized invariant updates apply to the governed episode or domain context and do not retrain the base model; the governance update records the response, state transition, and audit pointer.Source: A. Executing the V&V Loop: Fluid… · pp. 8, 10
D. Illustrative Clinical Routing Cases
- Illustrative clinical cases show reduced-authority continuation for a documentation conflict, internal recovery before external validation for a medication conflict, and immediate Regulated intervention for a re-identification risk.These cases clarify V&V mechanics and are not part of the empirical validation.Source: A. Executing the V&V Loop: Fluid… · pp. 8–9
E. Asynchronous Versus Synchronous V&V
- Asynchronous validation permits reduced-authority work when risk is elevated but not critical, whereas critical divergence or a critical predicate violation blocks autonomous execution until an authorized reset.Non-contested work may continue while high-impact actions depending on an unresolved parameter remain unexecuted.Source: A. Executing the V&V Loop: Fluid… · p. 9
F. V&V Claim Boundary
- The V&V loop may update governance and an authorized active invariant, but it does not change base-model weights or prove intrinsic model safety.The safety claim rests on invariant monitoring, authority reduction, dossier-based validation, and shielded state transitions.Source: A. Executing the V&V Loop: Fluid… · p. 9
V. SMARTAUTONOMY GOVERNANCE AND FUZZY PERMISSION SCALING
- SMARtAutonomy replaces binary permission with graduated authority: Stable permits bounded execution, Meta-Cognitive attempts internal recovery, Assisted locks contested actions for external resolution, and Regulated revokes autonomy.Fuzzy routing preserves contested context for internal comparison and counterfactual analysis while preventing unresolved uncertainty from silently authorizing execution.Source: A. Executing the V&V Loop: Fluid… · pp. 10–11
A. Translating Epistemic Degradation into Governance Signals
- DNRS translates divergence from the active invariant and posterior uncertainty or instability into a composite risk signal before mapping it to a discrete autonomy state.The composite score weights geometric drift, KL instability, and hypothesis entropy; the aggregation rule is fixed for the governed episode and audit-logged.Source: A. Executing the V&V Loop: Fluid… · p. 11
- SMARtAutonomy preserves safe workflow continuity by keeping bounded action in Stable mode while restricting authority over contested actions instead of halting the entire system.Permission is scaled at the action level: non-contested work may continue while the disputed action is blocked or reduced in authority.Source: A. Executing the V&V Loop: Fluid… · pp. 12–13
- Meta-Cognitive mode uses a debounced internal-recovery window, then escalates to Assisted mode if risk remains unresolved, recovery budget expires, or external authority is required.Assisted mode locks or limits the contested action while non-contested work may continue; authorized external resolution can restore Stable operation or escalate unresolved critical issues.Source: A. Executing the V&V Loop: Fluid… · pp. 13–14
- Regulated mode revokes autonomy when critical divergence, a violated critical predicate, or a prohibited action makes autonomous continuation unjustified.Leaving Regulated mode requires explicit authorized reset, risk below the safe threshold, and satisfaction of all active SME predicates.Source: A. Executing the V&V Loop: Fluid… · p. 14
E. Effect of Permission Scaling on MoDA-Style Attention
- Fuzzy permission scaling can modulate invariant-aware attention, but this mechanism is optional and implementation-dependent rather than required by the LBCF guarantee.As permission falls, invariant-anchor visibility may increase and contested pathways may be dampened; black-box deployments can enforce governance through retrieval, tool-routing, or action controls instead.Source: A. Executing the V&V Loop: Fluid… · p. 15
F. Interface to the LBCF Finite Abstraction
- The baseline LBCF interface discretizes divergence, uncertainty, composite risk, and SMARt state into a finite K-bag state, treating permission as derived rather than an independent input.An independently imposed permission level may instead be included as an exogenous input; LBCF governs the finite abstraction, not raw activations or attention dynamics.Source: A. Executing the V&V Loop: Fluid… · p. 16
- Q-learning may rank candidate governance actions, but the LBCF shield—not the learned policy—filters executable actions and supplies the safety guarantee.The convergence claim depends on the finite abstraction and shielded descent condition.Source: A. Executing the V&V Loop: Fluid… · p. 17
A. Finite Governance Abstraction
The LBCF maps continuous risk and autonomy signals into a finite governance state, then shields actions to require energy descent or terminal routing. In the tested simulation, this abstraction converged without configured guard violations.
Finite Governance State
- LBCF quantizes divergence, uncertainty, composite risk, and the active SMARt autonomy gear into a finite K-bag governance state.The continuous signals are binned; the resulting state is used for shielded control rather than representing the full neural state.Source: A. Finite Governance Abstraction · p. 17
Shielded Q-Learning and Governance Actions
- Q-learning may rank candidate actions, but only shield-approved actions can execute, and the shield allows energy-descent or terminal-routing actions.The governance energy combines ordinal encodings of divergence, uncertainty, risk, and governance state.Source: A. Finite Governance Abstraction · p. 17
Clinical IDSS Transition Illustration
- In the clinical IDSS illustration, unresolved allergy conflicts route medication fields to Assisted mode, while critical clinical or PHI predicate failures route to Regulated mode.The examples illustrate finite LBCF transitions and are not empirical clinical validation.Source: A. Finite Governance Abstraction · p. 17, Table 7 · p. 18
LBCF Theorem
- Under a finite state space and feasible-shield assumptions, every contested action must descend through governance states or terminate in Assisted or Regulated mode.The theorem excludes indefinite autonomous continuation of the contested action, not unsafe behavior in the full neural activation space.Source: A. Finite Governance Abstraction · p. 18, Table 8 · p. 18
Limitations
- The results do not establish general ICS safety, attack detection, continuous neural-state convergence, or production validation; they are limited to the tested finite abstraction and perturbations.The simulation also omits live human review and PLC-level actuation, and uses fixed rather than site-adaptively tuned thresholds.Source: A. Finite Governance Abstraction · pp. 18–20
Simulation Results
- Across 120,000 simulated runs, LBCF achieved 100.0% convergence, mean energy descent around −0.0068, and zero configured guard violations.The results cover three parameter-grouping strategies under the tested thresholds and perturbation regimes.Source: Table 11 · p. 19, A. Finite Governance Abstraction · p. 19
Data Availability Statement
The study uses publicly available HAI 22.04 data and no proprietary, private, or personally identifiable datasets.
Data Availability Statement
- The findings use publicly available HAI 22.04 data; derived simulation outputs and intermediate materials may be available from the corresponding author upon reasonable request.No proprietary, private, or personally identifiable datasets were used, and no new external dataset repository was created.Source: Data Availability Statement · pp. 20–21
Related papers
This paper cites
- Attention Is All You Need2017
- Densely Connected Convolutional Networks2016
- Scaling Laws for Neural Language Models2020
- Constitutional AI: Harmlessness from AI Feedback2022
- Safe Reinforcement Learning via Shielding2017
- End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks2019