PaperlayerView exact revision on arXiv

Source-linked AI summary

SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents

Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou

arXiv:2609.04629v1cs.AIcs.LGeess.SY

TL;DR

LLM runtime gates are not well served by scalar violation scores when systems begin in violation and must recover through multiple actions. SiLR shadow-executes proposals and applies a product order over violation support and per-branch severity; it achieves structured recovery where terminal and scalar gates often stall, while the paper proves scalar surrogates cannot soundly represent that order.

  • Problem

    Runtime governance largely assumes systems start safe, leaving post-violation recovery admission—especially admission of intermediate progress—unaddressed.

  • Method

    SiLR deterministically shadow-executes each proposal and admits it using a product order that preserves overloaded-branch support and per-branch severity.

  • Results

    21/21 structured recoveries contrast with 9/21 for the best scalar gate and 0/21 for terminal admission, with the structured-versus-scalar advantage significant across the full 24-scenario benchmark.

  • Takeaways & Limitations

    Preserving violation geometry is required at both runtime admission and process-reward design points; scalar projection loses that structure.

  • Takeaways & Limitations

    The severity bound is a worst-case per-step drift envelope rather than a convergence guarantee, so reaching violation-free states depends on proposal quality.

Abstract

from arXiv · show

A runtime gate for an LLM tool agent is usually cast as a filter. In a ReAct loop a rejected proposal is followed by another at the same state, so the gate is a search operator over the proposal stream whose admission criterion shapes which trajectories are reachable. We study post-violation recovery admission, where progress must be admitted while the system is still in violation, and identify the scalar projection trap: an aggregate-score gate accepts a locally improving proposal and commits the trajectory to a plateau. SiLR instead shadow-executes each proposal and admits it under a product order over the branch-level violation state (overloaded-branch support and per-branch severity). We prove that no scalar surrogate is sound for this order, so the failure is representational, not a matter of threshold tuning. On mined Gym-ANM scenarios, SiLR recovers 21/21 multi-action episodes against 0/21 for terminal and 9/21 for the best scalar gate, significant across the full 24-scenario benchmark. The terminal-versus-structured dichotomy holds across three model families and in CityLearn. Because admission rests on deterministic simulation, the LLM lies outside the trust boundary: a magnitude-redistribution attack that defeats both scalar and support-only baselines is contained only by the full per-branch predicate. With two constraint families active, every tested scalar projection admits physically unsafe actions; support-only admits the largest fraction (63.2% of 42,410; product order 0). In the hardest dual-family traces, scalar gates recover only through that unsafe class. Reused as a GRPO process reward, it outperforms its count projection in every mined scenario and is the only tested reward whose ungated policy exceeds the untrained base (0.844 vs. 0.778). Scalar projection loses the violation geometry at both design points; only the full product order is structurally sufficient.

Introduction

SiLR addresses post-violation recovery, where runtime gates must admit progress while violations remain, rather than merely prevent unsafe actions. It argues that preserving violation geometry, instead of tuning scalar strictness, determines whether multi-step recovery remains possible.

  • Problem: Post-violation recovery requires admitting actions that leave the system unsafe but make progress toward recovery.Prior runtime governance assumes the system starts inside the safe set and does not address this inverted enforcement problem.
  • Problem: A terminal gate is safe but admits no intermediate step, deadlocking every multi-action episode in the mined Gym-ANM benchmark.The scalar relaxation is also representationally insufficient because changing its slack does not close the recovery gap.
  • Evidence: 21/21 structured recoveries contrast with 9/21 for the best scalar gate and 0/21 for terminal admission.The difference is significant across the full 24-scenario benchmark.
  • Approach: SiLR treats the gate as a search operator over the ReAct proposal stream and preserves a product order over branch-level violation states.The product order is presented as the representation required for admission, while scalar projection collapses the violation geometry.

Related Work

Prior shielding and runtime-enforcement systems provide safety checks, rollback, or recoverability mechanisms, but the cited work does not target post-violation recovery admission with SiLR’s branch-level product order.

  • Shielding and runtime guards: Existing shielding methods relax terminal gates through recoverability sets, forward simulation, or backup policies, but assume fixed action spaces or additional recovery machinery.These approaches differ from SiLR’s black-box LLM tool-call dispatch setting.
  • Positioning: Table 1 distinguishes systems by whether they retain support and severity geometry and whether they use only single-step shadowing without recovery policies or dynamics rollout.These dimensions summarize SiLR’s positioning against closest runtime-safety systems.
  • Runtime enforcement: SiLR targets post-violation recovery admission with single-step shadow simulation under a product order over branch-level violation state.The cited comparison positions this as distinct from prior runtime governance systems.
  • Grid-Agent and scalarization: Grid-Agent uses post-hoc rollback and support-level checking, whereas SiLR uses a pre-action admission predicate with per-branch severity protection.Grid-Agent’s check does not guard severity for individual branches.

Threat Model and Method

SiLR shadow-executes each proposed tool call and admits it only when support inclusion and componentwise severity containment preserve the branch-level violation order. The resulting invariant constrains admitted trajectories independently of LLM behavior, while scalarization cannot soundly represent the same order.

  • Trust boundary: The LLM is untrusted, while the simulator, verifier, checkers, and tool layer are trusted components outside the LLM’s control.The threat model permits compromised prompts and observations but excludes verifier, simulator, and tool-layer modification.
  • Shadow execution: Each proposal is evaluated by deterministic shadow execution on a deep copy, so rejected actions leave the live system untouched.The verifier applies the domain solver to the copied state before admission.
  • Violation geometry: The violation state retains overloaded-branch support and per-branch severity, with admissibility defined by support inclusion and componentwise severity comparison.PASS denotes terminal recovery, SAFE_PROGRESS denotes admissible nonterminal progress, and FAIL denotes an inadmissible proposal.
  • Scalarization gap: No scalar-threshold gate is sound for the product order because some antichain pair is necessarily ranked as comparable by any scalar projection.The constructive example shows that aggregate, sup-norm, and lexicographic candidates admit a non-improving direction.
  • Search behavior: The representational gap becomes behavioral when scalar admission accepts the first aggregate-improving proposal and commits the trajectory to a residual plateau.Structured admission rejects that proposal under the product order and keeps the per-step search active.
  • Admission predicate: SiLR implements tool validation plus a relaxed product-order test using α=1.05 and ε=10^-3, approaching the exact order as α →1 and ε →0.The support and severity predicates jointly form the computable admission test.
  • Admission invariant: Every admitted trajectory preserves the initial violation support, while severity obeys a bounded-growth invariant under shadow fidelity.The support guarantee is independent of the action source, including a compromised LLM.
  • Severity containment: The per-branch predicate rejects magnitude redistribution such as (10, 10) → (19, 0.1), despite the aggregate penalty decreasing from 20 to 19.1.Recovery remains robust over α ∈[1.05, 1.20] with 15/15 episodes.

Evaluation

Across benchmark recovery, compromise, and generalization tests, structured admission consistently outperforms terminal and scalar gates while containing unsafe proposals. Its advantage persists across model families and domains, with the full product order providing the strongest containment.

  • Recovery performance: 21/21 multi-action episodes were recovered by structured admission, versus 9/21 for the best scalar gate and 0/21 for terminal admission.At matched N=7, support-only recovered 20/21, while ungated recovery was 1/21.
  • Recovery performance: Recovery under scalar slack was nonmonotonic, ranging from 5/21 to 9/21 across η ∈{0, 0.05, 0.10, 0.20}, while structured admission achieved 21/21.Wider slack could admit an early penalty-improving proposal and lock the trajectory onto a residual plateau.
  • Residual outcomes: Across 480 benchmark episodes, structured admission drove episodes to zero residual, whereas scalar-admitted failures formed a nonzero-residual plateau and terminal episodes stayed at their initial violation state.The full benchmark recorded 120/120 structured recoveries against 96/120 for the best scalar gate.
  • Compromise containment: 43/120 magnitude-redistribution attacks passed the scalar gate, 11/120 passed support-only, and 0/120 passed the full structured predicate.The per-branch predicate ψ3 rejects concentrating steps that evade aggregate and support-level monitoring.
  • Cross-domain safety: 0/558 physically unsafe ANM actions and 0/11,311 CityLearn actions were admitted by the full product order, unlike sup-norm and scalar alternatives.In dual-family ANM states, support-only admitted 63.2% of 42,410 physically unsafe actions, while the product order admitted 0.
  • Generalization: Structured admission recovered 15/15 ANM episodes across three model families, while terminal admission recovered 0/15; the same terminal-versus-structured pattern transferred to CityLearn.In CityLearn, terminal admission succeeded only on the single-step scenario, whereas structured admission recovered between 10/15 and 15/15 depending on setting.

From Admission Gate to Process Reward

The product-order violation geometry is reused as a GRPO process reward, preserving branch-level recovery information that scalar and count projections lose. Across mined scenarios and stress regimes, the geometric reward better matches one-step recovery value and supports stronger trained recovery.

  • Reward construction: The verifier’s graded verdict and persisted branch state Φ define a GRPO process reward without verifier changes or task-specific outcome labels.
  • Reward construction: Support elimination outranks surviving-branch severity reduction, while drift penalties and per-family normalization preserve the admission order.
  • Reward fidelity: 0.000 reward-level confusion versus 0.25 for the count projection; the geometric reward dominates in all 24 mined scenarios (sign test p=1.2×10−7).
  • Policy outcomes: 0.844 versus 0.778: the geometric reward is the only tested signal whose trained ungated policy exceeds the untrained base.
  • Failure mechanism: 98% of the continuous action range is flat under the count reward, leaving scalar projections unable to attain full recovery in both multi-family stress regimes.
  • Stress-regime behavior: σL/σS ≈4 marks failure onset, while CityLearn’s ratio of 286 lies deep in the collapsed regime; the homogeneous limit removes the separation.

Conclusion

The paper identifies scalar projection as a representational failure at both runtime admission and process-reward design points. Structured product-order reasoning enables recovery and preserves intermediate safety under multi-family stress.

  • Conclusion: 21/21 structured-admission recoveries versus 9/21 for the best scalar gate and 0/21 for terminal admission.
  • Conclusion: The product order is required at both design points because scalarization commits trajectories to residual plateaus and collapses the process-reward advantage signal.

Attack-Family Constructions

The attack evaluation tests prompt, observation, stall, and magnitude-redistribution threats against verifier-level gates. The full product order contains unsafe actions across domains, while scalar and support-level gates admit unsafe cases.

  • Compromise suite: 0/60 attack episodes produced attack success, false recovery, or material worsening, with a 95% Wilson interval of [0.00, 0.06].
  • Witness-pool evaluation: The full product order admitted zero unsafe actions in both ANM and CityLearn witness pools, whereas strong scalar projections failed on support-expansion breaches.
  • Evaluation boundary: The witness-pool denominators are constructed and category-dependent, so absolute breach counts are not comparable across domains.

Proofs

The proofs show that scalar surrogates cannot soundly represent the product-order admission criterion, while admitted shadow-verified actions preserve support and bound severity drift. A separate result explains why count rewards collapse process-level advantages.

  • Scalarization gap: Any scalar surrogate totally orders an antichain, so at least one transition is admitted despite introducing a branch or inflating existing severity.
  • Admission invariant: Admitted actions preserve support inclusion and satisfy σi(st) ≤ α^tσi(s0) + εΣ_{k=0}^{t−1}α^k for each branch.
  • Admission invariant: The severity bound is a worst-case per-step drift envelope rather than a convergence guarantee; recovery still depends on proposal quality.
  • Advantage collapse: With B concurrent branches, count rewards take at most B+1 values, giving identical immediate rewards to actions within each count-equivalence class.

Robustness and Cross-Domain Generalization

Robustness tests show that structured admission remains effective across parameter ranges, scenarios, and domains, while scalar admission can plateau or rely on physically unsafe steps.

  • Benchmark scope: 153 single-action, 24 multi-action, 252 MPC-residual, and 171 trivial scenarios comprise the mined 600-scenario ANM pool.
  • Parameter robustness: 15/15 recoveries hold for α ∈[1.05, 1.20], with 14/15 at α=1.02.The magnitude guard tightens as α approaches 1 without disturbing support-driven recovery.
  • Cross-domain replication: In CityLearn, gate semantics transfer across three battery-storage scenarios, while one Gemma-3-12B exception stalls because proposal quality fails to reach the terminal action.
  • Scalar projection trap: In Scenario C, structured admission reaches full recovery while the scalar gate plateaus after admitting a locally improving proposal.The representative trace uses best slack η=0.05, and the plateau recurs across all four slack settings.
  • Dual-family safety: 689/689 oracle-recoverable dual-family states admit an all-safe recovery order under the product-order verifier.Thus, per-step safety does not itself make recovery infeasible in this audit.

Analysis

The analysis compares geometric and count-based reward signals, showing that preserving branch-level violation structure improves fidelity, robustness, and recovery across controlled and larger-scale settings.

  • Reward internalization: Only the full geometric reward keeps ungated recovery above the untrained base in the verifier-as-reward evaluation.The table measures internalization with the gate removed at evaluation.
  • Reward fidelity: Geometric reward confusion remains 0.000 across rollout policies and branch counts, while count confusion ranges 0.21–0.22 and grows with simultaneous violations.Count confusion rises from 0.347 at B=2 to 0.465 at B=10, with Pearson r=0.889.
  • Reward fidelity: Under additive Gaussian perturbation, geometric misordering rises above count’s 0.157 only after noise exceeds approximately 0.8σ.At zero noise, geometric misordering is 0.011.
  • Multi-family dynamics: The σ-heterogeneity failure emerges near σL/σS ≈4, saturates from 8 onward, and is pronounced at CityLearn’s measured heterogeneity of 286.The separation vanishes in the homogeneous limit.
  • Recovery grid: Geometric rewards achieve ≥0.97 recovery in all 16 coupling–floor cells, whereas count rewards reach ≤0.42 in 15 of 16.The count reward succeeds only at the weakest coupling, and an MLP reproduces the same ordering.
  • Cross-domain training: Across CityLearn training, geometric advantage correlates with true one-step action value at Spearman’s ρ=0.96 versus 0.79 for count.At equal violation count, count leaves 14% higher battery-family residual severity than geometric.
Loading 2609.04629v1…