PaperlayerView exact revision on arXiv

Source-linked AI summary

Approved Too Late: Verdict Staleness in LLM-Guarded Self-Adaptive Systems

Ilai Shraga, Roei Eshel, Lior Gorelik

arXiv:2608.26306v1cs.AI

TL;DR

The paper studies how approvals from LLM guardrails can become stale between checking and actuation in evolving self-adaptive systems. It distinguishes three freshness measurements and introduces FBS, which estimates an approval’s validity horizon from margin and feature volatility. Across five environments, verdict changes are nonzero, FBS reduces oracle-labeled expiry under fixed settings, and a separate LLM audit finds invalidity in every approval stream.

  • Problem

    An approval can be correct for the checked context and action yet become inadmissible by actuation as the plant evolves.

  • Method

    The paper uses fixed-action replay and three differently conditioned freshness measures, then gates approvals with FBS using safe-side margin and recent feature volatility.

  • Results

    5.3–48.4% all-candidate verdict-change rates occur at K=8 across five environments; FBS reduces oracle-labeled expiry in all five, while LLM audit invalidity is nonzero in all four streams.

  • Takeaways & Limitations

    A freshness contract must require both check-time correctness and validity at use time, with an explicit expiry rule and justified fallback.

  • Takeaways & Limitations

    The findings are limited to evaluated environments, scalar predicates, logged candidate distributions, replay shifts, and fixed FBS settings; deployment-specific calibration remains necessary.

Abstract

from arXiv · show

A large language model (LLM) guardrail for a self-adaptive system (SAS) may issue an approval that is correct at check time but stale by actuation. This creates an Execute-stage time-of-check to time-of-use (TOCTOU) hazard. We study verdict freshness: whether a guardrail verdict remains valid when used. We distinguish three quantities that answer different questions: all-candidate verdict change under fixed-action replay, oracle-labeled approval expiry on recorded closed-loop trajectories, and judge-conditioned use-time invalidity. Across five reproducible SAS environments, all-candidate verdict-change rates span 5.3-48.4% at a common replay shift of eight simulator steps. We introduce the Freshness-Bounded Shield (FBS), which estimates each approval's validity horizon from its safe-side margin and recent feature volatility, without an explicit plant-dynamics model. Using fixed settings documented in the artifact, FBS reduces oracle-labeled approval-expiry rates from 3.4-24.7% to 0-1.8% at the same shift. A separate audit of four LLM judges finds nonzero judge-conditioned use-time invalidity in every approval stream. We formulate a freshness contract: every approval must be correct at check time and remain valid at use time.

I. INTRODUCTION

The paper identifies verdict staleness as an Execute-stage TOCTOU hazard: an approval correct at check time may become invalid before actuation as the plant evolves. It formalizes freshness, distinguishes related measurement quantities, and proposes FBS as a lightweight mitigation.

  • An Execute-stage guardrail can correctly assess a context–action pair yet authorize an inadmissible action when the evolving plant reaches actuation.
  • The evaluation spans five reproducible SAS environments covering IoT adaptation, edge–cloud resource management, self-healing, object detection, and backend job dispatch.
  • The study measures freshness through all-candidate verdict change, oracle-labeled approval expiry, and judge-conditioned use-time invalidity.These quantities differ in their conditioning and measurement roles.
  • FBS estimates each approval’s validity horizon from safe-side margin and recent feature volatility without an explicit plant-dynamics model.It acts as a post-verdict gate that can apply a fallback when the estimated horizon is exceeded.
  • Related work addresses check-time assessment, information or tactic age, external-state revalidation, or runtime assurance, but does not directly establish approval validity under plant evolution.The paper positions FBS as complementary to these approaches rather than a direct baseline.

III. METHODOLOGY

The replay methodology holds a candidate action fixed while comparing deterministic reference-checker verdicts at the original and later recorded contexts. It separates unconditional verdict changes from approval expiry and judge-conditioned invalidity by using different conditioning sets.

  • Fixed-action replay asks whether a candidate admissible at check time remains admissible at replay use time t + K.The later context comes from the same recorded episode, rather than a counterfactual trajectory induced by delaying or replacing the action.
  • The all-candidate verdict-change rate averages reference-label changes before conditioning on a method’s pass set.
  • The oracle-labeled approval-expiry rate conditions on candidates passed by a method and reference-admissible at check time.
  • Judge-conditioned use-time invalidity is reported separately within each judge’s own approval set.Because the judge’s check-time error is not removed, this quantity is descriptive rather than an isolated temporal-expiry measure.

A. System, Threat Model, and Clock Model

The system model places an LLM guardrail between observation and actuation while the plant continues evolving. The clock model represents end-to-end latency as conservative integer observation shifts, evaluated across five step-driven environments.

  • System and threat model: At step t, the monitor emits observation o_t, the controller proposes action a_t, and the Execute-stage guardrail returns approval J_t on context x_t and action a_t.The context x_t includes the current observation and a short history; J_t=1 denotes approval.
  • System and threat model: The hazard is ordinary closed-loop plant evolution during the grounding interval, excluding prompt injection and deliberate adversarial state manipulation.
  • Clock model: End-to-end elapsed time Δ from observation acquisition to actuation is conservatively discretized into live replay age K_live using the control period T_ctrl.
  • Clock model: A 2 s latency with a 0.25 s control period yields K_live=8, while variable serving latency produces a distribution over K_live values.
  • Experimental scope: The experiments sweep K ∈ {1, 2, 3, 5, 8, 10}, using K=8 for the headline cross-environment comparison without treating it as a common physical duration.Eligible replay indices must remain within the same episode.
  • Experimental scope: The five environments are DELTAIOT, SIMTUNE, mRUBIS, SWITCH, and SIMDEX, covering network adaptation, resource management, self-healing, object detection, and backend dispatch.

C. Replay-Based Freshness Metrics

Replay-based metrics re-label admissibility at a later use time while conditioning on different candidate sets. The study separates all-candidate verdict change, oracle-labeled approval expiry, and judge-conditioned use-time invalidity.

  • The deterministic reference checker labels admissibility at check and replay use time, while logged-controller candidates preserve each environment’s state, history, and action distribution.
  • All-candidate verdict change averages verdict transitions over eligible logged candidates before conditioning on any evaluated pass set.It includes both expiry and recovery.
  • Oracle-labeled approval expiry measures, among method-passed candidates admissible at check time, the fraction inadmissible at replay use time.Recovery is treated as a missed opportunity rather than an unsafe stale approval.
  • Judge-conditioned use-time invalidity uses each judge’s approval set without additionally requiring reference admissibility at check time.It can therefore include approvals already inadmissible at check time as well as approvals that expire later.
  • The reference checker is an offline measurement device, whereas FBS estimates whether an upstream approval remains within a temporal horizon.The checker is not deployed in the loop and does not capture every safety-relevant property.

D. Freshness-Bounded Shield

The Freshness-Bounded Shield estimates an approval’s usable horizon from safe-side margin and recent scalar-feature volatility. It gates approval by that horizon, but remains a heuristic rather than a safety certificate.

  • FBS computes a signed safe-side margin from the selected scalar predicate’s threshold and uses recent absolute feature changes as a volatility signal.The margin is nonnegative on the admissible side and positive strictly inside the boundary.
  • The estimated horizon is a heuristic time-to-boundary proxy formed from margin and exponentially smoothed feature volatility.The artifact fixes α=0.1, ε, δmax, initialization, and update order.
  • An approval passes only when upstream approval is positive, margin is strictly positive, and replay shift K does not exceed the floored estimated horizon.Otherwise, the environment-specific fallback is applied.
  • FBS substitutes smoothed recent change for an unknown worst-case drift bound, so abrupt changes can outpace the estimate and invalidate a passed approval.

E. Methods Compared

The comparison uses a reference-open baseline and freshness-aware variants under paired seeds. FBS combines upstream approval with freshness gating, while ablations isolate margin, volatility, and fixed-age effects.

  • The reference-open baseline applies every candidate with positive upstream approval and performs no freshness check.In the oracle-labeled timing experiment, upstream approval equals the reference checker’s check-time verdict.
  • FBS uses the same upstream approval source as reference-open and additionally rejects candidates beyond the estimated freshness horizon.
  • The margin_only and vol_only variants retain only boundary-distance or recent-volatility information, respectively.
  • The e_deadline variant rejects every approval beyond a fixed age regardless of context and appears only in the DELTAIOT safety–utility audit.

F. Reproducibility and Traceability

The evaluation emphasizes reproducible closed-loop experiments and explicitly distinguishes research questions, metric families, and comparison scope. Reproducibility is supported by fixed seeds, paired comparisons, and artifact traceability.

  • All five environments pass a two-run reproducibility test under fixed seeds, with bit-exact or tolerance-bounded reproduction depending on simulator support.
  • Closed-loop comparisons use paired seeds, while the study separately asks about verdict change, FBS expiry reduction, and judge-conditioned invalidity.
  • The three metric families condition on different candidate sets, so numerical gaps across them are not effect estimates.The direct mitigation comparison is reference-open versus FBS within the oracle-labeled timing experiment.

A. RQ1: How Does Verdict Change Vary?

At K=8, verdict-change rates were nonzero across all five environments and varied from 5.3% to 48.4%, while FBS reduced oracle-labeled approval expiry to 0–1.8% from a 3.4–24.7% reference-open range. The observed curves show environment-specific temporal patterns, and the audited DELTAIOT result suggests FBS can avoid the reward loss seen with more conservative alternatives in that cell.

  • Verdict-change rates: 5.3%–48.4%: all-candidate verdict-change rates at K=8, spanning SIMDEX to DELTAIOT across all five environments.This is a common integer replay shift, not a common wall-clock age.
  • Verdict-change rates: DELTAIOT and SIMTUNE are early-saturating, mRUBIS is age-sensitive, SWITCH is approximately flat, and SIMDEX has low-base variation.These labels describe observed curve shapes and do not establish their underlying causes.
  • FBS results: 0–1.8%: FBS oracle-labeled approval expiry versus 3.4–24.7% for reference-open at K=8 in all five environments.Observed FBS-to-reference expiry-rate ratios range from 0 to 0.18, corresponding to relative reductions of 82–100%.
  • FBS results: margin_only reaches 0% approval expiry in every environment, but expiry rates alone cannot distinguish selective gating from conservative rejection.The single-signal variants use different pass sets, so lower expiry alone does not establish a safety–utility benefit.
  • Safety–utility audit: In the audited DELTAIOT K=8 cell, FBS matches reference-open’s −960 mean reward while achieving 0% approval expiry.margin_only and e_deadline reduce mean reward to −1050 and −1200, respectively; this trade-off is not established beyond the audited cell.

D. RQ3: Use-Time Invalidity in LLM Approval Streams

The audit finds use-time invalidity in every evaluated LLM approval stream, while FBS separately targets oracle-labeled approval expiry using freshness signals. These measurements use distinct conditioning sets and do not directly compare judge streams with FBS.

  • Judge-conditioned invalidity: Every audited LLM judge stream has a nonzero observed use-time invalidity rate.The audit covers four LLM judge backends under the same observation-shifting protocol.
  • Measurement distinctions: Oracle-labeled approval-expiry rates condition on method-passed candidates that are reference-admissible at check time.This conditioning defines the direct reference-open versus FBS mitigation comparison.
  • Judge-conditioned invalidity: 11.5%-36.8% is the observed use-time invalidity range at K=8 in DELTAIOT.Each rate is computed within that backend’s own approval set and may combine check-time judge error with temporal expiry.
  • Measurement distinctions: Judge-conditioned invalidity does not isolate temporal expiry, rank judge quality, or directly compare with FBS.The full backend-by-environment matrix is in the artifact, while same-stream Judge+FBS evaluation remains future work.
  • Freshness mitigation: FBS estimates an approval validity horizon from signed margin and recent feature volatility without an explicit dynamics model.Deployment monitoring should track live horizon, margin, volatility, fallback frequency, and task utility.
  • Freshness mitigation: FBS expires an earlier approval without another verifier call, but its horizon is heuristic rather than a safety certificate.Its practical value depends on the deployment’s fallback, and the DELTAIOT audit shows one safety–utility point rather than a general guarantee.

VI. LIMITATIONS

The study’s conclusions are bounded by fixed-action replay, selected simulators and predicates, sparse replay shifts, and fixed FBS settings. Several evaluations and deployment conditions remain unresolved.

  • Scope and evaluation design: The evaluation covers five step-driven simulators, sparse replay shifts, and FBS under fixed settings including α=0.1.Equal K values represent simulator-step shifts rather than equal physical durations.
  • Scope and evaluation design: Deployment-specific freshness estimates require measured latency traces, discretized Klive, and evaluation on the target plant and controller-induced action distribution.Sensitivity to FBS parameters and unsampled ages remains unmeasured.
  • Scope and evaluation design: Fixed-action replay relabels later recorded contexts rather than reconstructing trajectories induced by delaying, rejecting, or replacing actions.The reference checker provides deterministic labels for selected predicates but is not a complete safety oracle.
  • Method boundaries: FBS uses a scalar state feature as a proxy and therefore does not certify every dependency of the full checker.It is a proof-of-concept heuristic rather than a certified safety guarantee.
  • Method boundaries: Utility and fallback trade-offs are tested only on DELTAIOT at K=8.Judge-conditioned rates may mix check-time error with temporal expiry, and same-stream Judge+FBS composition is not evaluated.

VII. CONCLUSION

Across five reproducible SAS environments, approval freshness varies substantially under fixed-action replay, while FBS reduces oracle-labeled approval expiry under fixed settings. The paper concludes that Execute-stage assurance requires validity at use time, not only correctness at check time.

  • 5.3–48.4%: all-candidate verdict-change rates across five environments at replay shift K=8.The roughly ninefold spread reflects differences in plant dynamics, predicate structure, and logged candidate distributions.
  • FBS reduces oracle-labeled approval-expiry rates in all five environments under fixed artifact settings.In the audited DELTAIOT cell at K=8, FBS reaches 0% approval expiry while matching the reference-open baseline’s reported mean reward.
  • Nonzero judge-conditioned use-time invalidity appears in all four audited approval streams.The audit does not isolate temporal expiry from check-time judge error.
  • FBS remains a proof-of-concept heuristic requiring deployment-specific evaluation, calibration, and end-to-end Judge+FBS evaluation.The proposed freshness contract pairs check-time semantic approval with an explicit validity-at-use rule and justified fallback on expiry.
Loading 2608.26306v1…