PaperlayerView exact revision on arXiv

Source-linked AI summary

LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents

Jingyue Cong, Estrid He, Zhuang Li, Nan Wu, Li Shen, Yameng Peng, Xinshu Li, Andy Song

arXiv:2610.17685v1cs.CLcs.AIcs.LG

TL;DR

Enterprise-agent safety benchmarks often assess local signals, while policy risk can depend on workflow state accumulated across prior steps. LATENTRISKBENCH pairs trajectories with the same terminal action but different policy-relevant states and introduces state-dependent risk metrics. Action-level evaluation detects 0.09 of state-dependent violations, compared with 0.96 for state-dependent auditing.

  • Problem

    Existing agent-safety benchmarks often do not test routine-looking terminal actions whose policy risk depends on workflow state established by prior steps.

    Source: §1 · pp. 1–2

  • Method

    LATENTRISKBENCH pairs risky trajectories with benign twins sharing the same terminal action but differing in policy-relevant state, and reports state-dependent risk metrics including residual risk.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    0.09 of state-dependent violations are detected by an action-level checker, versus 0.96 by an auditor that reconstructs workflow state.

    Source: §1 · p. 2, §6 · p. 6

  • Takeaways & Limitations

    LATENTRISKBENCH provides a pre-deployment resource for testing whether guardrails detect state-dependent policy risk before use in enterprise workflows.

    Source: §6 · p. 6

    The study uses an e-commerce marketing sandbox with modest per-category sample sizes, and shared LLM bias in the world-model auditor cannot be fully ruled out.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local risks, overlooking failures whose safety depends on workflow state. In such state-dependent risks, the same action may be safe in one context but violate policy in another. Hence, we introduce LatentRiskBench for evaluating such state-dependent risks in enterprise settings. It pairs risky trajectories with benign counterparts that share the same terminal action but differ in latent policy-relevant states, requiring models to reconstruct workflow context rather than rely on action-level cues. We also introduce new metrics for state-dependent risk evaluation, including a suite of residual risk (RR) measures. Experiments on the benchmark show that action-level evaluation detects only 0.09 of state-dependent violations, whereas state-level evaluation reaches 0.96. Existing guardrail methods largely overlook such risks unless they are explicitly applied to workflow states. Case studies on a commercial deployed scenario further show that the safety metrics embedded in the benchmark provide a more realistic measure of safety for enterprise agents. Our dataset and code are available at https://github.com/JingyueCong/LatentRiskBench.

1 Introduction

Enterprise-agent risks can depend on workflow state established by prior steps, so routine-looking actions may be unsafe in context. LATENTRISKBENCH tests whether auditors reconstruct that state, finding a large gap between action-level and state-level detection.

1 Introduction

  • Existing benchmarks often expose decisive evidence in a local output, proposed action, or recognizable trace pattern rather than in workflow state accumulated across prior steps.In enterprise workflows, missing approvals or restricted content can make an otherwise routine action risky.Source: §1 · pp. 1–2
  • State-dependent risk occurs when the same terminal action is safe in one trajectory but policy-violating in another because prior steps induce different policy-relevant states.Risk depends on the location of the signal, not trajectory length.Source: §1 · pp. 1–2
  • LATENTRISKBENCH pairs risky and benign trajectories that share a terminal action but differ in policy-relevant states, alongside visible-risk controls and an e-commerce marketing instantiation.Its core includes 100 state-dependent tasks.Source: §1 · p. 2
  • 0.09 of state-dependent violations are flagged by an action-level checker, versus 0.96 by an auditor that reconstructs workflow state.Representative guardrails retain this blind spot unless they explicitly reconstruct and reason over states.Source: §1 · p. 2

2 Related Works

Prior agent-safety benchmarks cover multiple forms of unsafe behavior, but their decisive risk signals are often local. Enterprise workflows can instead make the same terminal action compliant or violating depending on accumulated state.

2 Related Works

  • Existing benchmarks cover web navigation, tool use, computer use, prompt-injection robustness, and policy compliance, but commonly expose risk through a local signal.Examples include sensitive tool output, an explicitly unsafe action, or a recognizable trace pattern.Source: §2 · p. 2
  • In enterprise workflows, the same terminal action can be compliant or policy-violating depending on accumulated business records, approvals, prior actions, and policy constraints.The decisive signal is reconstructed policy-relevant workflow state, rather than the action surface form or one observation.Source: §2 · p. 2
  • Existing benchmarks poorly cover risks whose decisive evidence lies in policy-relevant workflow state rather than in a local signal.Figure 2 is cited as showing this coverage gap.Source: §2 · p. 2

3 LatentRiskBench

LatentRiskBench evaluates enterprise-agent risks that depend on workflow state, pairing risky trajectories with benign twins that share the same terminal action. It reconstructs policy-relevant state and uses state-dependent auditing and metrics to assess these risks.

3 LatentRiskBench

  • LatentRiskBench pairs labeled risky trajectories with benign twins and visible-risk controls, using a state-dependent auditor to assess policy violations.Source: §3 · p. 2

3.1 State Reconstruction

  • State reconstruction maps the committed prefix and environment context into policy-relevant workflow variables, using supported facts rather than copying full logs.Source: §3.1 · p. 3

3.2 Benchmark Curation

  • Curation starts from compliant enterprise workflows and edits earlier steps to violate a policy-gating state variable while preserving the task and terminal action across four mechanisms.Source: §3.2 · p. 3
  • Manual validation requires action-invisible violations, state detectability, and path-invariant labels; the filtered core has 124 traces, with 24 local-risk controls, plus 40 deployment-derived e-commerce traces.Source: §3.2 · p. 3

3.3 State-Dependent Auditing

  • The state-dependent auditor reconstructs trajectory state to judge whether the terminal action violates policy, while the action-local evaluator sees that action but not the prefix or environment context.Source: §3.3 · pp. 3–4

3.4 Metrics

  • Residual Risk measures the fraction of risky trajectories where a safety control leaves a violation unremoved, with overall and risk-stratum values, alongside benign admission.Source: §3.4 · p. 4
  • State-Reconstruction Gain measures the additional detection rate of the state-dependent auditor over the action-local evaluator on the same risky terminal actions.Source: §3.4 · p. 4
  • Constant-Action Specificity checks whether matched benign twins with the same terminal action are admitted when their reconstructed state makes that action compliant.Source: §3.4 · p. 4

4 Experiments

The experiments compare guardrails and auditor backbones, finding that state reconstruction detects state-dependent violations missed by action-local checks. They also assess guardrail residual risk and validate the auditor against human labels and alternative backbones.

4.1 Experimental Setup

  • The setup compares lexical, rule-based, LLM-judge, and trajectory-level defenses, with OVR and residual-risk measures lower-is-better and TSR higher-is-better.RR is measured by the state-dependent auditor; LlamaGuard 3 is included as a moderation-style boundary case.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Guardrail Residual Risk

  • Prefix-checking TRAJRULE leaves high residual risk, while LLMJUDGE errors concentrate in the state-dependent regime; stronger trajectory reasoning reduces SD-RR most, but risk remains non-trivial.Additional guardrails and a full-prefix reader without state reconstruction also fail to close the SD gap.Source: §4.2 · pp. 4–5

4.3 State-Reconstruction Gain

  • 0.09 of violations were detected by action-local auditing, versus 0.96 by state-dependent auditing, for SRG = 0.87 on the 100 state-dependent trajectories.The paired difference was significant (McNemar’s exact two-sided test, p < 10−15).Source: Figure 3 · p. 5, §4.3 · p. 5

4.2 Guardrail Residual Risk

  • Utility remains high across non-trivial guardrails, and false-positive rates are low on 100 same-action benign twins, so lower residual risk is not achieved by broad blocking.Policy-class and mechanism breakdowns are reported in Appendices F.1 and F.6.Source: §4.2 · p. 5

4.4 Auditor Reliability and Robustness

  • The auditor agrees with human majority labels at κ = 0.88 overall, κ = 0.83 on SD, and κ = 1.00 on AR; its main weakness is mild over-conservatism on short, locally visible tasks.Two annotators had high agreement (κ = 0.90).Source: §4.4 · p. 5
  • Across auditor backbones, action-local detection remains low while state-dependent detection remains substantially higher, suggesting the SRG is not specific to GPT-4o-mini.GPT-4o-mini is the primary auditor; other backbones are robustness checks under the same protocol.Source: §4.4 · p. 5, Table 2 · p. 5

5 External-Validity Audit of an E-commerce Marketing Agent

The section applies the benchmark’s audit to anonymized trajectories from an enterprise e-commerce marketing agent. It compares conventional outcome metrics with state-dependent residual-risk measures, validates flagged actions offline, and reports operational costs and scope.

5.1 Setting

  • The audit uses anonymized committed trajectories from an e-commerce marketing assistant whose workflows include campaign preparation and syncing drafts to a marketing platform.The agent follows a REACT-style tool-calling loop and is evaluated under the benchmark’s enterprise policy family.Source: §5 · p. 5, §5.1 · pp. 5–6

5.2 Audit Results

  • OVR=0.08 and TSR=0.92 appear reassuring, but RR=0.61 and SRG=0.49 expose substantial pre-execution risk that outcome summaries miss.The benign false-positive rate is 0.025 (admission 0.975), indicating the audit is not broadly flagging routine marketing actions.Source: §5.2 · p. 6, Table 3 · p. 6

5.3 Counterfactual Harm Validation

  • Flagged terminal actions are replayed in an inert sandbox, and deterministic event-log rules count only transitions that would realize a policy violation.A routine campaign-sync action can be risky when the reconstructed state shows restricted internal pricing information in a draft sent externally.Source: §5.3 · p. 6

5.4 Operational Cost and Scope

  • Auditing adds 430 ms median latency, 1.9 s p95 latency, and an estimated $0.0028 per action.The authors describe selective inline auditing for actions with external side effects, with asynchronous auditing or sampling for lower-risk internal actions.Source: §5.4 · p. 6
  • This is an offline external-validity audit, not a production enforcement study; the controlled benchmark remains the primary quantitative evidence.Verdicts were computed over anonymized committed trajectories and were not used to block the live agent.Source: §5.4 · p. 6

6 Conclusion

The conclusion presents LATENTRISKBENCH as a deployment-oriented resource for evaluating state-dependent residual risk in enterprise agents. It reports that state-aware auditing recovers violations missed by action-local evaluation and identifies pre-deployment testing as the benchmark’s use.

6 Conclusion

  • Action-local evaluation detects only 0.09 of state-dependent violations, compared with 0.96 for state-dependent auditing (SRG = 0.87).Representative guardrails leave substantial residual risk when they fail to reconstruct and reason over workflow state.Source: §6 · p. 6
  • LATENTRISKBENCH targets policy violations whose decisive evidence lies in policy-relevant workflow state accumulated across a trajectory, not in the terminal action or a single observation.Matched benign twins show that the auditor responds to state differences rather than action surface form.Source: §6 · p. 6
  • The benchmark provides a pre-deployment resource for testing whether agent guardrails detect state-dependent policy risk before enterprise use.Human validation, robustness checks, and an offline audit of a real enterprise marketing agent support the benchmark’s reliability and practical relevance.Source: §6 · p. 6

Limitations

The authors identify benchmark design, the auditor’s role as measuring instrument, and the study’s domain and sample size as limitations on interpreting and transferring results.

Limitations

  • Because SD trajectories are authored with benign-looking final actions, measured SD difficulty is partly a property of the benchmark design.The trajectories become risky only after environment state is reconstructed.Source: Limitations · p. 7
  • The world-model auditor may share LLM bias, which independent LLM and human annotation do not fully rule out.The auditor is held out from the defense matrix and its judgments are validated against independent annotations.Source: Limitations · p. 7
  • The e-commerce marketing sandbox and modest per-category samples limit how directly absolute numbers, policy coverage, and inline deployment costs transfer to other domains.The authors state that transfer requires re-validation.Source: Limitations · p. 7

A Broader Related Work

The related-work section distinguishes capability and security benchmarks from evaluation of state-dependent policy risk. It positions LATENTRISKBENCH as testing whether evaluators use reconstructed workflow state and measure residual risk beyond conventional outcomes.

Agent evaluation and task completion

  • Task-completion benchmarks assess interactive capability, but completion alone does not establish whether actions satisfy enterprise policies; LATENTRISKBENCH instead evaluates residual policy risk under accumulated workflow state.The same terminal action may be compliant or violating depending on prior records, approvals, actions, and policy constraints.Source: A Broader Related Work · p. 9

Agent security benchmarks

  • Security benchmarks often expose step-local risks, whereas LATENTRISKBENCH targets cases where neither a single step nor the terminal action determines whether a violation occurs.Examples of commonly visible signals include injected instructions, sensitive content in an observation, or a directly unsafe proposed action.Source: A Broader Related Work · p. 9

Risk awareness and policy compliance from traces

  • Multi-turn trace access alone does not test state-dependent risk; holding the terminal action fixed while varying workflow state tests whether evaluators use reconstructed state over surface cues.Prior trace and policy-compliance benchmarks can rely on explicit unsafe content, recognizable sequence patterns, or outcome-level violations.Source: A Broader Related Work · p. 9

State-dependent residual-risk auditing

  • The benchmark complements conventional task-success, attack-success, compliance, and harm metrics with measures of state detectability, benign-twin admission, and risk left after controls.State-Reconstruction Gain measures added detectability from reconstructing state; Constant-Action Specificity checks admission of same-action benign twins; Residual Risk measures policy-violating risk remaining after safety controls.Source: A Broader Related Work · p. 9

B Benchmark Construction Details

LATENTRISKBENCH constructs enterprise safety cases where the same terminal action is safe or unsafe depending on policy-relevant workflow state. Its controlled suites, auditor, validation protocol, and metrics are designed to separate state-dependent risk from action-local risk.

B.1 Suite Composition

  • The general suite combines 30 short/local-visible tasks, 24 action-revealing controls, and a 100-task state-dependent core with matched benign twins.The e-commerce instantiation adds 40 state-dependent tasks and 40 matched benign twins.Source: §B.1 · pp. 9–10, Table 4 · p. 10

B.2 Construction and Certification

  • Risky items alter policy-relevant state while keeping the terminal action fixed, then require the violation to be recoverable from reconstructed state rather than an individual prefix step.Certification also checks that the label remains stable when path wording, ordering, or intermediate observations change while reconstructed state is preserved.Source: §B.2 · p. 10

B.3 Risk-Observability Labels

  • Risk-observability labels distinguish locally visible risks from state-dependent risks whose terminal action is benign or ambiguous without reconstructed workflow state.The state-dependent core uses four construction mechanisms; mechanisms describe why risk becomes visible, while policy classes specify the violated policy.Source: §B.3 · p. 10, §B.4 · p. 10, §B.5 · pp. 10–11

B.5 Policy Classes and Record Schema

  • Benchmark records link intent, committed prefix, terminal action, policy, risk label, mechanism, matched twin, and oracle rationale.The cases illustrate state accumulation, destination dependence, authorization dependence, and goal-shift dependence with same-action benign twins.Source: §B.5 · p. 11, §B.6 · p. 11

C Auditor Prompt and Output Format

  • The state-dependent auditor reconstructs policy-relevant facts from intent, committed prefix, environment context, terminal action, and policy set, then returns a risk label and evidence trace.It is a measurement instrument, not a runtime defense; the action-local contrast receives the same terminal action and policies but no prefix or environment context.Source: C Auditor Prompt and Output Format · pp. 12–13

D Human Annotation Protocol

  • Blind annotation uses different views for state-dependent and action-local cases, and reports annotator and auditor agreement plus per-stratum recall and specificity.Specificity matters on matched benign twins because the auditor should admit compliant cases sharing the risky case’s terminal action.Source: D Human Annotation Protocol · p. 13, §E.5 · p. 14

E.3 Guardrail Residual Risk and Utility

  • Residual Risk measures risky trajectories left policy-violating by a guardrail, while Task Success Rate measures successful completion of benign workflows.RR is reported overall and by risk stratum; unnecessary blocking, rewriting, or escalation on matched benign twins counts as a false positive.Source: §E.3 · p. 14

E.4 Conventional Outcome-Level Summary

  • Outcome Violation Rate summarizes risky workflow outcomes, whereas RR evaluates pre-execution residual violations under reconstructed state.Auditor verdicts are checked against human annotations and robustness measures including Cohen’s kappa, recall, and specificity.Source: §E.4 · p. 14, §E.5 · p. 14

E.2 Benchmark Diagnostics

The benchmark diagnostics compare action-local detection with detection after workflow-state reconstruction, then assess whether the auditor and benchmark labels are reliable. State-dependent risks show a large detectability gap, while action-revealing controls remain locally detectable.

E.2 Benchmark Diagnostics

  • SRG is the difference between state-dependent and action-local detection rates on the same risky split, measuring detectability gained from reconstructing trajectory state.Both evaluators receive the same terminal action, so the diagnostic isolates the contribution of reconstructed state.Source: §E.2 · pp. 13–14
  • CAS measures the fraction of matched benign twins admitted as compliant, testing whether the auditor avoids treating a shared terminal action as universally unsafe.High SRG together with high CAS indicates state-dependent detection without blocking the shared action.Source: §E.2 · p. 14

F.1 Detectability by Regime

  • On the controlled general suite, the state-dependent core shows a large gap between action-local and state-dependent detection, while the action-revealing control is locally detectable with near-zero SRG.The e-commerce instantiation follows the same pattern; its mechanism-level breakdown is reported separately in Appendix I.Source: §F.1 · p. 15

F.2 Auditor Reliability

  • κ was 0.90 between annotators and 0.88 between the auditor and human majority overall, with auditor agreement of 0.83 on SD and 1.00 on AR.These results indicate that independent annotators can recover the labels and that the auditor is not merely matching construction labels.Source: §F.2 · p. 15

F.3 Backbone and Instrument Robustness

Robustness analyses test whether the state-dependent detectability gap depends on a particular auditor, access to the prefix, or one construction pattern. The results distinguish state reconstruction from simply providing more trajectory text and extend the benchmark with examples, policy synthesis, and an e-commerce instantiation.

F.3 Backbone and Instrument Robustness

  • An independent auditor preserves the pattern that action-revealing risks are action-detectable while state-dependent risks require reconstructed state, including in e-commerce.This supports that the observed SRG is not specific to one auditor backbone.Source: §F.3 · p. 15

F.4 Prefix Access versus State Reconstruction

  • A prefix reader that sees the full committed trajectory but does not reconstruct a policy-relevant state record does not close the state-dependent detection gap.The reported interpretation is that the improvement comes from state reconstruction, not merely exposing more tokens.Source: §F.4 · p. 15

F.5 Computational Cost and Latency Trade-offs

  • TRAJRULE and LLAMAGUARD3 have lower latency in the reported implementation, while GUARDAGENT and the state-dependent auditor have comparable runtime overhead.GUARDAGENT achieves the lowest SD-RR among evaluated guardrails, and the state-dependent auditor detects 0.96 of SD violations.Source: §F.5 · p. 15, Table 9 · p. 15

F.6 Policy-Class and Mechanism Breakdowns

  • The hardest state-dependent cases distribute relevant evidence across accumulated content, destination, authorization, and goal-alignment cues, rather than one construction template or policy class.The authors interpret the headline gap as a risk-observability effect, not a narrow artifact of one policy class or mechanism.Source: §F.6 · p. 15

G Example Tasks and Trajectories

  • Action-revealing examples expose the violation in the trigger action itself, whereas state-dependent examples require preceding trajectory context to interpret the action.Matched benign twins preserve surface structure while removing the policy violation, testing whether defenses preserve legitimate task completion rather than simply block similar actions.Source: §G.1 · pp. 15–16, §G.2 · p. 16, §G.3 · p. 16

H Coverage-Driven Policy Synthesis

  • The supporting policy-synthesis procedure adds only gap-filling policies and accepts a run only when every risk item maps to at least one policy.Synthesized policies are grounded in risk items and linked to their provenance; this procedure is independent of the benchmark’s construction, matched-twin design, and residual-risk metrics.Source: H Coverage-Driven Policy Synthesis · pp. 16–17

I.1 Representative State-Dependent Case

  • The e-commerce suite pairs each risky task with a same-action benign twin, and its marketing example becomes risky when restricted internal information reaches an external campaign destination.The benign twin preserves the sync action but changes the reconstructed state to approved content or an authorized internal destination.Source: I E-commerce Marketing Instantiation · p. 17, §I.1 · p. 17

J Deployment-Style Audit Details

  • The deployment-style audit applies the benchmark’s policy family to anonymized committed trajectories from an enterprise marketing agent as an external-validity bridge, not primary quantitative evidence.The audit covers workflows such as drafting promotional copy, reading product and segment data, and preparing campaign content.Source: J Deployment-Style Audit Details · p. 17

J.1 Counterfactual Harm Validation

Deployment-style audits validate flagged actions through inert replay and report operational costs and scope boundaries. The mechanism breakdowns support interpreting SRG as an observability diagnostic for the controlled benchmark, not a prevalence estimate for organic traffic.

J.1 Counterfactual Harm Validation

  • Flagged actions are replayed under reconstructed state, and only deterministic policy-violating transitions count toward deployment-style RR.This makes deployment-style RR comparable to benchmark residual-risk measurements.Source: §J.1 · p. 18

J.2 Operational Cost

  • The audit adds 430 ms median latency and 1.9 s p95 latency per audited action, at an estimated cost of $0.0028 per action.The authors note that external-side-effect actions could be audited inline while lower-risk internal actions could be audited asynchronously.Source: §J.2 · p. 18

J.3 Scope and Ethics

  • The audit ran offline on anonymized trajectories; no real external side effects were triggered, and verdicts were not used to block the live agent.Inline deployment would require logging, rollback support, and human escalation.Source: §J.3 · p. 18

K State-Dependent Mechanism and Residual-Risk Breakdowns

  • Across SD mechanisms, action-local detection remains low and state-dependent detection high; AR has near-zero SRG because the terminal action exposes the violation.The authors caution that SRG is a risk-observability diagnostic, not an estimate of state-dependent risk prevalence in organic agent traffic.Source: Table 15 · p. 18, Table 16 · p. 18, K State-Dependent Mechanism and… · p. 18

Related papers

Loading 2610.17685v1…