Source-linked AI summary
G-CARB: Graph-Localized Conformal Agent Risk Budget for Compositional Harm
Zijun Yu, Yu Gu, Vahid Partovi Nia, Masoud Asgharian
TL;DR
Multistep tool use can turn a private read into a later leak, while step-level loss can fail to reflect episode harm. CARB calibrates stopping against a ledger of harm incurred before stopping, and G-CARB selects scorer evidence along observable dependencies. On AgentDojo replay, it roughly halves scorer-input records at intermediate budgets while improving autonomous completion relative to full-prefix scoring; random context of the same size performs similarly.
Problem
A private read can become a leak through a later outgoing action, and action-level labels or rates can miss episode-level harm.
Source: §1 · pp. 1–2Method
Results
G-CARB roughly halves scorer-input records at intermediate AgentDojo risk budgets while improving autonomous task completion relative to full-prefix scoring; random context of the same size achieves similar gains.
Source: §6 · p. 7Takeaways & Limitations
The controlled examples show that retaining relevant dependency evidence can further avoid stopping benign work.
Source: §6 · p. 7The guarantee bounds expected declared ledger loss under exchangeability, not harm in every episode or harm outside the ledger.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Small language model (SLM) agents need safety controls that track consequences across tool calls with little monitoring overhead. A private read, for example, becomes a leak when a later action sends that data outside the system. We introduce CARB (Conformal Agent Risk Budget), which calibrates when to stop an agent using a ledger of harm incurred before stopping. Under exchangeable episodes, standard conformal risk control bounds this declared loss in expectation over calibration and a future episode. G-CARB selects scorer evidence along observable dependencies from private sources to outgoing actions. The ledger still covers the entire executed history, and computing the gate score requires no additional language-model inference. On AgentDojo replay with two 14B backbones, G-CARB roughly halves scorer-input records at intermediate risk budgets while improving autonomous task completion relative to full-prefix scoring; random context of the same size achieves similar gains. Controlled examples show how retaining the relevant dependency can further avoid stopping benign work.
1 Introduction
Multistep tool use can turn a private read into a leak, so safeguards must connect proposed outgoing actions to earlier reads. CARB calibrates stopping against harm accumulated before stopping, while G-CARB selects scorer evidence along observable dependencies and keeps the ledger global.
1 Introduction
- A private certificate read, a note carrying its field, and an external send form a leak across tool calls.Preventing the leak requires connecting the proposed send to the earlier read.Source: §1 · p. 1
- Conformal risk control bounds the expected loss it is given, so a step-rate target can miss episode-level harm.Dividing a correctly labeled harmful event by many executed actions can make the harmful-step rate small while episode harm remains high.Source: §1 · p. 2
- CARB evaluates each stopping policy with a global ledger of harmful events across all actions executed before stopping.Harm incurred before stopping remains charged.Source: §1 · p. 2
- G-CARB selects scorer records along observable dependencies while the ledger continues to cover every executed action.The gate tracks dependencies without invoking another language model at every decision.Source: §1 · pp. 1–2
2 Related Work
Related work studies conformal controls for different agent outcomes and dependency-based defenses. G-CARB uses a graph to select evidence for a separate scorer, while calibration relies on a graph-independent harm ledger.
2 Related Work
- CARB applies conformal risk control to harm accumulated along the execution prefix a stopping policy permits.Prior agent applications target harmful executed actions, retained reasoning traces, completed runs, deployment relative to a safe policy, and other calibrated outputs.Source: §2 · p. 2
- Unlike dependency defenses that enforce security rules, G-CARB uses the graph to supply evidence to a separate scorer and calibrates with an independently defined ledger.The experiments use AgentDojo traces.Source: §2 · p. 2
3 Problem Formulation
The formulation lets a gate score a proposed action from prefix-visible history, then measures harm over the full prefix it permits. It motivates episode-level loss and relevant context because step rates can dilute harm and indistinguishable histories can force benign stops.
3 Problem Formulation
- A fixed selector chooses evidence from history and the proposal, and a fixed scorer determines whether the proposed action executes or the gate stops.The gate sees only history and proposal before execution; the returned observation becomes available afterward.Source: §3 · p. 3
- A stopped-prefix ledger records declared harm events in the executed prefix and is non-decreasing as that prefix grows.Its primary loss is the union indicator over harm categories; once an event occurs, it remains charged through stopping or later recovery.Source: §3 · p. 3
- Long episodes can keep every permitted executed-step rate at or below α while all-execute expected episode loss exceeds α.The construction uses length K ≥⌈1/α⌉ episodes with no-guard harm prevalence p > α.Source: §3 · p. 3
- When harmful and benign histories have matching gate-visible scores, an episode-loss bound forces the matched benign sink not to execute with probability at least 1 − α/p.The expected loss is pq, where q is the probability the sink executes.Source: §3 · p. 3
- The formulation therefore motivates episode-loss calibration and evidence that distinguishes harmful from benign histories.The two obstructions show that step-rate dilution and missing context can undermine episode-level control.Source: §3 · p. 3
4 Methodology
CARB replays logged episodes and applies conformal risk control to stopped-prefix ledger losses, selecting the largest feasible threshold. G-CARB localizes scorer evidence to observable dependencies, while retaining the same global ledger and replay requirements.
4 Methodology
- CARB replays logged episodes at each threshold and selects the largest threshold whose corrected calibration loss is at most α.The correction adds one worst-case loss to the calibration sum, and higher thresholds can only extend the permitted prefix.Source: §4 · p. 4
- G-CARB supplies nodes on source-to-proposal paths when the proposal is a reachable sink; otherwise it supplies the proposal and its ancestors.The graph is built from prefix-visible fields, with frozen tool metadata identifying private sources and designated sink types.Source: §4 · p. 4
- In the certificate–note–email example, G-CARB retains the reused private-field chain and omits a disconnected public task, while the ledger evaluates every executed record.An email carrying only the public document instead follows the public branch.Source: §4 · p. 4
- The scorer uses fixed weighted binary features, and computing its score requires no language-model inference.G-CARB and Full-prefix share the graph builder, node types, scorer, grid, and ledger; only selected records differ.Source: §4 · p. 4
- Under exchangeable episodes and a fixed pipeline, stopped-prefix conformal risk control bounds expected ledger loss over calibration and a future episode.The corrected criterion requires α ≥ 1/(n + 1); below this floor it is infeasible.Source: §4 · p. 4
- Replay matches online stopping only when the wrapper is silent and side-effect-free before rejection and observes the same prefix-visible fields.Warnings, prompt edits, or retries before stopping can change later actions, so evaluating those interactions requires trajectories collected with them.Source: §4 · p. 5
5 Experiments
The experiments test whether episode-level ledger calibration controls risk and whether restricting scorer context preserves task completion. On AgentDojo, context restriction improves autonomous success at intermediate budgets; constructed suites examine when retaining relevant dependencies avoids benign stops.
5 Experiments
- AgentDojo replay uses 984 ungated episodes per 14B backbone, with 200 shared seeded assignments split into 492 calibration and 492 test episodes.Methods share the assignments, ledger, and threshold grid; results are interpreted conditional on the fixed corpus and scorer.Source: §5 · pp. 5–6
- CES tests 80 constructed leaks against 240 matched non-leaks, while graph-braid interleaves harmful chains with unrelated benign branches in 2,000 trajectories.These suites test specified failure mechanisms at harm frequencies chosen by construction; grouped CES results are descriptive because its splits fall outside Corollary 1.Source: §5 · p. 5
- At three of four displayed points, executed-step CRC mean episode loss matched no guard (.133, .133, and .225), while ledger-calibrated methods stayed below budget.The AgentDojo comparison changes both calibration target and context; the constructed-suite comparisons do not isolate the effect of changing the target.Source: Figure 2 · p. 6, §5 · p. 6
- CtxRed measures omitted scorer-input records, while autonomous success counts logged tasks completed before handoff, including completions that coexist with ledger events.CES benign stops are measured over non-leaks; graph-braid counts first stops on disconnected benign branches over all trajectories.Source: §5 · pp. 5–6
- At intermediate budgets, context restriction improves autonomous success; at budgets .05 and .10, G-CARB supplies roughly half the full-prefix record count.Equal-size random context closely follows G-CARB, while fixed-window context stays nearer Full-prefix.Source: §5 · p. 6, Figure 2 · p. 6
- In CES and graph-braid, every gated method stops harmful sinks, while G-CARB avoids benign stops in both constructions.Ledger-current matches G-CARB on graph-braid, and Full-prefix matches it on CES clean; G-CARB handles both by retaining relevant history and excluding unrelated records.Source: §5 · pp. 6–7, Figure 3 · p. 7
6 Conclusion
CARB calibrates stopping against accumulated harm, while G-CARB selects dependency evidence for scoring and retains a ledger covering the executed history. AgentDojo replay shows improved task completion and roughly halved scorer-input records at intermediate budgets, but the evidence and guarantee have defined scope limits.
6 Conclusion
- CARB applies standard CRC to accumulated harm before stopping, and G-CARB selects dependency evidence while retaining that global loss.The AgentDojo replay reports improved task completion and roughly halved scorer-input records at intermediate budgets; random context performs similarly.Source: §6 · p. 7
- The guarantee bounds expected declared ledger loss under exchangeability, not harm in every episode or harm outside the ledger.The ledger lacks an independent error audit, and semantic or paraphrased flows may escape observable matching rules.Source: §6 · p. 7
- AgentDojo evidence uses a fixed corpus, one scorer, and two same-family 14B backbones; interactive interventions and total gate overhead remain unevaluated.Smaller backbones, alternative scorers, and outcomes after handoff remain evaluation priorities.Source: §6 · p. 7
A Theory and proofs
The theory motivates episode-level harm accounting by showing that low executed-step rates can coexist with high episode risk, and that indistinguishable harmful and benign histories constrain what a gate can do. It establishes stopped-prefix conformal risk control under exchangeability and describes the replay and graph-selection conditions used by G-CARB.
A.1 Control unit and matched histories
- A small executed-step rate can coexist with episode risk above the budget, while matching gate-visible histories forces a benign sink to be stopped with probability at least 1 − α/p.The counterexamples motivate calibrating episode loss and retaining evidence that distinguishes histories.Source: §A.1 · p. 11
A.2 Ordered-grid CRC and stopped-prefix calibration
- Conditional on exchangeable episodes and a frozen deterministic pipeline, stopped-prefix CRC bounds expected ledger loss; its corrected selection criterion is infeasible below α = 1/(n + 1).Dependence within an episode is unrestricted; exchangeability is required across episodes.Source: §A.2 · pp. 11–12
A.3 Replay equivalence
- Replay equivalence holds when paired runs share proposals, observations, graph contexts, scores, and randomness through rejection, but not when wrappers alter prompts or environments.The ledger agrees because it is deterministic on the common executed prefix.Source: §A.3 · p. 12
B.1 Calibration algorithm
- Stopped-prefix calibration replays each logged proposal through a frozen selector and scorer, stops before proposals exceeding the threshold, and charges earlier harm in the ledger.The algorithm applies standard CRC to the replayed losses.Source: §B.1 · p. 12
B.2 Prefix graph and source-to-sink selector
- G-CARB builds a temporally oriented graph from observable record relations and selects source-to-sink evidence, falling back to the proposal and its ancestors; these matching rules are not verified semantic dataflow.The graph excludes future outcomes and labels; shared identifiers or exact spans can add irrelevant edges, while paraphrase or unlogged state can omit needed ones.Source: §B.2 · p. 13
B.3 Frozen weighted scorer
The experiments use a frozen weighted scorer whose retained source-to-sink path feature differs from features that can raise scores for disconnected records. The record-count comparison concerns scorer inputs and excludes graph-construction and other wrapper costs.
B.3 Frozen weighted scorer
- Only a retained private-source-to-sink path activates the scorer’s .52 provenance feature, while earlier selected private sources, canaries, and injection text can raise scores even when disconnected.Selectivity with learned scorers or alternative context weights remains untested.Source: §B.3 · p. 13, Table 1 · p. 14
B.4 Record-count scope
- Full-prefix scoring supplies O(T^2) records over T decisions, versus O(Td) when each selected component has at most d records.These are conditional record-count bounds, exclude graph construction and other wrapper costs, and imply token or byte bounds only with bounded record length.Source: §B.4 · p. 14
C AgentDojo ledger, corpus, and preprocessing
The AgentDojo evaluation uses a deterministic ledger of executed, prefix-visible events and separates those labels from graph and scorer inputs. Its assignments and calibration targets define the scope of the reported diagnostics, with constructed-suite results not covered by the uniform episode-permutation corollary.
C AgentDojo ledger, corpus, and preprocessing
- The custom episode loss is the union of five binary event categories over executed records, and registered events persist after they occur.Events require executed actions and confirmed results; private reads without an executed sink do not register privacy flow.Source: C AgentDojo ledger, corpus, and… · p. 14
- The gate cannot use ledger labels or benchmark outcomes, and the official attack evaluator is excluded from scoring, calibration, and operating-point selection.The reported risk target is the operational prefix ledger, not terminal benchmark outcomes.Source: C AgentDojo ledger, corpus, and… · pp. 14–15
- Canary preprocessing differed between backbones, its effects were not quantified, and the ledger has not been independently audited against human annotations or benchmark outcomes.Exact-canary detection may miss paraphrased or transformed disclosure; the guarantee concerns the declared operational ledger, not all harms or semantic accuracy.Source: C AgentDojo ledger, corpus, and… · pp. 14–15
D.1 Frozen pipeline and assignments
- AgentDojo uses 200 seeded episode permutations with 492 calibration and 492 test episodes per backbone; these assignments summarize sensitivity of a fixed corpus, not independent datasets.Constructed CES results are descriptive held-out diagnostics because the uniform episode-permutation corollary does not cover grouped CES.Source: §D.1 · p. 15
- The equal-size random control retains the proposal and a deterministic pseudorandom subset of earlier nodes matching the graph selector’s size at each decision.Contexts must remain fixed across thresholds for replay, or prefix nesting can be lost.Source: §D.1 · p. 15
D.2 Calibration targets and context controls
- Executed-step CRC calibrates a running maximum of raw step rates over the threshold grid, while held-out results report the raw step rate.The raw rate can fall as harmless suffixes grow, motivating the nondecreasing calibration target.Source: §D.2 · p. 16
D.3 Reported metrics and denominators
This block defines the reported risk, intervention, utility, and context-reduction metrics, then reports AgentDojo and constructed-cohort results across operating points and controls.
D.3 Reported metrics and denominators
- Intervention is the fraction of trajectories with a rejected proposal; autonomous success counts logged task completion before handoff, which can coexist with a ledger event.Harm stop and benign-stop rates use different denominators, and no joint harm-free success rate is reported.Source: §D.3 · p. 16
F.1 Construction
- The context-reduction denominator is the counterfactual full-prefix record count on the same scored proposals, while assignment percentiles describe corpus sensitivity rather than future-deployment guarantees.The CES cohorts contain 80 leaks and matched non-leaks in the clean cohort, and 80 leaks plus 240 non-leaks in the near-miss cohort; graph-braid labels are set by construction.Source: §D.3 · p. 16, §F.1 · pp. 18–19, Table 8 · p. 19
E.1 Selected operating points
- At α = .05 and .10, G-CARB’s CtxRed is .565 and .597 for Qwen2.5, and .511 and .590 for Qwen3.The no-guard mean episode losses are .133 and .225 for the two backbones.Source: §E.1 · p. 17, Table 4 · p. 17
E.2 Thresholds and feasibility
- Corrected calibration risks meet their displayed budgets across 3,200 selections, but this does not guarantee every held-out assignment meets its budget.Numerically positive thresholds can still produce behavioral stop-all.Source: §E.2 · p. 17, Table 5 · p. 17
E.3 Paired assignment sensitivity and endpoints
- At α = .025, Qwen2.5 autonomous success is .0078 for G-CARB and .0098 for Full-prefix; at .20, both match no guard with U = .436.Qwen3 is behaviorally stop-all at .025; at .20 it remains gated, with U = .189 for G-CARB and .162 for Full-prefix.Source: §E.3 · p. 17, Table 6 · p. 18
E.4 Complete window and random-context controls
- Random context has mean ledger loss .035 for Qwen2.5 and .041 for Qwen3, equal to G-CARB at displayed precision, with no demonstrated natural-data advantage for dependency selection.The complete context-control grid compares selectors using CtxRed computed over each method’s own scored proposals.Source: §E.4 · p. 18, Table 7 · p. 18
F.2 Results and ties
Constructed-cohort results distinguish selector behavior across CES and graph-braid traces, while an evidence-removal audit tests whether observable source-to-sink paths remain available.
F.2 Results and ties
- On CES near-misses, G-CARB avoids benign stops where Full-prefix and Fixed-window-4 stop one third and Ledger-current and equal-size random stop two thirds.On CES clean, Full-prefix and Fixed-window-4 match G-CARB on loss, intervention, and utility; the equal-size comparison supports a role for record choice in these constructed cohorts under this scorer.Source: §F.2 · p. 19
- On graph-braid, Ledger-current matches G-CARB on loss, intervention, and utility; equal-size random has .079 benign-branch first stops and .905 intervention, versus .000 and .390 for G-CARB.Random and G-CARB are size-matched at each decision but have CtxRed .450 and .471 because their stopping points change the scored-proposal sets.Source: §F.2 · p. 19
G.1 Fixed-threshold evidence removal
- Removing both structured provenance and exact sink text reduces harmful-path recall to zero and yields mean ledger loss .500, equal to no guard on the balanced cohort.Either remaining observable channel retains all harmful paths in this construction; the threshold is calibrated in one cell and reused across the four evidence conditions, so this is not a four-cell conformal certificate.Source: §G.1 · p. 19, Table 9 · p. 20
G.2 Graph-braid perturbations with recalibration
The graph-braid sweep recalibrates separately under each perturbation, and the online check evaluates interface consistency rather than population risk. The discussion limits deployment claims to settings with validated ledgers and evidence channels.
G.2 Graph-braid perturbations with recalibration
- At 20% edge dropout, intervention reaches 1.0 and every episode is stopped, illustrating a possible utility cost when dependency evidence is degraded.The sweep recalibrates each condition; it does not test fixed-threshold transfer under distribution shift, and assignment standard errors are not deployment guarantees.Source: §G.2 · p. 20, Figure 4 · p. 21
H Online interface check
- Online decisions match deterministic same-code recomputation on all 135 primary proposals, but selected episodes do not establish exchangeability with a deployment population.The check is an integration diagnostic; independent ledger validation and population risk evaluation require separate evidence.Source: H Online interface check · p. 20
I Ethics and broader impacts
- Experiments use sandboxed synthetic records and canary strings, not real private data; deployment requires validating the ledger, evidence channels, and handoff process.The experiments do not evaluate outcomes after human handoff.Source: I Ethics and broader impacts · p. 21