Source-linked AI summary
Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations
Leon Goldberg, Gal Engelberg, Eden Yavin, Elad Elouz, Ariel Zadok, Konstantin Koutsyi
TL;DR
Cloud-security investigations often ask population-level questions whose answers require a complete inventory, while coding agents can investigate through live cloud APIs. This paper compares those agents with an offline-resolved security context layer on 28 AWS tasks using pooled, grounded coverage; the context layer scores higher, and the paper describes a sample-and-generalise pattern in agent answers.
Problem
Cloud-security population questions require complete inventories, but coding agents can reconstruct relevant structure from live APIs, leaving open what a purpose-built context layer contributes.
Source: §1 · p. 1Method
The study compares the Sola Security Brain, whose relational substrate is resolved offline, with Claude Code and Codex on the same 28 AWS investigations using blinded, tier-weighted, grounding-gated pooled relative recall.
Source: §1 · pp. 1–2, §4 · p. 6Results
0.549 ± 0.012 coverage for the Sola Security Brain compares with 0.340 ± 0.011 for Claude Code and 0.281 ± 0.006 for Codex, with identical ordering in every grading draw.
Source: §7 · p. 10Takeaways & Limitations
Both coding agents sometimes generalize from partial enumeration to an unhedged universal negative, disclosing sample size only in answer metadata.
Source: §6 · p. 9, §7 · p. 10The measure is relative recall over a merged judged set, not recall against the environment; findings omitted by all systems are invisible.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Cloud-security investigation is dominated by population tasks: which identities can read a data store, how many resources fail a control, what is reachable from another account. These resolve against a complete inventory, not a named object, so a partial answer to one is not a partial result but a different one. Coding agents can now be given read-only cloud credentials and asked to investigate directly, which raises the question of what a purpose-built security context layer still contributes. We evaluate the Sola Security Brain, a security intelligence layer whose relational substrate is resolved offline and whose security logic is evaluated against it at query time, against two coding agents operating the same live AWS environment through a read-only CLI, Claude Code and OpenAI Codex, over 28 investigation tasks. All three are scored by a blinded, tier-weighted, grounding-gated relative recall over one joint claim pool, so the scores share a denominator. The Sola Security Brain reaches 0.549 +/- 0.012 coverage against 0.340 +/- 0.011 for Claude Code and 0.281 +/- 0.006 for Codex, with the ordering identical in every grading draw. It leads 24 of 28 tasks from the cheapest model tier, at 17.8x and 20.8x lower cost per task than Claude Code and Codex. Beyond the aggregate, we describe an answer-level pattern we term sample-and-generalise: both agents enumerate a fraction of a large population, assert an unhedged universal negative, and disclose the sample size only in answer metadata rather than in the answer. In one task Claude Code reported that no bucket policies exist after checking four bucket families, in a sweep that sampled 40 of roughly 5,000 buckets, in an account where 65 buckets carry a wildcard-principal read grant.
1 Introduction
Cloud-security questions often require complete population-level answers, while coding agents reconstruct relevant structure from live APIs and security context layers query structures resolved in advance. The paper compares these approaches on 28 AWS investigations using pooled, tier-weighted coverage.
1 Introduction
- Population questions require complete inventories: an answer correct for only the examined subset is still wrong about the estate.The paper therefore evaluates coverage rather than correctness-style scoring, which could reward a correct answer about only a fraction of the population.Source: §1 · pp. 1–2
- The Sola Security Brain resolves a security representation offline, while coding agents reconstruct relevant structure from raw APIs when a question arrives.The approaches answer the same questions from different starting points.Source: §1 · pp. 1–2
- The open question is how a context layer performs on open-ended investigation and what it gains from its context, where existing benchmarks test exploring agents rather than context layers.Prior work establishes benefits from structured context in other settings, but does not examine assertions made when a population is only partly enumerated.Source: §1 · p. 2
- The study compared Sola Security Brain with Claude Code and OpenAI Codex on coverage across the same 28 AWS investigation tasks, measuring the share of findings each system surfaced.Source: §1 · p. 2
- Pooled relative recall scores each system’s tier-weighted, validated share of the joint set of claims made by all three systems.Claims are pooled, blinded, weighted by security consequence, and validated before they count toward coverage.Source: §1 · p. 2
2 Related Work
Prior agentic-security benchmarks assess agents exploring environments, while related work documents coverage and budget failures without examining population-level security assertions. This paper situates its contribution in the gap between partial evidence and universal claims about live resource inventories.
2 Related Work
- Agentic-security benchmarks test investigation across settings but generally use reference labels, not pooled finding sets, and do not compare agents against a context layer.ExCyTIn-Bench [4] is a structural analogue, but its graph is not given to the investigating agent.Source: §2 · p. 2
- Agents resolve 25.2% of scenarios [15], while doubling a defensive-investigation spend cap moves defensive score by less than a point and raises tool calls by a tenth [16].Source: §2 · p. 3
- Documented coverage failures include near-zero exhaustive collection by more than ten systems [7] and premature stopping or unawareness of stopping conditions [8].Other studies report that raising tool-call caps alone does not raise accuracy [20], while more searching can worsen abstention on unanswerable queries [21].Source: §2 · p. 3
- Related work describes incomplete review and undisclosed gaps [24], over-closure on universal queries [25], and false absence claims by tool-using agents [26].The paper’s security-specific distinction is that an exposure reported absent may not be revisited, and sample size may appear only in metadata.Source: §2 · p. 3
- Existing work does not examine agents asserting universal negatives about a partly enumerated population in a live security inventory, or where they disclose the sample size.The paper contrasts such exposures with missed code-review defects, which may be caught on a later pass.Source: §2 · p. 3
3 Experimental Setup
The evaluation compares an offline security representation with two live-CLI coding agents on a real enterprise AWS estate and analyst-authored tasks. The setup spans the security brain, agent architectures, environment, and population-oriented investigation themes.
3.1 The enterprise security brain
- The security brain builds a typed property graph from normalized, versioned records, connecting assets through validated relations and attaching findings, semantic attributes, and anomaly scores.Executable control predicates, attack traversals, and blast-radius patterns are evaluated against underlying data at inference time rather than stored as materialized results.Source: §3.1 · pp. 3–4
- The security brain resolves the substrate before questions arrive, so an agent need not rediscover schemas, reconcile identities, infer subsumption, or invent reachability traversals for each query.An agent querying the layer still plans, queries, follows findings, and refines its investigation.Source: §3.1 · p. 4
3.2 The coding-agent architecture
- Claude Code and Codex receive read-only credentials and CLI access, but no schema or inventory; each reconstructs task-relevant structure within a finite turn and time budget.The agents use the same prompt and receive only the task text.Source: §3.2 · p. 4
- The coding agents read current state directly and can derive relations from available APIs, but their budgets limit how many members of a large population they inspect.They select a subset and decide where to spend their calls.Source: §3.2 · p. 4
3.3 Environment and task set
- The three systems answer the same tasks in a live, multi-account AWS estate with no production customer data; practitioners reviewed and fixed the task set before runs.Tasks are short analyst-style prompts without schema, answer-shape, or completeness hints.Source: §3.3 · p. 4
- The 28 tasks span risk triage, secrets and keys, access and reachability, blast radius, control failures, and exposure and cross-account analysis.These themes require population coverage, effective-entitlement resolution, graph traversal, or relations spanning APIs and accounts.Source: Table 1 · p. 5, §3.3 · p. 5
4 Evaluation Protocol
The evaluation scores final answers using blinded, tier-weighted, grounding-gated pooled relative recall, with efficiency measured by duration and cost. Its claims are relative to the joint pool, not an enumerable gold set of all findings in the environment.
4 Evaluation Protocol
- Coverage is pooled relative recall [9] over claims extracted from the three systems’ final answers, with the arms blinded through randomized labels and order.Claims are weighted by security consequence, and shared claims can appear in multiple arms’ claim sets.Source: §4 · p. 6
- Claims receive tier weights for security consequence, while disagreements about analysis are neutralized; the score reflects security value surfaced, not claim count.The judge rates claims blind to the producing arm, using four consequence tiers from L0 to L3.Source: §4 · p. 6
- A claim earns credit only when contained in the answer, with its weight scaled by grounded-object faithfulness and containment.Grounding checks use text scrubbed of arm fingerprints.Source: §4 · p. 6
- The measure is not recall against the environment: findings omitted by all systems are invisible when no enumerable gold set exists.The model-driven scoring chain is run three times under independent seeds, and the arm ordering was identical in every draw.Source: §4 · p. 6
- Duration is end-to-end wall-clock time, while cost prices question-time inference tokens; both are also normalized by suite-total coverage.The Sola arm is priced at the Sonnet 4.6 rate, and the command-line arms are priced at their respective model output-token rates.Source: §4 · p. 6
5 Results
Across 28 tasks, the Sola Security Brain leads the coding agents in mean coverage and in five of the six themes. It is slower per task in raw wall-clock time, but more efficient when normalized by coverage.
Coverage
- 0.549 mean coverage for Sola exceeds 0.340 for Claude Code and 0.281 for Codex, with the ordering unchanged across three grading draws.Sola leads 24 of 28 tasks; Claude Code leads 3, and one is tied.Source: Table 3 · p. 7, §5 · p. 7
Coverage by theme
- Sola leads all six themes; the coding agents are closer to each other on five, while Codex edges Claude Code on blast radius, 0.287 to 0.278.Source: Table 4 · p. 7, §5 · p. 7
Cost and coverage-normalised efficiency
- Codex averaged 314,311 input tokens per task and cost $1.0936 per task at metered rates, versus Claude Code’s $0.5849.Twelve Codex tasks crossed the long-context threshold; no metered Sola cost is reported because its input tokens were not instrumented.Source: §5 · p. 7
Per-task coverage
- Per-task coverage gaps against Claude Code range from +0.58 to −0.15, with Sola’s scores spanning 0.22–0.83 and Codex’s 0.08–0.54.Seven tasks have a gap of 0.40 or more; three gaps are negative.Source: §5 · p. 7, Table 5 · p. 8
Duration
- 194s per task makes Sola the slowest arm in raw wall-clock time, but it uses 353s per coverage unit, versus 506s for Claude Code and 541s for Codex.Sola finished all 28 tasks with no empty answers or format or validation retries.Source: §5 · p. 7, Table 6 · p. 8
6 Discussion
The widest margins cluster on population questions that require complete-set sweeps, while live agents reconstruct only fragments; sample-and-generalise illustrates the risk of presenting partial evidence as a population-wide negative. The evaluation’s scope is bounded by its system-derived claim pool and single-vendor setting, and some tasks favor agents where the layer lacks pre-indexed relationships.
6 Discussion
- Both coding agents reconstruct fragments of population-wide “which, how many, rank” questions from live APIs, while answers that disclose a sample inspect roughly 40 resources of a type.These tasks require complete-set enumeration, including combining grant mechanisms and resolving wildcard and administrative subsumption.Source: §6 · p. 8
- In q12, Claude Code sampled 40 of roughly 5,000 buckets and reported no bucket policies, although 65 buckets carried wildcard-principal read grants.Sola identified several grant families, including 189 managed read-only rows, 1,410 matched inline Lambda-role rows, and five external accounts with cross-account read.Source: §6 · p. 9
- Claude Code leads three tasks and ties one, all involving relationships the layer does not pre-index, making its advantage depend on what has been modelled.The authors describe this as a coverage boundary rather than a ceiling because missing edges can be materialised.Source: §6 · p. 9
- The evaluation covers one cloud vendor; the authors expect, but did not test, a wider gap on multi-vendor estates where entity reconciliation and cross-system relations matter.They frame the reported gap as a lower bound, not as a measured multi-vendor result.Source: §6 · p. 9
- The layer’s cost advantage reflects offline normalization and relation inference amortized across later queries, while live agents repeat structure-reconstruction costs per task.For Claude Code, the Sola arm used 2,448 output tokens per task versus 8,646, a 3.5× difference; the remainder of the 17.8× cost ratio follows from model tiers.Source: §6 · p. 10
- The study’s relative-recall denominator comes from the three systems’ claims, so findings all three miss remain invisible; an independent reference set is needed to close this limitation.The authors also recommend more coding agents and ablations, including a graph-disabled ablation to isolate the layer’s contribution.Source: §6 · p. 10
7 Conclusion
Across 28 cloud-security tasks, the enterprise security brain achieved higher coverage than either coding agent, while the agents’ population-task coverage remained bounded by what they could enumerate within budget. The study also identifies sample-and-generalise in both agents and calls for a wider evaluation basis.
7 Conclusion
- 0.549 ± 0.012 coverage for the security brain exceeded Claude Code’s 0.340 ± 0.011 and Codex’s 0.281 ± 0.006 across 28 tasks, with the ordering identical in every grading draw.It led 24 tasks, was slowest in raw wall-clock time, and cost less per task and per unit of coverage than both agents.Source: §7 · p. 10
- Both agents showed sample-and-generalise: partial enumeration followed by an unhedged universal negative, with sample size disclosed only in metadata; its frequency was not measured.The authors note that the undisclosed gap can matter when an unsurfaced finding is exploitable.Source: §7 · p. 10
- A wider evaluation would include more coding agents, one-at-a-time ablations, and a reference claim set drawn independently of the tested systems.The authors identify these additions as ways to strengthen the claim further.Source: §7 · p. 10