On the Effectiveness of Kernel-Level Evidence for Agent Security

The problem

• Agent tool compromises can hide malicious actions behind legitimate-looking work, while defenses observing intent-layer signals may miss them or reject the wrong behaviors.The examples include credential exfiltration, a document-induced shell command, and a poisoned tool description.Source: §1 · p. 2

The evidence gap

• Prior systems instrumented kernel and application layers largely in isolation, and none released a paired-session corpus for comparing detectors on a common substrate.Whether kernel evidence discriminates agent-shaped attacks and whether Cross-View beats either single-layer view had not been systematically measured.Source: §1 · p. 2

Contributions

• ACE pairs kernel traces, served tools/list manifests, and agent transcripts across 4,047 sessions, 17 threat models, six delivery-vector families, and 14 OWASP categories.Three of the 14 categories have partial coverage.Source: §1 · p. 2
• Across four detector families, kernel evidence is discriminative alone and Cross-View generally improves over either single-layer view, with generalization to unseen attack families and transfer to another runtime.The layers are near-complementary: each falls below acceptable discrimination on some mechanics the other resolves.Source: §1 · p. 3

2 THE ACE CORPUS

• ACE pairs each session’s in-container syscall trace, served tools/list snapshot, and full agent transcript across 4,047 sessions and 17 threat models.The corpus spans six delivery-vector families and aligns to OWASP LLM and agentic threat taxonomies.Source: §2 · p. 3

2.1 THREAT MODEL

• The threat model covers package tampering, indirect injection against built-in tools, and user-direct malicious prompts, with a post-hoc auditor examining one completed session.The setting is single-agent, single-session retrospective detection.Source: §2.1 · p. 3

2.2 SESSION AS UNIT OF ANALYSIS

• A session is one threat-model, prompt, and variant execution in a fresh Docker container, and labels apply to the complete session rather than individual syscalls.Across 21 detector-view cells, AUROC was relatively stable when latent sessions entered the positive class and decreased more when dormant sessions were included.Source: §2.2 · p. 3

2.3 CAPTURE PIPELINE

• The capture pipeline runs each Claude CLI session in a fresh container with an in-container strace sensor attached to the tool process tree.Four artifacts persist per session, and sensor events average approximately 3,500 per session.Source: §2.3 · pp. 3–4

2.4 ATTRIBUTION IS THE DESIGN CONSTRAINT

• One strace per session container attributes events to that session: PID namespaces bound process descent, and separate logs prevent cross-session interleaving.Across 10 trials of three concurrent sessions, no recorded event carried another session’s payload tag.Source: §2.4 · p. 4
• Attribution does not guarantee containment, which is out of scope; the scope also excludes bare shared-host agents, pre-attached long-running daemons, and IDE- or desktop-embedded agents.Attribution assumes a functioning, untampered sensor; sensor-integrity monitoring against adversarial agents is future work.Source: §2.4 · p. 4

2.5 CORPUS SCOPE

• Across six delivery-vector families, ACE includes 17 threat models and 4,047 sessions; benign sessions match malicious sessions on median kernel features, with separation in the p90-and-above tails.The families include tampered MCP servers, file- and web-based injection, user-direct prompts, memory-based attacks, and retrieval or multi-step chains.Source: §2.5 · pp. 4–5

2.6 ATTACK-MECHANIC TAXONOMY

• ACE groups its 17 threat models into 12 sensor-observable mechanics and seven OWASP-aligned held-out folds, covering 14 of 25 threat categories, including three partially.Mechanics capture shared observable actions across delivery vectors; OWASP-coherent folds define the out-of-distribution evaluation.Source: §2.6 · p. 5

3.1 TASK FORMALIZATION

• The task scores captured sessions by AUROC against a malicious label: benign sessions lack a malicious patch, while malicious sessions require a matched fire predicate.The predicate can use kernel signatures, application-layer signatures, or a combination; structurally ambiguous sessions are excluded from the AUROC cohort.Source: §3.1 · p. 6

3.2 FEATURIZATION AND VIEWS

• The featurizer yields 86 structural features: 57 from syscall traces and 29 from application evidence, concatenated for Cross-View.Features use counts and cardinalities rather than mechanic-specific sentinel identities; LLM and SLM judges instead receive view-conditioned text renderings.Source: §3.2 · p. 6

3.3 EVALUATION REGIMES

• Evaluation uses stratified in-distribution folds and OWASP-aligned held-out-family folds, with the latter measuring generalization to attack shapes absent from training.Reported AUROC summaries average across folds with patch-clustered bootstrap standard errors.Source: §3.3 · pp. 6–7

3.5 RESULTS

• On Kernel-View alone, LoRA-fine-tuned Llama-3.1-8B reaches the strongest OOD AUROC overall, 0.922.Every non-linear detector is above chance on Kernel-View; all fine-tuned SLMs exceed 0.88 OOD AUROC.Source: §3.5 · p. 7
• Across four detector families, OOD AUROC exceeds 0.80 for the reported Cross-View and Kernel-View configurations, including Llama-3.1-8B Kernel-View at 0.922.The reported values include XGBoost Cross-View at 0.850, TabPFN Cross-View at 0.874, and Qwen3-80B K=1 Cross-View at 0.810.Source: §3.5 · pp. 7–8
• Cross-View achieves the highest AUROC for all 10 detectors in-distribution and seven of 10 out-of-distribution, while exceeding App-View for all 10 detectors.The benchmark compares App-, Kernel-, and Cross-Views across both regimes.Source: §3.5 · p. 7, Figure 2 · p. 8
• Under runtime transfer to ACE-XA, three of four detector families clear AUROC 0.75; TabPFN Cross-View is the exception at 0.662 ± 0.172.ACE-XA uses a different agent runtime and a 592-session companion corpus; TabPFN’s collapse traces to a path-count feature that differs between runtimes.Source: §3.5 · pp. 8–9
• Cross-View clears the α=0.75 signal threshold on all 12 mechanics and wins on 11, but the strongest single evidence layer varies by mechanic.Kernel resolves three mechanics where App falls below threshold; App resolves Retrieval reference injection, where Kernel falls below it.Source: §3.5 · p. 9, Table 2 · p. 9

4 CONCLUSION

• Every detector’s strongest Figure 2 configuration consumes kernel signal, while off-the-shelf server-workload rules are near-random on interactive-agent workloads.The authors recommend kernel evidence as a first-class detection input alongside application-layer signals.Source: §4 · p. 9

REPRODUCIBILITY STATEMENT

• The planned release includes ACE, source code, prompts, split definitions, trained detector weights, and result-regeneration scripts.The reproducibility statement also describes the corpus protocol, taxonomy, labeling policy, and evaluation procedures.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The corpus uses synthetic credentials, reserved TEST-NET addresses, loopback sinks, and validation to prevent patches from targeting real credentials or endpoints.The authors characterize release as dual-use and argue that defensive value is favored because the mechanisms are already publicly disclosed and paired benign/malicious data is lacking.Source: ETHICS STATEMENT · p. 10

USE OF LARGE LANGUAGE MODELS

• LLMs assisted only with grammar correction, proofreading, and minor stylistic refinements, not the research content.Source: USE OF LARGE LANGUAGE MODELS · p. 10

A.1 AGENT-ATTACK CATALOGS; A.2 INTENT-LAYER AGENT DEFENSES

• Prior agent-attack catalogs and intent-layer defenses largely operate at the protocol or tool-call layer; pre-deployment scanners also lack visibility into non-MCP delivery vectors.The related-work discussion contrasts those approaches with the paired runtime evidence examined here.Source: A RELATED WORK · p. 17, §A.1 · p. 17, §A.2 · p. 17

A.3 LEARNED MONITORING OF AGENT TRAJECTORIES; A.4 HOST INTRUSION-DETECTION CORPORA

• Trajectory monitors use tool-call evidence, while host-IDS research uses syscall or provenance traces; neither line provides the paired-session corpus needed for matched Cross-View evaluation.The discussed agent trajectory monitors consume schemas, invocations, and results, whereas traditional host-IDS corpora lack LLM-agent context and MCP sessions.Source: §A.3 · pp. 17–18, §A.4 · p. 18

B LIMITATIONS

• Pure tool-return manipulation, inter-call timing channels, whitespace steganography, and mid-session manifest changes fall outside the current sensor coverage.The three signature-free mechanics would require output-content classifiers or timing monitors; tools/list is captured only once at session start.Source: B LIMITATIONS · p. 18
• Attribution is session-level rather than per-tool-call, and sinkholed network traffic gives syscall fingerprints that differ from real-deployment captures.Per-tool-call attribution is out of scope, while the lab sinkhole makes external connections appear successful at the syscall layer.Source: B LIMITATIONS · p. 18
• Reproduction costs approximately $1,464–$8,545, while judge, exemplar-support, and alternate-runtime transfer coverage remain limited.Broader LLM-provider and runtime coverage remains untested; the alternate-runtime study covers four detector families on one runtime.Source: B LIMITATIONS · p. 19

C FUTURE WORK

• Future work proposes adding sensors for blind-spot mechanics, expanding ACE across attack and runtime types, and extending attribution beyond single-session containers.Proposed runtime targets include non-MCP plug-ins and multi-agent frameworks; proposed attribution alternatives include eBPF, late-attach ptrace, and IDE instrumentation.Source: C FUTURE WORK · p. 19

D BROADER IMPACT

• The paper argues the observation channels are production-deployment-ready and that the defensive value of ACE outweighs the marginal information it gives attackers.The planned corpus release uses synthetic credentials, TEST-NET addresses, and loopback-bound listeners to make fixtures air-gapped by construction.Source: D BROADER IMPACT · p. 19

E.1 THREAT MODEL DETAILS

• The threat model covers package tampering, indirect injection, and user-direct malicious prompts, with post-hoc detection of single-agent, single-session activity.Cross-session state, multi-agent coordination, inline blocking, and mid-session manifest mutation are outside scope.Source: §E.1 · p. 20

E FULL CORPUS COMPOSITION

• ACE maps 17 threat models onto 12 attack mechanics and seven OWASP-aligned folds; three built-in threat models with no malicious sessions remain training-only pure negatives.MCP tool-tampering fixtures contribute to every fold, while built-in threat models map to individual mechanics.Source: E FULL CORPUS COMPOSITION · p. 20, Table 3 · p. 21

E.3 SESSION LABELS AND FOLD DESIGN

• Across 4,047 sessions, 1,358 are malicious positives, 2,293 negatives, and 396 signal-bearing latent sessions are dropped; seven-fold testing holds out OWASP-aligned attack families.The 5-fold design measures in-distribution fit, while the 7-fold design measures generalization to attack shapes absent from training.Source: §E.3 · pp. 21–22

E.4 POSITIVE-CLASS ROBUSTNESS CHECK

• Adding latent sessions retained at least 95% of F AUROC across 21 detector-view cells (∆FL−F [−0.045, +0.010]), while adding dormant sessions caused larger decreases (∆FLD−F [−0.179, −0.081]).The dormant-inclusive AUROCs range from 0.496 to 0.787. Detectors were trained under F; only the scoring definition changed.Source: §E.4 · pp. 22–23, Table 5 · p. 23
• Restricting OOD scoring to 239 app-anchored malicious sessions preserved Kernel-View > App-View for every SLM and deep-tabular detector, with SLM gaps widening to +0.37–0.42 AUROC.On this cohort, classical detectors discriminate held-out attack families without literal IP addresses or pathnames: AdaBoost and XGBoost reach OOD Kernel-View AUROC 0.733 and 0.750, and OOD Cross-View AUROC 0.840 and 0.850, respectively.Source: §E.4 · p. 23

E.5 BENIGN CLASS LEGITIMACY CHECK

• Benign and malicious sessions have near-identical medians on six of seven kernel features, with class separation arising mainly in the p90-and-above tails.The remaining feature, openat, differs by 41 counts on a base of about 650 (about 6% relative).Source: §E.5 · p. 24, Table 6 · p. 24

E.6 OWASP COVERAGE DETAIL

• ACE represents 11 of 25 OWASP threat categories and partially covers three more, capturing only the OS-observable slice of those partial categories.Uncovered categories include threats targeting training pipelines, retrieval indexes, human interpreters, and identity infrastructure.Source: §E.6 · p. 24, Table 7 · p. 25

E.7 CAPTURE PIPELINE DETAIL

• ACE captures a 15-tracepoint strace stream alongside the served tool manifest, agent transcript, and user prompt, aligning artifacts by timestamp and session ID for downstream joins.The featurizer consumes 14 tracepoints; ptrace is captured for sensor-integrity monitoring.Source: §E.7 · pp. 25–26

E.8 SESSION-LEVEL ATTRIBUTION VALIDATION

• Across 30 concurrent-session runs, attribution precision was 100%, with zero strace lines carrying another session’s tag.The runs covered 10 trials of three concurrent sessions each; a host-side cgroup check independently confirmed the attribution result.Source: §E.8 · p. 27

E.9 SAFETY PERIMETER

• The corpus safety perimeter uses synthetic credentials, non-routable TEST-NET exfiltration targets, loopback-bound listeners, and static patch validation.Container isolation is a backstop, not the primary guarantee; the attacks reference only synthetic credentials and constrained network targets.Source: §E.9 · pp. 27–28

F DETECTOR INPUTS; G FEATURE LIST; G.1 KERNEL-LAYER FEATURES; G.2 APP-LAYER FEATURES

• Classical and deep-tabular detectors consume numeric session features, while LLM and SLM judges use view-conditioned text and Falco scores distinct fired rules from raw strace.The featurizer includes syscall counts, rates, pair ratios, and IP/path cardinalities; app-layer features include prompt structure such as length, word count, question marks, and imperative starts.Source: F DETECTOR INPUTS · p. 28, §G.1 · p. 29, §G.2 · p. 29

G.2 APP-LAYER FEATURES

• Application-layer features encode served-tool schema structure, tool-call and result trajectories, and novelty or excess system-style markers in tool results.Examples include schema counts and description-text patterns, call and error statistics, and result tokens absent from the prompt or served descriptions.Source: §G.2 · pp. 29–30

G.3 DESIGN CHOICES

• Network and filesystem features measure identity breadth rather than specific destinations or paths, while content-blind structural patterns make the feature set rename-invariant.The design avoids reading mechanic-specific sentinel strings by name, so renaming sentinels does not defeat the featurizer.Source: §G.3 · p. 30

H.1 FULL PER-CELL BENCHMARK

• The full benchmark reports AUROC and average precision across three evidence views, ten detectors, and in-distribution and held-out-family regimes.The tables report mean per-fold clustered bootstrap standard errors; Falco is reported on AUROC only.Source: §H.1 · p. 30, Table 8 · p. 31, Table 9 · p. 31

H.2 PER-MECHANIC SIGNAL CLASSIFICATION

• Per-mechanic signal labels use mean AUROC across ten non-Falco detectors, with a 0.75 threshold; every mechanic retains Cross signal across thresholds 0.70, 0.75, and 0.80.Lowering the threshold adds Kernel signal for Retrieval reference injection and Preference manipulation; raising it removes both single-layer views for Metadata prompt injection and Network redirect exfiltration.Source: §H.2 · p. 30, Table 10 · p. 32

H.3 SENSITIVITY TO CAPTURED USER PROMPTS

• Kernel-View and Cross-View retain higher AUROC than App-View for all six language models when evaluation uses sessions with captured initial user prompts.The restricted cohort contains 2,871 sessions before label and scoring exclusions; predictions and model inputs are held fixed.Source: §H.3 · p. 32, Table 11 · p. 32

H.4 PAIRED COMPARISONS BETWEEN EVIDENCE VIEWS

• Cross-View exceeds App-View for all ten detectors in paired OOD comparisons; against Kernel-View, intervals favor Cross-View for four detectors and Kernel-View for one.The other Kernel-View comparisons include zero, supporting kernel evidence while showing that combination benefits vary by model.Source: §H.4 · p. 33, Table 12 · p. 33

I DEEP TABULAR DETAILS

• TabPFN and TabICL are pretrained tabular foundation models evaluated in-context on ACE features, with train-from-scratch comparisons and configurations documented in this appendix.Their fold training rows serve as inference context, without gradient training on ACE.Source: I DEEP TABULAR DETAILS · p. 33

I.1 TABULAR DL ABLATION

• OOD Cross-View AUROC is 0.874 for TabPFN versus 0.850 for XGBoost, while MLP and FT-Transformer prefer App-View at 0.721 and 0.694.TabICL reaches OOD Cross-View 0.801; TabPFN exceeds MLP by 0.21 and FT-Transformer by 0.20.Source: §I.1 · p. 34, Table 13 · p. 34

I.2 MLP HYPERPARAMETER CONFIGURATION

• The selected MLP uses hidden layers (256, 128, 64, 32), dropout 0.4, and weight decay 10^-3, chosen by fold-0 Cross-View AUROC.All 16 configurations fell within a 0.011 AUROC band on the tuning fold.Source: §I.2 · p. 34, Table 14 · p. 35

I.3 FT-TRANSFORMER HYPERPARAMETER CONFIGURATION

• The selected FT-Transformer uses two blocks, block dimension 96, and dropout 0.1, chosen by fold-0 Cross-View AUROC.All 12 configurations fell within a 0.035 AUROC band on the tuning fold.Source: §I.3 · p. 34, Table 15 · p. 35

J LLM DETAILS

• The raw-stream ablation trains five deep-learning detectors on ACE’s seven OWASP-aligned OOD folds; LLM Table 8 uses one fixed prompt-and-shot configuration, expanded in Table 17.The sequence detectors use the same per-fold splits as the other detectors.Source: §I.4 · p. 35, J LLM DETAILS · p. 36

I.4 SEQUENCE-DL ABLATION ON THE RAW KERNEL STREAM

• 0.836 mean OOD AUROC for BERT on the R1 summary exceeds Qwen Kernel-View estimates (0.725–0.775) and nears the XGBoost Cross-View baseline (0.850).The four raw-stream detectors reach 0.517–0.578 mean OOD AUROC.Source: §I.4 · pp. 35–36, Table 16 · p. 36
• BERT-MIL (0.553), structured-event Set Transformer (0.578), and flat-token Set Transformer (0.576) fall within 0.025 AUROC, with no measurable lift from order preservation.The differences are within one bootstrap standard error.Source: §I.4 · p. 36
• The benign-only LSTM reaches 0.517 mean OOD AUROC, near chance, on ACE’s short, tool-diverse sessions.The passage contrasts these interactions with long-lived, single-process workloads targeted by prior syscall-IDS sequence work.Source: §I.4 · p. 36
• Across frozen LLM judges and trained deep sequence models, the ablations converge on session-level counts, cardinalities, and rates as a suitable representation for this workload.The interpretation reports that adding raw temporal ordering degrades rather than improves detection performance.Source: §I.4 · p. 36

• OWASP grounding improves every K=0 model × view × regime cell; OOD Cross-View gains are +0.226, +0.089, and +0.122 for Qwen3-235B, Qwen3-80B, and Qwen3-32B.The average lift across the three models is +14.6 percentage points.Source: §J.1 · p. 37
• With family-excluded examples, all three Qwen models improve on OOD Cross-View, reaching AUROC 0.723–0.829.The lifts over OWASP-grounded zero-shot are +0.082, +0.209, and +0.099 for Qwen3-235B, Qwen3-80B, and Qwen3-32B.Source: §J.1 · p. 37
• OWASP grounding plus family-excluded exemplars raises OOD Cross-View AUROC above either OWASP zero-shot or baseline few-shot for every Qwen model.The gains over the better of those configurations are +0.082 for Qwen3-235B, +0.163 for Qwen3-80B, and +0.099 for Qwen3-32B.Source: §J.1 · p. 37
• The OOD winning evidence view changes with prompt instructions and demonstrations: judges select App-View, Kernel-View, or Cross-View under different configurations.Under OWASP-K=1, Qwen3-235B and Qwen3-80B prefer Cross-View, while Qwen3-32B prefers Kernel-View.Source: §J.1 · pp. 37–38
• Under OWASP-K=1, frozen judges reach OOD Cross-View AUROC 0.829, 0.810, and 0.723 for Qwen3-235B, Qwen3-80B, and Qwen3-32B, without parameter updates.The evaluation uses references from attack families absent from the demonstrations.Source: §J.1 · p. 38

J.2 KERNEL-VIEW FEATURE REPRESENTATION ABLATION

• R1 is a bounded six-field trace summary; R2 deduplicates verbatim events, R3 tokenizes arguments while preserving event order, and R4 retains ordered verbatim events.The OWASP-grounded K=0 prompt is held fixed across the four representations.Source: §J.2 · p. 38
• R1 has the highest mean AUROC across Qwen judges: 0.669 versus 0.592 for R2, 0.585 for R3, and 0.583 for R4.R4 retains every event in order with verbatim arguments, yet trails R1 by 0.086 on average.Source: §J.2 · p. 38, Table 18 · p. 39
• R4 minus R3 is −0.002 mean AUROC overall, while identifier removal lowers Qwen3-235B by −0.072 and Qwen3-80B by −0.012.Qwen3-235B retains 0.671 of its 0.731 R1 AUROC under full identifier removal.Source: §J.2 · p. 39

J.3 LLM PROMPT TEMPLATES

• Both LLM prompt templates frame the judge as a security monitor; the OWASP-grounded version adds nine threat-description and observable-example blocks used across views.Examples include credential-path access, unexpected shell execution, and writes to side logs.Source: §J.3 · pp. 39–40
• App-View supplies the manifest and transcript, Kernel-View a compact syscall summary, and Cross-View both; judges’ verdict confidence is converted to a maliciousness score for AUROC.The conversion is c/100 for malicious and 1 − c/100 for benign verdicts.Source: §J.3 · pp. 39–40
• K=1 uses 14 labeled references; OOD references come from training rows excluding held-out families and test-session identities, while the shared scoring cohort retains only valid matched scores.OOD references are selected within each fold, and their headers give binary labels without family-name hints.Source: §J.3 · p. 40

J.4 DETECTION WITH EXAMPLES OF KNOWN ATTACK FAMILIES

• The comparison uses 14 fixed references covering families A–G, excludes reference sessions from scoring, and matches query sessions across reference protocols.The random-support evaluation also uses 14 references and a common cohort after excluding reference-session identities and common-score omissions.Source: §J.4 · pp. 40–41
• With OWASP grounding, Cross-View achieves the highest AUROC for all three models, reaching 0.811–0.842 without parameter updates.The reference-protocol comparison evaluates the settings as a whole because both family coverage and selected references differ.Source: §J.4 · p. 40, Table 19 · p. 41
• Across three seeded stratified-random support sets, Qwen3-Next-80B-A3B macro AUROC ranges from 0.837 to 0.845, compared with 0.842 for the canonical set.The random-draw mean is 0.841, and the worst observed reduction from the canonical macro AUROC is 0.0047.Source: §J.4 · p. 40, Table 20 · p. 41
• The robustness finding is limited to the fixed-size, category-balanced protocol; mechanics vary more than the macro average, and support sizes or category coverage are not compared.Source: §J.4 · p. 41

K SLM DETAILS

• The appendix documents SLM training and session-evidence rendering for ACE; five additional same-fold Llama adapters for ACE-XA follow the Appendix M.2 protocol.Source: K SLM DETAILS · p. 41

K.1 LORA TRAINING RECIPE

• LoRA adapters freeze base-model weights and train on verdict-only responses, with validation-AUROC checkpoint selection and early stopping.The recipe uses rank 64 and scaling 128; each adapter is trained under a single fixed seed.Source: §K.1 · p. 42

K.2 FEATURIZED TEXT RENDERING PER VIEW

• App-View renders the served manifest and agent trajectory, Kernel-View a compact syscall summary, and Cross-View concatenates both evidence blocks.When the 6,144-token cap binds, more dropped tokens come from App-View content because its rendering is typically longer.Source: §K.2 · p. 43

K.4 DISCUSSION

• 10 of 10 detectors’ OOD-winning views still consume kernel signal; three prefer Kernel-View, with the two SLM exceptions linked to adapter-capacity tradeoffs.Qwen2.5-7B is the only SLM to prefer Cross-View, at 0.920.Source: §K.4 · pp. 43–44

K.5 IDENTIFIER REDACTION IN KERNEL SUMMARIES

• 0.759 macro AUROC across seven held-out attack families follows identifier redaction, compared with 0.922 using the original kernel summaries.Redaction preserves counts and list structure while removing destination and pathname information as well as literal identities.Source: Table 21 · p. 44, §K.5 · p. 44

L COMPARISON WITH STANDARD DETECTION TOOLS

• The scanner under-fires and the runtime rule engine over-fires on this workload, so neither substitutes for a trained detector.The scanner has 15.9% recall at 16.3% FPR on the tampered-MCP subset; the rule engine has 100% recall and 100% FPR.Source: L COMPARISON WITH STANDARD DETECTION… · p. 44

L.1.1 SETUP

• Cisco mcp-scanner uses offline YARA rules on served tool descriptions and schemas; the comparison evaluates it on the tampered-MCP subset.The reported cohort contains 1,871 sessions after signal-bearing latent sessions are excluded.Source: §L.1.1 · pp. 44–45

L.1.2 PER-DETECTOR RECALL AND FPR

• At a 5% false-positive ceiling, XGBoost Cross-View reaches 0.936 recall, TabPFN Cross-View 0.923, and Llama-3.1-8B LoRA Kernel-View 0.993.The scanner’s native operating point is 15.9% recall at 16.3% FPR; Qwen reaches 0.732 recall at 7.3% FPR but has 0.007 recall under strict 1% and 5% ceilings.Source: Table 22 · p. 45, §L.1.2 · p. 45

L.1.3 CONFUSION OVERLAP ON 731 CONFIRMED-EFFECT MALICIOUS SESSIONS

• ACE catches 74.8–97.9% of the scanner’s missed confirmed-effect attacks, with complementary coverage highest for the SLM and lowest for Qwen.Across the four detector families, the scanner and ACE overlap on 75–115 of 731 attacks.Source: §L.1.3 · p. 46, Table 23 · p. 46, Table 24 · p. 46

L.1.4 WHY THE SCANNER CATCHES SO LITTLE

• YARA rules miss paraphrased injection wording, while 52 of 82 malicious tampered-MCP fixtures leave the served manifest byte-identical to baseline.Those logic-only patches include silent request logging, timing side channels, response-embedded injection, and credential-file reading during legitimate calls.Source: §L.1.4 · pp. 46–47

L.1.5 THE SCANNER HAS NO VISIBILITY INTO 51% OF ACE BY DESIGN

• Five of ACE’s six delivery-vector families have no MCP server, leaving 2,068 sessions with no manifest for a pre-deployment scanner to inspect.Runtime evidence is the only signal available for those sessions.Source: §L.1.5 · p. 47

L.1.6 DISCUSSION

• ACE detectors catch 74.8–97.9% of the scanner’s missed tampered-MCP attacks and cover the 51% of ACE sessions with no MCP server to scan.The scanner catches 15.9% of fired attacks in the tampered-MCP subset.Source: §L.1.6 · p. 47

L.2 RUNTIME RULE ENGINE (FALCO)

• Falco is presented as a representative runtime rule engine that scores kernel-side syscall evidence against a curated rule pack.The comparison quantifies its default rule pack under the paper’s fold designs.Source: §L.2 · p. 47

L.2.1 SETUP

• The ported Falco rules cover syscall-observable shell, sensitive-file, privileged-write, persistence, and outbound-network predicates.Rules requiring capture points absent from ACE, such as cloud APIs or Kubernetes audit events, are omitted.Source: §L.2.1 · p. 47

L.2.2 RESULT

• Falco’s rule-count score reaches Kernel-View OOD AUROC 0.572 ± 0.084, while its native threshold has 1.000 recall and 1.000 FPR.ACE-trained detectors reach Kernel-View OOD AUROC 0.733–0.922 under the same fold definitions.Source: §L.2.2 · p. 48

L.2.3 WHY DEFAULT RULES OVER-FIRE

• Falco’s server-workload assumptions trigger on baseline Claude CLI behavior, including shell spawns, credential-shaped file reads, and hidden temporary-file writes.Two rules have some positive lift on ACE’s network-egress signatures, but OOD fold AUROC ranges from 0.42 to 0.78.Source: §L.2.3 · p. 48

L.3 DISCUSSION

• The scanner and runtime rule engine fail in opposite ways, whereas trained detectors reach Kernel-View OOD AUROC 0.733–0.922 and Cross-View OOD AUROC 0.791–0.920.The learned scoring function makes the same syscall channel 16–35 AUROC points more discriminative than Falco’s default rule pack.Source: §L.3 · p. 48

M CROSS-AGENT GENERALIZATION ABLATION

• ACE-XA contains 592 labeled sessions captured with Devstral 2 123B in a Python ReAct scaffold, an alternate runtime with a different process shape from Claude Code CLI.The scaffold dispatches tool calls in-process and keeps intermediate results in memory rather than staging them under /tmp/.Source: M CROSS-AGENT GENERALIZATION ABLATION · p. 48, §M.1 · p. 49

M.1 ACE-XA CORPUS CONSTRUCTION

• ACE-XA covers user-direct, file-based, resource-and-audit-log, and tampered-MCP delivery vectors, using the paper’s strace configuration and evidence-based fire predicates.The corpus contains 367 malicious-fired positives and 210 negatives; 15 signal-bearing latent sessions are dropped from AUROC.Source: §M.1 · p. 49

M.2 EVALUATION PROTOCOL

• The transfer protocol conditions detectors on ACE sessions from a known attack-mechanic fold, then scores eligible ACE-XA sessions from that same fold plus shared benign sessions.This tests runtime transfer within known attack families, separately from the held-out-family evaluation; no ACE-XA sessions are used for training or selection.Source: §M.2 · pp. 49–50

M.3 PER-FOLD RESULTS

• On ACE-XA, three of four detector families retain mean AUROC above 0.75; XGBoost’s Cross-View score on fold A reaches 0.941.The table covers five OWASP folds and their unweighted mean.Source: Table 27 · p. 50, §M.3 · pp. 50–51

M.4 READING: WHY TABPFN UNDER-TRANSFERS

• TabPFN’s ACE-XA mean AUROC is 0.662, traced to a shifted /tmp path-count feature that is zero in 57% of sessions.On ACE alone, TabPFN reaches mean OOD AUROC 0.829 versus XGBoost’s 0.864; the ACE-XA comparison changes both corpus and conditioning regime.Source: §M.4 · p. 51, §M.3 · p. 51

N COMPUTE COST

• The compute-cost estimates cover ACE capture and ID/OOD evaluation, while excluding representation ablations and other exploratory runs.Rates used are point-in-time estimates and may change.Source: N COMPUTE COST · p. 51

N.1 SESSION CAPTURE

• ACE session capture used 16.7 serialized container-hours across 4,047 sessions, with approximately $164 in agent-side Bedrock spend.Capture was parallelized, so elapsed wall-clock time was shorter than the serialized total.Source: §N.1 · p. 51

N.2 LLM-JUDGE SWEEP

• The three-model LLM-judge sweep returned 211,386 responses, including 211,175 valid scores, at an estimated cost of approximately $288.The cost is a token-based estimate, not an AWS invoice.Source: §N.2 · p. 52, Table 28 · p. 52

N.3 SLM LORA TRAINING

• The 108 LoRA adapters required 368.6 GPU-hours; eight-way packing cost approximately $1,012, while fully serial accounting gives approximately $8,093.The eight-way packed figure reflects the operational configuration; the serial figure is an upper bound.Source: §N.3 · p. 52

N.4 CLASSICAL, DEEP-TABULAR, AND RULE-BASED DETECTORS

• Classical, deep-tabular, and rule-based detectors used under 5 CPU-hours across the full ID and OOD fold sweep.Their CPU-time is described as negligible compared with the LLM and SLM figures.Source: §N.4 · p. 52

N.5 AGGREGATE

• The estimated capture and ID/OOD evaluation total is approximately $1,464 with 8-in-parallel SLM training, rising to approximately $8,545 under fully serial accounting.All figures are approximate and use the stated point-in-time rates.Source: §N.5 · pp. 52–53

N.6 CROSS-AGENT AND EXEMPLAR-SELECTION STUDIES

• The cross-agent and exemplar-selection studies added 21,488 scored responses and an estimated $48.82 in Bedrock cost.The additional local Llama adapter runs totaled 2.96 GPU-hours and are excluded from the capture and ID/OOD evaluation costs.Source: §N.6 · p. 53