PaperlayerView exact revision on arXiv

Source-linked AI summary

NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs

Mohsen Seyedkazemi Ardebili

arXiv:2609.12582v1cs.CRcs.DC

TL;DR

Existing agent traces are mutable, non-replayable, and silent about removed secrets, despite evidence expectations in regulated settings. NovaFabric captures runs as sealed, replayable Run Capsules; evaluation reports tamper detection and fast provenance queries, while end-to-end planet-scale validation remains out of scope.

  • Problem

    Existing agent traces are mutable, lack a recipe for re-execution, and do not record whether captured secrets were removed, while regulated settings require evidence records.

    Source: §1 · p. 2

  • Method

    NovaFabric captures agent executions without modifying agent code into provider-neutral Run Capsules, then seals them with a holistic signature, trusted timestamp, append-only log, and redaction attestation.

    Source: §1 · p. 2

  • Results

    45.5 ms p99 over 10M edges (3.3× faster than a columnar baseline), with under 200 ms p99 over 100M edges, alongside model-offline replay and tamper detection across three mutation classes.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Run Capsules provide portable, queryable execution evidence that can be exported for third-party verification with stock tools.

    Source: §6 · p. 39

    The event pipeline remains unmeasured, so end-to-end correctness and performance at planet scale remain unvalidated and out of scope.

    Source: §4.11 · p. 35

Abstract

from arXiv · show

When an autonomous AI agent does something consequential, what can be proven about what it did? Agent-observability platforms capture traces, but a trace is mutable: alterable undetected, with no recipe for re-executing it, silent on whether captured secrets were removed. Regulation (EU AI Act, ISO 42001, NIST AI RMF) presumes records an independent party can check. We present NovaFabric, producing audit-grade execution evidence: provider-neutral, tamper-evident, replayable, shareable. It records an agent run, without modifying agent logic, into a portable Run Capsule (fifteen-entity schema), sealed with a holistic DSSE signature, RFC 3161 timestamp, Merkle log and redaction attestation. Sealed runs are re-executable under a four-mode replay protocol and exportable as an Evidence Bundle for third-party verification with stock tooling (specified, not evaluated). The contribution is integration, not new cryptography: OpenTelemetry, DSSE/in-toto and W3C PROV. We evaluate eight research questions at measured scope. Mocked replay serves every model response from the capsule (no live model call, 10/10) but is offline w.r.t. models, not the network; only 2/10 tool-using workloads completed; the gap is missing tool-response substitution. Tampering is rejected across three tested classes. Declared-stream completeness is 0.652 (95% CI +/-0.064, ten scenarios). A repaired rule pack redacts 14/14 credential types, preserving 9/9 decoys; diff localises 140/140 mutations. Blast-radius queries: 45.5ms p99 over 10M edges (3.3x faster than a columnar baseline), 167.9ms over 100M (1 client, n=30). A 314-machine, ten-region run finds capsule REST ingest lossless but capped at 61.6 req/s (p99 26.8s) by per-worker serialisation. Six defects found in NovaFabric and its evaluation corpus: four fixed, one withdrawn, one open. Verification is conditional on a stated trusted computing base.

1 Introduction

NovaFabric addresses the gap between mutable agent traces and evidence an independent party can check. It packages observable execution into a portable capsule that can be sealed, replayed, compared, queried, and exported, while keeping its claims within stated evaluation and scope boundaries.

1 Introduction

  • Existing agent traces are mutable, provide no recipe for re-execution, and do not record whether captured secrets were removed, leaving a gap to audit-grade evidence.The paper defines audit-grade evidence as tamper-evident, replayable, confidential through signed redaction attestation, and provider-neutral.Source: §1 · p. 2
  • NovaFabric captures an agent run without modifying agent code into a portable Run Capsule with a provider-neutral schema spanning fifteen entity types.The pipeline seals the capsule, supports four replay modes, enables structural comparison and provenance queries, and exports an Evidence Bundle for third-party verification with stock tools.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • NovaFabric integrates OpenTelemetry, DSSE, trusted timestamping, and W3C PROV rather than introducing new cryptographic primitives.The paper presents a holistic replayable execution capsule and cryptographic redaction attestation as mechanisms not provided by the surveyed prior art.Source: §1 · p. 3
  • The evidence covers observable events at the agent–system boundary, not model reasoning, and the system does not guarantee regulatory compliance.Agent-level results use a single machine; only capsule ingest, not the cluster-scale event pipeline, is measured at fleet scale.Source: §1 · p. 3
  • NovaFabric’s evaluation reports model-offline replay, tamper detection across three mutation classes, and provenance blast-radius queries at 45.5 ms p99 over 10M edges, 3.3× faster than a columnar baseline.The implementation is open source, while the evaluation corpus and measurement scripts are available on request pending a curated public reproduction package.Source: §1 · p. 4

2 Related Work

Related work contributes tracing, provenance, tamper-evident logging, reproducibility, and governance requirements, but the paper finds these capabilities remain fragmented. NovaFabric’s stated contribution is their integration into one execution-evidence substrate.

2 Related Work

  • The four bodies of work—agent observability, provenance, tamper-evident logging, and reproducibility and governance—each address part of the problem, but none combines the proposed audit-grade evidence capabilities.The scope is observable execution at the agent–system boundary; internal model reasoning is outside it.Source: §2 · p. 4

2.1 Observability and tracing for LLM agents

  • Agent tracing systems capture execution but generally leave records mutable; attestation designs address slices of integrity, while commercial and cloud tools lack forensic replay and cryptographic redaction attestation.The surveyed cloud services also lack a provider-neutral portable artifact; test-replay against a live model differs from forensic reconstruction from sealed evidence.Source: §2.1 · pp. 4–6

2.2 The “replay” terminology

  • NovaFabric uses “replay” for reconstruction from a sealed capsule, distinct from test-reexecution that feeds recorded inputs or saved state to a live model.The paper identifies mocked replay as deterministic because it substitutes stored model responses; it does not claim arbitrary live re-execution is reproducible.Source: §2.2 · p. 6

2.3 Provenance and lineage

  • NovaFabric maps its typed lineage edges to W3C PROV, adding agent-execution queries over sealed records that coarser pipeline-lineage models do not provide.The paper identifies provenance, blast-radius, and replay-chain queries as supported uses of this model.Source: §2.3 · p. 6

2.4 Tamper-evident logging and supply-chain attestation

  • NovaSeal composes established signing and logging primitives at agent-session granularity, rather than proposing a new signing scheme.Its design applies DSSE to a session capsule and adds a trusted timestamp and per-tenant Merkle log; the paper also describes witness cosigning for the log.Source: §2.4 · pp. 6–7

2.5 ML reproducibility and AI governance

  • Reproducibility and governance frameworks document runs or specify evidence requirements, but do not prescribe a machine-readable, tamper-evident format that third parties can verify.NovaFabric’s Evidence Bundle may support conformance activities, but compliance remains for qualified human review.Source: §2.5 · p. 7

2.6 Synthesis and gap

  • The synthesis frames NovaFabric’s contribution as integrating capture, sealing, replay, provenance, diffing, redaction attestation, and portable export—not as a new signing scheme or governance framework.Table 1 grades prior systems from published documentation, not deployment; it operationalizes a proposed definition rather than a neutral capability ranking. Concurrent work is described as complementary, and cluster-scale operation remains experimental and not validated at production scale.Source: §2.6 · pp. 7–8, Table 1 · p. 8

3 Architecture and Design

NovaFabric treats agent runs as evidence through a capture–seal–replay–diff–audit pipeline, recording observable activity at the agent–system boundary while excluding internal model reasoning. Its architecture is constrained by invariants for capture, storage, signing, and schema evolution, and its guarantees are conditional on a stated trusted computing base.

3 Architecture and Design

  • Audit-grade evidence is provider-neutral, tamper-evident, replayable, and confidential, but covers only observable agent–system boundary activity, not internal model reasoning.The operational pipeline is capture → seal → replay → diff → audit.Source: §3 · p. 8

3.1 Design goals and invariants

  • Verification is conditional on faithful capture, honest key custody, protocol-compliant timestamping and logging, retained objects, and independently obtained verifier keys and policies.A tenant signing-key holder can create a validly signed capsule asserting something false.Source: §3.1 · p. 9
  • Capsule bytes are authoritative and indexes are rebuildable, but distributed parent/child relations are not fully represented in capsules in this release.The missing relation is an identified unmet design goal; losing the capsule store is unrecoverable.Source: §3.1 · p. 9
  • Capture must not block the workload: it uses bounded local work, may shed events under an explicit loss policy, and exposes partial records through completeness metrics.Heavyweight processing is deferred from the compute node.Source: §3.1 · p. 9
  • Signing occurs at the collection boundary rather than on the compute node, while additive-only schema evolution preserves old-capsule readability and provider neutrality.Readers must tolerate unknown fields; removing or retyping fields requires a major schema version.Source: §3.1 · pp. 9–10
  • NovaFabric records externally scheduled runs; it is neither an orchestrator nor a training or inference platform, and an evidence bundle is not a compliance certificate.Source: §3.1 · p. 10

3.2 System overview

NovaFabric offers a self-contained tier and a cluster-scale tier that share capsule and bundle formats. The cluster pipeline keeps compute-node work light and moves heavyweight processing into service-plane layers.

3.2 System overview

  • The self-contained tier supports capture, validation, replay, diff, and lineage queries without a NovaFabric service-plane dependency, while replayed workloads may still use live tools or networks.The cluster-scale tier adds distributed ingestion and storage without changing the capsule format.Source: §3.2 · pp. 10–11
  • The self-contained operator loop uses six commands to record, validate, replay, compare, export, and verify runs.The CLI listing was verified against version 0.101.0.Source: §3.2 · p. 10
  • Capture can use a CLI wrapper, SDK decorator, OpenTelemetry ingestion, or transparent API/MCP proxy, with hooks for major model SDKs and common HTTP libraries.The first three mechanisms do not touch application source code.Source: §3.2 · p. 10
  • Redaction, signing, and graph construction are off the compute hot path; NovaSeal signing occurs at the node-collector boundary, not on the compute node.Source: Figure 2 · p. 11, §3.2 · p. 11
  • The five-layer cluster pipeline moves events from compute-node spool through node and cluster collectors to regional storage and global queries.The object store is authoritative; metadata and lineage indexes are derived views.Source: §3.2 · p. 11, Figure 2 · p. 11

3.3 The capsule data model

The Run Capsule is the durable, provider-neutral record for one process, built from a schema of fifteen entity types. Capsules also support distributed-run identity and evidence-graph relationships, though the shipped lineage representation has a parent/child linkage gap.

3.3 The capsule data model

  • The Run Capsule is the authoritative at-rest record for one process and instantiates fifteen provider-neutral entity types for reconstructing agent activity without a vendor SDK.Source: §3.3 · pp. 11–12
  • Capsule files include manifests, append-only event logs, environment locks, redaction attestations, replay policies, lineage edges, and asset references.A companion registry versions seven asset types using semantic versions and SHA-256 content hashes.Source: §3.3 · p. 12
  • Registry and lineage databases can be rebuilt from capsule directories for recorded relations, but distributed parent/child relations remain an exception.Source: §3.3 · p. 12
  • Distributed processes share a global_run_id, carry a capsule_role, and use parent_run_id to link workers to their parent.Source: §3.3 · p. 12
  • The graph model distinguishes contains, spawned, delegated_to, and replayed_from; delegated edges are excluded to sharpen blast-radius queries, but the shipped capsule schema uses a different seven-relation set.replayed_from is a NovaFabric-specific extension without a W3C PROV-DM equivalent.Source: §3.3 · p. 12

3.4 Event envelope and ingestion

The Event Envelope provides a language-independent wire contract for event identity, location, and trace correlation, with payload hashes for integrity checks. Ingestion separates local spooling from asynchronous collector work and accepts out-of-order or orphaned events under explicit reliability policies.

3.4 Event envelope and ingestion

  • Event Envelope v1 separates identity, location, and distributed-trace fields and includes a SHA-256 payload commitment; ULIDs order events within an emitter, not across emitters.The emitting node’s clock supplies the ULID timestamp, so identifier order does not establish cross-emitter causality.Source: §3.4 · pp. 12, 14
  • Envelopes map to OpenTelemetry LogRecords and CloudEvents messages, while a SHA-256 schema pin binds consumers to an exact schema revision.The round-trip mapping is designed to be lossless, but no conformance test is reported.Source: §3.4 · p. 13
  • Phase 1 writes and atomically commits an unsigned partial capsule to a local spool without contacting a daemon; Phase 2 asynchronously signs, uploads, indexes, and finalizes it.The phases separate agent-side from service-side work.Source: §3.4 · pp. 13–14
  • Delivery is at-least-once and events may arrive out of order, so consumers deduplicate by event_id and use ULID monotonic order within an emitter.Source: §3.4 · p. 14
  • Valid child capsules are accepted even when their parent has not arrived; after a bounded timeout, a synthetic parent placeholder is created and may later be replaced idempotently.A per-run fail-closed override is available for workloads that prefer the whole run to fail on the first failed child.Source: §3.4 · p. 14

3.5 Tamper-evident sealing and governance gates

NovaSeal composes established cryptographic primitives into a two-tier seal with timestamp, log, and redaction bindings, while governance gates and approval controls govern promotion and evidence use.

3.5 Tamper-evident sealing and governance gates

  • Ed25519 batch signatures protect ingestion events, while durable DSSE/ECDSA P-256 seals protect capsules and bundles with timestamp and Merkle-log bindings.The log proves deletion or reordering only for capsules that reached it and against retained checkpoints.Source: §3.5 · p. 14, Figure 4 · p. 15
  • NovaSeal adds crypto-agility, checkpoint witnesses, an append-only accumulator, portable identity, and delegated-authority chains to the trust chain.Verifier-pinned algorithm policy is needed to avoid downgrade risk when multiple algorithms are accepted.Source: §3.5 · pp. 14–15
  • Canonical serialization and RFC 8785 digests target reproducible verification across implementations, but no second implementation or shared conformance corpus yet exists.Signing occurs off the compute hot path, with tenant-specific keys scoping capsule authenticity to its owner.Source: §3.5 · pp. 15–16
  • Redaction attestations bind a confidential record of detected-and-removed secrets to the modified artifact, but cannot establish that no secret remains.Completeness depends on scanner, rule-pack version, and declared scope; uncovered locations or formats can leave a validly signed bundle containing a secret.Source: §3.5 · pp. 15–16
  • Promotion approval links proposal and approval DSSE envelopes by digest and requires distinct key fingerprints, but does not establish distinct human approvers.Identity binding and principal-level separation are delegated to deployment key-management and authorization configuration.Source: §3.5 · p. 16
  • OPA/Rego policy gates record decisions for promotion, mutating-tool replay, export, and dataset licensing; promotion follows draft-to-staging-to-production.The server tier uses OIDC and role-based access control, while storage adapters support immutable object-lock modes.Source: §3.5 · p. 16

3.6 Storage and derived indexes

NovaFabric stores signed capsule bytes as the source of truth and derives rebuildable metadata and lineage indexes from them.

3.6 Storage and derived indexes

  • Raw signed capsule bytes reside in a content-addressed S3-compatible object store, with an append-only manifest chain recording commits and supporting write-once-read-many storage.Downstream data is derived from these capsule bytes.Source: §3.6 · p. 16
  • A relational metadata index supports run, asset, and pointer lookups; tenant-and-time partitioning and mandatory row-level security isolate queries.The database is a derived index that can be rebuilt from the object store; the self-contained tier uses SQLite.Source: §3.6 · p. 16
  • Lineage queries use derived graph indexes, with recursive SQL for self-contained operation and graph backends for cluster-scale use.Indexes rebuild from capsule lineage edges, but those edges do not include every modeled relation; parent/child is not represented. Backend parity is an implementation property, not an evaluated claim.Source: §3.6 · pp. 16–17

3.7 The replay protocol

NovaFabric offers four replay modes with distinct execution and guarantee contracts, and records replay drift and lineage in each resulting capsule. Preconditions determine which mode is admissible; mocked replay is offline with respect to models, not the network.

3.7 The replay protocol

  • Exact replay uses captured model weights, image, data, and environment; it yields identical output bytes only when the original run was deterministic.It refuses unmet preconditions, including unresolved asset hashes or unavailable pinned images, and always requires explicit opt-in.Source: §3.7 · pp. 17, 19
  • Mocked replay reruns application logic with stored model responses and no model calls, but tool calls still reach the live environment.It is deterministic by construction only for model inputs when application logic reads no uncaptured entropy or environment; the mode is offline with respect to models, not the network.Source: §3.7 · pp. 17–18
  • Semantic replay compares outputs against captured ones by meaning using a configured judge model, and is nondeterministic with possible network cost.An optional budget can bound that cost.Source: §3.7 · p. 18
  • Forensic replay executes nothing and is the only mode guaranteed to make no network call or tool invocation.It reconstructs the captured timeline, inputs, outputs, and lineage for inspection.Source: §3.7 · p. 18, Figure 5 · p. 18
  • A six-rung mutation ladder substitutes dry-run stubs for tools above the permitted mutation class unless explicitly and verifiably allowed.A separate dry-run flag reports intended execution without executing; replay divergences are recorded in the new capsule’s replay_drift block.Source: §3.7 · p. 18
  • Replay-mode guarantees are scoped: only mocked is deterministic by construction, exact depends on original determinism, and semantic is nondeterministic by design.The selection tree makes forensic always reachable and exact least often; every branch except forensic still issues tool calls against the live environment.Source: Figure 5 · p. 18, §3.7 · p. 19

3.8 Evidence graph and provenance queries

Typed lineage edges support cross-run provenance queries and distinguish observed evidence from inferred structure. Graph-based diagnostics are investigation aids, not diagnoses or factuality judgments.

3.8 Evidence graph and provenance queries

  • The evidence graph supports provenance, blast-radius, replay-chain, and time-travel queries across executions.Blast-radius traversal follows causal edges and excludes delegated_to edges to avoid reporting independent delegated subtasks as affected.Source: §3.8 · p. 19
  • Lineage edges label provenance as observed or inferred, and the graph offers partial W3C PROV alignment and OpenLineage export.The design targets interactive, self-contained blast-radius queries within 500 ms at production graph sizes.Source: §3.8 · p. 19
  • Causal back-trace returns unverified candidate root causes, while claim grounding flags claims lacking preceding tool-span evidence as a structural proxy for hallucination risk.Neither heuristic establishes a diagnosis or factuality judgment.Source: §3.8 · p. 19

3.9 The Evidence Bundle: offline-verifiable audit packages

The Evidence Bundle packages a Run Capsule, lineage evidence, schemas, hashes, and attestations into a self-contained archive. Its offline verification procedure uses stock tools without NovaFabric, but independent implementation of that procedure has not been tested.

Bundle contents and integrity

  • A Merkle root over capsule files in canonical order binds the capsule, and three DSSE-wrapped in-toto attestations cover the run, redaction attestation, and lineage subgraph.The bundle also contains a byte-equal capsule copy, relevant lineage subgraph, referenced JSON schemas, and a hash-indexing manifest.Source: §3.9 · p. 19

Offline verification

  • A verifier can check hashes, signatures, and capsule schemas with stock tools and no NovaFabric installation, using schemas vendored in the bundle.This is a specification-level interoperability claim; an independently implemented verifier has not exercised it.Source: §3.9 · p. 19

Purpose and scope

  • The bundle is intended as an independently checkable evidence package for auditors or researchers.Its verifiability is designed to persist across NovaFabric versions, provided stock tooling for the underlying formats remains available.Source: §3.9 · pp. 19–20

3.10 Beyond the core pipeline: compliance export, enterprise integration, and self-diagnosis

Beyond the evaluated capture-to-audit pipeline, NovaFabric includes compliance-export, enterprise-integration, and self-diagnosis layers. These are reported as design and implementation contributions, not evaluated results.

Evaluation scope

  • The compliance, enterprise-integration, and self-diagnosis layers are not subject to evaluation claims in the paper.They are presented as design and implementation contributions around the evaluated pipeline.Source: §3.10 · p. 20

Compliance export

  • Compliance exporters render bundles against named regulatory and standards frameworks, marking each field as capsule-verified, operator-asserted, or unverifiable rather than adjudicating compliance.The evaluated-scope exporters include EU AI Act provisions, ISO/IEC 42001 and 42005, and a NIST AI RMF/Cloud Security Alliance profile.Source: §3.10 · p. 20

Enterprise integration

  • Enterprise integration provides API keys, SAML 2.0 SSO, SIEM egress, opt-in alert routing, and a thin TypeScript SDK.SSO is disabled by default and subject to Security-Architect review before production enablement; alerting is a no-op without configuration.Source: §3.10 · p. 20

Self-diagnosis

  • Self-diagnosis uses structural graph search to identify unverified root-failure candidates and flags model-span claims lacking preceding tool-span evidence.Both mechanisms are model-free diagnostic aids, not proofs of causation or truth.Source: §3.10 · p. 21

3.11 Deployment tiers and implementation status

NovaFabric reports its components using maturity labels and distinguishes the shipped inventory from the subset measured in Section 4. The implementation is experimental overall, with no planet-scale performance claim.

Deployment tiers and implementation status

  • No NovaFabric capability is labeled stable pending the v1.0 schema freeze, and the paper makes no planet-scale or billion-agent performance claim.Agent-level results are from one machine; graph results use a cloud host, and ingest results use a 314-VM, ten-region fleet.Source: §3.11 · p. 21
  • Table 3 defines experimental as working with interfaces or formats subject to change, prototype as not validated at target scale, and planned as not yet implemented.Maturity labels are not evaluation results.Source: Table 3 · p. 22
  • Only capture, sealing and tamper detection, replay, diff, redaction, lineage/graph backends, and capsule ingest carry measured results in Section 4.Compliance exporters, enterprise integration, and self-diagnosis are listed as implemented but unevaluated contributions.Source: Table 3 · p. 22, §3.11 · p. 22
  • The Run Capsule and Evidence Bundle are durable formats built over adopted standards, while the Event Envelope is an inter-component wire contract.The paper presents its novelty as integration rather than invention of competing protocols.Source: Table 4 · p. 23

4 Evaluation

NovaFabric’s evaluation measures capture, integrity, replay, lineage, regression localisation, and redaction on stored artifacts under fixed seeds, using a single CPU-only host with a cloud operating point for lineage. It reports tested-corpus redaction performance and fleet-scale capsule REST ingest alongside limits involving incomplete capture, replayed tool calls, and test scope.

4.1 Experimental setup

  • Ten agent scenarios exercised model, HTTP, command-line, and MCP calls; the implementation is open source, but the capsule corpus and benchmark scripts are available from the author on request.Agent runs used a locally served open-weight model, and reported measurements were computed from stored artifacts under fixed seeds.Source: §4.1 · pp. 24–25, §4 · p. 24

4.3 RQ1: Model-response substitution and replay completion

  • 10/10 mocked replays made no live model calls, but only 2/10 workloads completed because stored tool responses were not substituted and replayed tools reached the live network.Mocked replay is offline with respect to models, not the network; the other eight runs blocked on external I/O and timed out within 300 seconds.Source: §4.3 · pp. 26–27

4.4 RQ2: Tamper detection

  • 100% of tested mutations across three classes were detected: field modification, seal removal, and trusted-timestamp alteration.Verification rejected each tested mutation; the result makes no claim about untested classes.Source: §4.4 · p. 27

4.5 RQ3: Capture fidelity, cost, and a defect the evaluation surfaced

  • 0.652 ± 0.064 was mean per-scenario capture completeness, while model-call and MCP-call capture each reached 100% against declared expectations.The completeness interval is 95% over ten scenarios; it mixes missing capture surfaces with dropped events and is not a claim that 34.8% of events were lost. Mean capsule footprint was 76 KB; overhead median was −1.8% and mean +3.2% (95% CI [−10.5%, +16.9%]), too variable on the CPU-only host for a tight bound. Fixes restored network-event capture from 0/10 to 10/10 scenarios.Source: §4.5 · pp. 27–28

4.6 RQ4: Distributed parent/child lineage—unit-verified, not yet query-verified end to end

  • The attempted 20-worker lineage query failed; the capsule path lacks a parent/child relation, so end-to-end linkage is not demonstrated there.The linkage algorithm is unit-verified, but parent identity is absent from the child capsule manifest and lineage stream; end-to-end verification through the envelope path remains future work.Source: §4.6 · pp. 28–29

4.7 RQ5: Provenance query latency at scale

  • At 10M edges, depth-5 blast-radius p99 was 45.5 ms versus DuckDB’s 149.2 ms; at 100M edges, single-client p99 was 167.9 ms.At 100M edges, 32 concurrent clients raised p99 to 2.08 s while throughput rose from 7.8 to 16.7 qps. The sizes use different hardware, so the measurements do not establish a scaling law.Source: §4.7 · p. 29, Table 6 · p. 29, Figure 6 · p. 30

4.8 RQ6: Structural-diff regression localisation

  • 140/140 single-mutation capsule pairs were localised with zero spurious reports, yielding recall and precision of 1.00 across seven tested mutation classes.The Wilson 95% CI was [0.973, 1.000] for each; the synthetic, stratified corpus supports repeatability under its generator, not field-level recall.Source: §4.8 · p. 30

4.9 RQ7: Redaction recall—a second defect the evaluation surfaced

  • The repaired redaction corpus yielded 14/14 credential types removed and 9/9 decoys preserved; an earlier recall result was withdrawn because its scenarios did not test supported patterns.The original planted token matched none of the scanner’s fourteen credential patterns. Before repair, three broad rules also removed the SHA, MD5, and UUID decoys; the repaired pack removed none of the 90 decoy instances across ten capsules.Source: §4.9 · pp. 30–31
  • At rule-pack v0.5.0, redaction recall and precision were both 1.00 across ten runs, with zero decoys destroyed and the legacy UUID blind spot identified in 1/1 trials.The rules match prefixed vendor keys because legacy bare-UUID keys are structurally indistinguishable from run identifiers; over 5,000 generated keys and digests per rule, none was missed or falsely flagged. The synthetic corpus and small, author-designed decoy set do not establish field performance.Source: §4.9 · pp. 31–32, §4.11 · p. 36

4.10 RQ8: Evidence-plane ingest at fleet scale

  • Across a 9× offered-load span, accepted throughput moved only 1.036× (60.3–62.5 req/s), reaching p99 26,762 ms.The route missed the 200 ms sealing-step reference by 134×, although that reference is not an end-to-end objective for the REST route.Source: §4.10 · p. 33, Table 8 · p. 34, Figure 7 · p. 35
  • 8 workers / 130 ms predicts 61.5 req/s versus 61.6 measured; Little’s Law and a worker-count A/B support per-worker serialisation rather than host saturation.Increasing workers from 8 to 32 cut p50 from 1,167.6 ms to 672.4 ms, a 1.74× improvement for a 4× increase; the 130 ms service time remains the term to reduce.Source: §4.10 · p. 33
  • 41,309 measured uploads had distinct run IDs, no errors or duplicates, and a 400-capsule sample retained nested members in all 400 copies.Across the full campaign, 41,774 reported HTTP 201 responses matched 41,774 stored capsule directories; the 465-request difference came from a non-fleet smoke-test client.Source: §4.10 · pp. 33–34

4.11 Threats to validity and what we do not claim

  • The evaluation uses one model family and tier-specific hardware, leaves the event pipeline unmeasured, and tests redaction synthetically and structural diffs with one mutation per pair.The fleet result covers one server class and payload, and its metadata database had no tables, so metadata writes to a correctly migrated store remain untested; agent accuracy is out of scope.Source: §4.11 · pp. 35–36
  • Capture completeness remains 0.652 ± 0.064, and self-reported capture metrics do not independently establish correspondence with actual events.On the T1 host, the capture-overhead confidence interval includes zero, which indicates limited power to resolve the difference rather than proof of zero cost; a separate deployment matched provider call ledgers across thirteen time buckets, not for this workload.Source: §4.11 · pp. 34–36
  • Only 2/10 replayed workloads completed: replay substitutes model responses, not tool responses, and may contact reachable hosts; tamper tests cover three stated classes, not adaptive attacks.The tool-call counter overclaim was fixed, but forensic mode still overstates model calls mocked; the latter defect remains open.Source: §4.11 · pp. 36–37
  • Capsule import cannot demonstrate parent/child linkage end to end because the capsule schema has no parent/child relation, although the linkage algorithm is unit-verified.The event-envelope path represents the relation as SPAWNED_BY; reconciling the vocabularies and measuring linkage through that path remain future work.Source: §4.11 · p. 37

5 Discussion

NovaFabric’s guarantee covers observable boundary activity and detectable post-capture changes, not model reasoning or trust in the evidence producer. The discussion draws lessons from evaluation defects, proposes an unvalidated maturity model, and bounds the results by scale, performance, and workload scope.

5.1 What the evidence guarantee does and does not mean

  • The guarantee covers observable model, tool, file, network, and approval effects; it detects later capsule changes but does not capture reasoning or prevent a signing-key holder from producing a valid capsule.Verification therefore relies on a trusted computing base that includes capture, signing, timestamp and log mechanisms, and verifier key and policy.Source: §5.1 · pp. 37–38

5.2 A lesson from three of the evaluation-surfaced defects

  • Three evaluation defects showed that declared-ground-truth checks can reveal silent under-capture, invalid test corpora, and false success reports that “did it run?” misses.The authors argue that declared schemas and re-derivable ground truth make omissions measurable rather than trusting headline results or status codes.Source: §5.2 · p. 38

5.3 A maturity model for execution evidence

  • NOVA-EL proposes four execution-evidence levels, from structured capsules at L1 to transparency-logged, offline-verifiable bundles at L4, but has not been validated or adopted.NovaFabric reaches L1–L3 in the self-contained tier and implements L4 mechanisms experimentally.Source: §5.3 · p. 38

5.4 Limitations

  • Agent-tier results use one machine and SQLite, while fleet-scale testing covers only capsule ingest; the event pipeline remains unvalidated at production scale.Evaluation also covers one model family and workload pattern; capture-fidelity figures depend on workload, and sealing overhead is not reported.Source: §5.4 · pp. 38–39

6 Conclusion and Future Work

NovaFabric packages observable agent-run evidence into sealed, replayable capsules that can be checked with stock tools, subject to a stated trusted computing base. The conclusion reports tested integrity and replay properties alongside bounded evaluation results, and identifies scale, completeness, enterprise signing, and standards work that remains.

Conclusion

  • NovaFabric combines a provider-neutral Run Capsule with signatures, trusted timestamps, a Merkle log, redaction attestation, replay, provenance queries, and an Evidence Bundle for third-party verification.The authors state that verification remains conditional on the trusted computing base, including trust in the key custodian.Source: §6 · p. 39
  • Mocked replay served every model response without a live model call observed, but only 2/10 tool-using workloads completed; tamper detection covered three tested mutation classes.The conclusion also reports unit-verified distributed lineage linkage, unit recall and precision for structural-diff localisation, and single-client provenance queries under 200 ms p99 at 100M edges on a cloud host.Source: §6 · p. 39

Future Work

  • Future work includes removing per-worker serialisation, measuring the event pipeline, testing federated provenance, and broadening completeness evaluation across all fifteen entity types.The capsule ingest plane was lossless at 1,256-way offered concurrency but capped at workers/130 ms; mean capture completeness was 0.652 ± 0.064.Source: §6 · p. 39
  • Enterprise signing paths remain below production maturity, while standards work includes contributing agent entities to OpenTelemetry GenAI and refining the proposed NOVA-EL model.The paper says the compliance exporters render recorded facts but do not adjudicate compliance.Source: §6 · p. 40
Loading 2609.12582v1…