Source-linked AI summary
Can AI Agents Detect and Repair Artifact Drift in Network Experiments?
Tianzhu Zhang, Weichen Tao, Changgang Zheng, Yusheng Zheng, Long Chen, Xiaoyi Fan, Meikang Qiu
TL;DR
Network-agent evaluations can overlook whether modified experiment records remain supported, properly scoped, and traceable. The paper introduces NetArtifactBench, a deterministic benchmark of source-grounded repair tasks, and finds strong overall performance but a sharp difficulty boundary for implicit, cross-artifact repairs. It concludes that artifact integrity should be treated as a first-class requirement for network-system agents.
Problem
AI agents may complete immediate network tasks while leaving experiment records inconsistent with or unsupported by available evidence, undermining records reused in later work.
Method
NetArtifactBench evaluates agents on 52 source-grounded repair instances using frozen public packages, controlled violations, and deterministic contract scoring.
Results
Overall, 3,904 of 5,980 outputs pass the contract (65.3%), while agents struggle with implicit relations and coordinated changes across artifacts.
Takeaways & Limitations
Artifact integrity requires preserving substantive evidence support and the explicit record structure needed for later inspection and reuse.
Takeaways & Limitations
The controlled violations measure repair capability rather than the prevalence or natural distribution of integrity failures in real network workflows.
Abstract
from arXiv · showhide
In recent years, AI agents have evolved into capable assistants that carry out multi-step tasks in digital environments. The network systems community is beginning to explore these capabilities in operational and experimental settings. However, an agent operating in network systems should not be judged solely by whether it completes the immediate task. The experiment record it modifies must also remain trustworthy. We call this property artifact integrity: the record's claims must remain supported by the available evidence, confined to the scope established by that evidence, and traceable through the artifacts that encode their support. To make this property measurable, we introduce NetArtifactBench, which tests whether AI agents can repair inconsistent records derived from public network-system artifacts while preserving claims that remain supported. The benchmark contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations spread across several artifacts. We evaluate 23 agent configurations across three general-purpose AI agent runtimes using deterministic scoring. The average contract pass rate is 65.3 % across 5,980 outputs, but no agent runtime exceeds 30 % when repair requires recovering implicit relations and propagating changes across artifacts. These results reveal a sharp boundary between local correction and complete record-level repair. Therefore, we argue that artifact integrity should become a first-class design and evaluation requirement for AI agents operating on network systems.
1 Introduction
AI agents increasingly modify network experiment records, but outcome-centered evaluation can miss whether those records remain evidence-grounded, scope-bounded, and traceable. NetArtifactBench makes this artifact-integrity problem measurable through source-grounded repair instances requiring consistent propagation and preservation.
- Network agents increasingly interpret heterogeneous evidence, act through tools, and revise records describing network experiments.
- Current evaluations often ask whether an agent achieves the requested result without checking whether modified artifacts remain faithful to the evidence.
- Artifact integrity requires claims to remain supported by evidence, bounded by its scope, and traceable through supporting artifacts.
- Drift can propagate from one unsupported claim into later hypotheses, configurations, or measurements when records guide subsequent work.
- NetArtifactBench contains 52 source-grounded instances and tests diagnosis, cross-artifact correction, preservation of supported claims, and traceable records.
2 How Artifact Repairs Fail
Agent repairs can fail even when they appear locally plausible: stale claims, unsupported replacements, damaged support, broken traceability, or invalid output structure may remain. These failures show that repair must restore record-wide relations while preserving auditability and parseability.
- A repair may correct one artifact while leaving the same unsupported assertion elsewhere in the experiment record.
- Replacing an unsupported conclusion with its opposite remains invalid when the available evidence does not establish the underlying relation.
- Removing a broad claim can also delete valid information needed to preserve the tested scope and interpret the remaining observation.
- A substantively plausible correction still fails when its claimed support points to a nonexistent source location.
- A readable explanation is insufficient if the returned package is not parseable or does not identify repaired claims, labels, and supporting evidence.
- These failure modes may coexist, so valid repair must restore substantive relations and keep them explicit for later auditing and reuse.
3 Claim–Evidence–Scope Model
The paper models an experiment package as a graph linking claims, evidence, scope, provenance, and phases. A claim is integrity-preserving only when public artifacts establish a valid support path whose conditions contain the claim’s effective scope.
- An experiment package represents claims, evidence, scope constraints, and provenance or phase relations as a connected graph.
- Support may be direct or relational, requiring joins, metadata bindings, or state transitions across experiment phases.
- Matching identifiers alone do not justify cross-trace joins, object co-occurrence does not establish bindings, and phase observations must respect order and provenance.
- A claim satisfies artifact integrity only when a valid public support path exists whose conditions contain the claim’s effective scope.
- Explicit limitations can narrow a claim but cannot provide positive evidence or silently repair broader prose.
- Valid repairs propagate corrections, preserve supported claims, and avoid fabricating missing evidence or relations.
4 NetArtifactBench Design
NetArtifactBench turns the claim–evidence–scope model into a deterministic repair benchmark using frozen public packages with injected, source-grounded violations. Its contract scores diagnosis, repair, preservation, labels, parseability, and traceability across increasingly relational repair tiers.
- NetArtifactBench Design: Each task gives an agent a frozen public package and contract, requiring relation diagnosis, cross-artifact correction, preservation, and parseable, traceable outputs.
- Instance Construction: The benchmark contains 52 instances derived from Batfish, Zeek, P4C, and Open vSwitch repositories, with violations grounded in project semantics.
- Instance Construction: The structural tiers progress from direct correction through explicit relational repair and coordinated propagation to inferred relations.
- Machine-Auditable Scoring: The scorer checks structured findings, repair completeness, preservation of supported claims, residual violations, labels, parseability, and evidence references.
- Machine-Auditable Scoring: An output passes only when every violation is identified, every repair is completed, protected claims are preserved, and required artifacts are parseable and traceable.
5 Evaluation
The evaluation measures artifact-integrity repair across agent configurations, runtimes, and increasingly complex structural tiers. Overall performance is moderate, but reliability falls sharply when repairs require coordinated or inferred relations across artifacts.
- Experimental setup: 23 agent configurations were evaluated across 52 instances, producing 5,980 outputs under a deterministic contract.The configurations used Codex CLI, Cursor Agent, and OpenCode, with five executions per instance.
- RQ1: Overall Effectiveness: 65.3% of outputs passed the contract overall, with pass rates of 74.2% for Codex, 71.2% for Cursor, and 44.0% for OpenCode.These are aggregated across the evaluated instances and configurations.
- RQ2: Structural Complexity: Contract pass rates decline from Tier 1 to Tier 5: 98.9% to 26.3% for Codex, 93.8% to 28.4% for Cursor, and 60.8% to 17.0% for OpenCode.The harder tiers contain coordinated or inferred cross-artifact relations.
- RQ2: Structural Complexity: Across runtimes, pass rates decline as repairs progress from direct contradictions to following, propagating, or inferring relations across artifacts.This pattern defines a boundary between locally exposed corrections and more complex record-level repair.
- RQ3: Reliability Limits: At Tier 5, mean label accuracies exceed contract pass rates for every runtime, while contract rates are 26.3% for Codex, 28.4% for Cursor, and 17.0% for OpenCode.The corresponding Tier 5 label accuracies are 96.2%, 93.4%, and 60.2%, showing that partial completion can coexist with contract failure.
- RQ3: Reliability Limits: Among 2,076 output-level failures, 68.5% leave at least one required diagnosis or repair incomplete, and failure categories may overlap.Because one output can violate several obligations, the reported percentages do not sum to 100%.
6 Related Work
Related work applies LLMs and AI agents to network configuration, diagnosis, operations, measurement, and reproducible workflows. These evaluations generally center on immediate configuration, incident, workflow, claim, reproduction, or execution outcomes rather than experiment-record integrity.
- Network-system agents: NetConfEval and VPP evaluate configuration generation and correction, while NetAssistant, LLexus, OSS-GPT, Confucius, and ArachNet support network workflows.These systems cover diagnosis, operations, measurement, and multi-step network tasks.
- Network benchmarks: Cornetto, NIKA, and NetAgentBench make configuration repair, incident replay, and multi-turn configuration testable through dedicated benchmarks.Their evaluation units remain immediate configuration, incident, or workflow outcomes.
- Adjacent domains: SWE-bench, FEVER, Artisan, and ArtifactCopilot evaluate issue resolution, evidence-based claim classification, result reproduction, and artifact execution, respectively.NetArtifactBench instead evaluates whether the surrounding experiment record remains coherent, evidence-grounded, and supportable.
7 Conclusion
NetArtifactBench makes record-level repair measurable and shows that agents handle direct contradictions better than repairs requiring implicit relations or coordinated artifact changes. The controlled violations assess repair capability, not the natural prevalence of integrity failures.
- Implications: Artifact integrity requires preserving both substantive evidentiary support and the explicit structure needed for later inspection and reuse.The record must remain interpretable, verifiable, and reusable as agents modify it.
- Benchmark and findings: NetArtifactBench evaluates whether agents can identify broken claim–evidence relations, propagate corrections across artifacts, and preserve supported claims.The benchmark uses 52 source-grounded instances.
- Benchmark and findings: Strong general-purpose agents handle direct contradictions but struggle with implicit relations and coordinated changes across artifacts.Local correctness therefore does not guarantee a complete, auditable experiment record.
- Scope: The benchmark's controlled violations measure repair capability rather than the prevalence or natural distribution of integrity failures in real network workflows.This bounds what the evaluation can establish about deployment conditions.