PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Agentic Configuration Repair for Computer Networks

Rufat Asadli, Benjamin Hoffman, Ioannis Protogeros, Laurent Vanbever

arXiv:2606.06212v1cs.AI

TL;DR

Network misconfigurations contribute to costly outages, and complex network tasks remain challenging for monolithic LLM prompting. This paper evaluates agentic repair with dynamic context retrieval, iterative editing, and verifier feedback. It reports average gains in repair efficacy and safety over monolithic approaches.

  • Problem

    Complex network tasks challenge monolithic LLM prompting, while misconfigurations remain a source of costly outages.

    Source: §1 · p. 1

  • Method

    The paper evaluates open- and closed-source LLMs with dynamic context retrieval, iterative editing, and formal verification as environment feedback.

    Source: §1 · pp. 1–2

  • Results

    12% average gain in restoring correct network behavior and 17% average reduction in safety regressions are reported over monolithic approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dynamic context management and continued verifier access positively affect safety, while their effect on efficacy is more nuanced.

    Source: §1 · p. 2

    Interactions among design choices introduce trade-offs that require further investigation.

    Source: §5 · p. 4

Abstract

from arXiv · show

Misconfigurations in computer networks remain a major source of critical Internet outages. Research is turning to Large Language Models (LLMs) to automate the complex, error-prone task of network configuration. However, even state-of-the-art models fail to resolve misconfigurations in large-scale, complex scenarios and often introduce new errors. In this work, we benchmark open- and closed-source LLMs augmented with formal network verification and context retrieval tools. We demonstrate that agentic architectures outperform base LLMs in repair efficacy (by 12% on average) and safety (by 17% on average), enabled by the ability to dynamically manage context and iteratively validate configuration repairs.

1. Introduction

Agentic configuration repair addresses the difficulty of resolving network misconfigurations amid complex infrastructure, noisy configurations, and the limits of single-shot prompting. The paper equips LLMs with context retrieval, iterative editing, and verifier feedback, and reports improved repair efficacy and safety over monolithic approaches.

1. Introduction

  • Misconfigurations cause costly outages, while growing infrastructure complexity and limits in modeling real-world behavior leave network correctness largely unrealized.The introduction also notes that erroneous LLM outputs can cause catastrophic outages, limiting adoption for network operations.Source: §1 · p. 1

Existing benchmarks

  • Monolithic prompting struggles with complex network tasks because noisy configurations obscure signals, single-shot attempts hinder correction, and edits’ forwarding effects are difficult to predict.The cited challenges concern configurations spanning thousands of lines and the semantic gap between edits and network-wide behavior.Source: §1 · p. 1

This work: Agentic repair for misconfigurations

  • The paper applies agentic repair to end-to-end configuration fixing, using dynamic context retrieval, iterative editing, and verifier access to examine network-wide effects without live deployment.It evaluates open- and closed-source LLMs on CORNETTO with these tools.Source: §1 · pp. 1–2

Key contributions

  • A 12% average gain in restoring correct network behavior and a 17% average reduction in safety regressions are reported for the agentic system over monolithic approaches.The authors also report that context management and continued verifier access positively affect safety, while their effect on efficacy is more nuanced.Source: §1 · p. 2

2. Problem Setting and Benchmark

The benchmark evaluates whether a system can repair broken network configurations so that they satisfy intended forwarding specifications. It covers large real-world topologies and measures repairs by simulating forwarding behavior and checking specifications, including whether fixes cause regressions.

Task definition

  • A task supplies a topology, broken configurations, and violated specifications; a repair must produce a configuration satisfying the specification set.The benchmark’s scenario also includes a golden configuration and a faulty version derived from it.Source: §2 · p. 2

Dataset

  • The benchmark contains 231 scenarios across real-world topologies, 27 fault types, and up to eight simultaneous faults; networks reach 754 nodes and 200K configuration lines.Faults touch less than 1% of configuration lines on average yet can disrupt up to 50% of specifications, making precise context management important.Source: §2 · p. 2

Evaluation method

  • Fixes are evaluated by simulating forwarding behavior and checking the ground-truth specifications; strict success requires a perfect fix score with zero regressions.The evaluation distinguishes restored specifications, unresolved violations, and previously healthy specifications broken by the fix.Source: §2 · p. 2

3. Agentic Setup

The agent is designed to retrieve relevant configuration context, make iterative edits, and use verification feedback during repair. It can inspect the network and working configuration on demand, with a fixed step budget and an option to submit its current work when that budget expires.

Agentic Setup

  • The design targets selective context retrieval, iterative editing with rollback, and verifier interaction to validate intermediate repairs.These design considerations are motivated by limitations of monolithic prompting.Source: §3 · pp. 2–3
  • The ReAct-style agent retrieves topology, violated specifications, and configuration files on demand, chooses tool actions based on the current state, and has a fixed step budget.If it does not submit a solution within the budget, its existing work is submitted.Source: §3 · p. 3

Dynamic context management

  • The agent can list routers, retrieve individual configuration files, and inspect topology, using violated specifications to guide context selection.The setup avoids feeding the full network-wide configuration to the agent.Source: §3 · p. 3

Iterative repair

  • Edits are applied to a working copy that the agent can inspect after each modification, enabling it to diagnose and repair issues iteratively.The approach contrasts with attempting all repairs in a single pass.Source: §3 · p. 3

Verification as feedback

  • The agent can invoke the verification pipeline during repair to see remaining specification conflicts, guide edits, or decide when to submit.The verification feedback uses the same three-fold specification decomposition as final evaluation.Source: §3 · p. 3

4. Preliminary Evaluation

The evaluation compares agentic repair with monolithic LLMs and probes how verifier feedback, context retrieval, and step budgets affect performance and behavior. Agents generally outperform monolithic models, but design effects vary by model, and longer interaction budgets carry cost and safety trade-offs.

Main results

  • Agentic repair consistently outperforms monolithic LLMs in fix score and regression rate, with open-source models improving up to 7× and regressions falling from as high as 34.2% to under 6%.The comparison covers four models on 231 scenarios; agents had 30 iterations per task, with verifier feedback disabled during repair for this baseline comparison.Source: §4 · p. 3

Design effects

  • Verifier feedback improves QWEN3.5-9B’s fix score and regression rate, while for GPT-5 MINI it reduces regressions but also lowers fix score.The ablation independently tests verifier feedback and context retrieval; Table 1 identifies these design choices.Source: §4 · p. 3, Table 1 · p. 4
  • With verifier feedback enabled, GPT-5 MINI reaches its best fix score of 49% with the full problem description prefilled, while QWEN3.5-9B scores 20% fix and 3% regression with targeted retrieval.The passages attribute GPT-5 MINI’s result to its long-context capability and note that smaller models can struggle to extract signals from large contexts.Source: §4 · pp. 3–4

Step budgets

  • More iterations generally improve fix scores, but QWEN3.5-9B’s regression rate rises with longer budgets, and step-budget analysis informs deployment costs.Multi-turn inference costs more than single-turn prompting, making the budget where fix scores saturate relevant to deployment.Source: §4 · p. 4, Figure 3 · p. 4

Agent trajectory

  • Across models, agents typically begin by gathering specifications and inspecting topology, then shift to patching and verification, and tend to exhaust their step budgets.The trajectory analysis aggregates tool-calling behavior across 30 steps.Source: §4 · p. 4, Figure 4 · p. 4

5. Limitations and Future Work

The paper identifies deployment-interface, safety-guardrail, and training-method questions for future work on agentic network repair.

LLM-network interfaces

  • Static analysis alone cannot diagnose transient and performance-related faults; richer interfaces such as live routing tables and packet traces could expand coverage.The stated boundary concerns faults beyond offline verification.Source: §5 · p. 4

Safety guardrails

  • More proactive invariant checks and human approval for high-risk changes could further improve the viability of real-world agentic repair.Source: §5 · p. 4

Fine-tuning

  • Step-level reinforcement-learning feedback is a possible training direction, with small open-source models also proposed as efficient, privacy-preserving alternatives to proprietary models.The cited step-level recipes assign feedback to intermediate steps rather than only final outcomes.Source: §5 · p. 4

Impact Statement

The impact statement frames the work as contributing toward safer LLM-agent adoption in automated network management.

Impact Statement

  • The paper aims to support safe adoption of LLM-based agents for automated network management.Source: Impact Statement · p. 5
  • Improved reliability of network operations is identified as a potential social outcome, relevant to Internet-reliant critical services.Source: Impact Statement · p. 5
  • The impact statement notes that network misconfigurations are currently a major cause of Internet outages.Source: Impact Statement · p. 5

A. Additional Results

The additional-results section lists supplementary diagnosis, trajectory, and cost analyses.

A. Additional Results

A.1. Diagnosis and Localization

Agentic diagnoses identify true faults at comparable recall but have lower precision and overall diagnosis scores than monolithic models. Agents spend most iterations inspecting files and use verifier feedback 2–3 times per task.

A.1. Diagnosis and Localization

  • Diagnosis scores and root-cause localization are evaluated separately, with localization measured as classification of misconfigured files.Three LLM judges assess diagnosis soundness and completeness against the ground-truth misconfigurations.Source: §A.1 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
  • Agents achieve comparable recall but lower precision than monolithic models, reducing their overall diagnosis scores.Soundness measures precision, while completeness measures recall; the authors associate lower precision with more false positives, potentially from hallucinations.Source: §A.1 · p. 7

A.2. Agent Trajectory

  • Agents spend most iterations inspecting files and use verifier feedback an average of 2–3 times per task.GEMMA 4 E4B uses roughly half its allocated step budget; the authors suggest further study of tool-use order.Source: §A.2 · pp. 7–8, Figure 7 · p. 8

A.3. Cost-Performance Trade-off

Agentic repair costs more per task, while GPT-5 MINI’s results suggest that 20 steps can capture much of the benefit at lower cost. Costs and outcomes vary with step budget and verifier feedback.

A.3. Cost-Performance Trade-off

  • Agentic inference costs are higher: GPT-5 MINI costs $0.091 per task versus $0.018 monolithic, while GLM-4.7 rises from $0.034 to $0.132.The authors identify choosing an effective step budget as important for controlling practical repair costs.Source: §A.3 · p. 8, Figure 8 · p. 8
  • At 20 steps, GPT-5 MINI reaches a 42.7% fix score and below 1% regression for about $0.050 per task; gains beyond 20 steps are marginal in this analysis.At 10 steps, the fix score is 28.5% versus 33.9% for monolithic, while regression falls from 13.3% to 2.0% at $0.023; the analysis covers one model and three budgets.Source: §A.3 · p. 8, Table 2 · p. 9
  • With verifier feedback disabled, the 30-step setup nearly doubles cost and reaches a 45.2% fix score, with regression rate decreasing by 3.1 percentage points.The retrieval-only variant consumes more tokens because information-rich context is added at each iteration.Source: §A.3 · p. 9

B. Experimental Details

The appendix documents experimental design choices and benchmark materials supporting the main results.

B. Experimental Details

  • The appendix reports CORNETTO test-set statistics and explains the selection of models spanning a capability spectrum.It also documents pipeline configurations for diagnosis, interaction-history management, and context sampling.Source: B. Experimental Details · p. 9
  • A comprehensive catalog of benchmark misconfigurations is included for completeness.The appendix presents these materials as support for the results in Section 4.Source: B. Experimental Details · p. 9
  • The experimental details cover both benchmark composition and agent pipeline configuration.The stated topics include test statistics, model selection, diagnosis, interaction history, and context sampling.Source: B. Experimental Details · p. 9

B.1. Dataset Statistics

The experimental appendix describes the benchmark, model selection, pipeline, fault catalog, and prompts and tools used for repair. The agent can retrieve context incrementally, edit configurations, and verify network behavior.

B.1. Dataset Statistics

  • The CORNETTO test set contains 231 scenarios across Topology Zoo topologies, 27 fault types, and up to 8 simultaneous faults per scenario.The scenarios are modeled on frequently observed real-world outages, and faults are described as minimal but highly disruptive.Source: §B.1 · p. 9

B.2. Model Selection

  • Four models span proprietary and open-source capability levels; agentic scaffolding improves fix scores and reduces regressions for all, with larger gains for lower-capacity open models.The selected models are GPT-5 MINI, GLM-4.7, QWEN3.5-9B, and GEMMA 4 E4B.Source: §B.2 · p. 9

B.3. Pipeline Configurations

  • Pipeline configurations use multi-judge diagnosis against ground truth, truncate tool observations older than five steps while retaining reasoning, and retrieve randomly sampled configurations on demand.Three LLM judges independently assess diagnosis; retrieval loads selected configurations incrementally rather than fitting all context into one prompt.Source: §B.3 · p. 10

B.4. Fault Types

  • The benchmark fault catalog draws on past incidents, RFCs, and troubleshooting manuals, and lists affected protocols, fault summaries, and expected network effects.The catalog is presented as a comprehensive listing of misconfigurations.Source: §B.4 · p. 11, Table 4 · p. 11

C.1. Main Agentic Prompts

  • The main agent prompt directs the model to inspect violated specifications, diagnose root causes, apply targeted patches, verify, and iterate or roll back as needed.The task is bounded by a maximum tool-call budget.Source: §C.1 · pp. 12–13

C.2. Tool Stack

  • The tool stack supports configuration inspection, violated-spec retrieval, patching, formal verification, rollback, and submission.Verification reports fixed and remaining specifications and any regressions.Source: §C.2 · p. 13

C.3. Prefilled Context Prompt

  • The prefilled-context prompt supplies network information, disables retrieval tools, and asks for explicit router modifications in YAML.The task is to restore correct forwarding behavior by diagnosing the issue and applying patches.Source: §C.3 · p. 14

Related papers

Loading 2606.06212v1…