PaperlayerView exact revision on arXiv

Source-linked AI summary

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

Chang Liu, Xiaohui Xie, Xinyi Chen, Yong Cui

arXiv:2608.23179v1cs.NIcs.AI

TL;DR

LLM-agent reliability for network configuration remains poorly understood, motivating realistic, risk-free evaluation beyond static command generation. NetConfArena provides an emulator-backed closed-loop benchmark with executable behavioral tests, and evaluation across 480 task instances and 3840 trajectories shows failures involving specification adherence and robust execution, not only command errors.

  • Problem

    LLM agents may disrupt production networks, but existing evaluations provide limited evidence about their reliability in realistic, iterative network configuration.

  • Method

    NetConfArena combines emulated multi-device network interaction, a compact standardized action interface, hidden executable behavioral tests, and an emulation-grounded pipeline for reusable parameterized tasks.

  • Results

    Across 480 seeded task instances and 3840 execution trajectories, agents solved many tasks but exhibited failures involving task-specification adherence, ineffective planning and execution, and responses to unexpected feedback.

  • Takeaways & Limitations

    Evaluating both final network behavior and execution reliability exposes challenges that command-generation benchmarks miss and supports more reliable agent design.

  • Takeaways & Limitations

    The benchmark uses controlled scenarios on small topologies, omitting production complexity such as rollback procedures, multi-vendor environments, and network-performance dynamics.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

I. INTRODUCTION

NetConfArena addresses the limits of static or simplified network-configuration benchmarks with an executable, closed-loop evaluation environment. It combines realistic multi-device tasks, hidden behavioral tests, trajectory analysis, and large-scale evaluation to expose failures beyond command generation.

  • Motivation: Network configuration correctness depends on resulting network behavior, not merely whether a single device accepts commands.Realizing high-level intent commonly requires coordinated updates across devices and protocols.
  • Motivation: Existing benchmarks often test static outputs or use emulators only for post hoc validation, underrepresenting interactive execution and diagnosis.These approaches primarily measure configuration knowledge or single-shot generation.
  • Benchmark Design: NetConfArena places agents in vendor-specific, multi-device, protocol-rich emulated networks with a compact action interface and hidden executable behavioral tests.The benchmark also records complete interaction trajectories for process-level analysis.
  • Benchmark Design: Its LLM-assisted, emulation-grounded pipeline converts human-oriented network materials into validated, reusable parameterized task templates.Controlled variations can change addresses, interfaces, and protocol parameters without changing task intent or test logic.
  • Evaluation: NetConfArena evaluates representative agents on 480 seeded instances derived from 96 templates and collects 3840 execution trajectories.The evaluation identifies failures involving protocol or command semantics, specification adherence, ineffective planning, and abandonment after unexpected feedback.
  • Illustrative Example: The routing example shows why successful configuration requires progressive changes, intermediate validation, and feedback-driven repair across interdependent protocol elements.The process must account for ACL matching, RIP offset-list placement, metric adjustment, and resulting route selection and reachability.

B. Task Formulation

NetConfArena models configuration as interaction between an agent and an executable network environment, then evaluates both final behavior and execution process. Its formulation responds to benchmark gaps involving closed-loop interaction, realistic protocol-aware scenarios, and integrated outcome-process assessment.

  • B. Task Formulation: A task τ is formulated as a closed-loop decision-making problem between an LLM agent and an executable network environment.The formulation defines a task using intent, environment, available actions, and hidden executable tests.
  • B. Task Formulation: The operator intent I may contain sub-intents such as reachability, routing-path selection, and route-propagation control.The environment specifies topology, initial configurations, vendor-specific syntax, and protocol semantics.
  • B. Task Formulation: The action space supports information gathering, configuration, validation, and task submission, with observations returned after each step.The agent acts under environment constraints rather than a single static-output interface.
  • B. Task Formulation: An execution trajectory records observations, agent rationales or plans, and issued actions until submission or an interaction-limit termination.This representation supports analysis of how the agent reaches its final state.
  • B. Task Formulation: Evaluation uses the final environment state to run intent-specific hidden tests and also considers process-level trajectory signals.Agents decide from observations rather than a global state representation.
  • C. Existing Benchmarks Are Insufficient: A holistic benchmark should provide closed-loop interaction, realistic protocol-aware multi-device scenarios, and both outcome and process evaluation.These requirements are labeled R1, R2, and R3.
  • C. Existing Benchmarks Are Insufficient: Existing benchmarks address different aspects but do not jointly test protocol-aware, multi-device configuration under evolving network states.The stated gap concerns integrated realistic closed-loop evaluation rather than any single benchmark capability.

III. NETCONFARENA FRAMEWORK

NetConfArena evaluates agents in high-fidelity emulated networks using vendor-specific CLIs, hidden deterministic tests, and a compact action space. Its architecture separates task setup, agent execution, and agent evaluation across Benchmark, LLM Agent, Environment, and Evaluator components.

  • Design choices: NetConfArena runs agent execution and evaluation in high-fidelity emulated networks with multi-device, vendor-specific CLI configuration.This exposes command syntax and cross-device protocol dependencies required by realistic configuration tasks.
  • Design choices: Hidden deterministic test cases determine each run’s outcome, enabling objective and reproducible scoring.The evaluator also analyzes execution trajectories for process-level metrics.
  • Design choices: The agent action space exposes five abstract actions while remaining neutral to agent architecture.The experiments instantiate the LLM Agent component with a standard ReAct-style agent, but the benchmark does not require that architecture.
  • Architecture: The framework comprises Benchmark, LLM Agent, Environment, and Evaluator components interacting across task setup, agent execution, and agent evaluation.The Benchmark defines parameterized task artifacts; the Environment instantiates and updates the emulated network; the Evaluator computes outcome- and process-level metrics.
  • Task setup: Each task instance gives agents an operator intent and topology while reserving the reference configuration and test cases for evaluation.Controlled variation of addresses, interfaces, devices, and protocol parameters preserves task intent and test-case logic across instances.

C. Interactive Agent Execution

Agents interact with isolated emulated networks through a compact closed-loop interface, repeatedly observing device feedback and network state before submitting configurations. NetConfArena evaluates both behavioral correctness and configuration-text similarity, while recording trajectories for process analysis.

  • Interactive execution: GNS3 creates isolated task-specific topologies and emulates vendor router syntax, device feedback, and protocol dynamics.This supports realistic configuration workflows while keeping execution in an emulated environment.
  • Interactive execution: The interface provides five abstract actions: get_running_config, apply_config, execute_validation, wait, and submit.Every action except submit returns an observation used to ground the next decision.
  • Interactive execution: Agents can inspect configurations, apply device commands, validate behavior, wait for convergence, and submit results through sequential interactions.Configuration and validation actions return device or environment feedback, and coordinated operations can target multiple devices in one turn.
  • Interactive execution: A maximum interaction-turn limit bounds execution and prevents indefinite runs.The agent repeatedly observes network state and chooses the next action until termination or the limit is reached.
  • Trajectory logging: Each trajectory records task context, reasoning, actions, and observations, exposing invalid commands, insufficient validation, and premature submission.These records support failure diagnosis and agent-improvement methods.
  • Outcome-level evaluation: The evaluator combines a primary test-case score with configuration F1 as a contrast metric.Test-case score measures the fraction of task-specific behavioral tests passed, whereas configuration F1 compares actual and reference configuration changes.

2) Process-level evaluation:

Process-level evaluation characterizes how agents use the closed-loop interface, measuring efficiency, action quality, completion behavior, and execution overhead. The benchmark’s task-construction pipeline supplies validated, reusable tasks for these evaluations.

  • Process metrics: Trajectory metrics measure turn counts and the proportions of retrieval, update, validation, waiting, submission, and null actions.These metrics describe how interaction effort is distributed across action types.
  • Process metrics: Action-quality metrics count rejected configuration or validation actions and consecutive repetitions of identical actions.Rejected actions often reflect unfamiliarity with vendor-specific CLI syntax, while repetition indicates failure to convert feedback into progress.
  • Process metrics: Completion mode classifies trajectories as early submission, interaction-limit hit, correct submission, or error submission.The categories distinguish whether agents attempted configuration, reached the turn limit, and passed the task.
  • Process metrics: Task overhead records the token cost and elapsed wall-clock time required for each execution.This captures resource consumption alongside interaction behavior and task outcome.
  • Task construction: Benchmark task construction requires natural-language intent, concrete topology, valid initial configurations, and executable test cases.Manual authoring is labor-intensive because interface names, address plans, and routing processes must be matched carefully.
  • Task construction: An LLM-assisted, emulation-based pipeline normalizes source materials, validates specifications and tests, and parameterizes reusable task families.The pipeline screens scenarios for topology, intent, and reference configuration before specification and test-case generation.
  • Task construction: Generated specifications are reviewed by humans and revised until reviewers identify no further issues.Review focuses on intent clarity, sufficient initial configuration, and alignment between test cases and configuration intent.

C. Task Validation and Refinement

NetConfArena validates candidate tasks in running emulated networks before converting them into constrained parameterized templates. This process checks topology, initial states, reachable behavior, and test predicates, while parameterization varies selected fields without changing the underlying objective.

  • Task validation: Automated scripts deploy candidate tasks in the emulator to verify topology and initial-configuration consistency.They apply the reference configuration and run task-specific tests to confirm that intended behavior is reachable.
  • Task validation: Emulation-based checks test whether device outputs are parsed correctly and predicates evaluate resulting protocol behavior as intended.Grounding test cases in real device behavior reduces the risk of preserving protocol misconceptions in generated evaluations.
  • Parameterization: Validated tasks become parameterized templates that preserve topology, task objective, and test-case logic while declaring fields allowed to vary.Variable fields include names, link media, subnet allocations, prefix lengths, and protocol attributes.
  • Instantiation: Template instantiation resolves parameters in dependency order using a value registry, device capabilities, and a seed before rendering executable artifacts.The algorithm raises an unresolved-dependency error when no remaining parameter can be resolved.
  • Instantiation: Physical interfaces are resolved against emulator capabilities so generated configurations remain compatible with device models and link-media choices.Logical parameters and physical interfaces are resolved before template artifacts are rendered.
  • Scope boundary: Parameterization reduces direct reuse of memorized snippets but does not itself create fundamentally different operational scenarios.Benchmark diversity instead comes from variation across templates, topologies, protocol features, and task objectives.
  • Benchmark output: The resulting benchmark contains 96 task templates spanning representative IP routing, MPLS, and IP Overlay scenarios.The taxonomy records protocol categories, subcategories, representative features, template counts, and evaluation targets.

V. EXPERIMENTS AND FINDINGS

NetConfArena evaluates LLM agents across standardized model and thinking-mode settings on executable network-configuration tasks. Results show strong but imperfect, setting-dependent performance, with smaller models struggling substantially and thinking helping inconsistently.

  • Experimental setup: The experiments evaluate eight model–thinking-mode settings using a common ReAct-style workflow and 480 shared task instances, producing 3840 trajectories.The study covers qwen3 and deepseek-v4 models with thinking disabled or enabled, deterministic decoding, and a 20-turn interaction limit.
  • Metrics: Test-case score measures final network-state satisfaction, while configuration F1 provides a complementary text-level metric and task pass rate counts fully successful instances.Metrics are averaged equally over the 480 task instances.
  • Outcome-level findings: 0.961 is the strongest test-case score, achieved by deepseek-v4-pro with thinking disabled alongside a 0.852 task pass rate and 0.796 configuration F1.deepseek-v4-flash with thinking enabled also performs strongly, reaching a 0.933 test-case score and a 0.794 task pass rate.
  • Outcome-level findings: qwen3-8B remains substantially weaker, with test-case scores of 0.337 and 0.457 and task pass rates of 0.046 and 0.094 without and with thinking, respectively.The gap between test-case score and pass rate indicates that partial credit can mask incomplete task execution.
  • Thinking-mode effects: Thinking changes performance by model: it improves qwen3-8B and qwen3-32B by 12.0 and 7.2 percentage points, slightly improves deepseek-v4-flash by 1.9 points, and reduces deepseek-v4-pro by 9.9 points.The degradation for deepseek-v4-pro is associated with increased early submission.
  • Overall finding: State-of-the-art agents complete many tasks through interaction with the executable environment, but performance remains imperfect and setting-dependent, especially for smaller models.The experiments therefore show capability without uniform reliability across evaluated settings.

2) Process-level analysis:

Process-level analysis shows that model choice and thinking mode shape how agents allocate actions, handle errors, submit tasks, and incur overhead. Stronger agents validate more, while thinking can improve weaker models but also increase malformed actions, interaction cost, and premature submission.

  • Action Distribution: Stronger DeepSeek settings spend approximately 52–59% of actions on validation, whereas qwen3-8B allocates a larger share to configuration updates.The distribution excludes get_running_config and submit actions; Figure 4 compares thinking-disabled and thinking-enabled bars for each model.
  • Action Quality: Invalid or unrecognized turns consume interaction budget without changing or inspecting network state, and thinking raises their share across all models.For qwen3-32B, null or unrecognized action turns rise from 0.5% to 13.0% with thinking enabled.
  • Action Quality: Weaker settings have higher error-action and repeat-action rates, while deepseek-v4-pro with thinking disabled has a 2.0% error-action rate and almost no repeated actions.For qwen3-8B, enabling thinking reduces error and repetition patterns, suggesting additional reasoning helps produce more executable actions.
  • Completion Mode: Weaker settings more often exhaust the 20-turn budget or submit incorrectly, with error-submission rates of 12.1–46.0% for qwen3-8B versus 3.3–14.8% for DeepSeek.deepseek-v4-pro is an exception: thinking increases early submission from 0% to 7.9%.
  • Task Overhead: Thinking substantially increases assistant-output token overhead for qwen3 models while improving test-case score, and its time overhead is especially pronounced for qwen3 compared with DeepSeek.Time comparisons use trajectories that successfully complete the task, while DeepSeek shows a more moderate increase in time cost.
  • Synthesis: Overall, thinking improves qwen3 models and slightly improves deepseek-v4-flash but hurts deepseek-v4-pro by increasing early submission and trades fewer repeated or low-quality actions for higher overhead and malformed-action risk.These effects show that thinking mode changes both outcome and process behavior rather than providing a uniform benefit.

C. Failure Taxonomy

NetConfArena’s failure taxonomy extends beyond protocol knowledge to include unproductive validation, premature abandonment, and deviation from task-provided specifications. These categories expose distinct planning and execution weaknesses and motivate both trajectory-based learning and harness-level improvements.

  • Taxonomy construction: The analysis combines aggregate metrics with trajectory inspection, configuration diffs, and evaluator feedback to identify four representative failure categories.This combination makes failures observable at both behavioral and outcome levels.
  • Protocol or command knowledge: Knowledge failures arise when agents lack the protocol, mechanism, or command knowledge needed to complete a task.One RIP summarization failure uses a BGP- or OSPF-style workaround instead of RIP’s interface-level ip summary-address rip command.
  • Unproductive hesitation: Unproductive hesitation occurs when agents repeatedly observe or validate without converting feedback into configuration progress, leaving too few turns for remaining subgoals.An eBGP trajectory repeatedly checks BGP summary output and advertised routes without completing all required neighbor work.
  • Premature abandonment: Premature abandonment occurs when agents submit after malformed actions or unexpected observations instead of repairing their action sequence.This instability can involve emitting multiple actions in one turn, receiving an error, and submitting rather than recovering.
  • Specification deviation: Specification deviation occurs when agents rewrite task-provided invariants such as addresses, prefixes, AS numbers, area identifiers, or required mechanisms.In an LDP VPLS example, replacing specified 192.168.x links with a self-designed 10.x /30 scheme causes test cases tied to the original next hops to fail.
  • Implications: The taxonomy shows that failures are not caused only by knowledge deficiencies, while validated trajectories can support model training and harness mechanisms can improve execution reliability and accountability.Successful trajectories can provide demonstrations, and partial or failed trajectories can provide preference signals from test-case scores and process metrics.

2) Improving the Agent Harness:

NetConfArena’s failures indicate that reliable network-configuration agents need support for managing multi-turn interaction, not only protocol knowledge. The benchmark motivates harness mechanisms that encode procedures, track state, enforce invariants, and broaden evaluation beyond agent-only functional correctness.

  • Failure Diagnosis: Many configuration errors arise from weaknesses in managing multi-turn interaction, not solely from missing protocol knowledge.The failures implicate how agents manage the interaction process across turns.
  • Harness Improvements: Reusable skills can bundle prerequisite checks, command templates, validation commands, and repair rules for protocol-specific tasks.These skills supply procedural knowledge required during network configuration.
  • Harness Improvements: An explicit task-state summary can keep agents aligned with unresolved subgoals during execution.The proposed harness maintains state information alongside agent interaction.
  • Harness Improvements: Safety checks can prevent actions from overwriting task-provided addresses, prefixes, AS numbers, or other invariants.The harness checks proposed actions before execution.
  • Scope and Positioning: NetConfArena evaluates interactive agents in executable environments, complementing verification and synthesis systems that target formal properties or configuration generation.Its focus is network feedback during interaction rather than text-only output.
  • Contribution: The benchmark combines emulator-backed interaction, executable test cases, and process-level diagnosis, and has been released open source.It assesses both task success and execution reliability.

APPENDIX

The appendix details NetConfArena’s prompt protocol, trajectory format, task-template structure, executable testing, and coverage across routing, MPLS, and overlay scenarios. It illustrates how agents act incrementally in emulated networks and how behavior is assessed beyond matching a reference configuration.

  • Agent Prompt for Evaluation: The evaluation wrapper combines a fixed system prompt with a parameterized prompt containing each instance’s scenario, topology, and goals.The system prompt defines the agent role, tool protocol, and one-action-per-turn constraint.
  • Agent Prompt for Evaluation: The agent can gather configurations, apply changes, validate state, wait for convergence, and submit the task result.Tool responses are JSON-formatted, and each turn permits exactly one executable action.
  • Execution Trajectory: A successful example expands into a 19-turn trajectory that passes all hidden test cases for a task with three per-prefix objectives.The trajectory uses observations and validation to adjust routing behavior across four routers.
  • Benchmark Details: Task templates contain natural-language intent, topology, initial configurations, a reference configuration, and executable test cases.The reference configuration supports analysis but is not the sole correctness criterion because multiple command sequences can achieve the same behavior.
  • Benchmark Details: Test cases target sub-intents such as route advertisement, selected next hop, neighbor state, and protocol counters to report partial completion and diagnose violations.Diagnostic commands are mapped to deterministic predicates.
  • Benchmark Details: The 96-template task suite covers IP routing, MPLS, and IP Overlay scenarios involving path construction, neighbor establishment, route propagation, filtering, label distribution, and overlay control.These categories exercise distinct forms of protocol-dependent configuration reasoning.

D. Generalizability Analysis

The generalizability analysis asks whether agent performance transfers across task categories and instances. Results show stable relative rankings across subcategories, while finer-grained task requirements explain more score variation than broad categories.

  • Research Question: RQ4 tests whether agent performance generalizes across task categories and different instances of the same template.The analysis uses main-evaluation test-case scores.
  • Variance Decomposition: Variance decomposition separates score variation across category, subcategory, template, and instance levels.The analysis follows the benchmark’s task-taxonomy hierarchy.
  • Variance Decomposition: 1.5%–7.7% of total variance comes from the top-level category component, with a median of 5.7%.The subcategory and template components together have a median contribution of 37.5%.
  • Ranking Consistency: The median Spearman rank correlation across 16 subcategories is ρ = 0.928, with 11 of 16 subcategories reaching ρ ≥0.9.Correlations compare subcategory-specific agent-setting orderings with the overall ordering.
  • Answer to RQ4: Within NetConfArena, relative agent-setting rankings appear stable across evaluated subcategories, while randomized instances of one template can produce different absolute outcomes.The conclusion distinguishes ranking stability from instance-level score variation.
Loading 2608.23179v1…