PaperlayerView exact revision on arXiv

Source-linked AI summary

NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

Ahmed Twabi, Yepeng Ding, Tohru Kondo

arXiv:2604.09678v1cs.NIcs.AIcs.FL

TL;DR

Network configuration benchmarks need to capture stateful, multi-turn behavior rather than rely only on static checks. NetAgentBench uses an FSM formalization to provide deterministic, bounded evaluation, and its evaluation of four LLM agents found frequent meltdowns and near-zero success beyond basic tasks.

  • Problem

    Existing evaluation lacks formal, state-aware ways to assess adaptive multi-turn exploration and behavioral stability in network configuration.

    Source: §I · p. 1

  • Method

    NetAgentBench models network evaluation with interacting FSMs and establishes determinism, correctness, and bounded execution for agents operating within time and turn limits.

    Source: §I · p. 1

  • Results

    21%–39% meltdown rates and near-zero success beyond basic tasks were observed across four state-of-the-art LLMs in 300 benchmark runs.

    Source: §IX · p. 9

  • Takeaways & Limitations

    NetAgentBench provides a mathematically grounded, state-centric foundation for rigorous evaluation and safer deployment of trustworthy autonomous network agents.

    Source: §IX · p. 9

    Parsing unstructured CLI output on closed-source platforms can be fragile across vendor OS versions or when tables are truncated.

    Source: §V · p. 6

Abstract

from arXiv · show

As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.

I. INTRODUCTION

Network configuration is stateful and time-dependent, yet existing evaluations often use static or one-shot checks that miss operational reliability and multi-turn behavior. NetAgentBench addresses this measurement gap with a state-centric FSM benchmark for bounded, multi-turn evaluation.

I. INTRODUCTION

  • The field lacks standardized, state-aware benchmarks for rigorously evaluating LLM agents’ operational reliability and behavioral stability across network-management tasks.The paper frames this as a fundamental measurement gap amid growing interest in LLM-based network configuration.Source: §I · p. 1
  • NetAgentBench models multi-turn benchmarking as interacting FSMs and proves determinism, correctness, and bounded execution within time and turn constraints.Tasks use literature-grounded configurations of varying difficulty, with success checked against intended properties in the final network state.Source: §I · pp. 1–2

II. BACKGROUND AND PROBLEM DEFINITION

  • Network behavior depends on current state and convergence time, so syntax checks and static evaluations can miss runtime failures and configurations that do not remain correct when reapplied.Static evaluations also cannot assess iterative multi-turn behaviors such as ineffective-command loops and destruction of partially correct states.Source: §II · p. 2, A. Why Static Benchmarks Are Inadequate · p. 2

B. The Idempotency Problem in Automation

  • A configuration can pass a static check on first application yet fail dynamic robustness testing when reapplication briefly disrupts an OSPF adjacency.In the example, reapplying passive-interface default temporarily makes interfaces passive before eth0 is re-enabled, potentially causing missed Hello packets and adjacency flaps.Source: B. The Idempotency Problem in Automation · p. 2

III. FRAMEWORK DESIGN

The framework coordinates infrastructure provisioning, an interactive network under test, and a benchmark controller as interacting state machines. Tasks pair a deterministic topology blueprint with required network properties, while the infrastructure FSM provisions the starting network or records an error.

III. FRAMEWORK DESIGN

  • NetAgentBench models the lifecycle with interacting FSMs for deterministic infrastructure provisioning, event-driven configuration in the SUT, and benchmark orchestration.The controller coordinates the infrastructure and SUT through the Initialize bridge function.Source: §III · p. 2, Figure 1 · p. 2

A. Benchmark Inputs

  • Each task provides a topology specification T as infrastructure commands and an intent specification P as required properties evaluated over network state.T determines the network arrangement of nodes, links, and interfaces; P contains predicates that evaluate true or false.Source: A. Benchmark Inputs · pp. 2–3

B. Infrastructure FSM (Minfra)

  • The Infrastructure FSM processes T sequentially from an initial state, reaching an accepting state when the topology is provisioned or an error state if provisioning fails.An accepting state requires at least one node and a deployed topology; its state records provisioned nodes and established links.Source: B. Infrastructure FSM (Minfra) · p. 3

C. System Under Test (SUT) FSM (MSUT)

The SUT models agent configuration as bounded, event-driven transitions among stable, spending, and error states. It evaluates the resulting configuration sequence against observable intent properties, with finite-state abstractions and bounded protocol convergence supporting the benchmark’s formal setup.

C. System Under Test (SUT) FSM (MSUT)

  • A configuration command moves the SUT from a stable state to a spending state, then an autonomous ε-transition resolves it to a stable or error state.Protocol convergence exceeding Tmax instead transitions the system to an error state.Source: C. System Under Test (SUT) FSM (MSUT) · pp. 3–4
  • The benchmark evaluates the ordered configuration-command sequence extracted from the agent’s interaction trace, rather than the opaque agent itself.The agent operates for at most K turns, and its final state is the state reached at termination.Source: C. System Under Test (SUT) FSM (MSUT) · pp. 3–4
  • The SUT’s finite abstraction uses finite topology and command spaces, categorized runtime values, and equivalence between states with identical read-operation observations.The observable-equivalence quotient is finite because the read operations and their output domains are finite.Source: C. System Under Test (SUT) FSM (MSUT) · p. 4
  • Accepting states are stable network states in which every intent property holds and protocol convergence is complete.The SUT starts in a stable state with a provisioned topology but no protocol configuration.Source: C. System Under Test (SUT) FSM (MSUT) · pp. 3–4

D. Evaluation Engine

The Evaluation Engine checks the final network state against intent properties using read-only observations, then scores the interaction trace. Its combined score weights completeness, robustness, and command validity, with incomplete solutions receiving no robustness score.

D. Evaluation Engine

  • The Evaluation Engine checks whether the final state satisfies the intent and computes a score from the evaluation.Property checks are non-invasive and rely on observations; under the stated Perfect Observability assumption, suitable read operations can check every property.Source: D. Evaluation Engine · p. 4
  • Incomplete solutions receive a robustness score of zero, while failed exploratory commands are tolerated if the final state satisfies the intent.Syntactic validity measures command validity across the interaction history and does not determine solution completeness.Source: D. Evaluation Engine · p. 4
  • The combined score uses tunable nonnegative weights wC, wR, and wX summing to 1; baseline experiments set each to 1/3 and averaged results into a final aggregated score.Source: D. Evaluation Engine · p. 5

E. Benchmark Controller (Mbench)

A Mealy-machine Benchmark Controller coordinates task loading, infrastructure provisioning, agent exploration, evaluation, and scoring. It advances through sequential lifecycle states and handles infrastructure errors by entering a failure state.

E. Benchmark Controller (Mbench)

  • The controller progresses through idle, provisioning, ready, exploration, evaluation, and scoring phases before reaching success or failure.Infrastructure errors during provisioning trigger the controller’s error state.Source: E. Benchmark Controller (Mbench) · p. 5
  • The Mealy-machine controller uses external status events and agent-turn events as inputs, and emits infrastructure, configuration, read, evaluation, and scoring actions.Its output function produces command sequences atomically.Source: E. Benchmark Controller (Mbench) · p. 5
  • The controller loads the task and topology, provisions infrastructure, initializes the SUT, and then hands control to the agent’s exploration loop.Agent exploration ends on STOP or exhausted budget, after which evaluation and scoring run.Source: E. Benchmark Controller (Mbench) · p. 5

IV. BENCHMARK PROOFS

The benchmark’s formal proofs target determinism, correctness, and bounded execution. Their observation-sufficiency lemma relies on Perfect Observability: each intent property can be determined from a finite set of read operations.

IV. BENCHMARK PROOFS

  • The authors aim to prove determinism, correctness, and bounded execution as foundational properties of the benchmark.They introduce two supporting lemmas before the proofs.Source: §IV · p. 5
  • For each intent property, a finite set of read operations suffices to determine whether it holds.The proof derives this from Perfect Observability, under which no property depends on unobservable internal state.Source: §IV · p. 5

A. Benchmark Determinism

The benchmark formalizes deterministic scoring and checks solution validity against final-state properties, while its practical robustness check is narrower than the universal definition.

A. Benchmark Determinism

B. Validity and Correctness

C. Benchmark Bounded Execution

Benchmark execution is bounded by explicit limits on exploration and property evaluation, so every finite-input run halts.

V. PRACTICAL BENCHMARK LIMITATIONS

Practical constraints include fragile CLI parsing and timeout-related false negatives; the benchmark is implemented on containerized networks and spans three certification difficulty tiers.

V. PRACTICAL BENCHMARK LIMITATIONS

  • Unstructured CLI parsing on closed-source platforms can be fragile across vendor versions or when tables are truncated, despite theoretical observability.Source: §V · p. 6
  • Aggressive timeouts risk false negatives during multi-turn convergence, although explicit read commands can help agents verify convergence.Source: §V · p. 6

A. Development

B. Task Generation

  • The curated suite has five configuration and troubleshooting tasks, mapping protocol complexity from RIP through OSPF to BGP across three certification tiers.Source: B. Task Generation · p. 7
  • Tasks cover four-router RIPv2 reachability, five-router multi-area OSPF and adjacency troubleshooting, and dual-homed BGP plus route-filtering troubleshooting.Source: B. Task Generation · p. 7

C. Experimental Configuration

Across 300 runs, GPT-5 leads the four evaluated models, but success remains low beyond basic tasks. Difficulty-related failures and behavioral traces reveal distinct convergence problems.

C. Experimental Configuration

A. Model Performance

  • GPT-5 has the highest mean score (µ = 0.508) and success rate (24.0%), significantly ahead of all competitors; no model nears reliable completion.Source: Table I · p. 7, A. Model Performance · p. 7
  • The performance results support RQ1 by showing that the deterministic benchmark separates stochastic agent performance at scale.Source: A. Model Performance · p. 7

B. Task Difficulty and Agent Convergence (RQ2)

C. Multi-Turn Behavioral Analysis (RQ3)

Multi-turn behavioral analysis exposes coherence loss, exploration meltdowns, and efficiency trade-offs that static evaluations miss. The findings also show how short runs can make low meltdown rates or high token efficiency misleading.

1) Coherence Collapse:

  • GPT-5 largely preserves monotonic progress, while Qwen3-Coder-30b’s 1.688 diagnostic-to-constructive ratio signals cycling; all models lose OSPF coherence near 1/4 properties.Llama-3.3 averages 3.6 turns per run, avoiding prolonged collapse at the cost of near-zero task completion.Source: Figure 5 · p. 8, 1) Coherence Collapse · p. 8

2) Exploration Meltdown:

  • GPT-5 has the highest meltdown rate (38.7%), while Qwen models show premature submissions and Llama’s low rate (21.3%) reflects short runs, not stability.Signals include repeated commands ≥4 times, destructive commands ≥3 within five, low-score stagnation beyond 25 turns, 10 diagnostic-only steps, or submission below 0.30.Source: Figure 6 · p. 8, 2) Exploration Meltdown · p. 8

3) Token Efficiency:

  • Llama-3.3 leads average score-per-1K-token efficiency, but early termination drives that advantage; GPT-5 has the highest absolute score yet lowest average token efficiency.The comparison uses token efficiency against overall score.Source: Figure 7 · p. 8, 3) Token Efficiency · p. 9

VIII. FUTURE WORK

  • The authors propose testing multi-agent designs to mitigate behavioral failures and expanding tasks to multi-vendor platforms and complex scenarios to assess generalization.Examples include hierarchical agents, Juniper and Arista platforms, SD-WAN, and large-scale BGP.Source: §VIII · p. 9

IX. CONCLUSION

  • Across 300 runs, meltdown rates of 21%–39% and near-zero success beyond basic tasks summarize four agents’ persistent behavioral failures.The failures include excessive diagnostic cycling, destructive spirals, and early aborts, which static one-shot evaluation misses.Source: §IX · p. 9
Loading 2604.09678v1…