Source-linked AI summary
FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets
Kuan-Hao Tseng, Niruth Bogahawatta, Yasod Ginige, Kunjan Patel, Kosta Dakic, Suranga Seneviratne
TL;DR
Existing network-troubleshooting benchmarks generally use accurate tickets and assume a fault exists, leaving agents’ ability to handle unreliable reports untested. FaulT-Bench evaluates agents across 200 scenarios and controlled persona rewrites using an automated emulation and judging harness. Agents handle genuine and misleading-fault cases relatively well but struggle sharply with healthy networks and vague tickets, revealing distinct failure modes and a need to treat tickets as hypotheses.
Problem
Existing benchmarks assume accurate fault reports and genuine faults, limiting evaluation of agents’ ability to verify unreliable user tickets and recognize that nothing is wrong.
Method
FaulT-Bench evaluates free-text diagnoses across 200 scenarios and eight topologies, rewrites 72 false-premise tickets into five personas, and scores agents through an automated Kathará/NIKA harness.
Results
Agents perform well on genuine faults and resist wrong-device or wrong-cause misdirection, but falsely diagnose up to 24% of healthy networks and phrasing changes accuracy by up to 14.5 percentage points.
Takeaways & Limitations
Treating a ticket as a hypothesis rather than an instruction is distinct from diagnostic skill and remains lacking in current troubleshooting agents.
Takeaways & Limitations
Fault-based evaluation cannot distinguish ticket verification from blind compliance when a genuine fault exists; the behaviors separate only for nonexistent faults.
Abstract
from arXiv · showhide
LLM-based agents are increasingly proposed for network fault diagnosis, but existing benchmarks evaluate them only on accurate tickets and always assume a fault is present, conditions rarely met in practice. We present FaulT-Bench, a benchmark of 200 troubleshooting scenarios across eight network topologies, five reimplemented from public practitioner labs, spanning genuine faults, false fault reports, incorrect device attribution, and incorrect root-cause claims. To isolate how ticket wording affects diagnosis, we further rewrite 72 false-premise tickets into five reporter personas that vary reporter confidence and verifiable detail one factor at a time, holding the network state fixed. Our automated harness deploys each scenario in Kathará, lets agents interact through the NIKA tool interface, and scores free-text diagnoses with an LLM judge across outcome, fix, and reasoning quality. Evaluating SADE, ReAct, and Claude Code, we find all three are near-saturated on accurate tickets and robust to misdirection, yet degrade sharply when the network is healthy and the ticket is wrong, probing until a benign condition can be promoted to a root cause rather than concluding nothing is wrong. Persona rewrites show that how a ticket is written matters more than what it claims: a confidently wrong report is handled about as well as an accurate one, while a vague, underspecified report degrades performance sharply. The three agents also fail differently, from constant over-diagnosis to unanswered runs, at very different cost. These results position FaulT-Bench as a benchmark for developing agentic systems that can reason reliably over the noisy, unreliable tickets of real-world network troubleshooting.
1 INTRODUCTION
FaulT-Bench addresses the limited realism of existing network-troubleshooting evaluations by testing agents on unreliable user tickets, including reports that are ambiguous or false. It combines diverse scenarios, controlled ticket rewrites, an automated harness, and evaluations showing strong performance on genuine faults but substantial difficulty recognizing healthy networks.
- Motivation: Existing benchmarks largely assume accurate fault reports and genuine faults, so they cannot test whether agents verify that a reported problem actually exists.Real trouble tickets may be ambiguous, incomplete, inconsistent, or misleading, while NIKA assumes every scenario contains a genuine fault and begins without a user-written report.
- Benchmark: FaulT-Bench contains 200 scenarios across eight topologies covering correct faults, false premises, wrong devices, and wrong causes.The dataset includes 80 Correct-Fault and 120 incorrect-reporting scenarios, plus 360 rewrites of 72 False-Premise tickets in five personas.
- Benchmark: The benchmark varies reporter confidence and verifiable detail independently by rewriting each False-Premise ticket into five personas while holding the network state fixed.This design isolates how reporting style affects diagnosis rather than changing the underlying network condition.
- Evaluation: Its automated harness deploys scenarios in Kathará, supports interaction through NIKA, and uses an LLM judge to score outcome, repair, and reasoning quality.The evaluation targets free-text diagnoses across three dimensions.
- Findings: Agents perform well on Correct-Fault, Wrong-Device, and Wrong-Cause scenarios but degrade substantially on False-Premise scenarios, often blaming healthy devices.The reported evaluation covers SADE, NIKA’s ReAct agent, and Claude Code.
- Findings: Agents falsely diagnose up to 24% of healthy networks, while ticket phrasing changes diagnostic accuracy by up to 14.5 percentage points.SADE falsely diagnoses 24% of healthy networks versus 11% for Claude Code; phrasing effects are attributed to verifiable detail rather than confidence.
2 RELATED WORK
Prior work spans network-automation methods, LLM troubleshooting agents, emulation platforms, and task-specific benchmarks. FaulT-Bench extends this landscape by evaluating free-text troubleshooting with unreliable reports and false-premise cases on reproducible network environments.
- Network automation: Earlier network-management automation includes zero-touch service management, intent-based networking, static analysis, and machine-learning approaches such as graph neural networks.These approaches precede newer LLM-based systems for network configuration and troubleshooting.
- LLM agents: LLM troubleshooting research includes zero-shot virtual administrators, ReAct-style tool loops, procedural policies, and root-cause analysis for cloud and microservice systems.Industrial deployments have also applied LLM agents to dialogue-based diagnosis and multi-agent event handling.
- Evaluation environments: Container-based emulators such as Kathará provide reproducibility and programmatic fault injection, whereas VM-based platforms such as GNS3 offer greater vendor-OS fidelity.FaulT-Bench translates five operational Cisco/GNS3 enterprise networks into Kathará/FRR labs and retains three NIKA topologies as controls.
- Benchmarks: Existing benchmarks target configuration synthesis, broad management, telecom troubleshooting, cloud incidents, or curated network faults rather than unreliable L2/L3 troubleshooting tickets.General AI-agent benchmarks contain no networking environments, and NIKA assumes genuine faults without user-written reports.
- FaulT-Bench: FaulT-Bench operationalizes abstention and robustness to vague or confidently wrong reporters through free-text tickets, real-derived and reference topologies, and rubric-based LLM judging.The related-work discussion identifies these failure modes as unaddressed by networking benchmarks.
3 FAULT-BENCH BENCHMARK
FaulT-Bench evaluates network-troubleshooting agents on realistic tickets across diverse emulated topologies, including genuine faults and misleading reports. Controlled persona rewrites isolate how confidence and verifiable detail affect diagnosis while the automated framework standardizes deployment, interaction, and grading.
- 3.1 Network Topologies and Ticket Types: FaulT-Bench contains 200 scenarios across eight topologies, including five reimplemented from publicly available practitioner-built laboratory networks.Each topology contributes 25 scenarios, with 10 correct-fault and 15 incorrect or misleading-ticket cases.
- 3.1 Network Topologies and Ticket Types: Each false-premise ticket is rewritten into five reporter-persona variants while the verified-healthy network state remains unchanged.Controls separately manipulate reporter confidence, suspected-cause language, and verifiable identifiers.
- 3.3 Evaluation Framework: The evaluation harness deploys scenarios in Kathará, lets agents investigate through NIKA tools, and grades free-text diagnoses against ground truth.The workflow prepares the topology, injects a specified fault when applicable, triggers the agent with the report, and evaluates its answer.
- 3.4 Performance Metrics: FaulT-Bench assesses diagnoses across outcome accuracy, repair recommendation quality, and reasoning quality.The framework is designed for automatic integration and evaluation of LLM-agent-based troubleshooting systems.
4 BASELINE AGENT EVALUATION ON FAULT-BENCH
The baseline evaluation compares three agents under identical tool and input conditions on accurate and misleading tickets. Accurate-fault cases are near-saturated, whereas incorrect-reporting cases expose materially different and noisier failure behavior.
- 4 Baseline Agent Evaluation: SADE, Claude Code, and ReAct receive byte-identical inputs and use the same 22-tool interface under a shared evaluation setup.The agents represent skills-augmented, Claude Code, and classic reason-and-act designs.
- 4 Baseline Agent Evaluation: 80 correct-fault scenarios answer RQ1, while 120 incorrect-reporting scenarios answer RQ2 and expose failure modes missed by accurate-ticket evaluation.Because individual runs are noisy, every E1 figure is reported as a three-pass mean.
- 4 Baseline Agent Evaluation: Each persona rewrite is paired with its standard-ticket counterpart on the same network state, making within-pair differences attributable to the manipulated reporting factor alone.Persona runs use repeated executions and freshly deployed networks.
- 4 Baseline Agent Evaluation: Runs without a parseable verdict are counted as timeouts and reported separately.All runs are graded by the same LLM judge used elsewhere in the evaluation framework.
5 RESULTS
Agents nearly solve accurately reported faults and remain robust to incorrect device or cause claims, but healthy-network tickets expose over-diagnosis, wording sensitivity, and uneven cost.
- Evaluation Setup: FaulT-Bench deploys scenarios in Kathará, supports NIKA-tool interaction, and scores diagnoses on outcome, repair, and reasoning quality.The benchmark includes 200 scenarios across eight topologies and evaluates SADE, ReAct, and Claude Code.
- Accurate and Misleading Tickets: Outcome scores on 80 Correct-Fault scenarios were 0.912 for SADE, 0.942 for CC-Baseline, and 0.941 for ReAct.Accurate-fault diagnosis is nearly solved and provides limited separation among agent designs.
- Accurate and Misleading Tickets: Wrong-Device and Wrong-Cause tickets preserved robust diagnosis, with outcome scores of 0.957 and 0.911 and average fix scores of 0.946, 0.919, and 0.897 across fault classes.Agents probed symptoms directly and discarded claims contradicted by the live network.
- False-Premise Tickets: False-Premise scenarios reduced mean outcome to 0.820 versus 0.932 for Correct-Fault scenarios because agents over-diagnosed benign network conditions.They disproved the reported symptom but continued probing until a genuine standing condition could be presented as a root cause.
- Reporting Personas: On identical healthy networks, the vague novice persona scored 0.675, while a definite but false cause behaved similarly to the standard technical ticket.The key distinction was whether the ticket supplied a verifiable claim for the agent to test.
- Reporting Personas: Adding confidence to novice tickets improved outcome by +0.039, whereas removing certainty and then identifiers reduced outcome by 0.182 and 0.141 and increased over-diagnosis to 35% and 47%.Across the control chains, uncertainty and vagueness mattered more than confidence.
- Diagnostic Effort and Cost: Vague tickets increased effort: CC-Baseline made 30 tool calls and read 549k tokens on no-detail rewrites versus 13 calls and 136k tokens on Correct-Fault scenarios.Timeouts were 9% of runs but consumed 23% of total spend, with 36% of CC-Baseline’s budget producing no diagnosis.
6 CONCLUSION
FaulT-Bench evaluates troubleshooting agents under realistic, unreliable tickets across diverse network states and topologies. The results show strong performance when faults leave network evidence, but persistent over-diagnosis or non-verdicts when no fault exists and tickets lack verifiable identifiers.
- 6 CONCLUSION: FaulT-Bench covers 200 scenarios across eight network topologies, including genuine faults and three kinds of incorrect reports.Five topologies were translated from public practitioner labs; incorrect reports include nonexistent faults, healthy-device blame, and wrong-cause claims.
- 6 CONCLUSION: When faults are present, agents localize them despite wrong-device or wrong-cause tickets because systematic probing recovers network evidence.The benchmark conclusion attributes this robustness to evidence left by genuine faults.
- 6 CONCLUSION: When no fault exists, agents disprove reported symptoms but continue probing until a benign condition is promoted to a root cause.This failure reflects difficulty concluding that nothing is wrong in a healthy network.
- 6 CONCLUSION: Removing verifiable identifiers reduces outcome score by 0.141, producing false diagnoses from SADE and ReAct and unanswered runs from CC-Baseline.CC-Baseline returns no verdict in up to half of its runs under this condition.
- 6 CONCLUSION: FaulT-Bench identifies treating tickets as hypotheses rather than instructions as a capability distinct from diagnosis that current agents lack.The benchmark therefore isolates a specific reliability gap in troubleshooting agents operating from unreliable reports.