Source-linked AI summary
An Experimental Study on the Functional Correctness of Network Device Configurations Produced by Large Language Models
Anastasios C. Politis
TL;DR
This preliminary study asks whether freely available LLMs can produce functional network-device configurations on their first response, without prompt engineering. It compares routing-table prefixes from three models against a manually configured reference: ChatGPT was fully effective, while Gemini and LeChat had missing prefixes.
Problem
Manual device configuration is difficult and error-prone, and the study asks whether LLM-generated configurations can make network systems operate as intended.
Source: §I · p. 1Method
Three freely available, standard-capability LLMs receive an initial prompt for routing configurations, which are evaluated by comparing resulting routing-table prefixes with a manually configured reference.
Source: §I · pp. 1–2, §V · p. 4Results
ChatGPT produced fully functional configurations, Gemini missed 66% of required prefixes on four devices, and LeChat left five of six prefixes missing across all devices.
Source: §V · p. 5Takeaways & Limitations
ChatGPT’s initial configuration was fully effective in this test, while Gemini’s output required user validation and LeChat struggled to produce accurate configurations.
Source: §V · p. 5The preliminary investigation is limited to routing configurations from three freely available LLMs in a predefined network setting.
Source: §I · pp. 1–2
Abstract
from arXiv · showhide
This paper presents a preliminary study on the efficacy of network device configuration produced by Large Language Models (LLMs). Currently, the study focuses on routing protocol configurations in a predefined small internetwork with Cisco-based routing devices. The capability of three freely available LLMs with standard capabilities (i.e., with no enhanced features) to produce effective setups is investigated. Prompt engineering methods are excluded in order to assess the "first-response" configurations of the models under consideration. The assessment method is based on the contents of the resulting routing table, aiming at measuring the device functional correctness. For that purpose, the Jaccard index is used to indicate the percentage of correct network prefixes contained in the routing table constructed by the model-based configuration commands. The work provides insights on the maturity and limitations of modern LLMs on aiding network administrators with network device configurations in real deployments.
I. INTRODUCTION
Network device configuration remains difficult and often requires vendor-specific CLI work, while intent-based alternatives add their own learning burden. This study asks whether freely available LLMs can produce functional routing configurations on their first response.
I. INTRODUCTION
- Manual configuration requires vendor-specific knowledge and can cause erroneous behavior, while intent-based tools require familiarity with additional high-level languages,,,.Legacy and proprietary equipment also means human CLI intervention remains frequent,,,.Source: §I · p. 1
- The central question is whether LLM-produced network device configurations are sufficient to make systems operate as intended.LLMs already accept natural-language guidance for tasks such as research, content creation, and programming.Source: §I · p. 1
- Repeated corrective prompting may improve configurations, but adds human effort and time, motivating evaluation of models’ initial responses without prompt engineering.The study frames first-response functionality as practically meaningful.Source: §I · p. 1
- The preliminary study examines routing configurations from three freely available, standard-capability LLMs: GPT, Gemini, and Le Chat.The models are assessed without prompt engineering to evaluate immediate configurations.Source: §I · pp. 1–2
II. RELATED WORKS
Related work explores LLM-assisted configuration translation and correction, as well as routing-configuration effectiveness. The approaches described include verifiers, heuristics, and prompt-based knowledge updates.
II. RELATED WORKS
- A GPT-4 system for Cisco-to-Juniper router configuration translation performed poorly, so its authors paired GPT-4 with a verifier providing local feedback to correct misconfigurations.Source: §II · p. 2
- A unified translation framework uses LLMs for vendor-specific commands without parameters and heuristics for commands with parameters.Source: §II · p. 2
- An LLM-agent framework fragments configuration files, extracts intents, and uses manual target-device information plus syntax checkers and verifiers to address hallucinations.Source: §II · p. 2
- A related experimental study evaluates GPT-4-Turbo routing configurations for FRRouting using prompts to update and extend the model’s knowledge.Source: §II · p. 2
III. METHODOLOGY
The study evaluates freely available LLMs using compact, identical prompts without iterative correction. It assesses functional routing outcomes by comparing resulting network-prefix sets with a manually configured reference.
III. METHODOLOGY
- The evaluation uses freely available LLM chatbots without advanced features because enhanced versions may require subscriptions.Source: §III · p. 2
- The study excludes sophisticated prompt engineering and continuous conversations, focusing instead on the initial output and avoiding the time demands of iterative prompting.Source: §III · p. 2
- A single compact prompt, accompanied by the network diagram, is used identically across the investigated LLMs.The paper notes that lengthy prompts have been shown to degrade LLM performance.Source: §III · p. 2
- Configuration-command similarity can misjudge functionality because different OSPFv2 command variants can enable the protocol on the same four interfaces.The passage gives variants ranging from one network statement to four host-specific statements; both can achieve the same goal despite a cosine text similarity of approximately 0.55.Source: §III · pp. 2–3, Figure 1 · p. 2
- The method compares routing-table prefixes against a manually configured reference because those entries represent reachability of remote networks.Source: §III · p. 3
- The modified Jaccard index assumes LLM-produced prefixes are a subset of the reference set, making the score the fraction of reference prefixes captured.The index ranges from 0 for no common prefixes to 1 for identical prefix sets.Source: §III · p. 3
IV. EXPERIMENTS
Experiments apply model-generated configurations to a predefined internetwork and compare resulting routing tables with a manually configured reference. The same prompt and attached topology diagram are used across three LLMs.
IV. EXPERIMENTS
- The test topology contains two autonomous systems with three routers each, using OSPF internally and BGP between border routers R11 and R21.Source: §IV · p. 3
- The network was built manually in GNS3, and its fully functional routing tables serve as the reference for evaluating model configurations.The reference tables are identified in Fig. 4.Source: §IV · p. 4, Figure 4 · p. 4
- ChatGPT, Gemini, and LeChat received the same concise prompt with the topology diagram attached, and their returned device configurations were pasted into router CLIs.Source: §IV · p. 4
V. EXPERIMENTAL RESULTS
The study compares LLM-generated routing tables with manually configured reference tables, using network-prefix similarity to assess routing correctness. ChatGPT produced fully matching prefixes, while Gemini and LeChat produced incomplete results with distinct missing-prefix patterns.
V. EXPERIMENTAL RESULTS
- The reference tables showed all intra- and inter-AS networks reachable from all routers, with OSPF and BGP routes represented by protocol codes.Directly connected prefixes were excluded from the Jaccard calculation because their presence depends on interface IP configuration, not routing.Source: §V · p. 4, Table I · p. 4
- ChatGPT’s generated routing tables had total similarity to the reference tables in network prefixes, yielding a fully functional internetwork.Differences in router IDs and OSPF interface-enabling commands did not cause malfunction.Source: §V · p. 5
- Gemini’s configurations left four of six required network prefixes missing from the remaining devices, with all reference O E2 entries absent.The border routers redistributed OSPF-learned prefixes toward BGP, but not BGP-learned remote-AS prefixes into local OSPF.Source: §V · p. 5
- LeChat’s configurations left five of six network prefixes missing from every device’s routing table.Observed misconfigurations included OSPF on BGP interfaces, absent redistribution commands, and AS numbers differing from the diagram; the AS-number and addressing mishaps were excluded from evaluation.Source: §V · p. 5
VI. CONCLUSIONS
The study tested first-response configurations from three freely available chatbots in a predefined network, comparing their routing tables with a manually configured reference. ChatGPT returned fully effective configuration sets, Gemini followed closely, and LeChat performed worst; the study proposes broader future evaluation.
VI. CONCLUSIONS
- The study inserted configurations from ChatGPT, Gemini, and LeChat into a predefined routing topology and compared resulting routing tables with a manually configured reference.Prompt engineering and extended chatbot conversations were avoided to assess initial responses.Source: §VI · p. 5
- ChatGPT returned fully effective configuration sets, Gemini followed closely, and LeChat produced incomplete routing tables attributed to hallucinations.Source: §VI · p. 5
- Future work will assess configurations for services beyond routing and include more LLMs, switches, and firewalls.Source: §VI · p. 5