TN-AutoRCA: Benchmark Construction and Agentic Framework for Self-Improving Alarm-Based Root Cause Analysis in Telecommunication Networks

introduction

• Telecom RCA is a multi-label classification task over graph-structured data, where noisy, interdependent alarms and cascading failures require domain knowledge.Existing general benchmarks do not capture telecom-specific challenges such as alarm storms and dependencies across physical and logical network layers.Source: introduction · pp. 1–2
• 58.99% is the top F1-score reported for a directly evaluated state-of-the-art LLM on the real-world telecom benchmark.The authors present this as a baseline that quantifies the task’s difficulty.Source: introduction · p. 2
• TN-RCA530 is introduced as the first large-scale, real-world benchmark for telecom RCA, featuring 530 scenarios grounded in expert-verified knowledge graphs, a verifiable construction methodology, and automated complexity grading.Source: introduction · p. 2
• 91.79% F1-score on TN-RCA530 is the final solution’s reported result, achieved with Auto-RCA’s iterative, failure-pattern-driven code repair.The evaluate-analyze-repair loop generates contrastive feedback from aggregate failure patterns to guide targeted repairs.Source: introduction · p. 2

Knowledge Graph Enhanced LLMs

• KG–LLM approaches align reasoning with graph paths, navigate knowledge graphs, or rerank generated candidates; integration is presented as a way to improve factual grounding and mitigate hallucinations.The cited approaches include path alignment (Shen, Wang, and Xia 2025), graph navigation (Shen et al. 2025), and candidate reranking.Source: Knowledge Graph Enhanced LLMs · p. 2

Root Cause Analysis (RCA)

• Telecom RCA differs from software RCA through alarm floods, complex topological dependencies, and stringent real-time constraints, while the field lacked a large-scale public benchmark.The passage contrasts telecom-specific needs with general benchmarks such as Defects4J (Just, Jalali, and Ernst 2014) and RCAEval (Pham et al. 2025).Source: Root Cause Analysis (RCA) · p. 2

LLM-based Agents for Automated Program Repair

• Agentic program-repair work uses modular systems, while Auto-RCA extends self-correction from failing-and-passing test pairs to whole-benchmark failure patterns.Self-Refine relies on an LLM’s intrinsic correction; ContrastRepair (Kong et al. 2024) uses contrastive test pairs.Source: LLM-based Agents for Automated Program… · pp. 2–3

LLMs for Telecommunication Networks

• Telecom LLM research includes specialized models and tool-assisted agents; the passage argues reliable RCA in dynamic settings requires external tools or structured knowledge.TAMO is cited as an example using agents with multimodal data for fine-grained RCA (Wang et al. 2025).Source: LLMs for Telecommunication Networks · p. 3

Benchmark Construction: The TN-RCA530 Dataset

• TN-RCA530 comprises 530 distinct tele-network failure scenarios modeled as knowledge graphs, with root causes selected from candidate nodes.Its construction follows veracity, comprehensiveness, verifiability, and complexity-discriminability principles.Source: Benchmark Construction: The TN-RCA530… · p. 3
• Cycle consistency grades scenarios by mapping cardinality: one-to-one, matching causes are Simple; one-to-many mappings with plausible alternatives are Difficult.The automated process removes subjective manual difficulty labeling and supports analysis across complexity levels.Source: Benchmark Construction: The TN-RCA530… · pp. 3–4, Figure 2 · p. 3

Data Representation and Statistics

• Each scenario is represented as a directed graph of resource, alarm, and root-cause nodes connected by equipment-dependency, alarm-generation, and causal-hypothesis edges.The inputs and ground-truth labels are stored separately in input.json and label.json.Source: Data Representation and Statistics · pp. 4–5
• TN-RCA530 has 29 Simple and 501 Difficult scenarios, with a long-tail root-cause distribution aligned with real-world operational data.The Difficult scenarios make up 94.5% of the benchmark.Source: Figure 3 · p. 4, Data Representation and Statistics · p. 5

The Auto-RCA Agentic Framework

• Auto-RCA analyzes aggregate failure patterns to guide hypothesis-driven code repairs through a modular, iterative evaluate-analyze-generate-validate workflow.The evaluator measures benchmark F1 and compiles incorrect outputs as Bad Cases; the orchestrator accepts a candidate only if its score improves.Source: The Auto-RCA Agentic Framework · pp. 5–6, Figure 4 · p. 5

Evaluation Framework and Metrics

• The evaluation framework submits all 530 scenario graphs to a model and requests standardized JSON identifying target alarms, inferred causes, equipment IDs, and proposed solutions.Root causes are exact-match tuples of cause description and equipment ID; the reported model score is macro-averaged F1 across scenarios.Source: Evaluation Framework and Metrics · p. 6

Experimental Setup

• Each experiment uses all 530 expert-verified scenarios, macro-averaged F1 as the primary metric, and five optimization rounds from the same initial code base.The benchmark includes 29 Simple and 501 Difficult scenarios; P@1 and R@1 are also reported.Source: Experimental Setup · p. 6

RQ1: Baseline Performance of LLMs on TN-RCA530

• 0.8621 is the identical Precision@1, Recall@1, and F1-Score@1 achieved by six models on Simple scenarios.The results indicate comparable performance on the less complex cases.Source: Table 1 · p. 7, RQ1: Baseline Performance of LLMs on… · p. 7
• On Difficult scenarios, Qwen3-235B leads in Recall@1 (0.9810) and F1-Score@1 (0.6049), while Qwen-QWQ leads in Precision@1 (0.4381).On Mixed scenarios, Qwen-QWQ leads in Precision@1 (0.4613), and Qwen3-235B leads in Recall@1 (0.9736) and F1-Score@1 (0.6254).Source: Table 1 · p. 7, RQ1: Baseline Performance of LLMs on… · p. 7

RQ2: Effectiveness of the Auto-RCA Framework

• 0.9179 (Gemini-2.5-Pro) exceeds the best Mixed-scenario baseline, 0.6254 (Qwen3-235B), for final F1-score.The framework result is reported after five refinement rounds.Source: Figure 5 · p. 7, RQ2: Effectiveness of the Auto-RCA… · p. 8
• 0.0000 marks a rejected code change after an error or performance decrease, so the framework retains the best solution rather than accepting a regression.The figure describes best-solution F1-score across five refinement iterations.Source: Figure 5 · p. 7, RQ2: Effectiveness of the Auto-RCA… · p. 8
• The best solution’s F1-score is 0.9540 on Simple, 0.9158 on Difficult, and 0.9179 on Mixed scenarios.On Difficult scenarios, Recall (0.9763) exceeds Precision (0.8920); the paper characterizes this as an aggressive diagnostic strategy with some false positives.Source: Table 2 · p. 8, RQ2: Effectiveness of the Auto-RCA… · p. 8
• Gemini-2.5-Pro’s 1M-token context window can retain more problem descriptions and interaction history than the 64K-token windows of Claude-Sonnet-4 and Qwen3-235B.The paper suggests this reduces truncation that may hinder code modifications.Source: RQ2: Effectiveness of the Auto-RCA… · p. 8

• TN-RCA530 provides a public, real-world standardized platform to measure and drive progress in telecom RCA, while Auto-RCA demonstrates a path to mastering the benchmark.Source: Conclusion · p. 8
• For complex, domain-specific reasoning, the paper argues that LLMs may perform better as components in autonomous agentic frameworks that iteratively refine verifiable solutions than through direct application.Source: Conclusion · p. 8
• Future work will extend TN-RCA530 with greater complexity and generalize Auto-RCA to other challenging graph-based problem domains.Source: Conclusion · p. 8