Source-linked AI summary
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge
Rishiraj Sengupta, Sotiris Chatzimiltis, Mohammad Shojafar, Xiatian Zhu
TL;DR
The paper asks whether lightweight, edge-deployable LLMs can perform open-ended 5G specification and fault-analysis tasks and whether their outputs can be evaluated reliably at scale. It evaluates three models across three free-text benchmarks using three independent frontier judges and agreement analysis. Fault diagnosis reaches at least 90% accuracy for all models, specification recall remains below 60%, and Gemini-3.1-Flash-Lite provides the strongest efficiency trade-off.
Problem
Lightweight LLMs must be assessed on 5G specification knowledge and free-text fault analysis, but open-ended responses lack fixed answer keys and deployment also depends on cost and latency.
Method
Three lightweight LLMs are evaluated across three free-text telecom benchmarks, with three independent frontier judges scoring responses and inter-judge agreement measured alongside deployment metrics.
Results
All three models reach at least 90% accuracy on fault diagnosis, while all score below 60% on zero-shot 3GPP and O-RAN specification recall.
Takeaways & Limitations
Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest cost and latency for production telecom deployment.
Takeaways & Limitations
Inter-judge agreement establishes consistency among judges rather than ground-truth correctness, motivating human-expert validation.
Abstract
from arXiv · showhide
Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations and recommended actions. LLMs have emerged as a promising approach to automating this, yet whether lightweight, edge-deployable models are capable of performing in-depth free-text diagnostics remains an open question. While existing benchmarks rely on restrictive MCQs with fixed answer keys, this paper evaluates 5G domain understanding and fault analysis in a free-text generation format. Transitioning to this paradigm requires evaluating lightweight, edge-deployable AI models on open-ended diagnostic reasoning, alongside a dependable framework to validate these text outputs at scale. To address this we evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORAN FT, 5G-Faults FT, and TeleInter FT. Three independent frontier judges score outputs, and pairwise inter-judge agreement is measured as an empirical test of the LLM-as-Judge methodology. All three models reach at least 90% accuracy on fault diagnosis, while zero-shot recall of 3GPP and O-RAN specifications remains the critical gap, with all models scoring below 60%. Mean inter-judge agreement is at least 0.90 across all runs, indicating that multi-judge LLM scoring produces consistent, reproducible grades for open-ended telecom responses. Operationally, Gemini-3.1-Flash-Lite offers the best efficiency trade-off, combining competitive accuracy with the lowest inference cost and latency, making it the most suitable candidate for production telecom deployments.
I. INTRODUCTION
The paper targets lightweight LLMs for open-ended 5G specification knowledge and fault analysis, while addressing how to evaluate free-text responses reliably at scale. It introduces free-text benchmarks, multi-judge scoring, and deployment-oriented metrics for this assessment.
- Lightweight LLM adoption in 5G requires evaluating specification knowledge and fault analysis alongside energy consumption, cost, latency, and privacy constraints.
- Free-text fault analysis requires explanations, diagnostic steps, and recommended actions that cannot be scored against fixed multiple-choice answer keys.
- Three lightweight models are evaluated across TeleQNA ORAN FT, 5G-Faults FT, and TeleInter FT in an open-ended telecom setting.
- Inference cost, latency, and token usage are reported alongside correctness as indicators of edge-deployment viability.
- Three independent frontier judges score responses, with inter-judge agreement used to test LLM-as-Judge reliability for telecom outputs.
II. RELATED WORK
Prior telecom benchmarks and operational studies largely use multiple-choice or specialized network tasks, while this work combines free-text evaluation with a telecom-specific multi-judge framework. The experimental pipeline operationalizes this approach through solver and scorer stages and reproducible evaluation runs.
- Related work: Telecom knowledge benchmarks such as TeleQNA and ORAN-Bench-13K use multiple-choice questions, while domain-adapted smaller models have challenged frontier-model assumptions.
- Related work: Operational studies address troubleshooting, annotated 5G fault analysis, and RAN-level inference, but do not evaluate free-text telecom responses.
- Evaluation framework: The framework passes datasets through a solver pipeline, scores responses with three independent judges, and computes accuracy, agreement, token, latency, and cost metrics.
- Research gap: No prior study applies LLM-as-Judge to telecom or measures inter-judge agreement on telecom free-text outputs.
- Experimental setup: Nine experiments evaluate each student-model and dataset combination using 100 samples, with complete inputs, outputs, scores, and token usage persisted in compressed logs.
B. Student Models
The study uses three commercially available, cost-optimized and low-latency student models selected as practical edge-deployment candidates. Each generates a structured free-text response directly from a zero-shot prompt without retrieval, tools, or additional inference-time computation.
- Student models: Three student models from Anthropic, OpenAI, and Google represent cost-optimized, low-latency offerings for potential edge deployment.
- Prompting: Each model receives an expert telecom system prompt followed by the task question.
- Prompting: Responses must separate a two-line Student Explanation from a one-line Student Answer.
- Prompting: The response structure lets judges assess factual correctness from the final answer while retaining the explanation for qualitative inspection.
- Inference setting: Student models operate zero-shot without few-shot examples, retrieval augmentation, or tool use.
D. LLM-as-Judge Evaluation
Because open-ended telecom answers lack fixed text targets, the paper uses three independent frontier judges to assess semantic correctness against reference answers. The benchmarks cover specification recall, fault diagnosis, and applied interpretation across 100-question samples.
- Evaluation rationale: String-matching metrics are unsuitable for generative responses without fixed ground truth, motivating LLM-as-Judge evaluation.
- Judge protocol: Three independent judges assign binary Correct or Incorrect grades and enable cross-provider agreement analysis.
- Judge protocol: The judges receive the question, student explanation, student answer, and reference answer, using the latter as a semantic guide.
- Benchmarks: TeleQNA ORAN FT tests precise recall of 3GPP and O-RAN standards, protocols, interfaces, and architecture.
- Benchmarks: 5G-Faults FT evaluates identifying fault causes, affected components, and recommended remediation from operational-log scenarios.
- Benchmarks: TeleInter FT evaluates interpretation of 5G concepts, configurations, and technical details across architecture and protocol layers.
F. Metrics
The evaluation reports accuracy, inter-judge agreement, token usage, latency, and API cost for each student-model and dataset pair. Accuracy and agreement are defined from binary judge grades across 100 samples and three judges.
- Metric Scope: The evaluation metrics are collected for every student-model and dataset combination across the study.The datasets are represented in the evaluation-dataset table.
- Accuracy: Accuracy assigns each sample a binary Correct or Incorrect score from each judge.The per-sample score is 1 for Correct and 0 otherwise.
- Accuracy: Mean accuracy aggregates per-judge accuracy across three judge models and 100 samples.N = 100 samples per run and J = 3 judge models.
- Inter-Judge Agreement: Inter-judge agreement measures whether two judges assign the same binary grade to a sample.The framework computes three pairwise agreement values per run across the judge pairs.
- Token Usage: Token usage totals reasoning and completion tokens for each evaluation run.For each sample, token counts are tracked by token type k.
- Operational Metrics: Generation latency is the average API response time per sample, while API cost is reported in USD for each model-dataset pair.Cost is extracted from OpenRouter activity logs and matched by model identifier and API key.
IV. RESULTS AND DISCUSSION
The results section compares the three student models across three benchmarks using accuracy, agreement, cost, latency, and token usage. Mean accuracy is summarized per student model.
- Results Overview: The evaluation results cover three benchmarks and three student models across accuracy, agreement, cost, latency, and token usage.The section presents and discusses these metrics across the benchmark-model combinations.
- Accuracy: Mean accuracy is summarized per student model in Table IV.The table reports accuracy as a percentage.
A. Accuracy
GPT-5.4-Mini achieves the highest overall mean accuracy, while benchmark results show that open-ended 3GPP and O-RAN specification recall remains substantially harder than fault diagnosis.
- 82.1% overall mean accuracy is achieved by GPT-5.4-Mini, followed by Gemini-3.1-Flash-Lite at 78.1% and Claude-Haiku-4.5 at 71.0%.
- All three models score below 60% on TeleQNA ORAN FT, making it the most challenging benchmark for zero-shot 3GPP and O-RAN specification recall.The results suggest retrieval augmentation or domain fine-tuning is needed before deployment.
- 98.3% accuracy is reached by GPT-5.4-Mini on 5G-Faults FT, the highest score reported across the benchmarks.This suggests 5G fault scenarios are more accessible to lightweight models than open-ended specification knowledge.
- TeleInter FT produces intermediate results, with Claude-Haiku-4.5 at 76.0%, 15 percentage points behind the leading models.
B. Inter-Judge Agreement
Inter-judge agreement is consistently high, supporting the reliability of the multi-judge evaluation while also revealing a provider-level grading divergence on some fault-analysis responses.
- 0.95–0.99 agreement is achieved by the two Gemini judges in every case, while TeleQNA ORAN FT shows cross-provider agreement of 0.930–0.970.The balanced cross-provider agreement supports interpreting TeleQNA ORAN FT’s low accuracy as a genuine model limitation rather than evaluator disagreement.
- 0.860 pairwise agreement occurs between GPT-5.5 and each Gemini judge on 5G-Faults FT for Claude-Haiku-4.5, compared with 0.980 between the Gemini judges.GPT-5.5 grades some Claude responses Incorrect while both Gemini judges accept them as Correct.
C. API Cost
Gemini-3.1-Flash-Lite has the most favorable operational profile, combining the lowest inference cost and latency with concise output, while Claude-Haiku-4.5 is consistently more expensive and slower.
- Below $0.10 per 100-sample run across all datasets, Gemini-3.1-Flash-Lite is consistently the most cost-efficient student model.Claude-Haiku-4.5 exceeds $0.42 in every case, while GPT-5.4-Mini sits between the two.
- Approximately 4.5x cheaper than Claude-Haiku-4.5 and 2.5x cheaper than GPT-5.4-Mini, Gemini-3.1-Flash-Lite offers the strongest accuracy-to-cost trade-off.Gemini-3.1-Flash-Lite achieves 78.1% overall accuracy, compared with GPT-5.4-Mini’s 82.1%.
- The lowest completion token volumes across all datasets are produced by Gemini-3.1-Flash-Lite, contributing directly to its latency and cost advantage.
- 3.4s–4.3s mean generation latency is achieved by Gemini-3.1-Flash-Lite in every benchmark, compared with Claude-Haiku-4.5 peaking at 12.3s on TeleInter FT.GPT-5.4-Mini reaches its highest latency of 9.1s on TeleQNA ORAN FT.
V. CONCLUSIONS
Across three telecom benchmarks, lightweight models performed strongly on fault diagnosis and network interpretation, while specification recall remained the critical gap. Multi-judge scoring was stable, but human-expert validation and improved specification handling remain future priorities.
- At least 90% accuracy was achieved on 5G fault diagnosis across all three lightweight models, with two models also reaching 90% on network interpretation.
- Below 60% zero-shot recall of 3GPP and O-RAN specifications remained the critical gap for all models.
- Inter-judge agreement ranged from 0.90–0.98 across all nine runs, indicating stable and reproducible grading of open-ended telecom responses.
- The reported judge agreement establishes consistency among judges rather than ground-truth correctness, motivating human-expert validation.
- Future work targets human-expert validation, retrieval-augmented or fine-tuned specification handling, autonomous OAM thresholds, and Agentic AI frameworks.