Source-linked AI summary
OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models
Yuhe Liu, Changhua Pei, Longlong Xu, Bohan Chen, Mingze Sun, Zhirui Zhang, Yongqian Sun, Shenglin Zhang, Kun Wang, Haiming Zhang, Jianhui Li, Gaogang Xie, Xidao Wen, Xiaohui Nie, Minghua Ma, Dan Pei
TL;DR
Evidence on LLM performance in IT operations is limited, despite their potential for Ops tasks. OpsEval builds a bilingual, multi-task benchmark and evaluates models with varied prompting techniques and an expert-aligned QA metric. Its results offer practical observations for model-selection decisions, while the evaluation metric correlates 0.9175 with expert assessments.
Problem
No benchmark evaluates LLMs on Ops tasks, and assessment is complicated by sensitive data, task diversity, prompt sensitivity, and QA metrics that may miss meaning and evidence.
Source: §1 · p. 2Method
OpsEval assembles a bilingual Ops benchmark from community and other sources, covers multiple tasks and abilities, and evaluates models with prompting techniques and an expert-aligned QA metric.
Source: §1 · p. 2Results
The FAE-Score correlation with expert assessments is 0.9175, compared with 0.6705 for BLEU and -0.3957 for ROUGE.
Source: §1 · p. 2Takeaways & Limitations
The benchmark’s observations help practitioners assess model suitability for an Ops sub-domain, the need for fine-tuning, and whether quantization compromises effectiveness.
Source: §1 · p. 2Evaluation on specific datasets and Ops contexts may not generalize to other LLM deployment environments.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
Information Technology (IT) Operations (Ops), particularly Artificial Intelligence for IT Operations (AIOps), is the guarantee for maintaining the orderly and stable operation of existing information systems. According to Gartner's prediction, the use of AI technology for automated IT operations has become a new trend. Large language models (LLMs) that have exhibited remarkable capabilities in NLP-related tasks, are showing great potential in the field of AIOps, such as in aspects of root cause analysis of failures, generation of operations and maintenance scripts, and summarizing of alert information. Nevertheless, the performance of current LLMs in Ops tasks is yet to be determined. In this paper, we present OpsEval, a comprehensive task-oriented Ops benchmark designed for LLMs. For the first time, OpsEval assesses LLMs' proficiency in various crucial scenarios at different ability levels. The benchmark includes 7184 multi-choice questions and 1736 question-answering (QA) formats in English and Chinese. By conducting a comprehensive performance evaluation of the current leading large language models, we show how various LLM techniques can affect the performance of Ops, and discussed findings related to various topics, including model quantification, QA evaluation, and hallucination issues. To ensure the credibility of our evaluation, we invite dozens of domain experts to manually review our questions. At the same time, we have open-sourced 20% of the test QA to assist current researchers in preliminary evaluations of their OpsLLM models. The remaining 80% of the data, which is not disclosed, is used to eliminate the issue of the test set leakage. Additionally, we have constructed an online leaderboard that is updated in real-time and will continue to be updated, ensuring that any newly emerging LLMs will be evaluated promptly. Both our dataset and leaderboard have been made public.
1 Introduction
OpsEval addresses the lack of a benchmark for evaluating LLMs on IT operations tasks, where data scarcity, task diversity, prompt sensitivity, and QA metrics complicate assessment. It combines a bilingual multi-task dataset with evaluations of prompting techniques and an expert-aligned QA metric, and reports practical observations for model-selection decisions.
1 Introduction
- Ops lacks an LLM benchmark, while sensitive data, numerous sub-domains and tasks, prompt sensitivity, and QA metrics that miss meaning and evidence complicate evaluation.The authors note that BLEU-style linguistic similarity may not capture key points or sufficient evidence in Ops answers.Source: §1 · p. 2
- OpsEval draws on an industry community and nine representative sub-domains, then annotates tasks and abilities and tests self-consistency, chain-of-thought, and few-shot in-context learning.Its FAE-Score assesses fluency, accuracy, and evidence.Source: §1 · p. 2
- OpsEval is a bilingual multi-task dataset in the Ops domain, covering 8 tasks and 3 abilities with 9,070 questions; 20% of QAs are released, while 80% remain undisclosed to prevent unfair evaluations due to data leakage [28].Source: §1 · p. 2
- In the reported evaluation, FAE-Score showed stronger congruence with human expert assessments, with correlation 0.9175 versus 0.6705 for BLEU and -0.3957 for ROUGE.Source: §1 · p. 2
- The evaluation offers practitioners observations for deciding whether models suit particular Ops sub-domains, whether fine-tuning is needed, and whether quantization compromises effectiveness.The paper frames these as practical lessons drawn from its benchmark results.Source: §1 · p. 2
2 OpsEval Benchmark
OpsEval builds an IT operations benchmark from multiple sources, preprocesses and categorizes its questions, and evaluates language models using varied prompting techniques and a dedicated QA metric.
2 OpsEval Benchmark
- OpsEval combines multi-source data collection, preprocessing, and evaluation with varied prompt-engineering techniques.The framework describes these stages as the path from collected data to model evaluation.Source: §2 · p. 2
2.1 Data Collection
- Company materials, certification exams, and operations textbooks supply questions, with partner companies contributing production materials and internal training documents.The company materials include operations tickets, error logs, internal documents, and staff-training tests.Source: §2.1 · pp. 2–3
2.2 Preprocessing
- Preprocessing removes similar or context-dependent questions, categorizes questions into eight tasks and three abilities, and uses expert review to inspect automated filtering and classification.Deduplication flags question stems with cosine similarity above 0.7; industry experts review duplicates, invalid items, and GPT-4 classifications.Source: §2.2 · pp. 3–4
2.3 Evaluation Settings
- Multiple-choice answers are scored by accuracy, while QA responses receive FAE-Score and are also compared with expert ratings, BLEU, ROUGE, and RAGAS.The evaluation varies zero-shot and 3-shot prompts across naive, self-consistency, chain-of-thought, and combined chain-of-thought/self-consistency settings.Source: §2.3 · pp. 4–5
2.4 FAE-Score
- FAE-Score evaluates QA fluency, keyword-based accuracy, and evidence overlap with related retrieved documents.Fluency considers grammar, coherence, clarity, style, and completeness; accuracy uses keyword matching and F1, while evidence uses ROUGE recall against retrieved documents.Source: §2.4 · p. 5
2.5 Open-source Policy
- The released subset contains 20% of the dataset, sampled across sources and sub-domains, while the complete dataset remains undisclosed.Proprietary content in the sample was reviewed to remove sensitive information; the paper states that withholding the rest protects against answer leakage.Source: §2.5 · p. 6
3 Result Analysis
OpsEval results show that strong performance on general-domain evaluations does not necessarily transfer to specialized Ops tasks, and prompt and quantization choices also affect results. The analysis highlights particular weaknesses in analytical reasoning and the limits of conventional QA metrics.
3.1 Overall Performance
- Closed-source GPT-4 and Claude-3-Opus perform well on OpsEval, while open-source LLMs generally score worse than on MMLU and CEval.Few-shot and chain-of-thought prompting can improve performance when models are tuned for these techniques; smaller, weaker models are less stable with advanced prompts.Source: §3.1 · p. 6
- Model selection should balance average performance with robustness across prompt settings.The paper reports substantial prompt sensitivity and notes that simpler prompts work better for smaller, weaker models.Source: §3.1 · p. 6
3.2 Performance on Different Tasks and Abilities
- LLMs perform better on General Knowledge than on specialized tasks, where accuracy drops and varies; 10B–30B models sometimes exceed models with at most 10B parameters.Performance also varies substantially among 10B–20B models, and Figure 6 summarizes results by parameter-size group.Source: §3.2 · p. 6, Figure 6 · p. 7
- Analytical Thinking is weakest for Automation Scripts and across the three major tasks, while 5G communication and databases are identified for further pretraining and fine-tuning.The authors recommend inference-related supervised fine-tuning and connecting specialized domain knowledge for Ops-agent capability.Source: §3.2 · p. 6
3.3 Performance on Question-Answering
- BLEU and ROUGE can reward repeated question patterns without capturing semantic importance, whereas FAE-Score evaluates fluency, accuracy, and supporting evidence.The authors consider FAE-Score suitable for large-scale qualitative evaluation in Ops.Source: §3.3 · p. 7
3.4 Performance on Different Quantization parameters
- 3.50% and 0.27%: GPTQ 4-bit quantization is lower than LLaMA-2-70B on English multi-choice accuracy in zero-shot and few-shot evaluation, respectively.For Chinese questions, LLaMA2-70B-Int4 is 3.67% lower in zero-shot and 5.18% lower in few-shot evaluation; the paper reports INT4 declines below 10% overall and says quantization above 3 bits can preserve performance while reducing costs.Source: Figure 9 · p. 8, §3.4 · p. 8
- 12.46% and 9.30%: LLaMA2-70B-Int3 has average English accuracy degradation versus LLaMA-2-70B and LLaMA2-70B-Int4, respectively.The authors describe the INT3 performance drop as significant and advise careful consideration in practical applications.Source: §3.4 · p. 8
4 Validation
The validation evaluates possible test-set leakage using a loss comparison and compares automated QA metrics with expert judgments. The authors report no detected OpsEval test leakage and closer expert alignment for FAE-Score than for BLEU or ROUGE.
4.1 Benchmark Leakage Test
- A negative ΔL, defined as test-split loss minus reference-set loss, indicates potential leakage; the authors report that OpsEval’s test set is unbiased and non-leaking.The reference set consists of GPT-4 rewrites that preserve the original questions’ meanings.Source: §4.1 · pp. 8–9
4.2 Expert alignment of FAE-Score
- FAE-Score rankings closely match expert evaluations, particularly for accuracy, while BLEU and ROUGE often misalign and RAGAS retains a gap.The validation results are presented in Table 5b.Source: Table 5 · p. 8, §4.2 · p. 9
5 Discussion
The discussion describes task-specific templates for automated QA generation and identifies limits related to deployment choices, dataset scope, privacy, model capability, and sub-domain balance. It also outlines planned dataset and scenario expansion.
5.1 Automated QA generation
- Task-specific templates have GPT fill fields using supplied operational knowledge instead of generating whole questions and answers.The authors say this keeps QA structure consistent while supporting detailed, context-specific Ops tasks; Figures 11 and 12 show the prompt template and examples.Source: §5.1 · p. 9, Figure 11 · p. 9, Figure 12 · p. 9
5.2 Threats to Validity
- Deployment parameters and prompts may affect results, and evaluation on selected datasets and Ops contexts may not generalize to other deployment environments.The authors note that some prompt optimizations may not fully reflect real-world settings.Source: §5.2 · p. 9
5.3 Future Work
- Privacy constraints limit real-world company data, agent and RAG inclusion is constrained by current models’ operational knowledge, and sub-domain distribution remains a balancing focus.The authors plan to expand datasets, scenarios, and deployment settings as well as pursue more real-world scenarios and a more balanced distribution.Source: §5.3 · p. 10
6 Related Works
The related-work section distinguishes general-ability benchmarks from domain-specific benchmarks and reviews examples of each. It notes gaps in existing IT-operations benchmarks, including limited scope and evaluation or leaderboard features.
General ability benchmarks
- General-ability benchmarks assess capabilities across tasks; examples include MMLU, HELM, BIG-bench, SEAL, and C-Eval.The cited benchmarks cover varying numbers of subjects, scenarios, and tasks, with some emphasizing expert assessment or Chinese knowledge and reasoning.Source: §6 · p. 10
Domain-specific benchmarks
- Domain-specific benchmarks assess field knowledge and task performance, but NetOps is limited to wired network operations and OWL lacks a real-time leaderboard and designed IT-operations QA evaluation.The section also cites FLUE, MultiMedQA, and CMB as finance- and medical-domain examples.Source: §6 · p. 10
7 Conclusion
OpsEval provides a broad benchmark for evaluating LLMs in IT operations, with partial data release intended to support initial evaluation while protecting dataset integrity. Its findings highlight QA evaluation with FAE-Score, model performance and robustness, and opportunities to extend the benchmark.
7 Conclusion
- OpsEval comprises 9,070 questions across multiple Ops sub-domains and tasks, with partial release intended to preserve dataset integrity.The benchmark underwent data leakage detection experiments.Source: §7 · p. 10
- FAE-Score emerges as a more reliable QA evaluation metric than traditional metrics, with potential to replace manual labeling in large-scale quantitative evaluations.Source: §7 · p. 10
- The paper’s observations support balancing fundamental-model selection across performance and robustness.Source: §7 · p. 10
- Failure-rate analysis across eight tasks and three abilities offers researchers insights and prospects for future work.Source: §7 · p. 10
- OpsEval’s adaptability allows integration of additional fine-grained tasks and supports continued research and optimization of Ops-tailored LLMs.Source: §7 · p. 10