Studying the Characteristics of AIOps Projects on GitHub

Research gap

• Prior work proposed AIOps solutions for operational tasks, but real-world AIOps practices had not been systematically studied.Source: §1 · p. 2

Contribution and motivation

• The study aims to help practitioners and researchers understand AIOps practice, choose solutions suited to application scenarios, and identify problems and research opportunities.Source: §1 · pp. 2–3

RQ1: GitHub metrics

• AIOps projects show faster growth and higher distributions of popularity metrics, pull requests, and issues than the baselines.The comparison covers GitHub metrics including programming languages and stars.Source: §1 · p. 2

RQ2: Inputs, techniques, and goals

• Monitoring data such as logs and performance metrics is the most used input, classical machine learning is the most adopted technique, and anomaly detection is a primary goal.Source: §1 · p. 3

RQ3: Code quality

• AIOps projects have higher rates of bugs, code smells, and technical debt than the baseline projects.Source: §1 · p. 3

2.1 Overview of our study

• The study gathers 119 AIOps projects through two rounds of keyword searching, manual verification, keyword expansion, and filtering.GitHub is the primary data source for the analysis.Source: §2.1 · p. 4
• The comparison uses randomly sampled machine-learning and general GitHub projects as baselines.Source: §2.1 · p. 4

2 Experiment Setup

• Quantitative and qualitative analyses address repository metrics, AIOps project goals, and code quality; machine-learning projects provide a baseline because many AIOps projects use ML techniques.Source: §2.1 · pp. 3–4

2.2 Collecting AIOps Projects

• The authors’ project selection began with 1,016 candidates collected through two rounds of AIOps-related searches and ended with 119 projects.Source: §2.2 · pp. 4–5, §2.2.6 · p. 7

2.2.1 Search AIOps projects (first round)

• The first-round GitHub search found 542 repositories labeled with “AIOps” across repository names, about sections, topics, and README contents.Source: §2.2.1 · p. 5

2.2.2 Manual verification (first round)

• Projects were screened for AIOps relevance, sufficient code, and non-toy status; manual verification retained 99 candidates from the initial 542.Two authors independently coded the repositories, discussed disagreements, revised decisions, and could refer unresolved conflicts to a third author.Source: §2.2.2 · pp. 5–6

2.2.3 Keyword extraction (pattern mining)

• Frequent-pattern mining of repository topics identified four keyword pairs used to broaden the search beyond projects explicitly labeled “AIOps.”The pairs combine anomaly detection, log analysis, machine learning, and metrics; the mining support threshold was 2.Source: §2.2.3 · p. 6

2.2.4 Search AIOps projects (second round)

• The second-round search using the four selected keyword pairs found 474 unique projects after duplicates were removed.Source: §2.2.4 · pp. 6–7

2.2.5 Filtering

• Projects in both search rounds were filtered for at least one star and one fork to exclude toy repositories while keeping the threshold low.The first-round verified set fell from 99 to 55 after filtering; filtering the second-round results before manual review reduced unnecessary coding effort.Source: §2.2.5 · p. 7

2.2.6 Manual verification (second round)

• The final AIOps set contains 119 projects, combining 55 identified with the “AIOps” keyword and 64 found through expanded keywords.The expanded search yielded 64 projects from 474 candidates.Source: §2.2.6 · p. 7

2.2.7 Measuring the reliability of our manual verification

• Cohen’s kappa rose from 0.68 before coder discussion to 0.84 afterward, indicating strong agreement after discussion.The paper uses Cohen’s kappa to measure agreement between two coders.Source: §2.2.7 · pp. 7–8

2.3.1 Machine Learning projects

• Machine-learning projects serve as a baseline because AIOps is an application domain of machine learning, and the search uses broad “machine learning” and “deep learning” keywords.Source: §2.3.1 · p. 8, §2.3 · p. 8
• The machine-learning baseline contains 383 projects sampled from a pool of 87,276 repositories using the AIOps project creation-date range.The sampling target was 383 repositories for a 95% confidence level and 5% confidence interval.Source: §2.3.1 · p. 9

2.3.2 General projects

• The general baseline covers 4,358,342 public repositories without restricting topic or programming language, with a target sample of 385 projects.The sample size corresponds to a 95% confidence level and a 5% confidence interval.Source: §2.3.2 · p. 9

2.3.3 Manual verification

• Baseline screening removed repositories lacking substantial code or representing homework and university projects; coders selected final sets of 383 ML and 385 general projects.Manual verification assessed a portion of each sample independently, with kappa scores of 0.81 for ML and 0.91 for general projects.Source: §2.3.3 · pp. 9–10

• The section organizes its research questions by motivation, approach, and results.Source: §3 · p. 10

3.1.1 Motivation

• The comparison addresses the lack of systematic study of real-world AIOps projects and aims to clarify their characteristics relative to two baselines.Source: §3.1.1 · p. 10

3.1.2 Approach

• The approach compares repository growth, primary programming languages, and repository metrics using GitHub REST API data and nonparametric statistical tests.The authors use Mann–Whitney U tests for distribution differences and Cliff’s delta for effect sizes.Source: §3.1.2 · pp. 10–12

3.1.3 Results

• AIOps projects grew faster than both baselines from 2017 to 2022; Python dominates at 71.4%, and Java is also used substantially at 10.1%.Source: Figure 2 · p. 11, Table 2 · p. 11, §3.1.3 · p. 12
• AIOps projects have 4 times the General baseline’s median stars (17 vs. 4) and forks (8 vs. 2), with statistically different distributions from both baselines.Source: Table 3 · p. 12, §3.1.3 · p. 14
• AIOps and baseline projects do not differ significantly in commits; contributor counts show a small effect versus ML and a negligible but significant effect versus General.Source: Table 3 · p. 12, §3.1.3 · p. 14
• AIOps projects have more pull requests and issues, and are larger than both baselines; their median size is 7.6 MB versus 1.0 MB for General projects.Source: Figure 3 · p. 13, §3.1.3 · p. 14
• Only 1.7% of AIOps projects are archived, compared with 1.8% of ML projects and 4.2% of General projects.Source: §3.1.3 · p. 14
• Overall, AIOps projects are more active and popular than both baselines, while growing faster in recent years and tending to be larger.Source: §3.1.3 · pp. 12, 14, 15

3.2.1 Motivation

• The qualitative analysis addresses uncertainty about how real AIOps projects combine data and technologies to achieve operational goals.The authors frame the analysis as a way to understand AIOps practice and inform future work across application scenarios.Source: §3.2.1 · p. 15

3.2.2 Approach

• The researchers manually code projects’ inputs, techniques, and goals using open coding, then discuss labels, revise coding, resolve disagreements, and organize categories.Two authors independently label 97 projects; unresolved disagreements go to a third author for a final decision.Source: §3.2.2 · pp. 15–17

3.2.3 Results

• Logs, performance metrics, and network traffic dominate AIOps inputs; monitoring data appears in 87% of projects, and logs are the most common subtype.Network traffic is used by 21% of projects and IoT sensor data by 8%; multimedia and codebase each account for 5%.Source: Figure 5 · p. 16, §3.2.3 · p. 18
• Classical machine learning is the most-used analysis category (54%), far ahead of deep learning (14%), statistical analysis (9%), time-series models (7%), and log parsing (7%).The categorization is shown in Figure 6.Source: Figure 6 · p. 16, §3.2.3 · p. 18
• Classical machine learning use skews supervised (59% versus 41%), while deep learning skews unsupervised (60% versus 40%); overall, 37% use supervised and 31% unsupervised learning.LSTM is the most-used unsupervised deep-learning algorithm, while ANN is the dominant supervised technique.Source: §3.2.3 · pp. 17–19
• For log handling, 64% of projects use customized parsers and 20% use Drain; ARIMA leads time-series methods, and LSTM is used unsupervised, primarily for anomaly detection.Only 2% use NLP approaches, with Word2vec and BERT among the most common techniques.Source: §3.2.3 · pp. 17, 19, 21
• Anomaly detection is the leading goal (60%), followed by monitoring (12%); detection commonly uses logs or network traffic, while prediction mainly uses performance metrics.Only 1% of projects have self-healing as their final goal.Source: §3.2.3 · pp. 22–23, Figure 8 · p. 22

3.3.1 Motivation

• The study motivates code-quality measurement as important because unreliable project code could make operational insights unreliable.The authors compare AIOps projects with ML and General baselines to identify quality patterns and differences.Source: §3.3.1 · p. 23

3.3.2 Approach

• The analysis used SonarQube across four dimensions—size, reliability, maintainability, and security—and measured metrics including LOC, bugs, code smells, vulnerabilities, and technical debt.Python notebooks were converted to .py files before analysis; 770 of 887 projects (87%) were analyzed, with failures mainly from removed repositories and Java compilation.Source: §3.3.2 · pp. 23–25

3.3.3 Results

• AIOps projects had more bugs, code smells, vulnerabilities, security hotspots, and technical debt overall, although vulnerability and security-hotspot differences from General projects were not statistically significant.Code smells accounted for 85% of AIOps issues; most projects had no vulnerabilities.Source: §3.3.3 · pp. 26, 28, 29, Figure 11 · p. 29
• Median bugs were 3 in AIOps projects versus 1 in both baselines, while median code smells were 51.0 versus 26.5 for ML and 14.0 for General projects.The higher counts also appear in LOC-normalized comparisons.Source: §3.3.3 · p. 26, Figure 9 · p. 27, Figure 10 · p. 28
• AIOps projects had median technical debt of 5.4 hours, compared with 2.9 hours for ML and 1.7 hours for General projects; the gap remained in LOC-normalized results.Mean technical debt was 80.5 hours for AIOps, 26.8 for ML, and 57.3 for General projects.Source: Figure 9 · p. 27, §3.3.3 · pp. 28–29, Figure 10 · p. 28
• Naming convention, unused code, and bad design were the top violated rule categories; 9 of the top 10 violated rules and categories were shared with ML projects.The authors suggest that shared violations make ML-system quality-assurance tools and techniques applicable to AIOps.Source: §3.3.3 · pp. 32–33

4 Discussion

• The discussion assesses findings against a stricter filter that retains more mature projects, alongside its review of AIOps challenges and future directions.Source: §4 · p. 33

4.1 AIOps: Where is it now?

• AIOps applications are growing faster than general-purpose GitHub projects and receive more attention than ML and general-purpose applications.The reported attention comparison draws on GitHub metrics.Source: §4.1 · p. 33
• Logs, performance metrics, and network-traffic data are the most common AIOps inputs, used by almost 70% of projects.The authors note that other input types, such as alarms, are used less often.Source: §4.1 · p. 34

Classical machine learning is the most common technique in AIOps

• Classical machine learning is the primary technique in 54% of AIOps applications, compared with 14% using deep learning.The authors recommend evaluating classical approaches rather than assuming deep learning is optimal, and caution against uninterpretable black-box models.Source: Classical machine learning is the most… · p. 34
• Anomaly detection is the primary goal for 60% of open-source AIOps applications; only 5% provide public datasets and 1% provide self-healing.Source: Classical machine learning is the most… · p. 34

4.2 AIOps: Challenges and future directions.

• Only 5% of projects aim to publish AIOps benchmark datasets, and the authors call for public real-world datasets using common inputs such as logs, performance metrics, and network traffic.They also plan a framework for generating customized datasets.Source: §4.2 · p. 35
• Only 2% of studied projects use NLP, although logs are an input in approximately one-third; the authors suggest exploring NLP for log analysis and modeling.Source: §4.2 · p. 35
• Only 1% of projects target self-healing, while anomaly detection, prediction, monitoring, and root-cause analysis require human action after identifying an incident.The authors encourage integrating self-healing with detection and prediction.Source: §4.2 · p. 35
• AIOps and ML projects share 90% of their most violated rules and rule categories, so existing ML quality-assurance tools may also be used for AIOps.Naming conventions are among the easier issues to address; cognitive complexity and duplicated strings can require structural changes.Source: §4.2 · pp. 35–36

Open-source AIOps solutions can benefit from broader use-cases

• Anomaly detection accounts for 60% of AIOps use cases, and the authors call for broader attention to other scenarios, including reducing tedious engineering work and improving automation.Source: Open-source AIOps solutions can… · p. 36

4.3 The impact of using a stronger filtering criterion.

• The stricter criterion retained projects with at least 5 stars, reducing the AIOps set from 119 to 85, ML from 383 to 215, and General from 385 to 183.The threshold was selected using the elbow point in the sorted AIOps star distribution.Source: §4.3 · p. 37
• Under the stricter filter, AIOps projects continued to grow faster than the ML and General baselines.This matched the trend in the initial results.Source: Figure 12 · p. 37, §4.3 · p. 38
• With stricter project filtering, AIOps projects had a median of 2 pull requests, versus 0 for ML and 1 for General; the difference from General was not significant.Source: §4.3 · p. 39
• With the stricter filter, AIOps projects still had more bugs, code smells, and technical debt than both baselines, and the authors report that the initial findings remained robust.Median bugs were 3 versus 1 in ML and 0 in General; median code smells were 57 versus 27 and 9, respectively.Source: §4.3 · pp. 39–41, Figure 14 · p. 40
• The stricter-filter findings again showed similarity between AIOps and ML quality issues, with no change to AIOps’s top-five violated rules and categories from the initial analysis.The rules and categories retained similar weights and project usage.Source: §4.3 · p. 40, Table 16 · p. 41

External validity

• The sample may omit AIOps projects outside public GitHub, including projects that do not use the relatively new “AIOps” term.The authors recommend examining other sources, such as published studies and closed-source projects, to complement the results.Source: §5 · p. 41
• The selection threshold balances filtering low-bar repositories against retaining enough projects, and sensitivity analysis found results consistent under stricter filtering.Repositories needed at least one star and one fork; stricter filtering would substantially reduce the available AIOps sample.Source: §5 · pp. 41–42
• The machine-learning baseline may miss repositories because it was collected using only “machine learning” and “deep learning” keywords.The authors suggest expanding to more diverse keywords.Source: §5 · p. 42

Internal validity

• SonarQube metrics may not fully represent project quality, and the tool may produce false positives or false negatives.A manual check of 200 issues sampled across the 10 most violated rules found no false positives in the analyzed instances.Source: §5 · p. 42
• Qualitative categorization may reflect personal deductions, though two authors followed a five-step process and achieved Cohen’s kappa of 0.84.The paper characterizes this agreement as strong and reliable.Source: §5 · p. 42

6.1 AIOps solutions.

• Prior AIOps work covers anomaly detection, failure prediction, root-cause analysis, and incident management, while surveys report growing research interest and recurring challenges.The reviewed approaches use varied data sources and methods, including logs, performance metrics, traces, and machine-learning models.Source: §6.1 · pp. 43–44
• Earlier anomaly-detection studies use logs, performance metrics, network traffic, or traces, while failure-prediction work usually relies on system performance metrics.The reviewed work also includes root-cause analysis using alarms or combined event data, and incident-management methods based on incident reports and detection frameworks.Source: §6.1 · pp. 43–44
• Despite prior AIOps solution research, the authors find no existing study of real-world AIOps projects and conduct an empirical study of GitHub projects.Source: §6.1 · p. 45

6.2 Characterizing GitHub projects.

• Prior GitHub studies characterize repositories and developers in other domains, but the authors describe their work as the first GitHub-based characterization of AIOps projects.Source: §6.2 · p. 45

6.3 Analyzing code quality using SonarQube.

• Earlier SonarQube studies examined smaller sets of applications or Apache systems, whereas this paper analyzes code quality across 6,101 projects.The authors argue that the larger project set makes their results more confident and robust.Source: §6.3 · p. 45

7 Conclusion

• AIOps projects on GitHub are relatively new and growing rapidly, while their code quality is poorer than that of the two baselines.The study also identifies common inputs, techniques, and goals of AIOps solutions.Source: §7 · p. 46
• The authors identify self-healing and ad-hoc coding as weak aspects that future efforts could address to improve AIOps quality.Source: §7 · p. 46
• Proposed extensions include studying industrial AIOps solutions, automating or augmenting solution development, applying machine-learning quality-assurance techniques, and interviewing practitioners and researchers.Source: §7 · p. 46
• The study’s datasets, code, and analysis are publicly available in a GitHub replication package.Source: §7 · p. 46