ShikumiMiner: Mining Recurring Implementation Patterns in AI Codebases

Afsana Tasnim; Sheikh Motahar Naim

Source-linked AI summary

ShikumiMiner: Mining Recurring Implementation Patterns in AI Codebases

Afsana Tasnim, Sheikh Motahar Naim

arXiv:2609.02789v1Sep 2, 2026cs.SE

TL;DR

Open-source LLM projects may contain recurring implementation patterns, but their structure in C++ local LLM codebases has been insufficiently characterized. ShikumiMiner combines AST and CFG evidence with multi-label classification to detect and compare these patterns across ten repositories, finding recurring patterns alongside substantial project variation and stronger discriminative signal from syntax.

Problem
Limited work has examined recurring implementation patterns in C++ local LLM codebases, motivating characterization of their structural and execution patterns.
Method
ShikumiMiner combines AST- and CFG-derived function-level features, study-specific pattern categories, weak labels, and multi-label Random Forest classification to profile repositories.
Results
Across ten repositories, ShikumiMiner identifies recurring patterns with substantial project variation, while AST-derived evidence provides the strongest discriminative signal.
Takeaways & Limitations
The framework provides a basis for more interpretable and reproducible comparison of local LLM implementations.
Takeaways & Limitations
The evaluation covers ten open-source C++ repositories and uses weak labels plus a single-annotator reference set, limiting generalization and annotation reliability.

Abstract

from arXiv · show

Large language models are paving the way towards innovation by understanding, analyzing, summarizing and generating content in the modern world. Currently there are thousands of LLM projects developed by engineers in open-source repositories. However, whether these LLM projects have underlying patterns or not remains a question. Exploring these underlying patterns will give new dimensions to the developers who aim to develop these LLM projects. In this paper, we propose ShikumiMiner, a static-analysis framework that combines Abstract Syntax Tree (AST) and Control Flow Graph (CFG) features to detect and compare recurring implementation patterns in C++ local LLM codebases. We analyze ten GitHub open-source repositories and classify functions into seven study-specific categories using a multi-label Random Forest model. Studying these patterns can provide useful insights for developers aiming to design LLM applications.

I. Introduction

ShikumiMiner addresses limited characterization of recurring patterns in C++ local LLM codebases by combining AST and CFG evidence for function-level analysis. It evaluates these patterns across ten open-source repositories and compares their structural and execution characteristics.

Research gap: Limited prior work characterizes recurring implementation patterns in C++ local LLM codebases using program representations.Existing studies examined ASTs, CFGs, data-flow graphs, and related representations mainly for broader code-analysis tasks.
Approach: ShikumiMiner combines syntactic and control-flow perspectives to capture code organization, execution paths, and structural dependencies.The framework is designed to address analyses that rely on only one representation.
Contribution: ShikumiMiner is an AST-CFG-based static-analysis framework that extracts function-level structural and control-flow features from C++ local LLM codebases.Its contribution is the application and integration of established AST and CFG representations for domain-specific pattern tracing.
Contribution: Recurring implementation structures are formalized as detectable pattern categories using combined syntactic and control-flow evidence.The categories are used to organize recurring implementation behavior in local LLM codebases.
Evaluation: ShikumiMiner is evaluated on ten open-source C++ local LLM repositories using function-level and project-level pattern distributions.The evaluation also examines project-level similarities and feature-pattern relationships.

A. Why Abstract Syntax Tree:

ASTs represent source-code syntax hierarchically, while CFGs represent possible execution paths; both provide established evidence for analyzing program structure and behavior. Prior work combines these representations in code analysis, but several approaches remain limited in deeper semantics or execution-level modeling.

AST: An AST is a hierarchical tree representing the logical syntactic structure of source code.AST-based analyses derive structural information from source code and support code representation tasks.
CFG: A CFG represents all possible execution paths through a program and supports analysis of how code behaves when it runs.CFGs have been used for source-code analysis, code-quality assessment, vulnerability detection, and function similarity.
Combined representations: Prior work combines AST and CFG representations because analyzing both can reveal important information about source code.Related approaches also combine lexical or syntactic information with program-dependence information.
Limitations: Deep learning approaches may encode code structure while remaining limited in capturing deeper semantics, control flow, data flow, or runtime behavior.CodeT5 mainly represents code as token sequences despite using AST-derived identifier labels, while other analyses focus mostly on syntax.

III. Proposed Methodology:

ShikumiMiner defines implementation patterns as recurring code-level structures associated with specific behaviors, using AST and CFG evidence to analyze C++ local LLM systems. It preprocesses source, extracts function-level syntactic and execution features, and fuses them into a unified representation.

Pattern definition: ShikumiMiner organizes recurring C++ local LLM implementation structures into seven study-specific pattern categories.The categories are Training Pipeline, Interactive Inference, Advanced Sampling, Model Loading and Validation, Memory Management, Context Management, and Quantization Workflow.
Source preprocessing: Source preprocessing excludes unrelated files so analysis focuses on core model, inference, memory, context, quantization, and deployment behavior.Figure 1 summarizes this workflow.
AST analysis: AST traversal extracts function-level syntactic features including declarations, control statements, calls, memory operations, and domain-specific LLM operations.The extracted evidence covers model loading, sampling, training, and quantization.
CFG analysis: CFG analysis captures execution-oriented features such as nodes, edges, branches, loops, exits, back edges, cyclomatic complexity, loop depth, and validation paths.These features characterize branching, iteration, and control-flow complexity.
Feature fusion: ShikumiMiner concatenates each function’s AST-based and CFG-based feature vectors to jointly examine syntactic and control-flow characteristics.The fused representation supports evaluation of whether the two evidence sources provide complementary information.

1) Weak-Label Generation:

ShikumiMiner treats pattern detection as multi-label classification because one function may express multiple implementation patterns. Domain-specific rules generate scalable weak labels from AST and CFG features, which supervise one binary classifier per category.

Weak-label generation: Pattern detection is multi-label because a single function may contain multiple implementation patterns.Model-loading and memory-management behavior can occur in the same function.
Weak-label generation: Domain-specific rules over extracted AST and CFG features generate weak labels for the seven implementation-pattern categories.Examples include optimizer or backward-propagation evidence for Training Pipeline and prompt-processing or tokenization evidence for another category.
Weak-label generation: Each weak label is set to 1 when its category rule is satisfied and 0 otherwise.The rule-based labels provide scalable supervision for corpus-wide classification experiments.
Multi-label classification: ShikumiMiner trains one binary Random Forest classifier per implementation-pattern category so multiple labels can be assigned to one function.The category-specific predictions are combined into a complete multi-label output.

3) Project-Level Pattern Profiling:

ShikumiMiner aggregates function-level predictions into repository profiles and evaluates pattern representation, cross-project generalization, and repository comparison across ten open-source C++ local LLM repositories. The setup combines ablation, LOPO, and manual-validation protocols.

Project-level profiling: Function-level pattern labels are aggregated by repository to construct profiles summarizing the seven patterns in each analyzed system.These profiles support comparisons across local LLM repositories.
Repository corpus: The ten repositories range from lightweight inference projects to production-oriented serving and deployment systems.The analyzed set includes llama.cpp, gemma.cpp, ONNX Runtime GenAI, OpenVINO GenAI, LMDeploy, gpt2.cpp, minchatgpt.cpp, InferLLM, SGLang, and DeepSpeed-FastGen.
Repository corpus: 11,257 function records are extracted across the ten repositories after retaining core source files and excluding documentation, tests, build scripts, and third-party dependencies.Each retained file is parsed with Clang LibTooling and represented using fused AST- and CFG-derived features.
Classification setup: The classification stage uses RandomForestClassifier with 800 trees and trains one binary classifier for each pattern category.Pattern-specific thresholds are selected from {0.02, 0.04, . . . , 0.80} by maximizing F1 on out-of-bag predictions.
Evaluation protocols: Evaluation compares AST-only, CFG-only, and fused AST+CFG representations, tests Leave-One-Project-Out generalization, and manually validates 250 sampled functions.The ablation uses 5-fold stratified cross-validation over 7,945 distinct vectors.

B. Research Questions:

The study asks whether combined AST and CFG evidence detects recurring patterns, how those patterns distribute across repositories, how well detection generalizes, and how project profiles compare. Results show distinct profiles with substantial project-level variation alongside recurring inference, loading, memory, and context patterns.

RQ1: Evidence integration: RQ1 evaluates whether fused AST+CFG representations affect recurring-pattern detection relative to AST-only and CFG-only representations.The comparison is conducted through an ablation study.
RQ2: Recurring patterns: RQ2 identifies recurring implementation patterns by analyzing function-level labels across the seven study-specific categories.The categories are examined for recurrence across analyzed repositories.
RQ3: Pattern distribution: RQ3 examines repository-level distribution by aggregating function-level pattern labels for each local LLM project.The resulting distributions are compared across repositories.
RQ4: Cross-project generalization: RQ4 measures generalization to previously unseen repositories using Leave-One-Project-Out evaluation.Each evaluation trains on nine repositories and tests on the remaining repository.
RQ5: Project similarity: RQ5 compares project-level implementation-pattern profiles using Jaccard similarity based on category presence.The analysis asks how similar local LLM projects are according to their detected profiles.
Results: Distinct implementation-pattern profiles emerge across the ten repositories, but their distributions vary substantially by project.Interactive Inference dominates most repositories, while InferLLM is dominated by Model Loading and Validation and DeepSpeed-FastGen by Memory Management.
Results: Local LLM systems share recurring inference, loading, memory-handling, and context-management patterns, while their relative emphasis differs with repository scope and implementation purpose.The normalized distribution is presented across the ten analyzed projects.

A. Manual Validation

Manual validation against independent source-derived labels shows moderate agreement, but performance differs substantially from weak-label evaluation and is constrained by limited annotation evidence.

58% of the 250 functions achieved complete agreement across all seven implementation-pattern categories.The 95% bootstrap confidence interval for micro F1 was [0.47, 0.60].
A second disjoint sample of 80 functions produced a macro F1 of 0.39 and micro F1 of 0.45.Pooling both samples (n = 330) yielded a micro F1 of 0.51 with a 95% confidence interval of [0.46, 0.57].
AST-only achieved a macro F1-score of 0.40 and micro F1-score of 0.46, slightly exceeding fused AST+CFG performance.The fused representation reached 0.39 macro F1 and 0.45 micro F1, while CFG-only reached 0.22 and 0.24.
CFG-only performed substantially worse for most individual categories, including Training Pipeline, Advanced Sampling, and Memory Management.Their F1-scores were 0.02, 0.12, and 0.22, respectively.
Broad control-flow properties provide limited discriminative power without syntactic context, while syntactic representation carries most aggregate discriminative information.The fused representation did not outperform AST-only under the current feature-concatenation strategy.

C. F1 Score:

ShikumiMiner evaluates implementation-pattern detection with precision, recall, and F1-score after constructing fused AST-CFG features through its static-analysis pipeline.

F1-score balances precision and recall for the multi-label categories, which have different frequencies.Precision measures correct predicted instances, while recall measures detected reference instances.
Fused AST+CFG features provide the per-pattern F1-score results summarized in Figure 3.Performance differs across patterns because the features are more useful for identifying some patterns than others.
The pipeline collects and filters repository files, constructs ASTs and CFGs, extracts and concatenates features, trains classifiers, predicts labels, and aggregates repository profiles.Algorithm 1 returns function-level predictions and project-level pattern profiles.
ShikumiMiner generates function-level predictions and aggregates detected labels into project-level implementation-pattern profiles.The profiles summarize occurrences of seven implementation patterns across analyzed repositories.

D. Leave-One-Project-Out Generalization:

Leave-One-Project-Out evaluation indicates limited cross-project generalization, while Jaccard analysis finds substantial overlap in pattern-category coverage across repositories.

D. Leave-One-Project-Out Generalization: 0.112 average Macro F1 and 0.167 average Micro F1 were achieved across ten held-out projects in LOPO validation.minchatgpt.cpp reached the highest Micro F1 of 0.318, followed by llama.cpp at 0.274.
D. Leave-One-Project-Out Generalization: Repository-specific AST- and CFG-level characteristics limit transfer to completely new repositories.The results suggest broader project variety and less project-dependent features may improve generalization.
E. Jaccard Index: Jaccard compares repositories by the presence or absence of detected implementation-pattern categories rather than their frequencies.Each repository is represented as a set of categories occurring at least once.
E. Jaccard Index: A Jaccard value of 1.0 indicates identical pattern-category coverage, whereas 0.0 indicates no shared detected categories.Higher values represent greater similarity in represented pattern types.
E. Jaccard Index: llama.cpp, gemma.cpp, LMDeploy, and DeepSpeed-FastGen each contain all seven categories and therefore have pairwise Jaccard similarity of 1.00.Four other repositories contain six categories, with Training Pipeline absent, and also have pairwise similarity of 1.00.
E. Jaccard Index: Jaccard similarity measures category-coverage similarity, not detailed pattern distributions or repository architecture.Repositories can score 1.00 despite substantial differences in relative pattern frequencies.

F. Spearman Correlation:

Spearman analysis tests whether structural and control-flow characteristics relate to the number of detected implementation patterns, finding only weak positive associations.

Spearman correlation treats each function as an observation and relates structural metrics to its number of detected pattern categories.The metrics include AST and CFG size, calls, loops, branches, and validation paths.
Function-call count has the strongest association at ρs = 0.30, while CFG edge count and node count each reach ρs = 0.27.The correlations are positive but weak across all ten analyzed features.
Cyclomatic complexity and CFG branch-node count each reach ρs = 0.26, and branch count reaches ρs = 0.25.AST depth is ρs = 0.23; loop count and CFG loop depth are both ρs = 0.21; validation-path count is ρs = 0.20.
No individual structural or control-flow metric strongly tracks detected pattern count, so structural size alone is not a strong proxy for pattern evidence.Larger and more complex functions receive slightly more pattern assignments, but the relationship remains weak.

VI. Discussion:

Across ten local LLM repositories, seven implementation patterns recur, but their prevalence varies by project and is driven more by syntactic and API-level conventions than distinctive control-flow structure. The evaluation also identifies limited cross-repository transfer, labeling and coverage constraints, and a basis for interpretable comparison.

Pattern Findings: Seven implementation patterns recur across all ten local LLM repositories, including inference, model loading, memory management, and context handling.Training and quantization appear more selectively, depending on repository purpose.
Pattern Findings: Syntactic evidence provides the strongest discrimination, while CFG features offer weaker discrimination and do not necessarily improve performance when concatenated.The findings indicate that control-flow information contributes mainly to execution-oriented characterization.
Threats to Validity: The taxonomy may miss implementation strategies because projects organize patterns differently, and rule-based weak labeling can introduce noise when evidence is incomplete or ambiguous.Pattern evidence may also be distributed across multiple functions.
Threats to Validity: Evaluation is limited to ten open-source C++ local LLM repositories, so findings may not generalize to industrial systems, other languages, or all LLM systems.The reference set contains 250 functions plus 80 held-out functions and was annotated by one annotator.
Threats to Validity: Repository size, category frequency, and unavailable CFGs may affect detected pattern counts, performance results, and repository-level coverage.Unavailable CFGs particularly affected repositories with CUDA extensions or specialized build dependencies.
Conclusion: ShikumiMiner provides a basis for more interpretable and reproducible comparison of local LLM implementations while revealing substantial variation across repositories.The study identifies recurring patterns through AST- and CFG-derived evidence.