PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeGraph: Open-Taxonomy Knowledge Graph for Source Code with Wikidata Grounding

Federico Pennino, Andrea Gurioli, Stefano Zacchiroli, Maurizio Gabbrielli, Paolo Ferragina

arXiv:2609.29474v1cs.SEcs.CLcs.IR

TL;DR

Billions of source files contain engineering knowledge that existing syntactic and token-level tools do not organize at scale. CodeGraph extracts open-taxonomy concepts, grounds them in Wikidata, and applies calibrated quality assurance; on Stack-Edu it produces a graph with ∼158 M nodes and ∼1.02 B edges across 14 languages.

  • Problem

    Existing tools are largely syntactic or token-level, leaving algorithms, paradigms, design patterns, and application domains in large code archives difficult to organize at scale.

    Source: §1 · p. 1

  • Method

    CodeGraph combines open-taxonomy LLM extraction, three-stage Wikidata grounding, and calibrated quality assurance in a typed property graph.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    ∼158 M nodes and ∼1.02 B edges across 14 languages; the graph reveals algorithmic primitives recurring across substantively different application domains.

    Source: §1 · p. 2, §9 · p. 10

  • Takeaways & Limitations

    The graph supports choosing precision-recall operating points for annotation and exposes cross-domain semantic patterns that token-level retrieval cannot expose.

    Source: §9 · p. 10

    Paradigm and design-pattern annotations are more difficult than domain and algorithm annotations, so the authors release them as exploratory signals rather than quantitatively reliable labels.

    Source: §8 · p. 10

Abstract

from arXiv · show

Public software repositories, like GitHub and Software Heritage Archive, store billions of files, yet extracting their implicit engineering knowledge ---i.e., the algorithms they implement, the paradigms they follow, the patterns they instantiate, and the application domains they serve--- remains challenging, as current tools are constrained to syntactic and token-level analysis. We present a pipeline for building an open-taxonomy semantic annotation of source code using a code-specialised Large Language Model. The extracted entities are grounded in Wikidata through a three-stage linking procedure: a deterministic SPARQL stage handles unambiguous entities, a Deep Research Agent resolves the residual long tail, and a hierarchy-rollup stage imports the parent-of closure of each resolved Wikidata identifier. The resulting annotations are materialised as a source-code-specific open-taxonomy knowledge graph. We further introduce a calibrated quality-assurance protocol that quantifies annotation precision by combining a small human gold set with an LLM-as-a-judge filter. We applied our pipeline to the 167 million files of the Stack-Edu corpus, creating the first known large-scale open-taxonomy knowledge graph for source code. Our graph, named CodeGraph, contains approximately 158 million nodes, which include around 145 million files, about 63,000 extracted concept entities (such as algorithms, paradigms, design patterns, and application domains), and roughly 19,800 grounded Wikidata entities. Furthermore, CodeGraph features approximately 1 billion typed edges that connect files to their respective concepts, link these concepts to their grounded Wikidata identifiers, and relate them to their parent categories, covering 14 programming languages.

1 Introduction

Billions of code files contain engineering knowledge that remains difficult to navigate because existing tools and resources expose mostly syntactic, token-level, or task-specific information. CodeGraph combines open-taxonomy semantic extraction, Wikidata grounding, and calibrated quality assurance to create a large-scale semantic graph of source code.

1 Introduction

  • The gap is a methodology that exposes algorithms, paradigms, patterns, and application domains across hundreds of millions of source files.Static analysis is largely syntactic, supervised learning needs impractically large hand-labelled datasets, and code LLMs have mostly been used assistively.Source: §1 · pp. 1–2
  • The pipeline extracts four open-taxonomy concept types from whole source files and materialises them with files as a typed property graph.The axes are application domains, algorithms, programming paradigms, and design patterns; free-form labels are reconciled after extraction.Source: §1 · p. 2
  • Wikidata grounding uses deterministic SPARQL for unambiguous concepts, a Deep Research Agent for ambiguous or empty results, and hierarchy rollup for resolved QIDs.The resulting dual conceptual layer links a local vocabulary to a global taxonomy.Source: §1 · p. 2
  • A calibrated quality-assurance protocol combines a small human gold set, verifier selection, and an LLM-as-a-judge filter to estimate annotation precision at corpus scale.The paper also presents semantic code search and automated algorithm selection as motivating applications.Source: §1 · p. 2
  • CodeGraph is two orders of magnitude larger than GraphGen4Code [1], which covers only ∼1.3 M Python files and is limited to syntactic artifacts, while CodeGraph adds rich semantic metadata.After consolidation, CodeGraph’s semantic layer comprises ∼37.4 K domains, ∼19.7 K algorithms, ∼6.0 K design patterns, and 193 paradigms, with ∼19.8 K concepts grounded in Wikidata.Source: §1 · p. 3

2 Related Work

Existing code resources tend to offer either large-scale raw corpora or smaller task-specific supervision, while code knowledge graphs often emphasize syntactic structure. CodeGraph adds open-taxonomy semantic annotation and selectively applies LLM reasoning to Wikidata linking’s ambiguous long tail.

Code corpora and code knowledge graphs

  • Large code corpora expose raw tokens or basic metadata, while richer supervision is generally task-specific; CodeGraph adds a semantic layer at pre-training scale.GraphGen4Code instead models intraproject classes, functions, and data- and call-flow edges, supporting different queries from CodeGraph’s concept-oriented search.Source: §2 · p. 3

LLM-based annotation at scale

  • CodeGraph uses open-taxonomy annotation, generating free-form labels for later reconciliation rather than predicting labels from a fixed taxonomy.The paper also decomposes extraction into orthogonal prompts and enforces structured outputs with explicit empty-array negatives.Source: §2 · p. 3
  • The paper adds a calibrated per-annotation reliability layer, a protocol it says, to its knowledge, had not been used for code-semantic annotation at this scale.Comparable two-tier extractor-and-verifier protocols had appeared in distantly supervised NLP.Source: §2 · pp. 3–4

Entity linking to Wikidata

  • For Wikidata linking, deterministic SPARQL with a class-hierarchy filter handles unambiguous candidates, while a Deep Research Agent resolves ambiguous or empty residuals.This reserves more expensive, judgment-based reasoning for the long tail.Source: §2 · p. 4

3 Experimental Setup

The experimental setup uses a quality-filtered Stack-Edu corpus and a code-specialised open-weight model to build CodeGraph. It excludes markup files from semantic extraction and scales inference across GPU nodes.

3 Experimental Setup

  • The setup comprises the Stack-Edu corpus and inference infrastructure required to produce annotations at corpus scale.Figure 2 reports the language distribution of CodeGraph’s retained source files.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Dataset

  • Stack-Edu contains approximately 167 M files and 125 B tokens; semantic extraction uses its 14 programming languages, excluding Markdown.The corpus is a curated subset of The Stack v2 selected by a learned quality classifier; the remaining ∼145 M files become CodeGraph’s :File nodes.Source: §3.1 · p. 4

3.2 Inference Infrastructure

  • Qwen3-Coder-30B-A3B-Instruct processes files with up to 16,384 tokens of context in a data-parallel Ray and vLLM deployment.The open-weight, code-specialised model is chosen for reproducibility and auditability; extraction used 256 NVIDIA A100 GPUs and approximately 400,000 GPU-hours.Source: §3.2 · p. 4

4 CodeGraph Construction Pipeline

CodeGraph transforms retained source files into semantically annotated graph objects through extraction, canonicalisation, and Wikidata grounding. Its local concept vocabulary is connected to a global Wikidata hierarchy.

4 CodeGraph Construction Pipeline

  • The construction pipeline converts retained Stack-Edu files into graph objects through semantic extraction, post-processing and canonicalisation, then Wikidata grounding.The running example traces a file annotated for its domain, algorithm and complexity, paradigms, and design patterns.Source: §4 · p. 4, §4.1 · p. 5

4.1 Open-Taxonomy LLM Annotation Stage

  • Four parallel, structured prompts extract application domains, algorithms, programming paradigms, and design patterns, leaving labels open rather than predefined.The prompts request empty arrays when a file provides no support for an axis; separating axes is reported to reduce drift and hallucination relative to one all-in-one prompt.Source: §4.1 · pp. 4–5

4.2 Typed Property Graph Construction

  • The typed property graph links files to canonical concepts, while derived algorithm categories and normalized complexities support cross-language aggregation.File-concept edges retain annotation provenance, and free-form complexity strings are mapped to a shared asymptotic vocabulary.Source: §4.2 · p. 5

4.3 Three-Stage Wikidata Grounding

  • Wikidata grounding connects locally canonicalized concepts to language-independent identifiers and a shared class hierarchy across all four concept types.The local and global layers support rolling fine-grained concepts up to broader families and comparing concepts across languages under a common identifier space.Source: §4.3 · pp. 5, 7
  • SPARQL resolves unambiguous concepts, an agent reviews ambiguous or empty results, and a two-hop hierarchy traversal adds parent and grandparent entities.The grounding stage imports 19,807 distinct Wikidata entities and materialises parent links as WIKIDATA_PARENT edges.Source: §4.3 · pp. 6–7

5 Quality Assurance Protocol

CodeGraph combines a small human gold set with an LLM verifier to filter annotations at corpus scale. Silver-tier acceptance is high, while Wikidata-restricted verification is stricter and highlights differences among semantic axes.

Verifier selection

  • Devstral-2 2512 was retained as verifier, achieving 80.6% accuracy, 82.1% precision, 96.0% recall, and 88.5% F1.The six models had F1 scores between 84.9% and 88.5% across 1,536 model–human comparison pairs.Source: Table 2 · p. 7, §5 · p. 7

Quality assurance protocol

  • The protocol uses the adjudicated gold set to assess task difficulty and compare candidate LLM verifiers, then deploys the best-performing verifier as a per-annotation corpus filter.The target concepts require semantic interpretation and may involve contextual or taxonomically fuzzy judgements; manually annotating the full corpus was estimated to require roughly 9,500 person-years.Source: §5 · p. 7

Human gold set

  • The human gold set contained 292 judgements across 127 files, with overall Fleiss’ κ=0.153 and especially weak agreement for design pattern and paradigm.Agreement was highest for domain (κ=0.213) and algorithm (κ=0.200); the authors therefore frame quantitative downstream analyses around domain and algorithm.Source: §5 · p. 7

LLM-based silver-tier deployment for re-evaluation

  • On 10,000 files, verifier acceptance was 93% for paradigm, 88% for algorithm and domain, and 82% for design pattern.Cross-language acceptance was 87–90%; the authors treat 68% as a conservative precision lower bound and 88% as a loose upper bound.Source: Figure 7 · p. 8, §5 · p. 8

LLM-based behaviour verification on Wikidata-grounded annotations

  • On Wikidata-grounded annotations, paradigm remained most reliable, while the other three axes dropped significantly; the authors treat these rates as a stricter precision proxy.They hypothesize that algorithm and design-pattern labels often describe a function or class but are judged against the whole file; they leave function-level evaluation for future work and recommend treating design-pattern claims as exploratory.Source: Figure 8 · p. 8, §5 · p. 8

6 Graph Statistics

Applied to Stack-Edu, CodeGraph produces a large typed graph with a consolidated semantic vocabulary and Wikidata links. Its files and extracted concepts span 14 languages, with a skew toward several languages and broad application domains.

Graph composition

  • The resulting graph has 158,202,834 nodes and 1,020,887,480 edges, including 144,910,008 file nodes after excluding about 22 million markup-language files.The original Stack-Edu corpus contained 167 million files.Source: §6 · p. 8
  • Concept counts reflect a consolidated vocabulary: formatting variants, casing, punctuation, and one-off phrasings are collapsed into canonical entries.Separators and stray punctuation are stripped and casing is normalized before near-duplicates are merged.Source: §6 · p. 8
  • The consolidated vocabulary contains 37,412 domains, 19,660 algorithms, 5,955 design patterns, and 193 paradigms, with most linked to Wikidata.The respective Wikidata-linked counts are 36,879, 17,492, 5,666, and 188.Source: §6 · p. 8, Figure 9 · p. 9

Derived node layers

  • The graph adds 268,394 algorithm-category nodes and 12,941,405 algorithm-complexity nodes; its average degree is 12.91.Source: §6 · p. 9

Corpus and semantic-layer distribution

  • The corpus covers 14 languages, led by Java (30.6%), Python (16.9%), and C++ (11.2%); the semantic layer is similarly skewed toward broad domains such as web development, databases, and data science.Those three languages account for 58.7% of files, while the semantic layer also has a long tail of specialized concepts.Source: §6 · p. 9

7 Use Cases

The use cases illustrate how CodeGraph’s typed semantic links could support retrieval and algorithm-selection tasks. They are examples rather than an exhaustive account of possible applications.

Use case #1: Semantic code search

  • For semantic code search, graph traversal can retrieve dynamic-programming implementations before embeddings rerank candidates and concept/domain relations diversify results.This may surface related implementations despite different terminology, and distinguish areas such as competitive programming, RNA structure prediction, and financial option pricing.Source: §7 · p. 9
  • The same retrieval approach could support code-to-code search by clustering snippets according to associated concepts and domains [9] [10].Source: §7 · p. 9

Use case #2: Automated algorithm selection

  • For automated algorithm selection, CodeGraph could surface algorithms used on analogous problems across 145 million annotated files, including domains beyond the input domain.The anomaly-detection example spans machine learning, data science, IoT, finance, and other operational domains; Figure 10 illustrates cross-domain technique recurrence.Source: Figure 10 · p. 9, §7 · p. 9

8 Discussion and Limitations

CodeGraph grounds most consolidated concepts in Wikidata, while domain and algorithm labels show sharply skewed, long-tailed distributions. The authors caution that niche concepts and some annotation axes are exploratory, and that scaling human annotation is impractical.

Wikidata coverage and label consolidation

  • Wikidata links 36,879/37,412 domains (98.6%), 17,492/19,660 algorithms (89.0%), 5,666/5,955 design patterns (95.1%), and 188/193 paradigms (97.4%).Rates are measured on consolidated vocabularies; grounded labels can further collapse to shared Wikidata identifiers.Source: §8 · p. 9

Long-tailed domain and algorithm distributions

  • 62.1M files are associated with web development, over twice database (27.1M), followed by data science (24.9M) and game (16.4M).From fifth position onward, domains flatten into a 6.5–8.2M-file shoulder before the long tail.Source: §8 · p. 10, Figure 11 · p. 10
  • Robust statistical claims are limited to high-frequency labels, while niche concepts are exploratory indicators rather than measurement foundations.This boundary follows from the pronounced long tails in both domain and algorithm distributions.Source: §8 · p. 10

Limitations of our study

  • The human gold set contains 127 stratified files; domain-competent review across four axes makes scaling beyond a few hundred files impractical.Each label requires assessing multiple candidate concepts and adjudicating borderline cases.Source: §8 · p. 10
  • The gold set suggests paradigms and design patterns are harder to annotate than domains and algorithms, so they are released as exploratory signals.Downstream quantitative analyses should prioritize the two more stable axes; design patterns perform less well in the cited acceptance comparison.Source: §8 · p. 10

Long-tailed domain and algorithm distributions

  • 6.0M files map to linear search, the top algorithm label; 75.1% of domains and 38.8% of algorithms have fewer than 1,000 associated files.The algorithm head is dominated by computational primitives such as string operations, input validation, JSON, and regular expressions.Source: Figure 12 · p. 10, §8 · p. 10

9 Conclusion

The paper presents a pipeline for constructing a typed semantic knowledge graph over large source-code corpora. Applied to Stack-Edu, it produces a graph with about 158 million nodes and 1.02 billion edges, while the full corpus release and extended analyses remain future work.

9 Conclusion

  • The methodology combines open-taxonomy LLM extraction, three-stage Wikidata grounding, and calibrated quality assurance with an explicit precision/recall operating point.Deterministic resolution handles the unambiguous vocabulary, while a Deep Research Agent is reserved for the long tail.Source: §9 · p. 10
  • ∼158 M nodes and ∼1.02 B edges across 14 languages make up the Stack-Edu graph, where algorithmic primitives recur across different domains in patterns token-level retrieval cannot expose.Producing the graph required approximately 400,000 GPU-hours across extraction, grounding, and verifier deployment.Source: §9 · p. 10
  • The paper includes pipeline source code and a small reproducibility sample, while the full corpus release, extended cross-domain analysis, and AI-training implications are left to future work.Source: §9 · p. 10

GenAI Usage Disclosure

The disclosure states that generative AI tools were used only to improve writing clarity and style.

GenAI Usage Disclosure

  • Generative AI tools were used only to improve writing clarity and style; analyses, experiments, tables, and figures were produced without AI assistance.Source: GenAI Usage Disclosure · p. 11
Loading 2609.29474v1…