PaperlayerView exact revision on arXiv

Source-linked AI summary

SIEVE: A Parameterized GitHub Corpus Builder for Software Engineering Research

Musfiqur Rahman, Mahi Begoug, Emad Shihab

arXiv:2610.22544v1cs.SE

TL;DR

Researchers lack a reusable way to build richly annotated GitHub corpora with control over repository and code selection. SIEVE extracts and annotates function-, method-, and class-level records under configurable criteria. A manual audit found 99.0%–100% accuracy across four dimensions, and its classifier achieved F1=0.9478 and AUROC=0.9902.

  • Problem

    Researchers often rely on laborious, study-specific scripts or pre-built datasets that do not provide unified, researcher-controlled corpus construction, including temporal filtering.

    Source: §I · p. 1

  • Method

    SIEVE is a parameterized GitHub pipeline that extracts code-level records and combines configurable discovery with structural, dependency, and optional AI-generated-code annotations.

    Source: §I · pp. 1–2

  • Results

    A manual audit found 99.0%–100% accuracy across four extraction dimensions, and the CodeBERT-based classifier achieved F1=0.9478 and AUROC=0.9902.

    Source: §I · p. 2

  • Takeaways & Limitations

    SIEVE provides reusable infrastructure for customized code corpora supporting structural and dependency studies, class-level benchmarking, AI-generated-code research, and temporally bounded LLM evaluation.

    Source: §VII · p. 7

Abstract

from arXiv · show

Software engineering research increasingly relies on large, richly annotated corpora of real-world code. Still, researchers typically assemble these with ad-hoc scripts or static, pre-built datasets that offer no control over corpus composition. This leaves a gap for a reusable tool that goes below the repository level and builds annotated, function- and class-level code corpora under researcher-specified criteria. We present SIEVE (Software Ingestion & Extraction for Verifiable Evaluation), an open-source, parameterized GitHub corpus builder that, unlike repository-level mining tools, works at code granularity: it locates individual functions, methods, and classes in repositories and extracts them as records, giving direct control over corpus construction based on several criteria, including widely used programming languages (Python, Java, JavaScript, C++), repository quality, and creation date. To support software engineering research and LLM evaluation on real-world code, SIEVE extracts CodeSearchNet- and OpenClassGen-compatible records, each annotated with 22 structural metrics and AST-derived features, and optionally with a CodeBERT-based P(AI-generated) score (F1=0.9478, AUROC=0.9902), alongside the third-party dependencies declared in each repository's package manifest. These annotations enable researchers to conduct structural analyses, investigate code dependencies, assess AI-generated code, and construct customized benchmarks for evaluating LLMs, including under temporal constraints. A manual validation of 400 records in four languages and two granularities shows 100% accuracy in code metrics and import detection, 99.0% in source completeness, and 99.5% in metadata accuracy. SIEVE is publicly available with a live demo at https://mrahman2025-sieve.hf.space.

I. INTRODUCTION

SIEVE addresses the need for reproducible, researcher-configurable GitHub corpora by extracting and annotating code at function, method, and class granularity. Its manual validation reports 99.0%–100% accuracy across four extraction dimensions, while related-work comparisons distinguish it from tools and datasets that cover only parts of corpus construction.

I. INTRODUCTION

  • GitHub corpus construction is laborious and may be incomplete; temporal filtering is especially relevant when evaluating models on code from repositories created after their training cutoff.Reported overlap between benchmark solutions and pre-training corpora motivates reproducible corpus construction with temporal filtering.Source: §I · p. 1
  • SIEVE builds configurable GitHub corpora from function-, method-, and class-level records, with control over language, granularity, repository quality, and creation date.Records include file paths, line spans, and commit SHAs.Source: §I · pp. 1–2
  • The pipeline integrates repository discovery, multilingual extraction, deduplication, structural metrics, AI-generated-code scoring, dependency extraction, and compatible record exports.Its exports support CodeSearchNet- and OpenClassGen-compatible formats.Source: §I · p. 2
  • A manual audit of 400 records across four languages and two granularities found 99.0%–100% accuracy across source completeness, metadata accuracy, import detection, and structural metrics.The integrated classifier achieved F1=0.9478 and AUROC=0.9902 on a held-out test set of 11,419 samples.Source: §I · p. 2

II. BACKGROUND & RELATED WORKS

  • Existing discovery tools, datasets, and AI-code detectors cover separate capabilities, while SIEVE combines discovery, extraction, structural analysis, dependency extraction, and AI-code detection in one pipeline.Unlike predefined datasets, its corpus construction supports configurable repository selection, code granularity, and temporal constraints.Source: §II · p. 2

III. TOOL ARCHITECTURE

SIEVE exposes a shared pipeline through three interfaces and organizes its work into four modules. Its configuration, data exchange, and opt-in stages are designed to keep modules testable and extensible.

III. TOOL ARCHITECTURE

  • The CLI, Python API, and Streamlit application call the same pipeline orchestrator rather than implementing separate pipeline logic.SIEVE is implemented in Python 3.11+ and released under the MIT license.Source: §III · p. 2
  • The orchestrator composes repository discovery, code extraction, corpus annotation, and deduplication/export into the SIEVE pipeline.These modules respectively find repositories, produce function- and class-level records, add annotations, and serialize the corpus.Source: §III · pp. 2–3, Figure 1 · p. 3
  • A validated configuration object and plain record data classes decouple modules; expensive stages are opt-in, and processing failures are isolated and reported.Modules can be tested in isolation, while progress callbacks decouple pipeline execution from the interfaces.Source: §III · p. 3

A. Configurable Repository Discovery

Repository discovery filters GitHub candidates using researcher-specified language, date, quality, and size criteria. Code extraction then parses selected repositories into function-, method-, and class-level records, with allocation and extension mechanisms for corpus building.

A. Configurable Repository Discovery

  • Repository discovery filters by language, creation-date range, stars, contributors, and optional repository cap, while excluding forks and archived repositories.It uses creation date rather than pushed date because pushed date reflects recent activity, not when the repository code was written.Source: A. Configurable Repository Discovery · p. 3
  • Optional engineered-project filtering applies license, release, contributor, pull-request, and issue criteria; lazy discovery stops at the requested cap.Repository creation date bounds the repository itself, not the provenance of its contents; a new repository may contain copied older code.Source: A. Configurable Repository Discovery · p. 3

B. Code Extraction

  • Tree-sitter parses selected source files, and syntax-tree traversal locates function, method, and class nodes; the resulting records follow CodeSearchNet- and ClassEval/OpenClassGen-compatible schemas.Records also retain repository, file, language, and commit information for version-stable permalinks.Source: B. Code Extraction · p. 3
  • When a corpus cap is set, a counting pass allocates records proportionally across repositories, and extraction buffers surplus records to refill shortfalls.Language-specific extractors emit shared record types; adding a language requires its extensions, grammar, extractor, and metric node-type tables.Source: B. Code Extraction · p. 3

C. Corpus Annotation

Corpus annotation adds structural, AST-derived, dependency, and optional AI-generated-code information to extracted records. Deduplication and export can then produce configured corpus files and run metadata.

C. Corpus Annotation

  • Each record receives 22 cross-language structural metrics spanning size and comments, control flow, Halstead measures, and Maintainability Index.The metrics are computed from each snippet’s tree-sitter AST.Source: C. Corpus Annotation · p. 4
  • Repository annotations include declared direct dependencies with version constraints and main, development, or optional kinds; each record also stores AST summary fields.Full nested-JSON AST export is optional because it dominates file size.Source: C. Corpus Annotation · p. 4
  • Records can optionally receive a CodeBERT-based P(AI-generated) score, and scoring failures do not abort corpus construction.The classifier is applied in batches and can be replaced without changing other modules.Source: C. Corpus Annotation · p. 4

D. Deduplication and Export

  • Optional MinHash deduplication drops snippets above a configurable estimated-Jaccard threshold, and exports JSONL, Parquet, or both with a manifest of configuration and run metadata.The default similarity threshold is 0.8, and the first occurrence in each cluster is retained.Source: D. Deduplication and Export · p. 4

IV. TOOL USAGE SCENARIO

SIEVE exposes one pipeline through a CLI, Python API, and web application. A shared scenario illustrates repository filtering, record caps, annotation, and export through these interfaces.

Interfaces

  • All three interfaces call the same underlying pipeline.The tool is open source under the MIT license and can be installed from source.Source: §IV · p. 4

Usage scenario

  • The illustrated scenario filters engineered-quality Python repositories by date, stars, and contributors, with limits of 20 repositories, 30 functions, and 30 classes.The scenario also requests LLM-generation scores.Source: §IV · p. 4

Command line

  • The CLI runs SIEVE with individual flags or a JSON configuration that supports reproducibility and takes precedence over flags.Its options include language and date filters, repository and record caps, engineered-project filtering, deduplication, and export format; LLM-score annotation is configured in JSON.Source: §IV · pp. 4–5

Python API

  • Researchers can pass a SIEVEConfig object to run_pipeline for scripted or notebook-driven workflows, which return run statistics and exported-file paths.API deduplication is off by default.Source: §IV · p. 5

Web interface

  • The web application exposes the same pipeline settings as form controls and provides corpus downloads in JSONL, Parquet, or both.Downloads include files by granularity and a manifest, or a single ZIP bundle.Source: §IV · p. 5

Deployment

  • The web interface is deployed on HuggingFace Spaces, and deployment follows passing tests on the main branch.The GitHub Actions pipeline runs 455 tests with 83% coverage on every commit.Source: §IV · p. 5

V. EVALUATION

SIEVE is evaluated through corpus extraction quality, the bundled AI-generated-code classifier, and a user survey. These dimensions address record correctness, classifier performance, and perceived usability.

Evaluation scope

  • The evaluation assesses the quality of the extracted corpus.This is one of three evaluation dimensions.Source: §V · p. 5
  • The evaluation measures the performance of SIEVE’s bundled LLM-generated-code classifier.This is one of three evaluation dimensions.Source: §V · p. 5
  • The evaluation includes a user survey of perceived usability.This is one of three evaluation dimensions.Source: §V · p. 5

A. Corpus Quality Validation

A 400-record manual audit found perfect accuracy for code metrics and import detection, with two localized deviations in source completeness and metadata. The validation covers four languages, two granularities, and four assessment dimensions.

Validation method

  • The audit sampled 400 records—100 each from Python, Java, C++, and JavaScript—split evenly between function/method and class granularity.Each record was assessed for source completeness, metadata accuracy, import detection, and code metrics; partial issues counted as incorrect.Source: A. Corpus Quality Validation · p. 5
  • Accuracy is reported by dimension, language, and granularity, with 95% cluster-bootstrap confidence intervals accounting for records clustered within repositories.The scoring groups contain 50 records per language-granularity cell or 400 records overall.Source: A. Corpus Quality Validation · p. 5

Validation results

  • 100% accuracy on all 400 audited records was achieved for code metrics and import detection; 30 of 32 language-granularity cells reached 100%.The two deviations were source completeness for C++ classes (92%) and metadata accuracy for Java functions (96%), totaling 6 of 1,600 ratings.Source: A. Corpus Quality Validation · pp. 5–6

Validation context

  • A separate interface run processed 12 of 20 discovered engineered repositories and yielded 60 records in 2.6 minutes.The records comprised 30 functions and 30 classes.Source: Figure 2 · p. 6

Validation results

  • The 92% source-completeness result for C++ classes reflects tree-sitter-cpp parse errors on some valid code, due to unresolved syntactic ambiguity without type information.The grammar selects one parse by design.Source: A. Corpus Quality Validation · p. 6

B. LLM Classifier Performance

SIEVE evaluates its bundled CodeBERT classifier for distinguishing human-written from LLM-generated code. On a held-out test set, it achieves F1 = 0.9478 and AUC-ROC = 0.9902, though the test set does not establish generalization to unseen LLMs.

B. LLM Classifier Performance

  • F1 = 0.9478 and AUC-ROC = 0.9902 on a held-out test set of 11,419 samples.Per-language F1 ranged from 0.9407 (C++) to 0.9515 (Java).Source: B. LLM Classifier Performance · p. 6
  • The test set shares generators and prompting protocol with training, so the results do not establish generalization to unseen LLMs.The classifier was trained on 114,186 snippets and evaluated on a 10% stratified split.Source: B. LLM Classifier Performance · p. 6
  • The classifier is intended to flag candidate snippets for downstream filtering, with recall higher than precision.Precision = 0.9257 and recall = 0.9709.Source: B. LLM Classifier Performance · p. 6

C. User Survey

In an anonymous survey of 26 SE researchers, respondents generally viewed SIEVE as easy to configure and useful. The results indicate perceived usability and usefulness, not measured effort savings.

C. User Survey

  • 25 of 26 respondents said they would use SIEVE in their research, and all 26 agreed contamination awareness matters.Seventeen said they definitely would use it, while eight would use it with improvements.Source: C. User Survey · p. 6
  • 21 of 26 respondents found configuration easy or very easy, all 26 agreed samples looked syntactically correct and complete, and 24 agreed it saved time.The survey was conducted online from September 1 to 30, 2026.Source: C. User Survey · p. 6
  • The survey supports conclusions about perceived usability and usefulness, not measured effort savings.It was a small, self-selected, anonymous sample, and most respondents' backgrounds could not be verified.Source: C. User Survey · p. 7

VI. TOOL APPLICABILITY

SIEVE's applicability spans research, engineering practice, and education. Its records and metrics support distinct analyses and activities, with structural metrics and temporal filtering carrying stated interpretive limits.

Researchers

  • Researchers can use SIEVE's unified pipeline and record schema for studies of code quality, dependencies, documentation, inheritance, repository cohorts, and temporally bounded LLM evaluation.A run manifest records configuration, while replacing a chain of separately configured tools lowers tooling effort and the number of tools to learn.Source: §VI · p. 7

Practitioners

  • Engineering teams can use comparable-project metrics to calibrate thresholds, repository manifests to examine declared dependencies, and classifier scores to prioritize code review.The classifier score is a probabilistic signal, not evidence of authorship.Source: §VI · p. 7

Educators

  • Educators can select real functions and classes for exercises and comprehension, and use structural metrics in maintainability assessment.Those metrics are proxies for student-perceived difficulty, and creation-date filtering may reduce—but does not guarantee avoidance of—code already seen by LLMs.Source: §VI · p. 7

VII. CONCLUSION AND FUTURE WORKS

SIEVE is presented as reusable, configurable infrastructure for building code corpora tailored to software engineering research. The conclusion reports high manual-validation accuracy and classifier performance, while future work targets broader language and declaration support and a more interpretable classifier.

VII. CONCLUSION AND FUTURE WORKS

  • Manual validation found 99.0%–100% accuracy across four dimensions, and the bundled classifier achieved an F1-score of 0.9478 on its held-out test set.The two validation deviations were traced to a C++ grammar limitation and the extractor's handling of enclosing declarations.Source: §VII · p. 7
  • SIEVE provides direct control over language, granularity, repository quality, and creation date for sourcing real-world code corpora.The paper describes it as reusable infrastructure supporting structural and maintainability analyses, dependency studies, class-level benchmarking, AI-generated-code studies, and temporally bounded LLM evaluation.Source: §VII · p. 7
  • Future work includes expanding beyond four languages, handling interfaces, enums, and structs, and replacing the classifier with an interpretable alternative.The authors also identify richer filtering, faster execution, and additional languages as requested improvements.Source: §VII · p. 7

TOOL AND DATA AVAILABILITY

  • The paper states that all artifacts are publicly available.The availability section lists a live demo, GitHub repository, and classifier weights.Source: TOOL AND DATA AVAILABILITY · p. 7
Loading 2610.22544v1…