PaperlayerView exact revision on arXiv

Source-linked AI summary

PR4Code: A Pull Requests Dataset for AI Code Generation

Benedetta Donato, Leonardo Mariani, Daniela Micucci, Oliviero Riganelli

arXiv:2610.39890v1cs.SE

TL;DR

AI systems still struggle with larger software features, and available datasets offer limited coverage of realistic feature-level development. PR4Code collects and characterizes curated Java and Python pull requests, with their histories and code changes, as an evaluation resource. Its cases vary in change size and file types, while the dataset’s time cutoff only mitigates possible training leakage.

Abstract

from arXiv · show

AI-driven code generation is evolving from handling small, localized code-completion tasks toward supporting the implementation of full software features. To support this transition, this paper introduces PR4Code, a dataset of 4,508 Java and 8,831 Python curated Pull Requests (PRs) collected from GitHub, each enriched with metadata, commit histories, and detailed code changes. Unlike resources focused on isolated code fragments, PR4Code captures the feature implementation process, delivering a large and diverse collection of real-world development tasks that can be used to assess AI-driven solutions. Our analysis reveals substantial variability in PR structure, commit granularity, and textual content. In addition, we release scripts to regenerate and update the dataset, ensuring reproducibility and maintainability.

I. INTRODUCTION

AI code-generation systems have advanced on small tasks, but full-feature development remains difficult and is poorly represented in existing datasets. PR4Code addresses this gap with curated, context-rich pull requests intended to support evaluation of feature-level development.

I. INTRODUCTION

  • Current AI approaches remain more effective on small code tasks than on full features, modules, or applications, which require architectural understanding and integration across dependencies.Existing datasets often capture isolated changes rather than the descriptions, commits, and coordinated artifacts involved in feature development.Source: §I · p. 1
  • PR4Code supports study of autonomous feature implementation, heterogeneous artifacts, and the effects of cross-referenced or incomplete descriptions on generated changes.These questions are difficult to address with existing benchmarks and are central to understanding limitations in feature-level development.Source: §I · pp. 1–2
  • PR4Code contains 13,339 curated pull requests and 46,334 commits across 1,055 Java and Python repositories, with descriptions, cumulative diffs, and metadata.The authors also release construction scripts to support reproducibility and future extensions.Source: §I · pp. 1–2

II. RELATED WORK

  • Compared with commit- and patch-level resources, PR4Code preserves multi-commit, cross-artifact pull-request context and spans Java and Python development activities.It is intended primarily as an evaluation testbed for generalization, complementing narrower fragment- and issue-focused benchmarks.Source: §II · pp. 2–3

III. DATASET CONSTRUCTION METHODOLOGY

PR4Code is built through four sequential stages, from selecting repositories and relevant pull requests to analyzing commits and their cumulative code changes.

III. DATASET CONSTRUCTION METHODOLOGY

  • The construction process selects active GitHub repositories and closed pull requests that represent feature development, bug fixing, or incremental improvement.It then analyzes each PR’s commits and the cumulative effect of those commits to produce the curated dataset.Source: §III · p. 3
  • Commit analysis and cumulative-change analysis capture both the development sequence and the final integrated changes within each pull request.Source: §III · p. 3
  • The process outputs a curated, publicly available dataset of pull requests associated with feature development in selected repositories.Source: §III · p. 3

A. REPOSITORY SELECTION

PR4Code selects popular Java and Python repositories and filters their closed pull requests for relevant development activity and sufficiently informative descriptions. It then records commit histories and their changes.

A. REPOSITORY SELECTION

  • Repositories had more than 500 GitHub stars and recent pull requests after October 3, 2023, and were primarily Java or Python.The popularity threshold was chosen to indicate community adoption, relevance, and maturity; the date cutoff was intended to reduce training-data leakage risk.Source: A. REPOSITORY SELECTION · p. 3

B. PULL REQUEST SELECTION

  • Closed pull requests were filtered for development-activity keywords, titles and bodies totaling at least 100 characters, and qualifying branch-name keywords.The filters targeted feature development, bug fixing, and incremental improvement.Source: B. PULL REQUEST SELECTION · pp. 3–4
  • The selection yields 13,339 pull requests: 4,508 from Java projects and 8,831 from Python projects.Source: B. PULL REQUEST SELECTION · p. 4

C. COMMIT ANALYSIS

  • For each commit, PR4Code records its identifier, timestamp, message, modified files, line additions and deletions, and textual references.These fields support reconstruction of development activity and cross-referencing between commit text and code changes.Source: C. COMMIT ANALYSIS · p. 4

D. CUMULATIVE ANALYSIS OF CODE CHANGES

PR4Code reconstructs each pull request’s cumulative changes by comparing the pre-PR and final merged snapshots. The dataset organizes files, metadata, and descriptions to retain task context.

D. CUMULATIVE ANALYSIS OF CODE CHANGES

  • The cumulative diff compares the project revision before a pull request with the final snapshot after its merged changes.For each modified file, the dataset records its path, status, line counts, textual patch, and retrieval URLs.Source: D. CUMULATIVE ANALYSIS OF CODE CHANGES · p. 4

IV. THE PR4Code DATASET

  • PR4Code stores pre- and post-change files alongside JSON change metadata, organized by repository and pull request.A GitHub example highlights extracted PR fields including its description, commits, changed files, and diff summary.Source: §IV · p. 4

A. DESCRIPTIONS IN PRS AND COMMITS

  • The description analysis covers 4,508 Java and 8,831 Python pull requests, plus 14,639 Java and 31,695 Python commits.The authors examine description length and content as input for realistic, context-aware code-generation prompts.Source: A. DESCRIPTIONS IN PRS AND COMMITS · p. 5

1) Keyword frequency

PR4Code’s keyword analysis captures diverse development activities, with adding and fixing most frequent and similar patterns across Java and Python. Its PR- and commit-level descriptions vary in length and serve complementary roles.

1) Keyword frequency

  • 78.49% of 888 manually checked keyword occurrences were coherent, with higher coherence for fix (92%) and refactor (94%) than update (61%) or merge (39%).The 300-PR sample was balanced across Java and Python; a PR could contain multiple keywords.Source: 1) Keyword frequency · p. 5
  • Adding and fixing are the most frequent keywords, while Java and Python show similar patterns in how developers describe code evolution.The analysis treats keyword frequency as a characterization of activity, not a measure of PR importance or complexity.Source: 1) Keyword frequency · p. 5, Figure 3 · p. 6

2) Length and structure of textual descriptions

  • Java PR bodies average 19.3 lines (median 13), versus 21.3 (median 15) for Python; titles average 51 characters in Java and 48 in Python.Descriptions range from extremely short to more than 30,000 characters, while titles tend to be compact.Source: 2) Length and structure of textual… · p. 5, Figure 4 · p. 6
  • PR titles and descriptions provide context for assessing AI agents that plan and generate code for full feature implementations.Source: 2) Length and structure of textual… · p. 5
  • Commit descriptions are typically one or two lines and provide a more concise, operational view of changes than PR descriptions.Average commit-message lengths are 86 characters in Java (median 48) and 105 in Python (median 43).Source: 2) Length and structure of textual… · p. 5

3) Hyperlinks and cross-references

PRs often link to external artifacts and use structured templates, so their text can include more than plain prose. The characterization also considers commit and code-change attributes.

3) Hyperlinks and cross-references

  • 2,487 of 4,508 Java PRs and 4,576 of 8,831 Python PRs contain an external link; 55% and 52%, respectively, include at least one external reference.Issue-closing expressions occur in 1,201 Java and 2,402 Python PRs.Source: 3) Hyperlinks and cross-references · p. 6
  • Figures 5, 6, and 7 describe distributions of PR-description character counts and commit-description line and character counts.Source: Figure 5 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
  • Markdown checklists appear in 2,431 Java PRs and 5,106 Python PRs, while Co-authored-by markers appear in 147 and 495, respectively.Signed-off-by tags occur in 529 Java PRs and 492 Python PRs.Source: 3) Hyperlinks and cross-references · p. 6

B. CHARACTERISTICS OF THE PRS

1) Commits per PR

Most PRs involve a few commits and files, but line-change sizes are highly variable. The dataset spans localized changes and larger restructuring tasks, supporting evaluation across task sizes.

1) Commits per PR

  • Java PRs average 3.25 commits (median 1), with 54% containing one commit; Python averages 3.59 (median 2), with 47% containing one.Only 7.7% of Java and 10.5% of Python PRs exceed eight commits; the maximum is 30, so the median better reflects typical PR size.Source: Figure 8 · p. 7, 1) Commits per PR · p. 7

2) Files modified per PR

  • Java PRs modify 3.47 files on average and Python PRs 4.26; 75% of Java and 70% of Python PRs affect only one or two files.A few PRs modify hundreds of files, typically in refactoring or large merges.Source: 2) Files modified per PR · p. 7, Figure 9 · p. 8

3) Lines added, removed, and changed

  • The distribution of added, deleted, and modified lines is extremely skewed, with outliers and PRs changing more than 100 lines of code.This range includes both small-scale maintenance and large-scale restructuring, supporting evaluation across task sizes and complexity levels.Source: 3) Lines added, removed, and changed · pp. 7–8, Figure 10 · p. 8

4) Type of modified files

PR4Code shows that pull requests often modify files beyond language-specific source code, tests, and comments. Commit-level comparisons also reveal heterogeneous file changes and broadly comparable Java and Python characteristics.

4) Type of modified files

  • 31.8% of Java PRs and 23.5% of Python PRs modify no language-specific source files; 44.4% and 50.6%, respectively, modify source files only.Mixed source and non-source changes account for 23.8% of Java PRs and 25.6% of Python PRs.Source: 4) Type of modified files · p. 8

5) Presence of test files

  • 38.2% of Java PRs and 33.4% of Python PRs include changes to at least one test file.Test changes were identified through filenames or hosting folders containing the string “test.”Source: 5) Presence of test files · p. 8

6) Comment-related changes

  • 49.2% of Java PRs and 54.6% of Python PRs modify comments, while comment-only changes are below 1%.The authors highlight the need for AI systems to generate and update comments alongside source code.Source: 6) Comment-related changes · pp. 8–9

C. CHARACTERISTICS OF COMMITS

  • 26%–30% of commits modify only non-code files; code-only commits account for 56.1% in Python and 50.8% in Java.Mixed commits involving auxiliary files account for about one-fifth; the authors attribute Java’s higher non-code share to more complex build and deployment configurations.Source: C. CHARACTERISTICS OF COMMITS · p. 9
  • Java and Python PRs have broadly comparable commit structure, code churn, description size, and file-modification patterns, with negligible observed effect sizes.Files changed per commit and commit-message length show the largest differences, but their effect sizes are still negligible.Source: C. CHARACTERISTICS OF COMMITS · p. 9

D. EXTERNAL VALIDATION OF DATASET CHARACTERISTICS

The external validation compares PR4Code with SWE-bench and a random GitHub PR sample. It finds broader task and repository coverage in PR4Code while supporting its textual filtering choices.

Comparison with SWE-bench

  • PR4Code contains 13,339 PRs from 1,055 Java and Python repositories, while SWE-bench has 2,294 issue–PR pairs from 12 Python repositories.PR4Code covers varied activities and artifacts beyond SWE-bench’s focus on executable issue-resolution tasks.Source: D. EXTERNAL VALIDATION OF DATASET… · p. 9, Table 2 · p. 10

Random GitHub PR sample

  • 19% of random GitHub PRs had empty bodies and 27% had titles and bodies shorter than 100 characters, supporting a minimum text threshold for AI prompts.The random sample was 300 closed PRs balanced across Java and Python and subject to PR4Code’s temporal constraint.Source: D. EXTERNAL VALIDATION OF DATASET… · p. 9, Table 3 · p. 10
  • 71% of the random PRs contained at least one selection keyword, suggesting those terms are common in general GitHub activity rather than artifacts of PR4Code filtering.The comparison supports the view that PR4Code selects PRs with explicit intent and sufficient context while retaining common GitHub terminology and structure.Source: D. EXTERNAL VALIDATION OF DATASET… · pp. 9–10

V. DISCUSSION

PR4Code offers varied real-world feature-development tasks and supports several evaluation scenarios. Its scope and selection procedures also create generalizability and dataset-characterization boundaries.

V. DISCUSSION

  • PR4Code spans changes from a few to hundreds of thousands of lines, varied numbers of files and code locations, heterogeneous artifacts, and PR- and commit-level descriptions.The dataset can support feature-level generation, incremental commit modeling, heterogeneous artifact handling, and evaluation with cross-referenced descriptions.Source: §V · p. 10
  • The dataset focuses on popular open-source Java and Python repositories, which may limit generalizability to less popular projects, other languages, or proprietary settings.Repository selection may favor well-structured, actively maintained projects and underrepresent noisy, poorly documented, or irregular development processes.Source: §V · pp. 10–11
  • Minimum-text and keyword filters may favor PRs with explicit, descriptive language and exclude shorter or less structured valid development tasks.Models evaluated on the dataset may encounter clearer input descriptions than they do in practice.Source: §V · p. 11
  • Keyword-based categories describe observed activity types, but may not reflect task complexity or practical relevance, particularly for rare critical activities.The authors advise against treating these categories as direct measures of task importance or difficulty.Source: §V · p. 11
  • The October 3, 2023 creation-date cutoff mitigates, but does not eliminate, training-data leakage risk.LLMs may use partially undisclosed or continuously updated corpora, so the cutoff is a best-effort mitigation.Source: §V · p. 11

VI. USING PR4Code

PR4Code provides artifacts for reconstructing repository tasks and comparing generated changes with reference implementations. A small experiment demonstrates its use while finding mixed outcomes on selected PRs.

A. USAGE WORKFLOW

  • A typical workflow uses the PR description as model input, retrieves the repository’s base version, generates changes, and compares outputs with cumulative diffs or individual commits.Structured metadata and repository links support flexible, reproducible experimentation.Source: A. USAGE WORKFLOW · p. 11

B. SMALL-SCALE APPLICABILITY EXPERIMENT

  • The applicability experiment selected 10 PRs and used a four-stage agentic pipeline for planning, context retrieval, code generation, and compilation and test execution.The selected PRs were localized, buildable, testable, and limited to at most three files; the experiment was intended to demonstrate usability, not representative model performance.Source: B. SMALL-SCALE APPLICABILITY EXPERIMENT · p. 11
  • 6 of 10 generated patches compiled, 4 passed target tests, and manual inspection found the requested modification correctly implemented in four cases.The authors report that the results demonstrate dataset usability and reveal a gap between implementing requested changes and passing tests.Source: Table 4 · p. 12, B. SMALL-SCALE APPLICABILITY EXPERIMENT · p. 12

VII. CONCLUSIONS

PR4Code is a curated pull-request dataset intended to support research on AI code generation and software evolution. It includes heterogeneous cases, from large, multi-file PRs to changes involving non-source artifacts.

VII. CONCLUSIONS

  • PR4Code is a curated dataset of pull requests and associated data for research on AI-based code generation and software evolution.The paper presents the dataset as a foundation for investigating key research questions in these areas.Source: §VII · p. 12
  • Its heterogeneous cases include large PRs spanning many files and locations, as well as modifications involving non-source code artifacts.Source: §VII · p. 12
  • The dataset is publicly available, and future work will use it to study current AI techniques and support next-generation models for realistic, feature-level changes.Source: §VII · p. 12

Related papers

Loading 2610.39890v1…