RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing

1 Introduction

• Building repository-level datasets with execution feedback is difficult because integration testing requires building entire repositories, a challenging process that can limit dataset scale.Prior approaches require substantial manual effort or have low automated success rates.Source: §1 · p. 1
• REPOST isolates a repository function and its local dependencies in a separate script, generates tests, and iteratively improves and filters evaluation scripts for execution and test quality.The sandboxed function is checked to ensure its functionality remains unchanged.Source: §1 · pp. 1–2
• Sandboxing simplifies external dependencies and permits debugging separate scripts without modifying the original repository, preserving naturally occurring repository context for code generation.Automation also supports collecting training data at scale and constructing live benchmarks to avoid contamination issues.Source: §1 · p. 2
• REPOST-TRAIN contains 7,415 functions from 824 repositories, while REPOST-EVAL contains 296 functions from 99 repositories.The paper describes REPOST-TRAIN as the largest repo-level code generation dataset with execution support, to the authors’ knowledge.Source: §1 · p. 2
• 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval gains follow training on REPOST-TRAIN; GPT-4o reaches 39.5 Pass@1 on REPOST-EVAL, which benchmarks 12 models.The reported gains are for Qwen2.5-Coder.Source: §1 · p. 2

2 The REPOST Framework

• REPOST constructs environments by sampling repositories and extracting functions, sandboxing functions and local dependencies, generating tests, and checking executability, functionality, and test quality.Evaluation scripts can be run in a shared Docker environment while models retain access to the original repositories.Source: §2 · p. 2

2.1 Repository and Function Curation

• Repositories are sampled as non-forked, MIT-licensed Python projects under 10M in file size; functions tied to GPUs or cloud tasks are filtered, and each repository contributes at most 30 functions.The train and evaluation sets began with 1,000 and 200 repositories, respectively.Source: §2.1 · pp. 2–3

2.2 Sandboxing: Key to Environment Setup

• Sandbox testing puts the target function and its local dependencies in a separate script, extracting direct and indirect dependencies through the call graph and prompting an LLM to aggregate code fragments.For external APIs, files, or databases, the LLM creates mock connections or example files; the target function’s functionality is checked against the original.Source: §2.2 · p. 3, Figure 3 · p. 3

2.3 Test Generation

• Tests use generated inputs for equivalence testing against the sandboxed ground-truth function, with mock-class instances available as inputs when APIs are mocked.Test checks include calling the target function and having at least three assertions; the final quality step checks coverage and correctness.Source: §2.3 · pp. 3–4

2.4 Quality Control and Filtering

• Examples with execution errors or ground-truth test failures are debugged iteratively and filtered if errors remain; branch coverage is improved toward 80% for training and 100% for evaluation.AST identity is used as a sufficient functionality-equivalence check; for the training set, examples with different ASTs may also pass an LLM comparison, while those are filtered from evaluation.Source: §2.4 · p. 4

3.1 Statistics

• REPOST-TRAIN is reported as the largest repo-level dataset with execution feedback, while comparison with R2E on the same inputs shows REPOST yields more repositories and examples.The authors attribute its greater feasibility and scalability to installing only the dependencies needed for each function.Source: §3.1 · pp. 4–5
• Standalone functions make up 28.1% of REPOST-TRAIN and 26.4% of REPOST-EVAL, close to the 27% estimate for GitHub functions.The authors also report high test numbers and branch coverage, and examples with varied lengths and dependencies.Source: §3.1 · p. 5

3.2 Quality Verification with Human Study

• Human annotators agreed with GPT-4o on functionality checks for all 13 examples that passed its check, and judged 9 of 10 test-check-passing examples to have high-quality tests.The reported human–GPT-4o agreement scores were 0.9179 for functionality and 0.7750 for test checks.Source: §3.2 · p. 5
• Human participants solved 81.5% of the sampled examples, which the authors interpret as evidence that most were reasonable and not too complicated.The remaining examples included cases involving unfamiliar tasks or libraries and unclear function intent.Source: §3.2 · p. 5
• The cited passage reports that examples are solved in the first submission, while 37.0% require more than 5 submissions to solve.Source: §3.2 · p. 6

4 Code Generation Training Experiments

• Rejection Sampling (Self) generates candidate solutions with the trained model, while Rejection Sampling (Distill) uses stronger models; only solutions passing the tests are retained for fine-tuning.The experiments use GPT-4o and Claude-3.5-Sonnet to generate and debug candidates for Distill.Source: §4 · p. 6, §4.1 · p. 6

4.1 Experimental Setup

• The experiments evaluate StarCoder2-7B and Qwen2.5-Coder-7B on HumanEval, RepoEval's function split, and REPOST-EVAL using Pass@1.RepoEval uses Oracle context; the Distill candidates are generated and debugged using GPT-4o and Claude-3.5-Sonnet.Source: §4.1 · p. 6

4.2 Main Results

• 5.5% Pass@1 on HumanEval, 3.5% on RepoEval-Func, and 3.0% on REPOST-EVAL are the reported Qwen2.5-Coder gains from REPOST-TRAIN.Source: §4.2 · p. 6
• 4.3% Pass@1 on HumanEval is the reported advantage of Rejection Sampling (Distill) over SFT.Rejection sampling also outperforms fine-tuning on the original GitHub function in the reported experiments, including with self-training.Source: §4.2 · p. 7
• Self-training generally performs below distillation; the authors obtained 1,573 additional training targets with StarCoder2 versus 3,606 by combining GPT-4o and Claude-3.5.They suggest sampling more candidates or using different contexts as future ways to obtain more examples.Source: §4.2 · p. 7

4.3 Result Analysis

• Rejection Sampling (Distill) performance increases as training data scales, and rejection sampling outperforms SFT at the tested data scales.Figure 4a compares Pass@1 on RepoEval across training-example counts.Source: Figure 4 · p. 6, §4.3 · p. 7
• With 2,000 examples, Sample-by-Example covers 678 repositories and outperforms Sample-by-Repo, which covers 221; rejection sampling further improves performance.Figure 4b compares Pass@1 on RepoEval for the two sampling methods and reports further enhancement with Rejection Sampling (Distill).Source: Figure 4 · p. 6, §4.3 · p. 7

5.1 Experimental Setup

• The benchmark prompts models with repository context built from modules on which the target function depends, then executes solutions and computes Pass@1 as the fraction passing all tests.Source: §5.1 · p. 7

5.2 Experimental Results

• 39.53 Pass@1 for GPT-4o and a 5.07 Pass@1 gap between GPT-4o and DS-R1-Qwen-32B show substantial room for improvement on RepoST-Eval.Table 6 presents the code-generation results for open-source and proprietary models.Source: §5.2 · pp. 7–8
• Both models benefit from self-repair, but gains become limited after three iterations, while the performance gap between them grows with more debugging iterations.The authors conclude GPT-4o is stronger in both code generation and debugging.Source: §5.2 · p. 8
• Both models achieve much higher performance when provided with generated docstrings; original docstrings average 36.82 tokens/3.95 lines, while GPT-4o-generated docstrings average 130.84 tokens/9.92 lines.With more detailed docstrings, the performance gap between the two models becomes larger.Source: §5.2 · p. 8
• When a function’s intended specification is unclear, completing it as intended may be impossible, causing both weak and strong models to fail.Source: §5.2 · p. 8

Code Generation Training

• Online algorithm-problem datasets use execution feedback to construct training targets (Ni et al., 2024; Zhang et al., 2024) or reward signals (Liu et al., 2023a; Jiang et al., 2024).Repo-level training environments such as SWE-Gym (Pan et al., 2024) manually configure entire repositories, limiting dataset scale because setup is difficult to automate.Source: §6 · p. 8
• RepoST uses sandbox testing to set up only the necessary dependencies for individual GitHub functions, reducing environment-setup difficulty and improving scalability.Source: §6 · p. 8

Code Generation Benchmarks

• Repository-level benchmarks cover tasks including code-patch generation (Zhang et al., 2023; Li et al., 2024b), issue solving, test generation (Jain et al., 2025), and environment setup (Bogin et al., 2024).Live evaluation and LLM-assisted setup still require building the entire repository, constraining dataset scale.Source: §6 · p. 8

• Sandbox testing constructs environments from naturally occurring repositories and provides execution feedback for both training and evaluation.The method is automatic and supports collecting training data at scale and constructing live evaluation benchmarks.Source: §7 · pp. 8–9
• 5.49%/3.49% Pass@1 gains for Qwen2.5Coder on HumanEval/RepoEval follow training with execution feedback from REPOST-TRAIN.Source: §7 · p. 9
• REPOST-EVAL benchmarks 12 code LLMs, and GPT-4o achieves 39.53 Pass@1, indicating substantial room for improvement on the benchmark.Source: §7 · p. 9

Scale of the Datasets

• The datasets use only 1,000/200 GitHub repositories, although REPOST could in principle extend to a wider range.Source: §8 · p. 9

Context for Code Generation

• Training and benchmarking use only ground-truth context; other context types could be used to augment training or benchmark models.Source: §8 · p. 9

Coding Agents Training and Evaluation

• The study focuses on standard code generation rather than training or evaluating coding agents, which the datasets could also support.Source: §8 · p. 9

Multiple Repo-level Tasks

• The framework is applied to function generation, leaving tasks such as issue solving, code refactoring, and code translation for future work.Source: §8 · p. 9

A.1 Data Construction Details

• The appendix lists prompts for isolating target functions and local dependencies in separate evaluation scripts and generating tests for those scripts.These are described in Tables 14 and 15.Source: §A.1 · p. 12
• Additional prompts debug evaluation scripts when the ground-truth implementation fails and improve test coverage when branches are missing.These are described in Tables 16 and 17.Source: §A.1 · p. 12
• Quality-check prompts assess whether sandboxed and original functions match in functionality and whether generated tests correctly verify the implementations.These checks are described in Tables 18 and 19.Source: §A.1 · p. 12

A.2.1 Quality Check Agreement Check

• For the functionality-equivalence check, reviewers assessed 20 randomly sampled RepoST-Eval examples using the LLM checker prompt, allowing minor differences such as sanity checks or print information.Test-correctness checks sampled 10 LLM-Yes and 10 LLM-No cases, and excluded printed or logged values because exact matching is difficult.Source: §A.2.1 · p. 12

A.2.2 Solvability Check

• In the solvability check, participants received dependency context and execution feedback, could revise or give up, and were asked about tool use and task difficulty.They were not shown the evaluation script and were asked not to use AI models.Source: §A.2.2 · p. 12

A.3.1 Evaluation Details

• Evaluation uses Oracle context: RepoEval-Func retrieves in-repository fragments using the target function as query, while RepoST-Eval uses dependency-only context following R2E.Source: §A.3.1 · p. 12

A.3.2 Training Details

• The QwenCoder training prompt receives a line break after the prompt to prevent the target output’s first token from being a line break.The passage also specifies batch size 32, warm-up ratio 0.1, and one training epoch.Source: §A.3.2 · p. 12

A.3.3 Model Behavior Analysis

• Rejection sampling leads the model to generate more comments than the base model or SFT, which the authors interpret as more interpretable and well-documented code.The explanation offered is that GPT-4o and Claude-3.5 generate more comments than the human-written function when producing candidates.Source: §A.3.3 · p. 12, Table 10 · p. 13