PaperlayerView exact revision on arXiv

Source-linked AI summary

PIPer: On-Device Environment Setup via Online Reinforcement Learning

Alexander Kovrigin, Aleksandra Eliseeva, Konstantin Grotov, Egor Bogomolov, Yaroslav Zharov

arXiv:2509.25455v1cs.LGcs.AIcs.SE

TL;DR

Automated environment setup remains difficult, limiting scalable configuration of real software repositories. PIPER combines supervised distillation and reinforcement learning with a lightweight judge reward. The resulting Qwen3-8B performs on par with or better than several-times-more-expensive models, while the proxy reward and single-model-family evaluation bound the evidence.

  • Problem

    Environment setup remains challenging, and limited automation success constrains the scalable use of real repositories for software-engineering benchmarks.

    Source: §1 · p. 1

  • Method

    PIPER distills successful scripts from a larger model, then uses RL with an execution-free LLM judge reward to refine the smaller model.

    Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4

  • Results

    PIPER performs on par with or better than more expensive models including GPT-4o and Qwen3-32B.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The authors present PIPER as an on-device-sized environment-setup model with a favorable performance-cost ratio that can run locally.

    Source: §1 · p. 2, §5.2 · p. 7

    The reward is an LLM-based proxy; ground-truth runtime feedback could provide richer training signals and potentially further improve performance.

    Source: §7 · p. 9

Abstract

from arXiv · show

Environment setup-the process of configuring the system to work with a specific software project-represents a persistent challenge in Software Engineering (SE). Automated environment setup methods could assist developers by providing fully configured environments for arbitrary repositories without manual effort. This also helps SE researchers to scale execution-based benchmarks. However, recent studies reveal that even state-of-the-art Large Language Models (LLMs) achieve limited success in automating this task. To address this limitation, we tune a specialized model for environment setup. We combine supervised fine-tuning for generating correct Bash scripts and Reinforcement Learning with Verifiable Rewards (RLVR) to adapt it to the task of environment setup. On EnvBench-Python, our method enables Qwen3-8B (a model runnable on consumer hardware) to perform on par with larger models-Qwen3-32B and GPT-4o. The training code and model checkpoints are available online: https://github.com/JetBrains-Research/PIPer.

1 INTRODUCTION

Environment setup limits the scaling of interactive software-engineering datasets because repositories must be configured to run their code. Automated setup could support scalable use of real repositories, but reported success remains low; this work combines SFT and RL to improve small open-source models.

1 INTRODUCTION

  • Scaling interactive software-engineering datasets requires configuring systems to execute sample code, a process this paper calls environment setup.The authors identify acquiring sufficient data, rather than increasingly sophisticated filtering, as a bottleneck.Source: §1 · p. 1
  • Automated environment setup could scale benchmarks using real repositories, but reported success remains limited: SWE-Rebench reports 31% success on Python repositories, while EnvBench's best result is 6.69% (22 out of 329).The EnvBench result was achieved by GPT-4o in an agentic workflow.Source: §1 · pp. 1–2
  • Manual setup takes substantial effort, while synthetic augmentation trades realism for scale in benchmark construction.SWE-Bench includes only 12 Python repositories, and collecting and maintaining it required substantial manual effort.Source: §1 · p. 1
  • The proposed SFT-plus-RL method improves the base model by more than 9× and performs on par with a model four times larger and strong closed-source baselines.The authors also report generalization across different datasets and publicly release code, weights, and generated scripts.Source: §1 · p. 2

2 DATASET

The paper evaluates environment setup across EnvBench-Python and Repo2Run, and uses Terminal-Bench to assess transfer to broader terminal tasks. The benchmarks differ in repository sets, success criteria, and interaction settings.

2 DATASET

  • EnvBench-Python evaluates scripts on 329 GitHub Python repositories using successful execution and a clean Pyright import check as its correctness criteria.Approaches receive full repository context and the base environment configuration; their outputs are dependency-installing shell scripts.Source: §2 · p. 2
  • Repo2Run covers 420 non-overlapping GitHub Python repositories and determines setup success through pytest test collection rather than static import analysis.The paper adapts the benchmark to a non-agentic task formulation as well as its original agentic setting.Source: §2 · p. 2
  • Terminal-Bench comprises 80 real-world command-line configuration tasks, with success determined by task-specific tests in a Docker environment.The evaluation uses the original implementation and the multi-turn Terminus 1 scaffold.Source: §2 · p. 2
  • Terminal-Bench tests whether a pipeline designed for single-turn Python package installation transfers to out-of-distribution, multi-turn terminal command tasks.Its tasks include compiling code, training models, and setting up servers.Source: §2 · pp. 2–3
  • The training overview contrasts SFT, which learns from teacher completions, with RL, which scores generated scripts and updates the model using REINFORCE++.For SFT, teacher and student receive repository prompts; in RL, a rule-based reward function scores each completion.Source: §2 · p. 3, Figure 1 · p. 3

3 METHOD

PIPER uses a two-stage training process: distillation on successful scripts from a larger model, followed by RL with a lightweight judge reward. The stages address scarce ground-truth scripts and costly container-based execution feedback.

3 METHOD

  • PIPER first distills executable scripts from a larger same-family model into the smaller model, then applies RL to refine its environment-setup capabilities.The pipeline extracts shell scripts from model outputs and treats a failed parse as an empty script.Source: §3 · p. 3

3.1 SUPERVISED FINE-TUNING

  • Ground-truth setup scripts are costly to obtain for hard repositories, so SFT uses distillation from a larger teacher rather than relying on expert scripts alone.EnvBench provides only a small number of expert-generated scripts, and even strong models solve only a small portion of the dataset.Source: §3.1 · p. 3
  • SFT uses 2,500 randomly selected evaluation-rollout pairs after filtering out completions without scripts or scripts that exit unsuccessfully.The solutions come from a larger Qwen3-32B model; the resulting checkpoint is the foundation for RLVR training.Source: §3.1 · pp. 3–4

3.2 REINFORCEMENT LEARNING

  • Direct execution rewards require isolated containers and create substantial technical and computational overhead at RLVR scale.The authors instead use an execution-free LLM-as-a-Judge reward designed to mimic rule-based evaluation criteria.Source: §3.2 · p. 4
  • A study of GPT-4o scripts on 40 repositories identified 11 failure patterns and three repository configuration challenges, including missing codebase dependencies and optional development packages.Reported frequent failures included unresolved imports from dependencies absent in configuration files (25%) and optional development dependencies (22.5%).Source: §3.2 · p. 4
  • The judge receives the extracted script and repository context, then predicts the script's exit code and number of Pyright issues to emulate EnvBench evaluation.The reward is based on these predicted outcomes.Source: §3.2 · p. 4

4 EXPERIMENTS SETUP

Experiments train Qwen3-8B on EnvBench tasks and evaluate it on EnvBench-Python, Repo2Run, and Terminal-Bench. The evaluation varies in repository sets, success criteria, and whether the task is single-turn or agentic.

4.1 TRAINING SETUP

  • Training uses EnvBench tasks without ground-truth labels, reserving 96 repositories for validation and using 228 for training.The model receives rule-based reward scores; five EnvBench repositories are omitted because of technical issues, and the authors report no strong indication of memorization.Source: §4.1 · p. 4
  • The study focuses on Qwen3-8B in non-thinking mode and uses a zero-shot scaffold that generates one shell script without environmental feedback.The prompt includes the task description, repository context, and base-environment information.Source: §4.1 · pp. 4–5
  • SFT uses full-weight cross-entropy training for five epochs on one H200 GPU, while RL uses REINFORCE++ for 45 training steps on four H200 GPUs.The RL reward judge uses GPT-4.1; the authors leave broader model-family and objective comparisons for future work.Source: §4.1 · p. 5

4.2 EVALUATION SETUP

  • EnvBench-Python reports pass@5, avgFixRate, and # Failed, with all metrics except pass@5 averaged over five runs.Repo2Run uses pass@5 with pytest collection as its success criterion; Terminal-Bench uses pass@10 and custom evaluation commands.Source: §4.2 · p. 5
  • The evaluation compares against GPT-5, GPT-4o, GPT-4o-mini, and Qwen3 models with 8B, 14B, and 32B parameters.Qwen3 baselines are evaluated in non-thinking mode for consistency.Source: §4.2 · p. 5
  • PIPER achieves the best performance on Repo2Run, while SFT-based models underperform on Terminal-Bench's multi-turn setting.Table 1 reports results for base models and tuned Qwen3-8B models; Repo2Run success requires a zero exit code and no test-collection errors.Source: Table 1 · p. 6

5 RESULTS

PIPER is competitive with much larger models on EnvBench at lower cost, and its training stages each contribute to performance. Cross-benchmark results show strong single-turn transfer but weaker results on multi-turn Terminal-Bench tasks.

5.1 TRAINING DYNAMICS

  • Validation rewards rise substantially from step 0 to 45; the base model starts near the reward minimum despite formatting compliance, while SFT starts with highly rated scripts and training shows little overfitting.The reward ranges from −1 for malformed scripts to 1 for perfect performance.Source: §5.1 · p. 6, Figure 2 · p. 6

5.2 EVALUATION RESULTS

  • The combined SFT-and-RL checkpoint significantly outperforms either phase alone; avg@5 rises from 2.6 for base Qwen3-8B to 13 with SFT or 11.8 with RL alone.Table 2 reports EnvBench results across training setups and metrics.Source: §5.2 · p. 7, Table 2 · p. 7
  • PIPER pass@5 (27) exceeds GPT-5 pass@1 (25), while PIPER's inference cost is more than 14 times lower.PIPER pass@3 also exceeds pass@2 for both GPT-4o and Qwen3-32B.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 GENERALIZATION

  • On held-out EnvBench and two additional datasets, PIPER performs better than or on par with strong baselines, with no strong signs of memorization.The held-out EnvBench results are detailed in Section C.Source: §5.3 · p. 7
  • On Repo2Run, PIPER records 103 successes versus 32 for base Qwen3-8B and 71 for Qwen3-32B; on Terminal-Bench, SFT-only records 2 successes versus 8 for base, while RL-only reaches 9 and combined training 4.The authors report that RL provides more robust generalization across interaction paradigms than SFT alone.Source: §5.3 · pp. 7–8

5.2 EVALUATION RESULTS

  • PIPER is competitive with Qwen3-32B and GPT-4o across EnvBench metrics, with comparable-or-better performance at a fraction of their cost and the option to run locally.Figure 3(b) compares average pass@1 over five runs with price per 1M output tokens.Source: §5.2 · pp. 6–7, Figure 3 · p. 8

6 RELATED WORK

Related work spans environment-setup benchmarks and methods, alongside research on RLVR and combinations of reinforcement learning and supervised fine-tuning.

Environment Setup

Reinforcement Learning with Verifiable Rewards (RLVR)

7 LIMITATIONS AND FUTURE WORK

The study is limited to one model, a single-turn scaffold, and a lightweight LLM-based proxy reward; the authors identify broader model coverage and runtime feedback as future directions.

Models

  • The framework is evaluated only with Qwen3-8B in non-thinking mode, leaving other model families, sizes, and reasoning models to be explored.The authors note that probing those alternatives could further verify the study’s range of applicability.Source: §7 · p. 9

Scaffold

  • Experiments use a single-turn scaffold, while extending RLVR to multi-turn interaction is proposed as a future progression.Prior work suggests multi-turn agents that iteratively use environmental feedback could bring significant improvements.Source: §7 · p. 9

Proxy Rewards

  • The lightweight LLM-based reward avoids computational overhead from scaling containerized execution, but ground-truth runtime feedback could provide richer training signals and further gains.The authors describe runtime feedback as a likely source of richer signals, not as an evaluated result.Source: §7 · p. 9

8 CONCLUSION

PIPER combines supervised distillation and RLVR with a lightweight LLM-as-a-Judge reward for environment setup. The resulting on-device-sized model matches or exceeds several-times-larger models and retains reasonable performance on Terminal-Bench.

8 CONCLUSION

  • PIPER combines SFT distillation for script writing with RLVR and a lightweight LLM-as-a-Judge reward that mimics execution checks without direct environment feedback.The two-stage training is described as requiring no ground-truth data.Source: §8 · p. 9
  • The resulting on-device-sized model performs on par or better than several-times-more-expensive models, including GPT-4o and Qwen3-32B.The conclusion reports this comparison without providing a numerical margin.Source: §8 · p. 9
  • PIPER retains reasonable performance on out-of-distribution Terminal-Bench in an agentic scaffold, indicating improvement beyond environment-setup-specific performance.The authors interpret this result as evidence of broader terminal-manipulation capability rather than task-specific overfitting.Source: §8 · p. 9

9 REPRODUCIBILITY STATEMENT

The reproducibility statement makes checkpoints, evaluation results, and training and evaluation code available. The implementation appendix also describes the zero-shot scaffold and how it gathers repository context.

9 REPRODUCIBILITY STATEMENT

  • Checkpoints for the RL-only, SFT-only, and final PIPER models, raw evaluation results, and the training and evaluation code are publicly available.The authors also plan to publish complete training run logs upon acceptance.Source: §9 · p. 9

A IMPLEMENTATION DETAILS

  • The implementation appendix introduces additional experiment details and specifies that the zero-shot scaffold follows Eliseeva et al. (2025).Its prompt uses the same baseline-script and Dockerfile context variables as that work.Source: A IMPLEMENTATION DETAILS · p. 14, §A.1 · p. 14

A.1 SCAFFOLD DETAILS

  • The scaffold gathers repository context with fixed commands in the target Docker environment, including repository listings and project configuration files.The listed files include setup and dependency metadata, Python-version declarations, environment files, and Dockerfiles.Source: §A.1 · p. 14

A.2.1 SFT TRAINING

The appendix details SFT training, points to the RL configuration, and describes the evaluation implementations used for the three benchmarks.

A.2.1 SFT TRAINING

  • SFT fine-tunes Qwen3-8B for five epochs with AdamW, cosine scheduling, learning rate 5 × 10−5, weight decay 0.01, gradient accumulation of 4, and batch size 4.Training uses bfloat16 and a 5% validation split; the training data cover 227 of 228 unique repositories, with a median of 11 samples per repository.Source: §A.2.1 · pp. 14, 16
  • The zero-shot prompt uses the same baseline-script and Dockerfile context variables as Eliseeva et al. (2025), while repository context is collected by fixed commands in the target Docker environment.Figure 4 presents this prompt scaffold.Source: Figure 4 · p. 15

A.2.2 RL TRAINING

  • RL sampling uses Qwen3's recommended non-thinking-mode parameters, with full configuration files and code available in the reproduction package.Table 4 is identified as the VeRL RL-parameter table.Source: §A.2.2 · p. 16, Table 4 · p. 16

A.3 EVALUATION DETAILS

  • Evaluation uses the original EnvBench implementation, an EnvBench-suite extension for Repo2Run's pytest test-collection check, and Terminal-Bench 0.1.1 with Terminus 1.The Repo2Run evaluation reuses the zero-shot scaffold and prompts; Terminal-Bench uses its original implementation.Source: §A.3 · p. 16

A.4 LLM-AS-A-JUDGE REWARD IMPLEMENTATION

The reward uses GPT-4.1 to simulate script outcomes, while the failure study separates script, repository, and evaluator issues. On held-out validation, PIPER retains improvement over Qwen3-8B and remains comparable to Qwen3-32B and GPT-4o.

A.4 LLM-AS-A-JUDGE REWARD IMPLEMENTATION

  • GPT-4.1 was selected for the LLM-as-a-Judge reward; adding repository context or agent tools brought no noticeable improvement.The judge simulates script execution in EnvBench Docker and predicts exit code and unresolved import issues.Source: §A.4 · pp. 16–17

B EMPIRICAL STUDY OF ENVIRONMENT SETUP FAILURE PATTERNS

C TRAIN/VALIDATION PERFORMANCE

  • PIPER variants retain substantial improvement over Qwen3-8B and are comparable to Qwen3-32B and GPT-4o on held-out validation, ranking second or third across considered metrics.GPT-5 remains strongest; combined SFT and RL outperforms SFT-only and RL-only checkpoints on per-run metrics.Source: C TRAIN/VALIDATION PERFORMANCE · p. 17, Table 6 · p. 19

Related papers

Loading 2509.25455v1…