PaperlayerView exact revision on arXiv

Source-linked AI summary

Dockerless: Environment-Free Program Verifier for Coding Agents

Wenhao Zeng, Yuling Shi, Xiaodong Gu, Chao Hu, Chaofan Wang, Yuhao Cui, Hongting Zhou, Mengnan Qi, Jianqiao Wangni, Zhaojian Yu, Shuzheng Gao, Kai Cai, Shilin He

arXiv:2606.28436v1cs.SEcs.AI

TL;DR

Execution-based patch verification requires costly per-repository environments and is unreliable for codebases without reproducible setups or usable tests. Dockerless instead verifies patches through agentic, evidence-grounded repository exploration, and its environment-free pipeline matches environment-based post-training while outperforming open-source verifiers by 14.3 AUC points.

  • Problem

    Execution-based verification incurs substantial per-repository setup costs and is unreliable or infeasible for codebases lacking reproducible environments or usable test suites.

  • Method

    Dockerless derives verification questions and uses parallel sub-agents to gather repository evidence before aggregating a correctness score for each candidate patch.

  • Results

    Dockerless outperforms the strongest open-source verifier by 14.3 AUC points, while its fully environment-free post-training pipeline matches environment-based post-training performance.

  • Takeaways & Limitations

    Dockerless supports coding-agent post-training with zero per-repository setup, including SFT trajectory filtering and RL reward computation.

Abstract

from arXiv · show

Program verifiers play a central role in training coding agents, including selecting trajectories for supervised fine-tuning (SFT) and providing rewards for reinforcement learning (RL). Standard execution-based verification requires running unit tests inside per-repository environments such as Docker images, incurring substantial environment setup costs. We propose Dockerless, an environment-free agentic patch verifier that evaluates generated code patches without executing them. Rather than simply matching candidate patches to references, Dockerless judges patch correctness using evidence gathered through agentic repository exploration. On a verifier evaluation benchmark, Dockerless outperforms the strongest open-source verifier by 14.3 AUC points. Using Dockerless as both the SFT trajectory filter and the RL reward enables a fully environment-free post-training pipeline. The resulting model reaches 62.0%, 50.0%, and 35.2% resolve rate on SWE-bench Verified, Multilingual, and Pro, respectively. It surpasses the Qwen3.5-9B baseline by 2.4, 8.7, and 2.9 points, matching environment-based post-training.

1 Introduction

Dockerless is an environment-free agentic verifier that judges code patches by actively exploring repository evidence instead of executing tests or matching surface-level diffs. It supports environment-free SFT filtering and RL rewards, outperforming the strongest open-source verifier by 14.3 AUC points while enabling post-training comparable to environment-based methods.

  • Motivation: Execution-based verification incurs substantial overhead from Docker image construction, dependency resolution, test identification, and execution-script maintenance.Automated pipelines also succeed on only a limited share of candidate repositories.
  • Motivation: Existing environment-free verifiers score patches from surface-level information without inspecting repositories, limiting their suitability for complex SWE tasks.Shared base images reduce setup costs but do not eliminate the verifier bottleneck.
  • Dockerless: Dockerless derives verification questions from the issue and reference patch, dispatches parallel sub-agents to gather repository evidence, and aggregates their answers into a correctness score.Its inputs include the issue description, reference patch, and candidate patch.
  • Post-training: Dockerless enables a fully environment-free post-training pipeline for rollout collection, SFT trajectory filtering, and RL reward computation with zero per-repository setup.The pipeline runs on a minimal base image.
  • Results: 14.3 AUC points: Dockerless outperforms the strongest open-source verifier on a verifier evaluation benchmark.The resulting fully environment-free post-training pipeline achieves performance comparable to standard environment-based post-training.

2 Methodology

Dockerless replaces repository-specific test execution with an environment-free verifier that scores candidate patches using evidence from agentic repository exploration. The verifier is trained on execution-labeled trajectories and then used for environment-free rejection-sampling fine-tuning and reinforcement learning.

  • Environment-free verification: Standard verification executes held-out tests in Docker-based repository environments, but constructing these environments is prohibitively expensive and many codebases lack reproducible setups or usable test suites.The environment-based signal is binary: it indicates whether the tests in the repository-specific environment pass under the candidate patch.
  • Verifier design: Dockerless asks repository-grounded verification questions, gathers sub-agent answers, and aggregates the evidence to judge whether a candidate patch resolves the issue.Questions cover the intended fix location and behavior, confirming tests or assertions, and possible regressions elsewhere in the repository.
  • Verifier training: Teacher-generated question-answer-judge trajectories are rejection-sampled to retain only judgments matching execution labels, producing end-to-end training data for a shared backbone.The backbone is shared across question generation, sub-agent exploration, and final judging, and is trained with next-token cross-entropy over the full output sequence.
  • Environment-free post-training: Environment-free RFT collects rollouts in a minimal Linux image and uses Dockerless scores to retain the top-K final patches for SFT, avoiding per-repository environment instantiation.This replaces unit-test-based rejection sampling with Dockerless as the rejection signal.
  • Environment-free post-training: Environment-free RL uses Dockerless scores as GRPO rewards for rollouts collected without repository-specific environments, averaging M independent evaluations to improve reward stability.For each rollout’s final patch y_i on issue x, the reward is r_ϕ(x, y_i).

3 Experimental Settings

The experiments evaluate agent resolve rates on three SWE-bench variants and verifier quality on a balanced 776-sample trajectory benchmark. They use OpenHands under environment-based and environment-free protocols, compare against same-scale agent and verifier baselines, and build on Qwen3.5-9B with rejection-sampled training data.

  • Benchmarks: The evaluation covers SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro for agent resolve rate.
  • Benchmarks: 776 samples form a balanced trajectory-level benchmark for evaluating the verifier, including 500 from SWE-bench Verified and 276 from Multi-SWE-bench Flash.
  • Evaluation protocol: OpenHands serves as the default agent scaffold with a maximum of 150 turns across the evaluation protocols.Environment-based evaluation uses each repository’s Docker image, whereas environment-free evaluation uses minimal Ubuntu 22.04 LTS with only the base-commit checkout.
  • Baselines: Agent models are compared with SWE-Gym-7B, SWE-Dev-7B, SWE-Lego-8B, and Qwen3.5-9B, all under 10B parameters.
  • Baselines: Dockerless is compared with four zero-shot frontier LLM judges and four trained verifiers.The named frontier judges are DeepSeek-V3.2, Kimi-K2.5, GLM-5, and GPT-5.4; trained baselines include SWE-Gym Verifier, R2E-Gym Verifier, OpenHands Critic, and DeepSWE Verifier.
  • Implementation details: Qwen3.5-9B is the backbone for Dockerless and downstream post-training, while Dockerless training uses rejection-sampled trajectories from 3.7K execution-labeled issues and K=2−4 verification questions.

4 Results

Dockerless enables environment-free verification and post-training while matching or exceeding environment-based and open-source baselines. Its strongest results come from agentic repository exploration, effective trajectory filtering, and low-overhead reward evaluation.

  • Post-training results: 62.0%, 50.0%, and 35.2% resolve rates on SWE-bench Verified, Multilingual, and Pro establish Dockerless-RL-9B as the strongest open-source performance.The model improves over Qwen3.5-9B by +2.4, +8.7, and +2.9 points, respectively.
  • Verifier evaluation: 81.0 AUC on SWE-bench Verified and 72.1 AUC on Multi-SWE-bench Flash outperform every verifier baseline.Dockerless improves over the strongest trained open-source verifier by 14.3 and 9.2 AUC points, respectively.
  • SFT and trajectory filtering: 60.6 vs. 58.2 on Verified, 47.7 vs. 44.3 on Multilingual, and 35.3 vs. 32.0 on Pro show Dockerless 4K outperforms Random 4K.All 16K reaches 58.8, 41.3, and 31.9, indicating that unfiltered env-free trajectories do not improve over the base model.
  • SFT and trajectory filtering: 60.6 vs. 60.0 on Verified, 47.7 vs. 48.3 on Multilingual, and 35.3 vs. 33.9 on Pro show Dockerless filtering matches env-based SFT data.Dockerless 4K selects trajectories from an env-free pool, while Env-based 4K uses trajectories collected with per-repository environments.
  • Verification questions: 81.0 AUC at K=4, up from 78.3 AUC at K=0, shows verification questions improve Dockerless, while 79.6 at K=6 and 80.3 at K=8 fluctuate.Additional questions can introduce redundant or noisy evidence.
  • RL latency: 2308s average agent-rollout time dominates RL latency, while Dockerless reward evaluation adds 41–180s and accounts for only 7.2% of per-rollout time.Dockerless is slower to evaluate than the other verifier rewards, but reward evaluation remains a small share of total rollout cost.

5 Related Work

LLMs have advanced from code generation to real-world software engineering, while SWE agents are commonly post-trained with SFT followed by RL. Related verifier work ranges from fixed-prompt scoring models to agentic repository-aware approaches, with earlier methods not using tools or repository inspection during scoring.

  • LLMs have progressed from generating simple code snippets to addressing real-world software engineering tasks.
  • SWE agents are typically post-trained using a two-stage SFT-then-RL recipe on scaffolds such as SWE-agent and OpenHands.SFT uses curated or execution-filtered trajectories.
  • LM verifier research spans execution-trained classifiers, a scaled 30B mixture-of-experts critic, group-wise textual reasoning, and rubric-supervised or RL-distilled variants.These methods score patches from a fixed prompt.
  • Earlier fixed-prompt verifiers do not call tools or inspect the repository at scoring time.

6 Conclusion · B ENV-FREE EVALUATION RESULTS B.0.0.0

Dockerless is an agentic patch verifier that evaluates code by actively exploring repositories without requiring per-repository environments. It also supports an environment-free coding-agent post-training pipeline by filtering SFT trajectories and providing RL rewards.

  • 6 Conclusion: Dockerless scores patches through active repository exploration without requiring a per-repository environment.Its verification process avoids per-repository environment setup.
  • 6 Conclusion: Dockerless serves as both the trajectory filter for SFT and the reward signal for RL.Together, these roles enable fully environment-free post-training for coding agents.
  • 6 Conclusion: The resulting post-training pipeline is fully environment-free for coding agents.The paper frames this pipeline as an alternative to per-repository environment-based processing.

A Frontier-model env-base vs. env-free

Across four frontier models on SWE-bench Verified, Multilingual, and Pro, env-free evaluation retains most env-based performance despite removing per-repository environments. This makes agent rollouts feasible at scale, while correctness filtering and RL remain blocked without test execution.

  • Four frontier models were evaluated under env-based and env-free settings on SWE-bench Verified, Multilingual, and Pro.The env-based setting used per-repository Docker images and held-out tests; env-free evaluation used minimal Ubuntu without repository dependencies or test execution.
  • 13.9 points of resolve rate was the maximum env-free cost across four models and three benchmarks, versus 7.1 points on average.
  • GPT-5.4 stayed within 3.0–4.0 points of its env-based counterpart across all three benchmarks.
  • Env-free rollout collection is largely feasible at scale, but without test execution the verifier lacks a built-in correctness signal for filtering or RL rewards.Dockerless is introduced to close this verifier-side gap.

B Env-free evaluation results

The paper re-evaluates the Table 1 models under env-free conditions using only a minimal Ubuntu 22.04 LTS image and the repository checkout. Dockerless-RL-9B preserves its cross-environment lead as the strongest sub-10B model on every benchmark.

  • Setting: Env-free evaluation uses a minimal Ubuntu 22.04 LTS image with only the repository checkout at the base commit.It omits per-repository Docker, a test runner, and pre-installed dependencies.
  • Ranking: Ranking is preserved across environments, with Dockerless-RL-9B remaining the strongest sub-10B model on every benchmark.Its env-free scores are 53.8, 42.3, and 30.6.
  • Ranking: Dockerless-RL-9B leads Dockerless-SFT-9B by 1.2–1.8 points under env-free evaluation.The passage reports this margin across benchmarks.
  • Ranking: Dockerless-RL-9B also leads the env-based- model under env-free evaluation.The supplied passage truncates the comparison before reporting its full baseline name or margin.

C DATASET CONSTRUCTION C.3 … C.3 Verifier evaluation benchmark

Dockerless evaluates patches without per-repository environments and supports an environment-free training and deployment pipeline. Its verifier benchmark and deployment results show stronger performance and greater robustness than environment-trained alternatives.

  • C.3 Verifier evaluation benchmark: 3.8, 5.6, and 3.4 points are the respective advantages over Env-SFT-9B on Verified, Multilingual, and Pro under env-free deployment.Dockerless-SFT-9B and Dockerless-RL-9B were trained without test execution and remained the strongest models when deployment also forbade test execution.
  • C.3 Verifier evaluation benchmark: 9.4, 7.1, and 6.8 points are the average env-based-to-env-free score drops for Env-SFT-9B, Dockerless-SFT-9B, and Dockerless-RL-9B, respectively.Models trained on env-free rollouts show smaller deployment gaps than the environment-trained baseline.
  • C.1 Agentic verifier training data: 3.7K unique issues comprise the cleaned verifier-training corpus assembled from execution-labeled patches in SWE-Gym and Multi-SWE-RL.Examples are retained when predicted verdicts match held-out-test verdicts, with trajectory-length filtering, malformed-exchange removal, and a negative-to-positive ratio capped at 4:1.
  • C Dataset Construction: The benchmark is disjoint from the training corpus, which uses SWE-Gym and Multi-SWE-RL while evaluation uses SWE-bench Verified and Multi-SWE-bench Flash.This separation distinguishes verifier training data from the evaluation benchmark.
  • C.2 Env-free rollout data: 16K env-free rollouts are collected on SWE-Rebench-v2 from OpenHands using only the issue and base-commit repository in a minimal Linux image.No per-repository Docker image is used, while standard developer utilities may provide execution feedback.
  • C.3 Verifier evaluation benchmark: 500 SWE-bench Verified and 276 Multi-SWE-bench Flash samples form a balanced trajectory-level benchmark for comparing Dockerless with prior verifiers.Trajectories come from SWE-agent and OpenHands runs, split 1:1, with each issue–candidate-patch pair labeled positive or negative through standard evaluation.

D Training Details … D.3 Env-free RL

Dockerless training combines an agentic verifier, environment-free SFT, and GRPO-based RL. The verifier explores repositories through parallel sub-agents, while its scores filter rollouts for SFT and provide rewards for RL.

  • D.1 Agentic verifier: 150 optimizer steps produced the best verifier checkpoint on a held-out validation split.The verifier starts from Qwen3.5-9B and uses standard next-token cross-entropy on filtered trajectories.
  • D.1 Agentic verifier: 2–4 verification questions are generated per scoring call, with one sub-agent exploring the repository per question in parallel.Dense scores come from the softmax over the “0” and “1” verdict-token logits.
  • D.2 Env-free SFT: M=2 independent agentic passes score each candidate rollout’s final patch, with failed passes discarded before averaging.Dockerless then ranks the 16K-rollout pool and selects the top-ranked 4K rollouts globally for DRFT.
  • D.2 Env-free SFT: The SFT model is initialized from Qwen3.5-9B and trained with standard maximum-likelihood on DRFT.It uses the same AdamW configuration as the verifier.
  • D.3 Env-free RL: G=8 rollouts are sampled for each issue, and every rollout receives Dockerless scores from M=2 independent agentic passes.Failed passes are dropped, and the remaining dense scores are averaged into the per-rollout reward rϕ(x, y_i).
  • D.3 Env-free RL: The RL policy is initialized from the SFT model, and group-normalized advantages drive GRPO updates using Dockerless as the per-rollout reward source.The policy update follows the standard GRPO objective without test execution.

E Per-Language Analysis · F Latency distribution · G Prompt Templates

The analysis finds that env-free SFT matches env-base performance on high-volume languages but trails on compilation-heavy Rust and C, where compiler diagnostics provide distinctive supervision. Latency distributions are nearly identical across reward sources, while the prompts implement evidence-seeking question generation, read-only repository exploration, and binary patch judging.

  • E Per-Language Analysis: Rust (+7.0) and C (+13.3) are the two languages where SFT (w/ env) clearly wins, while Python, Go, JavaScript, Java, and PHP remain within ±2.5 points.TypeScript −13.3 and C++ −8.3 arise from splits with only 30 and 12 instances, respectively, so they are not treated as evidence.
  • E Per-Language Analysis: Compiler diagnostics explain the Rust/C gap: env-base trajectories observe type errors and link failures, whereas env-free trajectories infer them from source alone.The passage attributes the residual value of env-base supervision to compiler signal rather than test execution per se.
  • E Per-Language Analysis: The headline “env-free matches env-base” holds across high-volume languages but understates a real 7–13 point gap on compilation-heavy languages.The proposed direction is to surface compiler-style feedback in the env-free pipeline rather than scale env-base data further.
  • F Latency distribution: All three reward-source latency distributions overlap almost completely, with a single mode around 2400–3000s and a heavy tail reaching the hard timeout.Figure 10 covers 7680 RL-training rollouts and attributes the long right tail to slow rollouts rather than the verifier.
  • F Latency distribution: The reward source shifts mean per-rollout wall-clock time by less than 150s, so end-to-end RL is bottlenecked by the slowest rollouts in each group.Wall-clock time includes rollout plus reward.
  • G.1 Question generation prompt: The question generator uses the issue description and reference patch to produce 2–4 diagnostic questions tagged as location, behavior, test evidence, or edge case.Each question includes a short rationale and is intended to distinguish correct, incomplete, and incorrect fixes.
  • G.2 Sub-agent exploration prompt: The sub-agent performs ReAct-style, read-only repository exploration, issuing one non-interactive shell command per response to seek tests, failure locations, code context, and documentation.It returns reasoning in a THOUGHT section and ultimately submits an answer through a SUBMIT_ANSWER heredoc.
  • G.3 Final scoring prompt: The judge conditions on the issue, golden patch, candidate patch, and collected Q&A context, then emits a binary verdict whose logits define the continuous score rϕ(x, y).The required answer format is 0 for not solving the issue or 1 for solving it.
Loading 2606.28436v1…