VERA: Scaling Verifiable Environments for Agentic co-Evolution

1. Introduction

• Long-horizon medical research involves dozens of dependent steps, but existing environments often score only outcomes and provide no stage-level verified evidence.The introduction identifies the lack of precise, verifiable environments as a bottleneck for learning research workflows.Source: §1 · pp. 1–2
• VERA converts benchmark trajectories into restartable, stage-wise sandboxes, then alternates model training with harness edits using rubric evidence and verifier-gated acceptance.Auto-generated rubrics, executable checks, and an agent judge inspect environments before scaling.Source: §1 · p. 2, Figure 1 · p. 2
• Across tested backbones, domain performance rises with scale; only at 27B are general capabilities retained and mostly improved, and VERA transfers to unseen workflows.The unseen-workflow evaluations include Automation-Bench and AgentClinic.Source: §1 · pp. 2–3
• The authors report the first large-scale verifiable environments for joint LLM–harness self-evolution and release a corpus of 9,000+ environments.The introduction describes these as long-horizon training environments.Source: §1 · p. 3
• 10.3 and 13.0 Overall points separate the 9B co-evolved agent from the strongest single-axis baseline on CoWork and MedResearch; VERA-27B scores 71.6 on AutoCoWorkBench and 80.7 on AutoMedBench.At 27B, VERA surpasses Claude Opus 4.8 on AutoCoWorkBench and comes within 1.2 points on AutoMedBench.Source: §1 · pp. 2–3, Figure 2 · p. 3

2.1. VERA: Co-Evolution Method

• VERA alternates rubric-rewarded model training with skill edits, attributing failures to one axis at a time and retaining only verified updates.Model training freezes the harness; harness testing freezes the model, and edits must improve their target stage without end-to-end regression.Source: §2.1 · pp. 4–5

2.2. Auto-Rubric

• Binary rubric checks score observable action arguments, execution effects, artifacts, and output quality, serving as both task completion criteria and evidence of success.Executable or official checks override judge scores when they fail.Source: §2.2 · pp. 5–6
• Auto-Rubric converts benchmark workflows into independently restartable stage sandboxes and full-workflow environments, admitting only those that pass executable checks and Agent Judge verification.Stage sandboxes retain the starting workspace, query, and history; the end-to-end version uses an LLM surrogate for time-consuming tool calls.Source: §2.2 · p. 6

• Evaluation uses separate domain agents and training sandboxes, with instances excluded from training across CoWork, MedResearch, and transfer benchmarks.CoWork includes AutoCoWorkBench and SWE-Bench Verified; MedResearch includes AutoMedBench and MedXpertQA Text; Automation-Bench and AgentClinic assess transfer.Source: §3 · p. 6
• Table 3 compares VERA-27B and frontier models using benchmark-default runners, while Figure 2 evaluates complete agents with productivity harnesses.The reported rankings are specific to the evaluated configurations.Source: Table 3 · p. 7, §3 · p. 7
• At 9B, comparisons cover prompting, harness optimization, model training, and joint evolution under shared data and matched fixed-harness RL budgets; full co-evolution is tested at 4B, 9B, and 27B.Harness-only and LLM-only variants at 9B and 27B isolate each component; capability retention is measured on AIME 2026, ALFWorld, GPQA-Diamond, and IF-Bench.Source: §3 · p. 7

4.1. VERA Agentic Co-Evolution Outperforms Single-Axis Optimization

• 10.3 points: VERA-Code Agent scores 31.0 Overall versus ADAS’s 20.7 on AutoCoWorkBench; on AutoMedBench, VERA-Med Agent scores 69.1 versus 56.1 for harness-only.The full method also leads on SWE-Bench Verified and MedXpertQA Text, with 54.2 and 50.8 Pass@1, respectively; official scoring protocols differ from training rubrics.Source: §4.1 · p. 7

4.2. VERA Scales to 27B and Reaches Frontier-Level Performance

• On benchmark-default runners, VERA-27B scores 71.6 on AutoCoWorkBench and 80.7 on AutoMedBench, while productivity-harness comparisons place VERA highest Overall in both domains.It ranks second on Automation-Bench with 43.8% pass rate and third on AgentClinic with 76.4% accuracy; frontier rankings depend on runner configuration.Source: §4.2 · p. 8

4.3. Stronger Backbones Improve Agent Performance

• VERA improves over its backbone at every tested scale, while 27B has the strongest domain performance and capability retention among tested backbones.At 27B, CoWork improves on all four general-capability benchmarks; Med improves on AIME and GPQA-Diamond but loses 4.0 points on ALFWorld and 0.7 on IF-Bench.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. VERA RSI Cost Analysis

• 54.7% of estimated Medical Research RSI cost is environment scaling and curation, versus 27.9% for training infrastructure and 17.3% for verification and attribution.The approximately $240k total is a replacement-cost estimate for this pipeline, not audited expenditure or a general cost-scaling law; the timeline is illustrative.Source: §4.4 · p. 8, Figure 4 · p. 9

5.1. Recursive Self-Improvement: Weights, Harness, or Both

• Prior self-improvement methods target weights, harnesses, or both; VERA coordinates those axes through stage-wise rubric evidence, alternating updates, and development-set acceptance checks.Examples include GRPO (Shao et al., 2024), Meta-Harness (Lee et al., 2026), and COS-PLAY (Wu et al., 2026b).Source: §5.1 · p. 9

5.2. Verifiable Rewards and Judges

• VERA uses per-stage rubrics and workspace-inspecting agent judges for training, with failed executable checks overriding model judgments.Earlier rubric rewards score single responses, while agent judges had been used for evaluation rather than training (Zheng et al., 2023a; Zhuge et al., 2024).Source: §5.2 · p. 9

5.3. Environments for Training Agents

• Unlike static benchmarks and test-suite-based coding environments, VERA converts coding and medical trajectories into five-stage restartable sandboxes with verified rubrics.Admission requires that a sandbox run and score from observable evidence.Source: §5.3 · p. 10

• Environment counts and construction methods are not varied or compared.Source: §6 · p. 10
• Harness runtime settings, including reasoning effort, remain fixed and unabated.Source: §6 · p. 10
• Judging, verification, and attribution rely on GPT-5.6-Sol, and sensitivity to judge choice is untested.Executable checks constrain its judgments.Source: §6 · p. 10
• Full baseline comparisons are limited to 9B; a 27B co-evolution run costs approximately 5,000-8,000 GPU-hours.Source: §6 · p. 10
• The study compares learning algorithms, not alternative VERA update operators or reward formulations.Rubrics-as-Rewards remains fixed in the loop.Source: §6 · p. 10

7. Conclusion

• VERA uses Auto-Rubric to turn task trajectories into restartable sandboxes whose evidence-grounded feedback identifies failures and guides alternating model and skill updates.That feedback also supports verification of proposed changes.Source: §7 · p. 10
• Co-evolution outperforms updating either the model or harness alone across CoWork and MedResearch, with further gains at larger model scales.Source: §7 · p. 10

Appendix Contents

• The appendices include post-training data examples, training details, baseline configurations, and report-driven co-evolution and verifier and harness case studies.Source: Appendix Contents · p. 14

A.1. AutoCoWorkBench

• AutoCoWorkBench Lite has 40 end-to-end evaluation tasks, separate from training, gating, and PawBench-derived pools.The adapted SWE-bench tasks do not replace the standalone evaluation using the official SWE-bench protocol.Source: §A.1 · p. 15, Table 5 · p. 15
• CoWork episodes produce tool traces and workspace artifacts; the pinned evaluator scores the task while a trajectory judge assigns Agentic and five stage scores.The release protocol averages reports from three independently scored seeds.Source: §A.1 · p. 15

A.2. AutoMedBench

• AutoMedBench workflows use five stages and process cases within task tracks; the experiments use the seven-track Lite release.Case counts describe track workloads, not independent episodes or equal-weight contributions to the aggregate.Source: §A.2 · p. 15, Table 6 · p. 16
• AutoMedBench scores final artifacts and five-stage trajectories per repeat-track cell, then macro-averages valid cells equally on a 0-100 scale.Infrastructure-invalid cells are excluded, and private reference artifacts remain with the evaluator.Source: §A.2 · p. 16

B. Evaluation Protocols and Agent Harness

• The section identifies benchmark domains and evaluation protocols as the scope of the next section’s protocol description.Source: Table 7 · p. 16

B.1. Evaluation Harness Design

• Benchmark-default and productivity-harness comparisons use different configurations, and some model-harness pairs score lower after switching to a productivity harness.Harness friction is a possible explanation, not an established cause; results do not establish a general harness-quality ranking.Source: §B.1 · pp. 16–17

C.1. Stage Structure

• Restartable practice is organized into plan, setup, verify, execute, and submit stages, with task-specific actions and deliverables rather than equal-length trajectory segments.A rubric item receives credit only when recorded execution or workspace artifacts support the required behavior.Source: §C.1 · pp. 17–18

C.2. CoWork Rubrics; C.3. MedResearch Rubrics

• CoWork stage scores average ten equally weighted binary checks, while medical stage tables use six equally weighted binary items and may include specified hard gates.The medical registry adapts benchmark criteria and adds controls such as bounded pilots, output coverage, and verified submission.Source: §C.2 · p. 18, §C.3 · p. 19

C.4. Scoring Scope and Evidence Validity

• A valid observed failure can receive a low or zero score, whereas missing or invalid evidence, malformed judge output, or infrastructure failure blocks reward admission until resolved.This preserves unsuccessful attempts without treating unavailable assessment as demonstrated failure.Source: §C.4 · p. 19

D. Post-Training Data Example; D.1. SFT; D.2. RL; D.3. Data Curation

• SFT supervises one assistant decision from visible history, while RL starts from a state and reward contract and generates a new continuation rather than copying a teacher action.Curation checks preserve source history, verify resets, and exclude unsuitable candidates.Source: §D.1 · p. 20, §D.2 · p. 20, §D.3 · p. 21

E.1. RL Hyperparameters; E.2. Reward

• VERA uses full-parameter on-policy GRPO, and its end-to-end reward combines process and completion signals under a gate, with domain-specific settings and failure conditions.Group advantages are computed once per trajectory; tied groups receive zero advantage, and an all-zero-signal batch produces no update.Source: §E.2 · p. 21, Table 12 · p. 21

E.3. Co-Evolution Components; F.1. Compared Configurations and Shared Protocol

• VERA admits environments only after execution and scoring checks, attributes failures to model or harness reports, and uses verifiers to accept rewards, skill edits, and model checkpoints.The comparison draws from a shared admitted sandbox bank, while VERA keeps Rubrics-as-Rewards fixed and guides stage-wise training through attribution.Source: §E.3 · pp. 21–22, §F.1 · p. 22

F.3. ADAS Adaptation

• All 40 archived ADAS candidates were evaluated, but 13 lacked complete verifiable assessments and every eligible candidate scored zero, so the first registered candidate was selected by the tie rule.The search outcome does not establish a development performance gain.Source: §F.3 · p. 23

F.5. Full CoWork and MedResearch Results

• 31.0 Overall for Base + VERA LLM + VERA Harness (Ours) exceeds 20.7 for Base + ADAS harness on AutoCoWorkBench.The CoWork line is evaluated separately; Table 14 reports Overall/Agentic/Task scores.Source: Table 14 · p. 24
• 69.1 Overall for Base + VERA LLM + VERA Harness (Ours) exceeds 56.1 for Base + VERA Harness (LLM frozen) on AutoMedBench.The MedResearch line is evaluated separately; Table 15 reports Overall/Agentic/Task scores.Source: Table 15 · p. 24

G. Report-Driven Co-Evolution Example

• Figure 5 maps the LLM Report to model training and the Harness Report to skill edits in an illustrative synthetic CoWork example.The example follows the update and acceptance rules described in Section 2.1.Source: Figure 5 · p. 25

H. Verifier Case Studies

• The examples use historical development records and synthetic tests, so they do not establish accuracy for the main GPT-5.6-Sol configuration.Source: H. Verifier Case Studies · p. 26
• An incomplete real rollout received zero completion credit and was not flagged for cheating.Execution records showed that the task did not meet completion requirements.Source: H. Verifier Case Studies · p. 26
• In synthetic tests, admitting unfinished work without claiming success was not flagged, while requesting undeserved credit triggered a manipulation flag and zero reward.The paired inputs both described unfinished work with no tests run; the verifier also blocked the request for undeserved credit.Source: H. Verifier Case Studies · p. 26
• Completion records agreed with execution checks for all 64 audited rollouts, but that consistency check did not measure whether every cheating judgment was correct.The synthetic tests yielded two valid decisions after four earlier failed assessments, and the attempts used changing development configurations.Source: H. Verifier Case Studies · p. 26

I. Harness Case Studies

• In a historical comparison on six development tasks, the candidate completed 0/6 versus the baseline’s 1/6 and was not retained.The counts came from a saved experiment summary; full execution logs were unavailable for rechecking, and this single comparison does not establish acceptance reliability across repeated trials.Source: §I · p. 27
• The synthetic example starts a new stage workspace where an earlier output is mentioned but has not been opened there.Stage S3 saves trial-results.json; Stage S4 starts in a new workspace.Source: §I · p. 27
• A proposed skill edit instructs the agent to retrieve and inspect an earlier output, attempt recovery if it is missing, and report failure if recovery fails.For wrong contents, the instruction is to resolve the mismatch before continuing.Source: §I · p. 27
• The file-check instruction is only a proposed edit, not the edit tested in Case 3, and its performance effect has not been measured.The file-check scenario is synthetic, while the first case is an actual development comparison.Source: §I · p. 27