SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo

1 Introduction

• Unlike existing benchmarks that return experiment outcomes immediately, physical-facility trials take months, occupy scarce equipment, and cannot be revised once started.The benchmark addresses the absence of a setting combining long, scarce commitments, parallel scheduling, and noisy measurements available before the outcome.Source: §1 · p. 1
• SlowLab tests agents on a one-year campaign across four simulated greenhouse compartments, with hidden site conditions and a final policy scored in unseen weather.Agents choose when and what to plant, whether to stop crops, and which measurements to read; Full and Endpoint feedback provide different levels of process records.Source: §1 · p. 1, Table 1 · p. 2
• The three language models finished 3.99–5.60 EUR/m2 below the prior-informed baseline, and their in-season readings never changed what they planted.They ran the facility as two synchronous batches; no method, including the baseline, profited from in-season process feedback.Source: §1 · p. 2, Figure 1 · p. 2

Benchmarks of experimenting agents

• SlowLab makes trials occupy one of four units for half a year, prevents revision, and scores a final policy in weather the agent never saw.BoxingGym and SciGym return outcomes immediately at no query cost, while Vending-Bench models delayed deliveries rather than physical experiments.Source: §2 · pp. 2–3

Language models against classical optimisers

• CARE gates language-model proposals against a reference policy, while prior work found language models below classical Bayesian-optimisation baselines in gene-perturbation and molecular design.SlowLab’s baseline uses trust-region Bayesian optimisation with a prior from a standard grower setting.Source: §2 · p. 3

Crop simulators and scientific agents

• Crop simulators such as WOFOSTGym and GreenLight-Gym frame crop models as control problems, whereas SlowLab evaluates experiment design, monitoring, and evidence reading.Scientific-agent suites often start from existing data, while autonomous laboratories connect planners to instruments.Source: §2 · p. 3

3.1 Design principles

• Crops occupy compartments for 180 days, and a 365-day campaign leaves room for about two waves; stopping early discards a crop without salvage.Four compartments run in parallel, while margins arrive only at closure and in-season process measurements are noisy; site parameters remain hidden and scoring uses unseen weather.Source: §3.1 · p. 3

3.2 Problem formulation

• Each experiment grows a crop under a fixed policy in one of four compartments from a planting date, and its contribution margin is released only when the crop closes.A campaign allows at most 16 experiments, 32 decision calls (start, stop, recommend), and 128 interface calls; the three evaluation weather years are not used in the campaign.Source: §3.2 · pp. 3–4

3.3 A campaign step by step

• The campaign harness lets language models start or stop crops, advance time, observe channels, and recommend a policy, with invalid actions counted as the method’s result.Analysis tools summarize the agent’s own data without advancing time, and a private evaluator scores the final recommendation.Source: §3.3 · pp. 4–5
• While crops run, eight channels are measured every 300 seconds; Full feedback exposes records as they arrive, whereas Endpoint feedback reveals only status and crop totals at closure.Messages pass an information firewall and are hash-audited so campaigns can be replayed.Source: §3.3 · pp. 4–5

3.4 Reference points

• The fixed reference is a standard grower setting without experiments; the best-known reference is a numerical-search policy on a random subset of test sites.The fixed reference indicates whether experiments paid off, while the best-known reference indicates remaining headroom and is a lower bound on attainable score.Source: §3.4 · p. 5

3.5 Validity of the benchmark

• With three evaluation years, the fixed reference scored 29.34 EUR/m2 and the main baseline 33.62 EUR/m2; searched best-known policies lay 7.56 EUR/m2 above the fixed reference.The per-site standard error of paired differences was 0.76 EUR/m2 on development sites, below the 2 EUR/m2 smallest effect of interest.Source: §3.5 · p. 5
• Twelve formal campaigns replayed with bit-identical traces and settlements on another host, and eight formal evaluations rerun after the study were bit-identical.The simulator, evaluator, and harness are deterministic given recorded model replies.Source: §3.5 · p. 5

Methods

• The three language models were evaluated against prior-informed local BO, which starts from the fixed reference and uses a development-fitted kernel unavailable to the models.The baseline uses the same interface, budget and tools; the models start from their own pre-experiment recommendations.Source: §4 · p. 5

Evaluation results

• Table 2 reports mean recommended-policy scores over 48 test sites in EUR/m2, averaged across three evaluation years; the fixed reference runs no experiments.Its score is identical in every column.Source: Table 2 · p. 6
• Language models ended near the grower setting and well below the baseline in final policy performance.Figure 3 reports site-level paired differences against the baseline and gains over each model’s pre-experiment recommendation.Source: Figure 3 · p. 6

Research questions

• The evaluation asks whether models find strong final policies, whether in-season measurements help, and whether poor recommendations reflect gathered evidence or how it is read.A Reader is the recommendation procedure applied to a campaign’s final evidence.Source: §4 · p. 6

Design and statistics

• The preregistered evaluation used 48 unseen sites, site-level paired tests and intervals, and a 2 EUR/m2 threshold for practical significance.After a field-order error, affected records were rerun under a new lock, and results use the corrected data.Source: §4 · p. 6

Evaluation results

• No method’s final policy improved from access to in-season process measurements.Figure 4 compares each method’s Full and Endpoint branches using site-level paired differences.Source: Figure 4 · p. 7

5.1 Language models end near the grower setting, well below a domain-informed optimiser (RQ1, E1)

• The models finished below the baseline by −5.60 EUR/m2 for DeepSeek, −4.71 for MiMo, and −3.99 for GLM; each difference had pHolm <0.001.Every interval lay beyond the smallest effect of interest.Source: §5.1 · p. 7
• The baseline started at 29.34 EUR/m2 versus the models’ 25.36–28.35, then gained +4.28 EUR/m2 versus their +1.27 to +2.65.The baseline’s final policies used 116 kWh/m2 per crop of supplemental light and 95 heat, versus 78–91 light and 147–160 heat for the models.Source: §5.1 · p. 7

5.2 No method profits from in-season process feedback (RQ2, E2)

• Full feedback did not improve any method’s final policy; all four Full-minus-Endpoint estimates were nonsignificant and below the 2 EUR/m2 effect of interest.The estimates were −0.30 for DeepSeek, −0.76 for GLM, −0.34 for MiMo, and −0.72 EUR/m2 for the baseline.Source: §5.2 · p. 7
• The models ran two synchronous batches, usually planting the next crop only after the first wave closed, so in-season readings could not change that planting.The schedule allowed mid-season planting, but Full branches read sensors only 5.9–7.7 times and called the process predictor 0.9–3.6 times on average before planting.Source: §5.2 · p. 7

5.3 Reading the final evidence is not the bottleneck (RQ3, E3)

• No pooled Reader swap gain differed significantly from zero, and all pooled intervals were within the 2 EUR/m2 effect of interest.Even the GP Reader gained at most +1.67 EUR/m2 on any model’s packets, a small fraction of the E1 gap.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4 Headroom, sensitivity and cost

• On searched sites, best-known policies averaged +7.56 EUR/m2 above the fixed reference; the baseline’s final policies were +4.83 relative to it.These best-known values are lower bounds, with repeated searches differing by 1.63 and 2.05 EUR/m2.Source: §5.4 · p. 8
• The E1 gap narrowed at 4 W/m2K side-wall heat exchange, and its sign held in every tested setting except MiMo at that heat exchange.Changing fruit dry-matter fraction also shifted the E1 differences; E2 was insensitive to that fraction.Source: §5.4 · pp. 8–9
• Campaign costs averaged USD 0.079 for DeepSeek, 0.048 for GLM, and 0.052 for MiMo; MiMo produced 12 format errors per Full branch versus 1.3 and 1.Source: §5.4 · p. 9

6 Discussion and limitations

• The results locate the bottleneck in where agents start, what they try, and when they plant, rather than in summarising campaign outcomes.Source: §6 · p. 9
• The baseline gap compares general-purpose agents with a domain-informed optimiser, not evidence that language models cannot do better.The baseline starts from the fixed reference and uses a development-fitted kernel, while the models receive neither; the gap also narrows with greater heat exchange between neighbouring spaces.Source: §6 · p. 9
• SlowLab is a simulation not calibrated to a real greenhouse, and its findings concern the tested facility, budget, action space, and models.The external check failed for temperature and humidity, and the sensor-noise model is an assumption.Source: §6 · p. 9
• The baseline was redesigned after pilot results, and Reader-swap packets came from different sites, limiting the strength of those comparisons.The study also tested only two environment sensitivities and cannot exclude provider-side changes during its two-part run.Source: §6 · p. 9

7 Conclusion

• On 48 unseen sites, three low-cost language models stopped near a standard grower setting, below the prior-informed local optimiser, and ran two synchronous batches that in-season measurements never changed.The benchmark and locked protocol are released so stronger agents can be measured on the same sites.Source: §7 · p. 9
• The campaign and evaluation records are slated for CC BY 4.0 release, while test-site and E3 seeds remain sealed to preserve unseen test sites.Code, locked configurations, and analysis outputs are available under the MIT licence.Source: §7 · p. 9

A Protocol, locks and audit

• The formal run completed 480 campaigns and 2,880 evaluations, with USD 18.70 total API spend under a USD 30 cap.Evaluation repeats across hosts were bit-identical, and an audit found every planned campaign and evaluation complete with no missing data.Source: A Protocol, locks and audit · pp. 11–12
• Code, configuration, prompts, model identities, seed commitments, and environment were locked before the run, with analysis code locked before any formal score existed.The preregistration documents themselves were not included in the initial lock; their hashes at that commit were recorded afterwards.Source: A Protocol, locks and audit · p. 11

B The original run and the correction

• A field-order mismatch roughly doubled day-90 predictor error from 5.8 to 12.6 EUR/m2 and reduced cross-season ranking accuracy from 0.81 to 0.69.It affected the process predictor and baseline; only 26 Full branches had received a predictor output, while Endpoint branches and shared day-0 designs did not use it.Source: B The original run and the correction · p. 12
• The fix mapped fields by name, added a deployed/offline path test, reran all 192 baseline campaigns, and replayed affected Full branches from the recorded conversations under a new lock.Changed recommendations were reevaluated and affected E3 packets rebuilt; the locked analysis was run once on corrected data.Source: B The original run and the correction · p. 12
• After correction, the baseline’s Full mean rose from 31.10 to 33.62 EUR/m2, all three E1 differences became significant, MiMo’s gain over its pre-experiment recommendation lost significance, and E3 Reader-swap losses disappeared.The original Reader-swap losses had been significant for all three models and largest on the baseline’s packets.Source: B The original run and the correction · p. 12

C Baseline redesign

• The plain asynchronous GP/BO scored −7.30 EUR/m2 relative to the fixed reference in the eight-site pilot, with repeat SD 11.84 EUR/m2, so it was replaced before test-site campaigns.The language models’ pilot repeat SDs were 4.20–6.30 EUR/m2.Source: C Baseline redesign · p. 12
• The replacement starts from the fixed reference, explores around the best tried policy, freezes GP hyperparameters from development data, and recommends only tried policies.The design changes were set using development sites only.Source: C Baseline redesign · pp. 12–13
• The selected 35% initial trust-region radius was the largest of the three candidates, whose mean development scores were 36.38, 37.79, and 38.13 EUR/m2.The fixed reference scored 34.92 EUR/m2 and plain BO 27.86 EUR/m2; the radius rule allowed no further candidates.Source: C Baseline redesign · p. 13

D External validity check

• Air-temperature RMSE was 2.579 °C against a 2.04 threshold and humidity RMSE was 9.983 percentage points against 8.50, while CO2 passed; benchmark numbers are not real-greenhouse predictions.The check drove the 2019 challenge-team actions through a one-shot temporal holdout with thresholds fixed in advance.Source: D External validity check · p. 13

E Agent interface details

• The harness gives each call a task, tool catalogue, state digest, six recent turns, and a maintained notebook, then routes the model’s JSON reply to an executor or analysis tool.Malformed replies and invalid actions count as the method’s own result.Source: E Agent interface details · p. 13, Figure 6 · p. 14
• Analysis tools summarize only evidence already received: crop records, channel summaries, outcome-model fits and predictions, process predictions, and feasible policy candidates.The tools do not advance time.Source: E Agent interface details · pp. 13–14
• The harness caps prompts at 80,000 characters and 200 model calls, and forces a final-day recommendation when the campaign budget is nearly spent.Older results become one-line stubs, while the six most recent are shown in full.Source: E Agent interface details · p. 14

F Example campaign

• Figure 7’s bars encode crops in compartments C1–C4, grey marks cleaning, and red ticks mark channel reads; the models plant the second wave only after the first closes, unlike the baseline’s mid-season staggering.Table 4 presents the Full branch of the test-site-0 DeepSeek campaign with the model’s stated reasons.Source: F Example campaign · p. 14, Table 4 · p. 14, Figure 7 · p. 15

• Crops are stopped if indoor air stays below 5 °C or above 40 °C for an hour, or if they fail physiologically.Source: G Environment specification · p. 15

Environment card

• The environment combines site parameters, audited weather, a pinned GreenLight model, the task contract, agent tools, and a private evaluator.Its test set contains 48 sealed sites; campaign transcripts and evaluation records are released, while private parameters and simulation traces are not.Source: H Environment card and broader impact · p. 15

Broader impact

• SlowLab is not calibrated to a real greenhouse, and its results are not advice for real greenhouse operation.Source: H Environment card and broader impact · p. 15
• The authors argue for caution about real-facility deployment because tested agents left most available improvement unused and did not act on in-season information.Source: H Environment card and broader impact · p. 15

Environment card

• Table 5 distinguishes declared assumptions without a calibration source from ranges constrained by published values.Source: Table 5 · p. 16