rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

1 INTRODUCTION

• The study targets research “taste”: judging which experiments are worth pursuing before their outcomes are known.Small experiments can test a direction cheaply, but are useful only when outcomes are not obvious and expectations are accurate enough to guide action.Source: §1 · p. 1
• Prior evaluations assess task execution, proposal judgments, or outcome prediction; this study compares agents’ expectations with outcomes for every proposal from a recent paper.The models propose five minimum viable experiments, or inexpensive tests intended to indicate whether a direction merits further investment.Source: §1 · pp. 1–2
• Across seven seed papers and 415 MVEs, 194 were judged sound, valuable, and new.The study also compares agents’ forecasts and rankings with those from a fresh instance of the same model given only the paper and registrations.Source: §1 · p. 2
• Agents’ first choices matched auditors’ most valued MVE more often than a random pick, but showed no evidence of beating the experiment written first.Their forecasts beat a base rate but were not measurably more accurate than those of the same-model judge.Source: §1 · p. 2

Research agent systems

• Research-agent benchmarks typically give agents a fixed task, while this study asks them to choose what to test from a seed paper.Other benchmarks assess fixed-metric tasks, reproducing published experiments, or research problems drawn from recent or unpublished work.Source: §2 · p. 2

Evaluating research ideas

• Research-idea evaluations often judge proposals before execution, but executing ideas from an earlier study removed the model ideas’ advantage over human ideas (Si et al., 2026a).Other studies report that automated judges only partly reproduce expert preferences, are little better than chance, or favor proposal approval.Source: §2 · pp. 2–3

Predicting experimental outcomes

• Outcome-prediction studies test forecasts on paired ideas, programs, numeric results, or training recipes, but some never check unselected hypotheses at full scale.Other work runs every candidate to train hypothesis scorers or learn from executed ideas; calibration studies on coding tasks find agents overconfident about success.Source: §2 · p. 3

Research teams

• Research-team systems divide hypothesis generation, critique, and ranking among agents, but reported gains depend on task and coordination, and equal-computation comparisons often favor a single agent.One lead-agent system improved research-style information-task performance at substantially higher token cost.Source: §2 · p. 3

3.1 SEED PAPERS

• Seven recent NLP papers with released code were selected, excluding work requiring large GPU training runs or models above 8B parameters.Agents received each paper’s full text, page images, and code access, but no summary, suggested research direction, or hypothesis.Source: §3.1 · p. 3

3.2 AGENTS AND HARNESSES

• Six models each worked under single-researcher and research-team harnesses; teams used two independent proposers, one critic, and a lead who finalized five experiments.The lead lacked tools, while agents in both harnesses used the same literature-search tool and had the same limits.Source: §3.2 · pp. 3–4

3.3 PROTOCOL

• Before execution, each agent registered five MVEs with a hypothesis, closest prior work, and a threshold event, then forecast each event and ranked the proposals.All MVEs ran regardless of rank; fresh same-model judges saw only the seed paper and registrations, without code, agent reasoning, forecasts, or model identity.Source: §3.3 · p. 4

3.4 EXECUTION AND EVALUATION

• Executors attempted every MVE under shared compute limits, and auditors judged delivered experiments for faithfulness, validity, and research value.Executors saw only the seed paper, code, and assigned MVE; reruns determined whether registered events occurred, while auditors used majority judgments for soundness and median research value.Source: §3.4 · pp. 4–5
• Three reviewers separately assessed each delivered MVE’s relation to the seed paper, idea obviousness, and evidence for its hypothesis.Table 1 compares single-researcher and research-team conditions, reporting mean counts of sound, valuable, and new MVEs and counts for other experiment categories.Source: §3.4 · p. 5, Table 1 · p. 5

4.1 QUALITY OF THE MVES

• GPT-6 Luna MVEs were sound as often as GPT-6.1 Sol’s, but their evidence was informative less often—38% versus 74%, respectively, under the reviewers’ informative-evidence criterion.The four larger models proposed about three sound, valuable and new MVEs in five, while the two smaller models proposed about one.Source: §4.1 · p. 5

4.2 KINDS OF EXPERIMENTS

• 66% of delivered MVEs built on the seed paper, 24% checked a claim, and 5% opened a suggested direction; GPT-6 Astra and GPT-6.1 Sol proposed two thirds of beyond-paper, non-obvious MVEs.44% matched an obvious follow-up; on one seed paper, all eight conditions of the four larger models first proposed the same correctness check.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 REGISTERED EVENTS AND EVIDENCE

• 25% of sound-MVE events occurred for GPT-6 Astra and GPT-6.1 Sol, versus 52% for Claude Fable 5.1 and Claude Opus 5.5; failed events still often yielded informative evidence.Reviewers found evidence informative for 50% of sound MVEs whose event failed, while evidence was inconclusive for 20% whose event occurred.Source: §4.3 · p. 6

4.4 FORECASTS

• The designing agent’s forecasts did not measurably improve on a fresh same-model reader’s (Brier-score difference −0.006; 95% CI −0.021 to 0.009).Both beat the cross-seed base rate (Brier score 0.240), while the mean of six judges’ probabilities scored 0.179 against the agent’s 0.192.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 FIRST CHOICES

• The first choice was auditors’ most valued MVE in 38% of conditions versus 20% for a random pick, but showed no evidence of beating the MVE written first.Agents wrote obvious follow-ups first in 61% of conditions versus 35% for the fifth; their first-choice event occurred in 47% of conditions versus 29% for the last choice.Source: §4.5 · p. 7

4.6 SINGLE RESEARCHERS AND TEAMS

• Teams proposed 0.71 more sound, valuable and new MVEs out of five on average, but the difference was not statistically significant (95% CI −0.10 to 1.52).Teams changed little in the kinds of experiments proposed; they more often used confidence bounds and stated lower probabilities.Source: §4.6 · pp. 7–8

4.7 EXECUTION AND RESOURCES

• A condition cost $15.31 for a single researcher and $23.93 for a team; per $100, they produced 13 and 11 sound, valuable and new MVEs, respectively.Executors delivered 397 of 415 MVEs on the first attempt; median implementation took 12 minutes, and auditor judgments of unsoundness varied by executor.Source: §4.7 · p. 8, Table 4 · p. 8

5 DISCUSSION

• Most experiments were sound, but most tested the seed paper or extended it in an obvious way, and few opened a new direction.The authors distinguish checking a paper’s claims from proposing new research and recommend evaluating these capabilities separately.Source: §5 · p. 8
• GPT-6 Astra and GPT-6.1 Sol more often proposed non-obvious experiments beyond the paper, while Claude Fable 5.1 and Claude Opus 5.5 stayed closer and the smaller models more often left questions open.The two models’ stricter registered tests passed less often; a small model teamed with a capable executor yielded about as many sound and valuable experiments per dollar as the most efficient large model, though fewer beyond the paper.Source: §5 · pp. 8–9
• A small model working as a team with a capable executor yielded about as many sound and valuable experiments per dollar as the most efficient large model, but fewer beyond the paper.Proposal costs differed far more between models than MVE execution costs.Source: §5 · p. 9
• Because every experiment ran under the same limits, rankings stated priorities but did not determine what was implemented, leaving budget allocation untested.The authors propose a follow-up giving agents a fixed research budget to allocate and revise after initial results, with human researchers as a baseline.Source: §5 · p. 9

6 LIMITATIONS

• All study labels come from models, so the evaluation lacks a human baseline.Source: §6 · p. 9
• Auditors judged soundness and value after seeing outcomes, and two of the three were also models evaluated and used as executors.Source: §6 · p. 9
• Every MVE was attempted regardless of rank, so rankings did not determine which experiments ran.An agent whose ranking determines what runs may rank with more reasoning.Source: §6 · p. 9
• Complete registrations may build part of the forecasting result into experiment design, and each condition ran only once, leaving repeat-run variability unmeasured.Source: §6 · p. 9

7 CONCLUSION

• Six frontier models proposed small experiments from recent NLP papers, forecast whether each would pass a threshold they set, and ranked them before a separate executor attempted every experiment.Source: §7 · p. 9
• Agents proposed experiments often judged sound, valuable, and new, but their own reading and search did not measurably improve forecasts or rankings over a same-model judge given only the paper and registrations.Source: §7 · p. 9

• Seed papers had to be arXiv cs.CL papers first posted between 1 August and 1 October 2026 and listed among Hugging Face daily papers.Source: §A.1 · p. 14
• Each eligible seed paper needed released code and to support informative follow-up experiments on one GPU within Appendix A.3 limits.Source: §A.1 · p. 14
• Table 5 lists the seven selected seed papers in the order they were run.Source: Table 5 · p. 14, §A.1 · p. 14

• An MVE is one registration in a fixed form, with the registration fields shown in Table 6.Source: §A.2 · p. 14, Table 6 · p. 14
• Each agent supplied its five registrations together as one JSON object.Source: §A.2 · p. 14
• A registered event consists of one statistic, a relation, and a threshold.Source: §A.2 · p. 14
• Agents used the greater-than relation for 384 MVEs and the less-than relation for 31.Source: §A.2 · p. 14

A.3 IMPLEMENTATION COMPUTE

• Executors used one A100 GPU, 12 CPU cores, and 142 GB of memory, with no keys for hosted API models.Source: §A.3 · p. 15
• Experiments could run open models of 8B parameters or fewer, with 60 model calls and a 45-minute machine runtime limit.Implementation time was limited to 90 minutes, followed by a 35-minute rerun on a new machine.Source: §A.3 · p. 15
• For the seventh seed paper, 53 of 60 MVEs were implemented and rerun on H200 machines under the same limits.Source: §A.3 · p. 15
• If an attempt delivered nothing, another executor made one independent attempt; machine failures were repeated rather than counted as attempts.Source: §A.3 · p. 15

The audits

• Three auditors judged faithfulness and validity by majority and research value by median, with research value scored from 0 to 3.Source: §B.1 · p. 15

Planted defects

• In a planted-defect check, the majority flagged 8 of 10 defective copies and none of their 10 originals.The check used copies of ten delivered MVEs from seed paper 1, rerun and audited in new contexts.Source: §B.1 · p. 15

Agreement between auditors

• Auditors agreed moderately on research value and validity but poorly on faithfulness.Table 8 reports Cohen’s kappa for each auditor pair.Source: §B.1 · p. 15, Table 8 · p. 15

The rerun

• For 407 delivered MVEs with both values available, 335 executor–rerun values were identical and 403 had the same registered event.Across all 412 delivered MVEs, the saved event matched the registered relation and threshold applied to the rerun value.Source: §B.1 · p. 15

B.2 NOVELTY

• Claude Opus 5.5 labeled 387 of 415 MVEs distinct and 28 close variants; GPT-6.1 Sol labeled 298 distinct, 115 close variants, and 2 equivalent, with kappa 0.27.Both judges searched arXiv as of the evaluation day; the novelty prompt defined equivalent, close variant, and distinct labels.Source: §B.2 · p. 16, §D.3 · p. 21

C THE REVIEW

• Reviewers received the seed paper, registration, and delivered experiment evidence without model, forecast, ranking, audit, or novelty information.Decisions used at least two of three reviewers; the protocol defined non-obviousness and whether an experiment went beyond the paper.Source: C THE REVIEW · p. 16

D.1 THE AGENT

• Agents registered five MVEs before forecasting and ranking them; team leads selected or revised five drafts after independent proposals and critique.Registrations included the hypothesis, method, statistic, threshold, controls, and the larger study the MVE would justify.Source: §D.1 · pp. 18–19

D.2 THE JUDGES

• Judges predicted each committed experiment’s event probability and ranked the five by worth pursuing, with experiments shown in random order.A separate engineer implemented each experiment from its registration without seeing the proposer’s reasoning or other experiments.Source: §D.2 · p. 20

D.4 EXECUTION

• Executors were instructed to preserve the registered hypothesis, method, event, controls, and evaluation, making only the smallest change needed if implementation was infeasible.The reported event was determined from the registered statistic’s value alone.Source: §D.4 · p. 22

D.5 EVALUATION

• Auditors were blinded to which models proposed or implemented each experiment and were asked to list every departure from registration and whether it could change results.The audit considered the registration, engineer’s notes, fresh-machine evaluation output, and submitted files.Source: §D.5 · pp. 22–23

B.2 NOVELTY

• Auditors classify whether an experiment reruns, checks, varies, explains, or builds on a paper’s claims, or pursues a new direction.The classification turns on the hypothesis, not controls, side analyses, or whether the experiment mentions the paper.Source: §D.5 · p. 25
• An idea’s obviousness is estimated by how often it would appear among ten follow-up lists from researchers who know the paper’s area and literature.The rubric distinguishes most (six or more lists), some (three to five), few (one or two), and none; complexity, polish, and whether it has been published do not determine obviousness.Source: §D.5 · p. 25
• Auditors assess who would change their actions or beliefs if the question were answered, from nobody to people across several research topics.They also check whether the plan relies on a claim about the paper that its text shows to be false.Source: §D.5 · pp. 25–26
• Design answerability is judged from the plan: auditors distinguish a clear yes from conditional answerability or a design that cannot answer its stated question.Small sample size alone is not a reason to deem a design unanswerable or conditional.Source: §D.5 · p. 26
• For each run, auditors assess whether the evidence supports, contradicts, mixes, or fails to resolve the hypothesis, then recommend a larger study, another experiment, repair, stopping, or no decision.A registered threshold’s being met or missed does not by itself determine the evidence judgment or next step.Source: §D.5 · pp. 26–27

D.5 EVALUATION

• The template asks for answerability and evidence judgments, plus a next-step choice such as a larger study, repair, or stop.It also includes placeholders for a registration, engineer notes, and protected evaluation.Source: §D.5 · p. 27