AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Motivation

• Existing benchmarks lack controlled tools, cost accounting, standardized interfaces, authentic product-informed tasks, and comprehensive baselines, weakening fair comparison of scientific agents.These gaps make it difficult to determine whether improvements reflect agent capability, information access, extra computation, or narrow evaluation design.Source: §1 · p. 2

AstaBench

• AstaBench contains over 2400 problems spanning the scientific discovery process, multiple domains, broad skills, and many real user requests, with a standardized interface for general agents.The suite exercises literature and data understanding, planning, tool use, coding, and search.Source: §1 · p. 2

Asta Environment

• The Asta Environment provides production-grade scientific-literature search tools for controlled, reproducible evaluation while isolating core agentic abilities from information access.The benchmark’s standard tools are designed to give agents comparable access to the supporting corpus.Source: §1 · p. 2, Table 1 · p. 3

Evaluation infrastructure

• The evaluation toolkit and leaderboard account for confounders by reporting time-invariant computation costs alongside agent performance and tool usage.This makes resource requirements part of the comparison rather than treating accuracy as the only outcome.Source: §1 · p. 2, Figure 1 · p. 3

Agent baselines

• The agent-baselines suite provides nine science-optimized Asta agent classes alongside numerous baselines for systematic comparison across architectures.The broader evaluation spans 57 agents across 22 agent classes, including task-specific and general ReAct-style systems.Source: §1 · p. 3

Findings

• The experiments show meaningful progress across some aspects, but science research assistance remains far from solved despite evaluation of 57 agents across 22 classes.AstaBench is intended as a continuing, cost-aware, transparent framework for measuring progress.Source: §1 · pp. 3–4

Holistic Agent Evaluations

• AstaBench differs from existing suites by covering broad scientific task categories and domains, using product-derived tasks, standard tools, cost-aware scoring, and many agent classes.The comparison includes holistic scientific reasoning, product usage, controlled tools, confounder-aware scoring, general agents, and agent counts.Source: §2 · p. 4
• Existing holistic and science-agent benchmarks include AutoGenBench, BixBench, BrowserGym, HAL, Inspect Evals, Lab-Bench, OpenHands Evals, ScienceAgentBench, Terminal-Bench, and the Vector Institute Leaderboard.These efforts are compared with AstaBench across scope, tooling, confounder accounting, general-agent support, and agent counts.Source: §2 · p. 4

Science Benchmarks and Agents for Science

• Many science benchmarks target narrower stages such as coding, data analysis, reproduction, ideation, or literature retrieval rather than the full research pipeline.AstaBench spans many of these categories and is described as the most comprehensive evaluation of scientific agent performance to date.Source: §2 · pp. 4–5
• AstaBench introduces an end-to-end task in which an agent completes a research project from an idea to a written report and code.The task complements benchmarks focused on narrower research problems.Source: §2 · p. 5

3 ASTABENCH: A HOLISTIC SCIENTIFIC RESEARCH BENCHMARK SUITE

• AstaBench provides holistic evaluation of scientific research agents through reproducible search tools, a standard benchmark interface, time-invariant cost reporting, and compatible general-agent baselines.The suite is designed for use with new general agents through the Asta Environment and agent-eval toolkit.Source: §3 · p. 5
• The suite contains 11 benchmarks spanning literature understanding, code and execution, data analysis, and end-to-end discovery.Tasks include scientific search, question answering, literature-review tables, experiment setup and reproduction, data science, hypothesis discovery, and full research pipelines.Source: §3 · p. 5
• Benchmark outputs are scored either by LLM judges against rubrics or through programmatic evaluation, depending on the task.LLM judging is used for PaperFindingBench, ScholarQA-CS2, ArxivDIGESTables-Clean, DiscoveryBench, and both E2E-Bench variants; other tasks use programmatic evaluation.Source: §3 · p. 5

4 ASTA ENVIRONMENT

• Asta Environment is presented as a realistic, reproducible scientific research environment with standardized tools, an evaluation toolkit, a leaderboard, and numerous agents.It is described as the first such environment for agents.Source: §4 · p. 5

4.1 STANDARD TOOLS FOR AGENTS

• The scientific corpus provides production-grade literature search and retrieval tools, while the computational notebook supports stateful Python and IPython execution in a sandbox.Corpus tools include search and paper-access operations; notebook variables persist across calls, with a five-minute default cell timeout.Source: §4.1 · pp. 5–6
• AstaBench pairs task-specific subsets of standard research tools with reproducible benchmark conditions, including date-cutoff controls and predefined corpus or notebook tools.Table 2 organizes benchmarks across four task categories and identifies the standard tools and exclusive date cutoffs used for reproducibility.Source: §4.1 · p. 5, Table 2 · p. 6
• The tools are decoupled from agents through MCP, allowing agents to combine host-environment research tools with sandboxed code execution.This design supports testing code-execution agents, including agents using the CodeAct pattern.Source: §4.1 · p. 6

4.2 AGENT-EVAL EVALUATION TOOLKIT & ASTABENCH LEADERBOARD

• agent-eval adds normalized cost calculation, benchmark-suite reporting, leaderboard support, and confounder reporting on top of Inspect-formatted benchmarks.Costs use model usages logged by Inspect and a frozen community-sourced price snapshot, including cache discounts but excluding latency-related discounts.Source: §4.2 · pp. 6–7
• The leaderboard categorizes submissions by agent openness and whether they use standard, custom, or fully custom tooling.A web interface supports external submissions, interactive plots, and tables.Source: §4.2 · p. 7

4.3 AGENT-BASELINES AGENTS SUITE

• The agent-baselines Agents Suite provides many agents from 16 agent classes through a standard Inspect-compatible interface.The suite groups Asta agents optimized for scientific research with numerous baseline agents.Source: §4.3 · p. 7

Agent Design

• Specialized science tools raise Asta v0 to 53.0%, about 9% above ReAct with gpt-5 at 44.0%, but require higher development and sometimes inference costs.The benefit is accompanied by a clear engineering and task-dependent cost tradeoff.Source: §5 · p. 7

Overall Results

• 53.0% is the best overall score for an open-source agent using closed LLMs, while open-weight LLM agents reach only 11.1%.Asta v0 achieves 53.0%, whereas Smolagents Coder with Llama-4-Scout-17B-16E-Instruct achieves 11.1%.Source: §5 · p. 9

Cost-Performance Tradeoffs

• 32% at $0.04 per problem makes ReAct with gpt-5-mini the most economical model, within 21% absolute of the best-performing models.The comparison highlights a substantial quality-cost tradeoff across agents.Source: §5 · p. 9

Model Effects

• gpt-5 improves ReAct substantially on four benchmarks but harms several specialized agents, indicating that model upgrades are not a reliable recipe for broad success.The gains are +13.4% on ScholarQA-CS2, +24.8% on SUPER-Expert, +25.3% on LitQA2-FullText-Search, and +21.1% on E2E-Bench-Hard.Source: §5 · pp. 7, 9

Literature Understanding

• Literature understanding is relatively strong: Asta Paper Finder outperforms ReAct on search tasks, while Asta Scholar QA, Elicit, and SciSpace Deep Review score about 85% or higher on ScholarQA-CS2.Literature review table generation remains weaker, with recall around 43%, although Asta Table Synthesis backed by gpt-5 wins and its gpt-5-mini version costs 13% as much.Source: §5 · p. 9

Code and Execution

• Coding and execution remain bottlenecks: all but two agents score below 25% on SUPER-Expert, with ReAct using gpt-5 reaching 41%.The results identify coding and execution as major unsolved problems for science assistance.Source: §5 · pp. 9–10

Data Analysis and End-to-End Discovery

• 34% is the maximum score for automated data analysis, while end-to-end completion reaches at most 5% despite per-step scores up to about 70%.With roughly 10 steps and 70% success per step, the implied full-experiment success is approximately 3%.Source: §5 · p. 10

Conclusion

• AstaBench combines rigorous benchmarking methodology, a controlled scientific research environment, standardized agents, and an evaluation toolkit for comparing research agents.The suite supports leaderboard submissions and continuous measurement of progress.Source: §6 · p. 10
• 57 agents across 22 architectural classes show meaningful progress on individual aspects, but agentic AI remains far from solving scientific research assistance.This is the paper’s central conclusion from the benchmark experiments.Source: §6 · p. 10

Future Work

• Future work targets contamination-resistant problems, human collaboration, biomedicine coverage, complex-context handling, improved grading, and continued evaluation of new models and architectures.The authors also aim to improve open-agent performance and develop techniques specialized for scientific research.Source: §6 · p. 10

ETHICS STATEMENT

• User queries for Literature Understanding tasks were collected with consent, and the work underwent legal review.The authors state that all material presented received legal review.Source: ETHICS STATEMENT · p. 10
• The authors credit adapted benchmarks and leveraged agents, and worked with creators where possible to evaluate existing agents fairly.Named examples include Elicit, Future House, and SciSpace.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Open-source benchmarks, agents, infrastructure, experiment logs, repository commits, and date-restricted corpus access support reproducible evaluation.The framework records specific commits, including data commits, while tools restrict access to supporting documents by date.Source: REPRODUCIBILITY STATEMENT · p. 11

The Use of Large Language Models (LLMs)

• Claude Code, GitHub Copilot, and ChatGPT were used for result analysis, plotting and table code, error and reference checks, and minor writing assistance.The disclosure concerns implementation and analysis support rather than the benchmark’s scientific claims.Source: The Use of Large Language Models (LLMs) · p. 11

• Benchmark suites should represent real-world complexity through broad tasks exercising reasoning, planning, tool use, search, coding, and data analysis.Source: A PRINCIPLES FOR BENCHMARKING AGENTS · p. 16
• Standardized, realistic, reproducible environments and tools are needed to distinguish agent capability from differences in information access or infrastructure.Source: A PRINCIPLES FOR BENCHMARKING AGENTS · p. 16
• Evaluations should report computational cost and tool usage because repetition and privileged information access can inflate apparent performance.Source: A PRINCIPLES FOR BENCHMARKING AGENTS · p. 16
• Standard task interfaces should provide complete instructions, required tools, and submission affordances so general agents can be integrated without task-specific adaptation.Source: A PRINCIPLES FOR BENCHMARKING AGENTS · p. 17
• Comprehensive baseline agents with standard interfaces are necessary to identify state-of-the-art systems and support future development.Source: A PRINCIPLES FOR BENCHMARKING AGENTS · p. 17

• Agent openness ranges from open-source, open-weight systems to closed systems available only through a user interface, reflecting decreasing reproducibility.Source: B EVALUATION TOOLKIT: OPENNESS AND… · p. 17
• Agent tooling distinguishes standard evaluation tools, custom interfaces, and fully custom tools, with literature and code environments constrained by AstaBench specifications.Source: B EVALUATION TOOLKIT: OPENNESS AND… · p. 17
• Table 4 reports macro-averaged overall results for agents solving all tasks, while bold entries identify Pareto-optimal agent-cost pairs.Source: Table 4 · p. 18
• Table 5 provides Literature Understanding search results, with additional results in Table 12 and Pareto-frontier markers for agent-cost comparisons.Source: Table 5 · p. 18

• Supporting tables cover overall agents solving all tasks and category-specific results for Literature Understanding, Code & Execution, Data Analysis, and End-to-End Discovery.Source: C SUPPORTING EXPERIMENTAL RESULTS · p. 18
• Reported tables use openness and tooling labels, 95% confidence intervals, and bold formatting to identify Pareto-optimal agent-cost pairs.Source: C SUPPORTING EXPERIMENTAL RESULTS · p. 18
• The Literature Understanding QA results include additional results in Table 13 and exclude agents without an API from LitQA2-FullText evaluation.Source: Table 6 · p. 19
• The Literature Understanding ArxivDIGESTables-Clean results are reported in Table 7 with additional results in Table 14.Source: Table 7 · p. 19
• Code & Execution results are summarized in Table 8 with additional results in Table 15, while Data Analysis DiscoveryBench results appear in Table 9 and Table 16.Source: Table 8 · p. 20, Table 9 · p. 20
• End-to-End Discovery results are reported in Table 10 with additional results in Table 17, and overall all-task results are also provided in Table 11.Source: Table 10 · p. 20, Table 11 · p. 21

D FULL EXPERIMENTAL RESULTS

• Full results include all tested agent configurations, score-cost plots, and Pareto frontiers showing the best agent available at each cost.Source: D FULL EXPERIMENTAL RESULTS · p. 21
• 95% confidence intervals are computed as ±1.96 × SE, with standard errors derived from evaluation-sample variance and analytically propagated for weighted category averages.Source: D FULL EXPERIMENTAL RESULTS · p. 21
• Category-level uncertainty propagation assumes task independence, which could slightly underestimate uncertainty.Source: D FULL EXPERIMENTAL RESULTS · p. 21
• Figures 3–8 compare score and cost across Literature Understanding, Code & Execution, Data Analysis, and End-to-End Discovery evaluations using means, 95% confidence intervals, and Pareto frontiers.Source: Figure 3 · p. 22, Figure 4 · p. 23, Figure 5 · p. 24

O T Agent Model PaperFindingBench LitQA2-FullText-Search

• The Literature Understanding analyses separate search, question answering, and ArxivDIGESTables-Clean benchmarks, with date-unpinned models marked by † where indicated.Source: Figure 3 · p. 22, Figure 4 · p. 23, Figure 5 · p. 24
• The Code & Execution analysis covers CORE-Bench-Hard−, SUPER-Expert, and DS-1000, with cost shown on a logarithmic x-axis.Source: Figure 6 · p. 28

E.2 PAPERFINDINGBENCH

• PaperFindingBench evaluates ranked scientific-paper retrieval across navigational, metadata, and semantic queries rather than semantic search alone.It contains 48 navigational, 43 metadata, and 242 semantic queries, drawn from usage logs and existing literature-search datasets.Source: §E.1 · p. 32, §E.2 · p. 34
• PaperFindingBench scores navigational and metadata queries with result-set F1, while semantic queries combine estimated recall and nDCG using their harmonic mean.Estimated recall is bounded through recall@k, and the final dataset score averages individual query scores across query types.Source: §E.1 · p. 32, §E.2 · p. 35

E.1 SHORT DESCRIPTIONS

• The suite covers literature understanding through LitQA2, ScholarQA-CS2, and ArxivDIGESTables-Clean, measuring question answering, cited long-form synthesis, and literature-review table generation.LitQA2 uses accuracy and recall@30, ScholarQA-CS2 averages four answer-and-attribution facets, and ArxivDIGESTables-Clean scores entailed reference-table statements.Source: §E.1 · p. 32
• Additional benchmarks test executing research repositories, reproducing paper analyses, routine data science coding, and hypothesis discovery from datasets.SUPER-Expert, CORE-Bench-Hard−, DS-1000, and DiscoveryBench respectively evaluate repository execution, reproducibility, code correctness, and hypothesis alignment.Source: §E.1 · pp. 32–33
• E2E-Bench tests a complete research cycle spanning ideation, planning, experiment design, implementation, execution, analysis, and final reporting.Its detailed problems support controlled agent comparisons but only weakly test ideation and planning; outputs are judged across reports, code, and artifacts.Source: §E.1 · pp. 33–34

E.2 PAPERFINDINGBENCH

• PaperFindingBench is designed around challenging, realistic information needs, including complex criteria, uncommon terminology, distracting context, and cases requiring full-text evidence.The benchmark mixes query types without labeling them, reflecting the goal that literature-search agents handle or route among different search challenges.Source: §E.2 · pp. 34–35

E.3 SCHOLARQA-CS2

• ScholarQA-CS2 addresses ScholarQA-CS limitations by collecting diverse key ingredients from candidate responses and introducing LLM-as-judge methods for relevance and improved citation evaluation.Its test set contains 100 filtered English computer-science research questions, while the previously published 100-question ScholarQA-CS dataset is retained for development with updated ingredient lists.Source: §E.3 · pp. 36–37, §E.3.1 · p. 38
• ScholarQA-CS2 averages citation recall, citation precision, answer relevance, and answer coverage, using LLM judges to assess whether claims are supported, citations are relevant, paragraphs address the question, and necessary ingredients are covered.Coverage rubrics are built by extracting, clustering, and weighting answer-critical and valuable ingredients from candidate systems; citation metrics do not verify paper authenticity or quote correctness.Source: §E.3 · pp. 36–37
• ScholarQA-CS2 shows moderate system-level human–model agreement (Kendall τ = 0.467), rising to 0.800 after excluding systematically disfavored Elicit outputs, while instance-level agreement reaches 68.1%.The combined metrics agree better with human rankings than individual metrics, suggesting that their complementary weaknesses are partially offset.Source: §E.3 · p. 37
• Holding systems out of rubric construction causes significant average drops of 2.5 points for Asta Scholar QA (w/ Tables), Elicit, and SciSpace Deep Review, but insignificant drops below 1 point for OpenAI Deep Research and Perplexity Sonar Deep Research.A four-system rubric produces a 5-point average drop across evaluated systems, whereas adding more systems reduces held-in bias, though its causes and fairest mitigation remain unresolved.Source: §E.3 · p. 37

E.4–E.7 BENCHMARK COMPONENTS

• The benchmark components adapt established tasks for controlled evaluation: ArxivDIGESTables-Clean samples 100 test instances with entailment-based recall, AstaBench Expert has 45 problems scored by exact match, and CORE-Bench-Hard− has 37 CPU-compatible instances.Curation excludes generic or unrecoverable table columns; the Hard version removes files such as run scripts and precomputed results, and GPU-required instances are removed.Source: §E.4 · pp. 38–39, §E.5 · p. 39, §E.6 · p. 39

E.8 DISCOVERYBENCH

• DiscoveryBench formalizes data-driven discovery as finding a context-specific relationship between variables and evaluates predicted hypotheses against gold hypotheses using entailment and equivalence judgments across six scientific domains.Its task design supports reproducible evaluation when facets of the discovery problem, such as the context, variables, or relationship, are absent from the dataset’s language.Source: §E.8 · p. 40

E.9 E2E-BENCH

• E2E-Bench builds 50 AI/NLP research tasks from LLM-generated ideas based on highly cited ACL papers, with automatic filtering and ranking followed by expert review, and scores research-step completion across reports, code, and artifacts.Overall completion scores are near zero because all required steps must succeed, despite individual-step accuracy reaching up to ∼70%.Source: §E.9 · p. 41, Table 20 · p. 41

E.10 E2E-BENCH-HARD

• E2E-Bench-Hard constructs 50 tasks with HypER, preserving its initial proposals without E2E-Bench’s simplification step before automatic ranking and expert review.The harder dataset retains underexplored task proposals refined from relevant paper excerpts and removes infeasible tasks through manual review.Source: §E.10 · pp. 41–42

F.1 ASTA AGENTS

• Asta’s agent suite spans task orchestration, paper finding, scientific question answering, table synthesis, coding, data analysis, planning-and-acting, and autonomous discovery.Asta v0 routes detected task types to specialized solvers, while other agents implement distinct workflows for literature, code, datasets, and computational experiments.Source: §F.1 · pp. 42–43
• Asta DataVoyager coordinates specialized planning, programming, execution, and analysis agents to explore hypotheses, run statistical tests, and interpret generated plots.An orchestrator maintains context and a multimodal model summarizes plots for downstream subagents.Source: §F.1 · p. 43
• Asta v0 uses text similarity to identify task types and dispatches them to specialized solvers, achieving 100% routing accuracy on the validation set.The selected solver is chosen for best expected performance based on preliminary experiments.Source: §F.1 · p. 43

F.2 BASELINE AGENTS

• The benchmark includes ReAct and Smolagents Coder baselines alongside commercial and open research systems, including Elicit, FutureHouse agents, OpenAI Deep Research, OpenSciLM, STORM, and others.ReAct provides a minimally architectural baseline, while Smolagents Coder represents actions as executable Python code; Faker validates scoring for End-to-End Discovery tasks.Source: §F.2 · pp. 43–44

F.3 ASTA PAPER FINDER

• PaperFinder analyzes queries, routes them among navigational, semantic, metadata, and author-based workflows, then ranks relevance-judged papers using query-specific criteria such as recency or citation count.The AstaBench version is a frozen, single-turn subset using AstaBench’s search APIs and differs from the live product in available indices, papers, configuration, and continuations.Source: §F.3 · p. 45

F.3.1 QUERY ANALYSIS

• The query analyzer uses parallel short LLM prompts to extract semantic, relevance, metadata, navigational, recency, and centrality properties for downstream search and ranking.Manual coding was more efficient and reliable than a more autonomous workflow, while unsupported metadata such as author affiliation is identified for user notification.Source: §F.3.1 · pp. 45–46

F.3.2 NAVIGATIONAL QUERIES

• Specific-paper queries run three parallel strategies—title-API lookup, LLM-grounded title lookup, and citation-based key-term search—then merge their candidate results.The strategies respectively use Semantic Scholar titles, corpus identifiers, and top-cited papers from matching sentences.Source: §F.3.2 · p. 46

F.3.3 SEMANTIC QUERIES

• Semantic search iteratively broadens retrieval through query rephrasing, snippet aggregation, citation snowballing, boundary-focused follow-up queries, and LLM relevance judgments with bounded short-circuiting.Candidate papers combine evidence from retrieved and citing snippets, while relevance is assessed against the original query and decomposed sub-criteria.Source: §F.3.3 · pp. 47–48

F.3.4 METADATA QUERIES

• Metadata-only or complex metadata queries use an LLM-generated structured work-plan executed through API calls, while simple venue and year constraints post-filter or rank semantic results.Simple metadata fields include year, year-range, authors, venues, and citation counts; complex constraints are represented as data structures beyond the paper’s scope.Source: §F.3.1 · p. 46, §F.3.4 · p. 48

F.3.5 FINAL RANKING

• Final ranking combines relevance judgments with query preferences, citation counts, and publication date using a heuristic.Source: §F.3.5 · p. 48

F.5 REACT PROMPT

• The documented baselines include a ReAct agent with an InspectAI prompt and a smolagents coder prompt that exposes AstaBench tools and requires structured final-answer submission.The accompanying source references identify implementations for the evaluated agents and solvers.Source: §F.5 · p. 50, §F.6 · p. 50

F.7 ASTA V0 ROUTING TABLE

• Asta v0 routes tasks by character-level lexical overlap, achieving 100% validation routing accuracy, to specialized solvers selected for expected performance across search, QA, table, and data-analysis tasks.The approach can confuse similarly formatted tasks, but routes them to the same sub-agent when appropriate.Source: §F.7 · p. 51
• Asta v0 retries with the next most similar task type when a predicted sub-agent produces no output, allowing up to three attempts.The orchestrator provides a fallback mechanism for sub-agents to opt out.Source: §F.7 · p. 51

F.8 VALIDATION OF LITERATURE UNDERSTANDING AGENTS

• Five scientific QA agents use an LLM formatting step to convert plain-text or degraded structured outputs into the required schema with sections and inline citations.The formatting step uses gemini-2.5-flash and is applied to You.com, Perplexity DR, OpenAI DR, and FuturHouse Crow and Falcon.Source: §F.8 · p. 52
• Asta Paper Finder’s expanded agent has been repeatedly validated on an internal superset evaluation set, which monitors recall and ranking regressions as user adoption grows.The internal set has been used throughout development and real-world usage, while increasing adoption provides additional corroboration of the agent’s effectiveness and evaluation methodology.Source: §F.8 · p. 52
• LitQA2-FullText evaluation is limited by task-subset and model-backbone differences, with PaperQA2 averaging 66.0 accuracy versus FutureHouse Crow at 72.0.The comparison uses FutureHouse, You.com, and Perplexity DR agents, and attributes the difference partly to fewer questions and newer state-of-the-art models.Source: §F.8 · p. 52
• Perplexity Sonar Deep Research is evaluated with high reasoning effort and search context, without academic-only search because that mode disabled web search during the August 3–7, 2025 runs.Because the API does not explicitly return quote snippets, evaluation treats citations as titles and URLs only.Source: §F.8 · p. 52

F.9 VALIDATION OF END-TO-END DISCOVERY AGENTS

• Cross-checking reports, code, and artifacts reduces false positives and false negatives in end-to-end rubric scoring.A criterion is met only when all three facets consistently include evidence that it was met; spot-checking judged the rubric 92% correct on 50 development items.Source: §F.9 · p. 52
• The scoring interface records each rubric item as met, not met, or unknown across three artifact classes, then averages final criterion verdicts into an overall score.The graphical presentation uses green, red, and yellow states for the three artifact checks and a final consistency verdict.Source: §F.9 · p. 52, Figure 10 · p. 53
• 16% of answers where the paper alone indicated a criterion was met were corrected to an overall failure by code or artifacts.Source: Table 21 · p. 53

G.2 EVALUATION ON FULL SET OF LITQA2 DATASET

• PaperFinder receives LitQA2 questions verbatim without multiple-choice options or task-specific query reformulation or tuning.Source: §G.2 · p. 53
• PaperFinder scores significantly higher on LitQA2 subsets whose relevant papers have full text available.The main results emphasize LitQA2-FullText-Search, where the corpus contains full text for all papers.Source: §G.2 · p. 55, Table 23 · p. 55
• On the original LitQA2 dataset, PaperFinder performs very similarly to the prior agent despite only abstracts being available for 114 of 199 queries.Source: §G.2 · p. 55

H EVALUATION TASK SAMPLES AND PROMPTS

• The evaluation materials provide example problems, rubrics, and detailed prompts for assessing scientific research tasks.The section presents higher-level task details through examples, scoring criteria, and prompt specifications.Source: H EVALUATION TASK SAMPLES AND PROMPTS · p. 55

H.1.1 EXAMPLE PROBLEM

• Literature-search tasks ask agents to find comprehensive, relevant, efficient results, with up to 250 papers permitted.Queries may concern technical mechanisms or specific scientific facts, while known-paper requests should return only the identified paper.Source: §H.1.1 · p. 55
• Search responses must use JSON containing each paper’s Semantic Scholar corpus ID and concise markdown evidence supporting its relevance.The evidence should quote only the minimum necessary text from the paper.Source: §H.1.1 · p. 55

H.2.1 EXAMPLE PROBLEM

• A second literature-search example applies the same comprehensive-search, paper-selection, 250-paper limit, and JSON evidence requirements to olfactory-neuron research.The example asks about active olfactory receptor genes’ contacts with Greek island regions in mouse olfactory neurons.Source: §H.2.1 · p. 56

H.3.1 EXAMPLE PROBLEM

• Research-report prompts require valid JSON sections with inline citations whose identifiers appear in the text and whose snippets come from reference documents.Citation metadata may include titles and additional information, while a separate References section is prohibited.Source: §H.3.1 · p. 56

H.3.2 EXAMPLE RUBRIC {

• The hallucination-and-bias rubric evaluates definitions, shared training-data causes, direct bias-to-hallucination links, propagation, bidirectionality, examples, and mitigation.Mitigation criteria include data preprocessing, algorithm selection, model evaluation, and high-quality diverse datasets.Source: §H.3.2 · pp. 57–58
• 0.81-0.83 correlation coefficients are given between intrinsic bias and extrinsic hallucination rates in the rubric’s examples.The rubric also includes healthcare, hiring, legal, and conversational examples, alongside strategies such as data curation, deduplication, augmentation, and integrated fairness-and-accuracy evaluation.Source: §H.3.2 · pp. 58–59
• The example rubric includes criteria for explaining mechanisms connecting bias and hallucination and discussing implications for high-stakes domains.Examples connect language- and vision-model hallucinations with learned correlations, statistical bias, and risks in healthcare, finance, security, and law.Source: §H.3.2 · pp. 59–60

H.3.3 EVALUATION PROMPTS

• The citation validator checks each claim against its inline citations, classifying citations as supporting or non-supporting and marking whether each claim is fully supported.It considers claims in tables, allows consecutive sentences to share citations, and skips high-level introductory sentences.Source: §H.3.3 · p. 60
• The relevance prompt identifies every paragraph that does not directly answer a query while retaining indirectly relevant context and useful background.It distinguishes closely related but task-mismatched content from contextual information that supports another part of the answer.Source: §H.3.3 · pp. 60–61
• The relevance evaluator requires every answer paragraph to be considered and returns irrelevant paragraphs with a reason and the exact complete answer text.Its structured output uses an `irrelevant_paragraphs` list containing `reason` and `answer_text` fields.Source: §H.3.3 · p. 61
• Responses are scored 0, 1, or 2 for each criterion, where scores represent failure, partial satisfaction, or perfect satisfaction of only the specified aspect.Examples illustrate what fully satisfying, partially satisfying, and failing a criterion look like across medical NLP, linguistics, and transformer-versus-RNN questions.Source: §H.3.3 · pp. 61–62

H.3.4 QUERY SELECTION

• The query-selection schema labels language, query type, field and subfield, completeness, clarity, and research stage.Research-stage labels include Ideation, Topic Understanding, Literature Search and Synthesis, Research Design, Data Analysis, and Project Write up.Source: §H.3.4 · pp. 63–64
• Query types distinguish information requests, search-term sequences, and testing requests, while field labels include Computer Science and related disciplines.The schema also asks whether a query concerns Computer Science or a closely related field.Source: §H.3.4 · pp. 63–64
• Scientific-field classification examples distinguish Computer Science, Biomedicine, Psychology, and “None of the above,” including representative subfields.Examples include artificial intelligence and software engineering; medical microbiology and genetics; and behavioral and clinical psychology.Source: §H.3.4 · p. 64

H.3.5 KEY INGREDIENT EXTRACTION AND CLUSTERING PROMPTS

• Ingredient-extraction prompts require identifying report-supported answer requirements, separating distinct query components, and marking critical requirements with SHOULD versus useful requirements with MIGHT.Ingredients should describe one requirement at a time, use appropriate verbs such as MENTION or DISCUSS, and avoid unnecessary specificity unless requested.Source: §H.3.5 · pp. 64–65
• The ingredient format requires sequential identifiers, requirement descriptions, examples with available citations, and null citations when no citation is available.Citations may use corpus identifiers, URLs, or author-year forms.Source: §H.3.5 · p. 66
• Clustering prompts consolidate ingredients into minimal non-overlapping key requirements, distribute examples, prune duplicates or irrelevant examples, and explain omitted ingredients.Ingredients may be grouped into one requirement or split across multiple requirements when they cover overlapping or distinct query components.Source: §H.3.5 · p. 66
• Clustering requirements must remain query-focused, self-contained, non-overlapping, and grounded in ingredient descriptions rather than examples alone.Background information is generally assigned MIGHT status, while requirements centered on paper citations are excluded.Source: §H.3.5 · pp. 66–67

H.4.1 EXAMPLE PROBLEM

• A multiple-choice example asks how much active olfactory receptor genes increase their contacts with Greek islands and requires returning only the chosen answer letter in JSON.The answer options are 2.0, 27, 1.7, 2.7, insufficient information, and 3.0 fold.Source: §H.4.1 · p. 68

H.5.1 EXAMPLE PROBLEM

• A paper-comparison example requests a JSON table with nine papers as rows and five dimensions comparing receiver operation policies for RFEHNs.The supplied papers concern wireless information and power transfer, including time switching, power splitting, beamforming, resource allocation, and energy-efficiency trade-offs.Source: §H.5.1 · pp. 68–69
• The example problem asks agents to analyze wireless information-and-power-transfer research, including optimization formulations, semidefinite relaxation, power splitting, and rate-energy tradeoffs.The sample spans multiuser beamforming, secure communication, receiver architectures, training-based SWIPT, dynamic power splitting, and opportunistic energy harvesting.Source: §H.5.1 · pp. 70–72

H.5.2 TABLE UNROLLING PROMPT

• The table-unrolling prompt requires identifying headers and rows, converting every non-ID cell into one numbered atomic natural-language statement, and preserving all table details without citations or additions.The output uses paper IDs and column headers while excluding icons and emojis.Source: §H.5.2 · pp. 73–74

H.5.3 EVALUATION PROMPT

• The evaluation prompt presents informative statements about scientific papers and asks whether a specified statement is true.The supplied prompt fragment does not specify the answer format beyond stating truth status.Source: §H.5.3 · p. 74

H.6.1 EXAMPLE PROBLEM

• The example experiment asks an agent to run probability-based prompt selection on SST-2 with opt-125m and report metrics under a single reduced-size experiment.Instructions limit preprocessing to the first 10 rows per set, training to one epoch, and execution to one experiment without tuning.Source: §H.6.1 · p. 74

H.7.1 EXAMPLE PROBLEM

• The reproducibility task requires rendering a manuscript’s R Markdown file, installing README requirements, saving results, and reporting specified article-level and figure-level outcomes in report.json.The task targets final reproducibility-check counts and the proportion of fully reproducible target values after author contact.Source: §H.7.1 · p. 75

H.8.1 EXAMPLE PROBLEM

• The programming task asks for PyTorch code that fills a batch-by-sequence-length-by-embedding tensor with 2333 after each sequence’s actual length.The input tensor has shape (10, 1000, 96), and the solution must be appended inside code tags without repeating the provided code.Source: §H.8.1 · pp. 76–77

H.9.1 EXAMPLE PROBLEM

• The dataset-analysis example describes NLSY79 data containing demographic, family, educational, health, residential, financial, and criminal-record variables for analysis.The supplied schema includes identifiers, parental education and occupation, race or ethnicity, sex, family size, criminal-justice indicators, height, weight, and class rank or size.Source: §H.9.1 · p. 77
• The example problem asks whether increased time preference leads to higher BMI.Source: §H.9.1 · p. 81
• Agents must return a JSON object containing a scientific hypothesis and a workflow summary.Source: §H.9.1 · p. 81
• The hypothesis should state relevant context, variables, relationships, statistical significance, and supporting numeric information derived from the dataset.Source: §H.9.1 · p. 81
• The workflow summary should describe the full analysis workflow and ensure that the dataset is loaded or delegated to another agent.Source: §H.9.1 · p. 81

H.10.1 EXAMPLE PROBLEM

• The simple-dag-enhancement task targets improved emotion recognition by adding content-based edge selection to the static DAG-ERC model.Source: §H.10.1 · p. 81
• Its proposed method adds a small number of edges using simple content similarity between utterances while retaining the original DAG structure.Source: §H.10.1 · p. 81
• The DAG-ERC task hypothesizes that sparse content-similarity edges can capture long-range conversational context and improve emotion recognition over static structure.It compares the enhanced DAG with original static DAG-ERC and a fully connected graph baseline.Source: §H.10.1 · p. 82
• The DAG-ERC experiment evaluates weighted-average F1 and micro-averaged F1 excluding the majority class, while analyzing added-edge distributions.Its variables include graph construction, similarity threshold, and edge limits, with model architecture and feature extraction controlled.Source: §H.10.1 · p. 82
• The DAG-ERC implementation starts from IEMOCAP and adds edges when RoBERTa-based cosine similarity exceeds a threshold, limiting additions to preserve sparsity.The proposed pilot uses 20 conversations, a threshold such as 0.8, and at most three additional edges per utterance.Source: §H.10.1 · p. 82
• The DAG-ERC task requires performance comparisons, edge-structure visualizations, error analysis, and investigation of which conversations benefit from added connections.The agent must return a report, code, trace, and supporting research artifacts in a specified JSON structure.Source: §H.10.1 · pp. 82–83

H.11.1 EXAMPLE PROBLEM

• The adaptive-reasoning task hypothesizes that combining Complexity-Based Prompting and Imitation Demonstration Learning improves language-model generalization on unseen reasoning tasks.Complexity-Based Prompting selects informative high-complexity reasoning paths, while Imitation Demonstration Learning retrieves similar examples and imitates their solution steps.Source: §H.11.1 · pp. 84–85
• The adaptive-reasoning evaluation uses unseen-task accuracy as its primary metric, with reasoning complexity and demonstration effectiveness as secondary measures.The procedure supports MINI_PILOT, PILOT, and FULL_EXPERIMENT modes, requiring human verification before full execution.Source: §H.11.1 · pp. 84, 86, 87
• The adaptive-reasoning experiment compares standard prompting, CBP-only, IDL-only, and integrated CBP+IDL conditions on complex multi-step reasoning problems.It uses a reasoning dataset such as 2WikiMultiHopQA, split into 60% training, 20% validation, and 20% test data.Source: §H.11.1 · p. 86
• The integrated system combines complex reasoning chains with semantically similar demonstrations and adaptively weights both methods according to question characteristics.CBP generates and votes among reasoning paths, while IDL retrieves detailed reasoning examples for unified prompts.Source: §H.11.1 · pp. 86–87
• The task requires testing differences between conditions with paired t-tests, Cohen’s d effect sizes, and bootstrap confidence intervals.Source: §H.11.1 · p. 87
• Agents must log prompts, responses, evaluation metrics, statistical results, and successful or unsuccessful cases for reproducibility.Source: §H.11.1 · p. 87
• The final report must summarize each condition, assess statistical significance, explain when and why the integrated approach performs better or worse, and recommend improvements.Source: §H.11.1 · p. 87
• Implementation guidance calls for NLTK, scikit-learn, a language model such as GPT-4, and robust error handling and logging.Source: §H.11.1 · p. 87
• Execution must begin in MINI_PILOT mode, proceed to PILOT only if successful, and require human verification before FULL_EXPERIMENT.Source: §H.11.1 · p. 87
• The required deliverables are a structured report, research code, a step-by-step action trace with results, and supporting research artifacts.Source: §H.11.1 · p. 88
• The response must use a JSON object with a `results` key containing `report`, `code`, `trace`, and `artifacts` fields in the specified formats.Source: §H.11.1 · p. 88