PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, Shumin Deng

arXiv:2608.31076v1cs.CLcs.AIcs.IRcs.LGcs.MAcs.SE

TL;DR

Open-ended scientific research tasks often omit the analyses, methods, and success criteria needed for well-supported results. AutoSciRub induces an evidence-grounded rubric before execution, then uses it for guidance, criterion-level verification, and iterative revision. It improves scores across ResearchClawBench and AstaBench configurations, while scientific core coverage slightly decreases in one analysis.

  • Problem

    Open-ended research instructions often leave scientific questions, methods, evidence requirements, and success conditions implicit, risking omitted analyses, inappropriate procedures, and unsupported claims.

  • Method

    AutoSciRub decomposes underspecified instructions into atomic scientific goals, grounds criteria in literature and task-visible data, and uses criterion-level verification to target revision.

  • Results

    AutoSciRub consistently improves tested configurations, averaging 2.08 points across three backbone LLMs under Codex, 2.95 points across three agent harnesses with DeepSeek-V4-Flash, and 16.8 points across three AstaBench agent harnesses.

  • Takeaways & Limitations

    Evaluation-first, evidence-grounded rubrics provide scalable criteria and effective guidance for autonomous scientific research across backbones and agent harnesses.

  • Takeaways & Limitations

    AutoSciRub improves operationalization but not scientific framing: scientific core coverage decreases slightly from 3.35 to 3.07.

Abstract

from arXiv · show

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).

1 Introduction

Open-ended research instructions leave scientific questions, methods, evidence requirements, and success conditions implicit, allowing plausible-looking but incomplete or unsupported outputs. AutoSciRub makes these requirements explicit before execution and uses criterion-level verification to guide targeted revision.

  • The challenge: Open-ended instructions can leave intermediate goals, methodological requirements, expected evidence, and success conditions unspecified.Agents may consequently omit essential analyses, use inappropriate procedures, or make unsupported claims.
  • The challenge: Existing expert-authored rubrics provide interpretable criteria but require substantial domain expertise and manual effort to construct.Automatically generated criteria also require grounding in literature, data, domain conventions, and execution constraints to remain complete, justified, and verifiable.
  • AutoSciRub: AutoSciRub decomposes instructions into atomic scientific goals and grounds them in literature, web evidence, task data, and environmental constraints.This produces a task-specific rubric that makes required analyses, evidence, and success conditions explicit.
  • AutoSciRub: Rubric-guided iterative revision evaluates evidence criterion by criterion, identifies unmet requirements, and provides targeted feedback for reports and supporting artifacts.The framework treats rubrics as intermediate scientific specifications rather than only post-hoc evaluation instruments.
  • Evaluation: 2.08 points average improvement was achieved across three backbone LLMs under the fixed Codex harness, while 2.95 points was achieved across three agent harnesses using the fixed DeepSeek-V4-Flash backbone.On AstaBench, scores improved by an average of 16.8 points across three agent systems.

2 Related Work

Scientific research agents have evolved from unified pipelines toward multi-agent and persistent long-horizon workflows, while benchmarks have expanded beyond fixed coding tasks toward research reproduction. Rubric-based evaluation adds explicit, fine-grained criteria for assessing complex research artifacts.

  • Autonomous Scientific Research Agents and Benchmarks: Early scientific-agent systems use single-agent pipelines or centrally orchestrated workflows for literature review, experimentation, and manuscript preparation.These systems coordinate multiple research stages within a unified process.
  • Autonomous Scientific Research Agents and Benchmarks: Multi-agent systems distribute ideation, critical discussion, experiment execution, and coordination across specialized roles.Persistent agents additionally use structured memory, cache-efficient context control, and hierarchical multimodal experience storage.
  • Autonomous Scientific Research Agents and Benchmarks: Scientific-agent benchmarks have progressed from executable coding and model-development tasks under fixed objectives to research reproduction requiring reconstruction of methods and experiments.This reflects a broader move toward evaluating end-to-end research capabilities.
  • Rubric-Guided Evaluation and Iterative Refinement: Rubric-based evaluation uses explicit, fine-grained criteria as a structured alternative to holistic judging.Expert-authored hierarchical rubrics decompose complex research artifacts into independently assessable components, while later benchmarks introduce atomic verifiable criteria and question-specific scoring or deduction points.

3 Method

AutoSciRub turns underspecified research instructions into executable, evidence-grounded rubrics and uses them to guide execution, verification, and targeted revision. Its pipeline combines atomic goal induction, literature grounding, task-data inspection, criterion synthesis, and criterion-level feedback.

  • Overview: AutoSciRub transforms an underspecified instruction into a task-specific executable rubric that guides research execution, verification, and iterative revision.The rubric operationalizes implicit scientific requirements and supports improvement of the generated report and artifacts.
  • Task Formulation: The task formulation represents each multi-domain research task as a high-level instruction paired with a task-visible environment of literature, data, search, code execution, and domain tools.The agent produces a scientific report together with supporting code, results, analyses, tables, and figures.
  • Automatic Rubric Induction: Rubric Skeleton Induction decomposes the instruction into traceable, non-overlapping atomic scientific goals without initially specifying methods, metrics, baselines, or expected results.Later stages determine how each goal should be established.
  • Automatic Rubric Induction: Scientific Literature Grounding organizes relevant methods, protocols, analyses, metrics, baselines, controls, robustness checks, and evidence forms by scientific goal.Retrieved sources guide experiment and evidence design but are not treated as experimental evidence for the current task.
  • Automatic Rubric Induction: Task-Data Exploration profiles available files and datasets, including formats, dimensions, fields, units, labels, conditions, cross-source relations, and constraints, to identify feasible analyses.The profile supports planning without assuming unavailable labels, conditions, or reference values; full experiments are deferred to execution.
  • Automatic Rubric Induction: Criterion Synthesis combines the skeleton, literature knowledge, and data profile into criteria linking goals with feasible analyses, evidence artifacts, metrics, comparisons, and satisfaction conditions.These criteria align established scientific practice with evidence producible from the current task.
  • Rubric-Guided Iterative Revision: Rubric-guided revision verifies each criterion against the current artifact, identifies specific evidence gaps, and feeds failed-criterion diagnoses into targeted updates.Verification considers required experiments, reported results, supporting artifacts, and whether conclusions are adequately supported.

4 Experiments

Across ResearchClawBench and AstaBench, AutoSciRub improves research-agent scores across models, harnesses, domains, and refinement stages. Its executable rubrics operationalize broad goals into verifiable requirements, while rubric-guided revision provides the largest cumulative gains, with scientific-core framing remaining a limitation.

  • Overall Performance: AutoSciRub improves ResearchClawBench scores across all six model–harness configurations and 49 of 60 domain-level paired comparisons.Gains remain consistent across backbone models, agent harnesses, and scientific domains.
  • Overall Performance: 16.78 points: AutoSciRub’s average improvement across 20 AstaBench tasks, while successful-task counts remain unchanged or increase.Claude Code and Codex rise from 18/20 to 20/20 completed tasks, while OpenClaw remains at 19/20.
  • Ablation Study: 20.36: Rubric-Guided Iterative Revision’s score, improving 3.11 points over the base and 2.05 over grounded-rubric induction.The largest incremental gain comes from identifying unmet criteria and targeting report and artifact revisions.
  • Ablation Study: 3.84: executable-rubric mean quality versus 2.20 for rubric skeletons across 40 tasks, with specificity increasing from 1.65 to 4.40.Evidence verifiability rises from 1.78 to 4.08, and actionability from 2.00 to 3.83.

5 Conclusion

AutoSciRub induces task-specific scientific rubrics and uses them as execution-time guidance for autonomous research agents. Experiments on ResearchClawBench and AstaBench show improvements across models and agent harnesses.

  • AutoSciRub decomposes underspecified research instructions into scientific goals, grounds criteria in external evidence, and guides verification and revision.

A.1 Benchmark and Evaluation Details

The evaluation spans ResearchClawBench and AstaBench across multiple models, agent harnesses, tasks, and official evaluation procedures. Comparisons hold task resources and execution environments constant while allowing AutoSciRub’s additional calls, and primarily assess output quality rather than budget-normalized efficiency.

  • Benchmarks: ResearchClawBench contains 40 end-to-end tasks spanning ten scientific domains, each based on a published target paper with accessible raw data.
  • Benchmarks: AstaBench evaluation uses a fixed, randomly sampled 20-task subset and official task-specific rubrics with an LLM-as-judge pipeline.
  • Configurations: ResearchClawBench generalization varies backbone LLMs under Codex and varies agent harnesses under a fixed DeepSeek-V4-Flash backbone.
  • Controlled comparisons: Paired vanilla and +AutoSciRub settings share task inputs, visible resources, workspace, and tools, but AutoSciRub adds rubric induction, verification, and targeted revision.
  • Scoring: Each task-configuration pair runs once, submissions receive three independent evaluator scores, and the arithmetic mean forms the task-level score.
  • Evaluation boundary: No shared cap is imposed on tokens, tool calls, runtime, or monetary cost, so primary evaluation focuses on output quality rather than budget-normalized efficiency.

A.2 Rubric Quality Evaluation

Rubric quality is evaluated separately for instruction-derived and executable rubrics across four complementary dimensions. Independent judge prompts use task context, visible-data descriptions, candidate rubrics, and evaluator-only benchmark checklists.

  • Evaluation setup: Instruction-derived rubric skeletons and task-specific executable rubrics are evaluated separately for every ResearchClawBench task.
  • Dimensions: Specificity, Actionability, Evidence Verifiability, and Scientific Core Coverage are scored independently on five-point ordinal scales.
  • Judge inputs: The judge receives the task instruction, task-visible data description, candidate rubric, and official benchmark checklist, which remains evaluator-side evidence.
  • Aggregation: Overall rubric quality is the unweighted arithmetic mean of the four dimension scores, each ranging from 1 to 5.
  • Dimension meanings: Specificity measures precision, Actionability measures executability, Evidence Verifiability measures inspectable support, and Scientific Core Coverage measures alignment with scientific success conditions.
  • Dimension distinctions: High specificity or actionability does not guarantee scientific alignment, and an aligned executable plan may still lack independently checkable evidence.

Complete Rubric-Quality Scoring Prompts

The scoring prompts independently operationalize four rubric-quality dimensions using explicit definitions, evaluation rules, score anchors, and JSON output requirements. They distinguish concrete specification, executable planning, verifiable evidence, and faithful coverage of the scientific core.

  • Prompt structure: Four standalone judge prompts each evaluate exactly one dimension: Specificity, Actionability, Evidence Verifiability, or Scientific Core Coverage.
  • Specificity: Specificity scoring checks whether methods, conditions, variables, metrics, comparisons, thresholds, and artifacts make the intended analysis unambiguous.
  • Specificity: Specificity score anchors range from generic or ambiguous rubrics at 1 to precise, unambiguous core analyses at 5.
  • Actionability: Actionability scoring asks whether available inputs support an executable route to required outputs without the agent inventing consequential methods, parameters, or decisions.
  • Actionability: Actionability anchors range from no executable route at 1 to an end-to-end operational plan specifying consequential steps, dependencies, decisions, and outputs at 5.
  • Evidence Verifiability: Evidence Verifiability scoring assesses inspectable, traceable evidence and penalizes conclusions that lack checkable criteria, diagnostics, or claim-to-evidence links.
  • Scientific Core Coverage: Scientific Core Coverage maps core success conditions to rubric items and distinguishes aligned, equivalent, partial, proxy, missing, and conflicting statuses.

A.3 Rubric-Guided Revision Evaluation

The evaluation compares rubric-guided revision with rubric-free self-refinement using criterion-level verification, targeted revisions, and adaptive stopping. The guided procedure inspects scientific goals, evidence, artifacts, and rubric items before revising reports.

  • Criterion-level verification: Rubric-guided revision uses a verification-only call to assess experiments, results, figures, quantitative values, conclusions, analyses, and required artifacts.The verifier also distinguishes task-generated evidence from claims supported only by prior literature.
  • Criterion-level verification: For each scientific goal, verification records coverage of experiments, results, figures or tables, conclusions, and mechanisms or analytical explanations.It additionally evaluates rubric items and generates stable feedback specifying remaining issues and corrective actions.
  • Criterion-level verification: Overall pass is an LLM judgment requiring sufficient goal-level support and satisfied or justified high-priority rubric items, while lower-priority items need not individually pass.The output schema includes an overall_pass field and structured goal and rubric checks.
  • Targeted revision: If verification fails, one targeted revision can modify code, rerun analyses, regenerate artifacts, correct results, revise claims, and update the complete report.Verification and revision are separate calls, and each revision receives the current workspace, rubric, report, artifacts, and verifier feedback.
  • Adaptive stopping: At most three revision rounds and four verification calls are allowed, with immediate termination when overall_pass=true.If earlier verification does not pass, a final verification follows the third revision.
  • Adaptive stopping: Rubric-guided evaluation produced 103 distinct checkpoints across 40 tasks: 40 checkpoint-0, 40 checkpoint-1, 17 checkpoint-2, and 6 checkpoint-3 reports.Later checkpoint means carry forward the latest real report and its previously obtained score for tasks that stop early.
  • Baseline comparison: The rubric-free baseline performs three review–revision cycles for all 40 tasks, producing 40 real reports at each of checkpoints 1, 2, and 3.Each resulting report is independently scored three times by the same GPT-5.1 judge.
  • Baseline comparison: The baseline is not strictly compute-matched or workspace-identical because it combines review and revision, always runs three rounds, and starts from a newly initialized workspace.Guided revision separates verification from revision, uses adaptive stopping, and retains the original code and output state.

B Rubric Output Formats

The paper presents two rubric output formats: an instruction-derived skeleton of atomic scientific goals and a task-specific executable rubric that adds operational requirements. The executable format connects goals to experiments, artifacts, metrics, data sources, conclusions, and constraints.

  • Instruction-Derived Rubric Skeleton: The instruction-derived rubric skeleton represents a task as atomic scientific goals with unique identifiers, concise titles, and concrete descriptions.An example goal characterizes the spatial concentration of cloud-seeding projects across U.S. states.
  • Task-Specific Executable Rubric: The task-specific executable rubric augments scientific goals with experiments, evidence artifacts, metrics, comparisons, data sources, expected conclusions, priorities, and execution constraints.Its format is illustrated separately from the instruction-derived skeleton.
  • Task-Specific Executable Rubric: The example task summary requires reproducing and validating empirical conclusions from a NOAA cloud-seeding dataset using script-based spatial, temporal, purpose, and deployment analyses.The specified dataset covers U.S. cloud-seeding projects from 2000–2025.
  • Task-Specific Executable Rubric: The example executable rubric selects an environment providing packages for tabular analysis, statistical computation, and geospatial visualization.The selected environment is named .venv-rcb-base.
  • Task-Specific Executable Rubric: The example goal requires grouping projects by state, computing rank, percentage share, and Gini coefficient, and generating a choropleth map and state-by-purpose heatmap.These experiments operationalize the spatial-concentration goal through reproducible calculations and visual artifacts.
  • Task-Specific Executable Rubric: Required artifacts include a spatial-distribution figure, a state-purpose heatmap, and a ranked table of leading states with project counts and percentage shares.The figure and table requirements specify both expected paths and descriptive content.
  • Task-Specific Executable Rubric: The rubric requires explaining western geographic concentration and regional variation in project purposes, while identifying associated factors without leaving these patterns undescribed.The spatial figure is expected to show strong western concentration and leading states including California, Colorado, Texas, and Utah.
  • Task-Specific Executable Rubric: The example expected conclusions state that cloud seeding is heavily concentrated in western states, California has the highest project count, and the distribution reflects water-resource management priorities.These conclusions are marked high priority because omitting them would lose primary spatial evidence for geographic-concentration claims.
Loading 2608.31076v1…