PaperlayerView exact revision on arXiv

Source-linked AI summary

Training AI Co-Scientists Using Rubric Rewards

Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, Chenxi Whitehouse

arXiv:2512.23707v1cs.LGcs.CLcs.HC

TL;DR

Language models struggle to generate research plans that satisfy explicit constraints and implicit requirements, especially for open-ended science without executable feedback. The paper extracts goals and rubrics from scientific papers and trains plan generators with rubric-guided self-grading. Finetuned plans were preferred for 70% of ML goals and improved by 12–22% on medical and new arXiv evaluations.

  • Problem

    Language models remain limited at generating plans for open-ended research goals, where rapid execution feedback and verification are often unavailable.

  • Method

    The method extracts research goals and goal-specific rubrics from scientific papers, then trains a plan generator with reinforcement learning using a frozen initial model as grader.

  • Results

    Finetuned plans were preferred for 70% of ML research goals, while frontier-model rubric evaluations showed 12–22% relative improvements across medical papers and new arXiv preprints.

  • Takeaways & Limitations

    Rubric-guided self-grading improved research-plan generation across scientific fields without requiring specialized execution environments.

  • Takeaways & Limitations

    Human expert evaluation covered only machine learning, while remaining results relied on frontier-model rubric grading rather than implemented plans.

Abstract

from arXiv · show

AI co-scientists are emerging as a tool to assist human researchers in achieving their research goals. A crucial feature of these AI co-scientists is the ability to generate a research plan given a set of aims and constraints. The plan may be used by researchers for brainstorming, or may even be implemented after further refinement. However, language models currently struggle to generate research plans that follow all constraints and implicit requirements. In this work, we study how to leverage the vast corpus of existing research papers to train language models that generate better research plans. We build a scalable, diverse training corpus by automatically extracting research goals and goal-specific grading rubrics from papers across several domains. We then train models for research plan generation via reinforcement learning with self-grading. A frozen copy of the initial policy acts as the grader during training, with the rubrics creating a generator-verifier gap that enables improvements without external human supervision. To validate this approach, we conduct a study with human experts for machine learning research goals, spanning 225 hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% of research goals, and approve 84% of the automatically extracted goal-specific grading rubrics. To assess generality, we also extend our approach to research goals from medical papers, and new arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Together, these findings demonstrate the potential of a scalable, automated training recipe as a step towards improving general AI co-scientists.

1 Introduction

The paper addresses how to train models to generate sound plans for diverse, open-ended research goals, where execution feedback is often unavailable. It proposes extracting goals and rubrics from papers and using rubric-guided self-grading, with improvements shown in human and cross-domain evaluations.

  • The approach targets a limitation of execution-based AI-for-Science systems: many scientific problems cannot be represented in predefined, high-fidelity digital environments.The paper positions plan generation and human critique as a more general alternative for open-ended research settings.
  • The method extracts open-ended research goals, constraints, and goal-specific rubrics from scientific papers, then trains a plan generator through reinforcement learning with self-grading.A frozen copy of the initial model grades generated plans using the extracted rubrics as privileged information.
  • 70% of research goals received plans preferred from the finetuned Qwen-3-30B-A3B-Instruct model over the initial model, with p < 0.01.Experts judged the finetuned plans sounder, more outcome-oriented, and more useful as references for graduate students.
  • 12–22% relative improvements over the initial model were observed on medical papers and new arXiv preprints using frontier-model rubric juries.The finetuned 30B model was competitive with Grok-4-Thinking but remained behind GPT-5-Thinking.
  • ResearchPlanGen is released as a scalable corpus built from machine learning, medical, and recent arXiv papers.The corpus is intended to support research on AI co-scientists.

2 Methodology

The methodology automatically converts scientific papers into research goals, rubrics, and reference solutions, then trains a plan generator with rubric-based self-reward. Grading combines paper-specific requirements with general quality guidelines and scores the fraction of satisfied rubric items.

  • 2 Methodology: A frozen initial model grades generated plans with extracted rubrics, enabling reinforcement learning without continuous expert supervision or real-world execution.The policy generates a plan p for goal g, while the grader maps the goal, plan, and rubric to a reward in [0, 1].
  • 2.1 Data Construction: The automated pipeline extracts candidate goals, rubrics, and reference solutions from papers, filters rubric items, and selects the highest-scoring training tuple.The selector evaluates components for quality and diversity before choosing the final sample.
  • 2.2 Rubric Grading: Goal-specific rubrics test both explicit requirements from the research goal and unstated features that valid plans should include.The grading protocol also applies seven general guidelines to the relevant plan section for each rubric item.
  • 2 Methodology: The approach replaces costly specialized trial-and-error environments with plan critique before implementation, while leaving judgments about novelty and value to human researchers.This design is motivated by open-ended scientific questions for which execution feedback is impractical.
  • 2.2 Rubric Grading: The grader marks a rubric item satisfied only when no general guideline violations are identified, making reward the fraction of satisfied items.Violation-based grading also supports grounded scores and fine-grained error analysis.

3 Experiment Setup

The experiments evaluate the approach across machine learning, arXiv, and medical research datasets using human experts and frontier-model rubric juries. Training uses separate domain models and assesses both within-domain performance and cross-domain generalization.

  • 3.1 Dataset: ResearchPlanGen: The released corpus spans ML, arXiv, and medical papers, supporting evaluation on diverse research domains.The ML subset contains 6,872 training goals and 685 test samples, while the arXiv subset includes 6,573 training and 1,496 test samples.
  • 3.1 Dataset: ResearchPlanGen: Medical research is included because trial-and-error is infeasible in this high-stakes domain, making planning assistance especially relevant.The medical papers were sourced from the PMC Open Access dataset and released after March 2023.
  • 3.2 Training Configuration: Separate models are trained for ML, arXiv, and medical domains to measure within-domain performance and cross-domain generalization.The main policy is initialized from Qwen-3-30B-Instruct-2507, with additional model families used to test generality.
  • 3.3 Evaluation Methodology: 25 experts evaluated 100 ML test samples, with three experts per sample and 45 minutes allocated to each annotation.They compared initial and finetuned plans across five criteria and provided overall 1–10 scores.
  • 3.3 Evaluation Methodology: Automated evaluation averages rubric scores from GPT-5-Thinking, Gemini-2.5-Pro, and Claude-4-Sonnet.The frontier-model jury follows the same rubric-grading structure described for training.

4 Results

Rubric-guided reinforcement learning improves research-plan quality across human and automated evaluations, generalizes across domains, and benefits from carefully chosen training components, while self-grading can over-optimize weaker graders.

  • 4.1 ML Experts Prefer Our Trained Model’s Plans: 70.0% ± 5.3% of human annotations preferred finetuned plans over initial-model plans, with scores of 7.89 ± 0.2 versus 7.31 ± 0.2.The finetuned model also satisfied 79.8 ± 2.5% of rubric items versus 73.8 ± 2.9% for the initial model.
  • 4.1 ML Experts Prefer Our Trained Model’s Plans: 84% of rubric items were judged at least necessary for a good plan, with average rubric-item quality of 4.3/5.This supports the effectiveness of automatically extracting goal-specific rubrics from research papers.
  • 4.1 ML Experts Prefer Our Trained Model’s Plans: Frontier-model jury agreement with human consensus was significantly above chance, with Cohen’s κ = 0.297.Agreement was higher for objective rubric guidelines than for subjective ones.
  • 4.2 Finetuned Models Improve Across Domains, and Become Competitive with Frontier Models: 10-15% relative improvements over the initial policy matched the much larger Grok-4-Thinking model, although GPT-5-Thinking performed substantially better than all tested models.These results came from automated rubric evaluations averaged across three frontier-model judges.
  • 4.2 Finetuned Models Improve Across Domains, and Become Competitive with Frontier Models: 15% relative improvement on ML tasks and 17.5% on ArXiv tasks came from the medical finetuned model compared with baseline, demonstrating significant cross-domain generalization.On ArXiv, the medical finetune even outperformed ArXiv-specific finetuning.
  • 4.3 Training Ablations: Self-graded scores continued increasing while stronger-grader scores diverged after about step 120, indicating over-optimization of the weaker reward model.Later plans added superfluous details and worsened cost and effort efficiency according to the held-out grader; step 100 was selected for the main model.
  • 4.3 Training Ablations: SFT worsened plan quality, whereas rubric-RL ablations improved scores from 12.0 to 21.7 and then to 23.3 with a stronger grader, reaching 29.7 after disabling the KL penalty.Removing filtering produced only a minor improvement, while nested guideline checking and instance-specific rubrics were important components.

5 Related Work

Related work uses executable environments and automated feedback to improve AI-for-science systems, while this paper extends rubric-based automated evaluation toward open-ended research-plan generation and human-aligned training.

  • AI for Research: AI-for-science systems commonly optimize well-defined objectives through repeated trials in executable environments, enabling advances such as AlphaFold and AlphaEvolve.This paradigm is less general when scientific tasks cannot be represented by predefined sandboxes.
  • Reinforcement Learning with Rubrics: Rubric-based reinforcement learning uses language models to extract research goals and instance-specific grading rubrics from expert-created scientific papers when curated datasets are unavailable or costly.The extracted rubrics provide automated rewards for finetuning.
  • Aligning with Human Preference via Automated Feedback: Automated rubric judges can improve model performance according to human evaluators, but their rewards must ultimately be aligned with human judgment.The paper therefore validates its automated training recipe through an extensive human study.

6 Limitations and Future Work

The study relies on expert preferences and frontier-model rubric grading rather than implementing research plans, with human evaluation limited to machine learning. Finetuned plans also become more complicated, motivating better evaluations and training methods.

  • Limitations: Human evaluation compares plans through expert preferences rather than by implementing them and observing outcomes.This evaluation was conducted only for machine-learning goals after the final model was developed; remaining results use frontier-model rubric grading.
  • Limitations: Human access and cost limited expert evaluation to one domain, while plan comparisons can remain subjective at the sample level.The authors report significant aggregate evidence for relative model comparisons despite this subjectivity.
  • Future Work: Future work should develop evaluations that are more accurate, economical, and capable of providing fast feedback.The authors also propose better learning algorithms and rubric-based training for tool-using co-scientist systems.
  • Future Work: Finetuning improves plans across many criteria but also makes them more complicated, partly because the local model grades cost and effort efficiency poorly.Practitioners can filter unnecessary steps, and future work could use structured feedback beyond numeric scores.

7 Conclusion

The paper trains research-plan generators from scientific papers using extracted goals and rubrics, then evaluates improvements within and across scientific domains. Its conclusion emphasizes generality while positioning the approach as complementary to specialized execution-feedback systems.

  • 7 Conclusion: The paper uses extracted research goals and grading rubrics from scientific papers to train a plan generator with self-grading.The procedure targets diverse open-ended research goals rather than a single executable scientific environment.
  • 7 Conclusion: Human experts confirm improved research plans for machine-learning goals, while frontier-model rubric evaluations show gains for medical papers and new arXiv preprints.The conclusion reports significant cross-domain generalization across eight arXiv domains.
  • 7 Conclusion: The approach complements execution-feedback methods by addressing scientific tasks that cannot be represented by specialized environments.Existing AI-for-Science systems often optimize well-defined objectives in end-to-end executable settings, whereas many scientific domains lack such sandboxes.
  • 7 Conclusion: The paper frames broad scientific planning as a potentially general capability that can benefit from training on diverse research tasks.This motivation is linked to transfer learning and the generality of scientific creative processes.

A.3 Filtering for Quality

The filtering pipeline scores extracted research goals, rubric items, and reference solutions, removes weak or redundant rubric items, and selects one high-quality sample per paper. The appendix then reports filtering quality and related model-comparison experiments.

  • Filtering for Quality: The filtering model scores research goals and rubric items for guideline violations, then removes items that are weak or overlap with others.It selects n − k items for removal and retains the top k = 10 rubric items for final scoring.
  • Filtering for Quality: The pipeline evaluates reference solutions for faithfulness to the source paper before selecting the best sample from up to three candidates per paper.The final sample maximizes averaged quality across the research goal, rubric items, and reference solution.
  • Filtering Results: Filtering consistently improves quality across all criteria, especially research-goal guidelines and reference-solution satisfaction.Table 6 compares average scores for all candidate samples with the selected sample per paper.
  • Additional Experiments: The appendix also evaluates training across Llama and Gemma backbones while retaining Qwen-3-30B as the reward model.The authors use this setup because the smaller Llama and Gemma models were poor, lenient judges.
  • Additional Experiments: Consistent improvements appear across model backbones, with especially large gains for Llama as training adds detail to initially vague plans.Figure 5 summarizes finetuning improvements using a jury of GPT-5-Thinking, Claude-4-Sonnet, and Gemini-2.5-Pro.
  • Additional Experiments: The appendix reports pairwise preference evaluation separately for three judges, using win rates and average scores for initial versus finetuned models.These evaluations provide transparency across GPT-5-Thinking, Claude-4-Sonnet, and Gemini-2.5-Pro.

B.2 Training Qwen-3-4B Instruct vs Thinking

The Qwen-3-4B instruct and thinking models achieve similar validation performance when trained with the Qwen-3-30B MoE reward model. The thinking model is substantially more expensive per training step, limiting its practical advantage in this setting.

  • B.2 Training Qwen-3-4B Instruct vs Thinking: Thinking requires more than 2x compute for the same number of training steps, with higher training time and memory demands.The authors therefore train instruct models in their main work because no differential gains were elicited at the available scale.
  • B.2 Training Qwen-3-4B Instruct vs Thinking: The authors hypothesize that emphasis on mathematical and coding tasks limits the thinking model’s ability to exploit additional test-time compute for research-plan generation.They expect thinking could help at larger scale.
  • B.2 Training Qwen-3-4B Instruct vs Thinking: Qwen-3-4B instruct and thinking models show very similar validation performance under the Qwen-3-30B MoE reward model.Figure 6 reports validation results scored by Claude-4-Sonnet.
  • B.2 Training Qwen-3-4B Instruct vs Thinking: Improving reward-model capability is identified as a future direction because better alignment with human grading might preserve training gains under human evaluation.This observation concerns the broader self-grading setup rather than a demonstrated result of the instruct-versus-thinking comparison.

B.3.2 Supervised Fine-Tuning Fails to Generalize

Supervised fine-tuning lowers validation loss but worsens rubric-graded content, whereas rubric-based reinforcement learning improves performance, especially with stronger grading and without KL regularization.

  • Supervised fine-tuning: SFT significantly worsens automatic-grader performance despite validation loss decreasing from 1.984 to 1.127.The selected checkpoint used learning rate 1e-5, batch size 32, and step 180.
  • Rubric and filtering effects: Specific and generic rubrics each improve scores across guidelines, while best-of-three filtering produces no significant improvement.Providing a reference solution also improves scores across guidelines.
  • Supervised fine-tuning: SFT increasingly mimics summarized reference-solution wording, satisfying fewer rubric items even as its validation loss improves.The summarized reference solution removed important details and rationale from the original.
  • KL ablation: Disabling the KL penalty improves results after sufficient training, avoiding later degradation on detailed and specific-solution guidelines.With KL enabled, some guidelines worsen with more training; this pattern does not occur when KL is disabled.

B.3.4 Ablating data generation

Data-generation ablations show that combining specific and generic rubric optimization is preferred across judges and criteria except feasibility, while reference solutions help and filtering adds only marginal benefit.

  • Rubric composition: Specific-plus-generic rubric optimization is preferred across judges and criteria except feasibility.The comparison uses preference evaluations on human annotation guidelines rather than only rubric-based scores.
  • Evaluation: The preference-evaluation results are summarized in Table 9.The table reports preference evaluations of the ablations.
  • Reference solutions and filtering: Providing the reference solution to the reward model improves scores across guidelines, whereas filtering provides only marginal benefit.These effects are reported in the ablations corresponding to Figures 11 and 12.

B.4 ArXiv Subject-wise Performance Breakdown

Finetuned models outperform the initial model across all eight ArXiv subjects, with subject-specific strengths and broad improvements beyond the domains targeted during finetuning.

  • Overall subject-wise results: Finetuned models consistently outperform the initial Qwen-3-30B-A3B model across all ArXiv subjects.Scores are averaged across Claude-4-Sonnet, Gemini-2.5-Pro, and GPT-5-Thinking, with stratified sampling ensuring equal subject distribution.
  • Subject variation: MATH is the most challenging category for all models, scoring significantly lower than CS or Q-BIO.The paper notes that formal mathematical domains may benefit from complementary training with verifiable rewards.
  • Subject-specific strengths: Med-Ft reaches 37.1% on Q-BIO, while ML-Ft reaches 35.4% on EESS and 35.7% on CS.The paper attributes these strengths to overlap between related biological, biomedical, and machine-learning literatures.
  • Cross-domain generalization: Improvements on subjects not explicitly targeted by domain-specific finetuning suggest broadly applicable research-plan generation principles.This conclusion is based on the general improvement observed across all subjects.

B.5 Analysis: Domain finetunes across general guidelines

Across general guidelines, domain-finetuned models usually outperform the initial policy and Grok-4-Thinking, though GPT-5-Thinking remains stronger overall and some guideline exceptions persist.

  • General guideline performance: Finetuned models outperform the initial policy and Grok-4-Thinking on most general guidelines, but not cost and effort efficiency or no ethical issues.On overall plan, performance is equivalent for the no-ethical-issues guideline.
  • Frontier-model comparison: GPT-5-Thinking outperforms the compared models across general guidelines, with especially strong gains where finetuning also improves.Scores are averaged across Claude-4-Sonnet, Gemini-2.5-Pro, and GPT-5-Thinking.
  • Evaluation format: Table 10 reports ArXiv subject results as the percentage of rubric items satisfied, averaged across three judge models.The table covers eight ArXiv subjects and all evaluated models.

B.6.1 Human-Judge agreement

LLM judges generally align with human overall preferences, but their agreement varies by criterion and is asymmetric for rubric satisfaction.

  • 75.3% agreement (κ = 0.44) makes GPT-5-Thinking the strongest individual LLM judge against human overall preferences.Gemini-2.5-Pro follows at 73.9% agreement, while Claude-4-Sonnet reaches 50.0%.
  • 78–84% agreement on Addresses Requirements exceeds the 50–66% agreement observed for the more subjective Clear Execution criterion.GPT-5-Thinking aligns particularly strongly on Predicted Outcomes (κ = 0.41) and Clear Execution (κ = 0.33).
  • 89–98% of human-identified rubric violations receive LLM agreement, compared with only 22–49% agreement when humans mark items satisfied.The judges are therefore systematically stricter than human experts when assessing satisfaction.

B.6.2 Inter-judge agreement

Frontier judges show moderate agreement on rubric satisfaction but low agreement on preference grading, with disagreement increasing for subjective criteria and non-ML domains.

  • κ ranges from 0.366 to 0.424 overall for rubric satisfaction, with highest agreement on objective guidelines such as handles all criteria.Agreement for handles all criteria reaches approximately κ = 0.57–0.63, whereas subjective guidelines fall below κ = 0.35.
  • κ values are generally below 0.30 for preference grading, despite raw agreement of 40–87% because finetuned models win most comparisons.The GPT-5-Thinking versus Claude pair has the highest agreement range at κ = 0.12–0.30.
  • ML has the most consistent inter-judge agreement, while ArXiv and PubMed vary more, especially for comparisons involving Claude.Across model families, agreement is primarily task- and domain-dependent rather than model-dependent.

B.7 Full Benchmark Evaluation Results

Across domains and judges, GPT-5-Thinking performs best, while finetuning improves Qwen-3-30B scores by 3–7 absolute percentage points, especially with domain-matched training.

  • 3–7 absolute percentage points: finetuned models outperform base Qwen-3-30B, with domain-matched finetuning producing the largest gains.The benchmark averages the percentage of rubric items fully satisfied across test samples and evaluates models with three judges.
  • GPT-5-Thinking is the top-performing judge across all domains, while Gemini-2.5-Pro scores are often 1.5–2× higher than Claude-4-Sonnet scores.GPT-5-Thinking is described as intermediate in leniency between Gemini-2.5-Pro and Claude-4-Sonnet.
  • Medical evaluations yield consistently higher scores across models, whereas Llama-3.3-70B underperforms Gemma-3-27B and Qwen-3-4B.These patterns appear in the full benchmark results across domains and model families.

C.4 Deliverables for each sample

The framework automates tool-documentation refinement through interaction feedback, but evaluations reveal important weaknesses in scalability, exploration breadth, and stopping criteria.

  • The framework achieves 7/7 on automated documentation updates and iterative feedback integration.The plan describes automation and closed-loop updates grounded in LLM-tool interactions.
  • The plan lacks active exploration mechanisms, scoring 3/7 for diversity across tool-usage scenarios.Passive monitoring may miss edge cases and uncommon usage patterns.
  • The framework receives 7/7 for trial-and-error tool learning through repeated simulated invocations.A distilled Llama-3-8B usage simulator generates invocations, executes them, and logs outcomes.
  • Overfitting prevention is incomplete because acceptance thresholds and A/B tests do not define when documentation modifications should cease.The assessment notes the absence of an alignment-based stopping rule and the risk of indefinite minor changes.
  • Cross-model compatibility is insufficiently specified and lacks validation details despite the modular interface design.The assessment does not explain how different LLM capabilities and requirements would be accommodated or tested.
  • Scalability remains weak: the framework scores 2/7 because per-tool agents, telemetry, clustering, and testing impose substantial costs.The assessment states that resource consumption likely scales at least linearly with tool count.

E.2 [ML] Training on Mistakes

The study compares training setups for research-plan and representation learning, emphasizing controlled data construction and structured objectives. It reports that carefully curated, reasoning-aware training signals improve outcomes while exposing failure modes from poorly contextualized examples.

  • Training on Mistakes: The controlled comparison matched retry and control datasets by problem type, difficulty, grade distribution, token count, and training settings.This design isolates the presence of errors and corrections as the main dataset difference.
  • Training on Mistakes: −0.8% final accuracy occurred on low-difficulty problems when retry data encouraged overfitting to common carry-over errors.The effect was statistically significant at p=0.03.
  • Training on Mistakes: Reasoning-aware conceptual errors improved the accuracy gap by +1.3% and raised BLEU-4 from 31.2 to 38.7 compared with arithmetic-only noise.The results support error traces with explicit correction and backtracking.
  • Training on Mistakes: +1.1% accuracy in medium-difficulty problems was achieved with conceptually grounded, trace-aligned retry data, with no cost in final answer correctness.The authors recommend filtering errors, enforcing trace consistency, and limiting error frequency.
  • Representation Learning: CECL jointly models ensemble-level and channel-level structure to learn representations across subjects with variable electrode configurations.Its design uses pretrained temporal embeddings, channel-aware augmentation, and dual-level contrastive learning.
  • Representation Learning: Human evaluators favored Plan B over Plan A, citing its subject-specific channel embeddings, hierarchical sampling, and downstream proxy task.One annotator scored Plan A 8/10 and Plan B 9/10, while another criticized Plan A’s assumptions and matching requirements.
Loading 2512.23707v1…