PaperlayerView exact revision on arXiv

Source-linked AI summary

Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training

Moein Salimi, Babak Hosseini Mohtasham, Amin Aghakasiri, Mahdi Naieni, Amir Hossein Qeysarbeigi, Mohammad Masih Shalchian Nazer, Zahra Azar, Mahdi Jafari Siavoshani, Mohammad Hossein Rohban

arXiv:2604.16723v1cs.AIcs.LG

TL;DR

LLMs show promise for scientific ideation, but current approaches have limited ability to produce genuinely innovative ideas, and reward design remains vulnerable to hacking. This paper uses RL post-training with a multi-agent judge and reports significant improvements over SFT and strong baselines in automatic and human expert evaluations.

  • Problem

    Current inference-time approaches have limited ability to produce genuinely innovative ideas, while designing rewards that capture novelty and feasibility without being exploited remains challenging.

    Source: §1 · p. 2

  • Method

    The framework uses RL post-training with a multi-agent judge that rewards whether generated ideas match target abstracts from accepted papers.

    Source: §1 · pp. 2–3

  • Results

    The method significantly improves over SFT and strong baselines in both automatic and human expert evaluations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that GRPO-based RL post-training enhances scientific idea generation and produces ideas that are significantly more novel than baseline approaches.

    Source: §5 · p. 10

    Evaluation is limited to ICLR and NeurIPS papers, both focused on computer science.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated potential in automating scientific ideation, yet current approaches relying on iterative prompting or complex multi-agent architectures often suffer from hallucination or computational inefficiency. A critical bottleneck in applying Reinforcement Learning (RL) to this open-ended domain is reward hacking -- where models exploit imperfect evaluation proxies to maximize scores without producing genuine scientific innovation. To address these limitations, we propose an RL framework explicitly tailored for high-quality scientific idea generation. We propose the first multi-agent reward function designed to serve as a judge, decoupling methodological validation from implementation details while providing strict binary rewards that are robust to reward hacking. To effectively optimize against this sparse signal, we utilize an unbiased variant of Group Relative Policy Optimization to mitigate artificial length bias. We grounded our training in ICLR-320, a curated dataset of problem-solution pairs extracted from ICLR 2024 proceedings. Experiments demonstrate that our framework significantly outperforms state-of-the-art baselines across expert-evaluated metrics of novelty, feasibility, and effectiveness.

1 Introduction

The introduction frames scientific ideation as a gap between LLMs’ promise and their limited ability to produce genuinely innovative ideas, while inference-time methods and imperfect evaluation complicate RL. It presents RL post-training with a multi-agent judge and reports improvements over SFT and strong baselines in automatic and human expert evaluations.

  • Reward hacking includes overly long, irrelevant, or generic proposals and placeholder phrases that can prompt the judge to generate the missing idea itself.Source: §1 · p. 2
  • The RL framework trains a compact LLM on ICLR 2024 examples, using an LLM judge to match generated ideas against target abstracts.The target abstracts come from accepted papers; the training subset is from ICLR 2024.Source: §1 · p. 2
  • Judge prompting and multi-agent deliberation are integrated into the online RL loop as its reward function, rather than being used only after generation.Source: §1 · pp. 2–3
  • The final multi-agent deliberative judge achieves 1.00 precision on the expert benchmark, addressing false-positive rewards that can drive reward hacking.Earlier single-call and reduced multi-agent judges had mean precision of 0.85 and 0.906, respectively.Source: §1 · p. 3
  • Inputs are problem-only research questions that state the scientific challenge while omitting the proposed solution or method.Source: §1 · p. 3
  • The framework significantly improves over SFT and strong baselines in both automatic and human expert evaluations.Source: §1 · p. 3

2 Related Work

Related work spans retrieval and agentic systems for idea generation, evaluation methods for open-ended scientific ideas, and RL approaches to model improvement. This paper adapts GRPO to scientific ideation and uses multi-agent debate as a training reward.

  • Prior idea-generation systems use retrieval, iterative refinement, or end-to-end agentic pipelines, while many rely on inference-time strategies or frozen models.Examples include SciMon Wang et al. (2024a), ResearchAgent Baek et al. (2025), The AI Scientist Lu et al. (2024), and CycleResearcher Weng et al. (2025).Source: §2 · pp. 3–4
  • Other approaches emphasize idea structure and creative generation, including Chain of Ideas Li et al. (2025), LDC Li et al. (2024), and SPARK Sanyal et al. (2025).Source: §2 · pp. 3–4
  • Assessing novelty and feasibility is difficult without human experts, and multi-agent debate has been used to improve evaluation quality.ChatEval Chan et al. (2024) is cited as an example of agents discussing open-ended questions to reach higher-quality evaluations.Source: §2 · p. 4
  • GRPO is described as a memory-efficient PPO variant that normalizes scores within groups and removes the need for a value-function critic.Source: §2 · p. 4
  • The paper adapts GRPO to research idea generation so models can internalize reasoning from open questions to methodologically sound, novel solutions.Source: §2 · p. 4

3 Methodology

The framework trains a research-idea generator through a cycle of candidate generation, debate-based binary evaluation, and Dr. GRPO policy optimization. Its judge separates methodological assessment from implementation details, while Dr. GRPO addresses length bias in the reward-driven updates.

3.1 Overview of the Framework

  • The five-stage pipeline curates ICLR 2024 questions and abstracts, generates ideas with Qwen2.5 14B, assigns binary judge rewards, optimizes with Dr. GRPO, and evaluates novelty, feasibility, and effectiveness.Final evaluation uses both LLM-as-a-judge and human expert review.Source: Figure 1 · p. 5

3.2 Multi-Agent Evaluation Framework (The Judge)

  • The judge simulates a peer-review committee, assessing methodology, novelty, and critical assumptions while ignoring dataset choices, metrics, and experimental setups.Two Analysts decompose and compare ideas; an Evaluator reviews their interaction and produces a binary reward with a reasoning trail.Source: §3.2 · p. 5
  • The multi-turn debate transforms evaluation into dynamic verification, reducing reward-signal variance and penalizing plausible-sounding but vacuous text.Adversarial cross-examination exposes logical gaps and hallucinated complexity that single-pass scoring or static majority voting may miss.Source: §3.2 · p. 6

3.3 Policy Optimization via Dr. GRPO

  • Dr. GRPO uses group scores instead of a separate value network, reducing memory overhead for the 14B-parameter generator.Standard GRPO’s sequence-level advantages can give longer outputs disproportionately large gradients regardless of quality.Source: §3.3 · p. 6
  • Length-normalized token-level advantages let each response contribute equally to the gradient regardless of length, while the clipped objective combines with binary judge rewards to reinforce methodologically sound ideas.The objective uses the token-level importance sampling ratio and clipping hyperparameter.Source: §3.3 · p. 6

4 Experiments and Results

The experiments compare the proposed model with baselines using automated and expert evaluations, then test how Judge architecture affects reward reliability.

4.1 Implementations

  • The method and baselines use either 4-bit Qwen 2.5 14B or full-precision Qwen 2.5 7B; a Best-of-10 approach compute-matches costly baselines.The Best-of-10 setup generates ten ideas per question and uses the same model to select one.Source: §4.1 · pp. 6–7

4.2 Data

  • The dataset retains high-scoring accepted papers classified as presenting novel ideas, then extracts their golden ideas and research questions from full texts.ICLR 2024 questions use Gemini 2.5 Pro; NeurIPS 2025 questions use DeepSeek-V3.1.Source: §4.2 · p. 7
  • RL post-training uses only the 320-sample ICLR 2024 training split; the study also includes 203 AI-Scientist papers and 30 PhD-contributed research questions.The RL and SFT dataset splits are summarized in Table 1.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Baselines

  • Baselines include quantized zero-shot Qwen, SFT, and agentic research systems; AI-Scientist v1 is limited to shared questions because it requires code-template context.The comparison includes a modified Research Agent and GPT Researcher.Source: §4.3 · pp. 7–8

4.4 Evaluation

  • The method outperforms baselines under both evaluation protocols across novelty, feasibility, and effectiveness, although pairwise feasibility is lowest across datasets while absolute feasibility remains comparable.The comparison uses unsloth-Qwen2.5-14B 4-bit quantized models.Source: Table 2 · p. 8, §4.4 · p. 9
  • On 30 open research questions, domain experts assigned the method the highest average scores across novelty, feasibility, and effectiveness.Experts rated anonymized ideas on a 1–5 scale.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 Ablation Studies on Multi-Agent Judge Architecture

  • Removing the Analyst lowers precision to 0.780, while removing the Evaluator leaves precision perfect but reduces recall to 0.033.The ablations indicate that methodological decomposition and final aggregation are both needed for reliable reward decisions.Source: §4.5 · p. 10
  • The study adopts 2 Analysts plus 1 Evaluator as its default Judge, achieving perfect precision with recall of 0.300 amid accuracy and test-time-compute trade-offs.Multiple Analysts generally improve precision, while recall peaks with a moderate committee size.Source: §4.5 · p. 10

5 Conclusion

The paper applies GRPO reinforcement-learning post-training to scientific idea generation and iteratively refines its evaluation protocol to mitigate reward hacking. Automatic and expert evaluations find the resulting ideas significantly more novel than baseline ideas.

  • Automatic evaluations and human expert assessments find the method generates significantly more novel ideas than baseline approaches.The evaluation protocol was refined after analyzing outputs across dozens of judge prompting strategies.Source: §5 · p. 10
  • The method uses GRPO for reinforcement-learning post-training of a large language model on scientific idea generation.Source: §5 · p. 10
  • The team manually analyzed outputs across dozens of judge prompting strategies and iteratively refined the evaluation protocol to mitigate reward hacking.Source: §5 · p. 10

Limitations

The study's scope is limited by its training data, base model, training starting point, and evaluation domains. The authors identify broader datasets, larger models, supervised fine-tuning, and cross-domain evaluation as future directions.

Limitations

  • RL post-training relies on 320 high-quality ICLR 2024 examples; more expert-validated scientific ideas could further improve performance.Source: Limitations · p. 11
  • The study fine-tunes a 4-bit-quantized Qwen 14B model, while larger or more capable foundation models may yield higher-quality and more diverse proposals.Source: Limitations · p. 11
  • RL post-training starts from a pretrained instruct model rather than a supervised fine-tuned checkpoint, which might better align the model with scientific discourse first.Source: Limitations · p. 11
  • Evaluation covers ICLR and NeurIPS papers in computer science, leaving generalizability to other scientific domains an open question.Source: Limitations · p. 11

A.1 Model Specifications

The appendix specifies the policy and reward models and summarizes GRPO's reward-normalization approach. Table 4 is identified as the reference for GRPO post-training hyperparameters.

A.1 Model Specifications

  • The policy is Qwen 2.5 14B loaded in 4-bit quantization, and GPT-OSS-120B generates the reward signal within the multi-agent framework.Source: §A.1 · p. 14

A.2 GRPO Hyperparameters

  • GRPO normalizes rewards across outputs generated for the same input, avoiding a separate value-function network; Table 4 lists its post-training hyperparameters.Source: §A.2 · p. 14, Table 4 · p. 14

B Data Construction

The training data comes from selected post-cutoff ICLR 2024 papers, paired with research questions extracted from abstracts. The generation prompt uses a structured chain-of-thought format before the final proposal.

B Data Construction

  • The dataset retains original-contribution papers accepted at ICLR 2024, using papers published after the model's September 2024 knowledge cutoff and filtering by review scores and paper type.GPT OSS 120B identifies survey papers and empirical studies that do not propose novel ideas for removal.Source: B Data Construction · p. 14
  • DeepSeek-V3.1 extracts each paper's core research question from its abstract, forming research-question and abstract input–target pairs.The question is the prompt, and the abstract is the target against which generated ideas are rewarded.Source: B Data Construction · p. 14

C Idea Generation Prompts

D LLM-as-a-Judge Details

The judge design evolved through several evaluation strategies toward a deliberative system that separates methodological decomposition from final aggregation. The final combined system achieved perfect precision on expert-annotated data.

D LLM-as-a-Judge Details

  • Prompt and architecture variants were tested on a fixed expert-annotated set of 177 triples over five runs, reporting mean precision, recall, F1, and accuracy.Precision measures the share of positive predictions that were correct; false positives are a key reward-hacking risk.Source: D LLM-as-a-Judge Details · pp. 14–15
  • The judge-development process compared single-call, embedding, structured-compliance, keyword-alignment, and venue-style novelty strategies before the final architecture.The strategies surfaced trade-offs including verbosity bias, threshold sensitivity, partial-compliance exploitation, conceptual misalignment, and novelty-manipulation artifacts.Source: D LLM-as-a-Judge Details · p. 15
  • Two debating Analysts decompose methodology before an Evaluator issues the binary decision, separating decomposition, critique, and aggregation.The final combined system's robustness is attributed to hierarchical structure and prompt refinement.Source: D LLM-as-a-Judge Details · p. 15
  • A single-LLM Analyst reached 0.906 precision but remained susceptible to reward hacking, showing prompt quality alone did not eliminate exploitation.The benchmark labels were determined by seven domain experts, with disagreements resolved by consensus meetings.Source: D LLM-as-a-Judge Details · p. 15
  • Perfect precision (1.0) on the hand-labeled dataset is reported for the final multi-agent judge.The table caption states that this eliminates false-positive rewards, which are critical to mitigating reward hacking.Source: D LLM-as-a-Judge Details · p. 15, Table 5 · p. 16

E Multi-Agent Reward System Details

The multi-agent judge uses role-separated discussion to assess methodological alignment, with an ablation favoring two Analysts and one Evaluator. The appendix also provides prompts for idea generation, agent roles, and idea extraction.

E Multi-Agent Reward System Details

  • Scaling analysis to two Analysts and one Evaluator maximized precision while maintaining competitive recall; removing the Analyst degraded precision, while removing the Evaluator collapsed recall.The selected configuration achieved precision 1.0 and recall 0.300.Source: E Multi-Agent Reward System Details · p. 16, Figure 3 · p. 17

F Prompts

System Prompt: Best of N

  • Best-of-N selection asks a reviewer to assess novelty only, identify each idea's divergence from the standard paradigm, and choose the most distinct idea.The prompt explicitly excludes feasibility, cost, and implementation risks from this selection.Source: System Prompt: Best of N · p. 21

System Prompt: Supervised Fine-Tuning

User Prompt: The Critic Agent

  • The Critic prompt compares the abstract's and generated idea's methodological assumptions and logic, then reassesses their alignment using prior critiques.Source: User Prompt: The Critic Agent · p. 25

System Prompt: The Evaluator Agent

User Prompt: The Evaluator Agent

  • The Evaluator's user prompt asks it to summarize participants' positions and reasoning before making its own assessment based on methodology and core contributions.Source: User Prompt: The Evaluator Agent · p. 27

System Prompt: Idea Extraction

  • The idea-extraction prompt requests the central research question, a brief reasoning trail, and the core novel methodology, while excluding evaluation context and performance comparisons from the method description.Source: System Prompt: Idea Extraction · p. 28

G Sample Generations

The appendix presents generated-idea examples for three representative research questions from the curated golden-RQ set, including outputs from the proposed method and several baselines.

Sample 1

Sample 2

Sample 3

H Additional Quantitative Analysis

The additional quantitative analysis compares mean pairwise scores and standard deviations across ICLR 2024 and NeurIPS 2025. The proposed method leads in novelty and effectiveness, while feasibility results favor other systems.

Additional Quantitative Analysis

ICLR 2024

  • On ICLR 2024, the figure caption reports a significant advantage for the proposed method in Novelty and Effectiveness.Source: Figure 52 · p. 40

Dataset comparison

  • Table 6 presents performance comparisons across datasets for unsloth-Qwen2.5-7B, with best scores bolded.Source: Table 6 · p. 41

Related papers

Loading 2604.16723v1…