PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Yuhan Wang, Zhengxi Lu, Yuchen Yan, Kaitao Song, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2608.31119v1cs.CL

TL;DR

Research-plan generation lacks scalable, verifiable feedback, while existing rubric pipelines risk criterion leakage and compress supervision into rollout-level rewards. PaperGym separates question and criterion sources, uses the rubric first for OPSD self-distillation and then for GRPO, and reports consistent gains across Qwen3 scales, stronger PaperGym-20k comparisons, and a 73.48 ResearchQA score for Qwen3-8B.

  • Problem

    Research planning lacks automatically verifiable answers, and existing pipelines can let models earn rubric rewards by paraphrasing questions while reducing rubric feedback to one scalar per rollout.

  • Method

    PaperGym builds training environments from papers by deriving questions from goals and backgrounds, criteria from methods and experiments, and applying the rubric through OPSD followed by GRPO.

  • Results

    Across Qwen3-1.7B, 4B, and 8B, the two-stage schedule improves five-benchmark averages by +5.6, +5.0, and +4.8 points over alternatives, while Qwen3-8B reaches 73.48 on ResearchQA.

  • Takeaways & Limitations

    PaperGym-20k models win 58.1% of three-way comparisons versus 28.2% for RubricHub Science, and the released resources include two held-out benchmarks.

Abstract

from arXiv · show

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.

1 Introduction

PaperGym addresses the lack of scalable, verifiable feedback for research-plan generation by separating question construction from rubric extraction and using rubrics for both training and evaluation. The resulting corpus and two-stage schedule improve rubric quality and model performance across benchmarks.

  • Motivation: Research plans lack automatically checkable answers, so expert review cannot provide reinforcement-learning feedback at training scale.Research planning specifies hypotheses, methods, and experiments, while its quality requires expert judgment.
  • Limitations of existing pipelines: 11.90% to 34.10% of existing criteria can be inferred from questions alone, enabling reward gains through paraphrase.Existing pipelines also assess experimental design mainly through generic guidelines rather than instance-specific criteria.
  • PaperGym framework: PaperGym reduces criterion leakage to 3.7% by synthesizing questions from research goals and backgrounds and criteria from methods and experiments.Its ten atomic binary criteria cover methodological innovation and experimental design.
  • Training results: +5.6, +5.0, and +4.8 points improve five-benchmark averages for Qwen3-1.7B, 4B, and 8B under the two-stage schedule.The schedule uses the rubric first as privileged context for self-distillation and then as the GRPO reward, outperforming supervised fine-tuning, either stage alone, and reverse ordering.
  • Training results: 58.1% of three-way comparisons favor PaperGym-20k models versus 28.2% for RubricHub Science models under the same training recipe.The trained Qwen3-8B reaches 73.48 on ResearchQA, above Kimi K2.6.
  • Resources: PaperGym-20k contains 20,000 instances, while PaperGym-Innov and PaperGym-Design separately evaluate methodological innovation and experimental design.The corpus is constructed across three domains, and the benchmarks are held out for evaluation.

2 Related Work

Prior work automates research through modular agents, search-based refinement, or training-driven augmentation, while rubric-based research-plan training has mainly used sequence-level supervision. PaperGym combines rubric-based OPSD with GRPO to provide denser guidance.

  • Research automation: Agent-based and search-based systems coordinate research stages or iteratively refine candidates while keeping the underlying model frozen.These paradigms bound output quality by the capabilities of the fixed model.
  • Rubric-based training: Rubric-driven research-plan methods use prompt-specific criteria, but prior approaches rely on sequence-level supervision.PaperGym addresses this limitation with rubric-based OPSD and GRPO for dense token-level guidance.
  • Training signals: GRPO uses group-level outcome rewards without a learned critic, but its sequence-level scalar treats all tokens uniformly.OPSD supplies a complementary denser signal through teacher-student conditioning.
  • Rubric-based training: Rubric-as-Rewards provides interpretable criterion-level supervision but requires multiple judge calls and can expose policies to reward hacking.Recent OPSD-based methods densify rubric rewards, although privileged information leakage remains a concern.

3 Method

PaperGym converts each paper into a training environment by separating question construction from rubric construction, then uses those rubrics in a two-stage training schedule. The dataset combines specialized and general criteria, while scorer analyses examine its composition and reliability.

  • PaperGym frames research-plan generation as a policy producing a coherent solution for a research question, with the rubric serving as its critic.
  • Data Preprocessing: The pipeline decomposes papers into Research Goal, Background, Research Method, and Experimental Design stages.It uses arXiv LaTeX source and a map-reduce procedure to produce coherent four-stage summaries.
  • Problem Construction: Questions use Research Goal and Background, while reference answers use Research Method and Experimental Design from disjoint sections.This construction prevents criterion leakage at its source.
  • Rubric Generation: Specialized rubrics measure methodological innovation and experimental design, combining question-conditioned and answer-grounded criteria before deduplication, ranking, and filtering.General rubrics enforce completeness, specificity, soundness, efficiency, and ethical safety across instances.
  • Rubric Generation: 20,000 instances span three domains, with approximately 50% CS, 25% Physics, and 25% Econ; specialized criteria target Method 63.8% and Experiment 36.2%.The dataset composition is illustrated in Figure 2.
  • Rubric Generation: Four scoring models were highly self-consistent across five temperature-0 runs, and Qwen3-8B reached nearly 80% agreement with Kimi K2.6.
  • Rubric-Centered Training: The first training stage uses rubric-conditioned OPSD, where a privileged teacher evaluates on-policy prefixes and the student matches it by minimizing KL divergence.Rubric principles preserve broader valid continuations than answer-conditioned distillation and align with the second-stage reward.
  • Rubric-Centered Training: The second stage applies GRPO with self-graded binary rubric verdicts averaged over specialized and general criteria, combining them with α = 0.7.The reward emphasizes task-specific scientific fit while retaining pressure on general proposal quality; GRPO standardizes rewards within groups and uses a KL penalty against a reference policy.

4 Experiments

Experiments evaluate PaperGym’s data construction and two-stage training across model scales, benchmarks, ablations, and training dynamics. The results show lower criterion leakage, stronger performance from OPSD+GRPO, and competitive Qwen3-8B results.

  • Main Results: +5.56, +5.04, and +4.81 average-score improvements are achieved by OPSD+GRPO on Qwen3-1.7B, 4B, and 8B, respectively.The schedule achieves the highest score on every test set at every scale and outperforms SFT, either stage alone, and the reverse ordering.
  • Main Results: Qwen3-8B scores 24.47 on PaperGym-Innov and 21.88 on PaperGym-Design, exceeding Intern-S1-Mini and Rebicon-Preview on both benchmarks.It also reaches 73.48 on ResearchQA, above S1-VL-RL at 72.09 and Kimi K2.6 at 73.19.
  • Main Results: PaperGym-20k models win 58.1% of Overall Score comparisons, versus 28.2% for RubricHub-trained models and 13.7% for the base model.All three models share the same training regimen, isolating the contribution of training data on ResearchPlanGen-ML.
  • Data Scaling: Performance rises with data scale from 16.12 to 19.21 on PaperGym-Innov and from 13.89 to 16.96 on PaperGym-Design.The scaling study uses GRPO with datasets increased from 0.5k to 15k instances sampled evenly across three domains.
  • Data Construction: Criterion leakage is 3.73%, 4.71%, and 4.97% on PaperGym-20k, PaperGym-Innov, and PaperGym-Design, versus 11.90%–34.10% for existing benchmarks.The reported leakage reduction is approximately 3–9×.
  • Ablations: The full rubric pipeline outperforms single-source and innovation-only variants, while replacing the rubric generator causes drops of -2.00 / -1.18.Using Qwen3-8B for extraction causes smaller drops of -0.97 / -0.67, identifying rubric quality as the critical bottleneck.
  • Training Dynamics: OPSD+GRPO converges faster and to higher final accuracy than OPSD alone, GRPO alone, and GRPO+OPSD.OPSD raises entropy during warm-up, while GRPO reduces entropy toward high-reward regions; the post-OPSD entropy drop is sharper.

5 Conclusion

PaperGym turns each research paper into a complete training environment by separating question and solution sources, using dual-dimensional rubrics twice during training. The resulting corpus and training approach improve benchmark performance, including for the trained Qwen3-8B.

  • 5 Conclusion: PaperGym decouples research questions from reference solutions by deriving them from background and goals versus methods and experimental design.This design eliminates criterion leakage at its source.
  • 5 Conclusion: PaperGym-20k contains 20,000 instances with a 3.7% criterion-leakage rate, reported as the lowest among open-source alternatives.
  • 5 Conclusion: PaperGym’s rubrics assess methodological innovation and experimental design, then serve as privileged context for self-distillation and as rewards for GRPO.
  • 5 Conclusion: Qwen3 models trained with this rubric-centered evolution improve on in-domain and external benchmarks, and the trained Qwen3-8B surpasses Kimi K2.6 on ResearchQA.

A Dataset Analysis

PaperGym’s specialized rubrics classify criteria across methodological innovation and experimental design, with methodological criteria receiving the larger share while experiments retain substantial representation.

  • A Dataset Analysis: Method concerns novelty in the proposed method, algorithm, model, framework, technique, theoretical contribution, or solution design.
  • A Dataset Analysis: Experiment concerns setup, metrics, baselines, ablations, dataset selection, human-machine comparisons, robustness, and related experimental designs.
  • A Dataset Analysis: 63.76% of 9,999 classified criteria are Method and 36.24% are Experiment across 1,000 sampled instances.
  • A Dataset Analysis: The rubric distribution emphasizes methodological innovation while still assigning substantial weight to experimental design.

A.2 Criterion Leakage Detection

Criterion leakage detection asks whether rubric criteria can be directly inferred from questions alone, and applies this judge-based measure across multiple benchmarks.

  • A.2 Criterion Leakage Detection: Criterion leakage is measured by asking an LLM judge to mark each rubric criterion as directly inferable from the question alone.
  • A.2 Criterion Leakage Detection: The leakage rate is the fraction of criteria judged directly inferable from the question alone.
  • A.2 Criterion Leakage Detection: The detection is applied to in-domain and out-of-domain benchmarks including HealthBench, RubricHub Science, ResearchPlanGen-ML, ResearchPlanGen-ArXiv, and ResearchQA.
  • A.2 Criterion Leakage Detection: PaperGym’s decoupling pipeline attains a substantially lower leakage rate than existing benchmarks.

B Training Details and Additional Ablations

An ablation of OPSD teacher inputs finds that rubric conditioning performs best, while adding the reference answer does not recover the lost performance.

  • B Training Details and Additional Ablations: The rubric-conditioned teacher outperforms the answer-conditioned teacher by +1.08 on Innov and +0.88 on Design.
  • B Training Details and Additional Ablations: Conditioning on the reference answer narrows the valid continuations preserved by distillation because it specifies a single realized solution.
  • B Training Details and Additional Ablations: Adding the reference answer alongside the rubric does not recover the rubric-conditioned teacher’s performance.

B.2 Parameter Sensitivity

The GRPO reward is sensitive to the balance between specialized and general rubric terms, with a 7:3 ratio performing best on both in-domain benchmarks.

  • Parameter Sensitivity: The sensitivity experiment evaluates the specialized-to-general reward ratio using a Qwen3-1.7B policy after 200 OPSD steps and 100 GRPO steps.
  • Parameter Sensitivity: Table 6 reports rubric scores on PaperGym-Innov and PaperGym-Design for the three tested reward-mixing configurations.
  • Parameter Sensitivity: 17.17 on PaperGym-Innov and 12.35 on PaperGym-Design are the highest scores under the 7:3 specialized-to-general ratio.The sweep compares ratios of 8:2, 7:3, and 6:4 while holding other settings fixed, confirming α = 0.7.

B.3 Stage-Ordering Swap between OPSD and GRPO

Across all three model scales and five benchmarks, OPSD→GRPO outperforms the reversed GRPO→OPSD ordering. The paper attributes this advantage to warm-started exploration and a broaden-then-optimize entropy curriculum.

  • Stage-ordering results: The forward OPSD→GRPO ordering wins on every benchmark at every scale, with five-benchmark average gains of +2.04, +0.99, and +1.31 points.The reverse-ordering comparison keeps data and hyperparameters identical while swapping only the stage order.
  • Stage-ordering results: The largest ordering penalty occurs at 1.7B, while PaperGym-Innov gains are +2.97, +2.27, and +1.92 across 1.7B, 4B, and 8B.
  • Mechanisms: Cold-start GRPO produces weak rollouts and noisy advantages, whereas OPSD warm-up yields rubric-aligned plans before policy optimization.The paper links this difference to unstable exploration and the KL penalty resisting the distribution shift needed to escape low reward.
  • Stage-ordering results: Figure 5 shows OPSD→GRPO leading throughout Qwen3-4B training and converging to higher final accuracy than GRPO→OPSD.
  • Reward scoring: GRPO rewards combine specialized and seven general rubric scores as 0.7 Rspec + 0.3 Rgen using deterministic self-scoring.A frozen base-policy copy grades responses criterion by criterion; the 4B model scores 1.7B responses because its self-scoring is unreliable.
  • Mechanisms: OPSD raises output entropy and GRPO reduces it, forming a broaden-then-optimize curriculum that the reverse ordering cannot recover after premature collapse.The entropy reduction during GRPO is substantially more pronounced after OPSD pre-warming.

C.1 Pairwise Win-Rate Evaluation Protocol

The pairwise evaluation compares trained and untrained Qwen3-1.7B variants on an independent benchmark using expert judges and criterion-level plan comparisons.

  • Data-quality comparison: 68.0% overall is the win rate of the PaperGym-trained OPSD model against the RubricHub-trained OPSD model.Both trained models use the same Qwen3-1.7B architecture and OPSD recipe, isolating the training corpus difference.
  • Training gains: 73.6% and 68.2% are the overall win rates of the PaperGym OPSD and two-stage models against the untrained base model.These comparisons use ResearchPlanGen-ML, which is independent of all training data.
  • Criterion-level effects: GRPO improves goal alignment from 66.0% to 71.1% and novel insight from 63.2% to 80.0% against the same base model.
  • Criterion-level effects: GRPO lowers scientific soundness from 72.0% to 62.1%, execution quality from 71.9% to 66.3%, and expected impact from 73.5% to 71.6%.The paper attributes this redistribution to innovation-heavy rubrics and reward hacking that produces longer, more complex responses.
  • Evaluation protocol: The pairwise prompt compares two plans and lets judges select the better plan or declare a tie for each criterion.

C.2 Three-Way Win-Rate Evaluation Protocol

The three-way protocol evaluates research plans from three models on the same scenario using randomized presentation, five criteria, expert judging, and overall blueprint scores.

  • Evaluation setup: 685 ResearchPlanGen-ML instances are evaluated through three-way comparisons with an LLM-as-a-judge protocol.Each question is paired with one response from each of the three compared models.
  • Evaluation setup: The three plans are randomly assigned to Plan A, Plan B, and Plan C to reduce positional bias.The judge is not told which model produced each plan.
  • Evaluation criteria: Plans are ranked on goal alignment, novel insight, scientific soundness, execution quality, and expected research impact.The criteria cover research objectives, originality, rigor, feasibility, and likely value as described by the protocol.
  • Aggregation: Overall scores from 1 to 10 measure each plan’s value as a research blueprint for an average graduate student.The protocol considers both scientific quality and practical usefulness.

D Data Construction Details

PaperGym constructs research-planning environments by separating question synthesis from reference-answer and rubric construction, then applies structured extraction and multi-stage summarization to paper sections. Its rubric-generation prompts require concrete design elements that cannot be inferred from the task alone.

  • Map extraction: The paper body is split at section boundaries, and Qwen3-235B-A22B extracts research field, background, solution, and experimental-design information from each section.The extraction requires sentence-level faithfulness, empty fields when evidence is absent, and escaped LaTeX for valid JSON output.
  • Reduce summarization: Per-section extractions are grouped by stage and reduced by Qwen3-235B-A22B into concise, coherent summaries using dedicated prompts.The reduce prompts separately target research field, background, solution, and experiment design, process, and analysis strategy.
  • Question-answer construction: The research question is synthesized from research field and background, while the reference answer is synthesized from solution and experimental-design stages.This construction draws the question and answer from disjoint paper sections to prevent answer leakage into the question.
  • Rubric construction: Each rubric must assess a specific design decision grounded in the reference solution and not stated, implied, or predictable from the experimental-design task alone.The prompt instructs the generator to discard any rubric that a reader could determine without seeing the reference solution.
  • Rubric construction: Rubrics must name concrete discriminating elements, such as a baseline category, dataset property, ablation, or control, rather than generic experimental requirements.Generic criteria involving baseline comparisons, evaluation metrics, or confounding control are explicitly rejected.
Loading 2608.31119v1…