PaperlayerView exact revision on arXiv

Source-linked AI summary

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister

arXiv:2605.10899v1cs.CLcs.LG

TL;DR

Deep research agents are difficult to train because their outputs lack ground-truth verification, their tool-augmented rollouts are long, and conventional post-training offers limited experience reuse. RubricEM addresses this with stagewise rubric-conditioned policy decomposition, denser stage-aware credit, and reflection-based memory, achieving 55.5 average among non-proprietary systems with its 8B model.

  • Problem

    Training deep research agents remains difficult because outputs lack ground-truth verification, feedback is coarse and delayed, and conventional post-training provides limited experience reuse.

    Source: §1 · p. 1

  • Method

    RubricEM uses rubrics to condition stagewise policy execution, assign process-level credit through SS-GRPO, and distill judged trajectories into reusable reflections via a shared-backbone meta-policy.

    Source: §1 · pp. 1–2

  • Results

    55.5 average: RubricEM-8B-RL achieves the highest score among non-proprietary deep research systems evaluated, surpassing strong open baselines.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    RubricEM provides a rubric-guided reinforcement-learning recipe for long-form research beyond verifiable rewards.

    Source: §6 · p. 11

    SS-GRPO depends on sufficiently aligned stagewise judging with bounded noise, since intermediate judging can introduce judge noise.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory. Based on this view, we introduce RubricEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution. RubricEM first makes research trajectories stage-aware by conditioning planning, evidence gathering, review, and synthesis on self-generated rubrics. It then assigns credit with Stage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback for long-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM.

1. Introduction

RubricEM treats rubrics as a shared interface for structuring deep-research execution, judging intermediate decisions, and converting experience into reusable guidance beyond verifiable rewards. Its stagewise policy decomposition, denser credit assignment, and reflection meta-policy produce RubricEM-8B, which performs strongly across four long-form research benchmarks.

Motivation

  • RubricEM addresses deep-research training challenges caused by unverifiable outputs, coarse delayed feedback over tool-augmented rollouts, and limited reuse of judged attempts.Deep research requires planning, searching, evidence evaluation, and long-form synthesis, while existing approaches largely rely on verifiable search proxies or imitation data.Source: §1 · p. 1

Rubric-guided policy decomposition

  • RubricEM uses self-generated rubrics across planning, research, review, and answer synthesis, turning flat trajectories into stage-specific decision modes and optimization units.The same rubrics guide search and synthesis, provide on-policy references for judging, and generate structured traces for reflection.Source: §1 · pp. 1–2

Stage-Structured GRPO

  • Stage-Structured GRPO scores Plan, Research, Review, and Answer with evolving stage rubrics, producing denser returns that combine local quality with downstream impact without a critic.This replaces broadcasting one terminal score to every token with stage-specific process-level feedback.Source: §1 · p. 2

Reflection Meta-Policy

  • Reflection Meta-Policy training jointly updates a shared backbone from judged rubric-grounded reflections and stores the best accepted reflection for within-episode refinement and cross-episode transfer.An asynchronous reflection branch trains alongside task-policy RL, while the rubric bank provides natural-language memory for future rollouts.Source: §1 · pp. 2–3, Figure 1 · p. 2

Results

  • RubricEM-8B achieves state-of-the-art performance among comparable open models across four long-form research benchmarks, surpasses strong prior RL systems with fewer training steps, and approaches Gemini and OpenAI Deep Research.The model is an 8B deep research agent trained with 1400 RL steps.Source: §1 · p. 3

2. Related Work

Prior deep-research post-training largely targets short-answer, verifiable-reward settings, while RubricEM addresses open-ended long-form research through finer-grained credit assignment and jointly trained meta-policy evolution. It also improves meta-policy efficiency for real-world research by removing cross-rollout dependencies and using reflection-based training.

The Post-training recipes of deep research agents

  • RubricEM targets open-ended, long-form deep research, extending DR Tulu (Shao et al., 2025) with fine-grained credit assignment and jointly trained meta-policy evolution.These additions provide denser learning signals and reusable guidance during long-horizon reinforcement learning.Source: §2 · p. 3
  • Most open-source and proprietary deep-research training efforts focus on short-answer or short-form tasks with verifiable rewards, imitation data, or supervision.The cited open-source efforts include Chen et al. (2025a), Han et al. (2025), Jiang et al. (2025), Jin et al. (2025), Song et al. (2025), and Zhao et al. (2025); proprietary systems include Google (2025), Moonshot AI (2025), OpenAI (2025), and Perplexity AI (2025).Source: §2 · p. 3

Credit assignment and meta-RL with language models

  • Existing fine-grained credit-assignment and Meta-RL methods are generally developed for verifiable or synthetic tasks, whereas RubricEM targets real-world research without simple automatic verification.RubricEM removes cross-rollout dependencies and designs an efficient reflection-training infrastructure to reduce meta-policy training overhead.Source: §2 · pp. 3–4

3. RubricEM

RubricEM structures deep-research trajectories around rubric-guided stages, uses stagewise judging for credit assignment, and reuses judged experience through a reflection meta-policy and rubric bank. Its design connects explicit reasoning scaffolds, adaptive rewards, and asynchronous memory updates for long-horizon research agents.

3.1. Preliminaries and notations

  • RubricEM models deep research as tool-augmented trajectories in which a language-model agent autoregressively produces actions and evidence-grounded long-form answers.Each action may be textual or a structured tool call, with subsequent tool outputs incorporated into the agent’s history.Source: §3.1 · p. 4

3.2. Structured Reasoning Scaffold

  • Explicit stage information strictly improves decisions when identical compressed contexts support different optimal actions across task-relevant stages.This addresses context aliasing that can cause inefficient exploration and compounding errors in flat long-horizon policies.Source: §3.2 · p. 4
  • RubricEM instantiates four rubric-guided stages—Plan, Research, Review, and Answer—where rubrics are generated, revised, and used to organize evidence collection, evaluation, and synthesis.Plan defines information checklists, analytical criteria, and negative constraints; Research evaluates evidence and may revise the plan; Review maps evidence to rubrics; Answer produces a cited response.Source: §3.2 · p. 5

3.3. Stage-Structured GRPO

  • Stage-Structured GRPO uses stage boundaries and rubric-guided LLM judgments to provide denser learning signals without requiring oracle intermediate rewards.It assigns each stage its own score while allowing downstream stages to contribute credit through a causal stage-dependence matrix.Source: §3.3 · p. 6
  • SS-GRPO helps when recovered intermediate process information outweighs cumulative judge misalignment, requiring sufficiently aligned stagewise judgments with bounded noise.The stagewise judge maintains separate evolving rubric buffers for Plan, Research, Review, and Answer and adapts them to the current policy distribution.Source: §3.3 · p. 6

3.4. Meta-Policy Training with Reinforcement Learning

  • RubricEM trains a shared-backbone reflection meta-policy whose accepted guidance enters an agent rubric bank, combining parametric policy updates with textual memory updates.A judge scores reflection candidates for within-episode refinement and cross-episode transfer, while only the highest-scored accepted reflection is stored.Source: §3.4 · pp. 7–8
  • The rubric bank supports both within-episode refinement and cross-episode transfer, while one-step-stale asynchronous execution keeps inference and training occupied without effectively increasing SS-GRPO wall-clock overhead.Bank items distill completed trajectories into outcome-aware reflections, and the asynchronous branch prepares the next reflection batch while current task rollouts proceed.Source: §3.4 · p. 8

4. Experiment

RubricEM is evaluated on four long-form research benchmarks using a DR Tulu-based setup, and RubricEM-8B-RL achieves the strongest average performance among evaluated non-proprietary systems while comparing favorably with proprietary systems.

4.1. Experimental Setup

  • The evaluation covers HealthBench, ResearchQA, DeepResearchBench, and ResearchRubrics within a DR Tulu-based infrastructure using Gemini-flash-grounded Google Search and Semantic Scholar.SFT uses short- and long-form data, whereas RL focuses exclusively on long-form queries sourced from diverse public datasets.Source: §4.1 · p. 8

4.2. Main Results

  • 55.5 average score makes RubricEM-8B-RL the best evaluated non-proprietary deep-research system, surpassing strong open baselines and outperforming Perplexity Deep Research on shared benchmarks while beating OpenAI Deep Research on DRB.It remains within 4.4 average points of OpenAI Deep Research overall.Source: §4.2 · p. 9, Table 1 · p. 9
  • RL raises the average score from 49.2 to 55.5 across all four benchmarks, surpasses the distilled Gemini-3.1-Pro SFT model, and uses 1,400 versus DR Tulu’s 1,900 RL steps.RubricEM starts from a stronger SFT checkpoint, but teacher and search-backend differences motivate controlled ablations in Section 5.Source: §4.2 · p. 9

5. Empirical Analysis

Empirical analyses show that RubricEM’s full RL recipe combines complementary gains from stagewise credit assignment and reflection-based experience learning. The rubric-guided scaffold improves distillation, RL stability, prompt-level research behavior, and inference-time reuse, while long-form training transfers to short-form search benchmarks.

5.1. RL Training Recipes

  • Under a matched 600-step budget, RubricEM (Full) performs best across benchmarks, while SS-GRPO and Meta-Policy each improve over Baseline-RL.The full recipe combines stagewise rubric credit with reflection meta-policy training, indicating complementary gains under matched compute.Source: §5.1 · p. 10

5.2. Structured Scaffolding and Inference-Time Experience Reuse

  • The structured rubric-guided scaffold improves SFT distillation, subsequent RL effectiveness, DRB prompting, and both cross-episode and within-episode reflection reuse.Without the scaffold, 600-step RL gains are small and unstable; Baseline-RL does not benefit from the same retrieval setting.Source: §5.2 · p. 10, Figure 6 · p. 11, §5.3 · p. 11

5.3. Short-Form Benchmark Performance

  • RubricEM transfers strongly to four out-of-distribution short-form search benchmarks despite using only long-form prompts for online RL.Long-form RL further improves RubricEM-SFT without short-form RL data, with the largest gains on more complex tasks, indicating transferable tool-use and evidence-grounding skills.Source: §5.3 · p. 11, Table 2 · p. 11

6. Conclusions … B.1. Scaffold Description and System Prompt

RubricEM addresses open-ended deep research by combining rubric-guided trajectory stages, stage-aware credit assignment, and reflection-based reusable guidance. Its scaffold operationalizes this approach through adaptive planning, iterative evidence evaluation, mandatory review, and citation-grounded synthesis.

6. Conclusions

  • RubricEM-8B shows strong benchmark performance, while ablations support its rubric-guided structure, stage-aware credit assignment, and reflection meta-policy.The framework structures long-horizon trajectories, assigns stage-aware credit, and distills judged attempts into reusable guidance through a shared-backbone reflection meta-policy.Source: §6 · p. 11

A. Additional Related Works

  • RubricEM targets the unresolved RL structure of unverifiable long-form research, complementing prior work focused on scalable search, workflows, or verifiable answer supervision.Relative to DR Tulu, RubricEM uses a structure–assign–evolve recipe intended to improve RL efficiency with fewer training steps.Source: A. Additional Related Works · p. 20
  • RubricEM provides semantic stage-level credit and a shared-backbone reflection meta-policy for open-ended research, avoiding step labels, learned critics, and sequential cross-rollout bottlenecks.Stage-specific judge scores and causal stage-dependent returns provide denser supervision, while judged trajectories become reusable rubric-grounded reflections.Source: A. Additional Related Works · pp. 20–21

B. Details on Structured Scaffolds

  • The scaffold decomposes each trajectory into four XML-marked stages: Plan, Research, Review, and Answer, each governed by distinct behavioral requirements.The stages organize planning, evidence gathering, self-evaluation, and final synthesis into a sequential outer workflow.Source: §B.1 · p. 21

B.1.1. Stage 1: Planning (Plan)

  • Planning combines exploratory thinking with a three-part structured plan containing deep analysis, a prospective rubric, and a complexity-adaptive research roadmap before the first tool call.Simple queries receive brief linear plans, whereas complex queries receive conditional look-ahead strategies; answering before searching is prohibited.Source: §B.1.1 · pp. 21–22

B.1.2. Stage 2: Research (Research)

  • Research repeatedly evaluates gathered evidence against the rubric, either continuing or revising the plan before another tool call, or proceeding when evidence is sufficient.The outer workflow remains sequential, but the Research stage supports multiple adaptive tool-use and state-evaluation iterations.Source: §B.1.2 · p. 22

B.1.3. Stage 3: Review (Review) / B.1.4. Stage 4: Answer (Answer)

  • Before answering, mandatory review verifies rubric satisfaction and converts retrieved evidence into a complexity-scaled writing plan, after which the answer follows the plan and uses inline citations.The review includes rubric_review and writing_plan; final responses must satisfy verified rubric criteria and ground nontrivial claims with citations.Source: §B.1.3 · pp. 22–23, §B.1.4 · p. 23

B.1.5. Cross-Stage Design Principles / B.1.6. Full System Prompt

  • Across the system prompt, the rubric controls trajectory substance while formatting controls surface form, with one action per turn, mandatory initial search, adaptive effort, and no simulated tool outputs.The prompt specifies rubric-guided planning, iterative searching, evidence evaluation, and answer synthesis, with outputs enclosed in answer tags and supported by citations.Source: §B.1.5 · p. 23, §B.1.6 · pp. 23, 24, 26

B.2. SFT Data Generation Process … C. Details on Stagewise Evolving Rubric Evaluation

This section describes RubricEM’s process for generating stage-structured SFT data through teacher-guided tool interaction and filtering, alongside its evolving rubric evaluation framework. It covers stage-local, adaptive rubric generation and independent, evidence-grounded scoring that distinguishes positive qualities from negative flaws across trajectory stages.

B.2. SFT Data Generation Process

  • The pipeline uses Gemini-3.1-Pro to generate complete stage-structured trajectories through repeated model generation, real tool execution, evidence evaluation, and final synthesis.Training queries come from approximately 13K diverse open-ended research questions, each paired with an exact_answer, short_form, or long_form format label.Source: §B.2 · p. 26, §B.2.1 · p. 26, §B.2.2 · p. 30

B.2.1. Teacher Model and Prompt Adaptation

  • Prompt adaptation separates first-round planning from later-round research and synthesis, requiring an initial structured plan and restricting each turn to one tool call or answer.Gemini scratchpad traces are converted to Qwen3-compatible think tags because Gemini’s internal reasoning traces are inaccessible.Source: §B.2.1 · pp. 27, 30

B.2.2. Multi-Round Generation Pipeline

  • Each trajectory begins with scratchpad, structured_plan, and one tool call; subsequent rounds process tool outputs and continue searching or produce review and answer until completion or 10 rounds.Google Search grounding and Semantic Scholar snippet search execute asynchronously, while invalid tool calls enter an error stream.Source: §B.2.2 · p. 30

B.2.3. Quality Filtering and Rejection Sampling

  • Rejection sampling discards trajectories lacking closing answers, valid non-final tool calls, required structural elements, or recovery from two consecutive tool errors.Accepted traces normalize tool names and preserve single error rounds followed by successful recovery.Source: §B.2.3 · p. 31

B.2.4. Training Data Format

  • Accepted trajectories are represented as three-message ChatML conversations using the unified Qwen3 prompt, the format-conditioned query, and the complete tool-augmented assistant trajectory.Tool outputs receive span-masking markers, preventing the model from learning to memorize search results.Source: §B.2.4 · p. 31
  • The process yields around 11k SFT samples, approximately 2k fewer than DR Tulu because repeated generation errors are filtered out.Tool outputs are masked, so only model-generated thinking, planning, tool calls, state evaluations, reviews, and answers receive gradient during training.Source: §B.2.4 · p. 31

C. Details on Stagewise Evolving Rubric Evaluation

  • The evolving rubric evaluator uses Gemini-3-Flash for adaptive rubric generation and stagewise scoring, maintaining persistent final-answer rubrics alongside capacity-limited active rubrics.For each batch question, it generates, adds, scores, and prunes rubrics based on discrimination.Source: C. Details on Stagewise Evolving… · p. 31, §C.1 · p. 32

C.2. Adaptive Rubric Generation

  • Adaptive rubrics are question-specific, discriminative, stage-local, non-redundant, and anti-hack, with stage prompts asking whether the agent understood, searched, audited readiness, and answered well.The judge may reference agents’ self-generated rubrics but cannot treat weak self-rubrics as scoring standards.Source: §C.2 · p. 32
  • Rubrics are generated separately for planning, research, review, and answering, using task- and trajectory-specific criteria rather than merely checking stage tags.Criteria include planning depth and constraints, research specificity and triangulation, review gap assessment, and answer correctness, comprehensiveness, readability, and evidence grounding.Source: §C.2 · p. 33
  • Self-generated rubrics guide evaluation but never define its standard: evaluators must not reward weak plans, penalize evidence-based improvement, or overfit rubric wording.Rubrics should favor fewer, sharper criteria, skip nondiscriminative dimensions, and target clear failure modes conservatively.Source: §C.2 · p. 33

C.3. Stagewise Scoring

  • Each trajectory is independently scored against persistent and active rubrics, with positive and negative criteria on a 0/1/2 scale and negative scores inverted in weighted stage aggregation.For each stage k, the resulting score R_i,k lies in [0, 1], and each judgment includes a brief justification grounded in specific trajectory evidence.Source: §C.3 · p. 34
  • Scoring partitions trajectories into Plan, Research, Review, and Answer spans, applies each rubric independently, and requires concrete evidence-based justifications with exact rubric indices.Positive criteria score exhibited quality, whereas negative criteria score flaws inversely so avoiding a flaw contributes positively.Source: §C.3 · p. 34

C.4. Buffer Management and Implementation · D. Asynchronous Reflection Pipeline and Windowed Curriculum · D.1. Training Pipeline Architecture

The implementation manages evolving stagewise rubrics with capacity-limited buffers, structured judge outputs, and parallelized generation and scoring. Its asynchronous pipeline overlaps rollout, judging, reflection, and policy updates across concurrent threads, while deferred reflections support meta-policy training.

C.4. Buffer Management and Implementation

  • Rubrics are retained by stage-specific capacity caps, removing the least discriminative criteria while preserving persistent rubrics and replacing stale criteria when questions recur.Caps are 3, 2, 2, and 3 for stages 1–4; discrimination is measured by score variance across the rollout group.Source: §C.4 · pp. 34–35
  • Structured JSON schemas govern rubric generation and scoring, ensuring stage-organized rubrics and indexed, justified scores without fragile post-hoc parsing.Both processes use Gemini’s structured output mode with Pydantic-derived schemas.Source: §C.4 · p. 35
  • At each training step, 32 unique questions yield 8 rollouts each, with 256 trajectories; rubric generation precedes parallel scoring, and every call uses exponential-backoff retries.Failed generation calls produce no new rubrics, while failed scoring calls fall back to terminal answer scores only.Source: §C.4 · p. 35
  • The rubric-generation and scoring phase takes approximately 5 minutes per step and overlaps gradient computation, using Gemini-3-Flash for cost and latency.Generation requires 32 calls per step, while scoring uses 256 lightweight calls over approximately 8–15 rubrics per trajectory.Source: §C.4 · p. 35

D. Asynchronous Reflection Pipeline and Windowed Curriculum

  • The training loop combines vLLM rollout generation, stagewise judge scoring for SS-GRPO rewards, policy updates, and asynchronously generated rubric-grounded reflections.Reflection generation adds compute that the system overlaps with the main training loop.Source: §D.1 · p. 35

D.1. Training Pipeline Architecture

  • Three concurrent threads separate gradient training, vLLM inference, and data preparation, enabling next-batch rollouts during deferred reflection training.The data thread performs asynchronous Gemini judging, launches reflection generation and judging, and prepares packed batches.Source: §D.1 · p. 35
  • The main thread alternates meta-policy updates on prior-step reflections with task-policy SS-GRPO updates, while reward scores return immediately and reflection scores continue asynchronously.This scheduling fully overlaps inference with reflection training during Phase A.Source: §D.1 · p. 35

D.2. One-Step Deferred Reflection Training … D.6. Prompt Templates

The implementation overlaps reflection processing with task-policy training, uses a windowed curriculum to make deferred self-guidance available, and persists rubric memories for retrieval. Prompt templates specify grounded reflection, judge evaluation, and distinct within-episode versus cross-episode guidance injection.

D.2. One-Step Deferred Reflection Training

  • One-step deferred training overlaps reflection generation, judging, and meta-policy updates with SS-GRPO task training, adding effectively no extra wall-clock overhead.Step N rollouts are updated immediately, while scored reflections train the meta-policy at step N+1 from a deferred buffer.Source: §D.2 · p. 36

D.3. Windowed Curriculum

  • The windowed curriculum separates new-query cross-episode retrieval from repeated-query within-episode retrieval across 2K steps, using K=3 to complete deferred reflection processing.The K-step gap ensures generation, judging, bank insertion, and deferred training finish before exact within-episode retrieval.Source: §D.3 · p. 36, Figure 7 · p. 37

D.4. Trajectory Sampling and Candidate Generation

  • For each new query, reflection generation uniformly samples one trajectory from the rollout group without length or score bias, matching inference conditions.The trajectory-sampling procedure is described as uniform over trajectory indices.Source: §D.4 · p. 36

D.5. Bank Persistence and Retrieval

  • The rubric bank uses CPU Qwen3-Embedding-0.6B embeddings with FAISS top-k similarity retrieval, exact SHA-256 question matching, overwrite updates, and checkpoint-consistent atomic persistence.The default cross-episode retrieval value is k=2; the bank is persisted every 10 steps and at each model checkpoint.Source: §D.5 · p. 37

D.6.1. Reflection Generation Prompt

  • Reflection prompts require grounded postmortems that judge correctness, coverage, research quality, synthesis, and communication, then produce reusable rubrics and concise transferable takeaways.The output contains exactly two blocks: reflection rubrics and reflection takeaways.Source: §D.6.1 · pp. 37–38

D.6.2. Judge Scoring Prompt

  • The judge scores reflection usefulness using diagnostic accuracy, specificity, and scope-and-balance criteria, rewarding grounded root-cause guidance rather than generic advice.The dimensions are weighted 0.4, 0.3, and 0.3, respectively.Source: §D.6.2 · pp. 38–39

D.6.3. Injection Formats

  • Retrieved guidance precedes the original question: exact prior-attempt reflections support self-improvement, while top-k similar-query items support adaptive analogical transfer.The two formats explicitly instruct agents to avoid past mistakes or adapt guidance to the current question’s needs.Source: §D.6.3 · pp. 39–40

E. Theoretical Analysis · E.1. Value of Stage Information

The theoretical analysis formalizes the value of exposing stage information under a compressed decision context. It shows that stage conditioning weakly improves decision value, with strict gains precisely when the context aliases stages with conflicting optimal actions.

E.1. Value of Stage Information

  • The formal setup assumes a measurable random decision history, finite action space, measurable compressed context and stage label, and integrable utilities.Finite actions and integrability ensure the relevant conditional values and expectations are well-defined; utilities may represent continuation values.Source: §E.1 · pp. 41–42
  • Under Assumption 1 and Definition 1, Lemma 1 establishes collapsing stage-conditioned utility to the flat information structure and is the only structural identity needed in proving Theorem 1.Source: §E.1 · p. 42
  • The analysis defines the aliasing gap as the difference between stage-conditioned and flat decision values at each context.The gap is measurable and nonnegative, providing the pointwise quantity used to characterize when stage information helps.Source: §E.1 · pp. 44–45
  • Theorem 2 shows that explicit stage information weakly increases decision value, with the gain equal to the expected stage-aliasing gap.The result compares information structures σ(C) and σ(C, Z) without assuming that compressed context and stage jointly summarize the full history.Source: §E.1 · pp. 41, 44, 47
  • Zero aliasing gap holds exactly when one action is optimal across every stage with positive conditional probability for almost every context.Thus, stage information has no value when all supported stages share a common maximizer.Source: §E.1 · pp. 46–47
  • The gain is strictly positive exactly when the compressed context aliases decision points whose stage-conditioned optimal actions disagree.This identifies explicit stage structure as beneficial when locally optimal actions vary across stages sharing the same context.Source: §E.1 · pp. 44, 47

E.2. Judge-Aligned Stage-Weighted Credit Assignment · 2. Lower bound for terminal broadcast relative to the oracle target: 𝑔term

This subsection formalizes full-rollout policy-gradient signals and compares terminal-broadcast credit with a stage-weighted oracle target. Under judge-alignment assumptions, stage-weighted credit improves gradient approximation when omitted intermediate signal exceeds cumulative judge misalignment.

E.2. Judge-Aligned Stage-Weighted Credit Assignment

  • Stage-weighted credit is analyzed over full rollouts, rather than a single random decision point, using stage-specific token sets and score-function sums.The analysis assumes differentiable policies, policy-independent environment dynamics, and square-integrable stage score-function sums.Source: §E.2 · p. 48
  • The framework distinguishes observed stagewise judge scores from latent true process scores and combines them through a causal stage-weight matrix with unit diagonal.The matrix excludes future-stage weights while allowing causal stagewise aggregation.Source: §E.2 · p. 48
  • The oracle contribution represents the stage-specific gradient induced by the full stage-dependent return, while judge-weighted and terminal-broadcast signals provide practical alternatives.The terminal-broadcast omission measures the oracle process-level signal lost by using only the final-stage score instead of the full stage-weighted target.Source: §E.2 · pp. 48–49
  • Judge alignment requires each observed stage score to approximate its latent true score in the gradient-relevant direction defined by the stage score-function sum.For every stage pair k≤j, the approximation is bounded by a nonnegative misalignment constant 𝜖_k,j.Source: §E.2 · p. 49
  • Stage-weighted credit yields a strictly better gradient approximation when omitted true intermediate signal outweighs cumulative judge misalignment.This is the informal statement of Theorem 3 comparing stage-weighted credit against terminal broadcasting.Source: §E.2 · p. 49

2. Lower bound for terminal broadcast relative to the oracle target: 𝑔term

  • The formal theorem establishes, for every stage, an error bound for the stage-weighted signal relative to the oracle target under the judge-alignment assumption.The supplied passage identifies this as the formal version of Theorem 3 and separately states the stage-weighted signal’s error relative to the oracle target.Source: §E.2 · p. 49

3. Comparison. If 𝐾 ∑︁ … F.2.1. Long-Form Benchmarks

The section establishes when stage-weighted credit more closely approximates process-level optimization and formalizes judge-gated co-evolution of task policy and rubric memory. It then specifies the training setup and evaluates four long-form benchmarks using rubric-based judging.

3. Comparison. If 𝐾 ∑︁

  • When intermediate-stage oracle signal is sufficiently strong relative to judge misalignment, stage-weighted credit is strictly closer to the intended process-level gradient than terminal broadcast.The benefit depends on the omitted intermediate-stage signal outweighing accumulated intermediate-judge errors.Source: §3 · pp. 49, 51

1. Query distribution for the task objective. There is a query random variable

  • The formal analysis models a fixed rubric bank, task rollouts, fixed reflection contexts, shared-backbone reflection generation, judged usefulness, acceptance gating, differentiability, and smooth objectives.These assumptions define the local setting for analyzing one parameter update rather than claiming every reflection is globally useful throughout training.Source: §E.3 · p. 51, §1 · p. 52

We also define the combined co-evolution objective

  • Under judge-gated positive transfer, task-improving and memory-improving updates mutually improve the shared task and memory objectives, while co-evolution can dominate task-only training with static memory.The condition requires accepted reflections to occur with nontrivial probability and their expected gradient contribution to align positively with the task gradient.Source: We also define the combined… · pp. 53, 54, 57

E.3. Judge-Gated Co-Evolution of Policy and Rubric Bank

  • The co-evolution mechanism relies on parameter sharing: task and reflection updates affect the same backbone, whereas fully decoupled parameter blocks eliminate the first-order mutual-improvement effect.The reflection update treats the sampled trajectory as fixed, so transfer arises through shared parameters rather than backpropagation through the trajectory.Source: §E.3 · p. 51, We also define the combined… · p. 57

F. Experiment Details

  • The experimental setup largely follows DR Tulu (Shao et al., 2025), adapting its training, evaluation, infrastructure, and baseline choices for the proposed components.The supplied setup passage provides this overall alignment but no comparative benchmark results.Source: F. Experiment Details · p. 57

F.1.1. Supervised Fine-Tuning

  • Supervised fine-tuning uses Qwen3-8B (Yang et al., 2025) with LLaMA-Factory (Zheng et al., 2024), DeepSpeed ZeRO-3, and tool-output span masking.Masking prevents gradients on search results, so the model learns to generate reasoning, tool calls, and answers.Source: §F.1.1 · pp. 57–58

F.1.2. Reinforcement Learning

  • The RL setup extends DR. Tulu’s codebase with stagewise credit assignment and meta-policy operations, uses 600-step two-node ablations, and trains the final RubricEM run longer on four nodes with rubric-only judge rewards.The training data contains ∼4.9K diverse deep research queries, and the design adds no auxiliary verifiable rewards.Source: §F.1.2 · pp. 58–59

F.2.1. Long-Form Benchmarks

  • Evaluation covers HealthBench, ResearchQA, DeepResearchBench, and ResearchRubrics, using benchmark-specific rubric, coverage, citation, or compliance judgments.The benchmarks span medical advice, scientific research, citation-grounded reports, and open-ended real-world research prompts.Source: §F.2.1 · p. 59

F.2.2. Short-Form Benchmarks … G. Algorithm

The appendix evaluates RubricEM’s zero-shot transfer, documents its distributed training and memory infrastructure, defines baseline categories and comparison caveats, and points to Algorithm 1 for the complete procedure.

F.2.2. Short-Form Benchmarks

  • RubricEM evaluates four short-form benchmarks zero-shot with the same agent pipeline and search tools as long-form tasks, using no short-form data during RL training.The benchmarks are SimpleQA, 2WikiMultihopQA, WebWalker, and DeepSearchQA, covering factual, multi-hop, navigation, and search-intensive questions.Source: §F.2.2 · p. 59
  • Citation-specific evaluation is left for future work because the method optimizes rubric-based rewards rather than precise academic citation, while remaining long-form benchmarks still assess citation and evidence use.SQAv2 is excluded as citation-centric, and citation quality is treated as one component of broader answer quality in other benchmarks.Source: §F.2.2 · p. 60
  • Main evaluations use direct zero-shot prompts without rubric-bank entries, reflections, or experience examples, isolating the task policy’s learned capabilities from inference-time context injection.Inference-time experience reuse is studied separately by explicitly injecting rubric-bank entries.Source: §F.2.2 · p. 60

F.3. Infrastructure

  • The infrastructure builds on DR Tulu’s distributed RL stack, adding GRPO with token-level loss aggregation, tool-output masking, and a 0.001 KL penalty for stability.The stack includes Ray, vLLM, asynchronous tool calling, sample packing, and one-step asynchronous training.Source: §F.3 · p. 60
  • RubricEM extends the stack with asynchronous stagewise judging, deferred reflection and meta-policy training, a FAISS-backed rubric bank, and checkpointed memory and dataloader state.Stagewise scoring parallelizes 32 rubric-generation and 256 trajectory-scoring calls per step, while deferred execution adds effectively no extra wall-clock overhead for meta-policy training.Source: §F.3 · pp. 60–61
  • The agent uses Google-grounded Gemini search for general web information and Semantic Scholar snippet search for scholarly evidence through a shared MCP backend.Evaluation serves checkpoints with vLLM on one GPU and executes tools through MCP.Source: §F.3 · p. 61

F.4. Baselines

  • Baselines span closed deep-research systems, fixed multi-stage pipelines, and open-weight end-to-end research models, with scores sourced from DR Tulu or rerun using the same pipeline.DR Tulu is the closest comparison because it shares training data and the open deep-research setting.Source: §F.4 · p. 61
  • The RubricEM–DR Tulu comparison differs in search tool and SFT teacher, so ablations isolate training-recipe effects from search-tool effects.RubricEM uses Google Search with Gemini grounding and Gemini-3.1-Pro, whereas DR Tulu uses Serper API and GPT-5.Source: §F.4 · p. 61

H. Limitations and Discussions

RubricEM frames LLM-generated rubrics as interfaces for structuring behavior, assigning semantic credit, and accumulating reusable experience during training. Its reported limitations include infrastructure instability, reliance on Gemini Flash judging, and risks from flawed rubrics and memories.

Limitations

  • Infrastructure instability occasionally caused API delays, inconsistent connections, server restarts, and additional staleness in asynchronous reflection, rubric, and judge-feedback pipelines.The authors characterize results as realistic but not fully ideal and suggest uninterrupted infrastructure could reduce stale feedback and improve efficiency.Source: H. Limitations and Discussions · p. 62
  • Using Gemini Flash for rubric generation, stagewise scoring, and reflection evaluation reduces cost but may limit stage-level credit and reflection quality on subtle long-form research tasks.Stronger, specialized, multiple, or calibrated ensemble judges could improve accuracy, but would increase cost and latency.Source: H. Limitations and Discussions · p. 62

Discussion and broader impact

  • Rubrics can structure agent behavior, assign semantic credit, and accumulate reusable experience, extending Meta-RL from inference-time prompting to training-time policy and memory updates.The authors suggest this recipe may transfer beyond deep research to writing assistance, data analysis, scientific review, tutoring, and complex tool-use workflows.Source: H. Limitations and Discussions · p. 62
  • Rubric-guided meta-policies may reinforce shallow preferences, biased standards, or overconfident synthesis when judges or criteria are poorly specified, while memories can propagate errors across tasks.Future directions include robust rubric generation, stronger or ensemble judges, auditable rubric banks, uncertainty-aware reflection, and safety-aware criteria.Source: H. Limitations and Discussions · p. 62

Related papers

Loading 2605.10899v1…