PaperlayerView exact revision on arXiv

Source-linked AI summary

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister

arXiv:2605.10899v1cs.CLcs.LG

TL;DR

Deep research agents are difficult to train when outputs lack ground-truth verification and feedback is coarse over long tool-augmented rollouts. RubricEM structures trajectories with stage-aware credit and reflection-based memory, and RubricEM-8B achieves strong performance across four benchmarks, leading comparable open models and approaching proprietary systems.

  • Problem

    Training deep research agents remains unclear because outputs lack ground-truth verification, feedback is coarse and delayed, and proprietary methods reveal little.

  • Method

    RubricEM uses rubrics to structure stages, assign process-level credit with SS-GRPO, and distill judged trajectories into reusable reflections via a shared-backbone meta-policy.

  • Results

    55.5 average score makes RubricEM-8B-RL the highest-scoring non-proprietary system, surpassing strong open baselines and approaching proprietary deep-research systems.

  • Takeaways & Limitations

    RubricEM provides a rubric-guided recipe for long-horizon reinforcement learning beyond verifiable rewards, with ablations supporting its proposed components.

  • Takeaways & Limitations

    SS-GRPO requires sufficiently aligned stagewise judging with bounded noise because intermediate judging can introduce judge noise.

Abstract

from arXiv · show

Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory. Based on this view, we introduce RubricEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution. RubricEM first makes research trajectories stage-aware by conditioning planning, evidence gathering, review, and synthesis on self-generated rubrics. It then assigns credit with Stage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback for long-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM.

1. Introduction

RubricEM addresses the challenge of training deep research agents beyond verifiable rewards by using rubrics as a shared interface for policy execution, judgment, and memory. It combines stagewise policy decomposition, Stage-Structured GRPO, and reflection-based meta-policy training, yielding an 8B agent with strong benchmark performance among comparable open models.

  • Motivation: Deep research agents must plan, search, evaluate evidence, and synthesize long-form reports, but training remains unclear beyond verifiable search proxies and imitation data.The central questions concern reinforcement learning beyond verifiable rewards, long-horizon credit assignment, and learning from experience.
  • Core perspective: RubricEM treats rubrics as a shared interface that guides planning and search, supports process-level judgment, and structures memory evolution.This extends rubrics beyond their prior use as judge-side criteria for final-response rewards.
  • Policy decomposition: The reasoning scaffold carries self-generated, task-specific rubrics through four stages: planning, research, review, and answer synthesis.Each stage defines a distinct decision mode and a natural unit for optimization, converting flat long-horizon rollouts into rubric-conditioned decision stages.
  • Stagewise optimization: Stage-Structured GRPO assigns denser credit by scoring Plan, Research, Review, and Answer with stage-specific rubrics instead of broadcasting one terminal score to all tokens.An evolving rubric buffer supports process-level feedback, and stagewise scores combine local stage quality with downstream effects.
  • Experience reuse: Reflection Meta-Policy training reuses judged experience by jointly training a shared-backbone reflection policy whose rubric-grounded candidates receive auxiliary rewards on reflection tokens.An asynchronous reflection branch trains alongside task-policy reinforcement learning without adding a sequential bottleneck.
  • Results: RubricEM-8B is an 8B deep research agent trained with 1400 RL steps that achieves state-of-the-art performance among comparable open models across four long-form research benchmarks.It improves over strong prior RL systems with fewer training steps and approaches proprietary systems such as Gemini and OpenAI Deep Research.

2. Related Work

Prior deep-research post-training largely targets short-answer tasks with verifiable rewards, while recent credit-assignment and Meta-RL methods generally remain grounded in verifiable or synthetic settings. These limitations motivate the paper’s focus on deep research beyond verifiable rewards.

  • The Post-training recipes of deep research agents: Most open-source deep-research training efforts focus on short-answer tasks with verifiable rewards.Proprietary systems mainly report scaling high-quality imitation data and training on verifiable short-form settings.
  • Credit assignment and meta-RL with language models: Recent agentic reinforcement-learning work increasingly emphasizes finer-grained credit assignment.
  • Credit assignment and meta-RL with language models: Most credit-assignment methods operate in verifiable settings with decomposable subgoals and reliable process-level supervision, while Meta-RL methods are typically evaluated on verifiable or synthetic tasks.The passage describes Meta-RL as training meta-policies during reinforcement learning.

3. RubricEM

RubricEM structures deep-research trajectories around rubric-guided stages and uses those stages to assign denser credit. It further turns judged experience into reusable natural-language guidance through a shared reflection meta-policy and rubric bank.

  • 3.1. Preliminaries and notations: Deep-research agents interact with tools over trajectories of textual segments or structured calls before producing evidence-grounded long-form answers.The policy autoregressively samples each action from the query, prior actions, and prior tool outputs.
  • 3.2. Structured Reasoning Scaffold: Explicit stage information resolves local-context aliasing when planning, searching, reviewing, and synthesis require different optimal actions, yielding strict value improvement where such aliasing occurs.A flat policy must infer its decision mode from local context, which can cause inefficient exploration and compounding errors over long horizons.
  • 3.2. Structured Reasoning Scaffold: RubricEM instantiates four sequential stages—Plan, Research, Review, and Answer—with local Research iteration and in-place plan revision.Plan generates rubrics and a research plan; Research evaluates evidence after each tool response; Review maps evidence to rubrics; Answer produces the cited response.
  • 3.2. Structured Reasoning Scaffold: Rubrics function as trajectory-wide control targets: they define information checklists, analytical criteria, and negative constraints, while guiding revision, review, and synthesis.The scaffold also defines SS-GRPO credit units and rubric-conditioned traces for the rubric bank.
  • 3.3. Stage-Structured GRPO: SS-GRPO partitions each rollout into K = 4 semantic stages and assigns stage-specific LLM-judge scores with causal downstream credit instead of broadcasting one final score to every token.Each stage retains its own score while receiving credit from downstream stages it enables; terminal reward broadcasting is a special case.
  • 3.3. Stage-Structured GRPO: Stage-weighted credit helps when recovered intermediate process information outweighs cumulative judge misalignment, so SS-GRPO requires aligned stagewise judging with bounded noise rather than oracle process rewards.Intermediate judging can improve credit assignment but also introduces judge noise.
  • 3.4. Meta-Policy Training with Reinforcement Learning: RubricEM reuses experience through a shared backbone serving as both task policy and reflection meta-policy, training reflections from judged rollouts with LLM-judge rewards.Accepted reflections become reusable natural-language guidance in an agent rubric bank, complementing parametric RL updates with textual memory updates.

4. Experiment

RubricEM is evaluated on four long-form research benchmarks using a DR Tulu-based experimental setup. RubricEM-8B-RL achieves the highest average score among evaluated non-proprietary systems, while reinforcement learning improves performance across all four benchmarks and the full recipe performs best in ablations.

  • Experimental setup: RubricEM is evaluated on HealthBench, ResearchQA, DeepResearchBench, and ResearchRubrics using a DR Tulu-based setup with Gemini-flash-grounded Google Search and Semantic Scholar.The setup generally shares DR Tulu’s infrastructure, training datasets, hyperparameters, and evaluation protocols.
  • Main results: 55.5 is RubricEM-8B-RL’s highest average score among evaluated non-proprietary deep research systems, surpassing DR Tulu-8B-RL, Tongyi DeepResearch-30B-A3B, and WebThinker-32B-DPO.On benchmarks with available scores, RubricEM-8B-RL also outperforms Perplexity Deep Research on average.
  • RL effectiveness: 49.2 to 55.5: reinforcement learning raises the average score from the structured SFT checkpoint and improves results on all four long-form benchmarks.RubricEM-8B-RL also surpasses its Gemini-3.1-Pro-distilled RubricEM-8B-SFT model on average.
  • Ablation study: Under a 600-step budget, each proposed RL-training component improves performance, and the full RubricEM recipe performs best.The ablation compares the proposed components within the RL training recipe.

5. Empirical Analysis

Empirical analyses show that RubricEM’s stagewise credit assignment, reusable experience, and rubric-guided scaffold contribute complementary improvements under matched training settings. The method also transfers effectively from long-form training to short-form search benchmarks and supports experience reuse across episodes.

  • RL component ablations: Under matched 600-step settings, SS-GRPO and Meta-Policy both outperform Baseline-RL, while RubricEM (Full) performs best across benchmarks.The comparison uses the same RubricEM-SFT initialization, training configuration, and fixed evaluation subsets.
  • RL component ablations: Stagewise credit assignment and reusable-experience learning provide complementary gains under the same training budget and compute.SS-GRPO replaces terminal reward broadcast with stagewise rubric credit, whereas Meta-Policy adds reflection training and rubric-bank retrieval.
  • Structured scaffold: The rubric-guided scaffold improves SFT distillation quality and makes subsequent RL more effective under matched 600-step settings.Without the scaffold, RL gains are small and unstable, indicating that rubric-conditioned stages structure exploration and credit assignment.
  • Inference-time experience reuse: With the same Gemini-3.1-Pro model and search backend, the scaffold outperforms a standard ReAct prompt on DRB.The analysis isolates this effect at the prompt level.
  • Transfer and experience reuse: RubricEM transfers strongly to out-of-distribution short-form search benchmarks, with long-form RL improving search-and-answer performance despite no short-form RL data.The gains are most pronounced on more complex tasks, and the learned meta-policy supports cross-episode transfer and within-episode refinement.

6. Conclusions … B.1. Scaffold Description and System Prompt

RubricEM is a rubric-guided reinforcement-learning framework that structures long-horizon deep-research trajectories, assigns stage-aware credit, and distills judged attempts into reusable guidance. Its scaffold operationalizes this approach through four ordered stages—planning, research, review, and answer—while related work positions it as a more training-efficient extension beyond verifiable rewards.

  • 6. Conclusions: RubricEM is presented as a rubric-guided RL framework for deep research beyond verifiable rewards.The conclusion identifies rubric-guided reinforcement learning as the paper’s central contribution.
  • 6. Conclusions: RubricEM structures long-horizon trajectories, assigns stage-aware credit, and distills judged attempts into reusable guidance through a shared-backbone reflection meta-policy.The framework’s analyses and ablations support its proposed recipe, with RubricEM-8B showing strong benchmark performance.
  • A. Additional Related Works: Prior deep-research work emphasizes real-web or scalable search, workflow and report-state design, and agentic SFT/RL or RLAIF built largely around verifiable or ground-truth-anchored supervision.The related-work discussion contrasts these directions with RubricEM’s beyond-verifiable-reward setting.
  • A. Additional Related Works: Compared with DR Tulu’s extended-RL foundation, RubricEM targets training efficiency through rubric-guided stages, denser stage-level credit, and reflection-based policy evolution.The paper states that this structure–assign–evolve recipe yields larger gains with fewer RL training steps.
  • B.1. Scaffold Description and System Prompt: The structured scaffold decomposes each trajectory into four semantically distinct, XML-tagged stages governed by stage-specific behavioral requirements.The four stages are Plan, Research, Review, and Answer.
  • B.1.1. Stage 1: Planning (Plan): Planning creates prospective rubrics before search, with effort scaling from brief criteria and linear plans for simple queries to detailed conditional strategies for complex ones.The planning stage includes exploratory thinking, structured analysis, rubric construction, and a research roadmap before the first tool call.
  • B.1.2. Stage 2: Research (Research): Research iteratively evaluates accumulated evidence against the rubric and plan, either continuing tool use, revising the plan after invalidated assumptions, or proceeding when evidence is sufficient.The outer workflow remains sequential, while the Research stage supports repeated tool-use and state-evaluation cycles.
  • B.1.3. Stage 3: Review (Review): Before answering, mandatory review maps evidence to rubric criteria, verifies satisfied checklist items, restates synthesis constraints, and produces a writing plan for the final cited response.The answer stage must follow this writing plan, satisfy verified rubric criteria, and ground nontrivial claims with inline citations.

B.2. SFT Data Generation Process … C. Details on Stagewise Evolving Rubric Evaluation

RubricEM generates stage-structured SFT trajectories through a teacher-driven, multi-round tool pipeline, then filters and reformats them into approximately 11k training samples. Its evolving rubric evaluator maintains adaptive stage-local criteria, scores trajectories against combined rubrics, and emphasizes discriminative process quality across research stages.

  • B.2. SFT Data Generation Process: The SFT data process uses teacher-generated, tool-augmented trajectories to instill RubricEM’s rubric-guided scaffold into Qwen3-8B.The pipeline produces complete stage-structured trajectories rather than isolated responses.
  • B.2.1. Teacher Model and Prompt Adaptation: Gemini-3.1-Pro generates trajectories with prompts separated between first-round planning and later-round evidence evaluation and synthesis.The separation prevents premature answering and focuses each API call on its current stage.
  • B.2.2. Multi-Round Generation Pipeline: The multi-round pipeline alternates teacher generation with real tool execution, routing google_search and snippet_search calls before continuing or answering.The loop stops at </answer> or after a maximum of 10 rounds.
  • B.2.2. Multi-Round Generation Pipeline: ∼15–25% of trajectories are discarded because Gemini often fails to produce a well-formed closing </answer> tag.Incomplete trajectories that reach the round limit are also discarded.
  • B.2.3. Quality Filtering and Rejection Sampling: Rejection sampling removes trajectories missing required answer or tool-call structure, malformed stage elements, or repeated tool errors.Missing </answer> is identified as the most common failure, while absent valid tool calls are especially common on simple factual queries.
  • B.2.3. Quality Filtering and Rejection Sampling: Accepted trajectories convert <scratchpad> to <think>, normalize tool names, and preserve only model-generated tokens for gradient training while masking tool outputs.The resulting ChatML conversation contains system, user, and assistant messages.
  • C. Details on Stagewise Evolving Rubric Evaluation: The evolving rubric evaluator generates adaptive rubrics by comparing same-question trajectories, scores each trajectory against persistent and active rubrics, and targets discriminative stage-local criteria.Stage guidance covers understanding the problem, effective adaptive search, readiness auditing, and final-answer quality; scoring uses weighted stage averages with positive and negative rubric handling.

C.4. Buffer Management and Implementation · D. Asynchronous Reflection Pipeline and Windowed Curriculum · D.1. Training Pipeline Architecture

The implementation manages evolving stagewise rubric buffers and uses structured, parallelized judging for efficient training. The training pipeline overlaps reflection generation with rollout inference and policy updates, while the appendix documents the associated retrieval curriculum.

  • C.4. Buffer Management and Implementation: When questions recur, adaptive rubrics from the improved policy can replace stale criteria, allowing judge standards to co-evolve with the policy.The approach extends evolving rubrics from answer-only evaluation to stagewise evaluation.
  • C.4. Buffer Management and Implementation: Rubric generation and scoring use Gemini structured JSON outputs with Pydantic-derived schemas, avoiding fragile post-hoc text parsing.Schemas organize rubrics by stage with titles, descriptions, and weights, and scores with indices and justifications.
  • C.4. Buffer Management and Implementation: Each training step processes 32 unique questions with 8 rollouts each, launching 32 rubric-generation calls and 256 scoring calls concurrently with retry and fallback handling.Calls use exponential-backoff retries up to 5 times; failures produce no new rubrics or terminal-answer-only scores.
  • C.4. Buffer Management and Implementation: Rubric generation and scoring take approximately 5 minutes per step and overlap with gradient computation, using Gemini-3-Flash for cost and latency.Generation requires 32 calls per step, while scoring requires 256 lightweight calls covering approximately 8–15 rubrics per trajectory.
  • D. Asynchronous Reflection Pipeline and Windowed Curriculum: The appendix documents the meta-policy training pipeline and windowed retrieval curriculum, whose loop combines rollout generation, stagewise judge scoring, policy updates, and overlapped reflection generation.Rollouts use multi-turn tool-augmented trajectories via vLLM, while stagewise scoring computes SS-GRPO rewards.
  • D.1. Training Pipeline Architecture: The architecture uses concurrent training and inference activity: the main thread alternates deferred-reflection meta-policy updates with current-rollout SS-GRPO updates while vLLM generates the next batch.During Phase A, next-batch inference proceeds while the training engine processes deferred reflections.

D.2. One-Step Deferred Reflection Training … D.6. Prompt Templates

RubricEM defers reflection training by one RL step and uses a windowed curriculum to ensure reflections are available before repeated-query retrieval. Its rubric bank supports cross-episode transfer and within-episode refinement, while prompt templates govern reflection generation, judging, and guidance injection.

  • D.2. One-Step Deferred Reflection Training: Reflection training is deferred by one RL step, allowing rollout generation, SS-GRPO scoring, reflection generation, and candidate judging to overlap asynchronously.The task-policy update proceeds immediately while reflection scoring continues in the background.
  • D.2. One-Step Deferred Reflection Training: This one-step staleness keeps inference and training continuously occupied, adding effectively no extra wall-clock overhead to the SS-GRPO loop.
  • D.3. Windowed Curriculum: The windowed curriculum separates new-query and repeat phases across 2K steps, preventing within-episode retrieval from preceding reflection generation and bank insertion.The K-step gap ensures the deferred pipeline completes before a query is replayed; experiments use K=3.
  • D.4. Trajectory Sampling and Candidate Generation: For each new query, one rollout trajectory is sampled uniformly at random for reflection generation, without length bias or score-based selection.
  • D.5. Bank Persistence and Retrieval: The rubric bank uses CPU Qwen3-Embedding-0.6B embeddings with an in-memory FAISS index for top-k cross-episode retrieval, while within-episode retrieval matches exact SHA-256 question hashes.The default retrieval count is k=2, and later reflections overwrite existing items for the same query.
  • D.5. Bank Persistence and Retrieval: The bank is persisted every 10 steps and at model checkpoints with atomic writes, then restored from the checkpoint matching the model’s global step.
  • D.6. Prompt Templates: Reflection prompts instruct the policy to judge final answers across correctness, coverage, research quality, synthesis, and communication, then produce grounded, transferable rubrics and takeaways.The required output contains exactly two blocks: reflection rubrics and 1-4 concise portable lessons.

E. Theoretical Analysis · E.1. Value of Stage Information

The analysis formalizes the value of exposing stage information under a compressed decision context. It shows that stage labels help exactly when the same context aliases stages with different optimal actions.

  • E.1. Value of Stage Information: The result compares decisions using compressed context C alone with decisions using the augmented information structure (C, Z), without requiring (C, Z) to summarize the full history H.The theorem isolates the gain from making the stage label explicit under a compressed state representation.
  • E.1. Value of Stage Information: The formal setup assumes a measurable history space, a finite nonempty action set, a standard-Borel compressed context, a finite stage-label set, and integrable utilities.These assumptions ensure the conditional objects, maximizations, and expectations used in the analysis are well-defined.
  • E.1. Value of Stage Information: For each action, the stage-conditioned utilities collapse to the corresponding flat conditional utility after averaging over stage probabilities given the context.This identity is the only structural lemma needed to relate stage-conditioned and flat decision values.
  • E.1. Value of Stage Information: The stage-aliasing gap is nonnegative and equals the improvement obtained by optimizing actions separately by stage rather than selecting one action from the compressed context alone.The proof derives the gap as a subtraction between stage-conditioned and flat values and establishes nonnegativity by comparing each stagewise maximum with every common action.
  • E.1. Value of Stage Information: The gap is zero for almost every context exactly when all stages with positive conditional probability share a common maximizing action.Stages outside the context’s support contribute nothing, so the characterization applies over the supported stage set.
  • E.1. Value of Stage Information: A strict positive gain occurs when a measurable set of contexts has supported stages whose stage-conditioned optimal actions do not share a common maximizer.The strictness condition compares stages with positive conditional probability, avoiding conclusions based on impossible stage events.
  • E.1. Value of Stage Information: The analysis interprets explicit stage structure as beneficial precisely when compressed contexts alias decision points with disagreeing stage-conditioned optimal actions.This is presented as the precise mechanism by which stage information improves reasoning under compressed local context.

E.2. Judge-Aligned Stage-Weighted Credit Assignment · 2. Lower bound for terminal broadcast relative to the oracle target: 𝑔term

This subsection analyzes full rollouts and formalizes how stage-weighted judge feedback approximates an oracle process-level policy-gradient signal. It shows that terminal-only broadcasting can omit intermediate signal, while stage-weighted credit improves approximation when omitted signal exceeds judge misalignment.

  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: The analysis extends the single-decision framework to full rollouts and their policy-gradient signals.It studies a generic rollout sampled from p_θ(τ), with stage-specific token sets and score-function sums.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: Stagewise judge scores are compared with latent true process scores under integrability and differentiability assumptions.The policy is differentiable, environment dynamics are parameter-independent, and the stage score-function sums are square-integrable.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: The causal stage-weight matrix enforces λ_k,j = 0 for j < k and λ_k,k = 1.Thus, stage k receives no weight from later-inapplicable indices and retains its own score with unit weight.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: The oracle process-level gradient contribution is defined using the stage-dependent return G_Λ, while the judge-induced and terminal-broadcast signals provide alternative estimators.The terminal-broadcast signal uses only the final-stage score Y_K rather than the full stage-weighted oracle target.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: Judge alignment requires observed score R_j to approximate latent score Y_j in the gradient-relevant direction defined by Γ_k.For each stage pair k ≤ j, the assumption bounds this directional misalignment by a nonnegative constant ε_k,j.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: Stage-weighted credit yields a strictly better gradient approximation when omitted true intermediate signal exceeds cumulative judge misalignment.This is the informal benefit stated by Theorem 3.
  • E.2. Judge-Aligned Stage-Weighted Credit Assignment: The formal theorem establishes bounds for every stage, including the error of the stage-weighted signal relative to the oracle target.Theorem 4 is presented as the formal version of Theorem 3, with its first result addressing stage-weighted approximation error.

3. Comparison. If 𝐾 ∑︁ … F.2.1. Long-Form Benchmarks

The paper establishes when stage-weighted credit better approximates process-level optimization, formalizes judge-gated shared-backbone co-evolution, and specifies training and evaluation procedures for RubricEM. Experiments use rubric-based rewards, fixed RL budgets for ablations, and four long-form benchmarks evaluated with rubric-oriented judges.

  • 3. Comparison. If 𝐾 ∑︁: Stage-weighted credit is strictly closer to the intended process-level gradient than terminal broadcast when judge misalignment is sufficiently small relative to omitted intermediate-stage signal.The advantage depends on intermediate-stage signal strength versus judge score errors; sufficiently large process signal outweighs imperfect-judge noise.
  • 1. Query distribution for the task objective. There is a query random variable: The co-evolution mechanism relies on a shared backbone, with task and reflection objectives updating the same parameters while sampled trajectories remain fixed during reflection updates.In a fully decoupled parameterization, the first-order mutual-improvement phenomenon disappears.
  • E.3. Judge-Gated Co-Evolution of Policy and Rubric Bank: Accepted reflections can mutually improve task performance and memory utility under a judge-gated local positive-transfer condition.The condition requires nontrivial acceptance probability and positive alignment between accepted reflections’ expected gradient contribution and the task gradient.
  • 1. Query distribution for the task objective. There is a query random variable: The formal analysis treats the rubric bank and reflection-context distribution as fixed locally and defines task, reflection-utility, transfer, and judge-acceptance quantities.The acceptance indicator marks reflections judged sufficiently useful for downstream adaptation or storage.
  • We also define the combined co-evolution objective: Theorem 5 shows that joint training yields a strictly larger first-order gain in adapted task value than task-only training with static memory when accepted reflections are sufficiently good.This identifies judge-gated accepted-reflection quality as the source of the co-evolution gain.
  • F.1.1. Supervised Fine-Tuning: RubricEM fine-tunes Qwen3-8B with LLaMA-Factory and DeepSpeed ZeRO-3, using span masking so gradients train reasoning, tool calls, and answers rather than search-result tokens.The supervised setup uses 5 epochs, learning rate 4 × 10−5, effective batch size 128, and maximum sequence length 16,384 tokens.
  • F.1.2. Reinforcement Learning: RL uses GRPO or SS-GRPO with rubric-judge rewards only, while ablations share a 600-step budget, two-node setup, and common RubricEM-SFT checkpoint.The RL configuration includes 8 rollouts per prompt, 32 unique prompts per step, and a final RubricEM run using four nodes for longer runs.
  • F.2.1. Long-Form Benchmarks: Evaluation covers four long-form benchmarks, including HealthBench’s 1000 medical questions and ResearchQA’s 756 scientific research questions, with rubric-based LLM judging.HealthBench reports overall rubric satisfaction across accuracy, completeness, context awareness, and communication.

F.2.2. Short-Form Benchmarks … G. Algorithm

The paper tests zero-shot transfer on four short-form benchmarks, using the same agent pipeline and tools without short-form RL data, and details the infrastructure, baselines, and Algorithm 1. Its evaluation excludes SQAv2 because the benchmark emphasizes citation precision rather than open-ended answer quality.

  • F.2.2. Short-Form Benchmarks: Four short-form benchmarks test out-of-domain transfer: SimpleQA, 2WikiMultihopQA, WebWalker, and DeepSearchQA.They contain factual, multi-hop reasoning, web navigation, and search-intensive questions, respectively, with exact-match or LLM-as-judge evaluation.
  • F.2.2. Short-Form Benchmarks: All short-form evaluations are zero-shot, use the long-form pipeline and search tools, and exclude short-form data from RL training.The evaluations are therefore fully out-of-domain.
  • F.2.2. Short-Form Benchmarks: SQAv2 is excluded because it primarily measures precise inline citations to academic papers rather than open-ended answer quality.The RL objective intentionally uses only rubric-based judge rewards, according to the benchmark note.
  • F.2.2. Short-Form Benchmarks: Test-time results use direct zero-shot prompts without rubric-bank entries, previous reflections, or other experience examples.The agent still follows benchmark protocols and uses external search tools when required.
  • F.3. Infrastructure: The infrastructure builds on DR Tulu’s Ray-based RL loop, vLLM, asynchronous tool calling, sample packing, and one-step asynchronous training.Training uses GRPO with token-level loss aggregation, tool-output masking, and a 0.001 KL penalty.
  • F.3. Infrastructure: Checkpointing saves model, optimizer, rubric-buffer, rubric-bank, and dataloader states to support consistent training resumption.Evaluation serves checkpoints with vLLM on a single GPU and executes tools through the MCP backend.
  • F.4. Baselines: Baselines span closed deep-research systems, fixed multi-stage pipelines, and additional recent models, with scores adopted from Dr. Tulu where available.DR Tulu is the closest comparison because it uses the same training data and open deep-research setting, despite differing tools and SFT teachers.

H. Limitations and Discussions

RubricEM’s training is sensitive to infrastructure instability, while its broader framework treats LLM-generated rubrics as interfaces for structuring behavior, assigning semantic credit, and accumulating reusable experience. The approach may transfer beyond deep research, but inherits risks from poorly specified rubrics and judges, including biased standards and propagated errors.

  • Limitations: Training with search-tool calls and external LLM judging is more sensitive to infrastructure instability than standard offline RL or supervised fine-tuning.API delays and inconsistent network connections made wall-clock execution and some rollout–judge latencies imperfectly controlled across RL steps.
  • Discussion and broader impact: LLM-generated rubrics can structure agent behavior, assign semantic credit, and accumulate reusable experience, supporting a training-time view of Meta-RL for language agents.The discussion frames rubrics as more than evaluation artifacts and positions reflection, memory, and self-improvement within training.
  • Discussion and broader impact: The rubric-guided recipe may transfer to writing assistance, data analysis, scientific review, tutoring, and complex tool-use workflows with multidimensional quality criteria.These domains share evaluation challenges that are hard to reduce to exact-answer correctness.
  • Limitations: Poorly specified judges and rubrics can reinforce shallow preferences, biased standards, or overconfident synthesis, while reusable memories may propagate errors across tasks.Future work should study more robust rubric-guided meta-policies.
Loading 2605.10899v1…