PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation

Changze Lv, Jie Zhou, Wentao Zhao, Jingwen Xu, Shihan Dou, Zisu Huang, Muzhao Tian, Xiaohua Wang, Yang Liu, Pluto Zhou, Tao Gui, Le Tian, Xiao Zhou, Xiaoqing Zheng, Xuanjing Huang, Jie Zhou

arXiv:2602.03619v2cs.CL

TL;DR

Reliable long-form DeepResearch generation lacks scalable, verifiable reward signals, while existing rubrics are either too coarse or costly to construct for each query. This paper learns query-specific rubric generators from paired human preferences using reinforcement learning and hybrid rewards. The learned rubrics better discriminate preferred reports and improve DeepResearch training under both ReAct and multi-agent workflows.

  • Problem

    Long-form DeepResearch generation lacks scalable, verifiable reward signals, and existing rubrics are either insufficiently granular or costly to construct manually for each query.

  • Method

    The paper constructs over 5,000 paired-report preference examples and trains query-specific rubric generators with GRPO using preference, format, and LLM-based quality rewards.

  • Results

    The learned rubrics outperform predefined and LLM-generated alternatives for preference discrimination and improve downstream DeepResearch training in single- and multi-agent workflows.

  • Takeaways & Limitations

    Human preferences can support reusable, scalable query-specific supervision when explicit golden signals are unavailable for complex report generation.

  • Takeaways & Limitations

    The preference formulation relies only on pairwise comparisons, which may not fully capture the complexity of human judgments; evaluations of novelty, creativity, factuality, and reasoning depth remain partly subjective.

Abstract

from arXiv · show

Nowadays, developing reliable DeepResearch-style long-form report generation remains challenging, as training and evaluation lack verifiable reward signals. Accordingly, rubric-based evaluation has become a common practice. However, existing approaches either rely on coarse, pre-defined rubrics that lack sufficient granularity or depend on manually constructed query-specific rubrics that are costly and difficult to scale. In this paper, we propose a pipeline to train preference-grounded query-specific rubric generators tailored for DeepResearch report generation. We first construct a dataset of DeepResearch-style queries annotated with human preferences over paired reports, and train rubric generators via reinforcement learning with a hybrid reward combining preference consistency, format validity, and LLM-based rubric evaluation. We evaluate the resulting rubric generators in two stages. First, on a held-out human-preference test set, the learned rubrics discriminate preferred from rejected reports more effectively than generic, prompted, or SFT-trained rubric alternatives. Second, when used as reward signals to train DeepResearch systems, our rubric generators yield substantial performance gains under both a simple single-agent ReAct framework and a complex multi-agent workflow on the DeepResearch Bench.

1 Introduction

DeepResearch report generation lacks scalable, verifiable reward signals, motivating preference-grounded query-specific rubrics. The proposed pipeline learns such rubrics and shows stronger preference discrimination and downstream performance across single- and multi-agent training workflows.

  • DeepResearch report generation requires multi-source reasoning, retrieval, integration, and coherent organization for complex open-ended queries.
  • Existing generic rubrics lack granularity, while manually authored query-specific rubrics require substantial expertise and effort to scale.
  • Pairwise human preferences offer scalable supervision, but the paper uses them to learn reusable rubric generators that provide finer-grained query-specific rewards.
  • The pipeline trains rubric generators on over 5,000 paired-report queries with GRPO and a hybrid reward combining preference consistency, format compliance, and LLM-based rubric quality.
  • Learned rubrics discriminate preferred reports more effectively than predefined or LLM-generated alternatives and improve downstream training signals for DeepResearch agents.
  • The contributions include a preference dataset, reinforcement-learned query-specific rubrics, and improvements in held-out discrimination and both single- and multi-agent workflows.

2 Related Work

Related work distinguishes verifiable short-form question answering from difficult long-form report generation and surveys fixed, query-specific, and rubric-reward approaches. Existing systems also include collaborative and open-source workflows for long-form research.

  • Short-Form Question Answering: Short-form DeepResearch tasks use verifiable targets that enable reinforcement learning with verifiable rewards.
  • Long-Form Report Generation: Long-form report generation requires synthesizing heterogeneous evidence, reconciling conflicts, reasoning across steps, and organizing coherent responses without reference answers.
  • Long-Form Report Generation: ResearchRubrics applies LLM-as-a-Judge to human-annotated general or query-specific rubrics for report evaluation.
  • Rubric-Based Evaluation: Prior work studies fixed and query-specific rubrics, with newer methods incorporating checklist-like rubrics as rewards in reinforcement-learning frameworks.

3 Method

The method constructs diverse DeepResearch queries and challenging candidate-report preference pairs, then trains a query-specific rubric generator with hybrid rewards. The generated rubrics provide query-specific reward signals for downstream DeepResearch training.

  • 3.2 Creation of the Preference Dataset Stage 1: Query Construction: The dataset begins with diverse, open-ended research queries requiring multi-step reasoning, evidence synthesis, and structured long-form reporting.Queries are generated from knowledge-graph entities and relations, then rewritten to diversify phrasing and naturalness across realistic information needs.
  • 3.2 Creation of the Preference Dataset Stage 2: Candidate Report Generation: Candidate reports are filtered for factual errors, citation problems, disorganization, and superficial aggregation before retaining challenging high-quality pairs for annotation.The procedure avoids trivially separable pairs so preference annotations provide more useful supervision for learning.
  • 3.3 Training Rubric Generators with Hybrid Rewards: The rubric generator samples structured weighted criteria with GRPO, combining preference consistency, format validity, and LLM-as-a-Judge quality rewards.Each rubric item contains a title, description, and importance weight; the hybrid reward is used to guide rubric generation.
  • 3.3 Training Rubric Generators with Hybrid Rewards: Preference consistency evaluates whether a generated rubric ranks the human-preferred report above the rejected report using weighted item-level ratings.Judge-assigned conformity scores are aggregated with rubric-item weights, and the preference reward is +1 for the correct ranking and −1 otherwise.
  • 3.4 Rubric-Based GRPO under the Multi-Agent Markov-State Workflow: The trained rubric generator supplies query-specific rewards for downstream DeepResearch systems, including the MaMs workflow with shared-policy search, state updates, and report generation.MaMs provides a controlled long-context workflow and an interface for assigning rubric-based rewards to generated reports.
  • 3.4 Rubric-Based GRPO under the Multi-Agent Markov-State Workflow: Table 1 evaluates preference accuracy/AUC and paired Cohen’s d on the human-preference test set, with human-defined general rubrics as a non-model baseline.The caption reports the baseline as Acc. = 48.78 and d = 0.192; the cited passage does not provide the remaining table rows.

4 Experiments

The experiments test whether learned rubrics capture human preferences, provide stronger rewards for DeepResearch training, and improve workflow-level performance. Results favor reinforcement-learned, query-specific rubrics and the MaMs workflow across preference discrimination, benchmark evaluation, and tool interaction.

  • Research questions: The experiments evaluate preference capture, reward informativeness, and the workflow-level gain of MaMs over ReAct.RQ1 concerns human-preference discrimination, RQ2 compares rubric-based training signals, and RQ3 compares MaMs with ReAct under shared rewards.
  • Experimental settings: The dataset uses topic-balanced query-level 8:1:1 train, validation, and test splits, with DeepResearch Bench providing 50 Chinese and 50 English queries.Preference accuracy and paired Cohen’s d are reported on D; DeepResearch Bench is evaluated on comprehensiveness, depth, instruction following, and readability.
  • Human-preference evaluation: Query-specific rubrics substantially outperform human-defined general rubrics on held-out preference accuracy and paired Cohen’s d.General rubrics achieve near-random preference accuracy and small effect sizes, whereas generated query-conditioned rubrics improve both measures.
  • Human-preference evaluation: GPT-5-generated and SFT-trained rubrics improve preference accuracy but retain relatively small paired Cohen’s d, indicating limited separation of accepted and rejected reports.These methods do not sufficiently capture fine-grained human-preference margins.
  • Human-preference evaluation: Preference-based reinforcement learning markedly increases paired Cohen’s d across both Qwen backbones, while hybrid rewards achieve the best preference accuracy.Preference-only RL remains close in accuracy and reaches the largest effect size for Qwen3-30B-A3B; format and LLM-quality rewards act as auxiliary constraints.
  • DeepResearch Bench: The RL-trained rubric generator achieves the best performance across all evaluation dimensions on DeepResearch Bench under the Tongyi-DeepResearch backbone.It outperforms human-defined general rubrics, GPT-5-generated rubrics, and the SFT-trained generator.
  • DeepResearch Bench: MaMs improves over corresponding ReAct runs under the same learned rubric rewards, and the MaMs system with RL-trained rubrics is strongest among reported open-source methods.The authors attribute this implementation-level gain to explicit state variables and chunk-level state updates, not to a fundamentally new agent architecture.

5 Conclusion

The paper addresses the lack of scalable preference-informed supervision for DeepResearch reports by learning query-specific rubric generators from human preferences. Combined with GRPO, format constraints, and LLM-based rubric evaluation, the method produces discriminative and adaptable signals in the reported experiments.

  • Conclusion: The paper learns query-specific rubric generators from human preferences instead of relying on predefined or human-annotated rubrics.The approach targets scalable preference-informed supervision without explicit golden signals.
  • Conclusion: GRPO combines human-preference information with format constraints and LLM-based rubric evaluation to produce discriminative and adaptable rubrics.The conclusion presents learning evaluative criteria as a direction for preference-aligned training in complex tasks.

Limitations

The approach remains limited by coarse pairwise preference signals and difficult-to-assess qualities such as novelty, creativity, factuality, and reasoning depth.

  • Pairwise comparisons may not fully capture the complexity of human judgments because the preference formulation uses only two-way comparisons.Rankings or graded scores could provide finer-grained preference supervision.
  • Evaluating novelty, creativity, factuality, and reasoning depth remains challenging because current assessments are partially subjective.The paper suggests combining more sophisticated LLM assessments with targeted human feedback to improve reliability and consistency.

A Case Study on Query Rewriting

The case study expands a short question into a structured DeepResearch request, while the described workflow iteratively searches, processes evidence chunks, updates state, and refines the report. The rubric prompt specifies query-derived, self-contained criteria for evaluating the resulting report.

  • A Case Study on Query Rewriting: The original Cheshire Cat question is rewritten into a study request covering the likely real-world breed, coat colors, and associated eye colors.The rewritten query asks for identification and a summary organized by coat color and typical eye color.
  • State Abstraction and Iterative Transitions: The MaMs workflow represents each research state with structured memory, an execution plan, and an incrementally evolving report.Transitions are modeled as st+1 = T(st, at), with tool execution and subsequent multi-agent processing.
  • Agent Modules and Chunk-based Process: A search agent selects actions and refines the plan, while a state agent processes long observations as semantically coherent chunks.Chunk-based processing addresses observations that exceed the LLM context window.
  • Agent Modules and Chunk-based Process: The state agent incrementally preserves existing memory while compressing and merging facts from each chunk, then updates the global state.After all chunks are processed, mt+1 = mt,K and pt+1 = pt,K.
  • Agent Modules and Chunk-based Process: The report agent drafts, corrects, and expands the report alongside state updates, separating information compression from narrative generation.The final report is produced when the maximum turns are reached or no further information acquisition is required.
  • Rubric Prompt: The rubric prompt requires query-specific, self-contained criteria covering relevance, organization, completeness, reasoning, clarity, and style.It also specifies 7–20 rubric items, category prefixes, and weights ranging from -2 to 5.

D Implementation Details

Implementation combines a post-training and inference stack with explicit preference-annotation procedures and comparisons against multiple rubric and agent baselines. The annotation protocol uses anonymized paired reports, calibrated annotators, majority voting, and quality controls.

  • Training and Evaluation Setup: Training uses slime2 with Megatron as the backend and SGlang as the inference backend, with an optimizer configuration update for reinforcement learning of MoE models.Hyperparameters are reported separately for rubric-generator training and DeepResearch-agent training.
  • Baselines: Human-preference evaluation compares human-defined, pointwise, pairwise, generated-rubric, SFT, and GRPO reward configurations.These baselines cover manual rubrics, direct preference judgments, prompted rubric generation, supervised targets, and reinforcement-learning rewards.
  • Baselines: The DeepResearch Bench comparison includes ReAct and MaMs workflows, while one DRTulu result is reported as a reference rather than a strictly controlled same-backbone comparison.The stated reason is infrastructure constraints preventing reproduction with the same backbone and external search stack.
  • Preference Annotation: Annotators read one query and two anonymized reports, then choose the report that better satisfies the query’s information need without assigning absolute scores or writing rubrics.Report labels and source metadata are randomized or removed to reduce ordering and system-identity bias.
  • Preference Annotation: The protocol calibrates 16 annotators and aggregates at least three independent judgments by majority vote, discarding pairs that remain unresolved after a fourth annotation.Calibration requires at least 80% agreement with author-produced gold labels.
  • Quality Control: Annotation quality is monitored with seeded gold pairs, inter-annotator agreement checks, and manual audits, while instructions explicitly control length, formatting, and citation-count biases.Annotators evaluate alignment, usefulness, completeness, and coherence without fixed weights.

G Metrics for Human Preference

Human-preference modeling is evaluated with ranking correctness and standardized score separation. Preference accuracy corresponds to pairwise AUC, while paired Cohen’s d measures the magnitude and stability of separation.

  • Preference Metrics: Preference accuracy measures whether scalar scores rank the accepted report above the rejected report, and is equivalent to pairwise AUC.It captures ranking correctness for human-preference judgments.
  • Preference Metrics: Paired Cohen’s d standardizes the score difference between accepted and rejected reports by the variability of those differences.The score difference is defined as Δi = S(r(i)acc) − S(r(i)rej).
  • Preference Metrics: The two metrics provide complementary views of preference quality: accuracy reflects ranking correctness, whereas Cohen’s d reflects score-separation strength at the query level.Together they assess both correctness and the magnitude and stability of preference separation.

H Rollout Speed-up for MaMs workflow

The MaMs workflow reduces rollout latency by overlapping asynchronous micro-batches under a concurrency limit. This changes the concurrent stage’s effective runtime from linear in dataset size to approximately |D|/C.

  • Rollout Speed-up: The Naive Linear Pipeline processes the dataset sequentially, causing network latency to accumulate linearly across samples.
  • Rollout Speed-up: The Linear Concurrent Pipeline partitions dataset D into configurable micro-batches and executes them across three processing stages.
  • Rollout Speed-up: An asynchronous event loop maintains at most C active tasks, launches available micro-batches, and admits new work as tasks complete.
  • Rollout Speed-up: O(|D|) to approximately O(|D|/C) reduces concurrent-stage complexity by overlapping high-latency API calls across micro-batches.The resulting runtime is bounded primarily by external API rate limits rather than local execution speed.

J Analysis on Entropy over Two RL Algorithms

The analysis compares GSPO and GRPO for rubric-generator training and examines the rubric structure used in the example. GSPO produces higher-entropy rollouts despite nearly identical rewards, motivating GRPO for stability.

  • Entropy comparison: GSPO produces higher-entropy rubric-generator rollouts than GRPO despite nearly identical reward values on generated samples.The paper attributes this to GSPO’s sequence-level importance weighting and clipping, which permit more diverse realizations with similar global rewards.
  • Algorithm choice: GRPO is adopted because rubric generation prioritizes stability and consistency, while GSPO’s greater rollout diversity is undesirable in this setting.
  • Example rubric: The example rubric requires coverage of common network failures, analysis of each failure, and a clear organizational structure.
  • Example rubric: Additional criteria address causal explanations, actionable troubleshooting, clarity for non-experts, professional tone, prevention, and concrete examples.
  • Example rubric: The rubric penalizes technical errors, listing without analysis, and irrelevant information, while optional criteria cover classification and preventive measures.

L Dataset and Rubric Statistics

The dataset and generated rubrics exhibit stable structural patterns. Rubrics average 13.3 items, and their weights combine positive quality criteria with negative penalties.

  • Rubric Count Distribution: 13.3 items per query is the mean rubric size across 5,651 train, validation, and test queries.The median is 13, 94.3% contain 10–16 items, and the full range is 8–23.
  • Rubric Count Distribution: The generator converges naturally to a stable level of evaluation granularity without explicit rubric-count constraints.
  • Rubric Weight Structure: 78.1% of the 75,251 rubric items have positive weights, while 21.9% have negative weights.Positive weights reward desirable qualities across critical, important, and optional tiers; negative weights penalize errors or omissions.
  • Rubric Weight Structure: The learned weight structure mirrors expert-designed rubrics by combining reward criteria with explicit penalty criteria without directly supervising this property.

M Future work

Future work targets richer preference supervision, more reliable assessment of difficult report qualities, reduced dependence on LLM evaluation, and broader transferability testing.

  • Future directions: Future preference modeling could move beyond pairwise comparisons to rankings or graded scores for finer-grained learning.
  • Future directions: Assessment could improve for novelty, creativity, factuality, and reasoning depth through stronger LLM evaluation combined with targeted human feedback.
  • Future directions: More principled self-consistency or hybrid human-LLM validation could reduce dependence on LLM-based evaluation and improve training stability and interpretability.
  • Future directions: Broader evaluations across scholarly search, domain-specific corpora, and alternative report formats could characterize transferability more fully.
Loading 2602.03619v2…