PaperlayerView exact revision on arXiv

Source-linked AI summary

RecRM-Bench: Benchmarking Multidimensional Reward Modeling for Agentic Recommender Systems

Wenwen Zeng, Jinhui Zhang, Hao Chen, Zhaoyu Hu, Yongqi Liang, Jiajun Chai, Dengcan Liu, Zhenfeng Liu, Shurui Yan, Minglong Xue, Xiaohan Wang, Wei Lin, Guojun Yin

arXiv:2605.11874v1cs.IR

TL;DR

Agentic recommenders make multi-step decisions, yet reward signals focused on final interactions leave intermediate capabilities unassessed. RecRM-Bench supplies four-dimensional training data, and RecRM-RL integrates the resulting reward signals; the authors report reduced training variance and improved performance on complex, multi-step recommendation tasks.

  • Problem

    Existing recommendation rewards focus on final user interactions, while the sparsity of feedback in multi-step agentic systems makes training unstable and complicates credit assignment.

    Source: §1 · pp. 1–2

  • Method

    RecRM-Bench supplies over 1.1 million entries across four dimensions, and RecRM-RL integrates independent reward models in a hierarchical reinforcement-learning framework.

    Source: §1 · p. 2, §4 · p. 5, §4.5 · p. 7

  • Results

    The proposed multidimensional RL framework uses holistic reward feedback, reducing training variance and improving agent performance in complex, multi-step recommendation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RecRM-Bench provides a large-scale foundation for training multidimensional reward models across instruction following, factual consistency, query-item relevance, and user behavior.

    Source: §1 · p. 2, §6 · p. 11

Abstract

from arXiv · show

The integration of Large Language Model (LLM) agents is transforming recommender systems from simple query-item matching towards deeply personalized and interactive recommendations. Reinforcement Learning (RL) provides an essential framework for the optimization of these agents in recommendation tasks. However, current methodologies remain limited by a reliance on single dimensional outcome-based rewards that focus exclusively on final user interactions, overlooking critical intermediate capabilities, such as instruction following and complex intent understanding. Despite the necessity for designing multi-dimensional reward, the field lacks a standardized benchmark to facilitate this development. To bridge this gap, we introduce RecRM-Bench, the largest and most comprehensive benchmark to date for agentic recommender systems. It comprises over 1 million structured entries across four core evaluation dimensions: instruction following, factual consistency, query-item relevance, and fine-grained user behavior prediction. By supporting comprehensive assessment from syntactic compliance to complex intent grounding and preference modeling, RecRM-Bench provides a foundational dataset for training sophisticated reward models. Furthermore, we propose a systematic framework for the construction of multi-dimensional reward models and the integration of a hybrid reward function, establishing a robust foundation for developing reliable and highly capable agentic recommender systems. The complete RecRM-Bench dataset is publicly available at https://huggingface.co/datasets/wwzeng/RecRM-Bench.

1 Introduction

Agentic recommender systems use multi-step reasoning, but reward models centered on terminal interactions leave process-level capabilities and intermediate decisions unassessed. RecRM-Bench and RecRM-RL address this gap with four-dimensional evaluation and integrated reward signals.

1 Introduction

  • Sparse terminal feedback makes training unstable and complicates credit assignment in multi-step recommendation, motivating explicit process-level rewards.The proposed rewards are intended to support stable and efficient learning.Source: §1 · pp. 1–2
  • Single-dimensional rewards tied to clicks or ratings omit instruction compliance and semantic query-item relevance, alongside user behavior prediction.The paper identifies syntactic compliance and reliable tool interaction as operational-integrity requirements.Source: §1 · p. 2
  • RecRM-Bench provides over 1.1 million entries across instruction following, factual consistency, query-item relevance, and user behavior prediction.The benchmark is designed for training and assessing multidimensional reward models for agentic recommenders.Source: §1 · p. 2
  • The proposed multidimensional RL framework uses holistic reward-model feedback, reducing training variance and improving agent performance in complex, multi-step recommendation tasks.The paper lists this framework alongside the benchmark and reward-model training framework as a contribution.Source: §1 · p. 2

2 Related Work

Prior work includes ranking-centric, simulation-centric, and conversational recommender agents, while benchmark efforts often assess isolated capabilities. RecRM-Bench brings multiple evaluation dimensions together.

2.1 Agentic Recommender Systems

2.2 Benchmarking Recommender Systems

  • Existing benchmarks often address isolated behavior or agent capabilities, whereas RecRM-Bench integrates instruction following, factual consistency, query-item relevance, and user behavior prediction.Traditional interaction datasets also miss the complex reasoning, tool integration, and multi-step decisions of agentic systems.Source: §2.2 · p. 2

3 Construction of RecRM-Bench

RecRM-Bench organizes real-world Meituan interactions into four specialized databases for evaluating agentic recommendation capabilities. Its construction combines rubric-based instruction scoring, hallucination and relevance labels, and fine-grained behavior and ranking signals.

3 Construction of RecRM-Bench

  • The benchmark derives four specialized databases from Meituan interactions, including 68,096 query-response pairs spanning explicit-merchant, explicit-product, and multi-condition intents.Figure 2 identifies instruction following, factual consistency, query-item relevance, and user behavior prediction as the four dimensions.Source: §3 · p. 3, Figure 2 · p. 3, §3.1.1 · pp. 3–4

3.1 Instruction Following Database Construction

  • Instruction-following assessment uses six refined rubric dimensions, followed by LLM judging and counterfactual synthesis to create 6,422 single-violation negatives from 2,000 compliant seeds.The labeled training set contains fine-grained compliance scores, with score distributions summarized in Figure 3.Source: §3.1.3 · p. 4, §3.1.2 · p. 4, Figure 3 · p. 4

3.2 Factual Consistency Database Construction

  • Factual consistency distinguishes fabricated or conflicting items from ungrounded or incorrect content claims, using 9,391 samples with overlapping hallucination labels and accurate responses.The collection includes 3,066 item-hallucination cases, 1,188 content-hallucination cases, and 6,165 accurate responses; one sample may contain both hallucination types.Source: §3.2.1 · p. 4

3.3 Query-Item Relevance Database Construction

  • The relevance database samples over 20,000 Meituan interactions across six service categories and assigns three relevance levels across seven dimensions.Human annotations aligned GPT-4.1 distillation, yielding 19,456 query-item-score triplets.Source: §3.3.1 · pp. 4–5, §3.3 · p. 4

3.4 User Behavior Database Construction

  • The user-behavior database combines prediction with item ranking, using user profiles, queries, candidates, and behavior labels across more than one million interactions.It includes 960,862 behavior-prediction samples and a 75,648-sample ranking subset, with profiles spanning demographics, consumer profile, long-term preferences, and real-time context.Source: §3.4.1 · p. 5, §3.4 · p. 5

3.4.1 Data Collection.

  • Only 23.1% of 4,163,922 item exposures produced positive outcomes, motivating process-level ranking labels as dense relative-preference signals alongside sparse terminal feedback.The authors state that this multi-layered structure supports reward models sensitive to preference variation and final conversions.Source: §3.4.2 · p. 5

4 RecRM-RL

RecRM-RL combines four independently trained reward models under ReAct, providing hybrid feedback focused on final responses rather than internal tool-calling trajectories.

4 RecRM-RL

  • RecRM-RL integrates instruction-following, factual-consistency, query-item relevance, and user-behavior reward models into a hybrid multi-objective feedback system.The ReAct-based optimization focuses on final-response quality rather than internal tool-calling trajectories.Source: §4 · p. 5

4.1 Instruction Following Reward Model

  • The instruction-following model uses rubric-based supervised fine-tuning to return verification reasons and overall and dimension-specific compliance scores.Cross-entropy training is intended to internalize requirements without manual or task-specific rule checking.Source: §4.1 · p. 6

4.2 Factual Consistency Reward Model

  • The factual-consistency model identifies discrepancies between agent responses and tool outputs or general knowledge using a specialized consistency database.It is optimized with a sequence-level cross-entropy objective.Source: §4.2 · p. 6

4.3 Query-Item Relevance Reward Model

  • The query-item relevance model scores how well recommended items match a user query, using human-annotated and LLM-distilled training data.Its scores can penalize irrelevant items and narrow the action space during reinforcement learning.Source: §4.3 · pp. 6–7

4.4 User Behavior Reward Model

  • The user-behavior reward mechanism provides dense supervision to stabilize training despite sparse direct user feedback.Source: §4.4 · p. 7

4.4.1 Item Ranking.

  • The item-ranking model scores retrieved candidates from user profiles and queries, using 82,993 samples and both click-prediction and list-ranking losses.The dual objective captures item-wise click probability and relative ranking relationships.Source: §4.4.1 · p. 7

4.4.2 Behavior Prediction.

  • The behavior-prediction model is fine-tuned on 1,000,000 feedback sequences to predict whether the positive item appears in the recommendation list.This list-based objective addresses sparse, noisy item-level signals.Source: §4.4.2 · p. 7

4.5 RecRM-RL Training Objective

  • The training objective gates behavioral rewards behind instruction compliance, factual consistency, and a relevance threshold.Only responses passing the foundational checks receive item-ranking and behavior-prediction signals; reward components are normalized to [0, 1].Source: §4.5 · p. 7

5.1 Evaluation of RecRM-Bench

The evaluation finds substantial differences across models and benchmark dimensions, while the optimized reward models outperform zero-shot baselines on reported tasks. Database validation results also show varying levels of agreement across dimensions and relevance classes.

5.1 Evaluation of RecRM-Bench

  • The evaluation compares zero-shot models and optimized reward models across benchmark databases using accuracy and F1 or accuracy, AUC, and ranking HR metrics.The study also validates instruction-following and query-item relevance database reliability.Source: §5.1.1 · p. 8, §5.1.2 · p. 8

5.1.2 Data Quality.

  • Instruction-following labels achieve weighted κ>0.61 overall, with constraint and process compliance agreement above κ>0.87.Agreement is lower for content-quality and style compliance.Source: §5.1.2 · p. 8
  • Query-item relevance distillation reaches weighted κ=0.71, ρ=0.76, and 77% raw agreement; F1 is higher for irrelevant and fully relevant items than weakly relevant ones.The reported F1-scores are 79.45% for irrelevant, 81.36% for fully relevant, and 70.59% for weakly relevant items.Source: §5.1.2 · p. 8
  • GPT-4.1 leads query-item relevance accuracy at 79.26%, while LongCat-Flash-Chat leads instruction following, item ranking, and factual consistency.LongCat-Flash-Chat reports 64.84% instruction-following accuracy, 62.48% item-ranking accuracy, 82.93% HR, and 67.34% factual-consistency accuracy; DeepSeek-V3.2 leads behavior prediction accuracy at 47.80%. Thinking models do better on complex relevance, while non-thinking models lead on the other databases.Source: §5.1.3 · p. 8

5.1.3 Evaluation Performance.

  • 89.36% query-item relevance accuracy and 86.78% item-ranking accuracy are achieved by Ours, significantly outperforming zero-shot baselines.The reported model performance remains imbalanced across databases, with accuracy exceeding 70% in query-item relevance but dropping sharply in others.Source: §5.1.3 · p. 8

5.2 Performance of RecRM-RL

RecRM-RL’s structured reward signals accelerate training convergence and improve final behavior scores, with relevance rewards contributing an especially notable gain.

5.2 Performance of RecRM-RL

  • Structured rewards substantially accelerate convergence over the baseline, with the largest learning-efficiency gain after adding Query-Item Relevance (R3).The authors interpret this as evidence that recognizing user intent is a prerequisite for effective exploration.Source: §5.2 · p. 9
  • The final behavior score improves 19% with all rewards and 7.8% after adding Relevance (R3).The 7.8% gain is particularly evident after integrating Relevance reward.Source: §5.2 · p. 9
  • Instruction Following (R2) and Factual Consistency (R5) stabilize early training and prune ineffective paths despite lacking a direct functional dependency on the final click.The authors say these rewards reduce optimization noise and help identify a policy within a valid candidate pool.Source: §5.2 · p. 9

5.3 Ablation Study

The ablations examine synthetic-data augmentation, reranker architectures, and model scale, finding that the best choice depends on the task and system-design priorities.

5.3.1 Impact of Data Augmentation.

  • 15.63% improvement in overall-score prediction accuracy follows inclusion of synthesized Instruction Following data, with smaller changes in content quality and style.The reported content-quality gain is 0.78%, while style decreases by 0.78%.Source: §5.3.1 · p. 9, Table 8 · p. 10

5.3.2 Impact of Base Retrievers.

  • The single-tower Reranker gains 14% AUC over the Embedding backbone, but this advantage reverses in multi-tower configurations; the final framework selects a generative reranker for flexibility.The authors cite prompt-based integration of multidimensional criteria without structural redesign and architectural consistency with the agent as selection priorities.Source: §5.3.2 · pp. 9–10, Figure 6 · p. 10

5.3.3 Impact of Model Size.

  • The 14B model excels at Query-Item Relevance, while 8B is optimal for Instruction Following and Behavior Prediction.The authors attribute the behavior-prediction result to sparse, noisy data and the 8B model’s stronger inductive bias for generalized preferences.Source: §5.3.3 · p. 10, Table 9 · p. 10

5.4 Failure Case Analysis

Failure analysis identifies distinct weaknesses in factual verification and complex intent matching, including over- and under-detection of hallucinations and unstable multi-condition reasoning.

5.4 Failure Case Analysis

  • Factual Consistency failures comprise over-sensitivity to item hallucination (56.6%), false negatives on item hallucination (20.2%), over-sensitivity to content hallucination (16.1%), and false negatives on content hallucination (7.1%).The authors identify insufficient cross-verification against reference item attributes as a systematic failure mode.Source: §5.4 · p. 10, Figure 8 · p. 11
  • Query-Item Relevance failures most often involve multi-conditional intent misjudgment (34.4%) and category misjudgment (25%).The authors link these failures to category and knowledge gaps and inconsistent weighting of multiple intent conditions.Source: §5.4 · pp. 10–11

6 Conclusion

The conclusion presents RecRM-Bench as a large-scale benchmark for multidimensional reward modeling and RecRM-RL as a hierarchical framework for integrating its reward signals.

6 Conclusion

  • RecRM-Bench contains 1,073,779 samples across four subdatabases, and RecRM-RL demonstrates integration of multidimensional reward signals for agentic behavior optimization.The four dimensions are Instruction Following, Factual Consistency, Query-Item Relevance, and User Behavior.Source: §6 · p. 11

A System Prompt Design for Data Generation and Reward Modeling

The section presents a prompt for instruction-following data augmentation and prompts for assessing five recommendation capabilities.

A System Prompt Design for Data Generation and Reward Modeling

Related papers

Loading 2605.11874v1…