PaperlayerView exact revision on arXiv

Source-linked AI summary

MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan, Ramin Giahi, Kai Zhao, Parth Parikh, Jianpeng Xu, Jason Cho, Evren Korpeoglu, Sushant Kumar

arXiv:2610.36389v1cs.AIcs.IR

TL;DR

Human evaluation of recommendation explanations is resource-intensive, while automated LLM evaluators can be inconsistent and multi-agent debates poorly coordinated. MAREval combines planned roles, moderated discussion, arbitration, and Monte Carlo sampling; the paper reports stronger human alignment and improved online engagement and conversion metrics.

  • Problem

    Manual explanation evaluation is expensive and impractical at scale, while automated LLM evaluation can be biased or inconsistent and multi-agent debates may lack structured coordination.

    Source: §1 · pp. 1–2

  • Method

    MAREval coordinates LLM evaluators through a Chain of Debate planner, a moderator, and an arbitrator coupled with Monte Carlo sampling.

    Source: §1 · p. 2

  • Results

    MAREval consistently outperforms state-of-the-art baselines across evaluation criteria and reports higher consistency with human evaluations plus measurable online engagement and conversion improvements.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Online A/B tests report measurable improvements in user engagement and conversion metrics for MAREval.

    Source: §1 · p. 2

    The described moderation approach focuses on textual summarization; embedding-based moderation is offered as a possible generalization.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Evaluating natural language explanations in recommender systems is essential for fostering user trust, transparency, and engagement. However, existing evaluation approaches like human evaluations, while accurate, are resource-intensive and impractical at the scale required by modern recommendation platforms. Also, automated methods using single-agent LLMs suffer from prompt sensitivity and inconsistent outputs. To address these challenges, we propose MAREval, a structured multi-agent framework for evaluating recommendation explanations using large language models. MAREval orchestrates (i) a planner agent that uses a novel Chain of Debate (CoD) prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans; (ii) a moderator agent that regulates discussions by mitigating prompt drift; and (iii) an arbitrator agent that aggregates outputs from multiple evaluation rounds and (iv) a Monte Carlo sampling method, improving robustness and alignment with human judgment. We conduct comprehensive evaluations on both public (TopicalChat) and proprietary recommendation datasets, demonstrating that MAREval outperforms state-of-the-art baselines. Comprehensive experiments on a public benchmark and a proprietary e-commerce dataset show that MAREval improves alignment with human judgments over strong singleand multi-agent baselines. Stability analyses indicate substantially lower variability across repeated trials. In a large human-annotation gate, MAREval meets production quality thresholds where prior evaluators fall short, and online A/B testing demonstrates statistically significant improvements in engagement and revenue metrics. These results establish MAREval as a scalable and reliable solution for human-aligned evaluation of recommendation explanations in real-world systems

1 Introduction

Recommendation explanations can build trust and guide decisions, but manual evaluation is costly at scale, while automated LLM evaluators face bias, inconsistency, and poorly coordinated debate. MAREval structures multi-agent evaluation with planned roles, moderated discussion, and arbitration plus sampling to improve human alignment and online outcomes.

1 Introduction

  • Manual assessment of explanation relevance, phrasing, and informativeness is expensive, time-consuming, and impractical for large datasets.Explanation quality matters because misleading or unclear explanations can frustrate users and decrease platform reliability.Source: §1 · p. 1
  • Single-agent evaluators exhibit bias and inconsistency, while multi-agent frameworks can lack coordination and produce noisy, variable evaluations.Identical explanations may receive different scores within the same framework.Source: §1 · p. 2
  • MAREval’s Chain of Debate has a planner analyze the discussion flow and generate synchronized, role-specific plans to promote logical consistency and complementary reasoning.This planning-first structure contrasts with ad hoc or loosely coordinated multi-agent interactions.Source: §1 · p. 2
  • A moderator summarizes agreement and contention to provide shared context, reduce redundancy and noise, and regulate later evaluations.The passage describes moderation as a dynamic memory mechanism that avoids overloading prompt context.Source: §1 · p. 2
  • Arbitration across debate rounds and Monte Carlo aggregation of independently sampled debates counter output variability and improve alignment with human annotations.Experiments report higher consistency with human evaluations, while online A/B tests show measurable improvements in user engagement and conversion metrics.Source: §1 · p. 2

2 Related Works

Prior work applies LLMs and multi-agent debate to evaluation, with related systems also used in recommendation domains. MAREval is positioned as a response to coordination and consistency problems in debate-based evaluators.

2 Related Works

  • LLM evaluation research includes single-model judges, debate, and self-refinement; MAD uses agent discussion to refine responses after concerns about self-refinement were identified.The cited account describes a “Degeneration of Thought” problem in Self-Refine.Source: §2 · p. 2
  • Peer Discussion and CHATEVAL use agent discussions or communication strategies for evaluation, while agentic recommendation pipelines cover several recommendation tasks, including explanation.The cited recommendation areas include context-aware recommendations, interactive engagement, and user-behavior simulation.Source: §2 · pp. 2–3
  • MAREval addresses prior debate systems’ prompt drift, redundancy, and inconsistent outputs with planned roles, moderation, arbitration, and Monte Carlo sampling.The paper characterizes the framework as end-to-end and optimized for general dialogue and recommendation-specific explanation evaluation.Source: §2 · p. 3

3 Methodology

MAREval’s methodology coordinates evaluation agents through synchronized planning, moderates debate context before arbitration, and aggregates repeated runs to reduce sampling-driven instability.

3.1 Chain of Debate

  • CoD uses evaluation criteria, context, and pipeline flow to generate synchronized, stepwise plans for evaluators, the moderator, and the arbitrator.The plans coordinate agents whose steps depend on information from other agents.Source: §3.1 · p. 3

3.2 Context Moderation and Arbitration

  • The moderator distills prior-round discussions into essential context, and the arbitrator uses the final summary with its plan, criteria, and instance context to produce the evaluation.The final moderator summary is passed to the arbitrator after T rounds.Source: §3.2 · pp. 3–4

3.3 Monte Carlo Sampling and Score Function

  • Monte Carlo sampling uses probability-weighted outcomes when token probabilities are available, or averages N independent full-pipeline runs for black-box models.The authors propose sampling to mitigate outlier results and report that both strategies reduce reliance on single trajectories for more stable outcomes.Source: §3.3 · pp. 4–5

4 Experiments

Experiments compare MAREval with five evaluator baselines on TopicalChat and a proprietary e-commerce dataset. MAREval leads in benchmark agreement and stability, meets human-annotation launch thresholds, and improves online business metrics.

4.1 Implementation Details and Datasets

  • The experiments use 108 proprietary e-commerce use-case phrases labeled for relevance and phrasing, and TopicalChat responses labeled on six conversational criteria.All results are based on a gpt-3.5-turbo model.Source: §4.1 · p. 5

4.2 Metrics and Benchmarks

  • Agreement with human annotations is measured using Cohen’s κ and Pearson’s ρ for binary outcomes, and Pearson’s ρ and Kendall-Tau’s τ for integer-range outcomes.Benchmarks include Vanilla-gpt, Chain-of-Thought, Self-Refine, G-Eval, and ChatEval.Source: §4.2 · p. 5

4.3 Overall Performance

  • MAREval outperforms all benchmarks, reaching ρ = 0.548 and τ = 0.478 for TopicalChat Overall Chat Quality, ρ = 0.338 and κ = 0.334 for PE Relevance, and ρ = 0.216 and κ = 0.208 for PE Phraseness.Table 1 compares framework variants and five benchmarks across both datasets.Source: Table 1 · p. 6, §4.3 · p. 6

4.4 Stability and Reliability Analysis

  • MAREval had the lowest average standard deviation across TopicalChat criteria at 0.0197, while G-Eval averaged 0.0227, 15.22% higher.Self-Refine had the highest average standard deviation at 0.1563.Source: §4.4 · p. 6, Table 2 · p. 7

4.5 Ablation Study

  • Removing moderation lowered groundedness correlations from ρ = 0.553, κ = 0.536 to ρ = 0.497, κ = 0.484; removing moderation and Monte Carlo sampling lowered Overall Chat Quality ρ from 0.548 to 0.456.The study also reports degradation when CoD is removed, attributing its relative advantage to planner-aware coordination.Source: §4.5 · p. 7, Table 3 · p. 8

4.6 Hyperparamers and Performance vs Latency

  • From 2 to 4 evaluator agents, Pearson correlation rose from 0.509 to 0.548 and Kendall-tau from 0.442 to 0.478; five agents averaged 8.16 seconds per single-thread run.Five Monte Carlo sample runs reached higher performance, with no significant marginal correlation improvement from more runs.Source: §4.6 · pp. 7–8

4.7 Human-Annotated Evaluation

  • MAREval-filtered explanations met all hard requirements (100% vs. 91% for G-Eval) and achieved 98.6% vs. 76.1% soft-criterion satisfaction.The results covered 5,000 MAREval-filtered and 2,500 G-Eval-filtered samples; G-Eval alone had not met launch requirements.Source: §4.7 · pp. 8–9, Table 4 · p. 9

4.8 Online Experiments

  • ATC increased by 1.18% (p=0.05) and GMV increased by 1.36% (p<0.01) in the online test of MAREval-validated explanations.The test covered approximately 300,000 high-traffic items across 14 product categories; controls saw the same items without explanations.Source: §4.8 · p. 9, Table 5 · p. 9

5 Conclusion

MAREval combines structured multi-agent evaluation with planning, moderation, arbitration, and Monte Carlo sampling. Experiments report stronger human-judgment alignment and lower variability, while an A/B test found engagement and revenue gains.

5 Conclusion

  • Across public and proprietary datasets, MAREval outperforms state-of-the-art baselines, improves alignment with human judgments, and reduces output variability.A large-scale A/B test reports a 1.36% lift in GMV and a 1.18% increase in ATC rate.Source: §5 · p. 9
  • MAREval combines a planner using Chain of Debate, a moderator that regulates discourse and context noise, and an arbitrator coupled with Monte Carlo sampling.Ablation studies validate the components’ complementary benefits.Source: §5 · p. 9
  • The authors position MAREval as a scalable, reliable, and practically effective framework for evaluating natural-language explanations in recommender systems.Source: §5 · p. 9

A Case Study I: MAREval Evaluation on Conversational Response

The case study shows MAREval evaluating conversational responses for understandability and naturalness through evaluator opinions, a moderator summary, and an arbitrated score. In the examples, evaluators sometimes disagree on understandability but agree on naturalness.

A Case Study I: MAREval Evaluation on Conversational Response

Criteria: Understandability

  • For Example 1’s understandability, two evaluators score the response 1 and one scores it 0; the moderator records the split and the arbitrator selects 1.Source: Criteria: Understandability · p. 12

Criteria: Naturalness

  • In Example 1, all three evaluators rate naturalness 2, citing awkward phrasing or missing punctuation, and the moderator summarizes these issues.Source: Criteria: Naturalness · pp. 12–13

Criteria: Understandability

Criteria: Naturalness

In the Example 2 naturalness evaluation, evaluators cite unclear phrasing and abrupt transitions; the moderator summarizes their concerns before arbitration.

Criteria: Naturalness

  • Two evaluators score the response 2 for naturalness, while one scores it 1; they cite unclear phrasing, missing explanation, and an abrupt transition.Source: Criteria: Naturalness · p. 14
  • The moderator summarizes the shared concerns about clarity, punctuation, and the transition between the 1960s and the comparison of football and bowlers.Source: Criteria: Naturalness · p. 14
  • The arbitrator assigns the final naturalness score of 2.Source: Criteria: Naturalness · p. 14

B Case Study II: MAREval Evaluation on Recommendation Explanations

The case study and evaluation prompts illustrate a multi-agent sequence in which evaluators assess a response, a moderator summarizes their opinions, and a judge selects an evaluation. The planner prompt asks for ordered evaluation steps that include the scoring rule.

B Case Study II: MAREval Evaluation on Recommendation Explanations

C.1 Planner Prompt

  • The planner prompt requests an algorithmic chain of evaluation steps that accounts for the pipeline and includes the scoring rule.The planner receives the chat context, response, criteria, and pipeline flow description.Source: §C.1 · p. 17

C.2 Evaluator Prompt

  • The evaluator prompt asks each evaluator to assess a response using chat context, criteria, and the planner’s steps, then end with a score.Source: §C.2 · p. 17

C.3 Moderator Prompt

  • The moderator prompt asks for a summary of two evaluators’ opinions for the judge, labeled as a summarized reason for evaluation.Source: §C.3 · pp. 17–18

C.4 Arbitrator Prompt

  • The arbitrator prompt asks the judge to select the best evaluation opinion from the evaluators’ responses and moderator summary, then output an integer score.Source: §C.4 · p. 18

D Real World Implementation

The section identifies a real-world deployment of MAREval and a comparison of its score correlations across backbone models on TopicalChat.

  • Figure 6 depicts real-world deployment of recommendation explanations using MAREval.Source: Figure 6 · p. 18
  • Table 6 reports Pearson correlation and Kendall–Tau with human judgments for Overall Score Criteria on TopicalChat.Source: Table 6 · p. 18
  • The table compares MAREval score correlations across backbone models.Source: Table 6 · p. 18

E Different Backbone Model Performance

MAREval’s alignment with human judgments varies across backbone models, with gemini-2.5-pro showing the strongest correlations among those compared.

Related papers

Loading 2610.36389v1…