Source-linked AI summary
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Zhongru Chen, Yuan Wu, Yi Chang
TL;DR
Existing recommendation benchmarks do not test whether LLM assistants can critique faulty premises grounded in visible recommendation evidence. RPCBench addresses this gap with an evidence-grounded benchmark and fine-grained evaluation framework, finding proactive detection to be the main bottleneck while longer reasoning can incur overthinking penalties.
Problem
Existing recommendation evaluations mainly assess ranking, generation, or preference satisfaction, while error-detection benchmarks are not grounded in recommendation-specific visible evidence.
Method
RPCBench provides 4,623 evidence-grounded instances from five recommendation domains with ten premise-failure types, evaluating detection, localization, handling strategy, and evidence faithfulness.
Results
Proactive detection is the main bottleneck; effective information density improves critique and faithfulness, while critique quality peaks at intermediate reasoning length and can degrade with overthinking.
Takeaways & Limitations
Reliable recommender LLMs should emphasize proactive premise critique and faithful use of visible evidence rather than merely longer reasoning or more attractive recommendations.
Takeaways & Limitations
RPCBench may be affected by pretraining contamination and residual LLM-generated query bias, and it is currently English-only; therefore it is a controlled diagnostic benchmark rather than an estimate of real-world premise-failure prevalence.
Abstract
from arXiv · showhide
Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommendation and test whether they can recognize flawed recommendation requests. Existing recommender benchmarks mainly assess ranking, generation, or preference satisfaction, while existing error-detection benchmarks are usually not grounded in recommendation-specific user and candidate evidence. To address this gap, we introduce RPCBench, a benchmark for evaluating Recommender-Premise Critique: the ability to detect, diagnose, and properly handle faulty premises in natural-language recommendation requests. RPCBench contains evidence-grounded test instances from five recommendation domains and covers ten types of premise failures. Each instance provides a visible recommendation context and a corrupted user query. We further design a fine-grained evaluation framework that measures proactive detection, error localization, post-detection handling strategy, and evidence faithfulness. Through a systematic evaluation of 11 LLMs, we find that proactive detection is the main bottleneck in Recommender-Premise Critique, and models perform worst on underspecified-premise errors. We also observe that target-critical information density matters more than redundant evidence, and that longer reasoning does not monotonically improve critique quality: performance peaks at intermediate reasoning length, while overly long reasoning is accompanied by an overthinking penalty. The code is available at https://github.com/ZhongruChen/RPCBench.
1 Introduction
RPCBench addresses a gap in recommendation evaluation by testing whether LLM recommenders can recognize and handle requests whose necessary premises are not established by visible evidence. It introduces an evidence-grounded benchmark and finds proactive detection to be the main bottleneck.
- Interactive recommendation assistants must handle natural-language requests containing richer preferences, constraints, and contextual assumptions than traditional clicks, ratings, or filters.
- Existing recommendation studies usually assume requests are answerable under available context, leaving proactive critique of faulty premises insufficiently tested.
- RPCBench contains 4,623 evidence-grounded test instances from five recommendation domains and covers ten types of premise failures.
- The benchmark evaluates premise-failure detection, cause localization, post-detection handling strategy, and faithfulness to visible evidence.
- Proactive detection is the main bottleneck, while effective evidence density and reasoning length are further analyzed as factors related to critique quality.
2 Related Work
Prior recommendation benchmarks emphasize utility, ranking, generation, or conversational alignment, while general error benchmarks do not fully capture recommendation-grounded premise critique. RPCBench combines proactive diagnosis, handling strategies, overthinking analysis, and evidence faithfulness in a recommendation setting.
- LLM Recommendation: Earlier LLM recommendation studies mainly evaluate ranking, sequential or direct recommendation, rating prediction, explanations, summaries, and generation quality.
- LLM Recommendation: Interactive recommendation evaluations broaden task coverage but do not systematically test proactive faulty-premise detection, cause localization, handling strategy, and visible-evidence faithfulness.
- Related error benchmarks study response errors or step-level diagnosis, but they do not jointly evaluate multiple handling strategies, explicit evidence boundaries, and reasoning-length effects in recommendation.
- RPCBench provides a recommendation-grounded framework covering ten fine-grained error types, proactive detection, localization, handling strategies, overthinking, and evidence faithfulness.
3 Method
RPCBench formalizes recommendation critique around visible user, candidate, state, and query evidence, then constructs controlled corrupted requests and evaluates both critique capability and evidence faithfulness.
- Task Formulation: Each instance is represented by user evidence H, candidate evidence C, and query Q, with all evidence restricted to the rendered visible payload.
- Task Formulation: A premise failure occurs when a necessary condition, constraint, assumption, or factual claim cannot be established from the visible evidence and query.
- Task Formulation: RPCBench groups failures into underspecified, inconsistent, unsupported, and boundary premises, further dividing them into ten fine-grained error types.
- Benchmark Construction: The benchmark uses five public recommendation datasets spanning movies, news, local businesses, e-commerce products, and books to form sample units and candidate scopes.
- Benchmark Construction: A generator creates solvable evidence-grounded queries and injects one targeted failure, after which cross-model review and manual deduplication produce 4,623 final samples.
- Evaluation: The evaluation separates critique capability from evidence faithfulness and measures detection, localization, handling strategy, and grounding in visible user, candidate, and state evidence.
CLAm + CSQm
The framework supplements core critique metrics with consistency checks and evidence-use measures, while human verification assesses the reliability of automatic LLM-judge scoring.
- CLAm + CSQm: CPCC retains a full-sample penalty for missed detections while rewarding correct fault diagnosis and appropriate handling strategies.
- CLAm + CSQm: The instance-level iCPCC combines normalized localization and strategy scores within each response before averaging.
- CLAm + CSQm: Spearman ρ = 0.9909 shows highly consistent rankings between CPCC and iCPCC.
- Evidence Evaluation: Evidence evaluation measures faithfulness to visible user, candidate, and state evidence, alongside fabrication and evidence-distortion rates.
- Evaluation Reliability: Three independent LLM judges achieved a content-level macro-average Fleiss’ κ of 0.7583, while human verification agreed with aggregated judgments on 82.60% of sampled responses.
4 Experiments
Experiments evaluate 11 LLMs on RPCBench using critique-capability and evidence-faithfulness metrics, controlled clean-query checks, error-group analysis, evidence ablations, and within-model reasoning-length trends. Results identify proactive detection and underspecified premises as key weaknesses, show gains from denser target-relevant evidence, and reveal non-monotonic effects of reasoning length.
- Experimental setup: 11 LLMs are evaluated using critique-capability and evidence-faithfulness metrics, including PDR, CLA, CSQ, CPCC, EFI, F1R, and FFR.Table 2 defines the reported metric blocks and their scales.
- Overall results: 0.5151 average PDR and 0.4376 average CPCC show weak proactive premise critique, although downstream localization and strategy quality are usually relatively high after detection.Qwen3.5-Plus achieves the highest CPCC at 0.5261 among the evaluated models.
- Overall results: 49.45% matched corrupted-query PDR versus 0.55% clean-query false-positive rate shows detection is selective but incomplete.The paired control uses 400 stratified clean–corrupted pairs.
- Breakdown by error group: 0.1001 mean PDR and 0.0595 mean CPCC make underspecification the hardest coarse error group, while unsupported premises reach 0.7684 mean PDR and 0.6165 mean CPCC.Models often treat missing constraints, ambiguous preferences, or incomplete conditions as ordinary intent rather than premises requiring clarification.
- Ablation study: +0.1384 CPCC and +0.0361 EFI result when auxiliary fields are removed while target-critical evidence is preserved.The same ablation reduces FFR by 0.0150 and F1R by 0.0623, supporting structured target-relevant density over redundant context volume.
- Reasoning length and critique quality: 0.5623 content CPCC at Q3 and 0.6214 reasoning CPCC at Q4 contrast with 0.2991 and 0.4037 at Q10, showing critique quality peaks at intermediate reasoning length.Detection peaks later around Q7, while adjusted curves preserve middle-length peaks and long-end declines.
- Reasoning length and critique quality: 0.1816 content CPCC and 0.1449 reasoning CPCC overthinking penalties show robust underperformance of the longest reasoning bin versus the best middle bin.EFI also declines in both settings, alongside sharper degradation in strict critique success.
5 Conclusion
RPCBench evaluates recommender-premise critique across evidence-grounded instances, finding proactive detection to be the main bottleneck and longer reasoning to have non-monotonic effects on quality.
- RPCBench contains 4,623 evidence-grounded test instances spanning five recommendation domains and ten premise-failure types.
- The benchmark evaluates 11 LLMs on critique capability and evidence faithfulness.
- Proactive detection is the main bottleneck for reliable recommender-premise critique.
- Increasing effective information density in the visible payload improves premise critique and evidence faithfulness.
- Performance peaks around an intermediate reasoning length, while overly long reasoning can degrade critique quality through overthinking.
Limitations
RPCBench is limited by possible pretraining contamination and LLM-generated query bias, which prevent it from fully representing real-world premise-failure distributions.
- Public recommendation data may have appeared in evaluated LLMs’ pretraining corpora, so memorized external knowledge cannot be fully ruled out.
- LLM-generated initial query pairs may retain generator-specific style, preference, or reasoning bias despite cross-model review.
- RPCBench is a controlled diagnostic benchmark rather than an estimate of premise-failure prevalence or linguistic distribution in real-world traffic.
- The benchmark is currently English-only, leaving multilingual and cross-cultural recommendation scenarios unexplored.
- Its five public datasets differ in visible evidence, and some premise failures can only be constructed where corresponding evidence exists.
Ethical Considerations
RPCBench includes safety-related cases for defensive evaluation, uses public evidence-limited data, and relies on general-purpose LLMs because the task requires free-form critique and handling strategies.
- Safety and compliance cases test whether models refuse, constrain, or redirect unsafe requests rather than elicit harmful recommendations.
- The benchmark avoids private user data and constructs samples from public recommendation datasets with evidence-limited payloads.
- General-purpose LLMs are used because RPCBench requires free-form premise critique, causal localization, and natural-language handling strategies.
- Specialized recommender models are excluded because their available interfaces primarily support ranking or constrained recommendation outputs.
- Reasoning-level analyses are limited to models whose providers expose reasoning content.
- Main evaluation uses final answers, while exposed reasoning fields are retained for separate analysis.
A.4 Benchmark Sample Statistics
RPCBench’s sample distribution is only partly balanced because premise-failure types depend on the evidence available in each dataset and on validated sample construction.
- The dataset-by-type distribution is not fully crossed because some cells are unsupported, naturally constructable, or approximately balanced.
- I4 appears only in Yelp because it requires snapshot-internal state evidence.
- U1 and I3 are more naturally diverse in metadata-rich Amazon and Goodreads than in MovieLens.
- U2 is exactly balanced across datasets, while I2 is nearly balanced.
- Raw dataset aggregates should not be interpreted as intrinsic domain-difficulty estimates.
B.1 Metric Definitions
RPCBench evaluates premise critique through detection, diagnosis, handling, and evidence-faithfulness measures, with judge aggregation and validation against human annotations.
- Detection: Proactive Detection Rate measures the full-sample rate at which responses detect faulty recommendation premises.Detected premises include flawed, missing, unverifiable, infeasible, conflicting, or unsafe premises.
- Localization and handling: Conditional Localization Accuracy measures diagnosis quality after detection, while Conditional Strategy Quality measures post-detection handling quality.Localization distinguishes root-cause, incomplete, and absent diagnoses; strategy categories include repair, clarification, and refusal.
- Evidence faithfulness: Faithfulness scores whether responses use visible evidence without distortion, external information, invisible fields, or fabricated facts.Mixed faithful and fabricated evidence receives the lowest overall faithfulness score.
- Evaluation reliability: Three independent LLM judges score detection, localization, strategy, and faithfulness, with majority voting for detection and aggregation rules for other measures.Human verification of 500 responses achieved 96.63% mean exact agreement, while judge–human agreement ranged from 71.40% to 95.20% across fields.
- Composite metrics: CPCC and iCPCC produce highly consistent model rankings, with Spearman ρ = 0.9909 and unchanged top-three models.Nine of 11 models retain exactly the same rank; only GPT-5.5 and DeepSeek-V4-Pro swap adjacent positions.
- Dataset variation: Performance varies by dataset: Yelp and Amazon have the highest CPCC, while MovieLens is hardest and weakest on faithfulness.MovieLens’s sparse item-side evidence is offered as a plausible explanation for greater difficulty suppressing unsupported external movie knowledge.
C.2 Minimal-valid Evidence Ablation
The minimal-valid evidence ablation tests whether critique depends on evidence structure rather than raw visible volume by removing auxiliary information while preserving target-critical facts.
- Ablation design: The paired ablation conservatively prunes auxiliary fields, long text, and redundant attributes while preserving evidence needed to keep each annotated premise failure valid.Four hundred instances are sampled, then the 11 models are rerun under the same scoring protocol.
- Results: Removing auxiliary evidence improves CPCC and EFI while reducing FFR and F1R.The result distinguishes raw evidence volume from evidence structure in premise-critique evaluation.
- Interpretation: Structured, target-relevant evidence benefits critique and faithfulness more than larger visible payloads containing redundant or weakly relevant fields.Additional richness regressions likewise find raw richness non-monotonic, while structured state evidence is positively associated with CPCC and EFI.
C.4 Fixed-Effect Reasoning-Length Analysis
The reasoning-length analysis models nonlinear relationships between within-model reasoning length and critique outcomes, then quantifies whether the longest reasoning group underperforms an intermediate group.
- Model specification: The fixed-effect model controls for model, error type, dataset, corrupted-query length, and visible-payload length while fitting a nonlinear function of log reasoning length.Binary outcomes use logistic regression, and continuous CPCC is estimated with OLS; adjusted curves visualize the specification.
- Nonlinear effects: Content CPCC and reasoning CPCC show significant nonlinear reasoning-length patterns, with joint-test p-values of 0 and 9.4e-9.The final specification reports matching significance for the binary outcomes.
- Length grouping: Reasoning-length analyses use within-model deciles or quintiles, so groups represent relative rather than shared absolute token lengths.The bootstrap analysis defines the best middle bin as the best-performing Q2–Q4 quintile and the longest bin as Q5.
- Overthinking penalty: The overthinking penalty compares the best middle-length score with the longest-length score, and bootstrap confidence intervals exclude zero for CPCC, strict success, and EFI.Positive penalties indicate underperformance by the longest quintile relative to the best middle-length quintile.
- Critique suppression: Internal critique is sometimes suppressed in final answers: 6,625 of 31,172 reasoning-enabled responses show hidden awareness, yielding CSR = 0.2691.Suppression is highest for U errors at 0.7318, compared with 0.2274 for I, 0.1915 for B, and 0.1249 for X.
- Representative case: A representative qwen3.5-plus case detects and localizes an I1 contradiction in reasoning but surfaces no critique in the final response.Reasoning scores are D/L/S = (1, 2, 2), whereas final-answer scores are (0, 0, 0).