Source-linked AI summary
Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
Ante Kapetanovic, Tomislav Duricic, Andro Mercep, Emanuel Lacic
TL;DR
LLM reranker gains in conversational recommendation are difficult to interpret when retrieval and inference protocols vary. The paper compares rerankers and baselines in controlled ReDial pipelines, finding that candidate generation, scoring policy, pool size, and decoding materially shape both measured effectiveness and stability. It therefore argues that these protocol choices should be reported alongside headline results.
Problem
LLM-based conversational recommender comparisons often under-report candidate generation, candidate access, scoring policy, and decoding, despite their sensitivity to these choices.
Method
The paper evaluates proprietary, open-weight, and fine-tuned LLM rerankers against CF and sequential baselines in a shared ReDial retrieve-then-rerank pipeline while varying pool size, retriever, and temperature.
Results
Under a matched semantic top-250 pool, Claude Opus 4.6 reaches NDCG@10 of 0.1497 versus 0.0939 for EASE, while open-weight rerankers remain below EASE; retrieval and decoding choices also substantially alter outcomes.
Takeaways & Limitations
Candidate generation, candidate-pool size, scoring policy, and decoding configuration should be treated as core experimental variables and reported with LLM CRS results.
Takeaways & Limitations
The experiments cover a single movie-domain benchmark, ReDial, with a 6,924-item catalog and do not systematically measure cost or latency.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as rerankers in conversational recommender systems, yet measured gains depend strongly on the retrieval and inference protocol. On the ReDial conversational movie recommendation benchmark, we compare proprietary, open-weight, and fine-tuned LLM rerankers with collaborative-filtering and sequential baselines in a shared retrieve-then-rerank pipeline. We vary candidate-pool size, first-stage retriever, and decoding temperature. With a shared semantic top-250 candidate pool and strict candidate-aware scoring, the best proprietary reranker reaches NDCG@10 of 0.1497, compared with 0.0939 for the strongest non-LLM baseline. The same reranker reaches 0.2925 in zero-shot generation, showing that unconstrained scoring can yield a much larger apparent advantage than matched-pool evaluation. No evaluated open-weight LLM outperforms the tuned shallow autoencoder baseline under this protocol. For the strongest proprietary and open-weight rerankers, switching from semantic to collaborative-filtering candidates raises NDCG@10 by more than 50%, showing that measured reranker performance is highly sensitive to candidate generation. For the best proprietary reranker, raising temperature from 0 to 1.0 increases top-10 Jaccard distance from 0.0900 to 0.1240 while mean NDCG@10 changes negligibly, whereas weaker LLMs show larger degradation. These ReDial results support treating candidate generation, candidate-pool size, scoring policy, and decoding configuration as required reporting fields rather than implementation details.
1 Introduction
The paper argues that reported LLM gains in conversational recommendation depend on retrieval, candidate-pool, scoring, and decoding choices. It studies these factors in a shared ReDial retrieve-then-rerank pipeline.
- Motivation: Reported LLM gains depend on which candidates are retrieved, how many are shown, whether scoring is candidate-restricted, and how decoding is configured.These protocol choices can move ranking quality by more than the gap separating top systems.
- Motivation: LLM reranking comparisons are rarely controlled for candidate generation, while stochastic decoding can make outputs vary with repeated sampling and candidate order.The paper frames list-level stability as an under-reported complement to accuracy.
- Study design: The study evaluates proprietary, open-weight, and fine-tuned LLM rerankers against collaborative-filtering and sequential rerankers on ReDial.All models are compared within a shared two-stage pipeline while candidate-pool size, retriever, and temperature vary.
- Headline findings: Under the matched semantic top-250 pool, Claude Opus 4.6 reaches NDCG@10 of 0.1497 versus 0.0939 for EASE, while all evaluated open-weight rerankers fall below EASE.Four proprietary models significantly outperform EASE after Holm correction.
2 Methodology
The methodology evaluates rerankers in a controlled two-stage pipeline with configurable candidate pools, retrievers, scoring constraints, and decoding temperatures. ReDial dialogues, catalog items, retrieval diagnostics, and repeated generations support both effectiveness and stability analysis.
- Dataset and inputs: The evaluation uses ReDial’s standard test split of 1,025 dialogues and a 6,924-movie catalog, with accepted recommendations masked as targets.Each model produces ten ranked movie titles from family-specific inputs.
- Dataset and inputs: LLM prompts combine liked titles, the masked dialogue, and, except in zero-shot generation, a shuffled candidate list; cK denotes K candidates, c0 none, and cAll the full catalog.Candidate-aware prompts restrict outputs to listed titles.
- Two-stage pipeline: A candidate generator returns a pool C_u^K, after which a reranker orders that pool and emits the top-10 recommendation list.CF and sequential models can rerank semantic pools or generate top-K pools for LLM reranking.
- Candidate generation: Content-based filtering generates the primary pools by embedding movie metadata with all-mpnetbase-v2 and ranking items by a liked-minus-disliked preference score.The main comparison uses K=250, with K∈{500,1000} and full-catalog reranking for sensitivity analysis.
- Rerankers: The study compares proprietary, open-weight, and fine-tuned LLMs with CF, sequential, unreranked CBF, and popularity baselines.The fine-tuned reranker is Qwen2.5-7B-Instruct adapted with LoRA.
- Metrics and stability: NDCG@10 is primary, while Hit@10, item coverage@10, popularity@10, CandRecall@250, Oracle NDCG@10, bootstrap intervals, and corrected paired tests provide additional evaluation.Temperature experiments use 20 generations per prompt across temperatures {0, 0.5, 1.0, 2.0} on a fixed 30-prompt c250 subset, measuring Jaccard distance@10 and position disagreement@10.
3 Results
The results show that proprietary LLMs lead under a fixed semantic candidate pool, while retrieval strategy and decoding temperature substantially shape measured effectiveness and stability.
- 0.1497 NDCG@10 is Claude Opus 4.6’s score under the matched semantic top-250 pool, versus 0.0939 for EASE; every open-weight reranker falls below EASE.
- 57–89% is the proprietary NDCG@10 increase from c250 to cAll, while zero-shot Opus reaches 0.2925 compared with 0.2644 for cAll.Pool expansion changes target availability, candidate composition, prompt length, and ranking difficulty simultaneously.
- 52% and 59% are the NDCG@10 gains for Claude Opus 4.6 and Llama-3.3-70B when semantic candidates are replaced with EASE candidates.Retriever choice changes downstream performance even when candidate-pool size and reranker are held fixed.
- 28.8% of relevant items enter semantic top-250 pools, compared with 48.6% for EASE and 50.9% for SASRec.EASE yields the best NDCG for both tested rerankers, while SASRec has the highest recall and oracle ceiling.
- 0.0900 to 0.1240 is Claude Opus 4.6’s Jaccard distance@10 increase from temperature 0 to 1.0, while mean NDCG@10 changes little.Llama-3.3-70B’s Jaccard distance@10 rises from 0.0230 to 0.7600 as temperature reaches 2.0, alongside an NDCG@10 decline.
4 Discussion and Conclusion
The discussion argues that retrieval and evaluation protocol can affect measured LLM recommender gains as much as model choice, while accuracy and stability respond differently to decoding. The study therefore supports reporting pipeline details and notes that its evidence is limited to one movie-domain benchmark without systematic cost or latency measurement.
- Retrieval and evaluation: 52–59% NDCG@10 gains follow switching first-stage retrievers across both tested rerankers, showing that candidate generation materially shapes downstream quality.The higher-recall SASRec pool nevertheless trails EASE after reranking, indicating that candidate availability, composition, and reranker–pool compatibility jointly matter.
- Retrieval and evaluation: 0.2925 versus 0.1497 illustrates how zero-shot scoring and strict candidate-aware scoring can produce different headline LLM gains.The comparison does not isolate candidate access from scoring policy because both differ between c0 and c250.
- Stability: Mean NDCG@10 can remain nearly unchanged as temperature increases while Jaccard distance@10 and position disagreement@10 rise.For Llama-3.3-70B, Jaccard distance@10 increases from 0.0230 to 0.7600 while NDCG@10 falls from 0.0750 to 0.0490.
- Scope and limitations: The experiments use ReDial, a single movie-domain benchmark with a 6,924-item catalog, and do not systematically measure cost or latency.Future evaluations are proposed for larger catalogs, other domains, conversational settings, and joint quality, cost, and latency comparisons.
- Reporting implications: LLM-based CRS evaluations should report the retriever, candidate-pool size, scoring policy, decoding configuration, prompt template, and exact model and provider identifier.The paper argues that pipeline choices can affect measured gains as much as model choice.
GenAI Usage Disclosure
The paper discloses using generative AI tools for code implementation, data analysis, LaTeX editing, grammar checking, and related-work identification. The authors state that they reviewed and verified all AI-assisted outputs and retain responsibility for the work.
- Generative AI tools supported code implementation, data analysis, LaTeX editing, grammar checking, and related-work identification.
- The authors reviewed and verified all AI-assisted outputs and take full responsibility for the work.