Source-linked AI summary
Beyond Ranking Accuracy: Evaluating LLM-Cited Feature Rationales for Next Basket Repurchase Recommendation
Yanan Cao, Anay Dombe, Murali Mohana Krishna Dandu, Shreeranjani Srirangamsridharan, Sinduja Subramaniam, Yogananth Mahalingam, Evren Korpeoglu, Kannan Achan
TL;DR
The paper asks whether LLMs can rank next-basket repurchases and provide outcome-grounded rationales beyond ranking accuracy. It evaluates structured behavioral evidence with standalone scoring and cross-model feature masking, finding that LLMs trail supervised rankers while rationale quality can improve unevenly. The resulting evidence supports using LLMs as validated explanation components rather than primary rankers.
Problem
Next-basket recommendation emphasizes ranking accuracy, but users may also need concise evidence explaining why a previously purchased item is recommended now.
Method
The study evaluates LLM scores and cited features from structured repurchase evidence across three datasets, comparing rationale selections with attribution methods through cross-model feature masking.
Results
Off-the-shelf LLM scores consistently trail supervised rankers, while prompt and evidence changes improve outcome-grounded feature selection in some datasets without consistently matching attribution baselines.
Takeaways & Limitations
Supervised models should rank candidates, while LLMs can translate validated outcome-relevant features into concise explanations.
Takeaways & Limitations
Feature masking measures predictive sensitivity under a particular perturbation scheme, not causal effects or faithful representation of the LLM’s internal reasoning.
Abstract
from arXiv · showhide
Next-basket repurchase recommendation is commonly formulated as a ranking task: given a customer's purchase history, the system ranks previously purchased items that may be needed again. In production settings, however, ranking accuracy is only one component of recommendation quality. Customers may also benefit from concise evidence about why an item is recommended now. Large language models (LLMs) offer a potential way to surface such evidence through feature-based, human-readable rationales grounded in interpretable behavioral signals. We construct repurchase features spanning cadence, frequency, recency, user behavior, and item popularity, and evaluate LLMs on two public grocery datasets and one proprietary retail dataset. We investigate (1) whether off-the-shelf LLMs can use these features as next-basket scorers relative to heuristic and supervised rankers, and (2) whether LLM-cited features carry outcome-grounded ranking signal. For the latter, we compare LLM-cited features with model-specific attribution methods under a cross-model feature-masking protocol that measures ranking degradation after masking selected features. Our results show that LLM scores are not competitive with supervised rankers, suggesting that off-the-shelf LLMs should not be used as standalone repurchase recommenders. However, changes in prompt and evidence representation can improve outcome-grounded feature-masking results in some settings even when ranking performance does not improve; the effect is dataset-dependent and does not consistently match attribution baselines. These findings suggest a practical role for LLMs as validated explanation components rather than primary rankers, with rationale quality evaluated separately from ranking accuracy.
1. Introduction
The paper evaluates whether LLMs can score repurchase candidates and select outcome-relevant behavioral evidence, treating explanation quality as distinct from ranking accuracy.
- Repurchase recommendations need evidence about purchase timing, frequency, and current need in addition to ranking accuracy.
- LLMs are evaluated as feature-based scorers and rationale generators using interpretable signals such as cadence, frequency, recency, user behavior, and popularity.
- Fluent rationales may cite plausible features that are not relevant to future repurchase outcomes.
- The framework converts purchase histories into structured features, then evaluates scores with ranking metrics and rationales with cross-model feature masking.
- RQ1 compares off-the-shelf LLM scoring with heuristic and supervised rankers, while RQ2 tests whether cited features remain outcome-relevant across model classes.
- The contributions position rationale evaluation as separate from ranking quality and compare LLM-cited features with attribution methods across target model families.
2. Related Work
Prior explainable-recommendation work uses textual, attribute-based, collaborative, and interaction evidence, while this paper studies directly selected structured behavioral features for repurchase explanations.
- 2.1. Explainable Recommendation: Explainable recommendation aims to provide rationales that help users understand why an item is recommended.
- 2.1. Explainable Recommendation: Repurchase explanations commonly rely on frequency, recency, cadence, inter-purchase intervals, and reorder behavior as structured predictive signals.
- 2.2. LLM-Based Recommendation Explanations: Recent LLM recommendation systems generate fluent and personalized explanations, often alongside an existing recommender model.
- 2.3. LLMs as Interfaces to Existing Feature Attributions: In prior interfaces to feature attributions, LLMs primarily verbalize explanations produced by external attribution methods.
- 2.3. LLMs as Interfaces to Existing Feature Attributions: This paper instead asks whether an LLM can independently select behavioral evidence that remains relevant across tree-based and neural recommendation models.
- 2.4. Rationale Groundedness and Feature Attribution: The evaluation treats LLM-cited features as rationales and tests their ranking signal through a common cross-model feature-masking protocol rather than assuming one attribution method is ground truth.
3. Methodology
The methodology represents each customer-item pair with pre-target structured features, compares supervised and LLM scoring functions, and evaluates LLM outputs as both scores and feature rationales.
- 3.1. Task Formulation: Each candidate is a previously purchased item, represented by a feature vector built only from information available before the target basket.
- 3.1. Task Formulation: A supervised ranker maps structured features to a score, while an LLM reads a natural-language evidence card constructed from the same features.
- 3.2. Feature Construction: All models use 28 interpretable engineered features to provide a controlled interface instead of requiring the LLM to extract temporal patterns from raw purchase sequences.
- 3.3. Evidence Representation and Prompting: The base evidence card presents raw feature values, whereas the norm-prompt variant adds percentile representations relative to customer-specific or global comparison sets.
- 3.1. Task Formulation: The LLM outputs a behavioral support score, exactly three cited feature names, and a natural-language explanation without observing the supervised score or future label.
- 3.3. Evidence Representation and Prompting: Prompt and evidence variants are selected on a separate validation set using NDCG@5, then fixed for held-out evaluation without exposing future labels during inference.
4. Experimental Setup
The evaluation spans three repurchase datasets and compares LLM scorers with heuristic and supervised rankers, while testing cited features through cross-model masking gaps.
- Datasets: The study evaluates two public grocery datasets and one proprietary retail dataset, with candidates drawn from each customer’s purchase history.Features use only information preceding the target basket, whose items provide repurchase labels.
- Datasets: LLM scoring sets contain 100 held-out customers per dataset because LLM inference is costly.Supervised rankers use customer-level splits to prevent user leakage.
- Ranking evaluation: RQ1 compares LLM support scores with a personal-frequency baseline, XGBoost, and a vanilla neural network using the same 28 features.Ranking quality is measured with Precision@k, Recall@k, and NDCG@k.
- Feature-rationale evaluation: For RQ2, selectors provide exactly three features, enabling equal-budget comparisons among LLM citations, TreeSHAP, integrated gradients, and random selection.The attribution methods include both target-model-specific and cross-family baselines.
- Feature-rationale evaluation: Selected features are replaced by their training-set medians, and the resulting masking gap measures degradation in ranking quality.Larger gaps indicate that selected features carry more ranking signal for the target model, tested across XGBoost and VNN.
5. Results
Supervised rankers outperform off-the-shelf LLM scorers, while LLM-cited feature quality varies by dataset and can diverge from ranking performance. On Instacart, generated explanations accurately reflect cited evidence, but selecting outcome-grounded features remains harder.
- RQ1: LLM Scores as Ranking Signals: Across all three datasets, XGBoost and VNN achieve the strongest ranking results, remaining substantially more effective than off-the-shelf LLM scorers.Table 2 reports recommendation quality on the 100-user LLM scoring sets.
- RQ1: LLM Scores as Ranking Signals: On Instacart, GPT-4o beats the personal-frequency baseline on most metrics but remains below supervised models; on DC and proprietary data, LLM scorers generally trail that baseline.LLMs therefore extract some structured-feature signal without consistently converting it into effective candidate rankings.
- RQ1: LLM Scores as Ranking Signals: Prompt and evidence refinement does not improve GPT-4o ranking performance, suggesting percentile features and prompt changes alone are insufficient for competitive standalone ranking.The model may identify salient features for individual pairs while struggling to calibrate comparable scores across candidates.
- RQ2: Outcome-Grounded Feature Rationales: Feature-masking results are inconsistent: norm-prompt GPT-4o improves over base GPT-4o on 7 of 18 Instacart metrics and all 18 proprietary metrics, but exceeds the cross-family baseline on only 10 and 4 metrics, respectively.The effect of percentile-based evidence is dataset-dependent and does not provide a general solution.
- RQ2: Outcome-Grounded Feature Rationales: On DC, LLM-selected features produce near-zero or negative gaps on the XGBoost target, while norm-prompt improves over base GPT-4o on only 8 of 18 metrics.Model-specific attribution methods still identify substantial ranking signal in this setting.
- Text-to-evidence groundedness: On Instacart, generated explanations achieve reason-group precision of 0.951 and numeric consistency of 0.996, but recall is 0.740.The text usually represents selected evidence accurately, although some cited evidence is omitted.
6. Discussion and Limitations
Repurchase explanations should foreground interpretable behavioral evidence, but ranking quality, rationale quality, and textual fluency are distinct evaluation axes. The proposed hybrid role for LLMs is constrained by dataset-dependent evidence effects, masking limitations, automatic text evaluation, limited sample coverage, and untested user impact.
- Repurchase explanations should emphasize cadence, frequency, recency, inter-purchase intervals, and reorder behavior rather than product semantics alone.
- Ranking quality and rationale quality can diverge, so evaluating only ranking or textual fluency can mischaracterize explanation value.
- Percentile-based evidence views may improve feature selection in some settings, but effects are dataset-dependent and do not improve ranking performance.
- A constrained hybrid design uses supervised models for ranking, feature-level evaluation for validation, and LLMs for concise explanations.
- Feature masking measures predictive sensitivity under a perturbation scheme, not causal importance or faithful representation of LLM reasoning.
- Text-to-evidence analysis is automatic and limited to Instacart, and the experiments do not establish user trust, decision benefits, or superiority over templates.
- Only 100 customers per dataset were evaluated, and the explored prompt and evidence variants were not exhaustive.
- Results may differ for other LLMs, ranking architectures, feature sets, or recommendation domains, motivating broader studies and controlled user evaluations.
7. Conclusion
The conclusion finds that off-the-shelf LLMs are weaker standalone repurchase scorers than supervised rankers, while their cited rationales sometimes contain outcome-relevant signal. These findings support validating evidence before using LLMs to verbalize user-facing explanations.
- Off-the-shelf LLM scores consistently trail supervised rankers across two public grocery datasets and one proprietary retail dataset.
- Providing structured behavioral evidence is insufficient to make LLMs reliable standalone repurchase recommendation models.
- LLM-cited features sometimes carry outcome-relevant ranking signal, while percentile-based evidence can improve feature selection in some settings.
- These gains are dataset-dependent and do not consistently match or exceed cross-family attribution baselines.
- On Instacart, generated explanations closely reflect cited structured evidence, with high reason-group precision and numeric consistency.
- The supported design has supervised models rank items, feature-level evaluation validate evidence, and LLMs translate it into concise rationales.
Declaration on Generative AI
Generative AI was used in the evaluated system and for limited writing assistance, while experimental results and evaluation outcomes were not altered by generative AI.
- Gemini 2.5 Pro and GPT-4o produced feature-based scores, cited rationales, and textual explanations from structured behavioral evidence cards.
- Generative AI was also used for limited language polishing and grammar correction, but not to alter results, tables, or evaluation outcomes.
A. Prompt Template
The prompt frames the LLM as a repurchase analyst for one shopper–product pair, requiring a bounded score, three evidence features, and two explanation forms. It evaluates behavioral dimensions using actual evidence values while handling conflicts and selection constraints explicitly.
- The LLM receives a structured evidence card for one shopper–product pair and estimates next-order repurchase likelihood.
- It evaluates cadence, frequency, recency, user context, and item popularity before identifying three strongest individual features.
- The output contains a 0–1 support score, exactly three valid feature names, a customer-facing explanation, and analyst-facing reasoning.
- Evidence Evaluation Guidance: Cadence assesses regularity and timing, while N/A cadence values are treated as insufficient evidence.
- Evidence Evaluation Guidance: Frequency, recency, and user context assess purchasing habits, elapsed time, and how shopper behavior changes item-level interpretation.
- Evidence Evaluation Guidance: Item popularity is supporting evidence rather than a substitute for shopper-specific evidence.
- Evidence Evaluation Guidance: Conflicting evidence should be reflected in the score, such as frequent purchase combined with very recent purchase.
- Feature Selection Rules: Feature selection uses exact valid names, observed values, material score effects, and interpretable nonredundant features without automatically favoring dimensions.