Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation

1 INTRODUCTION

• Figure 1 contrasts direct prediction of user feedback with an objective that adds explicit reasoning about user preferences before prediction.The distinction is the placement of preference reasoning before feedback generation.Source: Figure 1 · p. 1, §1 · p. 1
• Existing RecLLMs can fail in complex cases because direct-feedback objectives encourage pattern-based predictions susceptible to data biases and spurious correlations.The introduction links this objective to difficulty generalizing to low-frequency or cold-start items.Source: §1 · p. 1
• Deliberative Recommendation trains LLMs to reason about user preferences and item features before predicting feedback, using verbalized feedback as supervision for personalized reasoning.The paper adopts freely available user-generated rationales, such as reviews or conversations, as reasoning guidance.Source: §1 · pp. 1–2
• Reason4Rec uses three collaborative experts with tailored training strategies, and experiments on three datasets validate its effectiveness for prediction accuracy and reasoning quality.The experts address Preference Distillation, Preference Matching, and Feedback Prediction.Source: §1 · p. 2

2 RELATED WORK

• The paper frames its focus as deliberative recommendation, connecting LLM-based recommendation, deliberative alignment, and review-based recommendation.The related-work overview names these three adjacent areas.Source: §2 · p. 2

2.1 LLM-based Recommendation

• In-context reasoning methods use strategies such as chain-of-thought and self-reflection, while fine-tuned approaches often compress reasoning into one step and lack fine-grained preference supervision.Reason4Rec instead decomposes reasoning into multiple steps and uses verbalized feedback to align reasons with user preferences.Source: §2.1 · p. 2

2.2 Deliberative Alignment

• Deliberative alignment differs because recommendation preferences are highly personalized and diverse, and numerical feedback or pairwise preferences alone are insufficient to supervise reasoning.The paper proposes verbalized user feedback as reasoning supervision.Source: §2.2 · p. 3

2.3 Review-based Recommendation

• Review-based systems commonly encode reviews into user and item embeddings, whereas Reason4Rec produces explicit textual reasoning and also uses reviews to supervise its outputs.The paper contrasts implicit review representations with interpretable, step-wise text.Source: §2.3 · p. 3

3 TASK FORMULATION

• Deliberative recommendation requires reasoning about user preferences and item features before predicting feedback, with user reviews providing fine-grained preference guidance for rating prediction.The paper studies the classic rating-prediction task.Source: §3 · p. 3
• For each user-item pair, the input includes the user's and item's historical interactions, and the LLM produces both a reasoning process and a predicted rating.Historical interactions are represented by paired ratings and reviews associated with users and items.Source: §3 · p. 3
• The target-item review supervises reasoning during training but is excluded during testing to prevent data leakage.The paper notes that explicit reasoning may increase prediction time and computational cost, while potentially improving accuracy, reliability, and interpretability.Source: §3 · p. 3

4.1 Summarizer for Preference Distillation

• The summarizer distills historical reviews into aspect-preference summaries of user preferences and item features, which can be stored offline to reduce repeated processing.Teacher-generated summaries provide its supervised fine-tuning targets.Source: §4.1 · p. 4
• Summaries generated by the three tested models had average semantic similarity of 0.92, indicating that teacher-model choice had little impact on this step.Source: §4.1 · p. 4

4.2 Reasoner for Preference Matching

• The reasoner matches user and item summaries, while a generation-then-filter strategy selects review-aligned reasons for its fine-tuning.ChatGPT generates candidate reasons; a reward model filters them using the target-item review, with the user’s rating supplied as a hint after initial generation.Source: §4.2 · p. 4

4.3 Predictor for Feedback Prediction

• The predictor combines summaries from user and item histories, historical ratings, and the reasoner’s output, then learns to predict the target rating.The prompt includes average historical ratings, and supervised fine-tuning aligns predicted scores with the target rating.Source: §4.3 · p. 5

### Average Past Ratings ###

• Logit-weighted decoding converts rating-token probabilities into a continuous expected rating and avoids decoding the predictor’s fixed output prefix.The method weights each possible rating by its probability; restricting decoding to the rating token reduces inference time and computational cost.Source: ### Average Past Ratings ### · p. 5

4.4 Reward Model for Reason Judgment

• A reward model scores reasons for alignment with the target-item review, and a threshold selects high-quality reasons while balancing their quantity and quality.It is trained to predict user ratings from the target review; its training data is separated from the reasoner’s to reduce rating prediction by memorization.Source: §4.4 · pp. 5–6
• The reward model compares rating-prediction errors from a generated reason and the target review against the ground-truth rating.The evaluation score measures their prediction difference; a threshold determines which reasons are treated as high quality.Source: §4.4 · p. 6

4.5 Overall Workflow of Reason4Rec

• The workflow trains a summarizer, reasoner, and predictor sequentially, then uses their outputs and historical rating statistics to predict ratings.The summarizer learns from teacher-generated summaries; filtered reasons supervise the reasoner, and ground-truth ratings supervise the predictor.Source: §4.5 · p. 6

• The experiments ask how Reason4Rec compares with baselines in rating-prediction accuracy and generated-reason quality.Source: §5 · p. 6
• They examine why Reason4Rec’s multi-step reasoning strategy is essential and how it compares with one-step reasoning.Source: §5 · p. 6
• They evaluate the reason-generation method and reward model for incorporating user-verbalized preference feedback.Source: §5 · p. 6

5.1.1 Datasets.

• Experiments use all Music interactions and recent subsets of Book and Yelp, with timestamp-based 8:1:1 train, validation, and test splits and 5-core filtering.The Book subset covers the last two months and Yelp the last six months; test interactions occur after training and validation interactions.Source: §5.1.1 · p. 6

5.1.2 Evaluation Metrics.

• Rating prediction is evaluated with MAE and RMSE, while BLEURT and GPTScore measure semantic alignment between generated reasoning and target-item reviews.GPTScore, computed using GPT-4o, ranges from 0 to 100, with higher scores indicating better alignment.Source: §5.1.2 · p. 6

5.1.3 Baselines.

• Baselines include review-based recommendation methods and LLM-based methods, including GPT-4o, Deepseek-R1, Rec-SAVER, and EXP3RT.Source: §5.1.3 · pp. 6–7

5.1.4 Implementation Details.

• Fine-tuned LLM methods use Llama3-8B with QLoRA and the same 12,000 sampled user-item pairs; reward-model training uses 8,000 non-overlapping pairs.QLoRA updates about 45M parameters, less than 1% of the 8B model, and reduces GPU memory overhead by more than 70% versus full-parameter fine-tuning.Source: §5.1.4 · p. 7

5.2 Main Results (RQ1)

• Reason4Rec reduces MAE by approximately 0.02 and RMSE by approximately 0.01 versus traditional methods on all three datasets.The authors attribute these results to reasoning about user preferences and item features before prediction.Source: §5.2 · p. 7
• Reason4Rec achieves the best RMSE and MAE against reasoning-enhanced methods on all three datasets, except Music MAE, where DeepSeek-R1 is lower.DeepSeek-R1’s RMSE is more than 0.1 higher, according to the authors.Source: §5.2 · p. 7
• Reason4Rec’s generated reasons outperform all baselines on GPTScore and BLEURT.The authors report that this indicates stronger alignment with users’ true preferences.Source: Table 2 · p. 8, §5.2 · p. 8
• Reason4Rec improves Preference Alignment over DeepSeek-R1 by 0.18 points in human evaluation; both methods score above 4.0 in Reasonability and Persuasiveness.Three independent judges rated 100 cases from the Books and Music datasets on a 0–5 scale.Source: Table 3 · p. 8, §5.2 · p. 8
• Reason4Rec achieves the best NDCG@5 and Precision@5 on both Music and Book ranking tasks.The test set pairs items rated at least 4 with randomly sampled unseen items in candidate sets of 20.Source: Table 4 · p. 8, §5.2 · p. 8

5.3.1 Impact of Multi-step Reasoning (RQ2)

• Removing any expert raises MAE and RMSE on Music and Book; removing the Predictor raises both significantly, while removing the Reasoner particularly increases MAE.The ablations remove or combine the Summarizer, Reasoner, and Predictor steps.Source: §5.3.1 · pp. 8–9, Figure 3 · p. 9
• Both one-step reasoning and single-prompt CoT increase errors on Music and Book; CoT has a 2× larger test-set GPTScore drop than Reason4Rec.The authors report comparable training-set reasoning quality between CoT and Reason4Rec.Source: Figure 4 · p. 9, §5.3.1 · pp. 9–10, Table 6 · p. 9

5.3.2 Effect of Reason Generation Approaches (RQ3)

• All four alternative reason-generation approaches underperform Reason4Rec on Music and Book; review-inferred reasons do worse than using no reason, and history-inferred reasons do worse on most metrics.The authors suggest target-review-derived reasons may rely on preferences absent from historical interactions; unrefined GPT-generated history reasons are also ineffective.Source: Table 7 · p. 10, §5.3.2 · p. 10

5.3.3 Effectiveness of Reward Model (RQ3)

• Reward-model rating-prediction accuracy ranks reason types in the same order as their known similarity to target-item reviews.The tested reason types range from Review-Guided through Review-Inferred and Hint-Inferred to History-Inferred.Source: Table 8 · p. 10, §5.3.3 · p. 10

5.3.4 Impact of Reward Threshold

• On Yelp, τ=0.02 leaves too little training data, while τ=0.10 compromises reason quality; the reported optimal τ differs by dataset: 0.08 for Yelp and 0.2 for Book.The authors recommend retaining about 50% of candidate reasons initially, then searching locally on validation data.Source: §5.3.4 · pp. 10–11, Table 9 · p. 11

5.3.5 Impact of Interaction Volumes.

• Increasing history from five to ten interactions significantly improves accuracy; beyond ten, MAE and RMSE changes are limited to 0.004 and 0.008.The authors suggest ten interactions may be adequate on Music and Book.Source: Figure 6 · p. 11, §5.3.5 · p. 11

5.3.6 Inference Cost Analysis

• Reason4Rec averages about 6 seconds and 148 generated tokens per prediction, with comparable inference time and token counts to Rec-SAVER and EXP3RT but fewer tokens than Deepseek-R1.The Summarizer is preprocessed, and the Predictor decodes only one token in the final step.Source: Table 10 · p. 11, §5.3.6 · p. 11

5.3.7 Case Study

• In the long-tail case, Reason4Rec identifies a mismatch between user preferences and item features and predicts dislike despite the user’s mostly five-star history.The target item has only one historical interaction; direct prediction instead relies on the user’s rating pattern and misses review-content preferences.Source: Figure 7 · p. 11, §5.3.7 · p. 11

6 CONCLUSION AND FUTURE WORK

• Across three real-world datasets, Reason4Rec shows superior prediction accuracy and reasoning quality using three collaborative experts for Preference Distillation, Preference Matching, and Feedback Prediction.The framework aligns its reasoning process with users’ true preferences using reviews.Source: §6 · pp. 11–12
• Reason4Rec uses only reviews as verbalized feedback; future work proposes adaptive history selection and graph-based interaction histories for richer reasoning signals.The paper also suggests relational correlations, periodic events, and structure augmentation for handling islanded nodes.Source: §6 · p. 12
• Reason4Rec has comparable or lower inference costs than Rec-SAVER and EXP3RT, but the authors identify efficiency as an area for improvement.They propose exploring LLM acceleration algorithms tailored to Deliberative Recommendation.Source: §6 · p. 12
• Reason4Rec currently predicts user feedback; future work could use its explicit preference reasoning to construct more interpretable user simulators.Source: §6 · p. 12
• The authors propose deployment on real-world recommendation platforms such as OpenP5 to explore practical integration and support broader evaluation.Source: §6 · p. 12