PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Latent Reasoning for LLM-based Recommendation

Yang Zhang, Wenxin Xu, Xiaoyan Zhao, Wenjie Wang, Fuli Feng, Xiangnan He, Tat-Seng Chua

arXiv:2505.19092v2cs.AI

TL;DR

LLM recommendation methods need reasoning but are constrained by the cost of explicit CoT supervision and generation. LatentR3 replaces textual reasoning with compact latent representations optimized through staged SFT and RL, and experiments report improved performance with minimal inference overhead.

  • Problem

    Recommendation methods face difficult CoT-data collection and high inference latency from generating explicit textual reasoning.

  • Method

    LatentR3 learns compact latent reasoning end to end using SFT initialization followed by RL with a modified GRPO using perplexity-based continuous rewards and batch-relative advantages.

  • Results

    LatentR3 applied to D3 achieves superior performance across all metrics and three datasets, while the reported framework significantly improves LLM-based recommendation methods.

  • Takeaways & Limitations

    Latent reasoning is presented as a practical, latency-efficient alternative for integrating reasoning into LLM-based recommendation without explicit CoT data.

  • Takeaways & Limitations

    Experiments are limited to relatively small datasets, and latent reasoning is less interpretable than explicit CoT.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks, sparking growing interest in their application to preference reasoning in recommendation systems. Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data. However, these methods face significant practical limitations due to (1) the difficulty of obtaining high-quality CoT data in recommendation and (2) the high inference latency caused by generating CoT reasoning. In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning. This approach eliminates the need for explicit CoT generation and improves inference efficiency, as few latent tokens can effectively capture the entire reasoning process. Building on this idea, we propose \textit{\underline{R}einforced \underline{Latent} \underline{R}easoning for \underline{R}ecommendation} (LatentR$^3$), a novel end-to-end training framework that leverages reinforcement learning (RL) to optimize latent reasoning without relying on any CoT data. LatentR$^3$ adopts a two-stage training strategy: first, supervised fine-tuning to initialize the latent reasoning module, followed by pure RL training to encourage exploration through a rule-based reward design. Our RL implementation is based on a modified GRPO algorithm, which reduces computational overhead during training and introduces continuous reward signals for more efficient learning. Extensive experiments demonstrate that LatentR$^3$ enables effective latent reasoning without any direct supervision of the reasoning process, significantly improving performance when integrated with different LLM-based recommendation methods. Our codes are available at https://github.com/xuwenxinedu/R3.

1 INTRODUCTION

Existing recommendation methods use explicit CoT supervision, but obtaining reliable reasoning data and generating textual reasoning at inference are both difficult. LatentR3 addresses these constraints by learning compact latent reasoning with reinforcement learning and reports improved recommendation performance.

  • Recommendation methods commonly fine-tune LLMs with explicit textual CoT data.
  • Explicit CoT methods face high inference latency and costly, unreliable supervision because feedback usually contains only final outcomes and preferences are subjective.
  • Latent reasoning uses hidden representations to encode complex reasoning compactly, reducing the need for lengthy textual reasoning and improving inference efficiency.
  • LatentR3 learns latent reasoning without explicit CoT data through end-to-end reinforcement learning from weak final-feedback signals.
  • The proposed framework is reported to improve LLM-based recommendation performance without explicit CoT data.

2 PROBLEM DEFINITION

The recommendation task represents users, interaction histories, and next-item targets as textual inputs and outputs. LatentR3 introduces intermediate reasoning intended to support efficient recommendation without explicit reasoning annotations.

  • Each recommendation instance contains a user u, interaction history h, and next interacted item y.
  • Historical interactions are converted into a textual prompt x, which the LLM uses to generate a next-item recommendation.
  • The framework represents the LLM’s intermediate thoughts as reasoning output r and the final recommendation as predicted item ŷ.
  • LatentR3 aims to learn recommendation-oriented reasoning directly from recommendation data without explicit CoT annotations.

3 METHODOLOGY

LatentR3 combines a latent reasoning architecture with a two-stage SFT-then-RL training strategy for recommendation. Its LR-GRPO variant samples continuous latent reasoning, uses perplexity-based continuous rewards, and computes batch-relative advantages to improve training efficiency and exploration.

  • 3.1 LATENT REASONING ARCHITECTURE: LatentRATT extracts final-layer hidden states to generate autoregressive latent reasoning tokens aligned with the LLM input embedding space.The generated latent tokens are fed back into the LLM as reasoning inputs.
  • 3.1 LATENT REASONING ARCHITECTURE: After generating N latent reasoning tokens, the model concatenates them with the prompt for next-item prediction; N = 1 can achieve strong performance.N controls the length of the latent reasoning sequence.
  • 3.2 TRAINING: LatentR3 first applies SFT to initialize meaningful latent reasoning, then uses pure RL to explore more diverse reasoning paths.The SFT warm-up is intended to reduce instability and collapse risks associated with training RL from scratch.
  • 3.2.2 RL-BASED TUNING: LR-GRPO samples continuous latent reasoning via reparameterization, replacing textual outputs with latent vectors drawn around the original reasoning vector.The original latent reasoning is retained as the first sample, while additional samples add Gaussian noise controlled by σ.
  • 3.2.2 RL-BASED TUNING: LR-GRPO computes advantages against the batch-level average reward and updates the policy using a GRPO-like objective with KL regularization.Batch-relative comparison is used to avoid unreliable positive advantages when all samples in a group have low quality.
  • 3.2.2 RL-BASED TUNING: The modified reward uses negative perplexity on the ground-truth answer, avoiding full autoregressive answer generation and providing continuous rewards.Each sampled latent reasoning vector is evaluated by predicting the ground-truth tokens conditioned on the prompt and sampled reasoning.

4 EXPERIMENTS

Experiments evaluate LatentR3 across Amazon recommendation datasets, baselines, and controlled analyses. The method improves recommendation performance, benefits unpopular items, and achieves efficiency with compact latent reasoning.

  • Main results: LatentR3 applied to D3 outperforms all existing approaches across every metric on all three datasets.
  • Main results: 17.0% relative improvement for BIGRec and 8.4% for D3 show that LatentR3 substantially improves different LLM-based recommenders.
  • Main results: Un-tuned LLM recommenders perform substantially worse than traditional methods, while tuning enables LLM-based baselines to outperform traditional models in nearly all cases.
  • Efficiency: One latent token keeps inference cost close to non-reasoning baselines, whereas explicit CoT increases cost 25-fold on Toys and nearly 30-fold on Games.
  • Performance by item popularity: LatentR3 achieves significantly larger improvements on unpopular than popular items when compared with BIGRec.The analysis treats unpopular items as more challenging because existing methods already perform well on popular items.
  • Ablation study: Removing LatentRATT or RL reduces performance, while replacing batch-based advantage estimation with the original GRPO advantage performs comparably or worse than the SFT-only variant.These ablations were conducted on the Toys and CDs datasets.
  • Reasoning length: Increasing latent reasoning length from K=0 to K=2 improves performance, but the gain from K=1 to K=2 is much smaller than from K=0 to K=1.The reasoning-length experiments report NDCG@5 and HR@5 on Toys and CDs.

5 RELATED WORK

Related work places LLM-based recommendation into three paradigms: in-context learning, agent-based frameworks, and fine-tuning-based methods.

  • Existing LLM-based recommendation methods include in-context learning, agent-based frameworks, and fine-tuning-based approaches.

6 CONCLUSION

LatentR3 replaces explicit chain-of-thought reasoning with compact latent representations and combines architectural innovations with two-stage reinforcement learning. Experiments report improved performance with minimal additional inference cost, while the method remains limited by dataset scale and interpretability.

  • LatentR3 replaces explicit chain-of-thought reasoning with compact latent reasoning representations.
  • The framework integrates the LatentRATT layer with a two-stage reinforcement learning strategy for preference reasoning without costly CoT supervision.
  • LatentR3 significantly enhances existing LLM-based recommendation models while adding inference cost equivalent to generating one extra token.
  • Experiments are limited to relatively small datasets, and latent reasoning is less interpretable than explicit CoT.

ETHICS STATEMENT

The work states that it follows the ICLR Code of Ethics and uses anonymized, publicly available interaction data without personally identifiable information. It also notes the need for additional privacy protections in broader future applications.

  • The research does not involve human subjects or sensitive personal data.
  • User interaction data are anonymized, publicly available, and do not contain personally identifiable information.
  • Future broader applications should consider additional user privacy protections.

A DATASET PRE-PROCESSING

The experiments use Amazon domain-specific datasets and apply 5-core filtering within a dynamic temporal partitioning process. Dataset statistics are presented in Table 3.

  • The experiments use Amazon datasets from the Toys, CDs, Games, and Instruments domains.
  • Each subset is preprocessed to achieve 5-core data integrity.
  • A sliding time window processes data from October 2017 to October 2018 while iteratively applying 5-core filtering.
  • Table 3 reports dataset statistics.

B COMPARED METHODS AND IMPLEMENTATION DETAILS •

The evaluation targets top-N recommendation and compares traditional sequential models with LLM-based approaches. LatentR3 is implemented on BIGRec and D3 using specified training and decoding choices.

  • Compared Methods: The top-N recommendation evaluation compares traditional sequential models with state-of-the-art LLM-based recommendation approaches.
  • Compared Methods: Traditional baselines include Caser, GRU4Rec, and SASRec, while LLM-based baselines include Base, COT, AlphaRec, BIGRec, and D3.
  • Compared Methods: LatentR3 is implemented on both BIGRec and D3 because existing recommendation reasoning methods do not match the top-N setting or rely on explicit CoT supervision.
  • Implementation Details: LLM-based methods use Qwen2.5-1.5B as the backbone, with supervised fine-tuning performed using AdamW and selected learning rates.
  • Implementation Details: The implementation adopts constrained decoding for BIGRec after finding better performance than grounding-based decoding on the CDs dataset.

C LEARNING EXPLICIT REASONING WITHOUT COT SUPERVISION VIA RL

The paper compares reinforcement learning for explicit CoT reasoning without CoT supervision against baseline methods and LatentR3. Although explicit reasoning improves over the base LLM, COT-RL does not reliably improve performance and may degrade it.

  • COT and COT-RL both outperform the base LLM in the reported comparison.
  • Table 4 compares BIGRec performance under grounding and constrained decoding on the CDs dataset.
  • RL without direct CoT supervision fails to improve explicit reasoning performance and can even degrade it.The authors attribute this to the difficulty of learning long CoT chains and handling output-formatting requirements.

D THE ANALYSIS OF STATISTICAL SIGNIFICANCE

The statistical analysis evaluates LatentR3 against BIGRec under random sampling across five runs. The averaged results remain statistically significant in favor of LatentR3.

  • p < 0.001: LatentR3 remains significantly better than BIGRec when random sampling is enabled.The comparison uses averaged results across five experimental runs.
  • Table 6 reports t-test results for BIGRec and LatentR3 on the CDs dataset, with significance marked at p < 0.01.

E PERFORMANCE ON LARGER LLM

Additional analyses examine LatentR3 with larger language models, inference cost, item popularity, reasoning length, and the original GRPO algorithm. The results indicate strong gains with a larger model, near-baseline inference time, improved performance across item groups, diminishing returns from extra latent tokens, and substantially lower RL training cost than original GRPO.

  • E PERFORMANCE ON LARGER LLM: 40.7%: LatentR3 achieves an average performance gain on the CDs dataset with the larger Qwen2.5-3B model.The larger-model comparison uses LoRA-based tuning to control training expenses.
  • F INFERENCE EFFICIENCY: Inference time is almost identical to non-reasoning methods because LatentR3 adds only one latent token by default.The comparison uses 100 samples from each of four datasets on a single A100 GPU.
  • G PERFORMANCE ACROSS ITEM POPULARITY: LatentR3 performance is evaluated separately on popular and unpopular items across four datasets.Popular items are defined as the top 20% by training-set frequency.
  • H INFLUENCE OF REASONING LENGTH: Increasing latent reasoning length from K=0 to K=2 improves performance, but the gain from K=1 to K=2 is smaller.Experiments report NDCG@10, HR@10, NDCG@5, and HR@5 on Toys and CDs.
  • I COMPARISON WITH THE ORIGINAL GRPO: LatentR3 achieves comparable performance to original GRPO while reducing training cost to approximately one quarter.Original GRPO requires full-model tuning to reach the reported performance.
Loading 2505.19092v2…