PaperlayerView exact revision on arXiv

Source-linked AI summary

EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment

Haokai Ma, Aoqi Hu, Yueao Xing, Ruobing Xie, Yonghui Yang, Teng Tu, Lei Meng, Tat-Seng Chua

arXiv:2608.14011v1cs.IRcs.AI

TL;DR

Generative recommendation has not fully exploited future behaviors as sequentially dependent supervision across horizons. EchoRec addresses this with chained multi-token prediction and cycle-consistent holistic-preference alignment, achieving effectiveness across three real-world datasets and coherent multi-item generation.

  • Problem

    Existing generative recommenders leave unresolved how to exploit future behaviors as sequentially dependent supervision across horizons and how to verify holistic-preference alignment.

  • Method

    EchoRec chains horizon-aware multi-token prediction branches and uses cycle-consistent projectors to align their predictions with holistic preference.

  • Results

    EchoRec demonstrates effectiveness on three real-world datasets and naturally acquires coherent, non-redundant multi-item generation ability.

  • Takeaways & Limitations

    Sequentially dependent future-behavior supervision and cycle-consistent alignment provide a framework for capturing horizon-aware preferences in generative recommendation.

  • Takeaways & Limitations

    One-way alignment can absorb discrepancies between subspaces while leaving the decoding representation unaligned, motivating cycle-consistent verification.

Abstract

from arXiv · show

Generative recommendation autoregressively generates the semantic IDs of the target item, unifying preference modeling and index retrieval within the shared token space. Recent attempts have introduced Multi-Token Prediction (MTP) into this field, yet they primarily inherit its efficiency merit, leaving its potential as dense supervision unexplored. Unlocking this potential hinges on whether future behaviors qualify as informative supervision. Our analysis reveals that future behaviors carry a semantic echo of the current one far above that of random pairs, which nevertheless decays along horizons under intent transitions, making them informative yet order-dependent signals. Motivated by this, we propose EchoRec, which empowers MTP with cycle-consistent holistic preference alignment across multi-horizon for generative recommendation. It comprises two synergistic modules. Horizon-aware Preference Generation (HPG) sequentially chains lightweight auxiliary branches upon the base recommender, where each branch conditions on its predecessor to respect preference evolution. Verifiable Holistic-Preference Alignment (VHA) further consolidates them into the holistic preference and echoes it back through cycle-consistent projectors to suppress spurious alignment, with theoretical guarantees that exclude the rank-collapse form of spurious alignment under an invertible transport, enabling the holistic preference to be retained in the decoding representation. All auxiliary components serve as disposable scaffolding discarded at inference, introducing negligible online serving overhead. Extensive experiments on three datasets demonstrate the superiority of our EchoRec, together with its naturally acquired multi-item generation ability. Our code and datasets will be available upon acceptance.

1 Introduction

Generative recommendation unifies preference modeling and item retrieval by autoregressively generating semantic IDs, while recent MTP-based approaches mainly target efficiency or expressiveness. EchoRec instead uses sequentially dependent multi-horizon supervision and cycle-consistent holistic-preference alignment to address unresolved future-behavior and spurious-alignment challenges.

  • Generative Recommendation: Generative recommendation tokenizes items into discrete semantic IDs and generates target items token by token, unifying preference modeling with index retrieval in a shared token space.Semantically related items are anchored to overlapping semantic IDs, supporting recommendation in sparse scenarios.
  • Open Challenges: Existing GR attempts leave unresolved how to exploit future behaviors as sequentially dependent supervision across horizons and how to verify holistic-preference alignment.The introduction identifies future-behavior dependence across horizons and one-way alignment as fundamental challenges.
  • Open Challenges: One-way projector alignment can absorb discrepancies between representation subspaces, producing spurious alignment while leaving the representation oblivious to holistic preference.The paper states that no existing GR study offers a mechanism to rule out this failure mode.
  • EchoRec: EchoRec empowers MTP with cycle-consistent holistic-preference alignment across multiple horizons, propagating preferences forward as sequential supervision and echoing them back for verification.Its Horizon-aware Preference Generation module extends the framework to sequentially dependent future-horizon supervision.
  • Evaluation: Experiments on three real-world datasets evaluate EchoRec through multi-horizon prediction, component ablation, and efficiency analyses targeting robustness, universality, and design superiority.The study reports effectiveness and examines the contribution and efficiency of each component.

2 Related Works

Generative recommendation represents items with shared discrete semantic IDs and generates target items conditioned on them. Prior work derives these IDs from content, discretizes item representations, or aligns behavioral and semantic information.

  • Generative Recommendation: Generative recommenders assign each item shared discrete tokens, called semantic IDs, to support target-item generation.Semantic IDs discretize item semantics into a shared token space.
  • Generative Recommendation: VQ-Rec and SeeDRec represent items using transferable codes and sememes derived from content information.These methods use content-based tokenization schemes for item representation.
  • Generative Recommendation: TIGER discretizes item representations into semantic IDs with RQ-VAE and predicts target items using a Transformer-based generator.The method combines residual vector quantization with autoregressive generation.
  • Generative Recommendation: EAGER and LETTER introduce contrastive alignment objectives to integrate behavioral and semantic information.Their objectives explicitly connect recommendation behavior with item semantics.

3 Preliminary

This section introduces multi-token prediction as dense, longer-range supervision and motivates a sequential design for generative recommendation. Future behaviors exhibit semantic echoes of current interactions, while parallel prediction risks homogenizing multi-horizon outputs.

  • Multi-Token Prediction: MTP predicts multiple future tokens in one step, extending next-token prediction with longer-range dependencies during training.Its generic objective uses a prediction horizon n.
  • Sequential MTP Design: Sequentially chained MTP modules preserve causal dependencies across prediction depths and provide denser future-token supervision compatible with generation.This design feeds each module’s predecessor representation into the next prediction stage.
  • Future-Behavior Informativeness: Future-item SID overlap remains above random-pair levels across larger offsets and consistently across codebooks, indicating semantically anchored future behavior.The lift is defined as observed-pair overlap divided by randomly sampled pair overlap; 1.0× denotes random-level similarity.
  • Homogenization Pitfall: Parallel MTP homogenizes multi-horizon predictions, whereas sequential rollout reduces inter-horizon overlap and avoids underperforming at the immediate future horizon.Figure 3 compares RPG-Par. and RPG-Seq. using top-10 Jaccard similarity and horizon-wise performance on Game.

4 Methodology

EchoRec extends a parallel-token generative recommender with sequential multi-horizon branches that model preference transitions, then aligns their consolidated preference back to the decoding representation. Cycle-consistent projectors make this alignment verifiable and theoretically exclude rank-collapse spurious alignment under invertible transport.

  • Horizon-aware Preference Generation: The auxiliary branches reuse MTP-0’s item pool, semantic-ID space, token embeddings, output head, and decoding graph, adding only lightweight projection and Transformer modules.This preserves the original architecture and decoding pipeline while keeping the added branches lightweight.
  • Horizon-aware Preference Generation: HPG appends sequential auxiliary MTP branches that predict later items while conditioning each horizon on its predecessor to capture evolving preferences.MTP-1 predicts i_n+2 using i_n+1, and MTP-2 predicts i_n+3 from MTP-1 representations and a further-shifted sequence.
  • Verifiable Holistic-Preference Alignment: VHA aggregates the three branch representations into a holistic preference and explicitly pulls the original decoding representation toward it, while stop-gradient prevents reshaping auxiliary supervision.HPG otherwise delivers future-behavior supervision to the original branch only indirectly, leaving its decoding representation without an explicit consolidated target.
  • Verifiable Holistic-Preference Alignment: One-way alignment can satisfy its objective through projector subspace discrepancies without changing the decoding representation, making spurious alignment generic rather than incidental.The supplied analysis identifies this as an unverifiable failure mode of one-way alignment.
  • Verifiable Holistic-Preference Alignment: Under invertible transport, cycle consistency pins the projected alignment minimizer to one positive direction and excludes rank-collapse spurious alignment as the round trip approaches identity.The deviation from the back-projected preference is bounded by the transport condition number, the inverse-round-trip error, and the preference norm.
  • Optimization Objective: HPG combines the original and future-horizon losses as LHPG = L0 + λ1L1 + λ2L2, and EchoRec adds LVHA to form the final optimization objective.λ1 and λ2 control the contributions of the future-behavior supervision terms.

5 Experiments · 5.1 Experimental Setups

EchoRec is evaluated on three chronologically split Amazon Reviews ’23 datasets using multi-item leave-one-out testing, standard ranking metrics, and comparisons against six generative recommenders. All methods share identical data, representations, splits, and evaluation protocols, while EchoRec trains with three horizons but is evaluated only with MTP-0.

  • 5.1 Experimental Setups: Three Amazon Reviews ’23 datasets reserve each user’s last three interactions for testing, the fourth-to-last for validation, and earlier interactions for training.The split extends leave-one-out evaluation to multi-item prediction while preserving chronological order.
  • 5.1.2 Baselines: EchoRec is compared with six generative recommenders: EAGER, TIGER, LETTER, ETEGRec, SETRec, and RPG.The compared methods are representative generative recommendation baselines.
  • 5.1.2 Baselines: Evaluation uses Hit Rate (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K) at k∈{10, 20} under the all-ranking paradigm.All-ranking is adopted to avoid selection bias.
  • 5.1.4 Implementation Details: All methods use identical environments, data splits, pre-computed SID representations, and evaluation protocols on each dataset.Each method is independently run three times with different seeds.
  • 5.1.4 Implementation Details: EchoRec retains its three-horizon training objective within the training split, receives no extra supervision beyond baselines, and is evaluated solely with MTP-0.This protocol isolates the proposed training objective while keeping supervision matched.
  • 5.1.4 Implementation Details: Table 2 reports EchoRec’s performance against six baselines across the three datasets, marking statistically significant backbone improvements with ∗ at p < 0.01.The significance test uses paired t-tests.
  • 5.1.4 Implementation Details: Figure 5 evaluates EchoRec and its ablation versions on Game, Baby, and Arts, indicating that all components are effective.The ablation study covers the three named datasets.

5.2 Performance Comparison

EchoRec is evaluated with two diverse backbones against six generative-recommendation baselines across three datasets. It consistently and significantly outperforms its base recommender across all reported datasets and metrics, attributed to HPG’s sequential supervision and VHA’s verifiable alignment.

  • Performance Comparison: EchoRec is implemented with two diverse backbones and compared against six GR baselines across three datasets in Table 2.The comparison covers all reported datasets and metrics.
  • Performance Comparison: EchoRec consistently and significantly outperforms its base recommender on all datasets and metrics.The results imply that future-behavior signals are converted into recommendation gains.
  • Performance Comparison: The gains are attributed to HPG’s sequentially dependent supervision and VHA’s verifiable alignment of informative future-behavior signals.Together, these components convert future behaviors into recommendation improvements.

5.3 Ablation Study

The ablation study shows that multi-horizon supervision, one-way alignment, and cycle consistency each contribute to EchoRec’s performance, with the full model performing best across datasets. Replacing HPG with alternative multi-horizon structures further supports sequential chaining for coherent, non-redundant multi-item generation.

  • Module Ablation: EchoRec−VHA consistently surpasses EchoRec−HPG−VHA across all datasets, showing that HPG’s sequentially dependent supervision makes future behaviors informative signals.EchoRec−HPG−VHA equals the base recommender, while EchoRec−VHA retains only multi-horizon supervision.
  • Module Ablation: Appending one-way alignment further improves performance, indicating that the consolidated z_fcarries holistic information beyond the decoding representation.The result empirically rules out a target-side shortcut concern because aligning toward a shortcut target would yield no improvement.
  • Module Ablation: EchoRec achieves the best performance on all datasets, showing that cycle consistency further suppresses spurious alignment left unresolved by the one-way objective.The ablation compares EchoRec−L𝑐𝑦𝑐, EchoRec−VHA, EchoRec−HPG−VHA, and the full EchoRec model.
  • HPG Structure Ablation: HPG exhibits the lowest average inter-horizon overlap, even below RPG-Seq., indicating horizon-specific intent transitions rather than replicated immediate predictions.RPG-Par. decodes all horizons from shared context, whereas RPG-Seq. rolls out predictions without horizon-specific supervision; HPG additionally enables coherent, non-redundant multi-item generation.

5.4 Robustness Analysis

EchoRec remains robust across loss-weight choices, noisy histories, and different recommendation backbones. Moderate VHA weights are optimal, while EchoRec consistently improves over its base recommender and RPG under the tested conditions.

  • Loss-weight sensitivity: VHA performance follows a unimodal trend, peaking at λtrace = 0.5 and λcyc = 0.3, while EchoRec surpasses its base recommender throughout.Smaller weights underuse alignment or cycle constraints, whereas larger weights distract optimization from next-item prediction.
  • Noise robustness: 54.89% relative improvement on Baby HR@10 at η = 20% exceeds the 17.68% improvement at η = 0%, showing gains under mild-to-moderate noise.EchoRec consistently outperforms RPG across the tested noise levels in the mild-to-moderate range.
  • Backbone robustness: EchoRec instantiated on SETRec consistently and significantly outperforms SETRec across all datasets and metrics despite their fundamentally different identifier structures.This robustness follows from adding auxiliary branches and objectives without modifying the base recommender itself.

5.5 In-depth Analysis

EchoRec’s in-depth analysis finds that its auxiliary training cost remains affordable and illustrates how holistic preference alignment improves next-item and multi-horizon recommendations. Case studies show more precise same-series ranking and continued generation across horizons.

  • Efficiency Analysis: EchoRec increases training latency from 11.25ms to 32.81ms on Baby versus RPG, but this training-only overhead is treated as a one-off investment.The added cost comes from auxiliary branches and alignment objectives.
  • Case Study: On Baby’s next-item case, EchoRec ranks “Diapers (Size 5, 96 Count)” first and “Diapers (Size 5, 25 Count)” immediately after, unlike RPG’s generally relevant recommendations.The passage attributes this improvement to holistic preference internalized by VHA, which captures temporal progression rather than only recent-interaction semantics.
  • Case Study: In a Baby multi-horizon case, EchoRec correctly generates “Square Corner Playmat” at the immediate horizon and continues with diverse recommendations.Figure 9 marks ground-truth items in red.

6 Conclusion

The paper investigates dense supervision from Multi-Token Prediction for generative recommendation, motivated by future behaviors semantically echoing current interactions. Because this echo decays across intent transitions, EchoRec models preference evolution sequentially across horizons.

  • 6 Conclusion: Future behaviors semantically echo current interactions, making them informative supervision for generative recommendation.This empirical observation motivates unlocking the dense-supervision potential of Multi-Token Prediction.
  • 6 Conclusion: The semantic echo decays under intent transitions, requiring sequentially dependent modeling across horizons.The conclusion frames this dependency as necessary for modeling evolving preferences.
  • 6 Conclusion: EchoRec treats preference modeling as an acoustic echoing process to enhance generative recommendation.The proposed approach is designed to unlock Multi-Token Prediction’s dense-supervision potential.
Loading 2608.14011v1…