PaperlayerView exact revision on arXiv

Source-linked AI summary

Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, Yu Shi

arXiv:2402.17152v3cs.LGcs.IR

TL;DR

Industrial DLRMs use heterogeneous, high-cardinality features but scale poorly with compute, motivating a different formulation for large-scale recommendation. The paper introduces Generative Recommenders, HSTU, and supporting training and inference methods that recast ranking and retrieval as sequential transduction tasks. Generative Recommenders achieve 12.4% production metric improvements, while the approach supports greater model complexity and compute scaling across three orders of magnitude.

  • Problem

    Most industrial DLRMs scale poorly with compute despite extensive engineered feature sets and vast training data, while recommendation systems also face heterogeneous features and dynamic billion-scale vocabularies.

  • Method

    The paper unifies heterogeneous features into sequences, reformulates ranking and retrieval as generative sequential transduction tasks, and introduces HSTU with cost-amortizing training and inference algorithms.

  • Results

    12.4% production metric improvements were achieved, with deployed Generative Recommenders 285x more complex while using less inference compute than traditional DLRMs.

  • Takeaways & Limitations

    Generative Recommenders and HSTU show that user actions can serve as an underexplored modality for recommendation modeling and support recommendation foundation-model development.

  • Takeaways & Limitations

    The evaluation methodology differs significantly from industrial streaming settings because full-shuffle and multi-epoch training are generally impractical there.

Abstract

from arXiv · show

Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

1. Introduction

The paper identifies poor compute scaling in industrial DLRMs and proposes Generative Recommenders, HSTU, and M-FALCON to reformulate recommendation and improve efficiency. These methods support substantially more complex deployed models, production metric gains, and empirical compute scaling across recommendation tasks.

  • Most industrial DLRMs scale poorly with compute despite extensive engineered features and vast training data.
  • Generative Recommenders treat user actions as a new modality and cast industrial ranking and retrieval as generative modeling problems.
  • 12.4% topline metric improvements were achieved by deployed models three orders of magnitude more computationally complex than prior state-of-the-art.
  • HSTU modifies attention for large, non-stationary vocabularies and delivers 5.3x to 15.2x speedup over FlashAttention2-based Transformers on 8192-length sequences.
  • M-FALCON uses micro-batching to serve 285x more complex Generative Recommender models with 1.50x-2.99x speedups under the same inference budget as traditional DLRMs.
  • The techniques were validated on synthetic datasets, public datasets, and multiple production surfaces, with Generative Recommenders significantly outperforming DLRMs in large-scale industrial settings.

2. Recommendation as Sequential Transduction Tasks: From DLRMs to GRs

The paper unifies heterogeneous recommendation features into chronological sequences and reformulates ranking and retrieval as sequential transduction tasks. Generative training reduces repeated encoder computation, while the formulation addresses dynamic vocabularies, target-aware ranking, and long-sequence cost.

  • 2.1. Unifying heterogeneous feature spaces in DLRMs: Generative Recommenders consolidate DLRM categorical and numerical features into a single unified time series.
  • 2.2. Reformulating ranking and retrieval as sequential transduction tasks: Sequential transduction maps chronologically ordered input tokens to output tokens, with undefined outputs represented by ∅.
  • 2.2. Reformulating ranking and retrieval as sequential transduction tasks: Retrieval predicts a distribution over the next content, while negative user responses and non-engagement features complicate direct autoregressive supervision.
  • 2.2. Reformulating ranking and retrieval as sequential transduction tasks: Ranking is formulated as p(ai+1|Φ0, a0, Φ1, a1, . . . , Φi+1) by interleaving items and actions to enable early target-aware interaction.
  • 2.3. Generative training: Transformer-based sequential training can reach O(N^3d + N^2d^2) time complexity for maximum sequence length N, making it costly for recommendation.
  • 2.3. Generative training: Generative training amortizes encoder costs across multiple targets and reduces computational complexity by an O(N) factor through sampling.

3. A High Performance Self-Attention Encoder for Generative Recommendations

HSTU is a simplified, fused encoder that replaces heterogeneous DLRM modules with pointwise projection, spatial aggregation, and pointwise transformation. Its pointwise attention, sparsity-aware kernels, Stochastic Length, and M-FALCON target the computational costs of long histories, large batches, and many candidates.

  • HSTU stacks residual layers containing Pointwise Projection, Spatial Aggregation, and Pointwise Transformation sub-layers.The design uses a single modular block in place of heterogeneous DLRM components.
  • HSTU replaces DLRM feature interactions with normalized attention-pooled features gated by pointwise transformations.The interaction is expressed as Norm (A(X)V (X)) ⊙U(X), motivated by the difficulty of approximating dot products with learned MLPs.
  • 3.1. Pointwise aggregated attention: Pointwise aggregated attention preserves the number of prior target-related data points as a preference-intensity signal, unlike softmax normalization over the entire sequence.On synthetic nonstationary streaming data, gaps as large as 44.7% were observed between softmax and pointwise attention setups.
  • 3.2. Leveraging and algorithmically increasing sparsity: Ragged attention kernels exploit skewed user-history lengths by grouping computations into GEMMs of varying sizes, yielding 2-5x throughput gains.The approach fuses back-to-back GEMMs and performs fully ragged attention computations.
  • 3.2. Leveraging and algorithmically increasing sparsity: Stochastic Length increases sequence sparsity by selecting subsequences from temporally repetitive histories, reducing encoder cost while preserving model quality in tested settings.Lower α values apply to longer sequences, including sequences up to length 8,192; applying SL to training targets the higher cost of training.
  • 3.4. Scaling up inference via cost-amortization: M-FALCON amortizes candidate-dependent inference costs, enabling 285x more complex target-aware ranking models at 1.5x-3x throughput under a constant inference budget.The approach addresses ranking, while retrieval can use existing amortizable or efficient search algorithms.

4. Experiments

Experiments show that HSTU and Generative Recommenders improve recommendation quality and efficiency across public and industrial streaming settings. Their advantages persist with stochastic sparsification and scale to substantially larger models than traditional DLRMs, although public-dataset evaluation differs from production streaming.

  • 4.1.1. Traditional Sequential Settings: HSTU significantly outperforms SASRec with matched configurations and improves further when scaled up on MovieLens and Amazon Reviews.HSTU-large uses 4x the layers and 2x the heads of the matched configuration.
  • 4.2. Encoder Efficiency: 64%-84% sparsity from Stochastic Length reduces sequence tokens while changing main-task NE by no more than 0.002 (0.2%).At α = 1.6, length-4096 sequences become length 776 most of the time.
  • 4.2. Encoder Efficiency: HSTU enables over 2x deeper networks than Transformers through reduced activation memory usage.The comparison uses identical encoder configurations and FlashAttention-2 for the Transformer baseline.
  • 4.3. Generative Recommenders vs DLRMs: Generative Recommenders significantly outperform industrial DLRM baselines offline and deliver 12.4% wins in online A/B tests.The evaluation reports both adding a GR source and replacing the existing main DLRM source.
  • 4.3. Generative Recommenders vs DLRMs: Using only GR features significantly degrades DLRM performance, while interaction-only GRs underperform GRs by 2.6% in NE on the main consumption task.The results support the value of high-cardinality user actions and GR’s unified feature space.
  • 4.3. Generative Recommenders vs DLRMs: GRs achieve 1.50x/2.99x higher QPS when scoring 1024/16384 candidates despite being 285x more computationally complex.The efficiency gain is attributed to HSTU and M-FALCON.
  • 4.4. Scalability: GRs scale to 1.5 trillion parameters while DLRM quality saturates at about 200 billion parameters, with recommendation metrics following compute power laws across three orders of magnitude.The scaling result covers retrieval Hit Rate@100 and Hit Rate@500 and ranking NE.

5. Related Work

Related work spans sequential recommenders, efficient sequential transduction, and language-model-inspired recommendation. Prior systems generally simplify behavior sequences or embed sequential components within DLRMs, motivating a fully sequential generative formulation for industrial recommendation.

  • Sequential Recommendation: Earlier sequential recommenders reduce user behavior to a homogeneous item sequence, while industrial systems primarily use pairwise attention or sequential encoders inside DLRMs.This distinction separates academic sequential recommendation from industrial DLRM practice.
  • Efficient Sequential Transduction: Efficient-attention research addresses self-attention’s O(n^2) scaling through factorized, low-rank, alternative sequential-transduction, and hardware-aware formulations.HSTU’s elementwise gating is specifically inspired by FLASH.
  • LLMs for Recommendation: LLM-related recommendation work explores in-context learning, instruction tuning, transfer learning, and textual user-behavior representations.Reported benefits include zero-shot or few-shot recommendation and scaling behavior on medium-scale datasets.

6. Conclusions

The paper concludes that Generative Recommenders formulate ranking and retrieval as generative sequential transduction tasks supported by efficient HSTU encoders and serving algorithms. These systems improve production metrics, scale beyond traditional DLRMs, and may support unified recommendation, search, and advertising models, while broader privacy and user-goal implications remain framed as expected benefits.

  • Conclusions: GRs formulate ranking and retrieval as sequential transduction tasks trained generatively, using HSTU and M-FALCON to support efficient scaling.HSTU is reported as 5.3x-15.2x faster than state-of-the-art Transformers on length-8192 sequences.
  • Conclusions: GRs deploy models 285x more complex than prior systems while using less inference compute and producing 12.4% production metric improvements.The conclusion also reports superior scaling performance compared with traditional DLRMs.
  • Future Scope: A unified feature space across recommendation, search, and ads is presented as a path toward foundation models spanning these domains.The paper links this possibility to simplifying heterogeneous features.
  • Potential Implications: The authors suggest reduced reliance on heterogeneous features could improve privacy and user experiences, while sequential attribution could reduce content misaligned with long-term goals.These are presented as potential implications rather than demonstrated outcomes.

B. Generative Recommenders: Background and Formulations

This appendix contrasts academic sequential recommenders and industrial DLRMs with Generative Recommenders, which unify heterogeneous features and model ranking and retrieval as sequential generative tasks. It also outlines GR feature construction, target-aware causal attention, and joint sequence modeling, while noting that full listwise generation remains future work.

  • B. Generative Recommenders: Background and Formulations: Classical DLRMs use heterogeneous feature spaces with feature interaction, pooling or target-aware attention, and multi-expert multi-task modules.The appendix contrasts these architectures explicitly with GRs.
  • B.1.1. ACADEMIC RESEARCH (TRADITIONAL SEQUENTIAL RECOMMENDER SETTINGS): Early academic sequential recommenders such as GRU4Rec retain positive events and primarily address retrieval-like next-item prediction.GRU4Rec used clicked items or sufficiently watched videos as positive sequence events.
  • B.1.1. ACADEMIC RESEARCH (TRADITIONAL SEQUENTIAL RECOMMENDER SETTINGS): SASRec applies an autoregressive Transformer to positive-item sequences with binary cross-entropy and randomly sampled negatives, followed by related bidirectional and pretraining variants.BERT4Rec and S3Rec are cited as subsequent developments in similar settings.
  • B.1.2. INDUSTRIAL APPLICATIONS AS PART OF DEEP LEARNING RECOMMENDATION MODELS (DLRMS).: Industrial sequential components typically use short histories inside DLRMs, with reported lengths of 20, 1,000, and 100 versus 8,192 in this work.DLRMs compensate partly through precomputed profiles, vector stores, and contextual, user, and item features.
  • B.1.2. INDUSTRIAL APPLICATIONS AS PART OF DEEP LEARNING RECOMMENDATION MODELS (DLRMS).: Industrial DLRMs define losses over user/request–candidate pairs, commonly using multi-task ranking losses and two-tower retrieval, whereas fully sequential ranking had not succeeded at billion-DAU scale before this work.The appendix classifies these systems as DLRMs despite differences in architectures, features, and losses.
  • B.2. Formulations: Ranking and Retrieval as Sequential Transduction Tasks in Generative Recommenders (GRs): GRs address prior formulations by incorporating features beyond interacted items and interleaving content and action sequences for target-aware causal autoregressive attention.This combines broader industrial feature coverage with sequential modeling.
  • B.2. Formulations: Ranking and Retrieval as Sequential Transduction Tasks in Generative Recommenders (GRs): GRs model the joint distribution of contents and actions, with next-action prediction corresponding to ranking and next-content prediction corresponding to retrieval.The formulation supports generative modeling over the full alternating sequence.
  • B.2. Formulations: Ranking and Retrieval as Sequential Transduction Tasks in Generative Recommenders (GRs): Directly sampling recommendation sequences, for example with beam search, is proposed as an advantage over traditional listwise settings, but full evaluation is left for future work.The comparison includes listwise methods such as DPP and reinforcement learning.

C. Evaluation: Synthetic Data

The synthetic evaluation models streaming recommendation data with a dynamic vocabulary and category-dependent engagement patterns. HSTU substantially improves hit rate over standard Transformers, while softmax attention reduces performance.

  • Synthetic data construction: The synthetic data follows a Dirichlet Process over a dynamically expanding item vocabulary to model streaming recommendation behavior.The process captures a “rich gets richer” pattern in user engagement histories and imposes streaming availability constraints on item ids.
  • Synthetic data construction: The experiment generates 1,000,000 records of length 128, using the first 90% for training and the final 10% for testing.Only 40% of item ids are initially available; the remaining ids become available progressively during streaming training.
  • Results: HSTU increases Hit Rate@10 by more than 100% relative to standard Transformers.The comparison uses the synthetic dataset, where timestamps are unavailable and the timestamp-related HSTU component is ablated.
  • Results: Replacing HSTU’s pointwise attention with softmax significantly reduces Hit Rate@10.This result supports the importance of pointwise attention-like aggregation mechanisms in the synthetic setting.

D. Evaluation: Traditional Sequential Recommender Settings

In traditional sequential recommendation settings, the evaluation compares HSTU with Transformer, recurrent, and bidirectional self-supervised baselines. HSTU significantly outperforms the evaluated alternatives, while SASRec remains highly competitive.

  • Evaluation setup: The evaluation compares HSTU with SASRec, GRU4Rec, and BERT4Rec in traditional sequential recommendation settings.SASRec represents causal autoregressive self-attention, GRU4Rec recurrent modeling, and BERT4Rec bidirectional Cloze-based self-supervision.
  • Evaluation setup: The comparison holds sampled-softmax negatives constant at 128 for ML-1M and ML-20M and 512 for Amazon Books.Reported GRU4Rec and BERT4Rec results are reused from prior work, with fixed negative counts intended to support fair comparison.
  • Results: SASRec remains one of the most competitive approaches when sampled softmax loss is used.This conclusion is stated for traditional sequential recommendation settings.
  • Results: HSTU significantly outperforms the evaluated Transformers, RNNs, and self-supervised bidirectional Transformers.The result is reported for the traditional sequential recommender evaluation.

E. Evaluation: Traditional DLRM Baselines

The DLRM evaluations use production-like ranking and retrieval baselines with extensive heterogeneous features, while also studying sequence selection and sparsity. Feature-weighted subsequence selection gives the best reported model quality.

  • DLRM baselines: The DLRM baselines approximate production configurations developed through years of research and engineering for a platform with billions of daily active users.The ranking and retrieval baselines use large heterogeneous feature sets and established architectural components.
  • Ranking baseline: The ranking baseline uses approximately one thousand dense features and fifty sparse features, including sequential modules, mixture-of-experts, and Deep & Cross components.The described setup incorporates target-aware pairwise attention and residual connections.
  • Feature comparison: The DLRM baseline feature sets are strict supersets of the features used by all Generative Recommender candidates.This applies to the main studies and the scaling studies.
  • Subsequence selection: Feature-weighted subsequence selection produces the best model quality among the evaluated selection methods.The selection methods choose a subsequence of length L from the full user history using recency-based, random, or feature-weighted criteria.
  • Sequence sparsity: Stochastic Length is evaluated for its effect on sequence sparsity using a representative industry-scale configuration with 30-day engagement history.The study also reports 60-day and 90-day history settings and defines s2 as one minus attention-matrix sparsity.

F.3. Comparisons Against Sequence Length Extrapolation Techniques

The paper compares Stochastic Length with zero-shot and fine-tuned sequence-length extrapolation methods under matched training sparsity. Stochastic Length significantly outperforms these alternatives in recommendation settings with high-cardinality ids.

  • Compared methods: The evaluation compares Stochastic Length with zero-shot and fine-tuned extrapolation methods based on NTK-aware or NTK-by-parts RoPE modifications.HSTU variants with and without relative attention bias are evaluated using the corresponding extrapolation procedures.
  • Experimental protocol: All zero-shot and fine-tuned models train on sequence length 1024 and evaluate on sequence lengths 2048 and 4096.Stochastic Length settings are selected to match the data sparsity of the comparison methods.
  • Experimental protocol: The comparison reports NE differences between models trained with induced data sparsity and models trained on the full data.The induced sparsity conditions include Stochastic Length, zero-shot extrapolation, and fine-tuning.
  • Results: Stochastic Length significantly outperforms fine-tuning and zero-shot approaches for sequence-length extrapolation.The authors attribute this empirically to the large vocabulary, where alternative methods fail to learn good representations for older ids.
  • Efficiency: The efficient HSTU attention kernel avoids materializing h × N × N intermediate attention tensors during the backward pass.It builds on memory-efficient and FlashAttention mechanisms while exploiting sparse input sequences.

H. Microbatched-Fast Attention Leveraging Cacheable OperatioNs (M-FALCON)

M-FALCON accelerates target-aware causal autoregressive inference by batching ranking candidates while preventing candidates from attending to one another. Microbatching and cache reuse reduce the cost of scoring large candidate sets.

  • Batched inference: Target-aware ranking can batch candidate inference instead of scoring each of m candidates separately at cost O(mn^2d).The modified attention mask enables parallel processing while retaining each candidate’s dependence on the shared user history.
  • Batched inference: M-FALCON modifies causal self-attention so candidate tokens cannot attend to one another, preserving independent target-aware predictions.A single forward pass over the user history and a candidate microbatch produces the same candidate results as separate forward passes.
  • Microbatching: Microbatching divides m candidates into ⌈m/b_m⌉ groups of size b_m, retaining O(n^2d) runtime when O(b_m) = O(n).This targets ranking stages with candidate sets reaching tens of thousands.
  • Caching: KV caching reuses cached K(X) and V(X) for the sequentialized user history across microbatches and requests.Cached forward passes compute projections only for the final b_m tokens while reusing history states.
  • Scope: M-FALCON applies beyond HSTU and GRs to other target-aware causal autoregressive self-attention models.The paper presents it as a broadly applicable inference optimization algorithm.

H.1. Evaluation of Inference Throughput: Generative Recommenders (GRs) w/ M-FALCON vs DLRMs

The evaluation compares Generative Recommenders using M-FALCON with DLRMs in large-scale inference settings. Throughput scales sublinearly with candidate count in a practical region, and the combined optimizations deliver substantial production speedups despite much higher model complexity.

  • Microbatch scaling: 1.99x additional speedup is obtained relative to the b_m = m = 1024 single-microbatch baseline.The comparison is reported for the combined throughput optimizations.
  • Candidate scaling: GR throughput scales sublinearly with ranking candidates up to m = 2048 in the case study.Batched inference amortizes computation across candidates in this region.
  • Production throughput: 2.99x higher throughput is achieved by HSTU-based Generative Recommenders than DLRMs in a large-scale production setup.This result holds despite GRs being 285x more complex in FLOPs.
  • Evaluation setup: Figure 12 compares end-to-end inference throughput for DLRMs and GRs with M-FALCON in large-scale industrial settings.The figure is reproduced from Figure 6 to facilitate reading.
  • Evaluation setup: Figure 13 shows M-FALCON throughput scaling on a 285x-FLOPs GR model as m ranges from 1024 to 16384 with b_m = 1024.This isolates throughput behavior in large-batch settings.
Loading 2402.17152v3…