Source-linked AI summary
LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation
Lee Xiong, Zhirong Chen, Rahul Mayuranath, Shangran Qiu, Arda Ozdemir, Lu Li, Yang Hu, Dave Li, Jingtao Ren, Howard Cheng, Fabian Souto Herrera, Ahmed Agiza, Baruch Epshtein, Anuj Aggarwal, Julia Ulziisaikhan, Chao Wang, Dinesh Ramasamy, Parshva Doshi, Sri Reddy, Arnold Overwijk
TL;DR
Production ads recommendation must scale sequence models while preserving latency and integrating sparse signals. LLaTTE studies joint scaling across architecture and semantic features, then separates heavy upstream computation from lightweight online ranking; the paper reports predictable scaling, transfer across stages, and production improvements with low serving overhead.
Problem
Production recommendation systems must scale sequence models while integrating sparse features and meeting strict latency requirements.
Method
LLaTTE combines a scalable target-aware transformer with asynchronous upstream user modeling and lightweight online ranking using cached embeddings.
Results
0.25% improvement in Normalized Entropy is achieved on primary revenue-generating models with minimal serving overhead.
Takeaways & Limitations
Predictable scaling and upstream-to-downstream transfer provide a framework for scaling sequence models across production recommendation stages.
Abstract
from arXiv · showhide
We present LLaTTE (LLM-Style Latent Transformers for Temporal Events), a scalable transformer architecture for production ads recommendation. Through systematic experiments, we demonstrate that sequence modeling in recommendation systems follows predictable power-law scaling similar to LLMs. Crucially, we find that semantic features bend the scaling curve: they are a prerequisite for scaling, enabling the model to effectively utilize the capacity of deeper and longer architectures. To realize the benefits of continued scaling under strict latency constraints, we introduce a two-stage architecture that offloads the heavy computation of large, long-context models to an asynchronous upstream user model. We demonstrate that upstream improvements transfer predictably to downstream ranking tasks. Deployed as the largest user model at Meta, this multi-stage framework drives a 4.3\% conversion uplift on Facebook Feed and Reels with minimal serving overhead, establishing a practical blueprint for harnessing scaling laws in industrial recommender systems.
1 Introduction
LLaTTE addresses the challenge of scaling sequence models in production recommendation systems, where deep modeling must coexist with sparse features and strict latency constraints. It combines a target-aware transformer and a multi-stage architecture, finding predictable scaling behavior and measurable production gains.
- Production recommendation systems must reconcile deep sequence modeling with millisecond-level ranking latency and high-dimensional sparse features.
- LLaTTE integrates sequence modeling with FM-style architectures through a target-aware transformer and adaptive computational reduction.The design uses non-sequence sparse features and candidate information in query tokens, with adaptive pyramidal output extraction.
- A multi-stage design runs large upstream encoders asynchronously and supplies cached user embeddings to a lightweight online ranker.The upstream stage is triggered by high-value user events, avoiding request-time latency for its heavy computation.
- Sequence length is a primary scaling lever, while semantic content embeddings are a prerequisite for steeper scaling curves.The experiments vary depth, width, sequence length, and feature enrichment to study their interactions.
- 0.25% improvement in Normalized Entropy is achieved on primary revenue-generating models with minimal serving overhead.
2 Background and Related Work
Prior work established transformer-based sequential recommendation, feature-aware attention, and multi-stage serving, but did not systematically study joint scaling across model capacity, data richness, and deployment stages. LLaTTE positions itself as an empirical scaling framework for both online and asynchronous ranking.
- Transformer-based sequence modeling: Self-attention methods improved parallel processing and long-range modeling for sequential recommendation.
- Transformer-based sequence modeling: Target-aware attention mechanisms were developed for production CTR prediction and later extended with interest evolution modeling.
- Feature integration: Recent architectures combine sequential and non-sequential features, but some require tight coupling that limits independent sequence-module scaling.
- Multi-stage architectures: Embedding-based multi-stage systems decouple upstream representation learning from online serving, enabling larger models than real-time latency permits.
- Research gap: Prior multi-stage work had not explored systematic predictable scaling across multiple dimensions or quantified transfer through the inference bottleneck.LLaTTE examines capacity allocation between online and upstream stages and introduces a transfer-ratio metric.
3 Preliminaries: Ads Recommendation Stack
The ads recommendation stack predicts calibrated engagement probabilities from user, ad, interaction, and context features. User behavior is represented as a temporal action sequence, and evaluation uses Normalized Entropy alongside a weighted multi-task loss.
- The model predicts engagement probabilities such as CTR and CVR for downstream ranking and auction pricing.
- Inputs comprise user, ad, user-ad interaction, and request-context features, including sparse IDs, dense embeddings, float attributes, and sequences.
- A user sequence is an ordered list of timestamped actions containing behavioral and item-related metadata.Experimental sequence lengths range from 500 to 5000 actions.
- Training uses a weighted multi-task binary cross-entropy loss, with relative improvement in Normalized Entropy as the primary evaluation metric.
- Normalized Entropy is average log loss normalized by the entropy of the empirical CTR.The empirical positive rate p is used in the normalization.
4 Experimental Backbone: The LLaTTE Paradigm
LLaTTE is a modular sequence-modeling backbone that keeps the non-sequence module and task heads fixed while scaling the sequence module. Its latent-attention and pyramidal-reduction mechanisms support long histories across asynchronous upstream and latency-constrained online deployments.
- LLaTTE comprises a sequence module, a non-sequence module, and task heads for prediction.
- The scaling study varies sequence-module depth, width, and length while holding the DHEN backbone and task heads fixed.
- The sequence module uses Multi-head Latent Attention and Adaptive Pyramidal Output to process contexts longer than 1000 under production constraints.
- Tokenization and context fusion create sequence and query-token inputs, while deeper layers selectively trim older tokens before fixed-size readout.Pyramidal reduction concentrates computation on recent events and query tokens.
- The upstream model uses high-capacity variants and full self-attention, whereas online ranking uses aggressive pyramidal trimming to meet latency budgets.The upstream sequence module consumes more than 45× the sequence FLOPs of the main ranker.
5 Scaling Framework and Methodology
The framework evaluates scaling across compute, capacity, temporal horizon, information density, and cross-stage transfer. It models normalized-entropy improvements with compute and measures how upstream gains survive the multi-stage information bottleneck.
- Scaling formulation: The framework models Normalized Entropy improvements relative to a production baseline as a power law of sequence-module compute.Compute is measured in FLOPs, with a scaling coefficient α representing the efficiency of a scaling dimension.
- Scaling axes: Capacity scaling varies transformer depth L and embedding width d to identify parameter-efficient allocations and width thresholds for effective depth scaling.The study searches for an aspect ratio L/d that maximizes parameter efficiency in the sparse-feature regime.
- Scaling axes: Temporal scaling treats sequence length T as the information horizon and tests whether larger models can use increasingly distant events up to T →5000.This distinguishes diminishing returns from continued gains as histories become longer.
- Scaling axes: Information-density scaling compares sparse ID tokens with tokens enriched by dense semantic embeddings, treating signal quality as a multiplier on α.The comparison tests whether semantic richness changes the effectiveness of architectural scaling.
- Cross-stage transfer: Transfer Ratio τ measures how effectively upstream improvements translate into online ranking gains while accounting for production interference such as asynchronous latency penalties.It is defined as downstream NE improvement divided by upstream NE improvement.
6 Experimental Results
Experiments show that balanced capacity, longer histories, semantic features, and increased compute improve recommendation quality in distinct but interacting ways. Overall, sequence recommendation exhibits predictable scaling, with sequence length and content quality especially influential.
- Capacity allocation: Sufficient width enables depth scaling: at d ≈256, added depth becomes substantially more effective, whereas narrow models show diminishing returns.Increasing width alone toward d = 1024 yields limited NE gains.
- Capacity allocation: Deep-Balanced configurations with d ≥256 maximize performance and compute efficiency, while extreme width or depth allocations produce diminishing returns.Table 1 reports the Deep-Balanced configuration as achieving the best performance at -0.17%.
- Temporal horizon: NE decreases monotonically as sequence length T increases across model depths, and deeper models obtain larger gains from additional context.The result supports using longer histories when model capacity is sufficient.
- Sequence composition: Balanced mixtures of high-frequency views and low-frequency conversions outperform pure-view and pure-conversion sequences at fixed T = 1000.Views provide dense, recent coverage, while conversions are valuable but temporally sparse.
- Content-aware scaling: Semantic content features make architectural scaling more effective: ID-only models gain little from added depth, whereas content-enriched models show substantially larger gains.The paper identifies semantic features as a prerequisite for extracting value from increased depth, sequence length, and compute.
- Global compute scaling: Recommendation performance follows a predictable power-law relationship with compute, with sequence length showing the steepest scaling slope.Width is foundational for exploiting other dimensions, while semantic features steepen scaling across dimensions.
- Global compute scaling: The combined findings establish sequence-based recommendation as a scalable paradigm that converts compute into predictable performance gains.The scaling hierarchy is used to optimize the multi-stage architecture under production constraints.
7 Multistage Sequence Model: Towards Maximizing RoI in Production
The multistage architecture separates large upstream user modeling from latency-constrained downstream ranking, allowing sequence-model scaling to continue under a fixed information bottleneck. Upstream improvements transfer predictably to ranking performance, with efficiency shaped by sequence length, compute allocation, and candidate context.
- Motivation and architecture: The downstream ranker serves at trillion-request scale under strict latency constraints, limiting its sequence module to a few layers and roughly T ≈400 events per source.The upstream model instead processes longer histories asynchronously and provides cached embeddings to the ranker.
- Motivation and architecture: Upstream modeling compresses thousands of historical events into a fixed-size vector with d_transfer = 2048, creating a strict information bottleneck.The downstream model combines these cached representations with fresh short-horizon signals and ad/context features.
- Upstream scaling laws: Downstream models exhibit steeper total-FLOP scaling slopes because sequence computation represents ≈30% of their FLOPs, versus ≈90% upstream.Scaling the sequence module therefore produces larger relative improvement per total FLOP downstream.
- Upstream scaling laws: Sequence-only depth and width scaling efficiencies are consistent across upstream and downstream settings, while upstream sequence-length scaling retains only ≈50% of downstream efficiency.Without candidate context, the upstream encoder must compress history into a generic representation rather than perform candidate-aware attention.
- Transfer efficiency: A 0.14% upstream improvement produced a 0.07% downstream improvement, corresponding to a transfer ratio of τ ≈50%.The transfer ratio measures conversion of upstream representation quality into downstream ranking accuracy.
- Architecture robustness: Matched-compute upstream configurations achieved similar upstream gains and identical downstream gains, indicating that compute allocation between depth and sequence length can be traded flexibly.The reported configurations used ≈12 GFLOPs/sample and produced −0.14% versus −0.13% upstream gains, with −0.07% downstream gains for both.
- Design policy: The resulting policy uses short sequences and shallow depth online, while scaling total sequence-modeling compute asynchronously upstream.This strategy continues the scaling frontier after the latency-constrained ranker exhausts its local budget.
8 Production Deployment
Production deployment pairs a compact online LLaTTE ranker with an asynchronous upstream embedding service. This preserves request-time latency while converting larger upstream sequence models into measurable ranking and conversion gains.
- Online ranking: The online ranker attends to short user histories capped at T ≈400 together with ad and context features under a strict per-request latency budget.It captures fresh user intent and ad-specific interactions while remaining smaller than the scaling-study models.
- Asynchronous upstream modeling: The asynchronous embedding service updates upstream user representations on high-value events and hosts deeper, longer models on a dedicated H100 cluster.Updates are not computed per request, allowing the upstream model to avoid the online latency path.
- Latency: Offloading heavy sequence modeling adds only a feature lookup to the ranking path, with no measurable change in P99 ranking latency versus the baseline.The larger upstream models run at lower QPS and are heavily batched on the H100 cluster.
- Online gains: 4.3% conversion uplift on Facebook Feed and Reels corresponded to approximately 0.25% NE reduction in multiple large-scale A/B tests.The result validates the two-stage strategy as a way to convert additional sequence-modeling compute into production value under strict serving constraints.
9 Conclusion & Future Work
The paper presents LLaTTE as a scalable paradigm for sequence learning in recommender systems and analyzes scaling across model and sequence dimensions. Its production strategy allocates most computation to offline user modeling while transferring gains to a lightweight online model.
- Conclusion: LLaTTE scales sequence learning in recommender systems and analyzes depth, width, sequence length, composition, and richness.The analysis argues that effective scaling requires balancing these dimensions simultaneously.
- Conclusion: The multistage deployment strategy trains an expensive user-only model offline and infrequently, then transfers its gains to a lightweight online production model.This allocation is intended to maximize production impact while preserving online serving constraints.
- Future work: Future work targets efficient long-context kernels, reinforcement learning, scalable infrastructure, and the upper bounds of recommender-system scaling laws.The authors position these directions alongside the continuing expansion of sequence modeling in large-scale recommendation systems.
A.1 Non-Sequence Module
The module combines sparse, dense, float, and sequence-derived features before transformer processing, using adaptive token reduction to control computation. Its attention analysis examines how model focus varies across user-history time and finds both recency and recurring daily patterns.
- Feature representation: The initial representation concatenates embedded sparse, dense, float, and sequence-summary features.Sparse categorical features use an embedding lookup, while the resulting non-sequence representation is formed by concatenation.
- Transformer input: Query tokens incorporate candidate-ad, request, user, and learned seed information before entering the transformer.Sequence and query tokens are concatenated into the transformer input, with token contents depending on the stage.
- Pyramidal schedule: Adaptive pyramidal attention progressively trims older tokens, reducing attention costs as sequence length decreases across layers.The schedule supports full self-attention, pyramidal attention over recent actions plus query tokens, and final cross-attention over query tokens.
- Pyramidal schedule: Offline models retain more full self-attention, whereas online ranking models apply more aggressive early pyramidal trimming before final cross-attention.This division reflects different computation patterns across upstream and online stages.
- Attention analysis: Attention weights exhibit sharp 24-hour spikes, suggesting recurring user interests and behaviors at similar times of day.Figures 5 and 6 report attention weight distributions by hours before the request and average attention per event.