PaperlayerView exact revision on arXiv

Source-linked AI summary

HELIX: Purified and Unified - Rethinking Feature Interaction and Sequence Modeling for Large-Scale Recommendation

Yuntao Zheng, Miao Zhang, Yadong Ding, Yanchuan Tang, Lixiyu Chen, Hao Wang, Quan Li, Shiying Cai, Yue Lin, Jiayu Li, Yu Feng, Wentao Yang, Rongkun Xing, Jiekai Wang, Mingge Zhang, Feiling Gong, Xiang Gao, Jinyu Dong, Yajing Zhang, Pengfei Ren, Yinzhou Wang

arXiv:2609.37183v2cs.IR

TL;DR

Industrial ranking models face limits when scaling feature interaction or sequence modeling alone. HELIX interleaves both capabilities while preserving reusable user-side sequence computation. The paper reports predictable scaling gains and meaningful online business impact after deployment in TikTok’s e-commerce recommendation system.

  • Problem

    Industrial ranking models scale feature interaction and sequence modeling, but either axis alone has limited scaling ceiling and a suboptimal scaling-law slope.

    Source: §1 · p. 3

  • Method

    HELIX interleaves sequence retrieval and feature interaction, directing information from reusable sequence states to candidate-conditioned mix-tokens.

    Source: §1 · p. 3, §6 · p. 13

  • Results

    Offline experiments show predictable scaling behavior, and deployment in TikTok’s e-commerce recommendation system confirms meaningful online business impact.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    HELIX is presented as an effective, production-ready approach to scaling recommendation ranking models along both modeling axes.

    Source: §6 · p. 14

Abstract

from arXiv · show

Industrial recommendation ranking models typically scale along two modeling axes: feature interaction over heterogeneous user, item, context, and cross features, and sequence modeling over long, informative, and multi-type user behavior histories. We find that scaling either capability in isolation is insufficient, as each exhibits a limited scaling ceiling and a suboptimal scaling-law slope. We conjecture that achieving a more favorable scaling-law slope requires jointly scaling both axes. To support this, we present HELIX, a purified and unified architecture for large-scale recommendation. HELIX interleaves sequence retrieval and feature interaction while enforcing one-way information flow from reusable sequence states to candidate-conditioned mix-tokens. This design preserves cross-depth communication between the two modeling axes while keeping user-side sequence computation amortizable, enabling flexible and asymmetric scaling of sequence modeling and feature interaction. Deployed in TikTok's e-commerce recommendation system, HELIX consistently improves offline CTR AUC, CVR AUC, and other ranking metrics. In online A/B tests, it achieves an approximately 6% increase in e-commerce video GMV per user.

1 Introduction

Industrial ranking models rely on both feature interaction and sequence modeling, yet scaling either axis alone has limits. HELIX interleaves the two while preserving reusable user-side computation, and the paper reports scaling gains and improvements in TikTok e-commerce recommendation.

1 Introduction

  • Scaling feature interaction or sequence modeling alone leaves the other axis as a bottleneck, while earlier approaches also face limits in interaction quality, behavior retrieval, or reusable computation.The introduction notes rapid saturation or optimization difficulty for larger feature-interaction models and limited gains from deeper sequence capacity.Source: §1 · p. 3
  • HELIX interleaves sequence retrieval with feature interaction and sends information one way from reusable sequence states to candidate-conditioned mix-tokens.A candidate-independent behavioral-history cache is repeatedly queried by mix-tokens, which then perform Mixup-PerToken-FFN interaction.Source: §1 · p. 3
  • HELIX preserves user-side computation reuse, supporting M-FALCON and user-level training amortization in production systems.The sequence-modeling branch remains reusable while the feature-interaction branch is candidate-conditioned.Source: §1 · p. 3
  • The paper reports scaling-law analyses in which HELIX converts added model capacity and compute into consistent ranking-quality gains.The analyses are presented as evidence for scaling both modeling axes together.Source: §1 · p. 3
  • HELIX deployment in TikTok’s e-commerce video recommendation system yields significant offline and online improvements.The introduction identifies offline and online gains without specifying their metric values here.Source: §1 · p. 3

2 HELIX

HELIX scores candidate videos using non-sequential features and two types of behavior histories. Its architecture tokenizes these inputs separately, interleaves sequence retrieval with feature interaction, and retains a candidate-independent user-history cache for reuse.

2 HELIX

  • For each user and candidate video, HELIX predicts multiple click-, conversion-, and GMV-related targets from non-sequential features and user and candidate-conditioned behavior histories.The task setting is fine ranking in an e-commerce video recommendation system.Source: §2 · p. 4

2.1 Architecture Overview

  • HELIX maintains separate mix-token and sequence-token streams, then interleaves retrieval from candidate-conditioned histories and a reusable user-side cache with feature interaction.Information flows from user-side sequence states to candidate-side mix-tokens, allowing the cache to be shared across candidates in a request and across user samples during training.Source: §2.1 · p. 4

2.2.1 MixTokenizer

  • The MixTokenizer combines non-sequential feature embeddings with mean-pooled sequence summaries, then forms mix-tokens using independently parameterized per-token projections and normalization.Per-token parameterization preserves heterogeneous feature-group semantics and distributions.Source: §2.2.1 · pp. 4–5

2.2.2 SeqTokenizer

  • The SeqTokenizer maps each sequence separately and adds sequence-specific positions before concatenation, avoiding a global positional order across different sequence types.U-only histories can be amortized across candidates or samples, unlike candidate-conditioned U×C sequences.Source: §2.2.2 · p. 5

2.3 MixFormer

MixFormer alternates candidate-relevant sequence retrieval with feature interaction among mix-tokens. It uses cross-attention for retrieval and Mixup-PerToken-FFN for heterogeneous feature interaction.

2.3 MixFormer

  • Each MixFormer block applies cross-attention to retrieve candidate-relevant information from U×C sequences, followed by MPTF to interact among mix-tokens.Heterogeneous mix-tokens are normalized before entering the block stack, with no output normalization after the final block because the decoder normalizes its input.Source: §2.3 · p. 5

2.3.1 Cross-Attention

  • HELIX attention uses QK normalization, grouped-query attention, and gating; cross-attention takes mix-tokens as queries and U×C sequence tokens as keys and values.Query projections are parameterized per mix-token, while U×C tokens use position-wise K/V/G projections.Source: §2.3.1 · pp. 5–6

2.3.2 Mixup-PerToken-FFN

  • MPTF mixes channel chunks across input tokens and applies a token-wise SwiGLU FFN, enabling explicit cross-token communication without self-attention’s quadratic attention map.The token-wise projections retain feature-group heterogeneity, and stacking MPTF with cross-attention interleaves feature interaction and sequence retrieval.Source: §2.3.2 · p. 6

2.4 SeqFormer

SeqFormer separates user-history encoding from candidate-side decoding. Its encoder builds a reusable user-side K/V cache, which decoder mix-tokens repeatedly query while alternating retrieval and feature interaction.

2.4 SeqFormer

  • SeqFormer encodes U-only histories into a reusable user-side K/V cache, which candidate-side mix-tokens can repeatedly retrieve from.The decoder alternates cache retrieval with MPTF feature interaction, then mean-pools the representation for task-specific prediction towers.Source: §2.4 · p. 6, §2.4.2 · p. 7

2.4.1 SeqFormer Encoder

  • The encoder removes padded positions before its blocks and progressively reduces query states while retaining current states as keys and values.Gather selects recent valid states under per-sequence token budgets, later blocks retain a shrinking prefix, and the final block projects retained states into the K/V cache.Source: §2.4.1 · pp. 6–7

2.4.2 SeqFormer Decoder

  • The decoder reuses cached keys and gated values without reprojecting them, while progressively refined mix-token queries retrieve richer patterns from the same cache.Attention handles sequence retrieval and MPTF handles feature interaction; alternating them prevents mix-tokens from over-attending to themselves and weakening retrieval.Source: §2.4.2 · p. 7

3 Experiments

Experiments compare HELIX with production ranking architectures offline, examine scaling and design choices, and evaluate HELIX-L in an online A/B test. HELIX outperforms the compared architectures across CTR and CVR metrics, scales predictably, and improves online business metrics.

3.1 Experimental Setup

  • Offline evaluation uses an anonymized TikTok e-commerce video dataset, with CTR and CVR reported as representative classification tasks.Quality is measured by AUC and UAUC; efficiency by non-embedding parameters and pre-amortization per-sample training FLOPs.Source: §3.1 · p. 8

3.2 RQ1: Overall Offline Performance

  • HELIX outperforms all compared architectures across CTR and CVR metrics in the internal-dataset comparison.Table 1 reports relative quality lifts over RankMixer + Transformer; parameters and per-sample training FLOPs are absolute.Source: §3.2 · p. 8, Table 1 · p. 8

3.3 RQ2: Scaling Behavior

  • CTR and CVR AUC improve smoothly with compute and remain unsaturated at HELIX-XXL; RankMixer falls behind as compute grows, while OneTrans trails HELIX-L despite using more compute.HELIX scales width, depth, and retained sequence-token budget across S/M/L/XL/XXL; Figure 2 fits power-law trends to HELIX configurations.Source: §3.3 · pp. 8–9, Figure 2 · p. 9

3.4 RQ3: Which Design Choices Matter?

  • At matched parameter and compute budgets, MPTF improves CTR and CVR AUC by 0.05% and 0.02%, respectively, over self-attention.The final design also uses gated attention after the value projection.Source: §3.4 · p. 10, Table 3 · p. 10
  • K/V-cache-only cross-attention yields a 0.02% AUC gain on both CTR and CVR; the final-layer cache improves CTR AUC by 0.01% while keeping CVR AUC unchanged.The final-layer cache matches all-layer retrieval in ranking quality while reducing memory overhead and allowing asymmetric scaling of sequence modeling and feature interaction.Source: §3.4 · p. 10, Table 3 · p. 10
  • Non-parametric Gate-Up-Norm stabilizes training and improves CTR AUC by 0.02% and CVR AUC by 0.01%.Value-side gating improves CTR and CVR AUC by 0.03% and 0.02%, respectively; QK-Norm improves them by 0.03% and 0.01%.Source: §3.4 · p. 11

3.5 RQ4: Online Impact

  • In a randomized full-traffic A/B test, HELIX-L raises video GMV per user by 5.9932% against the production baseline.It also lifts video clicks per user by 2.0591%, video paid orders per user by 6.0964%, and video GPM by 5.4624%.Source: §3.5 · p. 11, Table 4 · p. 11

4 Training and Serving Optimizations

HELIX reduces sequence-modeling costs through query compression, padding-free encoder execution, and reuse of candidate-independent user-side computation. These optimizations support efficient scaling in both training and serving.

4 Training and Serving Optimizations

  • The optimization strategy combines pyramidal query compression and padding removal with candidate-independent reuse in serving and training.M-FALCON shares the K/V cache across candidates; user-level training amortizes encoder computation across samples from the same user.Source: §4 · p. 11

4.1 Pyramidal Query Compression

  • Per-sequence query budgets retain recent valid states, and later blocks progressively shorten the query prefix while attending to all current-layer states.Recent or high-intensity actions tend to be more informative; selected queries in the first block can still absorb evidence from unselected states.Source: §4.1 · pp. 11–12

4.2 Variable-Length Sequence Execution

  • The encoder packs valid states once, avoids processing padding between blocks, and pads the resulting cache to a fixed maximum length for decoder retrieval.Fused variable-length kernels and the variable-length attention interface support this path; the decoder receives a static-shape cache.Source: §4.2 · p. 12

4.3 User-Side Computation Amortization

  • Reusing the encoder cache changes the cost for n instances from n encoder evaluations to one, while MixFormer and Decoder computation remain instance-specific.M-FALCON maps candidate queries directly to the shared cache during serving, while user-level training evaluates the U-only encoder once for samples grouped by user.Source: §4.3 · pp. 12–13
  • At p = 0.95 and n = 100, theoretical speedup reaches approximately 16.8× under the Amdahl-form amortization model.Here, p is the amortizable fraction attributable to the SeqFormer Encoder; increasing that fraction through asymmetric modeling raises potential speedup.Source: §4.3 · p. 13

5 Related Work

Prior recommendation models developed feature interaction and sequence modeling as complementary scaling directions. HELIX follows unified architectures but assigns retrieval and interaction distinct roles while retaining reusable user-side computation.

5 Related Work

  • Feature-interaction methods evolved from classical factorization and cross networks [7] [18] to tokenized interaction backbones, alongside transformer-style sequence models [3] [23].The passage also names Wukong, RankMixer, TokenMixer-Large, and UniMixer [8] [11] [24] [28] among interaction backbones.Source: §5 · p. 13
  • Unified architectures connect the two modeling axes; HELIX separates repeated sequence retrieval from mix-token interaction while keeping its user-side K/V cache reusable [10] [15] [17] [22] [25].This design supports practical amortization through M-FALCON and user-level training, while allowing asymmetric scaling.Source: §5 · p. 13

6 Conclusion

HELIX jointly scales feature interaction and sequence modeling through interleaved retrieval and interaction with one-way information flow. The conclusion reports predictable offline scaling behavior and meaningful online business impact in TikTok e-commerce video recommendations.

6 Conclusion

  • HELIX interleaves dedicated sequence retrieval with heterogeneous feature interaction and directs information from reusable user-side states to candidate-conditioned mix-tokens.This lets mix-tokens benefit repeatedly from deep user-history modeling while preserving reusable user-side computation.Source: §6 · p. 13
  • Pyramidal encoding, a candidate-independent K/V cache, padding-free execution, M-FALCON, and user-level training support asymmetric scaling and reuse across training and serving.The cache can also be reused across generative retrieval, pre-ranking, and ranking stages.Source: §6 · pp. 13–14
  • Offline experiments demonstrate predictable scaling behavior, and deployment in TikTok’s e-commerce video recommendation system confirms online business impact.The conclusion characterizes HELIX as an effective and production-ready approach to scaling recommendation ranking along both modeling axes.Source: §6 · p. 14

B Detailed Online Metrics

Table 5 presents a representative subset of online A/B-test relative lifts for HELIX-L against the RankMixer + Transformer production control.

B Detailed Online Metrics

  • Table 5 covers relative lifts across business conversion, customer growth and retention, click-through, and downstream engagement; higher values are better.The passage identifies the selected metric categories but provides no individual table values.Source: Table 5 · p. 17, B Detailed Online Metrics · p. 17

Related papers

Loading 2609.37183v2…