PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds

Yifeng Zhou, Yuehong Hu, Zhixiang Feng, Junwei Pan, Kaihui Wu, Hanyong Li, Shangyu Zhang, Shudong Huang, Zhangbin Zhu, Chengguo Yin, Haijie Gu, Jie Jiang

arXiv:2604.13737v1cs.IRcs.AI

TL;DR

Recommendation must combine sparse static signals with evolving behavior histories, but naive unification can collapse sequential representations. TokenFormer unifies these inputs using BFTS and NLIR; experiments report state-of-the-art accuracy and improved dimensional robustness and discriminability.

  • Problem

    Unified recommendation must use sparse static signals alongside evolving behavior histories, yet naive joint modeling can collapse sequential representations.

    Source: §1 · pp. 1–2

  • Method

    TokenFormer places static fields, behavior tokens, and targets in one stream, combining BFTS attention with NLIR gated multiplicative interactions.

    Source: §1 · p. 2, §4.4 · p. 5

  • Results

    TokenFormer achieves state-of-the-art accuracy in experiments and Tencent Ads deployment while recovering representation dimensionality and improving discriminability.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results indicate that unified modeling can use informative static-feature priors while mitigating their collapse of behavioral representations.

    Source: §1 · p. 2

    Scaling plateaus beyond TokenFormer-L on KuaiRand-27K, where limited samples are insufficient to regularize the larger model and marginal overfitting appears.

    Source: §5.8 · p. 11

Abstract

from arXiv · show

Recommender systems have historically developed along two largely independent paradigms: feature interaction models for modeling correlations among multi-field categorical features, and sequential models for capturing user behavior dynamics from historical interaction sequences. Although recent trends attempt to bridge these paradigms within shared backbones, we empirically reveal that naive unifying these two branches may lead to a failure mode of Sequential Collapse Propagation (SCP). That is, the interaction with those dimensionally ill non-sequence fields leads to the dimensional collapse of the sequence features. To overcome this challenge, we propose TokenFormer, a unified recommendation architecture with the following innovations. First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers. Second, we introduce a Non-Linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states. Extensive experiments on public benchmarks and Tencent's advertising platform demonstrate state-of-the-art performance, while detailed analysis confirm that TokenFormer significantly improves dimensional robustness and representation discriminability under unified modeling.

1 Introduction

Unified recommendation must combine static fields with behavior histories without sacrificing sequential representations. TokenFormer addresses this tension with BFTS and NLIR, and experiments report improved accuracy and reduced dimensional collapse.

1 Introduction

  • Recommendation must combine sparse user, item, and context signals with evolving behavior histories, while existing approaches often retain internal separation.The paper identifies a need for native field-field, sequence-sequence, and sequence-field interactions in one computational manifold.Source: §1 · p. 1
  • Naive unification improves mutual-information discriminability but steepens sequential spectral decay and lowers effective rank, exposing Sequential Collapse Propagation.Figure 1 contrasts a sequence-only Transformer, vanilla unification, and the proposed designs.Source: Figure 1 · p. 1, §1 · p. 2
  • TokenFormer unifies static fields, behavior tokens, and target attributes, using BFTS and NLIR to improve dimensional robustness and discriminability.BFTS uses full attention in lower layers and shrinking-window sliding attention in higher layers; NLIR applies gated multiplicative interactions.Source: §1 · p. 2
  • Across public datasets and Tencent Ads, TokenFormer improves recommendation accuracy while substantially mitigating dimensional collapse under unified modeling.The analyses cover attention patterns, dimensional robustness, and representation discriminability.Source: §1 · p. 2

2 Related Work

Related work develops feature interaction and sequential modeling along separate trajectories, while studying representation collapse and exploring increasingly unified architectures. TokenFormer frames both paradigms within a shared interaction template.

2.1 Feature Interaction and Sequential Modeling

  • Feature-interaction models progressed from factorization and neural crossings toward dense-scaling backbones, while sequence models advanced through attention, target-aware methods, long-history systems, and sliding-window attention.The related work also notes growing use of gating and a shared motif of relevance, context aggregation, and gated interaction.Source: §2.1 · p. 2

2.2 Representation Collapse in Recommendation

  • Representation collapse places embeddings in a low-dimensional subspace, reducing item or user discriminability; prior work links collapse and feature entanglement to degraded recommendation quality [31].Nonlinear interaction operators are described as a basis for preserving representation rank during feature crossing [45].Source: §2.2 · p. 2

2.3 Toward Unified Modeling of Fields and Sequences

  • Hybrid systems and recent unification methods often retain token-type separation, whereas TokenFormer treats fields and behaviors as one stream under shared gated attention.The shared formulation subsumes feature interaction and sequential attention as projection, relevance weighting, and context mixing.Source: §2.3 · pp. 2–3

3 Problem Setup and Preliminaries

The setup groups recommendation inputs into non-sequential fields, behavior tokens, and target features, then views existing models as covering different subsets of their interactions.

3 Problem Setup and Preliminaries

  • Recommendation inputs comprise non-sequential fields F, sequential behavior tokens T, and target features V, with existing models typically operating on only part of this entity space.Feature-interaction models mainly consume (F, V), while sequential models mainly consume T or (T, V).Source: §3 · p. 3
  • The setup distinguishes field-target interactions, behavior self-attention, target attention over behaviors, and cross-feature F↔T interactions before proposing interactions over F∪T∪V.The cited examples include FM-family feature interactions [11] [18] [30] [44], Transformer-style sequential models [5] [23] [40] [47], and target-aware models [3] [9] [54] [55] [56].Source: §3 · p. 3

4 TokenFormer

TokenFormer casts static fields, behavior histories, and target features into one token stream processed by stacked unified interaction blocks. Layer-dependent attention and multiplicative interaction shape how those blocks integrate and refine representations.

4.1 Unified Token Stream

  • The unified backbone models intra-group and inter-group interactions across static fields, behavior tokens, and target features in one architecture.The interaction set includes F↔F, T↔T, V↔V, T↔V, F↔T, and F↔V.Source: §4.1 · p. 3

4.1.1 Unified Entity Stream.

  • Input length comprises field, history, target, and delimiter tokens; unified RoPE assigns static fields a shared prefix and behavior tokens chronological positions.The positional scheme is described as preserving static-feature invariance while retaining behavioral temporal evolution.Source: §4.1.1 · pp. 3–4

4.2 Unified Interaction Block

  • Each decoder-only Unified Interaction Block applies BFTS-masked attention and then nonlinear interaction to its attention output.TokenFormer stacks these blocks in a homogeneous backbone.Source: §4.2 · p. 4

4.3 BFTS Attention Mechanism

  • Shallow full attention builds global interactions before deeper shrinking windows localize refinement; non-sequence tokens are discarded after layer l_f.The deeper layers then devote attention bandwidth to behavioral evolution and target-aware reasoning.Source: §4.3 · p. 5

4.4 Non-Linear Interacted Representation

  • Sigmoid multiplicative gating of attention output adds high-order nonlinearity and is described as preserving feature-space rank richness.The resulting representation is intended to be more expressive for the subsequent stage.Source: §4.4 · p. 5

4.5 SwiGLU Feed-Forward Network.

  • After interaction, RMSNorm and a SwiGLU feed-forward stage retain a residual update consistent with the multiplicative interaction principle.For supervised positions, outputs are projected to action logits, softmax-normalized, and trained with cross-entropy.Source: §4.5 · pp. 5–6, §4.6 · p. 6

4.6 Unified Optimization Objectives

  • User-Centric training supervises densely across the stream, whereas New Impression Only applies loss to current candidate or terminal tokens and uses history as context.In New Impression Only, historical interactions do not contribute to loss gradients.Source: §4.6 · p. 6

5 Experiments

The experiments evaluate TokenFormer on KuaiRand-27K and Tencent advertising scenarios, examining recommendation performance, representation quality, attention allocation, efficiency, and scaling. Results report gains over baselines alongside evidence that BFTS and NLIR mitigate dimensional collapse under unified modeling.

5.2 Overall Performance Comparison

  • TokenFormer-Tiny exceeds the sequence-preserving Transformer by 5.00‰ AUC and HSTU-Ultra by 2.05‰ on KuaiRand-27K.The study also evaluates a 19k/2k/5k train-validation-test split and Tencent Ads datasets with billions of interaction logs.Source: §5.1 · p. 6, Table 1 · p. 7, §5.2 · p. 7

5.3 Impact of NLIR and BFTS on Representation Expressiveness

  • BFTS and NLIR consistently improve scaled MI across nearly all cluster counts, with their advantage widening at higher K.Static attributes also improve discriminability over sequence-only modeling, while naive joint modeling risks SCP.Source: §5.3 · pp. 7–8, Figure 3 · p. 7

5.4 Impact of NLIR and BFTS on Dimensional Robustness

  • Both BFTS and NLIR independently recover effective rank; BFTS limits low-rank noise propagation, while NLIR promotes feature decorrelation.Sequential-token rank declines with depth in both models, but the decline is substantially mitigated by NLIR.Source: §5.4 · pp. 8–9, Figure 5 · p. 8
  • Attention residuals consistently lift effective rank, while FFN residuals generally pull it toward an intermediate level.The residual effects are seen across measured stages and modulate the rank profile established by attention.Source: §5.4 · p. 8

5.5 Layer-wise Attention Analysis in BFTS

  • Vanilla Transformer revisits static tokens in deep layers, whereas BFTS uses broad shallow interactions and excludes non-sequential positions in deep layers.Shallow-layer interaction with non-sequential tokens is 52.7 versus 40.0 for Vanilla Transformer; its average receptive field expands from 40.0 to 46.4 late in the baseline.Source: Figure 6 · p. 9, Figure 7 · p. 9, §5.5 · pp. 9–10

5.6 Efficiency and Effectiveness of BFTS

  • The 2F2S configuration beats 4F by 0.85‰ AUC while reducing GFLOPs by 201.0‰; reversing the order to 2S2F lowers AUC by 6.18‰.Shrinking windows outperform uniform windows by 1.53‰ AUC.Source: Figure 8 · p. 10, §5.6 · p. 10

5.7 Ablation Study

  • NLIR alone adds +4.87‰ AUC and BFTS +4.91‰ over Vanilla Transformer, while all-layer SWA (4S) loses 36.35‰.In Tencent Ads, BFTS shifts relative AUC from -0.16% for full attention to +0.14%, and adding NLIR reaches +0.22%; online GMV rises 4.03%.Source: §5.7 · pp. 10–11, §5.9 · p. 11

5.8 Model Scaling and Empirical Observations

  • Performance improves as TokenFormer scales from Tiny to Large on KuaiRand-27K, but plateaus beyond TokenFormer-L; Tencent Ads scaling continues without premature saturation.The paper attributes the public-data plateau to limited sample volume and marginal overfitting of higher-capacity models.Source: §5.8 · p. 11

6 Conclusion

TokenFormer addresses Sequential Collapse Propagation in unified recommendation by combining BFTS attention with NLIR. Experiments and Tencent Ads deployment report state-of-the-art accuracy and recovery of representation dimensionality.

6 Conclusion

  • TokenFormer combines a bottom-full-top-sliding attention hierarchy with a non-linear interaction representation mechanism to address Sequential Collapse Propagation.The architecture unifies static fields, behavior tokens, and target attributes in one stream.Source: §6 · p. 11
  • Experiments and online deployment in Tencent Ads show state-of-the-art accuracy and recovery of the representation manifold’s intrinsic dimensionality.Source: §6 · p. 11
  • The paper presents TokenFormer as a robust blueprint for moving from heterogeneous expert ensembles toward unified recommendation backbones.Source: §6 · p. 11

A Complexity Analysis and Serving Optimization

BFTS reduces upper-layer attention cost through local windows while retaining global modeling in lower layers, and decoupled serving avoids repeated user-side computation across candidate ads. In online deployment, decoupled inference raises throughput from 126 QPS to 695 QPS.

Complexity analysis

Full attention vs. sliding-window attention

Backbone complexity under BFTS

Serving complexity under joint encoding

Serving complexity under decoupled encoding

  • Decoupled serving encodes user tokens once into N summary tokens, then combines them with each candidate ad for scoring.Its quadratic user-side cost is no longer multiplied by B, and candidate-specific cost depends on N rather than the original user sequence length L_u.Source: A Complexity Analysis and Serving… · p. 13
  • When B > 1 and N < L_u, the decoupled design is asymptotically more efficient than joint encoding.Source: A Complexity Analysis and Serving… · p. 13

Practical serving improvement

  • 126 QPS to 695 QPS, or a 5.5× speedup, is the reported throughput gain from decoupled inference in online deployment.The paper attributes the gain to removing repeated user-side computation together with BFTS architectural sparsification.Source: A Complexity Analysis and Serving… · p. 13

B.1 Mutual Information Diagnostics of Hidden Representations

The appendix describes mutual-information diagnostics for assessing how much task-relevant discriminative information model representations retain for each user action. It applies continuous and cluster-based estimates to extracted representations and aggregates action-level scores.

B.1 Mutual Information Diagnostics of Hidden Representations

  • The diagnostics assess how much task-relevant discriminative information model outputs retain for each user action.They offer complementary continuous and partition-based views of representation information.Source: §B.1 · pp. 13–14

Representation extraction

  • The analysis extracts raw logits, sigmoid-activated outputs, and penultimate-layer hidden states from test samples.Source: §B.1 · p. 13

Multi-label construction

  • Each user action is treated as an independent binary classification label for mutual-information estimation.The task includes actions such as click, like, follow, comment, and forward.Source: §B.1 · p. 13

Continuous MI estimation

  • The KSG estimator measures mutual information directly from continuous representations, avoiding information loss from discretization.Source: §B.1 · p. 14

Discretized MI via KMeans clustering

  • KMeans assigns representations to clusters for a complementary discrete mutual-information estimate, with multiple cluster counts used to assess sensitivity to discretization granularity.Source: §B.1 · p. 14

Aggregation and multi-seed analysis

  • Action-level MI scores are combined by a weighted mean; absent specified weights, positive sample counts are used, and results are averaged across random seeds.Source: §B.1 · p. 14

Block-wise diagnostics

  • Figure 9 tracks sequential-token effective rank through attention output, residual additions, and FFN output within each block.Source: Figure 9 · p. 14

B.2 Spectral Diagnostics of Hidden Representations

The appendix uses effective rank and normalized singular-value spectra to characterize representation geometry and dimensional robustness. It computes the diagnostics from sampled hidden representations across layers and stages within Transformer blocks.

Hidden representation extraction

  • The analysis retains item and candidate-item token states, samples at most 10,000 tokens, and uses hidden dimension 256 in its experiments.Source: §B.2 · p. 14

Singular value decomposition

  • The representation matrix is centered column-wise before singular value decomposition, whose singular values are ordered from largest to smallest.Source: §B.2 · p. 14

Effective rank

  • A larger entropy-based effective rank indicates variance spread across more latent dimensions, whereas a smaller value signals concentration in a low-dimensional subspace.Source: §B.2 · pp. 14–15

Normalized singular value spectrum

  • The normalized spectrum divides each singular value by the largest; slower decay indicates more evenly spread variance, while steeper decay indicates dominance by leading directions.Source: §B.2 · p. 15

Layer-wise and stage-wise analysis

  • Diagnostics are computed per layer and at attention output, FFN output, and final block output stages for intra-block analysis.Source: §B.2 · p. 15

B.2 Spectral Diagnostics of Hidden Representations

  • Effective rank measures subspace utilization, while the normalized singular-value spectrum shows how variance is distributed across singular directions.Together, they characterize dimensional robustness in learned hidden representations.Source: §B.2 · pp. 14–15

C Normalized Singular Value Spectrum Analysis

The appendix compares normalized singular-value spectra to examine how hidden-representation variance is distributed across latent dimensions. TokenFormer’s gated variant has a less top-heavy spectrum than the non-gated variant, complementing the effective-rank analysis.

C Normalized Singular Value Spectrum Analysis

  • The appendix uses normalized singular-value spectra to examine variance distribution across hidden-representation dimensions.This provides a complementary analysis of the representations.Source: C Normalized Singular Value Spectrum… · p. 15
  • Across all layers, gated TokenFormer has a less top-heavy spectrum and heavier tail than the non-gated variant, indicating more even variance distribution.Figure 10 presents the per-layer normalized singular value spectra.Source: C Normalized Singular Value Spectrum… · p. 15, Figure 10 · p. 15
  • The spectral distribution complements effective-rank analysis by showing how variance is allocated within the utilized subspace.Together, the observations support the claim that gating improves dimensional robustness by preserving richer, more balanced representations.Source: C Normalized Singular Value Spectrum… · p. 15

D.1 RoPE-based Positional Awareness

TokenFormer addresses interference from additive positional encodings with multiplicative RoPE, which encodes relative position through hidden-state rotations. The supplied analysis also describes how relative rotations support static-field interactions and motivates gating for dimensional robustness.

D.1 RoPE-based Positional Awareness

  • Additive positional interactions can introduce noise and compromise semantic geometry when high-dimensional states interact with lower-intrinsic-dimensional positional embeddings.The text argues that fine-grained relative dependencies matter for modeling user behavior evolution.Source: §D.1 · p. 15
  • Multiplicative RoPE rotates hidden states so relative distance is encoded in phase, while its orthogonal transformation preserves L2 norm and intrinsic dimensionality.The paper claims this avoids rank collapse while retaining the semantic manifold’s expressive power.Source: §D.1 · pp. 15–16
  • Static field tokens share position, yielding semantic dot-product interactions, while sequence-to-field relative rotations let the model vary static-profile reliance by sequence position.The text relates shared-position field interactions to multiorder Factorization Machine interactions.Source: §D.1 · p. 16

D.2 Gated Mechanism

  • The paper frames gating as supplying multiplicative interactions for sparse ID co-occurrences and theoretically preserving dimensional rank under unified modeling.It connects rank-collapse risk to unifying low-cardinality static features with dynamic sequences.Source: §D.2 · p. 16
Loading 2604.13737v1…