Source-linked AI summary
OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender
Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, Aixin Sun
TL;DR
Existing recommender architectures often scale sequence modeling and feature interaction separately, limiting their unified treatment. OneTrans combines them in a single tokenized Transformer with mixed parameterization and efficiency mechanisms, and reports strong industrial-scale performance with online GMV gains. Its scaling beyond strict online p99 latency constraints remains limited.
Problem
RecSys progress has largely treated sequence modeling and feature interaction as separate modules, restricting bidirectional exchange and unified optimization.
Method
OneTrans uses a unified tokenizer and Transformer backbone, mixed parameterization, pyramid pruning, and cross-request KV caching for joint modeling and efficient serving.
Results
OneTrans demonstrates near log-linear performance gains with increasing model size and achieves a 5.68% lift in per-user GMV in online A/B tests.
Takeaways & Limitations
OneTrans provides a single scalable architecture for jointly modeling sequential and non-sequential features in industrial recommendation ranking.
Takeaways & Limitations
Scaling substantially beyond strict online p99 latency constraints remains limited by online efficiency, leaving further system–model co-optimization for future work.
Abstract
from arXiv · showhide
In recommendation systems, scaling up feature-interaction modules (e.g., Wukong, RankMixer) or user-behavior sequence modules (e.g., LONGER) has achieved notable success. However, these efforts typically proceed on separate tracks, which not only hinders bidirectional information exchange but also prevents unified optimization and scaling. In this paper, we propose OneTrans, a unified Transformer backbone that simultaneously performs user-behavior sequence modeling and feature interaction. OneTrans employs a unified tokenizer to convert both sequential and non-sequential attributes into a single token sequence. The stacked OneTrans blocks share parameters across similar sequential tokens while assigning token-specific parameters to non-sequential tokens. Through causal attention and cross-request KV caching, OneTrans enables precomputation and caching of intermediate representations, significantly reducing computational costs during both training and inference. Experimental results on industrial-scale datasets demonstrate that OneTrans scales efficiently with increasing parameters, consistently outperforms strong baselines, and yields a 5.68% lift in per-user GMV in online A/B tests.
1 Introduction
Industrial recommenders commonly separate sequence modeling from feature interaction, limiting information exchange between sequential and non-sequential features. OneTrans unifies both functions in a scalable Transformer backbone with caching and mixed parameterization.
- Industrial ranking systems typically encode user behavior sequences, merge non-sequential features, and apply a separate feature-interaction module.
- Separate modules restrict bidirectional information flow and hinder unified optimization between sequence modeling and feature interaction.
- OneTrans jointly models sequential and non-sequential features within a unified Transformer backbone and single token sequence.
- Sequential tokens share Q/K/V and FFN weights, while each non-sequential token receives token-specific parameters.
- Cross-candidate and cross-request KV caching reduces complexity from O(C) to O(1) for sessions with C candidates.
- OneTrans reports near log-linear gains with model size and achieves statistically significant online improvements in production.
2 Related Work
Related RecSys work has developed sequence modeling and feature interaction largely as separate lines, with recent approaches seeking stronger scaling or cross-module exchange. OneTrans is positioned as a unified alternative to these designs.
- Early sequence models compress histories into fixed-length vectors, while self-attentive methods model dependencies across full user histories.
- Classical feature-interaction models provide efficient bounded interactions, but quality can plateau as additional cross layers are stacked.
- OneTrans uses a unified causal Transformer stack to combine sequential and non-sequential feature processing.
- InterFormer enables mutual signal exchange but retains separate modules and introduces architectural complexity and fragmented execution.
- Generative Recommenders model recommendation as sequential transduction, complementing DLRMs that use rich non-sequential features.
3 Methodology
The paper considers personalized ranking after candidate recall in a cascaded industrial recommender. The ranking model predicts candidate-specific scores from user history, non-sequential features, and trainable parameters.
- The ranking stage receives a candidate set, typically containing hundreds of items, after the recall stage.
- For each user–candidate pair, the ranking model predicts a score using historical behavior sequences and non-sequential features.
- Non-sequential features come from the user, candidate item, and context, while common targets include CTR and CVR.
3.1 OneTrans Framework Overview
OneTrans tokenizes sequential and non-sequential features together and processes them through a pyramid of causal Transformer blocks. Mixed parameterization and progressive sequence-token truncation support joint interactions and efficient computation.
- 3.1 OneTrans Framework Overview: A unified tokenizer maps sequential features to S-tokens and non-sequential features to NS-tokens in one computation graph.
- 3.1 OneTrans Framework Overview: The initial sequence concatenates L_S S-tokens and L_NS NS-tokens of dimensionality d, with learnable [SEP] tokens delimiting behavior sequences.
- 3.1 OneTrans Framework Overview: Mixed attention and feed-forward layers share weights across sequential tokens while assigning separate parameters to non-sequential tokens.
- 3.1 OneTrans Framework Overview: Causal attention lets NS-tokens attend over the full S-token history, enabling cross-token interaction before prediction heads consume final states.
- 3.1 OneTrans Framework Overview: Pyramid-style tail truncation progressively distills sequence information into a shorter representation aligned with non-sequential tokens.
- 3.1 OneTrans Framework Overview: The unified formulation supports intra-sequence, cross-sequence, multi-source, and sequence-feature interactions within one architecture.
3.2 Features and Tokenization
OneTrans converts sequential behavior data and non-sequential user, item, and context features into a unified token sequence. Separate tokenization and sequence-merging choices accommodate heterogeneous inputs and available timestamps.
- Feature preprocessing: OneTrans preprocesses raw inputs into embeddings, separates sequential and non-sequential features, and applies separate tokenizers.Non-sequential features represent user, item, or context attributes.
- Non-sequential tokenization: Non-sequential features include numerical and categorical inputs that are bucketized or one-hot encoded before embedding.The design produces L_NS non-sequential tokens of dimensionality d.
- Non-sequential tokenization: The Group-wise Tokenizer partitions features into semantic groups, whereas the Auto-Split Tokenizer projects all features once and then splits them.Auto-Split reduces kernel-launch overhead through a single dense projection; both choices are evaluated experimentally.
- Sequential tokenization: Sequential events combine item IDs with side information, and each behavior sequence uses a shared projection MLP before merging.Sequences may have different raw dimensionalities.
- Sequential tokenization: Aligned behavior sequences are merged either by timestamp-aware interleaving or timestamp-agnostic impact ordering with separator tokens.When timestamps are available, the timestamp-aware rule outperforms impact ordering in ablations.
3.3 OneTrans Block
Each OneTrans block is a causal Transformer that combines sequential and non-sequential tokens while using mixed parameterization. Shared parameters serve sequential tokens, while non-sequential tokens receive token-specific projections and feed-forward networks.
- Block structure: OneTrans blocks apply pre-norm causal Transformers to sequential S-tokens followed by non-sequential NS-tokens.RMSNorm is applied before processing to align token-type scales and stabilize optimization.
- Mixed parameterization: Sequential tokens share Q/K/V projections, whereas each non-sequential token uses token-specific projections.This mixed parameterization changes how Q/K/V are formed relative to standard multi-head attention.
- Causal attention: Causal masking lets each sequential token attend to earlier sequence positions, while each non-sequential token attends to the entire behavior history.For intent-ordered sequences, earlier high-intent behaviors can inform later behaviors.
- Mixed parameterization: The feed-forward network uses token-specific parameters for NS-tokens and a shared feed-forward network for S-tokens.The same shared-versus-token-specific strategy applies across the block's feed-forward layers.
- Unified block: OneTrans retains a single causal Transformer computation while allowing NS-tokens to aggregate behavior history through mixed QKV and FFN parameterization.The unified causal mask ties sequential and non-sequential tokens together.
3.4 Pyramid Stack
The pyramid stack progressively reduces the number of queried tokens across layers while computing keys and values over the full sequence. This concentrates processing on recent positions and reduces computation and memory.
- Pyramid schedule: At each block layer, only the most recent S-tokens issue queries, while keys and values are computed over the full sequence.The query set shrinks with depth.
- Pyramid schedule: The tail query set contains positions L−L′+1 through L, with L′ ≤ L.Only outputs for this tail set are retained after attention.
- Pyramid schedule: Retaining only tail-query outputs reduces token length to L′ and forms a pyramidal hierarchy across layers.The hierarchy follows the progressively shrinking query set.
- Benefits: The pyramid provides progressive distillation by funneling long behavior histories into a smaller tail of queries and consolidating information into NS-tokens.The design focuses capacity on the most informative events.
- Benefits: Attention cost becomes O(LL′d), FFN cost scales linearly with L′, and shrinking queries reduces FLOPs and activation memory.These savings arise from reducing the query set directly.
3.5 Training and Deployment Optimization
OneTrans reduces deployment cost by reusing shared request-side computations across candidates and requests, while memory-efficient attention and precision techniques support training and inference. Candidate-specific sequences are handled separately when they cannot reuse the shared cache.
- Cross-candidate caching: Samples from one request share S-tokens across candidates, while NS-tokens vary with the candidate item.This structure motivates a two-stage KV-caching design.
- Cross-candidate caching: Stage I processes S-tokens once per request and caches their key/value pairs and attention outputs.Causal masking is applied during this shared stage.
- Cross-candidate caching: Stage II computes candidate-specific NS-tokens, cross-attends to cached S-side keys and values, and applies token-specific FFN layers.Sequences such as SIM are pre-aggregated into NS-tokens because they cannot reuse the shared cache.
- Cross-candidate caching: KV caching amortizes S-side computation across candidates, keeping per-candidate work lightweight and improving throughput.The method eliminates redundant computations across candidates.
- Cross-request caching: Because behavior sequences are append-only, cross-request caching reduces sequence computation from O(L) to O(ΔL).ΔL denotes newly added behaviors since the previous request.
- Memory and throughput: FlashAttention-2, mixed-precision training, and activation recomputation reduce memory pressure and improve throughput at the cost of modest extra recomputation.FlashAttention-2 addresses attention I/O and quadratic activation footprint.
4 Experiments
The experiments evaluate OneTrans in large-scale industrial ranking against matched-compute baselines, using offline effectiveness, efficiency, ablations, and scaling analyses. OneTrans improves ranking metrics over separated sequence-modeling and feature-interaction systems while supporting unified design choices and scaling.
- 4.1 Dataset: Offline evaluation uses chronologically split production logs with impression-time feature snapshots, anonymized identifiers, and fixed-window click and order labels.CTR and CVR are evaluated using AUC and impression-weighted user-level AUC (UAUC), with daily next-batch predictions macro-averaged across days.
- 4.2 RQ1: Performance Evaluation: Baselines progressively strengthen feature interaction from DCNv2 through Wukong, HiFormer, and RankMixer, then strengthen sequence modeling from StackDIN through Transformer and LONGER.The comparisons use the same features and matched compute budgets, anchored on the production baseline DCNv2+DIN.
- 4.2 RQ1: Performance Evaluation: +1.13%/+1.77% CTR AUC/UAUC and +0.90%/+1.66% CVR AUC/UAUC are achieved by OneTransS over the production baseline.At comparable scale, OneTransS also outperforms RankMixer+Transformer with similar training FLOPs: 2.64T versus 2.51T.
- 4.2 RQ1: Performance Evaluation: +1.53%/+2.79% CTR AUC/UAUC and +1.14%/+3.23% CVR AUC/UAUC are achieved by OneTransL, the strongest reported overall improvement.The larger model uses 8 layers with width d=384, while OneTransS uses 6 blocks with width d=256 and targets approximately 100M parameters.
- 4.4 RQ3: Systems Efficiency: OneTrans unifies sequence modeling and feature interaction, yielding more reliable and compute-efficient improvements than scaling either component independently.The experiments also examine pyramid stacking, cross-request KV caching, and scaling behavior as part of the systems-efficiency questions.
- 4.3 RQ2: Design Choices via Ablation Study: Ablations test tokenizer, sequence fusion, [SEP] tokens, token-specific parameters, attention type, and pyramid stacking against OneTransS.The reported findings favor Auto-Split tokenization, timestamp-aware fusion, learnable [SEP] tokens when timestamps are unavailable, and token-specific parameters for non-sequential tokens.
4.4 RQ3: Systems Efficiency
OneTrans combines scaling-law analysis with system optimizations and online evaluation to assess whether larger unified models remain efficient. The results show strong scaling behavior, production-grade efficiency, and substantial business gains.
- Systems optimizations: Pyramid stacking and cross-request KV caching reduce training and serving costs by pruning sequential tokens and reusing sequence computation.FlashAttention improves training efficiency, while mixed precision with recomputation provides the largest serving gains.
- Systems efficiency: OneTransL maintains online efficiency comparable to the much smaller DCNv2+DIN baseline despite its larger unified Transformer backbone.This supports direct adoption of LLM-style optimizations in the scaled model.
- Scaling laws: Increasing input sequence length yields the largest gains, while depth generally improves performance more than width at the cost of greater serial computation.The depth–width choice should balance performance benefits against hardware-budget efficiency.
- Scaling laws: OneTrans and RankMixer show log-linear ΔUAUC trends, but OneTrans has the steeper slope and favorable performance–compute trade-offs.The comparison scales RankMixer+Transformer on the RankMixer side to 1B parameters.
- Limitations: Scaling substantially beyond the deployable OneTransL regime remains constrained by online efficiency, motivating future system–model co-optimizations.The authors report that further scaling is left to future work.
- Online evaluation: OneTransL delivers +5.6848% gmv/u and −3.91% latency in Feeds versus the RankMixer+Transformer control.In Mall, it achieves +3.6696% gmv/u and −3.26% latency.
5 Conclusion
OneTrans is presented as a unified Transformer backbone for personalized ranking that jointly models sequential and non-sequential features. Its design combines unified tokenization, mixed parameterization, efficiency mechanisms, and LLM-style optimizations for scalable deployment.
- Conclusion: OneTrans replaces encode–then–interaction with a unified Transformer backbone for personalized ranking.The backbone jointly performs sequence modeling and feature interaction.
- Conclusion: A unified tokenizer maps sequential and non-sequential attributes into one token sequence, with shared parameters for sequential tokens and token-specific parameters for non-sequential tokens.This mixed parameterization is part of the unified design.
- Conclusion: Pyramid scheduling, cross-request KV caching, FlashAttention, and mixed precision make the unified stack efficient at scale.The conclusion describes these mechanisms as supporting scalable recommender deployment.