PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDot: A Unified Network for Sequence Modeling and Feature Interaction in Large-scale Recommendation

Rongcheng Lin, Yan Sun, Jamey Zhang, Guanglei Xiong, Ivan Ji, Xianjie Chen, Shujian Bu

arXiv:2608.16797v1cs.IRcs.AI

TL;DR

Large-scale recommenders need a unified architecture for sequential histories and non-sequential feature interactions under latency constraints. UniDot represents both as tokens and combines parallel token-mixing and sequence-retrieval buses with an FM Highway, finishing runner-up with 0.83217 test AUC on the TAAC × KDD Cup 2026 Industrial track.

  • Problem

    Large-scale recommendation seeks one scalable, latency-bounded architecture that jointly models sequential behavior histories and non-sequential feature interactions.

  • Method

    UniDot embeds fields and behavioral events in one token space, then stacks parallel token-mixing and sequence-retrieval buses with FM Highway dot-product interactions.

  • Results

    0.83217 test AUC earned UniDot runner-up on the TAAC × KDD Cup 2026 Industrial track.

  • Takeaways & Limitations

    UniDot provides a single stackable design for jointly modeling sequence behavior and feature interactions in large-scale recommendation.

  • Takeaways & Limitations

    The contribution of the FuseFFN bus fusion is inconclusive because removing it slightly improves AUC while worsening LogLoss.

Abstract

from arXiv · show

Industrial recommenders rely on two model families that have evolved largely independently: feature-interaction models over multi-field user/item features, and sequential models over user-behavior histories. Production systems couple them only loosely. To unify the two, we present UniDot, a novel architecture for post-click conversion prediction built from the factorization-machine (FM) point of view: the embedding inner product---which powers collaborative filtering and lets a recommender generalize to unseen user--item pairs---is the same primitive as attention's query dot key scoring, so a single dot-product of tokens can underlie both feature interaction and sequence modeling. UniDot tokenizes non-sequential fields and multi-domain behavioral sequences into one shared token space and stacks a single macro-block in which a token-mixing bus and a sequence-retrieval bus (item tokens cross-attending the histories) run in parallel and exchange state each layer through an MLP-Mixer fusion, while an FM Highway carries explicit per-layer dot-product interactions around the residual stack directly to the classifier. The sequence side is embedded once per forward pass and shared by all consumers, bounding inference latency. Trained with a dual sparse/dense (Adagrad + Muon) optimizer, an auxiliary conversion-delay head, and multi-path mutual learning, UniDot finished as the runner-up on the Industrial track of the TAAC KDD Cup 2026.

1 Introduction

UniDot addresses the separation between feature-interaction and sequential recommendation models with one shared token space and homogeneous stackable backbone. Its FM-based architecture unified these signals and achieved runner-up status with a final-leaderboard AUC of 0.83217.

  • Motivation: Feature-interaction models learn crosses among mostly static user and item attributes, whereas sequential models capture dynamic user behavior through target-aware attention over histories.The introduction frames these as two largely separate traditions in large-scale recommendation.
  • Motivation: The challenge sought unified tokenization and a homogeneous, stackable backbone for sequential and non-sequential features under an inference-latency budget.UniDot was designed as a direct response using one stackable block and one shared token space.
  • FM Perspective: The FM inner product supports collaborative filtering by generalizing scores to user–item pairs unseen during training, motivating explicit dot-product interactions in UniDot.The introduction connects this property to sparse-positive conversion data and fast-moving ad inventories, where serving-time candidate pairs are often effectively new.
  • Architecture: UniDot stacks L homogeneous blocks in which token-mixing and sequence-retrieval buses exchange state through MLP-Mixer fusion, while an FM Highway routes per-layer dot products to the classifier.The sequence-retrieval bus lets item tokens cross-attend behavioral histories, while the token-mixing bus operates over user and item profile tokens.
  • Result: 0.83217 final-leaderboard AUC earned UniDot runner-up status on the Industrial track of the TAAC × KDD Cup 2026 challenge.The result was achieved without hand-crafted cross features beyond the released schema.

2 Related Work

Prior recommendation work developed feature-interaction models and sequential user-interest models largely as separate lines, followed by architectures that unify them in shared backbones. UniDot differs by preserving explicit collaborative-filtering-style dot products through cross-attention and an FM Highway.

  • Feature interaction: Matrix factorization and Factorization Machines established inner-product modeling for latent user–item affinity and generalization to unseen feature pairs.Factorization Machines extend matrix factorization to arbitrary multi-field features while retaining factorized interactions.
  • Feature interaction: DeepFM, Wide&Deep, xDeepFM, DCN-v2, AutoInt, FiBiNet, Wukong, and DHEN extended explicit feature interactions through shared embeddings, deeper crosses, attention, compression, or heterogeneous experts.These methods kept the inner product central while expanding interaction capacity.
  • Sequential user-interest modeling: DIN, DIEN, DSIN, and BST advanced sequential user-interest modeling with target-aware attention, interest evolution, and self-attention, while SIM, ETA, TWIN, and LONGER addressed long histories through retrieval or compression.The cited sequential models span target-aware pooling, evolution, self-attention, retrieval, and history compression.
  • Unifying feature interaction and sequence modeling: HSTU, InterFormer, HyFormer, OneTrans, and TokenFormer unified feature interaction with sequence modeling through generative sequence backbones, interleaved refinement, cross-attention, or homogeneous token streams.HyFormer cross-attends behavioral history, while OneTrans and TokenFormer fuse attributes, behaviors, and targets into one stream.
  • Unifying feature interaction and sequence modeling: UniDot keeps collaborative-filtering-style dot products explicit: cross-attention is interpreted as FM scoring between query and key tokens, and an FM Highway carries low-order interactions beyond the residual mixer.Prior unified approaches made the collaborative-filtering inner product implicit, whereas UniDot routes explicit interactions through a dedicated highway.

3 Problem Formulation

UniDot formulates post-click conversion prediction from user profiles, candidate items, and four behavioral domains, producing a conversion probability with binary cross-entropy training. Its objective also includes an auxiliary delay loss over rows with a subsequent action, while per-layer explicit interactions are preserved for readout.

  • Problem definition: Each example x = (u, i, S) combines a user profile, candidate item, and four behavioral domains, with binary label y = 1 indicating conversion.User features include IDs, aligned per-position weights, and pre-trained embeddings; item features include IDs and embeddings.
  • Problem definition: The model predicts conversion probability and uses binary cross-entropy as its primary training loss.The formulation fixes d as d_model and defines tokenization from raw inputs into d-dimensional tokens.
  • Readout: Per-layer FM Highway signals are concatenated rather than summed, allowing every layer’s explicit token dot-product interactions to reach the classifier undiluted.The classifier consumes the final-state readout, all highway signals, and a skip-embedding signal.
  • Objective: The auxiliary delay loss regresses log time-to-next-action for every row with a next action, including clicks and conversions rather than only positive examples.The target is log(1 + t_label − t_event), and the regression uses mean squared error.

4 Method: UniDot

UniDot unifies heterogeneous features and behavioral sequences in one shared token space, processed by parallel token-mixing and sequence-retrieval buses with per-layer fusion and FM-style dot-product readouts. Its sequence trunk is computed once and shared across consumers, while dual-path mutual learning improves inference through averaged logits with a near-equivalent single-path option.

  • Unified tokenization: UniDot maps heterogeneous inputs into one shared d-dimensional token space and scales depth by stacking L identical macro-blocks.Inputs include categorical features, multi-value ID lists, pre-trained embeddings, and behavioral sequences; learned token-axis compression forms compact user and item token sets.
  • Unified tokenization: Candidate-aware FAFE pooling makes position-invariant multi-value fields produce different tokens for each ranking candidate.It replaces static NCB pooling with DIN-style attention against the candidate, using a learned combination of field values.
  • Sequence encoder: The sequence encoder merges domains, captures local N-gram structure, filters positions with candidate-conditioned SwiGLU gating, and models causal dependencies with windowed Transformer attention.Its output views H(s) are refreshed by per-consumer projectors, and the sequence pipeline runs once per forward pass for all downstream consumers.
  • Parallel buses and fusion: Each macro-layer runs token-mixing and sequence-retrieval buses in parallel, exchanges information through FuseFFN, and carries explicit dot-product interactions through the FM Highway.The retrieval bus cross-attends item tokens to S sequence views, while the token-mixing bus uses Wukong with explicit pairwise dot-products.
  • Training and serving: 0.83217 two-path mean test AUC exceeds the 0.83184 single-path score, while the single path costs 1× and is only 0.033% below the mean.The two paths share dominant sparse embeddings, jointly train on the same batches, and average their logits at inference.

5 Experiments

UniDot achieved 0.83217 test AUC and finished runner-up, with incremental architectural refinements, dense scaling, and multi-path mutual learning contributing to the final gain. Ablations identify the FM Highway and dense capacity as especially important, while FuseFFN fusion remains inconclusive.

  • Overall results: 0.83217 test AUC earned UniDot the runner-up position, with the final submission retrained on all data using EMA weights and peaking at epoch 5.The result was attributed to the unified dual-bus architecture, FM Highway, depth, width, and multi-path mutual learning.
  • Incremental improvements: +1.10% was the largest single incremental gain from UniDot itself, while subsequent refinements and scaling produced a +1.82% total improvement.The listed refinements included FM-Highway dots, depthwise convolution, auxiliary loss, FAFE, EMA, dense scaling, and multi-path DML.
  • Component ablations: −0.127% AUC resulted from removing the FM Highway, the largest component-ablation cost in tiny-mode evaluation.Depth helped through the default six macro-layers, whereas eight to ten layers overfit the 4M-example set.
  • Component ablations: −0.053% AUC resulted from removing sequence cross-attention, suggesting the multi-channel pool captured much of the same signal.This ablation was conducted in tiny mode with single-path, d=64, in-distribution held-out evaluation, so only relative gaps matter.
  • Component ablations: +0.022% AUC followed removal of FuseFFN bus fusion, but LogLoss worsened, leaving its contribution inconclusive.The paper identifies static routing as the weak link and proposes an input-conditioned second-order fuser as the clearest open direction.
  • Scaling study: +0.135% AUC came from adding a second identical path under mutual distillation at d=64, exceeding the full width sweep’s +0.050% cumulative gain.Widening the dense path from 64 to 96 to 128 improved AUC monotonically with diminishing returns, while doubling embedding width alone produced no A/B win.

6 Conclusion

UniDot unifies sequence modeling and feature interaction in one stackable block by representing fields and behavioral sequences as shared-space tokens, jointly processing them through parallel buses, and routing explicit dot-product interactions to the classifier.

  • Architecture: UniDot unifies sequence modeling and feature interaction within a single stackable block.The passage presents this unified architecture as the paper’s central contribution.
  • Architecture: Non-sequential fields and multi-domain behavioral sequences become tokens in one shared space.This shared tokenization provides the common representation processed by the architecture.
  • Architecture: A parallel token-mixing bus and sequence-retrieval bus co-evolve and exchange deltas through a canonical MLP-Mixer each layer.The two buses jointly process the shared tokens while exchanging state layer by layer.
  • FM Highway: The FM Highway routes explicit dot-product interactions around the fusion path directly to the classifier.This is identified as UniDot’s defining idea in the conclusion passage.

A Dataset · A.1 Dataset

The Industrial-track dataset is a large-scale, fully anonymized Tencent advertising-log benchmark containing non-sequential user/item features and multi-domain behavioral sequences. Its schema includes aligned dense statistics for categorical IDs and extremely high-cardinality fields that directly shape UniDot’s input handling.

  • A Dataset: The challenge provides a large-scale, fully anonymized advertising dataset derived from real Tencent ad logs across two rounds, with competition on the Industrial track.Sparse features are anonymized integer IDs, while dense features are fixed-length float vectors; raw content and personally identifiable information are excluded.
  • A.1 Dataset: Each example represents a user–item interaction with non-sequential inputs and behavioral sequence inputs.The dataset therefore combines static feature fields with historical behavior for each interaction.
  • A.1 Dataset: Non-sequential features include single- and multi-valued user_int and item_int categorical IDs plus user_dense and item_dense continuous vectors.The dense vectors include pre-trained embeddings such as SUM and LMF4Ads on the user side and item embeddings on the item side.
  • A.1 Dataset: Behavioral inputs span four domains, seq_a to seq_d, each containing 9–14 fields and time-ordered events with timestamps and action types.These domain-specific histories provide the sequential component of each user–item example.
  • A.1 Dataset: Table 6 summarizes the Industrial-track dataset and schema for the second round, with first-round figures shown in parentheses.The table is explicitly identified as covering the dataset and schema across both challenge rounds.
  • A.1 Dataset: Table 7 documents the training and evaluation modes used with the dataset.It provides the dataset-related modes separately from the schema summary.
  • A.1 Dataset: Aligned user_dense arrays encode per-element statistics for matching user_int arrays rather than standalone features.Examples include dwell time and score, so UniDot consumes these values as per-position weights on ID embeddings in §C.2.
  • A.1 Dataset: ≈9.4M values are associated with fid 116, illustrating the extreme cardinality of some anonymized ID fields.This schema property directly affects the model’s input design and handling of high-cardinality features.

B Experimental Setup … C.6 Auxiliary conversion-delay head

The experiments use the TAAC × KDD Cup 2026 Industrial-track conversion task with AUC-ROC as the official metric and a dual Adagrad/Muon training setup. Implementation details emphasize reproducibility through feature-specific tokenization, embedding handling, high-cardinality skips, and an auxiliary delay objective.

  • B Experimental Setup: The setup uses 35M training and 12M test examples, reports official ROC-AUC plus LogLoss, and trains with dual Adagrad/Muon optimization and an auxiliary delay loss.Training uses effective batches of approximately 3k locally and 12k online, with 100-step warmup and λ=0.01 for the delay term.
  • C Implementation and Training Details: The implementation details are collected separately because they are largely orthogonal to UniDot’s core architecture but matter for reproducing the reported numbers.This subsection groups data-specific feature handling and training machinery.
  • C.1 Active configuration: The submitted configuration uses approximately 2.1B parameters, with 22 user-side tokens, 12 item-side tokens, 24 item-history tokens, and five behavioral sequences.The fid-axis compression produces a uniform 4·128 = 512-dimensional per-position width, while parameters are dominated by sparse embeddings.
  • C.2 Per-position weights (user_dense pairing): Ten user_dense arrays act as per-position multipliers aligned with same-numbered multi-value user_int fields, scaling each paired ID embedding before pooling.Count-like fids 62–66, 118, and 121 use a fixed log(1 + x)/10 transform.
  • C.3 Field-aware feature embedding (FAFE): FAFE applies candidate-aware DIN-style attention pooling to selected high-value behavioral ID lists, while remaining fields retain static NCB pooling.The candidate-aware fields include fids 15, 63–66, 115–118, 121, 122, concentrating adaptivity where it is expected to matter most.
  • C.4 Pre-trained embedding projection and normalization: Pre-trained dense vectors are standardized with fixed training-set statistics and projected through a two-layer Linear–GELU–Linear MLP before entering the shared token space.The resulting tokens receive shared per-token LayerNorm; the structured 320-dimensional LMF4Ads vector is instead tokenized as ten 32-dimensional sub-vectors with masking for zero padding.
  • C.5 High-cardinality skip embeddings: Fids above the 2M threshold use a shared hashing-trick table, feed pooled signal directly to the classifier through e_skip, and are re-initialized on cold restart.The item-id fid additionally uses a dedicated 2M-slot multiplicative-hash table feeding item tokenizers.
  • C.6 Auxiliary conversion-delay head: The auxiliary head regresses log(1 + (t_label − t_event)) with masked MSE over rows having a next action, supplying roughly 8× more auxiliary signal than conversion positives alone.It covers both next-click-without-conversion and conversion rows and contributes λL_delay with λ=0.01.

C.7 Optimization and training

UniDot combines sparse Adagrad embeddings with Muon- or AdamW-optimized dense parameter groups, while using repeated embedding re-learning and fresh warmups. Training also applies time bucketing, logit clamping, EMA model selection, and log-linear data scaling across 4M–32M examples.

  • Dual optimizer: Embedding parameters use Adagrad, while dense matrix weights use Muon and one-dimensional parameters use an AdamW auxiliary group.Muon uses decoupled weight decay of 1e-3; the AdamW auxiliary group has zero weight decay.
  • Data scaling: Both held-out AUC and LogLoss improve log-linearly as training examples increase across the full 4M–32M range.The result is reported for tiny mode with single-path d=64.
  • Re-learning and warmup: Embeddings are re-learned each epoch, with dense learning rates freshly warmed up after each re-initialization.The schedule lets re-learned embeddings settle before full-rate updates and matches the one-step-ahead train/test time gap.
  • Time bucketing: Continuous event-time deltas map to 64 embedding slots, with boundaries spanning approximately 1 second to approximately 1.5 years.Capacity is concentrated in the 1 h–18 month range using a recency-weighted grid; slot 0 represents padding.
  • Stabilization and selection: Forward logits are clamped to [−20, 20], and dense parameters maintain an exponential moving average with decay 0.999.When a held-out split exists, the higher-scoring live or EMA weights are published each epoch.

D Data-scaling study

With the model fixed at the tiny configuration, increasing the training-example cap from 4M to 32M produces a clean log-linear AUC gain and steadily lower LogLoss. The trend suggests UniDot’s unified block benefits from additional training data.

  • Data-scaling study: Held-out AUC rises log-linearly from 0.83657 at 4M to 0.84396 at 32M, while LogLoss falls from 0.2151 to 0.2115.Intermediate AUC values are 0.83899 at 8M and 0.84187 at 16M; the gain is about +0.0025 AUC per doubling.
  • Data-scaling study: The unbroken trend through the full 32M training-example cap suggests the unified block is data-hungry.The model remains fixed at the tiny single-path d=64 configuration while only the training-example cap changes.
Loading 2608.16797v1…