Source-linked AI summary
HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction
Yunwen Huang, Shiyong Hong, Xijun Xiao, Jinqiu Jin, Xuanyuan Luo, Zhe Wang, Zheng Chai, Shikang Wu, Yuchao Zheng, Jingjian Lin
TL;DR
Industrial LRMs need to combine long behavior sequences with heterogeneous features, but separated compression and interaction pipelines limit their integration. HyFormer unifies them through alternating Query Decoding and Query Boosting, and experiments report consistent gains over strong baselines with favorable scaling and online improvements. The method therefore offers a practical unified framework for industrial LRMs, while its short-sequence attention formulation trades contextual capacity for lower computational cost.
Problem
Industrial LRMs must jointly model long-range behavior sequences and heterogeneous features, but prevailing decoupled pipelines limit representation capacity and interaction flexibility.
Method
HyFormer uses global tokens and alternates Query Decoding over layer-wise long-sequence key-value representations with Query Boosting through efficient token mixing.
Results
HyFormer consistently outperforms strong sequence-based and token-mixing baselines under comparable parameter and FLOPs budgets, with superior scaling and significant online A/B-test gains.
Takeaways & Limitations
HyFormer provides a practical unified framework for long-sequence modeling and heterogeneous feature interaction in industrial LRMs.
Takeaways & Limitations
The short-sequence attention formulation trades contextual capacity for minimal computational cost.
Abstract
from arXiv · showhide
Industrial large-scale recommendation models (LRMs) face the challenge of jointly modeling long-range user behavior sequences and heterogeneous non-sequential features under strict efficiency constraints. However, most existing architectures employ a decoupled pipeline: long sequences are first compressed with a query-token based sequence compressor like LONGER, followed by fusion with dense features through token-mixing modules like RankMixer, which thereby limits both the representation capacity and the interaction flexibility. This paper presents HyFormer, a unified hybrid transformer architecture that tightly integrates long-sequence modeling and feature interaction into a single backbone. From the perspective of sequence modeling, we revisit and redesign query tokens in LRMs, and frame the LRM modeling task as an alternating optimization process that integrates two core components: Query Decoding which expands non-sequential features into Global Tokens and performs long sequence decoding over layer-wise key-value representations of long behavioral sequences; and Query Boosting which enhances cross-query and cross-sequence heterogeneous interactions via efficient token mixing. The two complementary mechanisms are performed iteratively to refine semantic representations across layers. Extensive experiments on billion-scale industrial datasets demonstrate that HyFormer consistently outperforms strong LONGER and RankMixer baselines under comparable parameter and FLOPs budgets, while exhibiting superior scaling behavior with increasing parameters and FLOPs. Large-scale online A/B tests in high-traffic production systems further validate its effectiveness, showing significant gains over deployed state-of-the-art models. These results highlight the practicality and scalability of HyFormer as a unified modeling framework for industrial LRMs.
1 Introduction
Industrial LRMs must jointly model long-range behavior sequences and heterogeneous features, but prevailing pipelines separate sequence compression from feature interaction. HyFormer unifies these operations through alternating Query Decoding and Query Boosting.
- Long-range behavior histories and heterogeneous features are central inputs for accurate industrial recommendation and search prediction.
- Existing architectures first encode long behavior sequences and then mix compressed sequence tokens with heterogeneous features, creating a separated modeling paradigm.
- Simplified query representations limit contextual information for long-term interest modeling, while adding queries degrades serving efficiency under KV-Cache and M-Falcon mechanisms.
- HyFormer unifies sequence modeling and feature interaction in one backbone using global tokens as a shared semantic interface.
- Query Decoding and Query Boosting alternate to enable bidirectional, layer-wise interaction between behavioral sequences and heterogeneous features.
2 Related Work
Related work develops scalable sequence models, feature-interaction networks, and partially unified recommendation architectures. However, production systems still commonly retain loosely coupled sequence and interaction components, while some unified designs sacrifice serving efficiency.
- Industrial LRMs commonly separate behavior-sequence modeling from downstream feature-interaction networks that consume encoded histories with heterogeneous features.
- DeepFM, xDeepFM, and DCNv2 model bounded-degree feature crosses, while Wukong and RankMixer emphasize scaling cross-module interaction capacity.
- HSTU and InterFormer explore unified recommendation architectures using contextual sequence transduction or learnable interaction tokens.
- Increasing query tokens with all non-sequence tokens causes a significant serving-efficiency drop in MTGR and OneTrans.
- Unified architectures remain an open challenge because industrial LRMs require full unification with minimal architectural overhead.
3 Methodology
HyFormer formulates recommendation prediction over behavioral histories, non-sequential descriptors, candidate items, and engagement labels. Its layers alternate Query Decoding over long-sequence representations with Query Boosting for heterogeneous token interaction.
- For user u, history S, non-sequential descriptors, and candidate item v, the LRM estimates the probability that the user engages with v.
- Model parameters are learned from historical data by minimizing binary cross-entropy between observed interaction labels and predicted engagement probabilities.
- Traditional LRMs sequentially compress behavior histories before applying downstream feature interaction, whereas HyFormer jointly models both stages in a unified framework.
- Each HyFormer layer uses Query Decoding to generate semantic global tokens and cross-attend to long-sequence key-value pairs, then uses Query Boosting for MLP-Mixer-style token mixing.
- The coupled design is reported to provide richer heterogeneous interactions, deeper sequential-structure utilization, and more favorable performance and computation cost than separate pipelines.
3.3 Query Generation
HyFormer generates semantic query tokens from heterogeneous features and pooled sequence context, then reuses and compresses queries to balance representational capacity with serving efficiency. Its architecture alternates decoding and token-mixing operations.
- Semantic grouping organizes input tokens by meanings such as user, context, or behavior, rather than uniformly splitting all flattened features.
- Query Generation concatenates non-sequential feature vectors and combines them with a pooled behavioral-sequence summary through a lightweight feed-forward network.
- The generated queries serve as inputs for long-sequence decoding, while global tokens connect heterogeneous features with sequence representations.
- HyFormer alternates Query Decoding with Query Boosting, which uses MLP-Mixer-based feature interaction to increase long-sequence query capacity.
- Feature selection and optional query compression keep the number of generated queries stable for serving efficiency.
- Deeper layers reuse previous-layer queries as progressively richer cross-attention outputs instead of regenerating them through MLPs.
3.4 Query Decoding
Query Decoding converts non-sequential features into semantic queries and uses them to extract target-aware information from long behavioral sequences. HyFormer supports encoding strategies that trade contextual capacity against efficiency while producing layer-wise key–value states for cross-attention.
- Query Decoding: Query Decoding transforms non-sequential features into semantic queries that attend over layer-wise key–value representations of long behavioral sequences.The decoded queries inject global contextual signals into sequence-aware representations for later interaction modules.
- Sequence Representation Encoding: Sequence features produce layer-specific key–value states that are recomputed at each layer, allowing representations to evolve with decoder depth.
- Sequence Representation Encoding: Full Transformer Encoding captures fine-grained interactions and long-range dependencies through full self-attention.
- Sequence Representation Encoding: LONGER-style Efficient Encoding replaces full self-attention with cross-attention between a compact short sequence and the full history.With L_H ≪ L_S, complexity is reduced from O(L_S^2) to O(L_HL_S).
- Sequence Representation Encoding: Decoder-style Lightweight Encoding uses attention-free feed-forward operations for latency-critical scenarios, trading contextual capacity for minimal computational cost.
3.5 Query Boosting
Query Boosting mixes decoded query tokens with heterogeneous non-sequential feature signals before the next layer. Its lightweight token mixing, per-token refinement, and residual connection progressively enrich representations while preserving linear computational complexity.
- Query Boosting: Query Boosting explicitly mixes information across query tokens and injects additional non-sequence-feature signals after Query Decoding.
- Query Boosting: The unified query representation concatenates decoded query tokens with non-sequential feature embeddings, with T = N + M tokens.
- Query Boosting: An MLP-Mixer-style operation aggregates information across token positions by partitioning each query token into T channel subspaces.
- Query Boosting: A per-token feed-forward module refines mixed queries with subspace-specific transformations while preserving linear computational complexity.
- Query Boosting: A residual connection stabilizes optimization and preserves the original decoded semantics before boosted queries enter the next HyFormer layer.
3.6 HyFormer Module
HyFormer stacks Query Decoding and Query Boosting into repeated layers that alternate sequence cross-attention with lightweight query refinement. This progressively produces richer semantic queries for downstream prediction.
- HyFormer Module: Each HyFormer layer first performs Query Decoding over layer-wise sequence key–value representations and then applies Query Boosting to the resulting queries.
- HyFormer Module: Stacked layers progressively refine semantic queries, enabling deeper layers to abstract long sequences with increasingly expressive representations.
- HyFormer Module: The top HyFormer output is fed into downstream MLPs for final predictions while integrating heterogeneous non-sequential features with long behavioral sequences.
3.7 Multi-Sequence Modeling
HyFormer processes heterogeneous behavioral sequences independently during each block, preserving sequence-specific semantics while deferring cross-sequence interaction to query-level token mixing. Figure 2 presents this multi-sequence design.
- Multi-Sequence Modeling: Industrial recommendation users may have multiple heterogeneous behavior sequences, such as video-watch and product-purchase histories.
- Multi-Sequence Modeling: HyFormer processes each behavior sequence independently in every block instead of merging sequences into a single stream.This preserves sequence-specific semantics during decoding and avoids explicit sequence concatenation.
- Multi-Sequence Modeling: Each sequence receives dedicated query tokens for Query Decoding, while later query-level token mixing handles cross-sequence interaction.
3.8 Training and Deployment Optimization
The system reduces long-sequence data-transfer and host-memory costs through feature deduplication, while asynchronous AllReduce overlaps synchronization with computation to improve GPU utilization.
- Feature deduplication exploits the fact that unique feature IDs typically comprise 25% of sequence tokens, reducing transfer costs and host-memory footprint.Features are stored in a compressed embedding table before graph execution.
- Asynchronous AllReduce overlaps step k gradient synchronization with step k+1 computation, mitigating idle time and communication bubbles.The approach maximizes GPU utilization but introduces one-step staleness for dense parameters.
- Dense parameters use gradients from the previous step, whereas sparse parameters update immediately after local gradient computation.This creates a small temporal inconsistency between dense and sparse parameter states.
- Empirical results indicate that the hybrid update schedule does not degrade convergence quality or model performance in practice.
4 Experiments
Experiments evaluate HyFormer on a billion-scale industrial CTR task against two-stage and unified baselines, using offline accuracy, efficiency, ablations, scaling analyses, and online A/B tests. HyFormer achieves the highest reported offline AUC with lower computational cost, benefits from richer queries and separate sequence modeling, scales favorably, and improves production metrics.
- Experimental Setup: The evaluation uses 3 billion samples from 70 days of Douyin Search logs, with user, query, document, cross, and sequential features.The primary sequences include a long-term search-and-click history capped at 3000 items, plus top-50 search and feed sequences.
- Experimental Setup: Offline comparisons cover separated two-stage models and unified-block models, evaluated with Query-level AUC, dense parameter counts, and training FLOPs.Baselines include LONGER or Full Transformer for sequence modeling and token-interaction modules such as RankMixer.
- Overall Comparison: HyFormer achieves the highest AUC while requiring only 3.9×10^12 total training FLOPs, outperforming two-stage and unified competitors under the reported evaluation.The strongest BaseArch combination, Full Transformer plus RankMixer, still trails HyFormer; the results attribute this to HyFormer’s bidirectional information flow.
- Ablation Study: Ablations show that target-only queries reduce AUC by 0.08%, removing cross-sequence pooling tokens reduces it by 0.05%, and sequence merging with shared queries loses 0.06%.Within HyFormer, expanding query information produces a 0.08% AUC gain, whereas the enriched-query baseline architecture gains only 0.03%.
- Ablation Study: Separate query tokens preserve distinctions among heterogeneous sequences and allow adaptive allocation of more global tokens to important sequences.This avoids forced alignment of side information or sparse dimensions across sequences and yields measurable offline gains.
- Scaling Analysis: HyFormer exhibits a steeper performance scaling slope than LONGER + RankMixer as parameters and FLOPs increase, with AUC rising steadily with FLOPs.The reported explanation links this scaling behavior to alternating stacked layers and bidirectional information flow between sequence and mixer modules.
- Scaling Analysis: Richer sequence side information benefits HyFormer more than LONGER + RankMixer across sequence lengths, with the advantage becoming stronger for longer sequences.The reported mechanism combines richer global information in sequence queries with bidirectional interaction between LONGER and Mixer modules.
5 Conclusions
HyFormer replaces the prevalent unidirectional sequence-then-feature-interaction pipeline with an alternating, bidirectional framework built around Global Tokens, Query Decoding, and Query Boosting. Offline and online experiments validate its effectiveness and support a higher scaling ceiling for industrial LRMs.
- 5 Conclusions: HyFormer alternates between decoding long sequences with Global Tokens and enhancing those tokens through cross-feature interaction.This iterative optimization reframes sequence modeling and feature interaction as co-evolving processes.
- 5 Conclusions: The architecture upgrades unidirectional information flow to a bidirectional, co-evolutionary paradigm for more thorough sequence modeling and feature interaction.
- 5 Conclusions: HyFormer also provides a flexible paradigm for multi-sequence modeling.
- 5 Conclusions: Extensive offline and online experiments validate HyFormer's superiority over the prevalent unidirectional information-flow paradigm.
- 5 Conclusions: The results raise the scaling ceiling for future industrial LRMs.