Source-linked AI summary
Topology-Masked Unified Backbone for Joint Feature Interaction and Multi-Domain Sequence Modeling
Zhihao Zhu, Dezheng Han, Jikang Xia, Shuaishuai Guo
TL;DR
Large-scale industrial CVR prediction needs joint modeling of heterogeneous feature interactions and multi-domain behavior dependencies, but existing approaches separate these aspects or incompletely unify their information space. MaskRec uses topology-masked unified token interactions, memory tokens, and candidate-conditioned query generation; experiments report stable improvements over the official baseline on the Tencent Advertising Algorithm Competition dataset.
Problem
Existing methods separate feature interaction from sequence modeling, while unified architectures may rely on module coordination or fail to organize heterogeneous information within one interaction space.
Method
MaskRec maps heterogeneous inputs into unified tokens, adds global and domain-level memory tokens, regulates attention with TopoMask, and generates candidate-conditioned domain-aware queries.
Results
MaskRec achieves stable performance improvements over the official baseline on the Tencent Advertising Algorithm Competition dataset.
Takeaways & Limitations
Topology-constrained token routing can unify feature interaction and multi-domain sequence modeling for industrial CVR prediction.
Takeaways & Limitations
TopoMask is manually specified from empirical structural priors and may not be optimal across different datasets or feature systems.
Abstract
from arXiv · showhide
Large-scale post-click conversion rate (CVR) prediction requires jointly modeling heterogeneous feature interactions and dependencies over multi-domain user behavior sequences. Existing industrial ranking models usually handle these two aspects with separate modules. Recent unified architectures attempt to incorporate them into a single framework, but such unification often relies on coordination between modules and does not fully organize all information sources within the same interaction space. To address this problem, we propose MaskRec, a topology-masked unified token interaction architecture for feature interaction and multi-domain sequence modeling. MaskRec transforms heterogeneous features, multi-domain behavior sequences, and contextual signals into unified token representations, and further introduces learnable global memory tokens and domain-level memory tokens as information aggregation nodes. Based on this unified token space, MaskRec designs a structured attention mask, TopoMask, which selectively enables or blocks attention connections according to the structural differences and modeling requirements of different information sources. In this way, heterogeneous feature interaction and multi-domain sequence modeling are performed within the same topology-constrained attention process. In addition, MaskRec incorporates a dual-path interactive query generation module to inject candidate-conditioned user--item interaction signals before the unified backbone. Experiments on the Tencent Advertising Algorithm Competition dataset show that MaskRec achieves stable improvements over the official baseline, validating the effectiveness of the proposed unified framework for industrial CVR prediction.
1 Introduction
MaskRec addresses the limits of separate and incompletely unified modeling by organizing heterogeneous inputs in a shared token space and regulating their interactions with TopoMask. It also adds candidate-conditioned query generation, and experiments report stable improvements over the official baseline.
- Separate modules for feature interaction and behavior-sequence modeling limit deep interactions among heterogeneous information sources.
- Existing unified architectures either overlook structural differences and interaction boundaries or rely on alternating coordination between modeling processes.
- MaskRec converts heterogeneous features, multi-domain behavior sequences, and contextual signals into unified tokens, with global and domain-level memory tokens as aggregation nodes.
- TopoMask selectively enables or blocks token attention connections to preserve source-specific structure while supporting controlled cross-source interaction.
- The dual-path query module injects candidate-conditioned user–item interaction information before the unified backbone and produces enhanced queries for behavior domains.
- Experiments on the Tencent Advertising Algorithm Competition dataset show stable performance improvements over the official baseline.
2 Related Work
Industrial ranking systems commonly separate behavior-sequence modeling from non-sequential feature interaction, while newer unified architectures reduce late fusion but still organize heterogeneous information imperfectly. The related work therefore motivates more structurally aware unification.
- Candidate-aware and self-attentive sequence models capture user interests from historical behaviors, especially when candidate relevance matters.
- Feature-interaction models provide efficient and scalable mechanisms for modeling feature crossing patterns in CTR/CVR prediction and industrial recommendation.
- Traditional pipelines encode or compress behavior sequences before fusing them with non-sequential features, weakening early and deep cross-source interactions.
- HyFormer reduces late fusion through alternating query decoding and query boosting, whereas OneTrans uses a unified token stream and single Transformer backbone.
- Hybrid and unified-backbone approaches still differ in how they organize heterogeneous information and may use general token interactions without explicit structural boundaries.
3 Method
MaskRec unifies heterogeneous features, contextual signals, multi-domain behavior sequences, memory tokens, and interaction-enhanced queries in a shared token space. TopoMask then regulates attention connections to preserve source-specific structure while enabling controlled cross-source modeling for CVR prediction.
- Problem Formulation: MaskRec estimates conversion probability from heterogeneous non-sequential features, contextual signals, and multiple behavior domains.The inputs include user and candidate-item attributes, associated context, and behavior sequences from K domains.
- Problem Formulation: The method targets unified modeling of feature interactions and multi-domain behavior dependencies while preserving their structural differences.The desired backbone shares an interaction space but avoids unconstrained full interaction among all tokens.
- Overall Architecture: MaskRec converts non-sequential features, contextual signals, and domain-specific behaviors into tokens, then adds enhanced query tokens and global or domain-level memory tokens.Memory tokens aggregate global or domain-specific information, while DualQ supplies candidate-conditioned auxiliary tokens before backbone processing.
- MaskRec Block and Prediction: The final prediction concatenates global-memory, domain-memory, and enhanced-query representations before classification.These readout signals respectively represent global context, domain-specific sequence summaries, and candidate-conditioned user–item interaction information.
- Dual-Path Interactive Query Generation: DualQ uses bidirectional user–item cross-attention, attentive pooling, and domain conditioning to construct interaction-enhanced query tokens.User-side tokens condition on candidate and context, candidate-side tokens receive user-context information, and domain modulation adapts the base queries.
- TopoMask Attention: TopoMask Attention selectively enables or blocks token-level connections, supporting source-local modeling, domain aggregation, and controlled cross-source interaction in shared layers.This replaces unconstrained token interaction with topology-aware routing that jointly optimizes heterogeneous signals within the unified backbone.
4 Experiments
MaskRec is evaluated on anonymized advertising logs for post-click CVR prediction using ROC-AUC, with controlled comparisons covering architecture, feature and temporal components, and memory capacity. It consistently outperforms the HyFormer-style baseline, while ablations show contributions from query generation, feature organization, temporal context, and memory tokens.
- Dataset and Evaluation Metric: The competition task estimates post-click conversion probability on large-scale anonymized advertising logs, using ROC-AUC as the primary metric.The official 0.9:0.1 training-validation partition is followed.
- Overall Comparison: MaskRec raises validation AUC from 0.831827 to 0.841253 and test AUC from 0.824902 to 0.834640 over the HyFormer-style baseline.The comparison keeps the same feature-processing and input-construction pipeline while replacing the backbone.
- Ablation Study: Removing DualQ decreases validation AUC by 0.001216 and test AUC by 0.000975, supporting the utility of candidate-conditioned interaction signals.The variant obtains 0.840037 validation AUC and 0.833665 test AUC.
- Ablation Study: Collapsing dense features into one token lowers validation AUC to 0.831861 and test AUC to 0.825470, indicating value in preserving internal dense-feature organization.The ablation removes grouped dense tokens, structured dense representations, aligned pair features, and the fid-87 residual path.
- Ablation Study: Removing sparse integer features lowers validation AUC to 0.839800 and test AUC to 0.833275, while removing temporal features lowers them to 0.841045 and 0.834468.Both feature families provide complementary information, although the temporal-feature degradation is modest.
- Sensitivity to Memory Token Capacity: Reducing memory tokens from 32/8 to 16/4 stays close to default performance, whereas 8/2 causes clearer degradation, especially on test data.The two values denote global and domain-level memory-token counts, respectively.
5 Conclusion and Discussion
The paper concludes that MaskRec unifies heterogeneous features, contextual signals, multi-domain sequences, memory tokens, and enhanced query tokens through TopoMask and DualQ. Its main limitation is that TopoMask topology is manually specified from empirical structural priors and may not transfer optimally across datasets or feature systems.
- Conclusion: MaskRec organizes heterogeneous signals and memory and query tokens in a shared space, using TopoMask for controlled information flow and DualQ for candidate-conditioned query generation.The paper reports stable improvements over the official baseline on the Tencent competition dataset.
- Discussion: TopoMask topology is manually specified from empirical industrial-recommendation priors and may not be optimal across different datasets or feature systems.The proposed future direction is learnable or adaptive topology construction.
A Appendix
The appendix documents practical modeling ideas and implementation choices explored for the Tencent Advertising Algorithm Competition system, extending the official HyFormer-style baseline.
- Appendix: The appendix records dual-query generation, field-aware dense tokenization, temporal context modeling, long-tail frequency bucketing, and optimization stabilization.It also documents implementation-level ideas and fine-grained design choices explored during the competition.
A.1 Dataset Details
The benchmark uses anonymized advertising logs to predict valuable post-click actions after a user clicks an advertised item, with submissions ranked by ROC-AUC.
- Dataset Details: For each clicked ad exposure, the task estimates whether the user subsequently performs an action such as purchase, form submission, or app installation.The benchmark is the Tencent Advertising Algorithm Competition post-click conversion ranking task.
A.1.1 Competition Task.
The task is large-scale post-click conversion rate prediction under sample-selection bias and severe class imbalance. AUC is emphasized as the ranking-oriented evaluation metric.
- CVR prediction estimates whether a user converts after clicking a candidate advertisement or item.
- Only about eight percent of clicked exposures are positive conversion samples, creating substantial class imbalance.
- AUC is more informative than accuracy for this imbalanced ranking task.
A.1.2 Dataset Scale, Labels, and Feature Schema.
The released data is industrial-scale, anonymized, and organized as a same-window split with heterogeneous user, item, pairwise, and behavioral features. Its validation setup primarily measures in-distribution generalization.
- The validation split contains approximately 2.214 × 10^6 rows.
- Training and validation cover the same six-day window, so validation is a row-group rather than future-period split.
- Each row includes sparse user and item fields, dense side information, pairwise features, and timestamped multi-domain behavior sequences.
- The positive conversion rate is approximately 8.06%, with negative samples outnumbering positives by about 11.4×.
A.1.3 Temporal and Behavioral Patterns.
Temporal signals and behavior-domain differences contain label-related information beyond sequence position or aggregate length. These patterns motivate temporal and domain-aware modeling.
- Traffic volume and conversion-rate peaks occur at different times of day.Evening traffic is busiest, whereas conversions are more concentrated in late-morning periods.
- Absolute and cyclic time signals carry label-correlated information beyond relative sequence positions.
- Behavior domains differ in how strongly sequence length separates positive and negative samples.Domains a and b show clearer separation, while domains c and d show weak or nearly no separation.
- The asymmetric domain patterns motivate domain-aware temporal, aggregation, and interaction modeling.
A.2 Supplementary Component Details
The supplementary system explores structured tokenization, candidate-conditioned queries, temporal sequence views, and capacity changes for industrial recommendation modeling. Competition-stage results favor meaningful field structure, temporal context, multi-view sequences, and selective residual refinement.
- A.2.2 ID-Frequency Bucketing: Frequency bucketing assigns identifiers discrete occurrence-count buckets and adds shared embeddings without removing identity embeddings.
- A.2.4 Field-Wise Dense Tokenization: Field-wise item-dense tokenization independently normalizes and projects each schema-defined field instead of transforming the 770-dimensional bundle jointly.
- A.2.5 Dual-Query Generation with Cross-Side Conditioning: DualQ conditions user-side and item-side token branches bidirectionally before pooling them into the downstream interaction query bank.
- A.2.6 Candidate-Aware Query Bias: Candidate-aware query bias injects target-aware sequence summaries into query representations before downstream cross-domain interaction.
- A.2.8 Multi-Domain Sequence Modeling: Independent domain tokenizers preserve domain identities, while stable time-bucket sorting enforces consistent temporal organization.
- A.2.10 Multi-View Sequence Modeling: Multi-view sequence modeling combines full retained histories with shorter recent subsets through independent interaction paths.
- A.3.3 Competition-Stage Results: Field-wise tokenization and the field-129 split produce the strongest reported late-stage improvements.
- A.3.3 Competition-Stage Results: Semantic grouping improves user-dense modeling more clearly than simply increasing token count.