Source-linked AI summary
User Representation via Cross Multi-source Behavior Pre-training for Mobile Games
Chengqi Yang, Yiran Qiao, Feng Liu, Xingyu Lou, Zijun Zhou, Xiaoyun Mo, Changwang Zhang, Jiayuan Xu, Jun Wang, Xiang Ao
TL;DR
Existing user-representation pre-training largely overlooks cross-source, multi-granular behavior on mobile devices, where intent spans heterogeneous streams. CM-PTM uses hierarchical cascaded mask-then-predict tasks that refine predictions from behavior source to app and action, and experiments report consistent gains in mobile-game recommendation.
Problem
Existing studies predominantly model single-source or app-level behavior, despite mobile-device intent spanning heterogeneous sources and granularities.
Method
CM-PTM organizes multi-source behavior sequences and hierarchically cascades predictions from the next behavior’s source through App category, App ID, and operation type.
Results
CM-PTM consistently outperforms strong baselines in offline evaluations and online A/B tests on large-scale OPPO mobile-game recommendation datasets.
Takeaways & Limitations
Combining multiple behavior granularities and sources is essential for accurately capturing device-level user representations in mobile scenarios.
Abstract
from arXiv · showhide
User representation pre-training has become a fundamental paradigm for alleviating data sparsity in downstream personalization tasks. However, existing studies predominantly focus on single-app or app-level behaviors, overlooking the inherently cross-source and multi-granular nature of user activities on mobile devices. At the device level, user intent emerges from complex interactions among heterogeneous behavior sources and hierarchical action structures, posing challenges that cannot be addressed by conventional app-centric modeling. To tackle this issue, we propose CM-PTM, a novel Cross Multi-source Behavior Pre-Training Model tailored for mobile game user representation learning on device-level behavioral logs. CM-PTM employs hierarchical cascaded mask-then-predict proxy tasks that first infer the source of the next behavior and then progressively refine predictions at the app-action level. This design enables unified modeling of cross-source dependencies and fine-grained behavioral dynamics within a single pre-training paradigm. Extensive experiments on large-scale real-world mobile datasets demonstrate that CM-PTM effectively captures users' endogenous interests and consistently delivers significant performance gains on downstream mobile game recommendation tasks.
I. INTRODUCTION
Mobile-device user intentions span heterogeneous behavior sources and granularities, making single-source and app-centric pre-training insufficient. CM-PTM addresses this setting with hierarchical cascaded prediction and is evaluated in real-world scenarios.
- Motivation: Existing user-representation pre-training studies predominantly model single-source or app-level behavior, whereas device-level learning integrates heterogeneous behaviors across a device.The paper frames device-level representation learning as a distinct problem setting rather than a simple extension of app-centric modeling.
- Motivation: Mobile-device logs combine system applications with fine-grained events and third-party applications with only coarse-grained observable events.System apps include App Store, Game Center, and Browser; third-party apps expose events such as installation, launch, and uninstall.
- Motivation: Cross-source interactions can distribute intent across games, the App Store, and the Browser, even when multi-step engagement does not result in installation.The illustrative case involves Honor of Kings and Zombie Frontier and suggests weak installation intent despite exploration across sources.
- Motivation: Chronologically aggregating all behaviors into one sequence overlooks structural distinctions and weakens intra-source continuity, degrading performance in ablation results.The paper links this failure to the mismatch between cross-source interleaving and the strong intra-sequence dependency assumption of many pre-training frameworks.
- Approach: CM-PTM organizes multi-source, multi-granularity sequences and progressively predicts the next behavior’s source, App category, App ID, and operation type.Each prediction stage is conditioned on the previous stage, forming a system-level architecture for asymmetric cross-source fusion.
- Evaluation: Offline experiments and online A/B tests evaluate CM-PTM in large-scale real-world industrial scenarios.The stated evaluation goal is to demonstrate effectiveness and robustness.
A. User Representation Pre-training
Prior user-representation pre-training commonly uses generative or discriminative self-supervision, while multi-behavior methods model heterogeneous behaviors but generally remain source-limited. The paper formulates learning a dense representation from chronologically ordered sequences across multiple behavior sources.
- Background: User representation pre-training learns dense embeddings from historical behavior for downstream tasks including recommendation, profile prediction, fraud detection, and click-through-rate prediction.The broader pre-training literature includes masked-behavior, next-behavior, and contrastive objectives.
- Related Work: Multi-behavior modeling combines heterogeneous user behaviors through GNN-, Transformer-, and hybrid architectures to capture correlations among behavior types.Prior work reports advantages over single-behavior approaches.
- Research Gap: Existing related approaches focus on behavior from a single source, limiting their applicability to the paper’s multi-source device setting.This limitation motivates a distinct device-level formulation.
- Problem Formulation: For each user, S_k^u denotes the chronologically ordered behavior sequence from source k.The formulation assumes behavioral data from n different sources are available.
- Problem Formulation: The task learns F_θ to map multi-granularity behaviors from multiple sources into a dense downstream user representation u.The representation has dimension d_u, and θ denotes the model parameters.
IV. METHODOLOGY
CM-PTM combines a multi-granularity user model with a cross-source cascaded prediction module for device-level mobile-game recommendation. Its inputs are source-organized behavior sequences represented as bounded event-embedding sequences with positional information.
- Architecture: CM-PTM has two components: a multi-granularity user model and a cross-source cascaded prediction module.The first integrates fine- and coarse-grained behaviors, while the second refines next-behavior prediction from source to fine-grained action.
- Architecture: The multi-granularity user model addresses the mismatch between fine-grained system-app logs and coarse-grained third-party-app logs.The stated challenge is preserving intra-source continuity while capturing inter-source intent correlations.
- Data Setting: The method is instantiated on OPPO device-level logs from third-party GameApps, Appstore, and Gamecenter for mobile game recommendation.These are the three sources used in the described setting.
- Behavior Embedding: For each source, CM-PTM concatenates per-event embeddings into a behavior-embedding sequence, where each event embedding represents one behavioral event.The embedding dimension is d, and source-specific sequences are denoted E_k^u.
- Behavior Embedding: Each embedding sequence is truncated to the most recent l behaviors when too long and zero-padded otherwise, followed by positional encoding.This imposes a fixed maximum sequence length before positional information is added.
2) Granularity-Aware Self-Attention:
The multi-granularity user encoder uses GASA to model correlations among same-granularity behaviors and CGFA to integrate coarse- and fine-grained patterns across sources.
- Granularity-Aware Self-Attention: GASA uses multi-head self-attention to identify correlations between behaviors of the same granularity.
- User Encoder: The encoder derives user representations from historical behaviors across different sources and granularities.
- Cross-Granularity Fusion Attention: CGFA uses multi-head cross-attention to integrate macrolevel coarse-grained patterns with micro-level fine-grained insights.
4) User Representation:
CM-PTM combines source-specific and cross-source representations with a cascaded prediction module that first predicts the next behavior source and then refines its action.
- User Representation: The encoder concatenates GASA and CGFA outputs to provide source-specific and cross-source association knowledge.Source-specific representations are used independently during pre-training, while downstream tasks concatenate all representations.
- Cross-Source Cascaded Prediction: The cascaded framework conditions fine-grained predictions on high-level intent, reducing each stage’s hypothesis space and mitigating cross-source label noise.
- Inter-Source Phase: The inter-source phase predicts whether the next behavior occurs in GameApps, Appstore, or Gamecenter.A shared source prediction tower produces logits and a softmax distribution for the source prediction task.
- Inter-Source Phase: Pre-training derives the source label from the held-out next behavior in the globally chronological multi-source log.The same source supervision is applied to each source-specific representation to encourage cross-source transition modeling.
- Intra-Source Phase: The intra-source phase predicts app category, specific app ID, and operation through sequential subtasks conditioned on prior task states.App ID prediction follows category prediction, while operation prediction is used when labels are available and meaningful.
C. Model Training
Training defines separate intra-source task sets and combines inter-source and intra-source losses with a balancing hyper-parameter, using training and validation data during pre-training.
- Task Sets: Third-party GameApps use T2, T3, and T4, whereas Appstore and Gamecenter use T2 and T3.T4 is omitted for Appstore and Gamecenter.
- Loss Definition: N(T) denotes the vocabulary size for sub-task T, while ˆp_k_T denotes its predicted probability distribution for source k.
- Loss Definition: The overall pre-training loss combines the two stages using λ as a hyper-parameter that balances them.
- Training Protocol: The pre-training stage uses only training and validation sets, with downstream scenarios reserved for evaluating the learned representations.
V. EXPERIMENTS
Experiments evaluate CM-PTM on large-scale real-world industrial datasets using self-supervised pre-training followed by downstream adaptation with LightGBM.
- Experimental Setup: CM-PTM uses a two-stage paradigm: self-supervised mask-then-predict pre-training followed by downstream adaptation.Pre-trained representations are frozen or fine-tuned and supplied to a LightGBM model.
A. Experimental Setting
The experiments use anonymized OPPO device-level behavior logs and evaluate mobile game recommendation across six tasks covering new and old users and games.
- Pre-training Dataset: Pre-training uses a one-week interval from 2023/07/13 to 2023/07/19 to provide rich and diverse behavior data.
- Dataset Separation: The datasets contain no temporal overlap between pre-training and downstream task data, supporting separate pre-training and evaluation periods.
- Downstream Task Datasets: The downstream evaluation contains six tasks, G1–G6, designed with popular blockbuster games for statistically significant offline evaluation.
- Definitions: Old games were released at least six months ago, new games within the past month, old users played within six months but not the last two weeks, and new users never played before.
- Downstream Scenarios: The scenarios cover attracting new users to old games, attracting old users to old games, and attracting new users to new games.
- Sampling: G1–G3 training data uses a 1:2 positive-to-negative ratio, while G4 uses the same ratio for old-user recommendation.
2) Baselines:
The study compares CM-PTM with generative and discriminative pre-training baselines using AUC and R@P0.95, reporting significant gains across most downstream tasks.
- Baselines: The baselines include masked-behavior, next-K-behavior, bidirectional self-attention, and contrastive pre-training methods.
- Evaluation Metrics: AUC and R@P0.95 are used for evaluation, with R@P0.95 measuring maximum recall when precision is at least 0.95.
- Statistical Significance: Improvements over the best baselines are significant under paired t-tests over five runs with p-value < 0.05.
- Overall Results: CM-PTM significantly outperforms all baseline models on G1–G5, improving AUC by 1.05% to 2.85%.The authors attribute this gain to cross-source cascaded prediction and hierarchical attention.
- Baseline Comparison: Generative pre-training methods generally outperform discriminative methods because they directly model behavior generation and temporal dependencies.
- New-Game Results: CM-PTM has suboptimal AUC on G6 but remains far ahead in R@P0.95, indicating stronger identification of potential positive samples.
C. Ablation Study
The ablation studies show that multi-granularity, cross-source modeling and cascaded prediction are important for CM-PTM’s performance, while phase weighting affects users differently.
- Data ablation: Removing either fine-grained or coarse-grained behaviors lowers AUC on G1 and G5, showing the value of combining multiple granularities.Using only fine-grained behaviors decreases AUC by 2.64 and 2.39 percentage points, while only coarse-grained behaviors decreases it by 0.57 and 2.28 percentage points.
- Sequence ablation: Chronologically merging all behaviors into one sequence decreases AUC by 1.76% on G1 and 1.52% on G5.The result highlights the limitations of conventional single-sequence modeling for heterogeneous cross-source behaviors.
- Task ablation: Removing T1 improves neither task, with AUC decreases of 0.52% on G1 and 1.87% on G5.For new games, removing T1 causes the largest performance decrease; for old games, its contribution ranks second among the four tasks.
- Task ablation: The cascaded design consistently outperforms parallel predictions, supporting progressive conditioning across heterogeneous behavior streams.Each sub-task and the cascade relationships contribute positively to overall performance, especially behavior source prediction.
F. Online A/B Test
The online evaluation reports that CM-PTM improves recommendation metrics over DeepFM on OPPO devices while adding minimal inference overhead. Its higher training cost remains within an acceptable industrial budget.
- Online A/B test: 1.28% AUC and 1.07% R@P0.95 improvements over DeepFM validate CM-PTM in OPPO’s online mobile game recommendation system.The A/B test randomly sampled 2.89 million active users over two weeks and split traffic evenly by user ID.
- Efficiency: CM-PTM requires more offline training time than most generative baselines but remains below MAFN and within an acceptable industrial training budget.The reported efficiency reflects strategies including skipping a separate pre-training test set and omitting T4 for App Store and Game Center behaviors.
- Efficiency: CM-PTM adds only d_u dimensions to downstream representations, introducing negligible online inference overhead relative to existing thousand-dimensional features.The method nevertheless consistently delivers substantial downstream performance gains.
- Overall outcome: CM-PTM consistently outperforms strong baselines in offline evaluations and online A/B tests, supporting practical applicability in OPPO recommendation scenarios.The conclusion attributes the results to capturing cross-source user intent from multi-source, multi-granular behavioral data.