Source-linked AI summary
Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts
Ziqing Qian, Haohang Chen, Shengqi Dang, Yuhan Xiong, Canyu Shen, Jiaying Lei, Nan Cao
TL;DR
Interest drift complicates user preference modeling because social-media preferences evolve across multiple time scales and span co-existing interests amid heterogeneous, sparse activity. DUMoE unifies multimodal temporal modeling with a sparse mixture-of-experts adapter and three-stage optimization. Across user interest classification and interaction prediction, it consistently outperforms strong baselines, while its pseudo-labeling and fixed temporal windows remain limitations.
Problem
Interest drift requires modeling heterogeneous multimodal behavior, multi-scale temporal dynamics, irregular sparse activity, and multiple co-existing interests in one user representation.
Method
DUMoE combines a backbone for static, short-term, and long-term signals with sparse expert routing for multi-interest disentanglement and three-stage training.
Results
DUMoE consistently outperforms strong baselines on user interest classification and interaction prediction tasks.
Takeaways & Limitations
Unified modeling of multimodal signals, temporal dynamics, and interest structure supports drift-aware user representation learning in the evaluated tasks.
Takeaways & Limitations
Pseudo-label quality may be affected by language-model annotation noise or domain-specific bias, and the backbone uses fixed temporal window lengths.
Abstract
from arXiv · showhide
Understanding user preferences from noisy and temporally evolving social media behaviors is fundamentally challenging due to interest drift, where user preferences shift across time and exhibit both multi-scale temporal patterns and diverse co-existing interests. To address this, we propose DUMoE, a unified framework for drift-aware multimodal user representation learning. Our model consists of (i) a temporal dynamics-aware backbone that captures and integrates static profiles, short-term behavioral signals, and long-term dependencies into a coherent representation, and (ii) a sparse mixture-of-experts (MoE) interest adapter that disentangles multiple latent interests via expert specialization and adaptive routing. Each expert models a distinct interest subspace, while a gating network dynamically selects and aggregates a sparse subset of relevant experts for each user. To enable stable and effective optimization, we further introduce a three-stage training strategy that decouples backbone learning, expert specialization, and gating optimization. Extensive experiments on real-world social media datasets show that DUMoE consistently outperforms state-of-the-art methods on both user interest prediction and interaction prediction tasks.
1 Introduction
Interest drift makes user preference modeling difficult because preferences combine long-term stability, short-term dynamics, and multiple co-existing interests. DUMoE addresses this with unified multimodal, multi-scale temporal, and multi-interest representation learning.
- Interest drift combines evolving preferences, multi-scale temporal dynamics, and multiple co-existing interests.
- Social media user modeling must handle heterogeneous modalities, irregular sparse activity, and temporal preference evolution.
- DUMoE jointly models multimodal signals, multi-scale temporal dynamics, and explicit and latent user preferences.
- The framework combines shared cross-modal encoding, recent-activity sequence modeling, historical aggregation, and sparse expert routing.
- The introduced dataset covers 15 domains, 14K+ users, 7.7M tweets, and 2.9M images with chronologically evolving interest annotations.
2 Related Work
Prior work advances multimodal representation learning, sequential behavior modeling, and user-interest disentanglement, but these directions are largely developed separately. The paper identifies a need to integrate heterogeneous content, temporal evolution, and diverse interests.
- Multimodal Representation Learning: Multimodal methods align textual and visual signals but are not designed to model evolving user preferences.
- Sequential User Behavior Modeling: Sequential models capture short-term or long-range behavioral dependencies, yet largely operate on item-level sequences with limited semantic expressiveness.
- User Interest Disentanglement: Multi-interest methods extract or disentangle multiple preference representations using routing, attention, mutual information, graphs, prototypes, or clustering.
- Summary: Existing multimodal, sequential, and disentanglement approaches rarely jointly consider temporal evolution and heterogeneous content.
3 Problem Formulation
The problem formulation represents each user through a static profile and a chronologically ordered multimodal interaction sequence. The learned representation is intended to capture interaction preferences, multi-scale temporal dependencies, and diverse interests.
- Each user u is associated with a profile p_u and a chronologically ordered interaction sequence S_u.
- The representation h_u captures interaction-level preference signals, evolving multi-scale temporal dependencies, and multi-interest structure.
- Interactions pair multimodal content with timestamps arranged in chronological order.
- Multimodal interaction content can include posted or retweeted text and image content.
- The profile contains static numerical attributes, such as follower counts, and textual descriptions, such as biographies.
4 Methodology
DUMoE combines multimodal encoding, multi-scale temporal modeling, and sparse expert routing to learn drift-aware user representations. Its supervised objectives and staged optimization jointly support interest classification, interaction prediction, and more stable expert specialization.
- Multimodal Encoding: The framework uses a multimodal encoder to map user posts and profile attributes into a shared embedding space.Posts combine textual and visual content, while profiles include numerical and textual attributes.
- Temporal Dynamics-Aware Backbone: The backbone integrates static profiles, short-term behavior, and long-term dependencies into a unified representation.A two-layer MLP encodes static features, an LSTM models recent behavior, and a Transformer captures long-range dependencies.
- Temporal Dynamics-Aware Backbone: Short-term and long-term interaction sequences use windows of lengths Lt and Ll, respectively, with Lt << Ll.Both windows end at the user’s current time step Tu.
- Interest Adapter: The sparse MoE adapter assigns lightweight experts to distinct latent interest subspaces and routes each user to a sparse subset of relevant experts.Residual expert transformations remain anchored to the shared backbone representation, while top-K′ routing selects dominant interests.
- Interest Adapter: The final representation is a gated aggregation of expert outputs, with gating weights normalized across experts and only a subset receiving non-zero weights.Each expert output is Ak(zu), and αu,k denotes its user-specific gating weight.
- Learning Objective and Training Procedure: Training combines interest classification and interaction prediction losses, then uses three stages for backbone pre-training, expert specialization, and gating optimization.The objective is L = Lcls + λ Lrec, with class-weighted focal loss for classification and negative-sampling BCE for interaction prediction.
- Learning Objective and Training Procedure: The stage-wise design decouples representation learning, expert specialization, and routing optimization to improve stability, clarify expert roles, and support personalized routing.Prediction heads are jointly optimized across all stages.
5 Experiments
Experiments evaluate DUMoE on multimodal X data for interest classification and future interaction prediction, then test its components through ablations. DUMoE consistently achieves strong ranking, retrieval, and distributional-alignment performance, while ablations support the value of multi-scale branches, sparse expert routing, and staged training.
- Experimental Setup: The evaluation uses a large-scale multimodal X dataset spanning 15 interest domains, with 14,015 users, 7,685,700 posts, and 2,890,668 images.Samples are built from chronological collections of 30 posts and GPT-5.1-assigned labels using preceding collections as context.
- Experimental Setup: Interest classification evaluates macro Recall, Hit@1, NDCG@3, and KL divergence, while interaction prediction reports Accuracy, Recall, F1, and BCE.The metrics cover top-rank accuracy, ranking quality, distribution alignment, classification, retrieval, and calibration.
- User Interest Classification: DUMoE achieves Hit@1 0.872, Recall 0.856, and NDCG@3 0.940, outperforming MIND and PTUM and improving Hit@1 by 8.1% over HORAE.It also reduces KL divergence to 0.323, a 31.4% reduction versus MIND’s 0.471.
- User Interaction Prediction: For future interaction prediction, DUMoE obtains Recall 0.926 and F1 0.888, surpassing PeterRec’s Recall 0.894 and F1 0.885.Accuracy is 0.883, comparable with PTUM, MIND, and PeterRec, while BCE is 0.276.
- Ablation Study: Single-branch ablations show Long-term Only performs best among isolated branches at Hit@1 0.862, while Short-term Only and Static Only reach 0.805 and 0.774.Long-term Only still trails the full model by 1.1%, indicating that the branches contribute complementary temporal information.
- Ablation Study: Removing the adapter or sparse gating lowers performance, while disabling stage-wise training yields Hit@1 0.865 and KL 0.341.Lcls alone favors interest classification but harms interaction prediction, whereas Lrec alone shows the opposite trade-off.
- Discussion: The discussion identifies dependence on GPT-5.1 pseudo-labels, fixed temporal windows, and a fixed expert count with static top-K′ routing as limitations.The paper proposes semi-supervised or self-supervised labels, adaptive windows, and dynamic expert allocation as future directions.
6 Conclusion
The paper presents a dynamic user profiling framework that jointly models explicit and implicit preferences across multiple temporal scales. It combines a multi-scale backbone, sparse MoE interest adapter, and three-stage training strategy, with experiments showing consistent improvements over strong baselines.
- The framework jointly models explicit and implicit preferences across multiple temporal scales.
- Its multi-scale backbone fuses static, short-term, and long-term features for dynamic user profiling.
- A sparsely routed MoE interest adapter decomposes multiple interests, while three-stage training supports stable specialization.
- Extensive experiments on user interest classification and interaction prediction show consistent outperformance of strong baselines.
- A large-scale multimodal X dataset with temporally evolved interest labels is contributed for evaluation and reproducibility.