Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts

Ziqing Qian; Haohang Chen; Shengqi Dang; Yuhan Xiong; Canyu Shen; Jiaying Lei; Nan Cao

Source-linked AI summary

Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts

Ziqing Qian, Haohang Chen, Shengqi Dang, Yuhan Xiong, Canyu Shen, Jiaying Lei, Nan Cao

arXiv:2608.25773v1Aug 26, 2026cs.LG

TL;DR

Interest drift complicates user preference modeling because social-media preferences evolve across multiple time scales and span co-existing interests amid heterogeneous, sparse activity. DUMoE unifies multimodal temporal modeling with a sparse mixture-of-experts adapter and three-stage optimization. Across user interest classification and interaction prediction, it consistently outperforms strong baselines, while its pseudo-labeling and fixed temporal windows remain limitations.

Problem
Interest drift requires modeling heterogeneous multimodal behavior, multi-scale temporal dynamics, irregular sparse activity, and multiple co-existing interests in one user representation.
Source: §1 · p. 1
Method
DUMoE combines a backbone for static, short-term, and long-term signals with sparse expert routing for multi-interest disentanglement and three-stage training.
Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 6
Results
DUMoE consistently outperforms strong baselines on user interest classification and interaction prediction tasks.
Source: §1 · p. 2, §6 · p. 9
Takeaways & Limitations
Unified modeling of multimodal signals, temporal dynamics, and interest structure supports drift-aware user representation learning in the evaluated tasks.
Source: §1 · p. 2, §6 · p. 9
Pseudo-label quality may be affected by language-model annotation noise or domain-specific bias, and the backbone uses fixed temporal window lengths.
Source: §5.6 · p. 9

Abstract

from arXiv · show

Understanding user preferences from noisy and temporally evolving social media behaviors is fundamentally challenging due to interest drift, where user preferences shift across time and exhibit both multi-scale temporal patterns and diverse co-existing interests. To address this, we propose DUMoE, a unified framework for drift-aware multimodal user representation learning. Our model consists of (i) a temporal dynamics-aware backbone that captures and integrates static profiles, short-term behavioral signals, and long-term dependencies into a coherent representation, and (ii) a sparse mixture-of-experts (MoE) interest adapter that disentangles multiple latent interests via expert specialization and adaptive routing. Each expert models a distinct interest subspace, while a gating network dynamically selects and aggregates a sparse subset of relevant experts for each user. To enable stable and effective optimization, we further introduce a three-stage training strategy that decouples backbone learning, expert specialization, and gating optimization. Extensive experiments on real-world social media datasets show that DUMoE consistently outperforms state-of-the-art methods on both user interest prediction and interaction prediction tasks.

1 Introduction

Interest drift makes user preference modeling difficult because preferences combine long-term stability, short-term dynamics, and multiple co-existing interests. DUMoE addresses this with unified multimodal, multi-scale temporal, and multi-interest representation learning.

Interest drift combines evolving preferences, multi-scale temporal dynamics, and multiple co-existing interests.Source: §1 · p. 1
Social media user modeling must handle heterogeneous modalities, irregular sparse activity, and temporal preference evolution.Source: §1 · p. 1
DUMoE jointly models multimodal signals, multi-scale temporal dynamics, and explicit and latent user preferences.Source: §1 · p. 2
The framework combines shared cross-modal encoding, recent-activity sequence modeling, historical aggregation, and sparse expert routing.Source: §1 · p. 2
The introduced dataset covers 15 domains, 14K+ users, 7.7M tweets, and 2.9M images with chronologically evolving interest annotations.Source: §1 · p. 2

2 Related Work

Prior work advances multimodal representation learning, sequential behavior modeling, and user-interest disentanglement, but these directions are largely developed separately. The paper identifies a need to integrate heterogeneous content, temporal evolution, and diverse interests.

Multimodal Representation Learning

Multimodal methods align textual and visual signals but are not designed to model evolving user preferences.Source: §2 · pp. 2–3

Sequential User Behavior Modeling

Sequential models capture short-term or long-range behavioral dependencies, yet largely operate on item-level sequences with limited semantic expressiveness.Source: §2 · p. 2

User Interest Disentanglement

Multi-interest methods extract or disentangle multiple preference representations using routing, attention, mutual information, graphs, prototypes, or clustering.Source: §2 · p. 2

Summary

Existing multimodal, sequential, and disentanglement approaches rarely jointly consider temporal evolution and heterogeneous content.Source: §2 · p. 3

3 Problem Formulation

The problem formulation represents each user through a static profile and a chronologically ordered multimodal interaction sequence. The learned representation is intended to capture interaction preferences, multi-scale temporal dependencies, and diverse interests.

Each user u is associated with a profile p_u and a chronologically ordered interaction sequence S_u.Source: §3 · p. 3
The representation h_u captures interaction-level preference signals, evolving multi-scale temporal dependencies, and multi-interest structure.Source: §3 · p. 3
Interactions pair multimodal content with timestamps arranged in chronological order.Source: §3 · p. 3
Multimodal interaction content can include posted or retweeted text and image content.Source: §3 · p. 3
The profile contains static numerical attributes, such as follower counts, and textual descriptions, such as biographies.Source: §3 · p. 3

4 Methodology

DUMoE combines multimodal encoding, multi-scale temporal modeling, and sparse expert routing to learn drift-aware user representations. Its supervised objectives and staged optimization jointly support interest classification, interaction prediction, and more stable expert specialization.

Multimodal Encoding

The framework uses a multimodal encoder to map user posts and profile attributes into a shared embedding space.Posts combine textual and visual content, while profiles include numerical and textual attributes.Source: §4.1 · pp. 3–4

Temporal Dynamics-Aware Backbone

The backbone integrates static profiles, short-term behavior, and long-term dependencies into a unified representation.A two-layer MLP encodes static features, an LSTM models recent behavior, and a Transformer captures long-range dependencies.Source: §4.1 · p. 4
Short-term and long-term interaction sequences use windows of lengths Lt and Ll, respectively, with Lt << Ll.Both windows end at the user’s current time step Tu.Source: §4.1 · p. 4

Interest Adapter

The sparse MoE adapter assigns lightweight experts to distinct latent interest subspaces and routes each user to a sparse subset of relevant experts.Residual expert transformations remain anchored to the shared backbone representation, while top-K′ routing selects dominant interests.Source: §4.2 · pp. 4–5
The final representation is a gated aggregation of expert outputs, with gating weights normalized across experts and only a subset receiving non-zero weights.Each expert output is Ak(zu), and αu,k denotes its user-specific gating weight.Source: §4.2 · p. 4

Learning Objective and Training Procedure

Training combines interest classification and interaction prediction losses, then uses three stages for backbone pre-training, expert specialization, and gating optimization.The objective is L = Lcls + λ Lrec, with class-weighted focal loss for classification and negative-sampling BCE for interaction prediction.Source: §4.3 · p. 5
The stage-wise design decouples representation learning, expert specialization, and routing optimization to improve stability, clarify expert roles, and support personalized routing.Prediction heads are jointly optimized across all stages.Source: §4.3 · pp. 5–6

5 Experiments

Experiments evaluate DUMoE on multimodal X data for interest classification and future interaction prediction, then test its components through ablations. DUMoE consistently achieves strong ranking, retrieval, and distributional-alignment performance, while ablations support the value of multi-scale branches, sparse expert routing, and staged training.

Experimental Setup

The evaluation uses a large-scale multimodal X dataset spanning 15 interest domains, with 14,015 users, 7,685,700 posts, and 2,890,668 images.Samples are built from chronological collections of 30 posts and GPT-5.1-assigned labels using preceding collections as context.Source: §5.1 · p. 6
Interest classification evaluates macro Recall, Hit@1, NDCG@3, and KL divergence, while interaction prediction reports Accuracy, Recall, F1, and BCE.The metrics cover top-rank accuracy, ranking quality, distribution alignment, classification, retrieval, and calibration.Source: §5.2 · p. 7

User Interest Classification

DUMoE achieves Hit@1 0.872, Recall 0.856, and NDCG@3 0.940, outperforming MIND and PTUM and improving Hit@1 by 8.1% over HORAE.It also reduces KL divergence to 0.323, a 31.4% reduction versus MIND’s 0.471.Source: §5.3 · pp. 7–8

User Interaction Prediction

For future interaction prediction, DUMoE obtains Recall 0.926 and F1 0.888, surpassing PeterRec’s Recall 0.894 and F1 0.885.Accuracy is 0.883, comparable with PTUM, MIND, and PeterRec, while BCE is 0.276.Source: §5.4 · p. 8

Ablation Study

Single-branch ablations show Long-term Only performs best among isolated branches at Hit@1 0.862, while Short-term Only and Static Only reach 0.805 and 0.774.Long-term Only still trails the full model by 1.1%, indicating that the branches contribute complementary temporal information.Source: §5.5 · p. 8
Removing the adapter or sparse gating lowers performance, while disabling stage-wise training yields Hit@1 0.865 and KL 0.341.Lcls alone favors interest classification but harms interaction prediction, whereas Lrec alone shows the opposite trade-off.Source: §5.5 · pp. 8–9

Discussion

The discussion identifies dependence on GPT-5.1 pseudo-labels, fixed temporal windows, and a fixed expert count with static top-K′ routing as limitations.The paper proposes semi-supervised or self-supervised labels, adaptive windows, and dynamic expert allocation as future directions.Source: §5.6 · p. 9

6 Conclusion

The paper presents a dynamic user profiling framework that jointly models explicit and implicit preferences across multiple temporal scales. It combines a multi-scale backbone, sparse MoE interest adapter, and three-stage training strategy, with experiments showing consistent improvements over strong baselines.

The framework jointly models explicit and implicit preferences across multiple temporal scales.Source: §6 · p. 9
Its multi-scale backbone fuses static, short-term, and long-term features for dynamic user profiling.Source: §6 · p. 9
A sparsely routed MoE interest adapter decomposes multiple interests, while three-stage training supports stable specialization.Source: §6 · p. 9
Extensive experiments on user interest classification and interaction prediction show consistent outperformance of strong baselines.Source: §6 · p. 9
A large-scale multimodal X dataset with temporally evolved interest labels is contributed for evaluation and reproducibility.Source: §6 · p. 9