Source-linked AI summary
Modeling Long-term User Behaviors with Diffusion-driven Multi-interest Network for CTR Prediction
Weijiang Lai, Beihong Jin, Yapeng Zhang, Yiyuan Zheng, Rui Zhao, Jian Dong, Jun Lei, Xingxing Wang
TL;DR
Long-term CTR modeling faces challenges from coupled interests, noise, and lengthy behavior sequences, while two-stage filtering can constrain the range of interests represented. DiffuMIN combines target-oriented multi-interest extraction with guided diffusion and contrastive learning; the authors report superiority in offline experiments and online A/B testing.
Problem
Long-term behavior modeling must address coupled interests, noise, and lengthy sequences, while two-stage filtering can constrain the user-interest space.
Source: §1 · pp. 1–2Method
DiffuMIN extracts multiple interests using target-derived orthogonal channels, then uses guided diffusion to generate augmented interests aligned with users’ preferences.
Source: §1 · p. 2, §3.5 · p. 6Results
Takeaways & Limitations
The authors conclude that DiffuMIN preserves and expands the constrained interest space in long-term behavior modeling, enhancing model capacity.
Source: §5 · p. 9The formulation represents users through behavior sequences and features such as age and gender, with target-item features and binary interaction labels.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
CTR (Click-Through Rate) prediction, crucial for recommender systems and online advertising, etc., has been confirmed to benefit from modeling long-term user behaviors. Nonetheless, the vast number of behaviors and complexity of noise interference pose challenges to prediction efficiency and effectiveness. Recent solutions have evolved from single-stage models to two-stage models. However, current two-stage models often filter out significant information, resulting in an inability to capture diverse user interests and build the complete latent space of user interests. Inspired by multi-interest and generative modeling, we propose DiffuMIN (Diffusion-driven Multi-Interest Network) to model long-term user behaviors and thoroughly explore the user interest space. Specifically, we propose a target-oriented multi-interest extraction method that begins by orthogonally decomposing the target to obtain interest channels. This is followed by modeling the relationships between interest channels and user behaviors to disentangle and extract multiple user interests. We then adopt a diffusion module guided by contextual interests and interest channels, which anchor users' personalized and target-oriented interest types, enabling the generation of augmented interests that align with the latent spaces of user interests, thereby further exploring restricted interest space. Finally, we leverage contrastive learning to ensure that the generated augmented interests align with users' genuine preferences. Extensive offline experiments are conducted on two public datasets and one industrial dataset, yielding results that demonstrate the superiority of DiffuMIN. Moreover, DiffuMIN increased CTR by 1.52% and CPM by 1.10% in online A/B testing. Our source code is available at https://github.com/laiweijiang/DiffuMIN.
1 INTRODUCTION
Long-term behavior modeling can improve CTR prediction, but coupled interests, noise, and lengthy sequences make it difficult to achieve both effectiveness and efficiency. DiffuMIN combines target-oriented multi-interest extraction with guided diffusion to explore user-interest space.
1 INTRODUCTION
- Coupled interests, noise interference, and lengthy behavior sequences challenge accurate interest extraction and make high-complexity mechanisms such as self-attention infeasible.Source: §1 · p. 1
- Two-stage models reduce efficiency and noise problems by filtering target-relevant behaviors, but single-perspective filtering constrains interest diversity and performance.Earlier single-stage attention approaches can also miss user interests because of simplified mechanisms.Source: §1 · p. 2
- DiffuMIN’s target-oriented extraction and guided diffusion generate augmented interests to preserve and expand the constrained user-interest space.Contrastive learning aligns generated interests with users’ actual preferences.Source: §1 · p. 2
- The method decomposes the target into orthogonal interest channels, routes behaviors by relevance, and aggregates the top-p% within each channel.Its diffusion module uses contextual interests and interest channels as guidance, while contrastive learning aligns augmented interests with actual preferences.Source: §1 · p. 2
- Offline experiments on three real-world datasets and online A/B testing are reported, with the authors stating that DiffuMIN achieves state-of-the-art performance.Source: §1 · p. 2
2 RELATED WORK
Prior work addresses long-term CTR modeling, multi-interest extraction, and generative recommendation, but the paper identifies limitations in interest coverage and applicability to long behavior sequences. It positions DiffuMIN as applying diffusion to user-interest modeling for CTR prediction.
2.1 CTR Models for Long-term Behaviors
- Two-stage CTR models select target-relevant behaviors, but the paper says current approaches constrain interest expression and overall performance.Simplified attention models may miss long-term behavior nuances; filtering-based methods include SIM, UBR4CTR, TWIN, and CoFARS.Source: §2.1 · p. 2
2.2 Multi-interest Modeling Methods
- Multi-interest methods use routing, attention, orthogonal channels, or clustering, but the paper notes their behavior-only focus or high complexity limits long-term CTR use.Examples include MIND, ComiRec, DMIN, Octopus, and Trinity.Source: §2.2 · p. 3
2.3 Generative Modeling Methods
- Generative recommendation work applies diffusion to items, embeddings, behavior sequences, and augmented data, while the paper states diffusion had not been applied to user interests for CTR prediction.Examples include DreamRec [67], DiffRec [66], DiffuASR, and CaDiRec.Source: §2.3 · p. 3
3 METHODOLOGY
The paper formulates CTR prediction from a user, a target item, and their interaction label. DiffuMIN’s architecture includes input, interest extraction, interest generation, calibration, and prediction components.
3.1 Problem Formulation
- The task is to predict the click-through probability for a user and target item given their binary interaction label.The user is represented by a behavior sequence and features such as age and gender; the target includes item and contextual features.Source: §3.1 · p. 3
- The prediction is represented by a model F(x; θ), where θ denotes the model parameters.Source: §3.1 · p. 3
- DiffuMIN’s architecture comprises an input layer, Orthogonal Multi-Interest Extractor, Diffusion Multi-Interest Generator, Contrastive Multi-Interest Calibrator, and prediction layer.Source: §3.1 · p. 3
3.2 Input Layer
The input layer initializes embeddings for behavior sequences, target items, and other features. Recent behaviors and the full behavior sequence then feed separate modeling paths for current and multiple interests.
3.2 Input Layer
- A uniform embedding table initializes behavior-sequence, target-item, and other-feature embeddings with a shared dimension d.The behavior matrix is E = [e_1, ..., e_l], and e_i is the embedding of behavior i.Source: §3.2 · p. 3
- The recent k behavior embeddings and target embedding enter a Transformer encoder for current interest, while the full sequence and target enter OMIE for multiple interests.Source: §3.2 · p. 3
3.3 Orthogonal Multi-Interest Extractor
The extractor addresses interest redundancy and incomplete coverage in long-term behaviors by separating user interests across target-derived channels. It matches behaviors to those channels, then routes, filters, and aggregates them into multiple interests.
3.3 Orthogonal Multi-Interest Extractor
- Orthogonal target-derived interest channels let OMIE extract interests from multiple perspectives, mitigating the reduction of interests common in two-stage approaches.The method emphasizes behavior-to-target relationships, which the paper describes as important in CTR scenarios.Source: §3.3 · p. 3
- Each channel is a basis embedding derived by linearly projecting and orthogonally decomposing the target embedding.The channels represent distinct target-oriented aspects of user interest, and their orthogonality reduces redundancy.Source: §3.3 · pp. 3–4
- Behavior-channel relevance scores are computed from the user behavior matrix and channel matrix, with each score representing one behavior’s relevance to one channel.These scores guide subsequent routing and channel filtering.Source: §3.3 · p. 4
- The extractor routes each behavior to its highest-scoring channel, retains the top-p% behaviors per channel, and aggregates the retained behaviors.Mean pooling is used for aggregation.Source: §3.3 · p. 4
3.4 Diffusion Multi-Interest Generator
The diffusion generator complements aggregated interests by modeling their distribution and generating augmented interests. Contextual interests and target-derived channels guide optimization and sampling, which starts from perturbed interests rather than Gaussian noise.
3.4 Diffusion Multi-Interest Generator
- The diffusion module complements aggregated interests by generating augmented interests that conform to user-interest latent spaces, addressing the space restriction that remains after OMIE.Contextual interests and interest channels condition generation to anchor personalized interest scopes and target-oriented types.Source: §3.4 · pp. 4–5
3.4.1 Diffusion Optimization Phase.
- During optimization, the forward process adds noise to aggregated interests, while the reverse process reconstructs them conditioned on contextual interests and the corresponding interest channel.The diffusion network uses self-attention and cross-attention to incorporate these conditions.Source: §3.4.1 · p. 5
3.4.2 Diffusion Sampling Phase.
- Sampling starts from a perturbed user interest rather than Gaussian noise and iteratively denoises it over T′ steps, where T′ << T, to produce an augmented interest.The sampling phase generates c augmented-interest perspectives per user; training performs optimization and sampling, while inference performs sampling only.Source: §3.4.2 · p. 5
3.5 Contrastive Multi-Interest Calibrator
Contrastive learning compares each aggregated interest with augmented interests to improve representation learning. It aligns an interest with the same user’s augmented interest while distinguishing interests across users.
3.5 Contrastive Multi-Interest Calibrator
- The calibrator uses contrastive learning to examine relationships between aggregated and diffusion-generated augmented interests.For an aggregated interest, its same-user augmented interest is positive and other users’ augmented interests in the mini-batch are negatives.Source: §3.5 · p. 6
- The contrastive loss aligns augmented interests with users’ actual preferences and enhances interest distinguishability across users.The loss is defined over users in a mini-batch.Source: §3.5 · p. 6
3.6 Prediction Layer
The prediction layer feeds long- and short-term interest representations into an MLP for CTR prediction. The model uses a sigmoid output and binary cross-entropy training loss.
3.6 Prediction Layer
- The prediction MLP receives aggregated interests, augmented interests, the encoded short-term interest, and e_other.The short-term interest is the last element of the Transformer encoder output.Source: §3.6 · p. 6
- The model applies a sigmoid function to produce its CTR prediction.Source: §3.6 · p. 6
- Binary cross-entropy is used for optimization, with N denoting the number of samples and y_i the label of sample i.Source: §3.6 · p. 6
3.7 Complexity Analysis
DiffuMIN’s added parameters come mainly from OMIE and CMIC projection layers and the DMIG diffusion network. With the stated configuration, its overall time complexity is approximately O(bldc), comparable to recent long-term behavior models.
3.7.1 Space Complexity.
- The added parameters primarily come from OMIE and CMIC projection layers and the DMIG diffusion network, with spatial complexities O(cd^2 + 2d^2) and L(8d^2 + 2dd_f).The passage characterizes the resulting parameter increase as minimal.Source: §3.7.1 · p. 6
3.7.2 Time Complexity.
- Given c << l and small T′, overall time complexity is approximately O(bldc), making DiffuMIN’s efficiency comparable to recent long-term behavior models.The component complexities are O(Blcd + Blc^2) for OMIE, O(BT′L(cd^2 + c^2d + cdd_f)) for DMIG, and O(bcd^2 + bc^2d) for CMIC.Source: §3.7.2 · p. 6
4 EXPERIMENTS
Across three datasets, DiffuMIN achieves optimal performance among the reported competitors, and a live test reports higher CTR and CPM after replacing SIM in the baseline. The experiments also examine module ablations, design variants, and user-interest visualizations.
4 EXPERIMENTS
- The experiments use two public datasets and one industrial dataset, compare CTR models including DIN, CAN, SIM, ETA, SDIM, and TWIN, and evaluate offline AUC and online CTR and CPM.Alibaba1 contains 22 days of shopping behavior; Ele.me contains 30 days of user behaviors; the industrial dataset covers a year of behavior records for users active in the last seven days. Models were trained for one epoch with shared feature dimensions and configurations.Source: §4.1.1 · pp. 6–7, §4.1.2 · p. 7, §4.1.3 · p. 7
4.2 Overall Performance (RQ1)
- DiffuMIN achieves optimal performance across the reported comparisons, with AUC differences from comparison models significant at the 0.05 t-test level.The experiments use Table 2 results across three datasets; the paper notes that AUC improvements of 0.001 are considered significant in CTR prediction.Source: §4.2 · p. 7, Table 2 · p. 7
4.3 Ablation Study (RQ2)
- Ablations indicate that target-oriented multi-interest extraction improves performance, multiple aggregated interests are essential, and augmented interests complement the original interests.The results also indicate that diffusion generates augmented interests, contrastive learning boosts representation expressiveness, and short-term interest extraction matters.Source: §4.3 · pp. 7–8, Table 3 · p. 8
4.4 Analysis of OMIE (RQ3)
- OMIE’s channel count, top-1 behavior routing, and top-p% channel filtering affect performance; target-attention aggregation adds no benefit, while self-attention can help at substantial time cost.Too few channels limit interest diversity, whereas too many can reduce channel effectiveness.Source: §4.4 · p. 8, Figure 2 · p. 8
4.5.1 Advantages of Diffusion Module.
- DiffuMIN-VAE does not outperform DiffuMIN on the industry and Ele.me datasets and only approximates its results on Alibaba.The comparison replaces DiffuMIN’s diffusion module with a VAE module.Source: §4.5.1 · p. 8, Table 4 · p. 8
4.5.2 Adaptation of Diffusion Module.
- Removing either contextual-interest or interest-channel guidance harms generation quality, especially when both are absent; generating from Gaussian noise also requires additional sampling steps for high-quality embeddings.Variant Z generates directly from Gaussian noise.Source: §4.5.2 · p. 8, Figure 2 · p. 8
4.6 Case Study (RQ5)
- In two industrial-user t-SNE cases, TWIN interests lie near targets, while DiffuMIN’s target channels lie near one or two of its interest embeddings.The paper attributes DiffuMIN’s broader interest-space exploration to channel-based behavior aggregation and generative extension.Source: §4.6 · pp. 8–9, Figure 3 · p. 9
4.7 Online A/B Test (RQ6)
- CTR increased 1.52% and CPM increased 1.10% after DiffuMIN replaced SIM in the live baseline, while inference time rose from 33ms to 35ms.The online A/B test ran for seven days in the live production environment.Source: §4.7 · p. 9
5 CONCLUSION
DiffuMIN models long-term user behaviors by extracting multiple interests and generating augmented interests. Offline experiments and online A/B testing demonstrate its superiority over existing models.
5 CONCLUSION
- Target-oriented multi-interest extraction disentangles and extracts multiple user interests from long-term user behaviors.The conclusion describes this method as the first stage of DiffuMIN.Source: §5 · p. 9
- A diffusion module guided by contextual interests and interest channels generates multiple augmented interests.The conclusion says this approach preserves and expands the constrained interest space in long-term behavior modeling.Source: §5 · p. 9
- Offline experiments and online A/B testing demonstrate DiffuMIN’s superiority over existing models.Source: §5 · p. 9