Source-linked AI summary
Future-Conditioned Recommendations with Multi-Objective Controllable Decision Transformer
Chongming Gao, Kexin Huang, Ziang Fei, Jiaju Chen, Jiawei Chen, Jianshan Sun, Shuchang Liu, Qingpeng Cai, Peng Jiang
TL;DR
Recommendation systems face uncertain future impacts and conflicting objectives, while changing goals make repeated evaluation and retraining burdensome. MocDT conditions autoregressive item-sequence generation on prioritized objectives; the authors report effective control and adaptation, while noting that objective feasibility and Pareto optimality are not fully verified.
Problem
Recommendation decisions have difficult-to-measure future impacts, and systems must reconcile potentially conflicting objectives such as accuracy and diversity.
Source: §1 · p. 1Method
MocDT uses offline interaction data to map prioritized future objectives to autoregressive item sequences, allowing objectives to change at inference without retraining.
Source: §1 · p. 2Results
Takeaways & Limitations
The approach provides a controllable recommendation strategy for generating sequences aligned with predefined multiple objectives.
Source: §6 · p. 9The study cannot fully verify the Pareto optimality of specified objectives or the feasibility of objectives provided to the model.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Securing long-term success is the ultimate aim of recommender systems, demanding strategies capable of foreseeing and shaping the impact of decisions on future user satisfaction. Current recommendation strategies grapple with two significant hurdles. Firstly, the future impacts of recommendation decisions remain obscured, rendering it impractical to evaluate them through direct optimization of immediate metrics. Secondly, conflicts often emerge between multiple objectives, like enhancing accuracy versus exploring diverse recommendations. Existing strategies, trapped in a "training, evaluation, and retraining" loop, grow more labor-intensive as objectives evolve. To address these challenges, we introduce a future-conditioned strategy for multi-objective controllable recommendations, allowing for the direct specification of future objectives and empowering the model to generate item sequences that align with these goals autoregressively. We present the Multi-Objective Controllable Decision Transformer (MocDT), an offline Reinforcement Learning (RL) model capable of autonomously learning the mapping from multiple objectives to item sequences, leveraging extensive offline data. Consequently, it can produce recommendations tailored to any specified objectives during the inference stage. Our empirical findings emphasize the controllable recommendation strategy's ability to produce item sequences according to different objectives while maintaining performance that is competitive with current recommendation strategies across various objectives.
1 Introduction
MocDT is presented as a way to generate item sequences conditioned on prioritized future objectives, addressing the challenges of uncertain future impact and competing recommendation goals. The introduction also describes data augmentation and evaluations of multi-objective sequence generation.
1 Introduction
- Recommendation systems struggle to measure future impacts directly and to reconcile conflicting objectives such as accuracy and diversity.Repeated recommendation and evaluation becomes more burdensome as objectives increase in number and complexity.Source: §1 · p. 1
- Earlier decision-transformer recommenders addressed single objectives, while multi-objective work did not focus on changing objectives during inference.The introduction also notes that evaluation focused on immediate metrics and left planning and long-term objectives such as diversity underexamined.Source: §1 · p. 2
- MocDT conditions autoregressive item-sequence generation on prioritized future objectives, avoiding retraining when objectives change.The approach draws on offline RL’s mapping of rewards to actions rather than forecasting rewards for each action.Source: §1 · pp. 1–2
- Three data-augmentation strategies are proposed to enrich offline interaction data when recommendation datasets lack comprehensive trajectories.The strategies are intended to improve generalization across recommendation objectives.Source: §1 · p. 2
- Across five method categories, MocDT generates trajectories adhering to predefined multi-objective goals.Evaluation uses randomly selected objectives and sequential item generation.Source: §1 · p. 2
2 Preliminary
The preliminary section introduces offline RL and decision transformers. Offline RL learns policies from fixed datasets of trajectories, while a decision transformer models action prediction as sequence modeling.
2.1 Offline Reinforcement Learning
- Offline RL learns a policy from a fixed dataset of decision trajectories, which is relevant when online interactions are risky or expensive.The RL problem is framed as a Markov Decision Process with state and action spaces, transitions, rewards, and a discount factor.Source: §2.1 · p. 2
2.2 Decision Transformer
- A decision transformer reformulates offline RL as sequence modeling and predicts actions conditioned on the current state, returns-to-go, and preceding trajectory.Training minimizes empirical negative log-likelihood on trajectories from the offline dataset.Source: §2.2 · p. 2
3 Method
MocDT treats interactive recommendation as a multi-objective decision problem and conditions item generation on prioritized future goals. Its architecture combines user, objective, and interaction-history representations, while synthetic trajectories augment offline data with alternative future paths.
3 Method
- Interactive recommendation is formulated as sequential decision-making, with user histories as states, recommended items as actions, and immediate indicators as rewards.The method addresses aggregate objectives such as diversity that cannot be represented by a reward on an individual state-action pair.Source: §3 · p. 3
3.1 DT as Future-Conditioned Recommender
- A sliding H-step future window captures both cumulative rewards and trajectory-level objectives, while a prioritized control signal guides recommendations without fixing session length.The framework models historical interactions as states, recommended items as actions, and immediate indicators such as ratings as rewards; aggregate objectives such as diversity require trajectory-level evaluation.Source: §3.1 · p. 3, §3 · p. 3
3.2 Model Architecture
- The Sequence Transformer combines encoded user, objective, history, and control inputs to decode item recommendations across the future horizon.The feature encoders represent user information, normalized objectives, and interaction history; a GRU encodes history, and an MLP constructs the control representation.Source: §3.2 · p. 3
3.3 Enhance MocDT with Synthetic Data
- Three augmentation strategies build alternative future paths from observed interactions: high-rating items, category-diverse items, and random items.The synthetic to-go sequences are length 2H and are concatenated with the original trajectory; they support exploration within the available offline data. The approach uses existing interactions rather than a trained model to fabricate data.Source: §3.3 · p. 4
4 Experiments
The experiments ask how MocDT compares with existing methods, controls specified long-term goals, and responds to augmentation and parameter choices. Evaluation uses three public datasets, defined rating and diversity metrics, and a range of recommender baselines.
4 Experiments
- The experiments investigate benchmark performance, objective control, augmentation effects, and key-parameter effects, with a case study illustrating model operation.The research questions also examine the augmentation rate.Source: §4 · p. 5
4.1.1 Metrics.
- Accuracy and diversity are assessed using listwise cumulative rating and diversity metrics.The metrics are formalized in Equations (1) and (2).Source: §4.1.1 · p. 5
4.1.2 Datasets.
- Evaluation uses MovieLens-1m, KuaiRand-Pure, and Zhihu-1m, whose item categories support calculation of the diversity metric.Because offline interaction data are sparse, completed user-item matrices are used for evaluation only, not model training.Source: §4.1.2 · p. 5
4.1.3 Baselines.
- Baselines span sequential, multi-objective, multi-objective RL, controllable, and Decision Transformer recommendation methods.The listed methods include BERT4Rec, RMTL, PHN variants, DT4Rec, and CDT4Rec.Source: §4.1.3 · p. 5
4.2 Overall Performance Comparison (RQ1)
Across three datasets, the evaluation compares Rating and Diversity for generated 10-item sequences. MocDT is presented as handling both objectives and explicitly controlling them, unlike the compared models.
4.2 Overall Performance Comparison (RQ1)
- For models without direct metric adjustment, the comparison selects results with similar Diversity scores; each Rating–Diversity pair comes from the same run.PHN and MocDT are assessed using specified objective points, with their corresponding Ratings identified by the described matching procedure.Source: §4.2 · p. 5
- MocDT is designed to consider Rating and Diversity simultaneously, unlike Bert4Rec, DT4Rec, and CDT4Rec, which focus on a single objective.The comparison covers Diversity-focused and Cumulative Rating-focused scenarios across three datasets.Source: §4.2 · p. 5, Table 2 · p. 6
Method
- MocDT integrates explicit objective control, while RMTL and PLE lack mechanisms for such control.The passage characterizes this control as enabling adjustments toward desired outcomes across multiple objectives.Source: Method · p. 6
4.3 Results on Controlling Ability (RQ2)
MocDT’s outcomes vary with the prioritized Rating and Diversity objectives, while their relative ordering remains stable across epochs. Dataset patterns differ, including opposing Rating–Diversity relationships.
4.3 Results on Controlling Ability (RQ2)
- Across nine objective points, their relative outcome positions remain invariant throughout training epochs, indicating stable multi-objective control.The evaluation generates 10 items for each point and averages results from the concluding 10 of 30 training epochs.Source: §4.3 · p. 6
- MovieLens results cluster by objective point, with distinct clusters indicating learned mappings from objectives to recommended items.The passage associates this clustering with a strong collaborative filtering signal.Source: §4.3 · p. 6
- On KuaiRand, increased Rating emphasis corresponds to lower Diversity, while objective points retain stable relative positions despite epoch-to-epoch fluctuations.The caption defines (1.0, 0.0) as maximizing Rating without emphasis on Diversity.Source: §4.3 · p. 6, Figure 3 · p. 7
- On Zhihu-1M, Diversity and Cumulative Rating increase together, unlike the inverse relationship reported for KuaiRand.The passage suggests the positive correlation may align with preferences in Zhihu’s content-rich environment.Source: §4.3 · pp. 6–7
4.4 Evaluation on Data Augmentation (RQ3)
The evaluation varies augmentation strategies and rates, reporting Cumulative Rating under two objective settings. Results differ across datasets, with gains, negligible changes, or decreases depending on the dataset and augmentation condition.
4.4.1 Effect on Augment Strategies.
- On KuaiRand-Pure, all four augmentation strategies similarly improve performance over non-augmented data.The paper attributes the gain to augmentation broadening the data and helping MocDT learn mappings from multiple objectives to actions.Source: §4.4.1 · p. 7
- On MovieLens-1M and ZhihuRec-1M, augmentation produces negligible Cumulative Rating changes; Rating-focused augmentation outperforms random and Diversity augmentation.The reported explanation for the small overall changes is that these datasets are already diverse.Source: §4.4.1 · p. 7
4.4.2 Effect on Augmentation Rates.
- As augmentation rates rise, MovieLens Cumulative Rating changes little, whereas KuaiRand metrics rise steadily.The passage links MovieLens’s pattern to its intrinsic diversity and adequate collaborative filtering signals, and KuaiRand’s to negatively correlated indicators.Source: §4.4.2 · p. 7
- On Zhihu, the metric decreases as augmentation volume grows; the paper attributes this to unconventional instances acting as negative samples and distorting training-data distribution.The passage connects this distortion to reduced ability to learn authentic user-preference patterns reflected in the test set.Source: §4.4.2 · p. 7
4.5 Ablation Study (RQ4)
The ablations show that Transformer depth mainly affects recommendation diversity, while shorter future-sequence horizons are easier to predict but do not capture the long-term scope the study targets.
4.5 Ablation Study (RQ4)
- Pairing one Step Transformer layer with one Sequence Transformer layer was the most effective tested structure.The study reports fundamentally similar results across tested combinations of transformer structures and layer counts.Source: §4.5 · pp. 7–8
- More Transformer layers tend to increase Diversity, while Cumulative Rating changes only within a narrow range.The reported layer comparison is in Table 3.Source: §4.5 · p. 8, Table 3 · p. 8
- Smaller H values performed better because their future-trajectory predictions diverge less, but the study chose H = 10 to retain long-term benefits.The authors state that very small H values are impractical when long-term strategy benefits matter.Source: §4.5 · p. 8, Table 4 · p. 8
4.6 Case Study
For one MovieLens user, recommendations shifted from high cumulative rating with no category diversity to greater diversity with lower rating as the prioritized objective changed.
4.6 Case Study
- The case study trained MocDT for 30 epochs and generated two ten-item lists for MovieLens user ID 6040.Source: §4.6 · p. 8
- For the top 10 recommended movies, objective point (1.0, 0.0) achieved cumulative rating 42.812 and Diversity 0; (0.0, 1.0) yielded Rating 21.607 and Diversity 0.902.Source: §4.6 · p. 8
- All ten movies in the (1.0, 0.0) list belonged to one category, whereas the (0.0, 1.0) list spanned categories.These outcomes illustrate how different objective priorities corresponded to different recommendation lists.Source: §4.6 · p. 8
5 Related Work
Related work spans multi-objective recommendation, controllable optimization, and Decision Transformers for recommendation; the cited discussion identifies dynamically changing inference-stage objectives as an open gap.
5 Related Work
- The section surveys multi-objective recommendation, controllable recommendation, and Decision Transformers in recommender systems.Source: §5 · p. 8
5.1 Multi-Objective Recommendation
- Competing objectives have been addressed through re-ranking, shared-weight architectures, and RL methods that constrain or dynamically weight objectives [24] [3] [28].The discussion frames the field's goals as correlated or competing.Source: §5.1 · pp. 8–9
5.2 Multi-Objective Controllable Recommender
- Controllable recommenders express preferences explicitly or implicitly; approaches include scalarization, Pareto hypernetworks, objective-conditioned diffusion, and LLM category control [36] [34] [5] [38] [29].The discussed approaches have stated dependencies or constraints, including reward-model design, computational complexity, training-target instability, or reliance on semantic understanding and collaborative filtering.Source: §5.2 · p. 9
5.3 Decision Transformer for Recommendation
- Prior recommendation DT work addresses single objectives or combines returns, but not dynamically changing inference-stage objectives; no industrial adaptation is reported [15] [22] [27].The stated gap concerns policies that must adjust to dynamic combinations of objectives in real time.Source: §5.3 · p. 9
6 Conclusion
The conclusion presents MocDT as a way to adjust multiple objectives during inference without retraining and reports controllable item-sequence generation, while noting that objective optimality and feasibility remain unverified.
6 Conclusion
- MocDT adds a control signal to Decision Transformer so it can generate item sequences aligned with predefined objectives without retraining.The conclusion reports empirical effectiveness in controlling and adapting multi-objective sequence generation.Source: §6 · p. 9
- The study also examines how data augmentation affects model learning across settings and contexts.Source: §6 · p. 9
- The authors cannot fully verify whether specified objectives are Pareto optimal or feasible for the model.Source: §6 · p. 9