Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

1 Introduction

• Prior LLM-based recommenders commonly convert historical behaviors into prompts, pair them with next-item targets, and fine-tune a standard LoRA module.Related work has focused mainly on improving prompts, leaving the choice of LoRA unexplored.Source: §1 · p. 2
• A shared LoRA can misalign across behaviorally distinct sequences, risking negative transfer; training separate LoRAs per sequence is impractical at millions of sequences.The paper observes gradient similarity clustering among sequences close in collaborative space, while more distant clusters tend to have dissimilar gradient trajectories.Source: §1 · p. 2, Figure 1 · p. 2
• iLoRA uses a sequence representation to guide expert participation weights, assembling an instance-wise LoRA for each sequence while keeping the total parameter count equal to standard LoRA.The framework is evaluated on LastFM, MovieLens, and Steam.Source: §1 · pp. 2–3

LLM-based Sequential Recommendation

• Sequential recommendation predicts a user's next item from past interactions by producing a probability distribution over candidate items.The recommender is trained to maximize the likelihood of the true next item.Source: §2 · p. 3
• Instruction tuning fine-tunes an LLM on explicit input-output pairs, with the input describing the interaction sequence and task and the output describing the predicted item.The objective optimizes next-token predictions autoregressively.Source: §2 · p. 3

Fine-tuning with Low-rank Adaption (LoRA)

• LoRA approximates pretrained-weight updates with trainable low-rank matrices while leaving the original weights frozen.The low-rank update is applied to query and value projection matrices in the attention sublayers, with a scaling factor controlling its influence.Source: §2 · p. 3

Fine-tuning with Hybrid Prompting

• Hybrid prompting combines LLM text-token representations with recommender-derived behavioral representations to bridge natural-language and user-behavior spaces.A trainable projector maps recommender item-ID embeddings into behavioral tokens that are concatenated with text tokens.Source: §2 · pp. 3–4

3 Methodology

• iLoRA combines multiple behavior-focused experts with a sequence-specific gate to customize LoRA updates and address negative transfer between disparate sequences.The resulting instance-wise updates are applied to the base LLM’s transformer layers.Source: §3 · p. 4, §3.2.3 · p. 6

3.1 Instance-wise Generation for Sequential Recommendation

• Treating each sequence as a separate task motivates instance-wise LoRA, while separate LoRA modules for millions of sequences would be impractical in resource and time.The method seeks sequence-specific customization without excessive computational overhead.Source: §3.1 · p. 5

3.2 Instance-wise LoRA with the Mixture of Experts Concept

• iLoRA divides LoRA’s projection matrices into expert arrays, uses a gate to score experts for each sequence, and combines them into customized LoRA weights.Different sequences may activate different experts.Source: §3.2 · p. 5

3.2.1 Splitting Low-Rank Matrices into Experts

• Splitting the projection matrices into expert pairs lets each focus on a hidden aspect of preference while preserving the parameter count of standard LoRA.Each expert’s partial rank is determined by the total LoRA rank and the number of experts.Source: §3.2.1 · p. 5

3.2.2 Generating Instance-wise Attentions over Experts

• A sequential recommender’s holistic representation of a sequence feeds a softmax gate that assigns expert attention scores; distinct sequences tend to yield different scores.Similar sequences tend toward analogous attention scores, and experts can specialize in diverse behavior patterns.Source: §3.2.2 · pp. 5–6

3.2.3 Aggregating Mixture of Experts as Instance-wise LoRA

• The gate-weighted expert projections form instance-wise LoRA parameters applied to the base LLM, dynamically customizing modules while keeping the standard LoRA parameter count.Each expert’s weight reflects its relevance to the particular sequence.Source: §3.2.3 · p. 6

4.1.1 Negative Transfer in Uniform LoRA & Instance-wise LoRA

• For LLaRA, nearby sequences cluster in gradient similarity while distant sequences have lower similarity; iLoRA’s near-zero similarities between some clusters indicate mitigation of negative transfer.Gradient similarity is used to reflect sequence proximity.Source: §4.1.1 · p. 7

4.1.2 Expert Showcase in Instance-wise LoRA

• Expert activation varies by sequence: Sequence 4 assigns Expert 1 42.5% and Expert 4 18.8%, while Sequence 2 assigns Expert 4 46.7%.Sequence 9 instead divides attention nearly evenly between Experts 2 and 3, at 31.1% and 31.5%; the authors conclude activation adjusts to sequence characteristics.Source: §4.1.2 · p. 8

4.2 Performance Comparison (RQ2)

• The performance comparison covers LastFM, MovieLens, and Steam, evaluating HitRatio@1 and ValidRatio against traditional sequential and LLM-based recommenders.The supplied passages describe the comparison scope but do not provide its outcomes.Source: §4 · p. 6, Table 1 · p. 7, §4.2 · p. 8

4.3.1 Effects of Gating Network

• Across all three datasets, sequence representations as gating guidance outperform random-initialized and token-collapsed embeddings.This comparison supports the designed gating network and MoE combination.Source: §4.3.1 · p. 9

4.3.2 Effect of Expert Numbers

• Four experts give the best performance across all three datasets; compared with two experts, HitRatio@1 rises by 5.2% on LastFM, 6.4% on MovieLens, and 2.2% on Steam.Eight experts perform similarly to or slightly below four, so the authors use four as the default.Source: §4.3.2 · p. 9
• On Steam, HitRatio@1 improves over training with one, two, or four experts, while the eight-expert model reaches decent performance rapidly and then shows no significant epoch-wise improvement.The authors speculate that eight experts may overemphasize personalization, reducing generalization and causing premature overfitting.Source: §4.3.2 · p. 9

5 Conclusion

• iLoRA integrates a mixture of experts into LoRA to adapt to diverse user behaviors and mitigate negative transfer.The framework dynamically adjusts to user behavior and tailors recommendations to individual preferences.Source: §5 · p. 9
• The authors present iLoRA as a more nuanced way to tailor recommendations to individual user preferences.Source: §5 · p. 9

6 Limitation

• Computational resources limited exploration of larger expert combinations and their potential effects on recommendation performance.Source: §6 · p. 9
• The effects of hard routing with a large number of experts were not extensively investigated.Source: §6 · p. 9
• Applicability to other recommendation systems or domains remains unexplored.Source: §6 · p. 9

7 Broader Impact

• iLoRA is presented as supporting sequence-tailored recommendations and customized solutions beyond recommendation systems.The passage names education, healthcare, and e-commerce as possible domains of application.Source: §7 · p. 10
• The authors warn that training-data biases may persist and potentially amplify existing recommendation biases.Source: §7 · p. 10
• The authors note that over-customization could limit diverse viewpoints and serendipitous discovery, while personalized recommendations raise privacy concerns.Source: §7 · p. 10

A Description of Figure 1

• Darker cells indicate higher gradient similarity, while proximity on the figure’s left reflects closeness between clusters in collaborative space.The dataset is partitioned into eight clusters, and Euclidean distance evaluates cluster proximity.Source: A Description of Figure 1 · p. 16
• The case study compares sequence pairs with cosine similarities of 0.86 and -0.75.For sequences containing more than three movies of the same genre, the analysis cross-matches sequence histories.Source: A Description of Figure 1 · p. 16
• The 0.86 pair shared thriller interest and two identical interaction-history items, whereas the -0.75 pair showed no noticeable preference similarity.Source: A Description of Figure 1 · p. 16

• Experiments cover LastFM, MovieLens, and Steam, datasets derived from real-world recommendation scenarios.They include music interactions and social tags, movie ratings and metadata, and gaming-platform ownership, playtime, and reviews.Source: B Experimental Design and Evaluation · p. 16
• Comparisons include traditional sequential models and LLM-based recommenders, including GRU4Rec[28], SASRec[24], Llama2[21], TALLRec[5], and LlaRA[9].The listed traditional models use recurrent, convolutional, and Transformer architectures; the LLM-based methods include fine-tuned and API-based models.Source: B Experimental Design and Evaluation · p. 16
• The study fine-tunes Llama2-7B [21]; traditional baselines run on one Nvidia A40, while iLoRA runs on one Nvidia A100.Experiments use Python 3.8 and PytorchLightning 1.8.6.Source: B Experimental Design and Evaluation · p. 16
• The evaluation uses HitRatio@1 on candidate sets containing the correct next item and 20 randomly selected non-interacted items, plus ValidRatio for responses within the candidate set.LLM-based recommenders generate one candidate item; traditional models select the item with the highest probability.Source: B Experimental Design and Evaluation · p. 16

C Experimental Setup of RQ1

• Recommendation sequences are jointly trained as multiple tasks in a basic LoRA framework, using batches of 128 and 64-dimensional SASRec representations to divide tasks.Source: C Experimental Setup of RQ1 · p. 17
• For large-scale multi-task analysis, 40k Steam sequences are clustered into 8 sub-datasets, and pairwise sequence-gradient cosine similarity is measured at checkpoints across 1k training steps.LoRA gradients bound to the same modules, such as gateproj, are averaged.Source: C Experimental Setup of RQ1 · p. 17

D Related Work

• Related work describes Transformer-based language models and their scale-related capabilities, including common-sense reasoning and instruction following, as motivation for exploring LLMs in recommendation.Source: D Related Work · p. 17
• Mixture of Experts is presented as a way to increase model capacity while activating only parts of the model, keeping computation the same or close to a dense counterpart.Source: D Related Work · p. 17

• Conventional sequential baselines use Adam with learning rate 0.001, embedding dimension 64, batch size 256, L2 regularization tuned by grid search, and results averaged across five random seeds.Source: E Statistics · p. 17
• LLM-based methods use maximum learning rates of 2e−4 on LastFM and 1e−4 on MovieLens and Steam, with projector parameters following the iLoRA learning-rate curve.Training uses a learning-rate warm-up strategy.Source: E Statistics · p. 17
• Training uses cosine annealing to decrease the learning rate smoothly, plus half-precision computation for memory efficiency and faster training.Source: E Statistics · p. 17