Source-linked AI summary
HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation
Yu Cui, Feng Liu, Jiawei Chen, Canghong Jin, Xingyu Lou, Changwang Zhang, Jun Wang, Yuegang Sun, Can Wang
TL;DR
LLM-based recommenders can model item semantics but have difficulty capturing collaborative signals across items. HatLLM applies hierarchical attention masks across shallow, middle, and deep layers; experiments on three real-world datasets report a 9.13% average improvement over existing LLM-based methods.
Problem
Method
HatLLM uses intra-item attention in shallow layers, original attention in middle layers, and cross-item attention in deep layers.
Source: §1 · p. 2Results
Takeaways & Limitations
HatLLM jointly models token-level and item-level dependencies through its progressive layer-wise attention strategy.
Source: §1 · p. 2, §7 · p. 9Directly masking intra-item attention can fail to yield significant gains and may slightly degrade performance, as attention can concentrate on nearby items rather than longer-range dependencies.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
Recent years have witnessed a surge of research on leveraging large language models (LLMs) for sequential recommendation. LLMs have demonstrated remarkable potential in inferring users' nuanced preferences through fine-grained semantic reasoning. However, they also exhibit a notable limitation in effectively modeling collaborative signals, i.e., behavioral correlations inherent in users' historical interactions. Our empirical analysis further reveals that the attention mechanisms in LLMs tend to disproportionately focus on tokens within the same item, thereby impeding the capture of cross-item correlations. To address this limitation, we propose a novel hierarchical attention masking strategy for LLM-based recommendation, termed HatLLM. Specifically, in shallow layers, HatLLM masks attention between tokens from different items, facilitating intra-item semantic understanding; in contrast, in deep layers, HatLLM masks attention within items, thereby compelling the model to capture cross-item correlations. This progressive, layer-wise approach enables LLMs to jointly model both token-level and item-level dependencies. Extensive experiments on three real-world datasets demonstrate that HatLLM achieves significant performance gains (9.13% on average) over existing LLM-based methods.
1 Introduction
LLMs can reason over item semantics for sequential recommendation, but their attention gives weaker weight to cross-item correlations than to tokens within the same item. HatLLM applies different attention masks across layers, and experiments on three benchmark datasets report an average improvement of 9.13% over state-of-the-art LLM-based recommenders.
1 Introduction
- LLM-based recommenders show weaker aggregated attention between different items than within the same item, limiting their modeling of collaborative signals.Figure 2 compares total attention proportion and average attention weight for intra-item and inter-item attention.Source: §1 · p. 1, Figure 2 · p. 2
- Earlier methods inject item embeddings from traditional recommenders, but compressing history information and dilution by surrounding tokens limit their effectiveness.Source: §1 · p. 2
- HatLLM uses intra-item attention in shallow layers, original attention in middle layers, and cross-item attention in deep layers.The layers progress from learning item semantics to modeling cross-item correlations, while retaining token-level language modeling in the middle.Source: §1 · p. 2
- 9.13% average improvement over state-of-the-art LLM-based recommenders on three benchmark datasets is reported for HatLLM.The method requires only minimal modifications to attention masking and introduces no additional overhead.Source: §1 · p. 2
2 Preliminary
Sequential recommendation predicts a user's next item from their chronological interaction history. LLM-based approaches represent histories as prompts and use attention to aggregate contextual information across tokens.
2.1 Task Formulation
- Sequential recommendation predicts the next item a user is likely to interact with from their chronological interaction history.Source: §2.1 · p. 2
2.2 LLM-based recommendation
- LLM-based recommenders convert item descriptions and interaction histories into prompts, then predict items through language generation or item embeddings.This method adopts embedding-based prediction because language generation has significant computational overhead; baseline comparisons report each method's superior strategy.Source: §2.2 · p. 3
- Supervised fine-tuning aligns LLMs with recommendation by pairing prompts with target items and optimizing language-modeling or cross-entropy loss.Source: §2.2 · p. 3
2.3 Attention Mechanism in LLMs
- Each token forms a weighted aggregation over preceding tokens using query, key, and value projections, with a causal mask enforcing autoregressive attention.The attention weights use scaled query-key dot products; the mask prevents attention to later tokens.Source: §2.3 · p. 3
3 Analyses on Attention Weights
Attention analyses find that LLM-based recommenders emphasize intra-item over inter-item attention, constraining their capture of cross-item collaborative signals. Positional proximity and training-data co-occurrence imbalance are identified as contributing factors, while injected collaborative tokens can receive little attention.
3 Analyses on Attention Weights
- Aggregated attention between tokens from different items is substantially weaker than attention within the same item, despite cross-item pairs being more numerous.The average attention weight per cross-item token pair is over seven times that of intra-item pairs.Source: §3 · p. 3
- Tokens within the same item are closer in sequence position, which is associated with attention concentrating on intra-item tokens.Average intra-item distances are 5.6 and 4.8 on Games and Clothing, versus cross-item distances of 30.03 and 29.42.Source: §3 · p. 3
- Figure 3 depicts shallow intra-item attention, middle original attention, and deep cross-item attention as the proposed framework's three components.Source: Figure 3 · p. 4
- Training data contains more co-occurring intra-item than cross-item token pairs, and fine-tuning further reduces cross-item attention weights relative to pretrained models.Pair frequencies are 6.8 versus 2.3 on Games and 8.1 versus 2.5 on Clothing.Source: §3 · p. 4
- Injected collaborative tokens in LLaRA receive average attention scores of 1.85% on Games and 1.14% on Clothing, while its performance gains average 3.80%.The approach relies on auxiliary models to compress collaborative information into embeddings, whose influence can be diluted by other tokens.Source: §3 · p. 4
4 Methodology
HatLLM uses layer-specific attention masks to model item semantics and cross-item dependencies within one LLM. Its discussions distinguish this design from separate item/user LLMs, stacked traditional models, and parallel generative-recommendation architectures.
4.1 Hierarchical Attention Masking Strategy
- HatLLM combines shallow intra-item attention, original attention in middle layers, and deep cross-item attention to model semantic and item-level dependencies progressively.Shallow layers block attention between different items; middle layers retain attention over all tokens; deep layers mask attention between tokens within an item.Source: §4.1 · p. 4
- Masking all intra-item attention can yield little gain or slightly degrade performance, as attention may concentrate on nearby items and impede long-range dependencies.Adjacent items may have disparate semantics, so excessive aggregation from neighbors may also adversely affect performance.Source: §4.1 · p. 5
- Using only each item's last token for cross-item attention reduces aggregation tokens, balances item representation length, and treats token-to-token attention as information propagation between items.The last token has already aggregated item semantics and preceding-token information, making it a natural item representation.Source: §4.1 · p. 5
4.2 Discussions
- Unlike Lite-LLM4Rec, HatLLM jointly models intra-item and cross-item dependencies in one end-to-end fine-tuned LLM while preserving middle-layer attention over all semantic tokens.The comparison also emphasizes retention of fine-grained semantic preference modeling.Source: §4.2 · p. 5
- HatLLM differs from direct LLM–traditional-model stacking by jointly optimizing its components without embedding transformation and compression that may lose information.The paper reports that HatLLM significantly outperforms the stacking strategy in its ablation comparison.Source: §4.2 · p. 5
- HatLLM targets LLM-based recommendation and modifies attention within one LLM, whereas DiscRec targets generative recommendation and uses parallel branches.DiscRec's branches fuse basic generative recommendation models with traditional collaborative models.Source: §4.2 · p. 5
5 Experiments
The experiments evaluate HatLLM on three Amazon recommendation datasets against traditional, LLM-enhanced, and LLM-based recommenders, then examine its components and layer-count choices. HatLLM outperforms the strongest LLM-based baseline by 9.13% on average, and the ablations and sensitivity analysis examine how its hierarchical design contributes to performance.
5.1.1 Datasets.
- The study uses Amazon Video Games, Beauty, and Clothing, Shoes and Jewelry datasets, applying 5-core filtering and length-11 sliding windows before timestamp-ordered 8:1:1 splits.Baselines include traditional, LLM-enhanced, and LLM-based recommenders.Source: §5.1.1 · p. 6
5.1.3 Evaluation Metrics.
- Performance is measured with HR@5/10 and NDCG@5/10; experiments use LLaMA3-3B for LLM baselines and LoRA fine-tuning for HatLLM.The authors also test HatLLM on LLMs at other parameter scales.Source: §5.1.3 · p. 6, §5.1.4 · p. 6
5.2 Performance Comparison (RQ1)
- HatLLM beats LLM-enhanced recommenders across metrics and datasets, outperforms Lite-LLM4Rec, and gains further over LLaRA, whose collaborative tokens are diluted.The authors attribute LLaRA’s limited gains to information loss from low-dimensional collaborative embeddings and dilution of its enhanced collaborative tokens.Source: §5.2 · pp. 6–7
- 9.13% average improvement over the strongest LLM-based recommender, with HatLLM outperforming recommenders across all metrics on three real-world datasets.Table 3 presents the three-dataset performance comparison; its caption defines “Impr.” as improvement over the best baseline.Source: §5.2 · p. 6, Table 3 · p. 7
- HatLLM outperforms representative LLM-based baselines on Llama3-1B, 3B, and 8B.Table 4 reports performance on LLMs at other parameter scales.Source: Table 4 · p. 7, §5.2 · p. 7
5.3 Ablation Study (RQ2)
- Removing IN, OR, or CR, or replacing CR with SASRec or CR-pre, significantly degrades performance; direct stacking also faces embedding mismatch and decoupled optimization.The authors associate CR-pre’s decline with weak long-range modeling and noisy aggregation of neighboring items.Source: §5.3 · pp. 7–8
- The proposed IN-OR-CR order outperforms the tested alternatives, progressing from item semantics through contextual semantics to collaborative signals.The authors describe this progression as moving from fine-grained token-level to coarse-grained item-level modeling.Source: §5.3 · p. 8
5.4 Hyperparameter Sensitivity (RQ3)
- Performance first rises then falls as shallow-layer count increases, with better results at 4 or 8 layers; typically, 1–3 deep layers achieve good results.Figure 5 analyzes performance across shallow intra-item and deep cross-item layer counts.Source: §5.4 · p. 8
6 Related Work
Sequential recommendation predicts users’ next items from historical behavior, and LLMs are used both directly and to enhance traditional recommenders. The related work describes these approaches and notes challenges in modeling collaborative signals and transferring LLM knowledge.
6.1 Sequential Recommendation
- Sequential recommendation predicts users’ next interests from historical interactions, with recurrent networks, CNNs, and Transformers among the sequence-modeling approaches.SASRec uses attention to weight interaction items, while DROS applies distributionally robust optimization.Source: §6.1 · p. 8
6.2 LLMs for Recommendation
- LLM-based recommenders use pretrained LLMs directly, with textual and embedding-based prediction approaches; token-level behavior modeling can weaken capture of collaborative signals.Textual methods can face inference-efficiency and hallucination challenges, while embedding-based methods predict item embeddings through a projection head.Source: §6.2 · pp. 8–9
- LLM-enhanced recommenders use LLM knowledge and reasoning to augment traditional models, but capability and mechanism gaps can hinder knowledge transfer.Examples include KAR’s semantic vectors, LLM-CF’s generated Chain-of-Thought, and DLLM2Rec’s knowledge distillation.Source: §6.2 · p. 9
7 Conclusion
The conclusion identifies token-level processing as a constraint on LLM-based recommenders’ ability to capture cross-item collaborative signals, and presents HatLLM’s hierarchical masks as its response. It reports experiments on three real-world datasets and proposes exploring attention mechanisms that capture semantic and collaborative signals in common layers.
7 Conclusion
- HatLLM assigns intra-item, original, and cross-item attention to shallow, middle, and deep layers to jointly model token- and item-level dependencies.The conclusion reports experiments on three real-world datasets and identifies common-layer attention for semantic and collaborative signals as future work.Source: §7 · p. 9
A.1 Baselines
The baselines span conventional sequential recommendation, LLM-augmented methods, and frameworks that align or transfer knowledge between LLMs and recommenders.
A.1 Baselines
- SASRec uses transformer self-attention for collaborative signals, while DROS applies distributionally robust optimization to address train-test shifts from dynamic environments.SASRec uses cross-entropy loss; DROS is model-agnostic across backbone recommenders.Source: §A.1 · p. 11
- KAR adds LLM-derived preferences and item facts through factorization prompting and hybrid-expert adaptation, while LLM-CF distills LLM knowledge and reasoning through in-context learning.LLM-CF uses instruction tuning while preserving LLM general abilities.Source: §A.1 · p. 11
- DLLM2Rec transfers LLM capabilities through ranking and collaborative-embedding distillation, while E4SRec processes item IDs and constrains generated outputs to valid candidates.Lite-LLM4Rec replaces beam search with an item projection head and uses a hierarchical LLM structure to reduce computational overhead.Source: §A.1 · p. 11
- SLMRec distills LLM knowledge into small language models, BIGRec grounds generated item tokens to actual items, and LLaRA aligns ID embeddings with textual features.SLMRec is described as compatible with quantization and pruning; LLaRA uses hybrid prompting to inject collaborative signals.Source: §A.1 · p. 11