Source-linked AI summary
Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders
Keito Kozaki, Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama
TL;DR
How last-item reliance is structurally expressed in causal self-attention recommenders remains unclear. The paper combines prediction-time diagnostics, full-block norm analysis, and residual scaling, finding that residual dominance links same-position information preservation to last-item reliance.
Problem
How information from different sequence positions is expressed in the final prediction representation remains unclear beyond attention weights alone.
Method
The study combines prediction-time positional diagnostics, full attention-block norm analysis, and inference-time residual scaling to probe information utilization.
Results
Residual addition sharply shifts representations toward same-position information, while residual scaling systematically changes contextual mixing, last-item reliance, and recovery of some final-position misses.
Takeaways & Limitations
Residual dominance provides a structural account of highly localized last-item reliance in causal self-attention recommenders at inference time.
Takeaways & Limitations
Residual scaling does not determine whether contextual information provides meaningful predictive signals or merely reflects degradation of the final representation.
Abstract
from arXiv · showhide
Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear. We combine prediction-time diagnostics with norm-based analysis of the full attention block. First, we show that SASRec-style models exhibit highly localized last-item reliance. We then find that, although self-attention aggregates contextual information, residual addition sharply shifts the full-block representation toward same-position contributions, which we term residual dominance. To probe this interpretation, we use inference-time residual scaling as a controlled diagnostic intervention. Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance, while reducing residual strength recovers a subset of final-position misses for which representations at non-final positions already rank the ground-truth item correctly. Our results provide a structural account linking extreme last-item reliance to residual dominance at inference time. The code is publicly available.
1 Introduction
The section establishes that causal self-attention recommenders, exemplified by SASRec, often rely disproportionately on the most recent interaction. It attributes this behavior to residual dominance, where residual connections preserve same-position information and reduce contextual mixing in the prediction representation.
- Last-item reliance: Causal self-attention recommenders frequently anchor predictions to the most recent interaction, with perturbing earlier positions causing minor changes and perturbing the final position causing sharp degradation.This localized behavior is documented through inference-time sequence perturbations and complementary ranking diagnostics.
- Last-item reliance: Predictive influence collapses onto the final item rather than gradually decaying across preceding positions, leaving nearby earlier positions with little contribution.This discontinuous positional pattern is not fully explained by recency bias or attention patterns alone.
- Residual dominance: Residual addition sharply reduces contextual mixing while preserving same-position information, providing a structural pathway for the final item to dominate the prediction representation.The analysis examines the complete attention block, including residual connections and normalization, using norm-based decomposition.
- Diagnostic intervention: Inference-time residual scaling keeps trained parameters fixed while varying residual contribution to probe systematic links among representation structure, contextual mixing, and last-item reliance.The intervention is presented as a mechanistic sensitivity probe rather than strict causal identification or a new training method.
2 Preliminaries and Overall Setup
The paper formulates sequential recommendation as next-item prediction from chronological user–item sequences and applies a common experimental setup across datasets, baselines, and diagnostics. Evaluation uses temporally separated, full-catalog ranking with standard top-K metrics.
- Task formulation: The task estimates a probability distribution over the next item v_t+1 given a user’s chronological interaction sequence S_u, with v_t as the most recent item.The final sequence position is denoted L, while preceding positions are L−1, L−2, and so forth for position-wise diagnostics.
- Datasets: Experiments analyze nine public datasets spanning varied domains, sparsity levels, sequence lengths, and temporal characteristics.The benchmark includes both short session-based interactions and long-term user histories, with implicit feedback throughout.
- Baselines: The study compares GRU4Rec, SASRec, and BERT4Rec as representative recurrent, causal self-attention, and bidirectional self-attention baselines.These models generate predictions from a single sequence-level representation, enabling direct comparison of prediction-time behavior across inductive biases.
- Evaluation protocol: All models are evaluated with a Global Time Split protocol using NDCG@K and HR@K at K=10 on the test set.The protocol strictly separates training and testing interactions in time to avoid temporal information leakage associated with leave-one-out evaluation.
- Evaluation protocol: Results use full-catalog ranking against the entire item set, exclude previously interacted items from top-K lists, and bypass filter-seen for Diginetica and Zvuk.Full-catalog ranking avoids sampling-based metrics that can produce inconsistent or misleading model comparisons.
3 Empirical Characterization of Last-Item Reliance
SASRec’s inference-time behavior is characterized using position shuffling and recency-localized hit-rate diagnostics. Across nine datasets, both diagnostics provide consistent evidence of highly localized reliance on the last interacted item, while remaining agnostic about its mechanism.
- Diagnostic framework: The section uses inference-time position shuffling and a recency-localized hit-rate metric to characterize SASRec’s positional reliance.The shuffling variants preserve only the final item or permute the entire sequence, isolating the final position’s influence.
- Position sensitivity: Across nine datasets, SASRec exhibits three behavioral regimes associated with data density and sequentiality.Sparse datasets show limited sensitivity to partial shuffling but sharp degradation under full shuffling, whereas weakly sequential datasets show negligible effects.
- Ranking diagnostics: HRLI@K measures how often the last input item appears in the Top-K recommendation list, while HRL2I@K measures the analogous frequency for the second-to-last item.These metrics quantify positional bias in predicted rankings and are computed without filter-seen to avoid excluding the measured item.
- Ranking diagnostics: Across nine datasets, causal self-attention models generally produce substantially higher HRLI@1 than HRL2I@1, unlike BERT4Rec and GRU4Rec, which show a much smaller gap.The analysis focuses on K=1, where HRLI@1 captures the sharpest signal of last-item localization.
- Interpretation: Together, the diagnostics consistently indicate highly localized last-item behavior in SASRec but do not identify the mechanism responsible.The subsequent analysis therefore examines how information from different positions is structurally composed within the attention block.
4 Structural Analysis of Last-Item Reliance
SASRec’s last-item reliance is structurally linked to residual dominance: attention aggregates contextual information, but residual addition shifts the full-block representation toward same-position contributions. Because prediction uses only the final-position representation, this preserved same-position information is anchored to the most recent item.
- Motivation: Attention weights alone cannot explain last-item reliance because they vary smoothly, whereas the observed reliance is highly discontinuous and reflects information usage in the final representation.The analysis therefore distinguishes where attention is placed from which information survives for prediction.
- Method: A norm-based decomposition quantifies each position’s contribution in the full attention-block output, accounting for transformed-value norms, residual addition, and layer normalization.The resulting mixing ratio measures contextual aggregation versus self-information retention.
- Residual dominance: Across datasets, attention outputs show substantial contextual aggregation, but residual addition sharply shifts the block output toward same-position contributions.Layer normalization changes the mixing ratio only marginally relative to residual addition, identifying the residual connection as the primary structural factor.
- Residual dominance: The complete block exhibits markedly stronger diagonal contribution patterns than attention-only outputs, indicating stronger preservation of same-position information in Toys and ML-1M.Figure 3 compares Attn with AttnResLN for both sparse and dense datasets.
- Position and layer analysis: Position-wise mixing varies modestly within layers, with maximum absolute differences of 0.0032 and 0.0047 on Toys and 0.0186 and 0.0054 on ML-1M across layers 1 and 2.The final-position ratio is not consistently lower and is slightly higher than preceding positions on ML-1M, supporting broad self-preservation rather than final-position-specific dominance.
- Structural account: Since SASRec forms next-item predictions exclusively from the final-position representation, its preserved same-position component is anchored to the most recently interacted item, explaining the discontinuous last-position privilege.This links broad self-preservation in the attention block to last-item reliance at the prediction interface.
5 Inference-Time Probing of Residual Contribution
Inference-time residual scaling systematically changes contextual mixing, last-item reliance, and ranking outcomes. Lower residual strength reduces reliance on the last item and recovers some final-position misses when earlier-position representations already rank the target correctly.
- Inference-time residual scaling is used as a controlled sensitivity analysis, not strict causal identification, to probe contextual mixing, last-item reliance, and ranking outcomes.The intervention varies residual preservation at every sequence position and Transformer layer.
- Decreasing α consistently increases the AttnResLN mixing ratio and decreases HRLI@1, linking weaker residual strength to greater contextual mixing and lower last-item reliance.This monotonic relationship supports residual dominance as a structural account of last-item reliance.
- As α decreases, HR@10 and NDCG@10 consistently degrade while last-item reliance declines, revealing a trade-off between final-position self-information and contextual information.Standard ranking metrics alone cannot determine whether contextual signals are meaningfully used or whether the final representation simply degrades.
- Approximately 24–46% of the standard HR@10 value corresponds to the largest single-position joint rate of correct non-final hits among final-position misses.Thus, potentially useful predictions occur at preceding positions even when final-position inference fails.
- Peak recovery of final-position misses occurs around α≈0.1–0.3 when sequences already contain a correct non-final hit under standard inference.Reducing residual preservation therefore recovers a subset of misses consistent with increased contribution from non-final predictive information.
- Across the section, residual strength systematically controls prediction behavior, correct non-final predictions appear when final-position inference misses, and reducing residual strength recovers some such cases.Validation-selected scaling reproduces the observed accuracy–recovery trade-off, supporting stable controllability without test-data access.
6 Related Work
Sequential recommendation progressed from RNN-, CNN-, and Markov-based models to Transformer architectures with causal self-attention, while related work studied dependency emphasis and last-item reliance. This work addresses the underexplored structural expression of that reliance in SASRec prediction representations and the full attention block with residual connections.
- Architectural background: Sequential recommenders evolved from RNN-, CNN-, and Markov chain–based models toward Transformer architectures using self-attention, with causal self-attention models such as SASRec widely adopted.Recent graph-based, generative, and LLM-based methods have expanded the field, but causal self-attention remains a core architectural choice.
- Attention dependencies: Prior studies examined which dependencies self-attention should emphasize, often highlighting local and short-term dependencies because unconstrained global attention can be noisy or suboptimal under sparse interactions.These studies primarily analyze or modify attention weights and attention modules.
- Last-item reliance: Recency-bias research introduced mitigation strategies and metrics such as HRLI, while showing that causal self-attention models can exhibit extreme last-item reliance.Later work found this behavior persists across architectures, embedding sizes, and loss functions.
- Research gap: Unlike prior work characterizing last-item reliance as an empirical or task-level property, this work investigates how it is structurally expressed in SASRec’s prediction representation.The analysis focuses on the full attention block, including residual connections, rather than attention mechanisms in isolation.
7 Conclusion
Causal self-attention recommenders exhibit highly localized last-item reliance at inference time. Norm-based analysis identifies residual dominance, together with the final-position-only prediction interface, as a structural account of this behavior, while residual scaling reveals systematic links among residual strength, contextual mixing, and last-item reliance.
- Causal self-attention recommenders often exhibit highly localized last-item reliance at inference time.
- Norm-based analysis identifies residual dominance as a broadly self-preserving tendency in the full attention block.
- Residual dominance and the final-position-only prediction interface provide a structural account of last-item reliance.
- Inference-time residual scaling reveals systematic relationships among residual strength, contextual mixing, and last-item reliance.