Source-linked AI summary
LSNR: Learned Sparse News Recommendations with Attentive User Modeling
David Vos, Andrew Yates, Maarten de Rijke
TL;DR
STAR’s concatenated news histories flatten users’ distinct interests and can exceed a language model’s context window. LSNR independently encodes articles and aggregates their sparse representations with attention and a user-level dense-projected expansion. On MIND and EBNeRD, it outperforms STAR, competes with dense recommenders in standard settings, and does better for short histories.
Problem
STAR represents a user by concatenating interaction history, which flattens distinct reading interests and can force truncation when long news articles exceed the language model’s context window.
Source: §1 · p. 1Method
Results
On MIND and EBNeRD, LSNR significantly outperforms STAR, remains competitive with dense recommenders in standard settings, and significantly outperforms them for short histories.
Source: §6 · p. 9Takeaways & Limitations
LSNR serves from an index an order of magnitude smaller than an uncompressed dense baseline, and its gains over STAR persist when short histories avoid truncation.
Source: §6 · p. 9The evidence comes from the small versions of two benchmarks in two languages; generalization to larger corpora and other domains remains future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Current learned sparse approaches for recommender systems, such as STAR, represent a user by concatenating their interaction history into a single query string. For news, this creates two bottlenecks: concatenation collapses a user's distinct reading interests into one flat query, and, because news articles are long-form text, concatenating several news articles quickly overruns the context window of the language model used as backbone and forces quality-degrading truncation. To address both bottlenecks, we introduce learned sparse news recommendations (LSNR). LSNR encodes each article independently and constructs a sparse user vector by combining learned, attention-based weighting of the per-article sparse representations with a dense-projected expansion at the user level. Experiments on the MIND and EBNeRD datasets show that LSNR significantly outperforms STAR, with these gains holding even in short-history settings where truncation is not a factor. Compared to state-of-the-art dense recommenders, LSNR is competitive in standard settings and it outperforms them when the user history is short. An ablation of the user encoder confirms that learned attention-based aggregation outperforms simpler pooling alternatives. Beyond effectiveness, an additional advantage of LSNR is that it serves from an index an order of magnitude smaller than an uncompressed dense baseline.
1 Introduction
LSNR addresses STAR’s flattened news histories with independent article encoding and attention-based user aggregation. On MIND and EBNeRD, it outperforms STAR, remains competitive with dense recommenders, and uses a substantially smaller index.
Motivation
- STAR’s concatenated history both flattens distinct reading interests and can exceed the language model’s context window, forcing quality-degrading truncation.LSNR instead models interactions at the individual-article level.Source: §1 · p. 1
Our approach
- LSNR encodes articles independently, then uses learned attention-based aggregation and dense-projected expansion to construct a sparse user vector.Article representations can be precomputed offline, leaving user-side aggregation at inference time.Source: §1 · pp. 1–2
Our findings
- 11%: LSNR improves NDCG@10 over STAR on MIND, and its advantage persists in short-history settings without truncation.In full-history settings, LSNR is within roughly half a percent of the strongest dense recommenders on every MIND metric and trails them by 2 to 3% on EBNeRD, while surpassing every dense baseline for short histories.Source: §1 · p. 2
Efficiency
- LSNR serves from an index an order of magnitude smaller than an uncompressed dense baseline.Its user-vector dimensions map to vocabulary terms, allowing direct inspection.Source: §1 · p. 2
Our contributions
- The user-encoder ablation finds learned attention-based aggregation outperforms simpler pooling strategies, with both pathways contributing to performance.Source: §1 · p. 2
2 Background
The background covers news recommendation and learned sparse representations. It contrasts content-based news recommenders with learned sparse retrieval and explains why history concatenation is ill-suited to news.
2 Background
- The background identifies news recommendation and learned sparse representations as its two relevant areas.Source: §2 · p. 2
2.1 News recommendation
- News recommendation commonly uses dense dual encoders; rapid article turnover makes ID-based collaborative filtering infeasible, while large language models’ inference latency makes them infeasible for the retrieval stage discussed here.Source: §2.1 · p. 2
- Dense news recommenders can aggregate clicked articles with attention, while LSNR constructs the user representation through attention in sparse lexical space.Source: §2.1 · p. 2
2.2 Learned sparse representations
- Learned sparse retrieval expands documents with latent terms predicted by language models, but STAR’s concatenated news history collapses interests and can force context-window truncation.Sparse dimensions correspond to vocabulary terms; their interpretability is qualified by analyses showing ranking signals may pass through loosely related or irrelevant tokens.Source: §2.2 · p. 2
3 Methodology
LSNR independently encodes news articles, then builds a sparse user representation by combining attention-weighted article vectors with a dense-projected user-level expansion. Its sparse representations support indexed candidate scoring and regularization of their sparsity.
3.1 News encoder
- Each article receives a SPLADE-based sparse representation and an auxiliary dense representation, while candidate articles use only the sparse vector.Sparse article representations max-pool term importance across tokens; the dense history representation is mean-pooled.Source: §3 · p. 2, §3.1 · p. 3
3 Methodology
- LSNR builds each user vector from two complementary pathways: weighted article-level sparse representations and a SPLADE projection of the aggregated dense history.The pathways retain direct lexical signals while allowing user-level semantic expansion beyond any single article’s representation.Source: §3.2 · pp. 3–4
3.2 User encoder
- The user encoder contextualizes dense history vectors, learns additive-attention weights, and uses them to aggregate the sparse and dense representations.It omits positional encodings, which the authors report do not improve performance.Source: §3.2 · p. 3
3.3 Scoring
- Candidate articles are scored by the sparse inner product with the user vector, and their user-independent representations can be precomputed for indexing.The sparse inner product is computed over an inverted index.Source: §3.3 · p. 4
3.4 Regularization
- LFLOPS regularization controls sparsity in both candidate-news and aggregated-user representations; average user vectors occupy less than 3% of the vocabulary space.Before retrieval-time pruning, average user vectors contain 722 non-zero dimensions on MIND and 563 on EBNeRD.Source: §3.4 · p. 4
4 Experimental setup
The experiments evaluate LSNR on English MIND and Danish EBNeRD, using sparse and dense baselines and protocols that test full-history performance, short-history robustness, efficiency, and user-encoder design.
4.1 Datasets
- MIND contains shorter title-and-abstract articles and moderate histories, whereas EBNeRD includes full article bodies and longer histories.The evaluation uses the small dataset versions and the official validation split for testing because official test labels are unavailable.Source: §4.1 · p. 4
4.2 Baseline methods
- Sparse comparisons include BM25 and STAR, while dense comparisons include NRMS, NAML, and LSTUR; NAML and LSTUR also use categories and user IDs.STAR is adapted by concatenating recent interactions until its 512-token context window is filled.Source: §4.2 · p. 4
4.3 Optimization details
- Models use English or Danish pretrained backbones, freeze the first nine of twelve backbone layers, and select learning rates with validation NDCG@10.STAR on MIND starts from a retrieval-pretrained checkpoint while LSNR starts from vanilla BERT.Source: §4.3 · p. 4
4.4 Evaluation protocols
- The evaluation reports NDCG@10, NDCG@5, and MRR, with paired two-tailed t-tests across per-user metric distributions.Accuracy metrics rank clicked and co-displayed non-clicked articles within impression logs; full-catalog retrieval is used only for efficiency measurements.Source: §4.4 · p. 5
- Short-history tests truncate test histories to the N most recent interactions for N=1–5, including N=1 where STAR and LSNR receive the same input.Efficiency tests measure index footprint, latency, and accuracy-versus-sparsity trade-offs; encoder ablations compare pathways and aggregation alternatives.Source: §4.4 · pp. 5–6
5 Results
LSNR outperforms sparse baselines on the full benchmarks and retains an advantage when histories contain only one interaction. These results cover full-history benchmarking and short-history behavior.
5.1 Benchmarking LSNR against sparse baselines
- LSNR significantly outperforms BM25 and STAR on both full benchmarks, gaining 11% over STAR in MIND NDCG@10 and over 10% on EBNeRD.Table 4 reports performance across the full MIND and EBNeRD benchmarks.Source: Table 4 · p. 6, §5.1 · p. 6
- On long-article EBNeRD histories, STAR barely edges BM25 in NDCG@10 (0.4704 vs. 0.4639) and trails it in MRR (0.3459 vs. 0.3503).The authors connect this weak performance to concatenation’s context-window bottleneck.Source: §5.1 · p. 6
5.2 LSNR in short-history settings
- With a single interaction and no STAR truncation, LSNR significantly beats STAR by 3.6% in MIND NDCG@10 and over 7% on EBNeRD.The comparison uses the same input for both models.Source: §5.2 · p. 6, Table 5 · p. 6
- As history length rises from 1 to 5, LSNR improves rapidly while STAR shows diminishing returns, illustrating different aggregation behavior.Figure 2 plots NDCG@10 for LSNR, STAR, and dense baselines on MIND and EBNeRD.Source: §5.2 · p. 6, Figure 2 · p. 6
5.3 Comparing LSNR with dense baselines
LSNR is competitive with dense recommenders in full-history settings and performs especially well when histories are short. The comparison spans MIND and EBNeRD.
5.3 Comparing LSNR with dense baselines
- With full histories, LSNR is within half a percent of NRMS and NAML on every MIND metric, but trails them by 2 to 3% on EBNeRD NDCG@10.LSNR remains well ahead of LSTUR on both datasets.Source: §5.3 · p. 7
- At N=1, LSNR significantly beats all dense baselines: by 2.5% over NRMS on MIND and 2.7% over NAML on EBNeRD.All methods converge around N=5; the short-history advantage is reported as statistically significant.Source: §5.3 · p. 7
- Overall, LSNR is competitive with dense models in standard settings and significantly superior when user histories are short.The authors describe this pattern as establishing LSNR as a reliable alternative to dense models.Source: §5.3 · p. 7
5.4 Efficiency analysis
The efficiency analysis examines user-vector pruning, index footprint, and retrieval cost. LSNR’s index is much smaller than the uncompressed dense baseline, while pruning outcomes differ by dataset.
5.4.1 Pruning the user vector.
- On MIND, tighter pruning lowers LSNR’s NDCG@10, yet even aggressive pruning remains substantially better than STAR’s best result.STAR’s accuracy is flat across thresholds; its user vectors average 15.6 NNZ.Source: §5.4.1 · p. 7
- On EBNeRD, tighter pruning improves LSNR accuracy; the authors suggest pruning may denoise broader-vocabulary vectors from longer histories and articles.STAR averages 30.0 NNZ there, but its accuracy does not benefit from pruning.Source: §5.4.1 · p. 7
5.4.2 Index footprint.
- LSNR’s index is an order of magnitude smaller than uncompressed NRMS on both datasets: 11.33 vs. 208.06 MB on MIND and 2.41 vs. 31.84 MB on EBNeRD.Relative to STAR, LSNR is comparable on MIND and roughly 14× smaller on EBNeRD.Source: §5.4.2 · p. 7
5.4.3 Retrieval cost.
- LSNR’s inverted-index retrieval is slower in absolute terms than dense HNSW, although its end-to-end latency remains within a few milliseconds per query at this corpus scale.The measurements use full-catalog retrieval; Figure 3 instead plots impression-based NDCG@10 against average user-vector NNZ as pruning varies.Source: §5.4 · p. 7, Figure 3 · p. 7, §5.4.3 · p. 8
5.5 Pathway ablation
The hybrid user encoder combines a lexical pathway that aggregates per-article sparse vectors with a semantic pathway that projects an aggregated dense representation into sparse space. It outperforms either pathway alone, with the size of its advantage varying by dataset and history length.
5.5 Pathway ablation
- The two pathways perform almost identically alone, differing by at most 0.014 NDCG@10, but their combination yields substantial gains.The results indicate that the pathways encode complementary rather than redundant signals.Source: §5.5 · p. 8
- The hybrid’s advantage is largest with full histories; at N=1 it narrows on MIND but remains substantial on EBNeRD.At N=1, gains over the lexical-only and semantic-only variants are 1.7% and 1.3% on MIND, and 6.5% and 3.6% on EBNeRD.Source: §5.5 · p. 8
- The full hybrid beats both single-pathway variants in all four dataset and history-length settings.On MIND full-history, it exceeds the lexical-only variant by 11.9% and the semantic-only variant by 11.3%; on EBNeRD full-history, the gains are 9.8% and 6.8%, respectively.Source: Table 8 · p. 8, §5.5 · p. 8
- EBNeRD’s longer articles provide more text per interaction, so even one article supports per-article aggregation and dense-projected expansion.The hybrid combines direct lexical signals from individual articles with user-level semantic expansion.Source: §5.5 · p. 8
5.6 User aggregation methods
The aggregation ablation compares additive attention with positional attention and three pooling alternatives while holding the semantic pathway constant. Additive attention is strongest across both benchmarks and history settings, while the parameter-free alternatives vary by dataset.
5.6 User aggregation methods
- Adding positional encodings degrades performance on both datasets, consistent with prior findings that strict temporal order is a weak news-recommendation signal [44].The authors report that this supports omitting positional encodings.Source: §5.6 · p. 8
- No parameter-free pooling strategy is robust across both datasets: max pooling fares best on MIND but worst on EBNeRD, while mean pooling and temporal decay rank lower on MIND.Mean pooling and temporal decay are slightly stronger on EBNeRD than on MIND.Source: §5.6 · p. 8
- Additive attention is the strongest aggregation method across both benchmarks and all reported settings.The ablation holds the semantic pathway constant and reports NDCG@10 for full-history and N=1 settings.Source: §5.6 · p. 8, Table 9 · p. 9
6 Conclusion
LSNR adapts learned sparse retrieval to news by encoding articles independently and aggregating them with learned attention. Across MIND and EBNeRD, it outperforms sparse baselines, competes with dense recommenders in standard settings, and is strongest against dense models for short histories; the authors also identify limits in benchmark scope and full-history performance.
6 Conclusion
- LSNR encodes each article independently and aggregates the per-article representations with a learned attention-based user encoder, addressing concatenation’s context-window bottleneck and interest collapse.The conclusion describes this as adapting learned sparse retrieval to long-form news text and user histories.Source: §6 · p. 9
- On MIND and EBNeRD, LSNR outperforms BM25 and STAR, retains its advantage at N=1, and competes with dense recommenders in standard settings while outperforming them for short histories.Its index is an order of magnitude smaller than the uncompressed dense baseline on both datasets.Source: §6 · p. 9
- The short-history advantage does not extend uniformly to full histories: LSNR matches the strongest dense recommenders on MIND but trails them by 2 to 3% on EBNeRD.The authors also report that a well-tuned dense HNSW index matches or exceeds LSNR’s retrieval accuracy at lower query latency; LSNR’s efficiency benefit is its compact index.Source: §6 · p. 9
- Evidence is limited to small versions of two benchmarks in two languages, leaving generalization to larger corpora and other domains for future work.The authors also note that on MIND the sparse-model comparison is not fully controlled because the models use different checkpoints.Source: §6 · p. 9
- Article categories and user demographics are proposed as possible additional signals, while direct evaluation of term-level user-interest controls is left to future work.The sparse vector’s vocabulary-term dimensions could in principle support inspecting or editing user interests.Source: §6 · p. 9
Resources
The authors share LSNR and baseline models, data-processing pipelines, and scripts and configurations intended to support reproduction of the reported experiments.
Resources
- The shared resources include LSNR and the STAR and dense-recommender baselines NRMS, NAML, and LSTUR.The resources are provided under an MIT license.Source: Resources · p. 9
- The release includes preprocessing and pre-tokenization pipelines for MIND and EBNeRD.Source: Resources · p. 9
- Training and evaluation scripts report performance and sparsity metrics and include an efficiency benchmark for index size and retrieval latency.A central configuration file, fixed seeds, chronological splits, and reproduction documentation are also shared.Source: Resources · p. 9