Source-linked AI summary
EB-NeRD: A Large-Scale Dataset for News Recommendation
Johannes Kruse, Kasper Lindskow, Saikishore Kalloori, Marco Polignano, Claudio Pomo, Abhishek Srivastava, Anshuk Uppal, Michael Riis Andersen, Jes Frellsen
TL;DR
News recommendation faces technical and normative challenges, including short-lived content, implicit feedback, and aligning recommendations with editorial and social values. The paper introduces EB-NeRD and a beyond-accuracy evaluation framework; its analysis finds that recommendation methods can produce substantially different news flows.
Problem
News recommendation faces technical challenges from rapidly expiring articles, implicit feedback, and the need to use article text, as well as normative challenges of aligning outputs with editorial and social values.
Source: §1 · p. 1Method
The paper presents EB-NeRD, a dataset built from Ekstra Bladet user behavior logs, and a preliminary framework for beyond-accuracy evaluation.
Source: §1 · p. 2Results
Recommendation methods produced different news flows: NRMS recommended 10.2% news-category articles, while CatBoost:D recommended 45.7%.
Source: §3.3.2 · p. 6Takeaways & Limitations
The analysis underscores the importance of understanding news content and modeling user interests for effective news recommendation.
Source: §4 · p. 6The static candidate list cannot capture dynamic effects such as diversity of exposure or fragmentation that require multiple interactions over time.
Source: §3.3.3 · p. 6
Abstract
from arXiv · showhide
Personalized content recommendations have been pivotal to the content experience in digital media from video streaming to social networks. However, several domain specific challenges have held back adoption of recommender systems in news publishing. To address these challenges, we introduce the Ekstra Bladet News Recommendation Dataset (EB-NeRD). The dataset encompasses data from over a million unique users and more than 37 million impression logs from Ekstra Bladet. It also includes a collection of over 125,000 Danish news articles, complete with titles, abstracts, bodies, and metadata, such as categories. EB-NeRD served as the benchmark dataset for the RecSys '24 Challenge, where it was demonstrated how the dataset can be used to address both technical and normative challenges in designing effective and responsible recommender systems for news publishing. The dataset is available at: https://recsys.eb.dk.
1 INTRODUCTION
News recommendation faces technical barriers from rapidly expiring articles, implicit-only feedback, and the need to use article text, alongside normative challenges of aligning recommendations with editorial and social values. EB-NeRD and a preliminary beyond-accuracy evaluation framework are presented to support research on both challenge types.
1 INTRODUCTION
- News recommendation must address rapidly expiring articles, implicit-only user feedback, and the need to use article text.These conditions make cold-start and changing user interests central technical challenges.Source: §1 · p. 1
- It remains unclear whether results from news aggregators generalize to classical digital news publishers with markedly different consumption and content profiles.The paper notes that many datasets are not public and are tailored to specific use cases.Source: §1 · pp. 1–2
- At scale, news recommenders perform an editorial function, creating a need to align their outputs with editorial and social values.Research on news-relevant beyond-accuracy metrics and integrating them into recommender objectives is still nascent.Source: §1 · p. 2
- The paper presents EB-NeRD, detailed dataset statistics, and a preliminary framework for beyond-accuracy evaluation.These contributions are intended to encourage research on both technical and normative challenges.Source: §1 · p. 2
2 RELATED WORK
The related work describes news recommendation’s text and changing-interest challenges, reviews the limited public dataset landscape, and introduces EB-NeRD as a publisher-based Danish resource. It details the dataset’s construction and its scale and content distributions.
2.1 News Recommendations
- News recommendation research addresses article-text representation and dynamic user interests, while methods have moved from content-based approaches toward deep learning that combines multiple traditions.Collaborative filtering is less common because news articles can lose relevance before enough behavioral data accumulates.Source: §2.1 · p. 2
2.2 Existing Datasets
- Public news recommendation datasets are few, and EB-NeRD differs from MIND by directly providing article bodies and by recording richer activity from publisher pages.MIND’s public version lacks article bodies, and it tracks front-page events only; Adressa and EB-NeRD include front-page and article-page activity.Source: §2.2 · pp. 2–3
2.3 Dataset Description
- EB-NeRD is a large-scale Danish dataset created by Ekstra Bladet to support news recommendation research and benchmarking.Source: §2.3 · p. 3
2.3.1 Dataset Construction.
- The dataset’s impression logs were collected over six weeks, with users filtered by click counts and assigned anonymized IDs using one-time salt hashing.The click-count bounds were 5–1,000 clicks during the specified three-week period.Source: §2.3.1 · p. 3
- Impression records include viewed and clicked articles, page context, time, device, subscription status, and available user demographics.The dataset also provides Danish article titles, abstracts, bodies, and metadata, enriched with entities, topics, and sentiment labels.Source: §2.3.1 · p. 3
- Training, validation, and test splits include impression logs, prior click histories, and available articles; a hidden set of 200,000 synthesized impressions is reserved for beyond-accuracy evaluation.Those impressions draw from 250 newly published articles.Source: §2.3.1 · p. 3
2.3.2 Dataset Statistics.
- EB-NeRD contains 1,103,602 unique users, 125,541 distinct news articles, and 37,966,985 impression logs; entertainment and news account for 45% of the catalog, while adding crime and sport yields 78%.In-view article counts are heavily skewed toward a small number of viewed articles per impression.Source: §2.3.2 · p. 3
3 BENCHMARKING AND ANALYSIS
This section examines EB-NeRD in the RecSys ’24 Challenge through ranking and beyond-accuracy evaluation. It describes the evaluated methods, how recommendations differ in news flow, and limitations of the preliminary analysis.
3.1 Methods
- The evaluation isolates individual components of challenge solutions, although teams using the same model or framework differed in data pipelines and feature engineering.Methods included Transformer and GBDT variants, HUIM, DIN, and NRMS; HUIM models long-term stable interests and short-term changing preferences.Source: §3.1 · pp. 4–5
3.2.1 Analysis.
- The ranking-task Clicks (59.70) and Read-time (59.40) baselines outperformed Random (49.98), and NRMS only slightly surpassed the most-popular-articles strategy.The table evaluates AUC, MRR, nDCG@5, and nDCG@10 across challenge methods and baselines.Source: §3.2.1 · p. 5, Table 3 · p. 5
3.3 Beyond-Accuracy
- The challenge’s preliminary beyond-accuracy framework evaluates diversity, novelty, serendipity, and coverage, alongside category, topic, and sentiment distributions.The framework adapts four information-retrieval metrics to news publishing.Source: §3.3 · p. 5
3.3.1 Dataset.
- Beyond-accuracy analysis uses a 200,000-user subset, including 50,000 account holders, with each user given one impression over the same 250 newly published articles.Account data enables segmentation by gender, age, and geography.Source: §3.3.1 · p. 5
3.3.2 Analysis.
- Recommendation methods produced markedly different news flows, including different shares of news and auto-generated articles and of negative-sentiment articles.NRMS recommended 10.2% news and 44.6% auto-generated content, while CatBoost:D recommended 45.7% news and 0.02% auto-generated content; CatBoostQL recommended 55.9% negatively-sentimented articles versus 30.6% for NRMS.Source: §3.3.2 · p. 6
- Across most challenge submissions, women’s news flows included less sports and more entertainment than men’s, with some methods showing larger differences.The Transformer recommended 7.1 percentage points more crime and 11.3 percentage points less sports content for women than men.Source: §3.3.2 · p. 6
- Beyond-accuracy scores varied substantially by method, while the reported differences did not appear to extend to demographic categories such as gender.Diversity ranged from 0.791 (CatBoost:D) to 0.588 (HUIM), and coverage from 0.152 (CatBoostQL) to 0.884 (NRMS).Source: §3.3.2 · p. 6
3.3.3 Discussion.
- The analysis is bounded by limited information-retrieval-derived objectives, static candidates that miss effects requiring repeated interactions, and single-run methods with differing pipelines.The authors also note that normative goals were not included as optimization targets because further research is needed on metrics and their correlation with socially relevant outcomes.Source: §3.3.3 · p. 6
4 CONCLUSION
EB-NeRD is a large-scale news recommendation dataset built from Ekstra Bladet user behavior logs, and the RecSys ’24 Challenge enabled extensive experiments on it. The paper concludes that accurate news-content and user-interest modeling matters, while initial beyond-accuracy analysis suggests recommender systems can affect news flow differently.
4 CONCLUSION
- EB-NeRD contains data from over 1 million users and more than 125,000 Danish news articles with titles, abstracts, and full bodies.The authors also report conducting extensive experiments and analysis through the RecSys ’24 Challenge.Source: §4 · p. 6
- The results underscore the importance of accurately understanding news content and modeling user interests for effective news recommendation.Source: §4 · p. 6
- Initial beyond-accuracy analysis is limited but suggests recommender systems can have significantly varying impacts on news flow.Source: §4 · p. 6
5 FUTURE DIRECTIONS
The paper identifies future work on scalable systems, news-specific recommendation methods, and value alignment. Its preliminary evaluation examines beyond-accuracy measures, sentiment, and category distributions, including differences between recommendations for men and women.
5 FUTURE DIRECTIONS
- EB-NeRD is intended to support research on news and user modeling, diversity, fairness, explainability, and privacy-preserving recommendation.The dataset is open-sourced to encourage research on technical and normative challenges in news publishing.Source: §5 · p. 6
- Benchmark methods rely predominantly on complex ensembles that may not suit real-world deployment because of computational demands and difficulty adapting to real-time environments.The authors call for practical, scalable solutions balancing performance and operational efficiency.Source: §5 · p. 7
- Only NRMS and HUIM were evaluated as models specifically designed for news recommendation, leaving opportunities to develop systems addressing the domain’s core challenges.The generalizability of results from news aggregators to classical digital news publishers remains to be seen.Source: §5 · pp. 7–8
- Future value-alignment research can develop beyond-accuracy metrics and optimization strategies closer to journalistic or democratic values; this paper evaluates generic objectives.The preliminary tables report diversity, serendipity, coverage, novelty, sentiment, and category distributions, including gender differences.Source: Table 4 · p. 7, Table 5 · p. 7, §5 · p. 8
A EB-NERD DATASET OVERVIEW
The dataset overview includes tables describing the article, behavior, and history data files. The behavior-file description distinguishes training and validation formats from the test set and notes an additional beyond-accuracy field.
A EB-NERD DATASET OVERVIEW
- Table 6 provides a detailed description of articles.parquet.Source: Table 6 · p. 10
- Table 7 describes behaviors.parquet; training and validation share a format, while test removes four features and adds an is beyond-accuracy field.The removed features are Article ID, Next read-time, Next Scroll Percentage, and Clicked Article IDs.Source: Table 7 · p. 11
- Table 8 provides a detailed description of history.parquet.Source: Table 8 · p. 11