Source-linked AI summary
Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation
Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, Harrie Oosterhuis
TL;DR
Carousel recommender evaluation lacks a metric that properly reflects both two-dimensional interface constraints and empirical browsing behavior. The paper reformulates N2DCG with constrained normalization and empirically grounded discounts, finding that the reformulation better matches eye-tracking behavior and simulated layout preferences. Its normalization procedure is intended for evaluation, not production recommendation generation.
Problem
Existing N2DCG adapts NDCG to carousels but uses an unconstrained ideal layout and discounts that do not reflect observed carousel browsing behavior.
Method
The paper constrains ideal layouts to valid category-specific carousels and develops discount functions incorporating mirrored F-pattern attention and interaction behavior.
Results
The reformulated N2DCG better matches empirical examination frequencies and more reliably predicts preferences between simulated carousel layouts.
Takeaways & Limitations
Carousel evaluation should account for both categorical layout constraints and empirically observed examination patterns.
Takeaways & Limitations
The Hungarian-algorithm computation is used only for normalization during evaluation, not to generate production recommendations.
Abstract
from arXiv · showhide
Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts. N2DCG has been proposed to address this gap by adapting NDCG to carousel-based recommendation, but it relies on unverified assumptions borrowed from the single-list web-search setting that do not transfer well to two-dimensional carousel layouts. We identify two substantial limitations of N2DCG: its ideal ranking, used for normalization, violates carousel constraints, and its discount function does not reflect user browsing behavior observed in empirical data. To address both limitations, we propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function. We validate the proposed metric, showing that it better reflects users' empirical behavior on real-world eye-tracking data and better predicts the comparison results of carousel layouts simulated based on empirical examination patterns.
1 Introduction: New Insights in Carousels
Carousel evaluation extends list-based ranking metrics into two-dimensional layouts, but empirical eye-tracking reveals attention patterns that prior assumptions do not capture. The paper reformulates N2DCG around carousel constraints and observed browsing behavior.
- Empirical motivation: Empirical studies show carousel examination does not follow a single global F-pattern: attention initially peaks at both top corners and shifts after swiping.This observed behavior motivates revising the position discount used by N2DCG.
- Metric background: N2DCG extends NDCG by combining item relevance with position discounts over a two-dimensional carousel grid.2DCG uses row and column coordinates, while the discount models visual attention and interaction effort.
- Empirical motivation: Existing discount functions penalize lower rows and farther-right columns, with action-aware variants additionally penalizing horizontal swipes and vertical scrolling.The parameters encode position and interaction costs associated with revealing items.
- Metric background: Standard N2DCG uses an ideal arrangement that maximizes 2DCG over all two-dimensional permutations, analogous to IDCG normalization in ranked lists.Its ideal layout assigns the most relevant items to positions with the highest discount factors.
- Contributions: The paper proposes constraint-aware normalization, corrects the discount’s failure to model mirrored F-pattern behavior, and evaluates four new position-discount formulations.The row-page discount is reported to align most closely with empirical examination frequencies and predict simulated layout preferences most reliably.
2 Limitations of the Original N2DCG
Original N2DCG normalizes carousel layouts against an unconstrained ideal that may not be displayable. This can depress scores for valid layouts when relevant items belong to different categories.
- Categorical constraint violation: Original N2DCG ignores that each carousel contains items from one category or theme.Its ideal normalization treats all grid positions as one global pool rather than preserving carousel structure.
- Categorical constraint violation: In a 2×3 example, unconstrained I2DCG places relevant items in top positions even when the resulting layout violates category constraints.The unconstrained optimum therefore represents a layout no carousel interface could display.
- Categorical constraint violation: The optimal valid layout reaches 2.2 2DCG, whereas the unconstrained optimum reaches 2.7.Using the larger unconstrained value for normalization lowers N2DCG scores of valid layouts.
3 Proposal: Category-Aware N2DCG
The proposed category-aware N2DCG restricts ideal layouts to valid carousel arrangements and computes normalization within that constrained space. Its normalization procedure is used for evaluation rather than production recommendation generation.
- Category-aware ideal layout: The category-aware variant constrains the ideal layout so every row contains items from the same category.This replaces the global pool of all two-dimensional layouts with a category-respecting subset.
- Category-aware ideal layout: Valid layouts assign category-specific carousels to display rows and rank items within each carousel.Items from different categories never share a row.
- Normalization procedure: The category-aware ideal score is the maximum 2DCG over constrained layouts, and N2DCGcat divides 2DCG by this constrained ideal score.The remaining optimization assigns category-specific carousels to rows while ranking each carousel by decreasing relevance on decreasing-discount positions.
- Normalization procedure: The row-assignment computation uses assignment algorithms such as the Hungarian algorithm during normalization.The paper explicitly states that this computation is not proposed for production recommendation generation.
4 Proposal: Mirrored F-Pattern N2DCG
The proposal reformulates carousel discounting to capture mirrored F-pattern attention and the interaction effort required to access items. It combines a re-indexed position discount with alternative penalties for horizontal swipes, vertical swipes, and row entry.
- Discount-function design: Four discount formulations model both mirrored F-pattern attention and the interaction effort of accessing carousel items.The formulations first adjust position discounting for attention shifts after swipes, then vary how user-action penalties enter the base discount.
- Mirrored F-pattern discount: Re-indexing mirrors each post-swipe page so newly displayed rightmost items receive the highest discount.The first page retains its original indexing; subsequent pages reverse their within-page order to reflect attention shifting from newly displayed items toward the left.
- User-action penalties: The action-penalty variants use horizontal and vertical swipes required to reveal an item as the basis for discounting.These counts represent additional interaction effort for items that are not currently visible.
- User-action penalties: Additive penalties place weighted swipe costs inside the logarithmic denominator of the discount.The weights γ and λ govern horizontal and vertical swipe penalties, respectively.
- User-action penalties: Multiplicative penalties apply per-swipe factors, while row-page discounting penalizes first-page departure and adds geometric decay across rows.The row-page variant treats the first horizontal swipe as costly without further penalizing subsequent pages, while also discounting carousel-row entry.
5 Experiments
The experiments fit discount functions to eye-tracking examination frequencies and evaluate how well the reformulated metric reflects empirical browsing behavior and layout preferences.
- Discount Fit to Examination Data: Figure 2 compares fitted discount functions using Spearman’s ρ, Pearson’s r, and MSE after max-normalizing empirical and computed discounts to [0, 1].The evaluation therefore targets relative attention decay rather than absolute magnitude.
- Experimental Setup: RecGaze provides eye-tracking data for modeling examination behavior in carousel interfaces, including horizontal pagination and vertical scrolling.The experiment focuses on free-browsing screens with ten genre-based carousels, fifteen movies per carousel, and three pages of five movies.
- Experimental Setup: Items are considered examined when at least one valid fixation falls within their area of interest, and position probabilities are estimated from observed examination frequencies.These frequencies both fit candidate discount functions and provide examination-based ground-truth position weights for simulation.
- Experimental Setup: Parameters are trained on 61 participants in Bratislava and evaluated on 26 participants in Amsterdam to test generalization across user groups.The location-based split separates parameter estimation from held-out evaluation.
- Discount Fit to Examination Data: The row-page discount d_RPD best matches empirical examination frequencies, achieving the highest correlations and lowest MSE among the evaluated formulations.Mirrored F-pattern functions capture repeated pagination-related drops and rises better than naive F-pattern functions.
- Metric Evaluation on Simulated Carousels: Across 20,000 simulated trials, the reformulated 2DCG consistently achieves higher pairwise preference accuracy than the original metric.For Δ ≥0.10, the original 2DCG still misjudges 1.3% of binary trials and 1.8% of graded trials.
- Metric Evaluation on Simulated Carousels: The reformulation replaces the unconstrained ideal normalization with category-aware I2DCGcat and N2DCGcat while using the empirically best-performing discount formulation.Category-aware normalization restricts ideal layouts to valid carousel arrangements.
6 Conclusion
The paper identifies two limitations of N2DCG and proposes a reformulation addressing categorical constraints and carousel browsing behavior. Validation supports the reformulation as a better choice for carousel recommendation evaluation, while future work targets behavior-aware simulator-based evaluation.
- The reformulation addresses N2DCG’s neglect of categorical constraints and its misalignment with actual carousel browsing behavior.
- Validation using empirical examination frequencies and simulated pairwise comparisons demonstrates the reformulation’s effectiveness.
- The conclusion suggests reformulated N2DCG as a better choice for evaluating carousel recommendation.
- Future work should extend metric-based evaluation to behavior-aware simulator-based evaluation with explicit models of user browsing and choice behavior.
GenAI Usage Disclosure
The authors used generative AI for language polishing and code cleaning and optimization, reviewed the generated content, and retain responsibility for the final manuscript.
- Generative AI assisted with language polishing and code cleaning and optimization during manuscript preparation.
- The authors reviewed all generated content and take full responsibility for the final manuscript.