Source-linked AI summary
From Click Modeling to Offline and Off-Policy Evaluation in Carousel Recommendation
Jingwei Kang
TL;DR
Carousel feedback is difficult to interpret because clicks reflect interface, exposure, layout, and position effects rather than preference alone. This thesis studies carousel behavior, develops observable-variable-based click-modeling and evaluation approaches, and finds that post-examination clicks remain position-dependent.
Problem
Carousel click modeling and evaluation lack a sufficient behavioral foundation because existing single-list assumptions and the limited available evidence do not fully explain interaction in carousel layouts.
Method
The thesis combines empirical carousel behavior analysis with a click-model design framework, a discrete-choice click model, a reformulated carousel metric, and planned off-policy evaluation methods.
Results
Post-examination click probability remains position-dependent, so clicks after examination cannot be treated as position-invariant relevance signals.
Takeaways & Limitations
The connected modeling and evaluation program aims to provide tools for improving carousel recommendation policies from logged interaction data.
Takeaways & Limitations
The thesis remains at a stage where the design of two planned off-policy evaluation routes and the broader evaluation methodology require further feedback.
Abstract
from arXiv · showhide
Carousel interfaces are widely used in modern recommendation systems. Unlike traditional interfaces that present a single ranked list, carousels simultaneously present several ranked lists to the user, as horizontally swipeable rows stacked on top of each other. In this design, the rankings are closely tied to the two-dimensional layout. Consequently, user behavior is shaped not only by item preference, but also by row organization, viewport constraints, and item context. This tight coupling between ranking and presentation complicates the interpretation of user feedback, introducing new challenges for recommendation evaluation. My PhD research aims to address these challenges by rethinking how carousel clicks are modeled and how carousel recommendation policies can be evaluated from logged interaction data. So far, I have studied how users interact with carousel interfaces and developed a click model design framework that prioritizes mathematical relationships between observed variables over latent behavioral assumptions. Building on these results, my ongoing work includes a project using discrete choice models to represent clicks as choices, alongside a project that develops carousel-specific offline metrics. As a next step, I plan to develop off-policy evaluation methods that estimate the performance of recommendation policies from logged interactions. Taken together, the expected contribution of my thesis is a connected body of work that links carousel click modeling with offline and off-policy evaluation, so that carousel recommendation policies can be improved more reliably.
1 Introduction
Carousel interfaces present multiple horizontally swipeable ranked lists in a two-dimensional layout, making clicks depend on interface structure as well as user preference. Eye-tracking evidence challenges the assumption that clicks after examination are position-invariant relevance signals.
- Carousels stack multiple horizontally swipeable ranked lists vertically, with each row organized around a topic such as a movie or music category.
- Offline and off-policy evaluation are complicated because carousel layouts, logging policies, and item exposure shape the meaning of observed clicks.
- Eye-tracking evidence shows that click probability remains position-dependent even conditional on examination.Position and layout affect examination and how examined items are interpreted, compared, and clicked.
- These findings motivate rethinking how carousel recommendation should be modeled and evaluated when standard click-modeling and evaluation assumptions do not hold.
2 Related Work
Prior research established behavioral and evaluation assumptions mainly for single-list and grid interfaces, while carousel-specific evidence and methods remain limited. This leaves carousel click modeling and off-policy evaluation as open research problems.
- 2.1 User behavior analysis: Eye-tracking research has produced widely used behavioral assumptions for single-list search and recommendation, including the examination hypothesis and F-pattern attention.Irregular grids show that examination behavior can instead depend strongly on layout.
- 2.1 User behavior analysis: Carousel behavior was poorly understood because RecGaze was the only eye-tracking dataset and had received limited analysis.
- 2.2 Click models: Existing click models are dominated by single-list graphical models, with some extensions to grids but only one published carousel model.The cited carousel model is the cascade-style Carousel Click Model.
- 2.2 Click models: Carousel clicks remain an open modeling problem because existing assumptions do not account for observed browsing behavior or position dependence after examination.
- 2.3 Offline and off-policy evaluation: Carousel offline metrics must generalize weighted-gain evaluation to two-dimensional layouts accessed through horizontal and vertical navigation.N2DCG is an existing NDCG-style metric incorporating carousel structure into position discount.
- 2.3 Offline and off-policy evaluation: Logged-feedback metrics also face exposure and selection bias, motivating off-policy evaluation of target policies from data collected under different logging policies.
- 2.3 Offline and off-policy evaluation: Carousel-specific off-policy evaluation remains a gap because users navigate a partially observed two-dimensional layout before clicking, while prior OPE work focuses mainly on single items or ranked lists.
3 Research Questions and Thesis Structure
The thesis is organized around two connected questions: how carousel clicks should be interpreted and modeled, and how carousel policies should be evaluated from logged interactions. Its papers link behavioral data, click modeling, offline metrics, and planned off-policy evaluation.
- RQ1: Rethinking Carousel Clicks: RQ1 asks how carousel clicks should be interpreted and modeled, covering user interaction and click-model design.
- RQ2: Evaluating Carousel Recommendation: RQ2 asks how carousel recommendation policies should be evaluated from logged interaction data through offline metrics and off-policy evaluation.
- Thesis structure: The thesis connects its papers so that behavioral studies inform both the choice model and metric, while the click-model framework feeds the choice model.
- Thesis structure: The choice model is intended to provide the basis for planned off-policy evaluation, while N2DCG reformulation and off-policy evaluation address complementary evaluation goals.
4.1 Chapter: Rethinking Carousel Clicks
This chapter establishes empirical foundations for carousel interaction and uses them to rethink click-model design. It combines eye-tracking analysis, a mathematical design framework, and a choice-based model for clicks.
- Empirical foundations: RecGaze combines eye tracking with clicks, cursor movements, and post-selection explanations from 87 participants across 40 carousel screens and 3,477 valid interactions.The study included three movie-selection tasks: free browsing, semi-free browsing, and direct search.
- Empirical findings: Four tested behavioral assumptions are refuted, including a global F-pattern, cascade-like examination after clicks, position-invariant post-examination clicks, and title-led navigation.The first page shows dual-focus attention, later pages show mirrored F-patterns, clicked items produce an L-pattern, and users often examine items without examining titles.
- Model design: The click-model design framework prioritizes mathematical relationships between observed variables over latent behavioral assumptions.It identifies global dependencies, sequentiality, and factorization as three design choices and places single-list, grid, and carousel models in one taxonomy.
- Choice-based model: The ongoing choice-based model represents clicks as a two-stage process: navigation generates a consideration set, then the user chooses among considered items.The consideration set begins with initially visible items and expands through horizontal swipes, vertical scrolling, or stopping.
- Choice-based model: The model accounts for joint consideration among items and navigation through two-dimensional layouts with partial visibility.Its challenges are parameterizing consideration-set construction and choice estimation when only the final click is logged.
4.2 Chapter: Evaluating Carousel Recommendation
This chapter addresses evaluation of carousel recommendation policies from logged interactions. It reformulates an existing offline metric and plans off-policy estimators for carousel-specific exposure and navigation structure.
- Chapter scope: The chapter addresses carousel evaluation through an offline layout metric and planned off-policy evaluation methods.The offline metric is currently under review, while the off-policy evaluation work is planned.
- Offline evaluation: The reformulated N2DCG corrects an ideal ranking that violates carousel constraints and a discount function inherited from single-list search assumptions.The reformulation tracks real-world eye-tracking behavior more faithfully and better predicts simulated carousel layout comparisons.
- Off-policy evaluation: Planned carousel OPE methods address a mismatch between standard slate estimators and layouts containing multiple stacked rankings with partial visibility.Standard estimators assume a single fully observable ranking, whereas carousels expose items through two-dimensional navigation.
5 Conclusion
The thesis connects empirical carousel behavior, click modeling, and recommendation evaluation. Its current scope includes a choice-based click model, off-policy evaluation directions, and a behavior-grounded N2DCG reformulation.
- Conclusion: The thesis combines empirical analysis, observable-variable click-model design, choice-based carousel modeling, and complementary off-policy evaluation directions.These components are presented as connected work on modeling and evaluating carousel recommendations.
- Conclusion: The behavior-grounded reformulation of N2DCG is currently under review.
- Conclusion: Feedback is sought on the design of the two planned off-policy evaluation routes and the broader evaluation methodology for carousel recommendation.