PaperlayerView exact revision on arXiv

Source-linked AI summary

Seq2Slate: Re-ranking and Slate Optimization with RNNs

Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed Chi, Elad Eban, Xiyang Luo, Alan Mackey, Ofer Meshi

arXiv:1810.02019v3cs.IRcs.LGstat.ML

TL;DR

Pointwise rankers do not readily capture how items interact in a slate, even though those interactions can shape its appeal. Seq2slate predicts an ordering sequentially, conditioning each selection on earlier ones, and the authors report scalable ranking improvements across benchmarks and a real-world recommendation setting.

  • Problem

    Pointwise ranking does not readily model how an item’s appeal depends on other items and their placement, which matters when only a few slate positions receive attention.

    Source: §1 · pp. 1–2

  • Method

    Seq2slate uses a pointer-network sequence model to select items sequentially, conditioning each choice on previously selected items, and trains from click-through feedback.

    Source: §1 · pp. 2–3

  • Results

    The authors report that seq2slate is highly scalable and can deliver significant improvements in ranking results, with experiments on benchmarks and a real-world recommendation system.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    Sequential decoding lets the model dynamically adjust item placement based on previous choices while accounting for high-order interactions.

    Source: §1 · p. 2, §6 · p. 15

    The authors identify Transformer architectures and alternative inference and training algorithms as directions whose performance remains to be explored for seq2slate.

    Source: §6 · p. 15

Abstract

from arXiv · show

Ranking is a central task in machine learning and information retrieval. In this task, it is especially important to present the user with a slate of items that is appealing as a whole. This in turn requires taking into account interactions between items, since intuitively, placing an item on the slate affects the decision of which other items should be placed alongside it. In this work, we propose a sequence-to-sequence model for ranking called seq2slate. At each step, the model predicts the next `best' item to place on the slate given the items already selected. The sequential nature of the model allows complex dependencies between the items to be captured directly in a flexible and scalable way. We show how to learn the model end-to-end from weak supervision in the form of easily obtained click-through data. We further demonstrate the usefulness of our approach in experiments on standard ranking benchmarks as well as in a real-world recommendation system.

1 Introduction

Pointwise ranking is efficient but can miss how items interact within a slate, especially when only a few positions receive attention. Seq2slate uses sequential decoding to condition each item’s score on earlier selections, and the paper reports training from click-through data and evaluation on benchmarks and a real-world recommendation system.

1 Introduction

  • Pointwise rankers score items independently and sort them, so they cannot readily represent dependencies between items in the resulting slate.These interactions matter when display area is limited or position bias leaves only a few highly ranked items likely to receive attention.Source: §1 · pp. 1–2
  • Pairwise and listwise losses consider item dependencies during training, but their ranking functions still assign pointwise scores at inference.Other interaction-aware methods may rely on simple interaction forms or use approximate, potentially expensive inference procedures.Source: §1 · p. 2
  • Sequential decoding conditions each item’s score on previously selected items, allowing seq2slate to model high-order interactions, including synergies and interference.The output sequence is the resulting ordering of candidate items.Source: §1 · p. 2
  • Seq2slate applies a sequence-to-sequence model to ranking, using candidate items as input and their ordering as output.The paper uses pointer networks and recurrent neural networks to capture dependencies while keeping inference manageable.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • The authors describe end-to-end training from click-through data and report experiments on learning-to-rank benchmarks and a real-world recommendation system.The approach uses weak supervision rather than requiring ground-truth rankings, which are more expensive to obtain.Source: §1 · p. 3

2 Ranking as Sequence Prediction

The paper formulates ranking as predicting a permutation of candidate items, with each selection conditioned on the prior slate. Its pointer-network architecture encodes candidates and decodes the ordering by pointing to remaining inputs.

2 Ranking as Sequence Prediction

  • Ranking is modeled as predicting a permutation of candidate items given their features and a query or context.The target ranking may optimize an engagement measure, such as the number of music tracks played.Source: §2 · p. 3
  • Each position’s item probability is conditioned on the items already placed, allowing the model to represent high-order dependencies without conditional-independence assumptions.The item set and its size may vary by instance, so the output vocabulary is not fixed as in standard sequence-to-sequence tasks.Source: §2 · pp. 3–4

Pointer-Network Architecture for Ranking

  • A pointer network uses attention to point to input items rather than choosing from a fixed output vocabulary.The encoder reads candidate vectors into latent states; the decoder uses an attention query to produce a distribution over the next item.Source: Pointer-Network Architecture for Ranking · p. 4
  • At each decoding step, seq2slate scores remaining items, masks items already selected, then feeds the chosen item’s embedding into the next decoder step.The first decoder step receives a learned “go” vector.Source: Pointer-Network Architecture for Ranking · p. 4
  • Sequential decoding has O(n^2) inference cost; the single-step alternative costs O(n), while a base ranker supplies the input ordering for re-ranking.Using a base ranker lets seq2slate focus its modeling capacity on interactions between items rather than individual items.Source: Pointer-Network Architecture for Ranking · p. 5

3 Training with Click-Through Data

Seq2slate can be trained from inexpensive click-through feedback rather than curated rankings. The section presents both a sampled REINFORCE objective and supervised training that gives feedback at each decoding step, with losses designed to reflect item order.

3 Training with Click-Through Data

  • Click-through data supplies relatively inexpensive training feedback; examples use binary click/no-click labels, while richer engagement feedback is possible.The section contrasts click logs with human-curated relevance scores, ratings, or full rankings.Source: §3 · p. 5
  • Because click labels do not provide ground-truth permutations, the model cannot use standard teacher forcing and instead trains directly from labels.The standard approach feeds ground-truth tokens into the next prediction step during training.Source: §3 · p. 6

3.1 Training Using REINFORCE

  • REINFORCE uses policy gradients to optimize the expected ranking measure for permutations sampled from the model.The sampled-gradient estimate uses a baseline to estimate expected reward and reduce variance.Source: §3.1 · p. 6

3.2 Supervised Training

  • Supervised training gives feedback at each decoder step, avoiding the need to wait until sequence completion as in reinforcement learning.Source: §3.2 · p. 6
  • Per-step losses can use cross-entropy or hinge loss, and other standard ranking surrogate losses, including ones for non-binary relevance labels.Cross-entropy encourages probability on positive labels; hinge loss favors higher scores for positive than negative labels.Source: §3.2 · p. 7
  • The sequence loss ignores items already selected, so no further loss accrues after a positive label is predicted; position weights can emphasize early ranks or restrict training to the first k steps.Source: §3.2 · p. 7
  • For sampled permutations, the expected loss is differentiable, and its gradient both lowers the probability of high-loss samples and reduces each sample's loss.A baseline approximating sampled loss is used for variance reduction.Source: §3.2 · p. 8
  • Greedy training selects the highest-probability remaining item at each step, but its loss is non-differentiable at score ties and otherwise differentiable almost everywhere.Source: Greedy Decoding · pp. 8–9

4 Experimental Results

Experiments evaluate seq2slate on learning-to-rank benchmarks and in a commercial recommendation system, including generated interaction data and a live test.

4.1 Learning-to-Rank Benchmarks

  • On diverse-clicks data with induced high-order dependencies, seq2slate significantly outperformed all tested ranking baselines.The reported benchmark results use generated binary labels rather than the datasets’ original relevance scores.Source: Table 1 · p. 10, §4.1 · p. 10
  • Across diverse-clicks training variants, cross-entropy generally performed best; REINFORCE was comparable on Yahoo, worse on Web30k, and took 4X longer to converge.Greedy and sampling policies showed no significant performance difference.Source: Table 2 · p. 11, §4.1 · p. 11
  • On benchmark data, the one-step decoder performed comparably to sequential decoding while reducing training time 4X and inference time 3X.For real-world data, one-step decoding was 2.5X faster per iteration in both training and inference.Source: Table 2 · p. 11, §4.1 · p. 12
  • The learned slates stayed close to the input ranking while demoting some items, presumably because of interactions with previously placed items.Source: §4.1 · p. 11
  • On similar-clicks data, seq2slate matched baseline performance on Yahoo less well but significantly outperformed baselines on the harder Web30k data.No change to the model or training algorithm was needed for this interaction variant.Source: Table 3 · p. 12, §4.1 · p. 13
  • Randomizing input order significantly worsened performance, suggesting that reranking is easier than learning a ranking from scratch.The shuffled-data experiment used 10 randomly ordered copies of each training example.Source: §4.1 · p. 12

4.2 Real-World Data

  • On real test data, seq2slate improved all reported performance metrics over the production ranker, and sequential decoding outperformed the one-step decoder.The test set was roughly the same size as the training set.Source: Table 5 · p. 13, §4.2 · p. 13
  • In a live experiment, seq2slate had significantly higher CTR than the production ranker in top positions.The comparison reports CTR differences by position.Source: Figure 3 · p. 14, §4.2 · p. 14

5 Related Work

Seq2slate applies sequential decoding to ranking, distinguishing it from approaches that encode candidate sets but assign pointwise scores. The related work also discusses permutation-learning methods whose extension to ranking without ground-truth permutations is unclear.

5 Related Work

  • Seq2seq models have been used for other prediction tasks, but seq2slate targets ranking and training from weak click-through feedback.Source: §5 · p. 14
  • Seq2slate learns a representation of the entire slate in the RNN state and can train item embeddings jointly with the sequence model.Source: §5 · p. 14
  • Sequential decoding distinguishes seq2slate from related RNN rankers that use a single decoding step, allowing scores to change with previously selected items.The cited comparison also notes differences in recurrent cells, input order, and training feedback.Source: §5 · pp. 14–15
  • Permutation-learning methods based on continuous relaxations target reconstruction of scrambled objects, and their extension to ranking without a ground-truth permutation is unclear.Source: §5 · p. 15

6 Conclusion

Seq2slate uses pointer networks and sequential decoding to adjust slate placements based on earlier choices, with training designed for weak user feedback. Experiments show scalable ranking improvements; the conclusion also identifies architectural, algorithmic, and off-policy extensions for future work.

6 Conclusion

  • The experiments show that seq2slate is highly scalable and can deliver significant improvements in ranking results.Source: §6 · p. 15
  • Pointer networks support sequential decoding that dynamically adjusts item placement given previous choices, while proposed sequence losses and gradient-based updates address weak click-through feedback.Source: §6 · p. 15
  • Future work includes testing Transformers, beam search, and newer seq2seq training approaches in the ranking setting.Source: §6 · p. 15
  • Off-policy correction would importance-weight training examples to address a mismatch between the logging policy and the policy evaluated during training.The expected sequence loss can be approximated from logged samples; the authors leave this extension to future work.Source: §6 · pp. 15–16

A Derivation of the Expected Loss

The passage introduces an expression for expected loss as a function of model scores S.

A Derivation of the Expected Loss

Related papers

Loading 1810.02019v3…