PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models for Page-Level Layout Decisions in E-commerce Search

Varun Joshi, Eva C. Song, ChengXiang Zhai

arXiv:2610.10920v1cs.LGcs.CLcs.IR

TL;DR

Evaluating whether and where to place secondary stacks in E-commerce search remains challenging, particularly for layouts not previously shown to users. This paper compares three language-model-based approaches using logged engagement, and finds that representation-based methods outperform prompt-based methods in predicting engagement.

  • Problem

    Page-level layout decisions are difficult to evaluate reliably offline because behavioral signals exist only for layouts already surfaced to users, while online experimentation is costly.

    Source: §1 · p. 1

  • Method

    The study frames layout evaluation as supervised prediction from logged engagement and compares direct prompt judging, prompt-derived feature classifiers, and representation-based methods.

    Source: §1 · p. 1

  • Results

    Representation-based methods significantly outperform all prompt-based methods across the reported metrics, including prompt-derived features with supervised training.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Within this study’s scope, lightweight semantic representations provide a reliable foundation for offline page-level layout evaluation.

    Source: §6 · p. 5

Abstract

from arXiv · show

E-commerce search pages are critical touchpoints for millions of online shoppers. While traditional search engines return a ranked list of results, modern E-commerce search pages increasingly incorporate recommender system modules -- for example, secondary stacks that surface alternative product groupings at specific positions. When introduced appropriately, secondary stacks can improve user engagement; however, suboptimal placement may disrupt browsing flow and degrade the primary results. Unlike traditional search ranking, where evaluation techniques such as interleaving are well established, evaluating page-level layout changes e.g., when and where to insert a secondary stack remains challenging without costly online A/B testing. To address this, we study offline methods for evaluating whether a given layout decision -- specifically, the inclusion of a secondary stack at a particular position -- is beneficial to users. We investigate language models as scalable evaluators by comparing direct prompt-based, prompt-derived feature, and representation-based methods. Our results show that representation-based approaches consistently outperform prompt-based judging in predicting user engagement, suggesting they provide a reliable foundation for offline layout evaluation in E-commerce search.

A Study of Prompt-Based and Representation-Based Methods

The supplied entries identify Varun Joshi and Eva C. Song as authors affiliated with Walmart Global Tech.

CCS Concepts

The paper’s CCS concepts include recommender systems, retrieval-result evaluation, and natural language processing.

  • The CCS concepts list recommender systems and evaluation of retrieval results under information systems.Source: CCS Concepts · p. 1
  • The concepts also include natural language processing under computing methodologies.Source: CCS Concepts · p. 1

Keywords

The paper’s keywords cover e-commerce search, page-layout evaluation, language models, and related methods; its reference identifies the workshop publication.

  • The listed keywords include e-commerce search, page layout evaluation, language models, representation learning, LLM-as-judge, and recommender systems.Source: Keywords · p. 1
  • The reference lists the paper in the OARS Workshop at RecSys 2026.Source: Keywords · p. 1

1 Introduction

Page-level choices such as whether and where to place a secondary stack remain difficult to evaluate offline. The paper frames language-model evaluation as a supervised prediction task and compares three approach classes against logged engagement.

1 Introduction

  • Evaluating secondary-stack placement remains challenging because page changes couple components and UI constraints and reliable assessment often requires costly online experiments.Historical engagement offers high-fidelity offline signals only for layouts already shown to users, leaving unseen layouts without corresponding behavioral signals.Source: §1 · p. 1
  • The paper presents the first study of language models for offline evaluation of page-level layout decisions in E-commerce search.Source: §1 · p. 1
  • The task is framed as supervised prediction grounded in logged user engagement.Source: §1 · p. 1
  • The empirical comparison covers direct prompt-based judging, prompt-derived feature classifiers, and representation-based methods.Experiments and ablations also examine explicit page-level features.Source: §1 · p. 1

2 Related Work

Prior work applies language models to relevance assessment and interface evaluation or generation, but does not evaluate concrete page-level layout choices using behavioral engagement data.

2 Related Work

  • LLM judging in information retrieval addresses query-document relevance, not page-level decisions involving interaction and presentation constraints [9].Prior work also examines reliability and bias in relevance judgments [10].Source: §2 · p. 1
  • HCI studies use language models for UI feedback or perception alignment, rather than predicting behavioral outcomes from concrete layouts using engagement logs [3] [6].Source: §2 · p. 2
  • Layout generation and UI understanding work does not assess whether specific choices improve engagement; interactive-retrieval modeling emphasizes formal models over empirical offline evaluation [5] [1].Source: §2 · p. 2

3 Problem Formulation

The task compares a search layout with a secondary stack inserted at a chosen position against the primary-results-only baseline, using observed Add-to-Cart engagement to label whether the stack is beneficial.

3 Problem Formulation

  • For each query, the test layout inserts candidate stack S after position i in ranked list P, and is compared with baseline P without the stack.Source: §3 · p. 2
  • The task is binary classification of whether the layout with the secondary stack is better, using Add-to-Cart (ATC) as the engagement signal.Source: §3 · p. 2
  • Language-model input includes the query, stack title and items, primary results, and key product attributes such as title and brand.Source: §3 · p. 2

3.1 Labeling scheme

  • Labels come from search logs where a secondary stack was displayed, and the scroll model counts only ATCs at or below its position.ATCs above the stack do not establish that users encountered it.Source: §3.1 · p. 2
  • ATC_S marks engagement with the secondary stack, while ATC_P+ marks engagement with a primary-stack item below it.Source: §3.1 · p. 2
  • An ATC from the secondary stack yields label 1; no stack ATC followed by a lower primary-result ATC yields label 0; otherwise the example is discarded.Discarding cases without evidence of scrolling past the stack limits labels to sessions where users demonstrably encountered it.Source: §3.1 · p. 2

4 Experimental Setup

The experiments compare direct language-model judgments, classifiers using prompt-derived features, and classifiers using pretrained embeddings under shared labeling and data splits. They use a balanced, held-out dataset of search requests with one displayed secondary stack.

4 Experimental Setup

  • All three approach classes use the same labeling scheme and data splits for a controlled comparison of effectiveness.They are direct prompt-based judging, prompt-derived feature classifiers, and representation-based methods.Source: §4 · pp. 2–3

4.1 Direct Prompting to produce final judgement

  • PDJ extends direct prompting with six scored criteria, including intent alignment, product relevance, overlap with primary results, and flow interruption cost.The criteria are scored on a 3-point scale; DP uses a basic system instruction, while PDJ uses the enhanced rubric.Source: §4.1 · p. 3

4.2 Prompt augmented features + Classification (PFC)

  • PFC uses PDJ's decomposed scores as features for simple classifiers trained on labeled data consistent with the held-out test labels.Source: §4.2 · p. 4

4.3 Representation-based embedding + Classification (REC)

  • REC embeds page-level prompt context with pretrained ModernBERT and adds a 32-neuron hidden layer for binary classification, using the same training and test sets as PFC.Source: §4.3 · p. 4

4.4 Data Description

  • The dataset contains 11,000 unique requests from one week of Walmart.com search traffic, split into 8,800 training and 2,200 held-out test examples with balanced labels.Each example had exactly one displayed secondary stack; the PFC–REC comparison controls for labels, splits, and downstream classifier architecture.Source: §4.4 · p. 4

5 Results and Discussion

On the held-out test set, representation-based embeddings with a shallow classifier outperform prompt-based approaches across the reported metrics. The analyses suggest prompt-derived features can improve with explicit signals, while pretrained embeddings already capture them.

5 Results and Discussion

  • Direct Prompting labeled every instance positive, whereas PDJ produced 12% negative decisions and a wider range of scores.The reported Figure 2 results use the models’ float scores.Source: §5 · p. 4
  • A regression analysis found equal weights on PDJ’s decomposed scores, suggesting the language model did not combine them optimally.PFC-X models improved on PDJ; their top features were flow interruption cost, repetition level, and intent alignment, with repetition level receiving a negative coefficient in linear models.Source: §5 · p. 4
  • The representation-based approach significantly outperforms all prompt-based methods across all metrics in a controlled comparison.Figure 2 reports the metrics with 95% bootstrap confidence intervals; Table 1 provides the numerical results.Source: §5 · p. 4, Figure 2 · p. 4, Table 1 · p. 5
  • A possible explanation is that page-level engagement prediction depends on semantic patterns correlated with historical engagement, which embeddings capture continuously.Prompt-based reasoning may add variance and rely on discrete intermediate outputs that do not align with the label.Source: §5 · p. 5

5.1 Ablation study

  • Removing explicit features degrades PFC models by 0.02–0.05 AUC, while adding them to REC-Snn changes AUC by only +0.002.The explicit features include secondary-stack type and insertion position.Source: §5.1 · p. 5

6 Conclusion

The paper studies offline evaluation of secondary-stack layout decisions in E-commerce search and compares prompt-based judging, prompt-derived classifiers, and representation-based methods. Representation-based methods perform more strongly across the experiments, and the findings support lightweight semantic representations as a foundation within this study’s scope.

6 Conclusion

  • This is the first study of offline methods for evaluating secondary-stack inclusion at specific positions on E-commerce search pages.Source: §6 · p. 5
  • Representation-based methods achieved stronger predictive performance than prompt-based judging across the experiments.Source: §6 · p. 5
  • The ablation analysis indicates prompt-based processing may lose structured attributes, while representation-based models capture them more efficiently.The authors conclude that lightweight semantic representations provide a reliable foundation for offline page-level layout evaluation within the study’s scope.Source: §6 · p. 5
Loading 2610.10920v1…