PaperlayerView exact revision on arXiv

Source-linked AI summary

Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search

Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, Karthik Subbian

arXiv:2206.06588v1cs.IRcs.LG

TL;DR

Shopping search relevance remains challenging because noisy results, ambiguous query intent, and diverse products complicate classifying items for a query. The paper introduces a multilingual dataset of manually labeled product-query pairs and three evaluation tasks, with baseline results. It presents the collection as a benchmark for product search research, while noting that human labels can be noisy.

  • Problem

    Noisy results, difficult query intent, and diverse products make correctly classifying products for shopping queries challenging, despite machine-learning advances.

    Source: §1 · p. 1

  • Method

    The paper introduces a multilingual dataset of difficult shopping queries with ESCI-labeled results and product information, and defines ranking, relevance-classification, and substitute-identification tasks.

    Source: §1 · p. 1

  • Results

    The paper provides initial baseline results for its three evaluation tasks, including ranking nDCG of 0.852 for the neural approach versus 0.551 for Terrier-BM25, a comparison the authors call unfair.

    Source: §4.2 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    The authors present the dataset as a large-scale benchmark intended to spur research on scalable, high-performing product-search models.

    Source: §5 · p. 5

    Human annotations are imperfect and potentially noisy; the authors estimate 91% overall agreement and observe leniency in assigning Substitute labels.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Improving the quality of search results can significantly enhance users experience and engagement with search engines. In spite of several recent advancements in the fields of machine learning and data mining, correctly classifying items for a particular user search query has been a long-standing challenge, which still has a large room for improvement. This paper introduces the "Shopping Queries Dataset", a large dataset of difficult Amazon search queries and results, publicly released with the aim of fostering research in improving the quality of search results. The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements. The dataset is multilingual with queries in English, Japanese, and Spanish. The Shopping Queries Dataset is being used in one of the KDDCup'22 challenges. In this paper, we describe the dataset and present three evaluation tasks along with baseline results: (i) ranking the results list, (ii) classifying product results into relevance categories, and (iii) identifying substitute products for a given query. We anticipate that this data will become the gold standard for future research in the topic of product search.

1 INTRODUCTION

Shopping product search remains difficult because noisy results, ambiguous query intent, and diverse products complicate relevance classification, while binary labels do not capture useful gradations. The paper introduces a multilingual dataset with manually labeled ESCI judgments and product information to support three product-search tasks.

1 INTRODUCTION

  • Search relevance matters because it can improve customer experience and engagement, while noisy results and query-intent ambiguity make shopping relevance classification challenging.Product diversity adds to the difficulty of matching items to a particular shopping query.Source: §1 · p. 1
  • Binary relevance misses intermediate cases, such as products that fail some query aspects but function as substitutes or complement an exact match.The ESCI labels distinguish Exact, Substitute, Complement, and Irrelevant products.Source: §1 · p. 1
  • The dataset supports ranking product results, classifying query-product pairs into Exact, Substitute, Complement, or Irrelevant, and identifying substitutes for a query.Each query has up to 40 results with ESCI judgments and Amazon catalog information, including titles, descriptions, and bullet points.Source: §1 · p. 1
  • The dataset combines real customer queries, multilingual breadth and per-query depth, manually assigned multi-valued relevance labels, and deliberately challenging query samples.It also provides categorical and textual product metadata at multiple representation levels, from short titles to longer descriptions, with links to Amazon catalog products.Source: §1 · pp. 1–2

2 SHOPPING QUERIES DATASET

The dataset combines difficult multilingual shopping queries with manually annotated product relevance judgments and catalog features. It uses baseline-informed query selection, human annotation, and task-specific versions and splits.

2.4 Splits and Statistics

  • The dataset covers English, Japanese, and Spanish queries and uses query-stratified splits of 70% training, 15% public test, and 15% private test.The private test is held out for evaluation; the public test is for tuning.Source: §2 · p. 2, §2.4 · p. 3

2.1 Query Selection Process

  • Queries were selected for baseline-model mistakes because random samples often yield near-perfect results, though this strategy is biased by the chosen baselines.The authors report that it produced more challenging queries in their experience than sampling tied to a particular model, such as active learning or adversarial approaches.Source: §2.1 · p. 2

2.2 Annotation of Relevance Judgements

  • At least three trained annotators labeled each pair, with majority vote used as gold; overall human-label agreement was 91%.About half of disagreements involved Irrelevant versus Substitute, while Exact/Not-Exact agreement exceeded 96%.Source: §2.2 · pp. 2–3

2.3 Product Description

  • Each query-product record includes an ESCI label and catalog fields such as product title, description, bullet points, brand, and color.The listed fields also include query and product identifiers and product locale.Source: §2.3 · p. 3

2.4 Splits and Statistics

  • 130,652 queries and 2,621,738 judgments form the larger version; the smaller version has 48,300 queries and 1,118,117 rows.The smaller version filters out simpler queries, while the larger version is a superset for classification and substitute-identification tasks.Source: §2.4 · p. 3
  • English queries make up 61.8% and 74.5% of the small and large versions, respectively, versus 16.6% and 11.6% Spanish and 21.5% and 13.9% Japanese.The average number of results per query is slightly smaller in English than in Spanish and Japanese, whose averages are comparable.Source: §2.4 · p. 3
  • The larger version is described as easier: it has 65.2% Exact matches versus 43.7% in the smaller version, and 20.1 versus 23.2 results per query.The dataset summary also reports that Exact is the most frequent label and Complement the least frequent.Source: §2.4 · p. 3

3 TASKS AND EVALUATION METRICS

The paper defines ranking, multiclass classification, and substitute-identification tasks for shopping search. It specifies relevance metrics for each task, including graded ranking gains and class-imbalance-aware classification evaluation.

3 TASKS AND EVALUATION METRICS

  • Together, the three tasks cover ranking product results, assigning graded relevance categories, and identifying substitutes.The paper presents these as ways to build ranking strategies and identify result categories relevant to customer experience.Source: §3 · p. 3

3.1 Task 1: Query-Product Ranking

  • Ranking orders up to 40 products per query from Exact through Substitute and Complement to Irrelevant, evaluated with nDCG gains of 1.0, 0.1, 0.01, and 0.0.The paper omits private-test information for this task while the competition is ongoing.Source: §3.1 · pp. 3–4

3.2 Task 2: Multiclass Product Classification

  • Multiclass classification assigns each query-product pair an Exact, Substitute, Complement, or Irrelevant label and is evaluated with micro-averaged F1 across classes.The paper gives class proportions of 65.20% Exact, 21.91% Substitute, 2.89% Complement, and 10.00% Irrelevant as the reason for micro-averaging.Source: §3.2 · p. 4

3.3 Task 3: Product Substitute Identification

  • Substitute identification is binary classification of each query-product pair as Substitute or non-Substitute, evaluated with F1.The task focuses on finding positive substitute cases.Source: §3.3 · p. 4

4 EXPERIMENTAL RESULTS

The section establishes baseline approaches for ranking, classification, and substitute identification, then reports public-test results across three languages. Neural ranking outperforms the default Terrie-BM25 baseline, while classification performance varies by language and all three tasks retain room for improvement.

4 EXPERIMENTAL RESULTS

  • The experiments provide initial standard ranking and classification baselines, with public-test results broken down for English, Spanish, and Japanese.Source: §4 · p. 4, §4.2 · p. 4

4.1 Baseline Models

  • Ranking uses a fine-tuned MS MARCO Cross-Encoder for US and multilingual MPNet models for Spanish and Japanese, alongside Terrie-BM25 across all locales.The ranking models use product titles; training labels map Exact to 1.0 and other relevance labels to 0.0.Source: §4.1 · p. 4
  • Classification and substitute identification use a classifier over frozen multilingual BERT query and product-title representations, with max pooling before concatenation and learned layers.The MLP uses a 128-neuron fully connected layer with 10% dropout, followed by a classification layer.Source: §4.1 · p. 4

4.2 Results

  • Task 3 also favors English, with language differences about half the size of Task 2’s; the authors see substantial room to improve all three baselines, especially classification.Neural Task 1 nDCG scores range from 0.840 to 0.857 across languages.Source: §4.2 · p. 4
  • Task 1 neural nDCG is 0.852 vs. Terrie-BM25 0.551, though default preprocessing penalizes Japanese; Task 2 F1 is 0.685 for English vs. 0.580 for Spanish.The table reports public-test nDCG and Micro F1 baseline scores across tasks; the authors caution that the Task 1 comparison is not fair because Terrie-BM25 lacks Japanese-specific preprocessing.Source: §4.2 · p. 4, Table 4 · p. 5

5 CONCLUSION

The paper presents the Shopping Queries Dataset as a large-scale benchmark for e-commerce product search. It describes the dataset, defines three KDDCup’22 evaluation tasks, and reports initial baseline results.

5 CONCLUSION

  • The paper introduces a large-scale benchmark intended to improve state-of-the-art algorithms for e-commerce product search.It describes the dataset and its statistics, presents three KDDCup’22 evaluation tasks, and provides initial baseline results.Source: §5 · p. 5

Related papers

Loading 2206.06588v1…