On Application of Learning to Rank for E-Commerce Search

• LambdaMART outperformed all other LETOR methods evaluated for E-Com search.Source: §1 · p. 1
• Crowdsourced relevance judgments were low quality for E-Com search, with many top-level labels receiving zero or one click despite repeated exposure.Source: §1 · p. 2
• Order-rate training was most robust across other feedback objectives, often matching target-signal training, while click rates were easier to predict than add-to-cart ratios.Source: §1 · p. 2
• Popularity-based features work effectively with relevance features, while selecting features to avoid query-attribute sparsity benefits LETOR.Source: §1 · p. 2

2.1 Overview of LETOR

• LETOR learns a parameterized ranking function from query-item features and relevance judgments, then applies it to future queries.Traditional retrieval functions such as BM25 can serve as features within the ranking function.Source: §2.1 · p. 2

2.2 Application of LETOR to E-Com Search

• Applying LETOR to E-Commerce search requires choosing models and training data that generalize to unseen queries and documents.The section also raises whether one model or segment-specific models are preferable.Source: §2.2 · pp. 2–3

2.2.1 Feature Representation:

• E-Commerce ranking features include query, product, and query-product properties, with query attributes matched against product attributes.The section highlights balancing product popularity against relevance and handling sparse query attributes; engagement signals are excluded as features due to overfitting concerns.Source: §2.2.1 · p. 3
• Popularity can help rank equally relevant products, but letting it outweigh critical query attributes such as category can be unacceptable.The section contrasts popularity among matching color variants with a mismatch on a more important attribute.Source: §2.2.1 · p. 3

2.2.2 Relevance judgments:

• Fine-grained E-Commerce relevance judgments are difficult because products matching a query can differ substantially in utility to individual users.For a query such as “14 inch laptop,” many products may be equally relevant to a rater even though purchase likelihood varies with product characteristics.Source: §2.2.2 · p. 4
• Search logs provide clicks, add-to-carts, orders, and revenue as distinct feedback objectives, while higher-quality signals tend to be sparse.The paper studies how these objectives relate and notes that no prior work had examined this tradeoff in E-Commerce search.Source: §2.2.2 · p. 4

3.1 Dataset

• The dataset contains 2.8K product queries and a 5M-product catalog, with up to 120 BM25F-retrieved products per query filtered to at least 100 impressions.After filtering, there were 94.2 documents per query; query-document pairs served as training instances.Source: §3.1 · p. 4
• Feedback rates and ratios were normalized and discretized into five relevance levels, with the highest click-rate product per query rated 4 and products with no clicks rated 0.Add-to-cart ratio uses clicks rather than impressions as its denominator.Source: §3.1 · pp. 4–5
• Click-rate relevance labels decline in frequency from rating 0 to 4, yet every bucket contains sufficient products for comparison.The distribution is averaged across 2.8K queries, with frequencies normalized per query.Source: §3.1 · p. 5, Figure 2 · p. 5
• The 562-feature set includes text-match, product, and query-product attribute features; most features have moderate-to-low click-rate information gain, with a few high-gain features.Engagement-based features were excluded to support generalization to previously unseen query-document pairs.Source: §3.1 · p. 5

3.2 Implementation of LETOR methods

• The study primarily uses LambdaMART and compares popular LETOR methods and classification approaches, reporting average NDCG@10 from five-fold cross-validation.Toolkits and default settings are described for the compared methods; LambdaMART configuration is given in Table 2.Source: Table 1 · p. 5, §3.2 · p. 6, Table 2 · p. 6

3.3 Research Qestions and Experiment Procedure

• The experiments test popularity-feature combinations, query-attribute sparsity, crowdsourced judgments versus behavioral signals, and training-testing combinations of feedback objectives.The procedure varies feature sets and combines feedback signals for training and testing to examine data quantity and signal quality tradeoffs.Source: §3.3 · p. 6

4.1 Comparison of LETOR methods

• LambdaMART achieved the highest test performance for each target objective, followed by AdaRank and RankNet.Table 3 compares ranking algorithms by NDCG@10 for click rate, cart add rate, order rate, and revenue.Source: §4.1 · p. 6, Table 3 · p. 7

4.2 Feature Representation

• Removing popularity features such as sales, reviews, and ratings significantly lowered training and testing NDCG@10 for order-rate prediction.The comparison used a 562-feature set and a version without popularity features; the authors conclude LambdaMART balanced relevance- and popularity-based features.Source: Table 4 · p. 7, §4.2 · p. 7
• Testing NDCG@10 improved by almost 12% for department C and around 13.4% for department F when training used the test-set intersection of attribute features.The intersection set includes only attribute features appearing in the test set.Source: Table 5 · p. 7, §4.2 · p. 7

4.3 Results on crowdsourcing

• Crowd-labeled ideal products received widely varying clicks despite sufficient top-two impressions, indicating crowdsourcing did not provide reliable E-Com relevance judgments.For five example queries, Table 6 bins products by click counts and reports impression summaries.Source: Table 6 · p. 8, §4.3 · p. 8

4.4 Cross Objective Learning Experiments

• Order-rate training was consistently optimal across test objectives, while add-to-cart-ratio training was consistently suboptimal; a practical strategy is to start with click-rate models and switch as order data becomes available.Click-rate models are useful initially because order rates can be sparse.Source: §4.4 · p. 8

5 RELATED WORK

• Prior E-Com search work covered economic models, adapted retrieval, representation learning, facet selection, beyond-relevance ranking, and diversification.The passage cites examples across these areas, including,, [31],,, and.Source: §5 · p. 8
• Web-search LETOR studies compared popular methods, including LambdaMART and RankNet, which are known to perform particularly well.The paper reports comparing these methods on E-Com data and finding LambdaMART outperformed the compared methods.Source: §5 · p. 8
• This work extends LETOR deployment guidance with E-Com findings on popularity features, sparsity-aware feature selection, and feedback signals; its sparsity results point to transfer learning [30].The paper also connects its crowdsourcing findings to prior work on differing E-Com relevance criteria.Source: §5 · pp. 8–9

6 CONCLUSIONS AND FUTURE WORK

• LambdaMART’s relative performance on product search was mostly consistent with web-search findings, confirming it as the best-performing method in the study.The authors recommend it for E-Com application systems and as a baseline for further research.Source: §6 · p. 9
• Popularity-based features were very effective, LETOR balanced them with relevance features, and avoiding sparse query attributes through feature selection was beneficial.Query-attribute sparsity challenged LambdaMART.Source: §6 · p. 9
• Crowdsourcing generated mostly unreliable relevance judgments for E-Com search, despite its usefulness for annotating Web search data.Source: §6 · p. 9
• Order-rate training was most robust across other objectives, click rates were easiest to predict, and add-to-cart ratios hardest; the authors suggest switching from click-rate to order-rate models as order data becomes available.Revenue behaved less robustly, and more test collections are needed to verify the findings.Source: §6 · p. 9
• Future work includes handling engagement-feature uncertainty tied to product age and impressions, as well as vendor-entered or automatically inferred attributes that may be missing or incorrect.Source: §6 · pp. 9–10
• Hidden relationships between engagement signals and other features may produce artificially high train/test performance, making their detection an industry-research challenge.Clicks, cart-adds, and orders were excluded from the feature set because they are closely related.Source: §6 · pp. 9–10
• Query sparsity remains an open challenge for generalizing to unseen queries, especially with thousands of product attribute values; reducing dimensionality and determining data needs require further study.Source: §6 · p. 10
• High variance in browsing depth motivates query-specific NDCG cutoffs, while product-versus-category query classes may warrant different models or evaluation measures.The paper identifies adaptive NDCG cutoffs as a future research direction.Source: §6 · pp. 9–10