Source-linked AI summary
Middleman Bias in Advertising: Aligning Relevance of Keyphrase Recommendations with Search
Soumik Dey, Wei Zhang, Hansi Wu, Bingfeng Dong, Binbin Li
TL;DR
Click and sales signals provide unreliable negative relevance evidence, while Search filtering shapes which advertised keyphrases enter click logs. The paper trains models against Search relevance judgments and compares encoder approaches; in an A/B test, bert-tiny improved several engagement measures against Jaccard while CTR remained neutral.
Problem
Click and sales data provide unreliable negative relevance signals, and Search filtering leaves click-trained models without examples of keyphrases it rejects.
Source: §1 · pp. 1–2Method
Results
Against Jaccard, bert-tiny increased impressions 20.17%, clicks 13.01%, bought items 26.04%, and CVR 11.6%, while CTR remained neutral.
Source: §4 · p. 4Takeaways & Limitations
The authors conclude that advertiser keyphrase relevance involves both Advertising and Search, and that alignment with Search matters because Search determines relevance in auctions.
Source: §1 · p. 2, §5 · p. 5The Jaccard baseline penalizes short head keyphrases, can eliminate single-token targets, and misses semantically aligned opportunities; the authors describe it as a stopgap rather than the most accurate model.
Source: §2 · pp. 2–3
Abstract
from arXiv · showhide
E-commerce sellers are recommended keyphrases based on their inventory on which they advertise to increase buyer engagement (clicks/sales). Keyphrases must be pertinent to items; otherwise, it can result in seller dissatisfaction and poor targeting -- towards that end relevance filters are employed. In this work, we describe the shortcomings of training relevance filter models on biased click/sales signals. We re-conceptualize advertiser keyphrase relevance as interaction between two dynamical systems -- Advertising which produces the keyphrases and Search which acts as a middleman to reach buyers. We discuss the bias of search relevance systems (middleman bias) and the need to align advertiser keyphrases with search relevance signals. We also compare the performance of cross encoders and bi-encoders in modeling this alignment and the scalability of such a solution for sellers at eBay.
1 Introduction
Advertiser relevance filters must contend with click data that gives unreliable negative signals and Search's separate relevance filter, which shapes which keyphrases generate logged clicks. The paper frames relevance as an interaction between Advertising and Search and proposes aligning their filters.
1 Introduction
- Advertising recommendations involve retrieval, relevance filtering, and ranking, with relevance shaping seller perception and auction competition.Sellers use recommendations to gain visibility beyond organic search placement.Source: §1 · p. 1
- Click and sales data can indicate relevance positively, but missing-not-at-random exposure makes a lack of clicks an unreliable negative signal.Buyers see items ranked by prior clicks, so low-ranked items may receive no clicks even when relevant.Source: §1 · p. 1
- Search acts as a middleman in auctions, and logged click data includes only keyphrases that pass Search’s relevance filter.Advertising still needs to filter keyphrases that Search considers irrelevant, although those examples are absent from click-based training.Source: §1 · pp. 1–2
- The authors redefine advertiser keyphrase relevance as an interaction between Advertising and Search and propose aligning the systems.Search ultimately determines which keyphrases are deemed relevant in the auction process.Source: §1 · p. 2
- Figure 2 distinguishes keyphrases rejected by Advertising, Search, or both, illustrating why their filters can disagree.The paper describes cases where Search rejects keyphrases that could be genuinely relevant, preventing an item from entering those auctions.Source: Figure 2 · p. 2, §1 · p. 2
2 Previous Experiments and Learnings
Earlier relevance approaches encountered limitations in both their training data and model behavior. Click-trained CTR modeling failed to align with Search, while the deployed Jaccard filter was a simple but imperfect stopgap.
2 Previous Experiments and Learnings
- The human-judgment BERT model used about 80,000 records, but category underrepresentation and low annotator agreement preceded seller complaints and its removal from production.eBay’s more than 100,000 categories require substantially more than 80,000 records for adequate representation.Source: §2 · p. 2
- The click-trained XGBoost CTR model failed offline on Search alignment and spot-checked human judgment, while analysis found missing Search-filtered pairs and inherited click-data biases.A Jaccard-index filter performed better and was released to production, despite also being a feature in the CTR model.Source: §2 · p. 2
- Jaccard’s token intersection-over-union penalized short head keyphrases, eliminated single-token targets, and missed semantically aligned opportunities.At eBay, 10% of keyphrases drive 90% of buyer engagement in clicks and sales.Source: §2 · p. 2
- Although easy to implement and integrate, Jaccard was not the most accurate relevance model and had been adopted as a stopgap.Source: §2 · p. 3
3 Data Curation and Experiment Design
The authors train models to predict Search’s relevance judgments for item–keyphrase pairs, using category-stratified data and comparing encoder designs and training objectives. They evaluate Search agreement alongside inference scalability to select a model for production consideration.
3 Data Curation and Experiment Design
- Training uses 24 million Search-labeled item–keyphrase pairs and evaluation uses 3 million, with labels denoting whether Search’s relevance judgment passed or failed.The data are stratified by category activity; the prediction inputs are an item title, category information, and its keyphrase.Source: §3 · p. 3
- The study compares bi-encoders, which encode item and keyphrase separately, with cross-encoders, which process them together.The item representation includes title and category information.Source: §3 · p. 3
- Bi-encoder training compares contrastive learning, a softmax classifier, and in-batch random negative sampling using click-derived positives.The click-based sampling data require at least one click, CTR of at least 0.05, and at least 30 impressions to reduce noisy examples.Source: §3 · p. 3
- Cross-encoder sizes are selected with inference latency in mind, and models are compared on Search agreement and scalability before an online A/B test against Jaccard.The experiment frames model selection as a trade-off between performance and scalability.Source: §3 · p. 3
4 Results
Cross-encoders outperformed bi-encoders in Search-relevance modeling, and deployed bert-tiny improved most buyer-engagement outcomes against Jaccard while CTR remained neutral.
4 Results
- The offline evaluation used 3 million item-keyphrase pairs with Search relevance judgments, reporting precision, recall, and F1.Precision measures alignment with Search’s relevance filter, while recall measures targeting potential lost through unnecessary filtering.Source: §4 · p. 3
- 0.74 precision and 0.80 F1 made bert-tiny the second-best model on those metrics; bert-mini led cross-encoders, while contrastive loss led bi-encoders.The authors report that bert-mini achieved the best cross-encoder precision and F1, and that contrastive-trained bi-encoders beat IRNS and Softmax-trained bi-encoders in F1.Source: Table 1 · p. 3, §4 · p. 4
- Bert-tiny was selected for release: daily-differential inference took 1.5 hours, versus 1 hour for bi-encoders and 2.5 hours for bert-mini.Serving separates delayed full and daily-differential batch inference from urgent NRT inference for newly created or revised items; GPU options were unsustainable due to cost.Source: §4 · p. 4
- 20.17% more impressions, 13.01% more clicks, 26.04% more bought items, and 11.6% higher CVR followed bert-tiny’s A/B test against Jaccard; CTR remained neutral.Each increase had p<0.05, while CTR had p=0.17; the model also reduced the number of keyphrases surfaced to sellers by 7%.Source: §4 · p. 4
- Bert-tiny reduced false positives and false negatives while increasing true positives; many reduced false negatives were shorter head keyphrases desirable for advertisers’ targeting potential.The full batch covers around 100 billion item-keyphrase pairs, while daily differential serves around 7 billion; differential latency determines production feasibility.Source: §4 · pp. 4–5
5 Conclusions
The paper frames advertiser keyphrase relevance as an interaction between Advertising and Search, identifies middleman bias in click-based training, and compares encoder approaches for aligning with Search.
5 Conclusions
- The authors identify click-only training limitations and middleman bias, frame relevance as an Advertising–Search interaction, and compare bi-encoders with cross-encoders for Search alignment.They report that simple two-layer cross-encoders significantly outperform more intricate bi-encoders, while lighter cross-encoders narrow the latency difference and provide better outcomes.Source: §5 · p. 5