Source-linked AI summary
Large language models can accurately predict searcher preferences
Paul Thomas, Seth Spielman, Nick Craswell, Bhaskar Mitra
TL;DR
Relevance labels need to reflect searcher preferences, but careful first-party feedback does not scale and third-party labels can be biased. The paper uses first-party gold labels to select LLM prompts for relevance labelling and finds that LLMs can outperform studied human labellers, with practical benefits for search ranking. Prompt wording materially affects performance, and the authors identify bias, overfitting, and limited evidence beyond web-text settings as boundaries.
Problem
Third-party relevance labels may misunderstand searcher preferences, while careful first-party feedback is high quality but does not scale.
Method
The paper collects first-party gold labels and uses them to evaluate and select LLMs and prompts for large-scale relevance labelling.
Results
LLM labels performed better against first-party ground truth than the studied human-labeller populations and supported improved ranker training.
Takeaways & Limitations
LLM-based labelling was useful at Bing for greater speed, reduced cost, and substantial improvements in the running search system.
Takeaways & Limitations
The evidence comes from web-text and test-set settings, so effectiveness for private corpora and more difficult applications remains unverified.
Abstract
from arXiv · showhide
Relevance labels, which indicate whether a search result is valuable to a searcher, are key to evaluating and optimising search systems. The best way to capture the true preferences of users is to ask them for their careful feedback on which results would be useful, but this approach does not scale to produce a large number of labels. Getting relevance labels at scale is usually done with third-party labellers, who judge on behalf of the user, but there is a risk of low-quality data if the labeller doesn't understand user needs. To improve quality, one standard approach is to study real users through interviews, user studies and direct feedback, find areas where labels are systematically disagreeing with users, then educate labellers about user needs through judging guidelines, training and monitoring. This paper introduces an alternate approach for improving label quality. It takes careful feedback from real users, which by definition is the highest-quality first-party gold data that can be derived, and develops an large language model prompt that agrees with that data. We present ideas and observations from deploying language models for large-scale relevance labelling at Bing, and illustrate with data from TREC. We have found large language models can be effective, with accuracy as good as human labellers and similar capability to pick the hardest queries, best runs, and best groups. Systematic changes to the prompts make a difference in accuracy, but so too do simple paraphrases. To measure agreement with real searchers needs high-quality "gold" labels, but with these we find that models produce better labels than third-party workers, for a fraction of the cost, and these labels let us train notably better rankers.
1 LABELLING RELEVANCE
Relevance labels are essential for evaluating and training search systems, but third-party assessors can misunderstand searcher preferences in ways that more data cannot fix. The paper proposes selecting LLMs and prompts using a small sample of careful first-party feedback, and reports that LLM labels can outperform human labellers against that ground truth.
- Relevance labels support offline search evaluation and training machine-learned retrieval systems.
- Biased third-party assessors may systematically misunderstand searcher needs, so collecting more labels from the same worker pool cannot resolve the problem.
- The standard quality-improvement process studies real searchers, identifies systematic labelling mistakes, and trains workers with guidelines or examples.
- The proposed approach uses careful first-party feedback to choose an LLM prompt whose generated labels best match real searcher preferences.
- LLMs performed better against first-party ground truth than every studied human-labeller population, including more-trained and monitored raters.
2 LABELLING RELEVANCE WITH AN LLM
The study tests whether an LLM can reproduce TREC-Robust relevance labels by varying prompt templates and their instructions and context. Prompts combine task guidance with query-document information, while the authors explicitly avoid claiming a single best prompt or model.
- The experiment asks whether an LLM can reproduce the original TREC-Robust labels assigned by expert human assessors.
- TREC-Robust contains 250 topics, using title queries and, in some prompts, additional description and narrative text.
- The evaluation uses official qrels from trained assessors, treating each assessor as the best judge of the associated query and topic meaning.
- Initial experiments sample 1000 qrels for each relevance label, while later experiments use judged documents ranked 1–100 in Robust 2004 runs.
- 2.2 Prompting: Prompt variants include task instructions, an optional search-quality-rater role, and query-document context based partly on TREC assessor instructions.
- 2.2 Prompting: The authors do not claim their prompt, LLM, or format is optimal, because minor paraphrases can materially change performance.
3 EVALUATING THE LABELS
The paper evaluates machine labels against human labels at document, pairwise-preference, query, and system levels, while emphasising that human agreement alone does not establish validity. It also compares labelling processes on practical criteria such as cost and scalability.
- The primary validity criterion is agreement with gold labels, measured through labels themselves, document preferences, and aggregated query- or system-level conclusions.
- Mean absolute error measures document-label disagreement, with 0 indicating exact agreement and 1 indicating maximal difference.
- 3.2 Document preference: Pairwise accuracy or AUC measures whether the model preserves human preferences between documents, where 0.5 is chance and 1 is complete agreement.
- Pairwise preferences remain useful when scoring schemes differ in scale or location, whereas MAE can be misleading under such mappings.
- Query-level evaluation uses normalized RBO after sorting queries from lowest to highest score, giving greater weight to disagreements about the worst queries.
- Agreement with human labels is useful but cannot determine whether disagreement reflects machine error, human-label noise, or both.
- The authors report LLM labelling as superior to crowd work on cost, time, reliability, scalability, and flexibility, and superior to experts on all listed criteria except debuggability.
4 RESULTS
LLM relevance-label quality varies substantially with prompt design and paraphrasing, yet carefully selected prompts can match human labels and preserve important query, run, and group rankings.
- 4.2 Effect of prompt features: Cohen’s κ ranges from 0.20 for prompt “R---M” to 0.64 for “-DNA-”, showing that prompt structure strongly affects labelling quality.The experiments tested 32 templates formed by toggling five prompt features.
- 4.2 Effect of prompt features: Aspects improve Cohen’s κ against TREC assessors by +0.21, while narrative adds 0.06, description adds 0.01, role decreases 0.04, and multiple judges decreases 0.13.These are feature-level effects measured independently of other features, and the paper notes that interactions are not additive.
- 4.4 Effect of document selection: Choosing the best prompt on one document split remained better than the baseline on another in all 1000 iterations, with “-DNA-” selected 829 times.For paraphrases, the best first-split variant beat the initial “-DNA-” prompt on the second split in 989/1000 iterations; variant #13 was selected 838 times.
- 4.5 Query difficulty and run effectiveness: LLM labels yield rankings broadly similar to human labels, with identical top-three runs under every metric and consistent top groups under P@10.Group rankings are more consistent than run or query rankings, while query rankings are harder to stabilize because many queries have similar scores.
- 4.5 Query difficulty and run effectiveness: Run-level Kendall’s τ ranges from 0.77 for MAP@100 to 0.86 for P@10, described as plausibly as good as most human labellers.The paper compares these values with prior human-assessor agreement ranges while noting that perfect agreement is unlikely.
5 WEB SEARCH AT BING
At Bing, LLMs are used with expert human oversight to produce scalable relevance labels from real-searcher ground truth, outperforming third-party labellers in accuracy, speed, throughput, and cost.
- 5.1 Experience with LLMs: LLMs were more accurate than every third-party labeller, including staff, and were faster, higher-throughput, and many times cheaper.Bing has used LLMs with expert human labellers for most offline metrics since late 2022.
- 5.2 Ground truth at Bing: Bing prefers breadth over depth: small datasets covering many real-like queries are preferred to larger datasets covering fewer simulated queries.The preference reflects a desire for queries resembling real web-search workloads rather than invented TREC-Robust needs.
- 5.2 Ground truth at Bing: Gold labels are gathered in situ from people with the search need, often while viewing full SERPs, making their judgments highly reliable.Sources include employees, contractors, and public feedback collected at or near the time of need.
- 5.2 Ground truth at Bing: Bing’s ground-truth corpus uses searcher-tagged good, neutral, and bad results to derive preferred-versus-non-preferred pairs for binary evaluation.The corpus contains queries, need descriptions, metadata, and at least two example results per query.
- 5.2 Ground truth at Bing: Three-level labels blur small and large preference differences, but searcher-context preferences are easier to collect and remain advantageous for evaluation.The paper evaluates labels independently of third-party judgments, allowing disagreements to be resolved against searcher-generated ground truth.
- 5.3 Monitoring the LLM system: LLM labels are monitored through weekly stratified re-labeling by trained assessors and a consistently re-labeled query set for rapid detection of changes.Disagreement patterns are investigated by a metrics team, and the system combines automation with manual verification.
- 5.3 Monitoring the LLM system: LLM relevance labels do not replace web-search safety systems addressing spam, misinformation, piracy, sensitive topics, and ranking diversity.The prompts support evaluation but are only one component of a web-scale search system.
6 POTENTIAL LIMITATIONS AND PITFALLS
The paper identifies unresolved risks involving bias, overfitting, attention differences, content homogenisation, and the ecological costs of LLM-based relevance labelling.
- Bias and harm: The extent of bias in LLM relevance labels is unknown and could intensify representational and allocative harms in search.Possible biases include underestimating the relevance of longer documents, and using varied prompts or models may not remove shared biases.
- Overfitting and system effects: Optimising toward LLM labels may overfit to model idiosyncrasies rather than improve relevance, especially as models influence ranking or web authors target them.The current data places LLM labels closer to ground truth than third-party assessors, but the risk may change with broader adoption.
- Overfitting and system effects: Differences between LLM and human attention could let website designers exploit model biases or drive undesirable shifts and homogenisation of online content.The paper identifies these as open questions rather than established outcomes.
- Environmental cost: The ecological costs of LLMs remain heavily debated and require further study.The paper treats environmental impact as an unresolved limitation of the approach.
7 CONCLUDING REMARKS
The paper argues that LLMs can provide accurate, lower-cost relevance labels when prompts are selected against gold judgments from real searchers. This approach has been productive at Bing, while requiring ongoing quality measurement, audits, and careful attention to scope.
- Using model-generated labels to retrain part of Bing’s ranker produced about six months’ relevance improvement in one step.
- The approach combines gold labels from searchers with prompt evaluation and selection for LLM labelling.
- LLM labelling retains human oversight and audits of examples, but assessing quality becomes difficult when model output exceeds third-party assessor quality.
- The experiments mainly cover web text, described test sets, and one capable model, so effectiveness in private corpora, running-system queries, and new models remains uncertain.
- As models improve and evaluation measures saturate, harder gold sets with finer distinctions may be needed.
- LLMs can label documents with performance comparable to TREC judges and notably better than crowd judges.
A.1 LLM-vs-human confusion matrix
The confusion matrix shows asymmetric agreement between one LLM prompt and TREC assessors: the model more often predicts “not relevant,” while positive predictions are more reliable.
- When the LLM predicted “not relevant,” agreement with TREC was 68%.
- When the LLM predicted “relevant” or “highly relevant,” agreement with TREC was 94%.
A.2 Effect of prompt length
After controlling for prompt features and true relevance effects, prompt length showed no substantial relationship with signed error. The estimated score shifts were statistically significant but practically negligible at median prompt length.
- Controlling for prompt features, prompt length showed no substantial correlation with signed error.
- −9×10−6 to 1×10−5 score shift per prompt character was estimated across prompt features.
- −0.05 to 0.06 was the corresponding score shift at median prompt length, with no practical significance given Table 1 MAEs.