Source-linked AI summary
Evaluating AI Recruitment Sourcing Tools by Human Preference
Vladimir Slaykovskiy, Maksim Zvegintsev, Yury Sakhonchyk, Hrachik Ajamian
TL;DR
AI sourcing tools are evaluated against traditional recruitment platforms because comprehensive systematic comparisons have been lacking. The study benchmarks search results using human and LLM-judge assessments and Elo ratings. It reports that AI-based tools outperform traditional platforms in candidate relevance, with Pearch.ai scoring highest and LLM-judge evaluations aligning with expert judgments.
Problem
Comprehensive systematic comparisons of AI sourcing tools with established platforms are lacking, leaving their relative performance uncertain.
Source: §1 · p. 1Method
The study compares sourcing-tool search results using human expert and LLM-judge assessments, then applies Elo ratings to measure relative effectiveness.
Source: §1 · p. 1, §6.2 · p. 7Results
AI-based sourcing tools outperform traditional recruitment platforms in candidate relevance, Pearch.ai achieved the highest performance scores, and LLM-judge evaluations aligned strongly with expert judgments.
Source: §7 · pp. 9–10Takeaways & Limitations
LLM-judge evaluations may support more scalable benchmarking by reducing the cost, time, and subjectivity associated with human annotation.
Source: §7 · p. 10Subscription pricing was only a rough proxy for cost, and the study identifies dollar cost per relevant candidate as a needed measure of compute efficiency.
Source: §7 · p. 10
Abstract
from arXiv · showhide
This study introduces a benchmarking methodology designed to evaluate the performance of AI-driven recruitment sourcing tools. We created and utilized a dataset to perform a comparative analysis of search results generated by leading AI-based solutions, LinkedIn Recruiter, and our proprietary system, Pearch.ai. Human experts assessed the relevance of the returned candidates, and an Elo rating system was applied to quantitatively measure each tool's comparative performance. Our findings indicate that AI-driven recruitment sourcing tools consistently outperform LinkedIn Recruiter in candidate relevance, with Pearch.ai achieving the highest performance scores. Furthermore, we found a strong alignment between AI-based evaluations and human judgments, highlighting the potential for advanced AI technologies to substantially enhance talent acquisition effectiveness. Code and supporting data are publicly available at https://github.com/vslaykovsky/ai-sourcing-benchmark
1 Introduction
Traditional recruitment search can be time-consuming and relies on Boolean queries and filters that may miss nuanced candidate attributes. This study benchmarks whether AI-based sourcing tools improve candidate relevance compared with LinkedIn Recruiter.
1 Introduction
- Traditional search relies on skilled use of Boolean logic, keywords, and filters, making searches iterative and nuanced attributes harder to capture.The example given is identifying candidates who have led sales teams of ten or more people.Source: §1 · p. 1
- RAG-based tools interpret job descriptions and profiles semantically, often producing more relevant results that are easier to refine without elaborate Boolean queries.Source: §1 · p. 1
- The study addresses the lack of comprehensive systematic comparisons by benchmarking AI sourcing tools against LinkedIn Recruiter using expert evaluations, LLM-judge assessments, and Elo ratings.Source: §1 · p. 1
2 Selection of Sourcing Tools
The study selected four sourcing search engines for comparison: LinkedIn Recruiter, JuiceBox, Exa.ai, and Pearch.ai. The authors describe this selection as representative but not exhaustive.
2 Selection of Sourcing Tools
- Four tools were selected for their specialization as sourcing search engines: LinkedIn Recruiter, JuiceBox (PeopleGPT), Exa.ai, and Pearch.ai.Source: §2 · p. 2
- LinkedIn Recruiter was described as the industry leader with over 1 billion users; JuiceBox was described as fast-growing, and Exa.ai as having raised $22 million.Source: §2 · p. 2
- Pearch.ai was the proprietary system, developed through two years of active R&D, and the authors acknowledge that the tool selection was not exhaustive.Source: §2 · p. 2
3 Selection of Search Queries
The benchmark built a diverse query set from anonymized Pearch.ai user traffic, reducing redundant reformulations through semantic clustering. Query selection also accounted for Pearch.ai’s training data and U.S.-based profile index.
3 Selection of Search Queries
- Text embeddings and k-means clustering grouped semantically similar anonymized Pearch.ai user queries, from which representative queries near cluster centroids were selected.The procedure filtered out frequently generated reformulations to reduce redundancy.Source: §3 · p. 2
- Queries were excluded from Pearch.ai training data, and some were amended with a U.S.-only constraint because its index primarily contained U.S.-based profiles.Source: §3 · p. 2
Example Search Queries
- Examples span backend engineers, founders who sold companies or completed an IPO, and customer-success managers with specified location, company-stage, and industry experience.Source: Example Search Queries · p. 2
4 Collection of Search Results
Searches were run through each tool’s web interface under a common collection procedure, with LinkedIn searches sometimes broadened when they returned no results. The study collected 1,735 results and summarized query and result volumes by tool.
4 Collection of Search Results
- The web UI was used across tools to keep conditions consistent and address missing APIs; LinkedIn searches with no initial results could be enhanced up to three times.The enhancement progressively relaxed criteria, including keyword, skill, title, geographic, and experience requirements.Source: §4 · pp. 2–3
- Outputs were reduced to LinkedIn profile links to standardize subsequent evaluations.Source: §4 · p. 3
- 1,735 search results were collected; some systems returned fewer than ten results for particular queries, and some LinkedIn links could not be opened.Source: §4 · p. 3
- Table 1 summarizes the number of search queries and result volumes for each sourcing tool.Source: Table 1 · p. 3
5 Comparing Search Results
This section describes paired candidate comparisons evaluated by expert recruiters and an LLM judge, then reports a strong positive relationship between LLM-judge alignment and human consensus.
5 Comparing Search Results
- A uniform random sample of 1,000 records captured query, rank, and paired source/document IDs; evaluators compared profiles at the same query and result position across systems.Table 2 shows a sample of the produced dataset.Source: §5 · p. 3, Table 2 · p. 3
5.1 Evaluation with LLM-judge
- The LLM-judge compared query/profile pairs using unchanged queries and LinkedIn-derived profile JSON enriched with Crunchbase company details, returning a binary choice and rationale.The prompt asked it to consider query requirements and whether candidates were overqualified or underqualified.Source: §5.1 · p. 4
5.2 Panel of Human Experts
- Eight Upwork recruiters were selected using experience and performance criteria; a concealed test batch was checked against majority vote, and the top half by majority alignment formed “Top Humans.”The recruiters represented junior through executive roles.Source: §5.2 · pp. 4–5
5.3 Correlation between Human Experts and LLM-judge
- Pearson correlation was 0.82 (p-value 0.0137) between recruiters' LLM-judge alignment and majority-vote alignment, supporting LLM-judge as an automated evaluator.The fitted relationship was strongly positive, and the chart shows a 95% confidence interval around the regression line.Source: Figure 1 · p. 6, §5.3 · p. 6
6 Ranking Sourcing Tools
The section turns pairwise preferences into global rankings using win-rate estimates and Elo. Across the evaluated settings, AI-native tools outrank LinkedIn Recruiter, with Pearch.ai receiving the highest Elo score.
6.1 Win-rate Matrix Estimation
- Pairwise win rates estimate the probability that an evaluator prefers the second tool; importance-weighted averages yield unbiased estimates that converge with sufficient sampling.The sampling probability for each tool pair is used in the estimator.Source: §6.1 · pp. 6–7
- Each tool’s global score averages its pairwise win probabilities, complementing the estimated rate when the tool order is reversed.This score reflects how frequently a tool wins against the others.Source: §6.1 · p. 7
- Human- and LLM-judged empirical win-rate matrices produced the same tool ordering, aligning LLM preference patterns with expert judgments.Figures 2 and 3 show the matrices based on human annotations and LLM-judge preferences, respectively.Source: §6.1 · p. 7, Figure 2 · p. 8, Figure 3 · p. 8
- Because pairwise win rates can cycle, the study uses Elo to produce a consistent, transitive global ranking.The described cycle is A outperforming B, B outperforming C, and C outperforming A.Source: §6.1 · p. 7
6.2 Elo Rating System
- Elo treats each comparison as a match, estimates expected wins from rating differences, and updates ratings after observed outcomes.The implementation uses K = 40 to control rating-update sensitivity.Source: §6.2 · pp. 7, 9
- Averaging 30 Elo calculations with randomized comparison orderings produced stable estimates, and the resulting expected win-rate matrix closely mirrored the empirical matrices.The close match suggests the Elo model fits the underlying data well.Source: §6.2 · p. 9
- AI-native tools outranked LinkedIn Recruiter in all settings, and Pearch.ai earned the highest Elo score.Figure 5 presents the final Elo-score rankings.Source: §6.2 · p. 9, Figure 5 · p. 9
7 Conclusion
The study reports that AI sourcing tools outperform traditional platforms on candidate relevance and query alignment, with Pearch.ai achieving the highest scores. It also presents LLM-judge as a human-aligned evaluator while identifying precision, cost-efficiency, and benchmark breadth as areas needing further evaluation.
7 Conclusion
- AI-based sourcing tools outperform traditional platforms in candidate relevance and query alignment, and Pearch.ai achieves the highest performance scores.Source: §7 · pp. 9–10
- LLM-judge evaluations align strongly with expert judgments and can reduce annotation cost, time, and subjectivity, enabling more scalable and continuous evaluations.Source: §7 · p. 10
- The evaluation primarily measures precision, so future studies should also measure recall across the full result set.Source: §7 · p. 10
- Subscription pricing is only a rough cost proxy; the authors call for compute-efficiency measures such as dollars per relevant candidate.Source: §7 · p. 10
- The benchmark covered a limited number of tools and queries, motivating broader systems, datasets, job roles, and query formulations.Source: §7 · p. 10
Disclosure
The authors disclose their affiliation with Pearch.ai, a tool evaluated in the study, and note steps taken to support objectivity and transparency.
Disclosure
- The authors are affiliated with evaluated system Pearch.ai; they note third-party expert annotations and publicly available procedures, data sources, and code as transparency measures.Source: Disclosure · p. 10