Source-linked AI summary
PeopleSearchBench: Evaluating AI-Powered People Search Platforms with Criteria-Grounded Verification
Tianyu Shi, Wei Wang, Zequn Xie, Shuai Zhang, Boyang Xia, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang, Zhenyu Yu, Lei Ding
TL;DR
People-search platforms are increasingly used, but no standardized benchmark rigorously evaluates whether they return qualified people with verifiable information. PeopleSearchBench evaluates platforms using criteria-grounded live-web verification, and finds that performance varies across scenarios while more diverse information sources are associated with greater robustness across heterogeneous tasks.
Problem
People-search evaluation lacks a shared benchmark, even though returned individuals have attributes distributed across sources that may change over time.
Source: §1 · p. 1Method
PeopleSearchBench evaluates four platforms on 119 multilingual queries using live-web checks of explicit criteria and the dimensions Relevance Precision, Effective Coverage, and Information Utility.
Source: §1 · p. 2Results
Performance varies substantially across scenarios, and systems drawing on more diverse information sources are more robust across heterogeneous people-search tasks.
Source: §1 · p. 2Takeaways & Limitations
The benchmark evaluates whether systems return relevant people, find enough qualified individuals, and provide actionable profile information.
Source: §1 · p. 2Web verification may systematically under-evaluate people with limited online presence, biasing evaluation toward publicly visible professionals.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
AI-powered people search platforms are increasingly deployed for recruiting, sales prospecting, and professional networking, yet no standardized benchmark exists for their rigorous evaluation. We present PeopleSearchBench, an open-source benchmark comprising 119 multilingual queries across four scenarios: corporate recruiting, B2B sales prospecting, expert search, and influencer discovery. A central contribution is Criteria-Grounded Verification, an evaluation methodology that decomposes each query into explicit, independently checkable criteria and verifies each returned individual via live web search, producing factual relevance judgments rather than subjective LLM-as-judge scores (Cohen's kappa = 0.84 with human annotators). We evaluate four architecturally diverse platforms along three complementary dimensions---Relevance Precision, Effective Coverage, and Information Utility---and find that multi-source search agents significantly outperform single-domain systems, particularly in influencer discovery where the performance gap is largest. Platform rankings are robust across ablations on scoring thresholds, dimension weights, and judge models. All code, queries, and evaluation prompts are publicly available.
1 Introduction
People search returns ranked sets of individuals who must satisfy multiple constraints and provide verified information for downstream decisions. The paper introduces a benchmark that evaluates people-search platforms through explicit criteria and external verification, addressing gaps in existing evaluation.
1 Introduction
- People-search results are ranked sets of individuals who must meet several constraints and provide verified information for downstream decisions.Examples include recruiting, sales prospecting, expert search, and creator discovery.Source: §1 · p. 1
- Existing retrieval evaluation focuses on documents or passages, whereas people search requires verifying real individuals’ attributes across sources that can change over time.Relevant attributes include current employer, role, location, seniority, publication record, and audience size.Source: §1 · p. 1
- A holistic LLM relevance score cannot substitute for checking whether a named person meets specific role, company, or quantitative constraints.The paper motivates decomposing queries into explicit criteria and verifying each returned person against external evidence.Source: §1 · pp. 1–2
- PeopleSearchBench provides 119 multilingual queries across recruiting, B2B sales prospecting, expert search, and influencer discovery, with criteria-based live-web verification of returned people.The benchmark also evaluates whether systems find enough qualified people and provide profiles usable for downstream action.Source: §1 · p. 2
- Criteria-Grounded Verification achieves κ = 0.84 agreement with human annotators, and the benchmark evaluates Relevance Precision, Effective Coverage, and Information Utility across four platforms.Results vary substantially by scenario, while platforms using more diverse information sources are more robust across heterogeneous tasks.Source: §1 · p. 2
2 Related Work
Prior benchmarks evaluate documents, passages, media, or single-goal agents, while people-search systems return real individuals whose attributes require independent verification. PeopleSearchBench addresses this gap by evaluating open-web people search across set relevance, factual verification, and information quality.
Information retrieval benchmarks
- Information retrieval benchmarks evaluate documents or passages as result units, unlike people search, where each result is a real individual with multiple attributes requiring independent verification.The section situates this contrast against TREC, BEIR, MTEB, and BERGEN.Source: §2 · p. 2
LLM-based and factual evaluation
- Factuality methods verify atomic claims in generated text, whereas Criteria-Grounded Verification decomposes the query into criteria and verifies real people using live web evidence.Its evidence concerns current employment, location, and role.Source: §2 · p. 2
Entity-centric and expert search
- Entity and expertise retrieval typically assumes fixed collections or closed corpora, while open-web people search identifies and verifies professional profiles across platforms.The related settings do not provide protocols for comparing autonomous people-search agents on open-web results.Source: §2 · pp. 2–3
Agentic AI evaluation
- Agent benchmarks assess single-goal completion or interaction efficiency, but do not jointly evaluate returned-person set relevance, per-result factual verification, and information quality.People search requires evaluating these aspects together.Source: §2 · p. 3
3 PEOPLESEARCHBENCH
PEOPLESEARCHBENCH evaluates people-search platforms using a 119-query multilingual benchmark spanning four scenarios. It verifies people against query criteria with live web evidence and assesses relevance, coverage, and information utility.
3.1 Query Dataset
- The benchmark comprises 119 queries across recruiting, B2B prospecting, expert search, and influencer/KOL discovery, with English, Portuguese, Spanish, and Dutch queries averaging 2.7 criteria.Influencer discovery tends to produce the largest performance differences because relevant data is scattered across social platforms.Source: §3.1 · p. 3
3.2 Criteria-Grounded Verification
- Criteria-Grounded Verification decomposes each query into checkable criteria, verifies each returned person against them using live web search, and averages the criterion scores into a relevance grade.The criteria are equally weighted; the pipeline also assesses whether profiles contain information sufficient for downstream action.Source: §3.2 · pp. 3–4
3.3 Evaluation Dimensions
- Relevance Precision uses padded nDCG@K so returning only a few perfect results does not inflate scores relative to returning more relevant people.Effective Coverage measures qualified people found per query, combining task completion with result volume.Source: §3.3 · p. 5
- Information Utility assesses whether profile data is complete and structured enough for downstream action, including criterion-specific evidence and source URLs.The three dimensions are combined by equal-weight averaging, and rankings are reported as invariant to weight perturbation in the ablation study.Source: §3.3 · p. 5
4 Experiments
Four platforms were evaluated under a common verification and statistical setup. Lessie ranked first overall and led all three dimensions, while the results showed tradeoffs among platform architectures.
4.1 Setup
- Four diverse platforms were assessed on up to 15 results per query using identical verification, January 15–22, 2025 evaluations, and bootstrap confidence intervals.The verification setup used Gemini 3 Flash Preview and Tavily Search; 95% confidence intervals used 1,000 bootstrap iterations.Source: §4.1 · pp. 5–6
4.2 Main Results
- Exa pairs 96.6±1.8% task completion with 58.1±2.6 Effective Coverage but 53.8±2.4 Relevance Precision; Claude Code has the lowest Information Utility (42.7±2.2).Juicebox has the lowest Relevance Precision (44.7±2.6), while its LinkedIn-style profile fields yield 50.9±1.9 Information Utility.Source: §4.2 · p. 6
- 65.2±1.5: Lessie ranked first overall, 18.5% ahead of Exa (55.0±1.8), and led all three dimensions; the first-versus-second differences were significant (p < 0.05).Claude Code scored 46.0±2.1 and Juicebox 45.8±1.9.Source: Figure 2 · p. 6, §4.2 · p. 6
- Lessie alone combines 100% task completion with the highest relevance; Exa trades high completion for moderate relevance, while Juicebox and Claude Code score lower on both.Figure 3 plots task completion against Relevance Precision, with bubble size indicating overall score.Source: Figure 3 · p. 6, §4.2 · p. 6
5 Conclusion
PeopleSearchBench combines Criteria-Grounded Verification with three complementary evaluation dimensions to assess four platforms across 119 queries. The evaluation finds architectural tradeoffs and stable rankings under tested design choices, and the materials are publicly released.
5 Conclusion
- PeopleSearchBench evaluates four platforms on 119 queries across four scenarios using Relevance Precision, Effective Coverage, and Information Utility.Criteria-Grounded Verification produces factual, reproducible judgments with κ = 0.84 human agreement.Source: §5 · p. 6
- The evaluation identifies architectural tradeoffs, with data-source diversity playing a key role in cross-scenario robustness.Ablations validate padded nDCG, qualified-result thresholds, and equal dimension weighting, showing stable rankings.Source: §5 · p. 6
- The authors release the code, queries, and evaluation prompts to support reproducible comparison as people-search systems evolve.Source: §5 · p. 7
Limitations
The benchmark has boundaries in query coverage, publicly indexed evidence, result depth, evaluation timing, utility scoring, and verifier diversity.
Limitations
- The 119-query set covers four scenarios but omits use cases such as academic collaborator search and angel investor identification.Source: Limitations · p. 7
- Public-web verification may systematically underrate people with limited online presence.Source: Limitations · p. 7
- Evaluation is capped at each platform’s top 15 results, so platforms returning more are assessed only on that subset.Source: Limitations · p. 7
- Results reflect one January 2025 evaluation window and may not generalize to later platform versions.Source: Limitations · p. 7
- Information Utility rewards per-result match explanations, potentially favoring platforms with built-in verification pipelines.Source: Limitations · p. 7
- The primary pipeline uses one LLM verifier, and the tested judges share similar architectural foundations despite consistent rankings across models.Source: Limitations · p. 7
Ethics Statement
The benchmark uses publicly available professional information, publishes aggregate scores rather than individual records, and recommends responsible data-handling practices.
Ethics Statement
- Queries target public professional information; private data are not scraped, and only aggregate platform scores—not personal records—are published.The authors also report informed consent and above-local-minimum-wage compensation for annotators, and encourage responsible data-handling policies.Source: Ethics Statement · p. 7
Scenario Breakdown
Performance varies substantially by scenario: recruiting is comparatively competitive, while influencer discovery shows the widest platform spread. Lessie is the only platform with consistent Relevance Precision across all four scenarios.
Recruiting
- Lessie leads Recruiting overall (68.2±2.8), while Juicebox ranks second (65.7±2.9) with the category’s highest Effective Coverage (75.3±2.7).All four platforms complete at least 90% of recruiting tasks.Source: Scenario Breakdown · p. 8, Table 4 · p. 9
B2B Prospecting
- In B2B Prospecting, Lessie’s Relevance Precision (62.8±2.9) leads Exa’s (50.0±3.2) by 12.8 points; task completion also falls for Juicebox and Claude Code.The passage attributes the wider gaps to B2B targets often lacking structured profiles in platforms’ usual sources.Source: Scenario Breakdown · p. 8, Table 4 · p. 9
Expert/Deterministic
- Lessie leads Expert/Deterministic overall (70.4±2.4) with its highest category Relevance Precision (79.0±2.3); Claude Code’s Information Utility is 38.5±3.4.Juicebox has the category’s lowest overall score (44.2±3.6) and 71.4±8.5% task completion.Source: Scenario Breakdown · p. 8, Table 4 · p. 9
Influencer/KOL
- In Influencer/KOL, Lessie’s Relevance Precision (65.2±3.1) is 2.45× Juicebox’s (26.6±4.0), and the overall-score gap reaches 31.2 points.Influencer data is dispersed across social platforms rather than concentrated in professional databases.Source: Scenario Breakdown · p. 8, Table 4 · p. 9
Cross-scenario consistency
- Lessie’s Relevance Precision ranges from 62.8–79.0 across scenarios (CV: 9.7%), while the other platforms show wider ranges and higher variation.Figure 4 visualizes the scenario-level Relevance Precision comparisons.Source: Scenario Breakdown · pp. 8–9, Figure 4 · p. 9
A Ablation Studies
Ablations show that platform rankings are stable across tested scoring parameters and aggregation schemes. Pipeline checks also find substantial human agreement, stable criteria extraction, and unchanged rankings across alternative judge models.
A Ablation Studies
- Platform rankings remain stable across tested configurations, and Lessie ranks first under all five dimension-weighting schemes.Removing Information Utility changes the ordering of Juicebox (43.3) and Claude Code (47.7), showing that profile completeness adds a distinct quality axis.Source: A Ablation Studies · p. 9
- Lessie remains first across tested nDCG cutoffs, with overall rankings stable for K ≤10; at K ≥20, scores decline sharply as most platforms return fewer than 15 results.Changing the qualified-result threshold preserves the top two rankings, and binary rather than ternary judgments lower Relevance Precision without changing rankings.Source: A Ablation Studies · p. 9, Figure 5 · p. 9
A.1 Pipeline Validation
- LLM-verifier agreement with human consensus reaches κ = 0.84 on criterion match and κ = 0.88 on qualified status.Across 26 criterion-level disagreements, 18 involved the LLM being more conservative on “partially met” judgments and 8 involved evidence humans missed.Source: §A.1 · pp. 9–10, Table 5 · p. 10
- Criteria extraction has 94.7% semantic equivalence across five runs, with 78.0% exact string match and a mean 2.73 criteria per query (SD = 0.41).The paper treats exact string match as a lower bound because paraphrasing is acceptable.Source: §A.1 · p. 10
- Alternative judges achieve κ = 0.87 for GPT-4o, κ = 0.85 for Claude 3.5 Sonnet, and κ = 0.79 for GPT-4o-mini; platform rankings remain identical.The tested judge models were compared on 200 person–query pairs.Source: §A.1 · p. 10
B Error Analysis
The error analysis identifies platform-specific failure patterns, while the appendix details ablations and evaluation costs. Web verification dominates cost and latency, though parallel execution reduces total time.
B Error Analysis
- Juicebox has the most false positives (24.6%) and, outside recruiting, false negatives and task failures concentrate there, reaching 41.4% false negatives in Influencer/KOL queries.Claude Code’s errors are often incomplete profiles (31.2%), while Lessie has 0% false-negative and task-failure rates across scenarios.Source: B Error Analysis · p. 10, Table 6 · p. 10
C Ablation Details
- The ablations cover qualified-result thresholds, padded nDCG cutoffs, weighting schemes, partial credit, and removal of Information Utility.The provided table caption states rankings are stable for K ≤15.Source: C Ablation Details · pp. 10–11, Table 8 · p. 10
- The appendix also tests binary-only judgments and scoring with only Relevance Precision and Effective Coverage, with tables reporting the partial-credit and Information Utility ablations.The supplied passages describe these test configurations but do not state their outcome values.Source: C Ablation Details · p. 11, Table 10 · p. 11, Table 11 · p. 11
D Cost and Reproducibility
- Full-benchmark web verification costs $89.40 and takes 1.8 hours; average verification cost is $0.86 per query, mostly from web search ($0.75).Criteria extraction costs $0.002 per query and LLM verification $0.11.Source: D Cost and Reproducibility · p. 11, Table 12 · p. 11
- Eight-way concurrency reduces evaluation time from 4.9 hours to approximately 1.2 hours because verification searches can be parallelized.Web verification dominates latency because each criterion requires an independent search.Source: D Cost and Reproducibility · p. 11, Table 13 · p. 11
E Error Analysis Details
The error breakdown highlights profile and coverage mismatches, with influencer discovery the most challenging scenario. Case studies illustrate keyword-driven false positives, limited single-pass recall, and handling of ambiguous evidence.
E Error Analysis Details
- In Recruiting, Juicebox has the lowest false-positive rate (6.2%), while Claude Code’s errors are dominated by incomplete profiles (38.5%).In B2B Prospecting, Juicebox’s task-failure rate is 15.6%, and Exa has 22.1% false positives when titles are ambiguous.Source: E Error Analysis Details · p. 11
- In Expert/Deterministic search, Claude Code has the lowest error rate (12.5%), while Juicebox has 28.6% task failure when targets lack LinkedIn profiles.Source: E Error Analysis Details · p. 12
- Influencer/KOL has the highest error rates across platforms; Juicebox’s false-negative rate is 41.4%, while Lessie has the lowest error rate at 18.5%.The passage attributes Juicebox’s misses to influencers rarely maintaining professional profiles and Lessie’s lower rate to multi-source coverage.Source: E Error Analysis Details · pp. 11–12
F Case Studies
- A Juicebox result matched “product manager,” “Singapore,” and “finance” but missed the query’s “fintech startup” constraint, exemplifying keyword matching without semantic constraint satisfaction.The passage describes this as a common failure for database-focused platforms relying on keyword matching over semantic understanding.Source: F Case Studies · p. 12
- For a NeurIPS diffusion-model query, Claude Code returned 3 correct names but missed 12 additional valid researchers found by Lessie and Exa; the case illustrates single-pass search’s coverage limitation.General-purpose agents may stop after initial results rather than continue searching exhaustively.Source: F Case Studies · p. 12
- When web sources conflicted about Anthropic co-founders, the verifier marked disputed claims “partially met” rather than forcing binary judgments.Source: F Case Studies · p. 12