Source-linked AI summary
No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation
Dimitri Staufer, David Hartmann, Ibrahim Baroud
TL;DR
LLM evaluations use person names, but uncontrolled evidential status can confound memorisation, retrieval, name priors, and wrong-person attribution. The paper introduces PUN, which constructs and validates plausible operationally unknown names through Wikidata-derived generation, web-enabled LLM screening, and controlled search. It reports 300 released names with controls, while the supplied evidence also documents validation-snapshot and naming-scope limitations.
Problem
Uncontrolled evidential status in person-name prompts can conflate memorisation, retrieval, name priors, and wrong-person attribution in LLM evaluations.
Method
PUN generates plausible First Last strings from Wikidata-derived components, screens them with a web-enabled LLM, and validates them using controlled search.
Results
300 accepted names are released with comparison controls, validation traces, reproducibility diagnostics, protocol ablations, and a 204-participant human agreement audit.
Takeaways & Limitations
PUN treats person-name evidential status as an experimental variable and supplies a baseline for evaluating prompts that retain person-name form without validated public full-name evidence.
Takeaways & Limitations
Operational unknownness is a validation snapshot dependent on provider, configuration, locale, and time, and the resource is restricted to Latin-script two-component First Last strings.
Abstract
from arXiv · showhide
Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential status is uncontrolled, measurements may conflate memorisation, retrieval, name priors and wrong-person attribution. We operationalise an unknown name as one with plausible First-Last form, no indexed full-name evidence, and no ambiguity signals under a documented validation run, and introduce PUN (Plausible Unknown Names), a protocol for constructing and validating such names, combining Wikidata-derived components, web-enabled LLM screening, and controlled search revalidation. We report acceptance rate, reproducibility, ablations, and a 204-participant human study, finding accepted names are more name-like than controls while participants recover person evidence in only 3% of cases. We release 300 names with comparison controls.
1 Introduction
Person names can carry public evidence, ambiguity, variant forms, and social associations, so uncontrolled name prompts make LLM evaluation results difficult to interpret. The paper introduces PUN to construct plausible names with controlled full-name evidential status and releases a validated resource.
- Names serve as inputs or targets across biography, privacy, hiring, anonymisation, bias, and abstention evaluations.
- A full-name string can identify public evidence, multiple namesakes, nearby variants, and cultural or demographic associations simultaneously.
- Uncontrolled evidential status can make model behaviour reflect retrieval, memorisation, social priors, wrong-person conflation, or unsupported generation.
- PUN defines plausible unknown names as attested two-component First Last strings with no exact-name evidence and no ambiguity signals under documented validation.
- The paper contributes an auditable protocol, diagnostics, and a showcase of 300 accepted names with comparison controls and a 204-participant human audit.
2 Operational Unknownness
The paper defines operational unknownness as the evidential status of a plausible full-name string under a specified validation run. Acceptance requires plausible form, no indexed public person evidence, and no full-name-level ambiguity signal.
- Operational unknownness refers to a plausible full-name string’s binary evidential status under a specified validation protocol run.The run fixes retrieval systems, settings, query forms, and decision rules for searching indexed public evidence.
- A candidate is accepted only if it is a plausible two-component First Last string with no indexed public evidence linking the exact name or configured equivalent to a person.
- Acceptance also requires no full-name-level ambiguity signal, including nearby spellings, reordered forms, search corrections, famous-name attractors, or unclear non-person referents.The criterion can reject names despite no exact-name trace when configured variants or nearby entities create ambiguity.
3 Background and Related Work
The paper situates PUN in research using person names as evaluation inputs while emphasizing that names vary in structure, meaning, and evidential reliability. A full name may identify multiple people, connect to variants, or carry demographic signals, and absent search evidence is not absolute.
- PUN’s narrow plausibility criterion covers attested components, First Last form, and explicit checks rather than sociolinguistic, legal, demographic, or community-specific plausibility.
- A full name can identify multiple people or one person across spellings, scripts, transliterations, aliases, abbreviations, and reordered forms.Name-disambiguation systems therefore reconcile variants while separating namesakes.
- Person names are used in LLM evaluations of biography, factuality, privacy, memorisation, anonymisation, hiring, bias, social inference, abstention, and unanswerability.
- Nonword or malformed probes avoid public-person evidence by moving away from plausible names, whereas PUN retains plausible name form while targeting operationally unknown strings.
4 The PUN Protocol
PUN generates plausible First Last candidates, applies local format checks, and uses web-facing screening and controlled searches to assign an auditable evidential verdict. It also provides comparison controls spanning public-person prominence and increasing surface distance.
- The PUN Protocol: PUN has three stages: candidate generation, local format checks, and web-facing validation.Candidates are built from attested Wikidata name components, filtered locally, then evaluated through web-facing procedures.
- Candidate generation: Candidate generation combines given-name and surname components from Wikidata human-entity name forms into First Last strings.A source-country key supports sampling and audit only, not demographic interpretation.
- Local checks: Local filters require exactly two Latin-script components, retain diacritics, bound total length, exclude overly common components, and check component frequency and duplicates.
- Web-facing validation: The web-enabled LLM screens candidates with “Who is {name}?” and labels responses found, ambiguous, fail, or no_info; only no_info candidates proceed to controlled search.The LLM can surface nearby referents but is not the evidential basis for acceptance because its retrieval and ranking are not fully observable.
- Web-facing validation: Controlled search rejects candidates when exact or configured forms appear in person-like contexts or when variants and corrections point to nearby full-name forms.Acceptance is based on fixed search evidence and decision rules rather than the LLM screening result alone.
- Comparison controls: Comparison names cover public-person prominence, literal full-name widespreadness, and twelve increasingly distant perturbation layers from orthographic edits to recombinations.Public-person controls use page views, article length, language count, sitelinks, and quoted exact-name search counts as proxies.
5 The PUN Resource
The showcase resource contains 300 accepted operationally unknown names with comparison and famous-name-distance controls. Its validation pipeline shows that most generated candidates are rejected for ambiguity or failed screening before controlled search leaves the accepted set.
- Resource: The resource releases 300 accepted names, 300 public-person comparison names, and 3,600 famous-name distance controls.The showcase set comes from one documented PUN run and is not a dataset of permanently unknown people.
- Validation yield: 52,726 generated candidates yielded 300 accepted names after local checks, web-enabled LLM screening, and controlled search.Only 1,781 candidates received no_info and proceeded to controlled search.
- Validation yield: 36,097 candidates were linked to nearby names or other referents, 2,449 to exact-person evidence, and 9,550 produced failed outputs.The web-enabled LLM screen therefore filtered candidates primarily through ambiguity, with additional exact-person and failed responses.
- Validation yield: Controlled search found 108 exact-person traces and 1,372 ambiguity signals among no_info candidates, leaving 300 accepted names.Accepted names represented 0.6% of generated candidates, or roughly one name per 176 candidates.
- Validation yield: Ambiguity motifs included similar names, partial-token matches, reordered forms, and nearby public referents, while failed responses included policy-like refusals.These patterns motivate the controlled-search ablations reported in Section 7.2.
6 Experimental Setup
The experimental setup measures validation reproducibility, controlled-search sensitivity, name-likeness, and human agreement. It combines repeated reruns and validator ablations with character-level and tokenizer-based diagnostics and a 204-participant study.
- Protocol reproducibility: Three repeated reruns on pre-specified cohorts estimate stochastic variation using fixed model versions, prompts, hyperparameters, and database snapshots.Agreement is reported for the labels found, ambiguous, no_info, and fail.
- Validator ablations: The controlled-search ablation uses 1,781 no_info candidates and removes one search guardrail at a time to measure additional acceptances.Generation-policy ablations are reported separately in Appendix A.10.
- Name-likeness diagnostics: Character-level name-likeness is measured with an LSTM trained on ParaNames, a multilingual corpus of 140 million names across 400+ languages.The diagnostic uses bits per character, where lower BPC indicates greater predictability under the name model.
- Name-likeness diagnostics: Tokenizer fragmentation is computed as |T(n)|/|n| and summarized by the median across 14 tokenizers.The setup uses tokenizer segmentation as a diagnostic for the relative rarity of name forms.
- Human agreement: The human agreement study included 204 participants from 27 self-reported language backgrounds and 31 countries of residence.Participants judged name plausibility and inspected evidence through prepared search queries and web-enabled chatbots.
7 Results
The validation and human audit support PUN’s operational distinctions, while reproducibility tests show instability is concentrated in web-enabled screening and ambiguity boundaries. Accepted names are name-like but have little recoverable person evidence.
- 7.1 Protocol Reproducibility: 75.0% end-to-end agreement contrasted with 98.2% classifier-only agreement, indicating rerun instability mainly comes from web-enabled model search context.Between-run standard deviation was 0.1 percentage points.
- 7.1 Protocol Reproducibility: 31.6% end-to-end agreement for no_info contrasted with 92.4% classifier-only agreement, with most switches becoming ambiguous rather than found.The no_info label therefore functions as a triage signal before controlled search; 57.7% switched to ambiguous and 1.4% to found.
- 7.2 Protocol Ablations: 300 of 1,781 web-LLM no_info candidates survived controlled search; removing variant-search guardrails would have accepted 942 instead.Exact quoted search alone would accept 1,115 names, or 815 additional names beyond the full validator.
- 7.3 Name-Likeness: Accepted PUN names had median BPC 3.16, more name-like than perturbation and non-name controls but less predictable than public-person names at 2.45.Lower BPC indicates greater predictability under the ParaNames-trained LSTM name model.
- 7.3 Name-Likeness: Accepted PUN names had median tokenizer fragmentation 0.36 pieces per character, close to public-person names and word-like non-names at 0.33.Fragmentation correlated with LSTM BPC across scored rows (Spearman ρ = 0.63) but weakly with exact-name search prevalence (Spearman ρ = −0.16).
- 7.4 Human Agreement: Participants judged released names plausible in 63.0% of ratings and recovered person evidence in 3.0% of controlled-search judgments.The stable_no_info_10 subset received 84.0% no_info and 2.5% found judgments; chatbot searches produced 2.0% found labels overall.
8 Discussion
PUN provides operationally unknown names as controlled evaluation inputs, while enabling follow-on studies of model behavior under reduced full-name evidence confounds.
- PUN is a measurement device rather than a catalogue of non-existent people.
- Accepted names remain name-like, while participants recovered person evidence in 97% of cases only after failing to do so.
- Accepted names can serve as negative controls in privacy audits because extracting claims for them may not measure leakage alone.
- The resource supports studies of synthetic facts, name bias, and machine unlearning with known inserted records or deletion targets.
9 Conclusion
The paper introduces PUN to construct plausible person-name prompts that are operationally unknown under a documented validation run. It argues that evidential status is an experimental variable and provides a baseline for evaluating LLM behavior without validated public full-name evidence.
- PUN constructs plausible person-name prompts that are operationally unknown under a documented validation run.
- Person names should not be treated as interchangeable prompt strings because their evidential status is an experimental variable.
- PUN supplies a missing baseline for measuring LLM behavior when prompts have person-name form but lack validated public full-name evidence.
- Future work should extend PUN beyond Latin-script two-component names and quantify provider and locale sensitivity.
Limitations
The paper’s operational unknownness is a validation snapshot whose scope depends on providers, configurations, locales, time, and a restricted naming format. PUN controls full-name evidence and ambiguity but does not remove social meaning or establish non-existence.
- Operational unknownness depends on provider, configuration, locale, and time, so released verdicts are documented validation snapshots rather than permanent properties.
- The resource is restricted to Latin-script two-component First Last strings and excludes many other naming practices.
- The 300-name set is a small showcase resource rather than a comprehensive catalogue, and users should revalidate names before high-stakes or time-sensitive use.
- PUN does not remove associations with language, region, gender, religion, class, ethnicity, or other perceived attributes.
Ethical Considerations
The resource is intended for aggregate evaluation of LLM behavior, with safeguards against overclaiming, misuse, and unnecessary disclosure of personal information. The authors also describe participant-data handling and limited use of LLM tools during preparation.
- Accepted names may coincide with real non-indexed individuals, so they must not be used to claim non-existence, fabricate biographies, or impersonate people.
- Public-person comparison controls should be used only as aggregate controls, not for individual-level claims about named persons.
- The intended use is aggregate evaluation of factuality, abstention, privacy-related behavior, and name-conditioned conflation.
- The protocol minimizes disclosure by releasing hashes, labels, query metadata, and summary evidence instead of unnecessary personal snippets.
- The human audit collected task responses and platform metadata for aggregate reporting, while no participant-level identifying information was released.
- LLM tools supported code completion, phrasing, visualization, and formatting, while authors retained algorithmic, analytical, and substantive-writing responsibility.
A Appendix
The appendix documents PUN’s construction, validation, diagnostics, release revalidation, and human-study analysis. It also records coverage and source-pool skews that bound interpretation of the released names.
- Coverage and source-pool composition: 40,715 eligible component slots remain across 26 source-country keys after filtering, with U.S., French, German, Italian, and Russian pools accounting for 70.5% combined.Before filtering, the mapped source pool contains 4,167,732 humans across 667 keys, and the top ten keys account for 56.3%.
- Candidate generation and local validation: The pipeline combines Wikidata-derived name components into two-token First Last candidates, then applies local validation before web-facing checks.Local checks reject invalid token counts, unsupported scripts, malformed punctuation, duplicates, titles, suffixes, excessive length, and components failing frequency or provenance constraints.
- Validation and release revalidation: PUN’s documented validation run fixes retrieval systems, settings, query forms, and decision rules, while release revalidation must distinguish independent evidence from self-indexing.The release timestamp should be recorded, and self-indexing traces ignored only when clearly attributable to the dataset release.
- Coverage and source-pool composition: The retained Wikidata labels are predominantly English-priority or multilingual, while script and country-key counts are extraction diagnostics rather than demographic estimates.English-priority labels comprise 86.2% and multilingual labels 7.8%; the appendix explicitly limits interpretation of these categories.
- Name-likeness diagnostics: The LSTM audit measures string-form plausibility rather than operational unknownness: lower bits per character indicate higher predictability under a character-level name model.Accepted PUN names remain closer to public-person names than non-name controls, but web-facing validation supplies the evidence for unknownness.
- Name-likeness diagnostics: Tokenizer fragmentation broadly agrees with the LSTM diagnostic, while its weak correlation with exact-name prevalence indicates that it is not a proxy for indexed full-name evidence.Accepted PUN names have low median fragmentation, similar to public-person names and word-like non-names, and much lower than random strings.
- Human agreement study: The human analysis uses 204 selected participants with complete packet coverage, including 1,224 plausibility judgments and 1,224 controlled-search judgments.The selected submissions have a median completion time of 8.2 minutes.