PaperlayerView exact revision on arXiv

Source-linked AI summary

The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion

Chenglong Ma, Xinye Wanyan, Danula Hettiachchi, Ziqi Xu, Jeffrey Chan

arXiv:2608.25245v1cs.IRcs.CL

TL;DR

LLM-generated initial queries may contain answer-side concepts unavailable to pre-search users, while existing metrics cannot separate this intrusion from legitimate human-tail variation. The paper introduces concept provenance to trace concept origins across 77,004 queries and finds localized retrieval effects but limited aggregate predictive power, making it a boundary-compliance diagnostic.

  • Problem

    Existing overlap, diversity, and effectiveness metrics cannot distinguish human-tail variation from candidate answer-side intrusion in LLM-generated initial queries.

  • Method

    Concept provenance assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones using dual extraction and validation pipelines.

  • Results

    Candidate answer-side concepts have disproportionate localized retrieval effects, with deletion d = -0.47 versus random deletion d = -0.34, yet explain less than 2% of aggregate metric variance.

  • Takeaways & Limitations

    Concept provenance is a boundary-compliance diagnostic rather than a predictor of aggregate evaluation shifts, and constrained post-generation selection achieves 99% elimination under tested conditions.

  • Takeaways & Limitations

    The provenance zones are relative to the observed UQV100 human-query distribution, so additional human sampling could reclassify some Z3auto concepts as human-tail.

Abstract

from arXiv · show

LLM-generated search queries are widely used to augment IR evaluation, yet they may contain concepts that presuppose answer-side document knowledge, violating the information-access boundary of pre-search users. Existing validation metrics, including overlap, diversity, and effectiveness, cannot distinguish rare human-tail variation from candidate answer-side intrusion. We introduce concept provenance, a framework that assigns query concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones, operationalizing a boundary that retrieval metrics alone cannot detect. Applying concept provenance to 77,004 queries across 100 UQV100 topics, 8 LLMs, and 5 prompt conditions with two extraction pipelines, we obtain a cross-pipeline token-HCIR Spearman rho of 1.0 over five condition means. Candidate answer-side concepts constitute 7.40 percent of non-generic concepts and appear in 97 of 100 topics, with topic explaining approximately 67 percent of variance. Human validation yields 68.2 percent relaxed precision, revealing two mechanisms: knowledge intrusion at 45.5 percent and deployment intrusion at 45.0 percent. Diagnostic probes show disproportionate localized retrieval effects, with deletion effect size d = -0.47 compared with d = -0.34 for random deletion, but these concepts explain less than 2 percent of aggregate evaluation variance. Concept provenance therefore serves as a boundary-compliance diagnostic rather than an evaluation-shift predictor. Under the tested conditions, no prompt condition eliminates intrusion; post-generation concept-provenance selection achieves 99 percent elimination.

1 Introduction

LLM-generated initial queries can violate the pre-search information-access boundary by using candidate answer-side concepts. Concept provenance distinguishes this intrusion from legitimate human-tail variation and frames prompt mitigation and validation as research questions.

  • Initial query formulation is bounded by the user's backstory and prior knowledge, excluding search results and candidate documents.
  • “NHS guidance prion disease” illustrates candidate answer-side concepts absent from the backstory and observed human queries but salient in relevant documents.
  • Existing overlap, diversity, and retrieval-effectiveness metrics cannot distinguish rare human-tail concepts from answer-side intrusion.
  • Concept provenance assigns concepts to backstory-supported, human-central, human-tail, and candidate answer-side zones using a priority-ordered protocol with automatic and manual validation.The framework is applied to 77,004 queries from eight LLMs under five prompt conditions.
  • The study asks how provenance zones are distributed and operationalized, whether intrusion changes retrieval outcomes, and whether prompting alone ensures boundary compliance.

2 Related Work

Prior work validates synthetic queries through diversity, overlap, effectiveness, and broader evaluation alignment, but does not identify where individual concepts originate. This paper adds provenance-based validation to separate human-tail variation from candidate answer-side intrusion.

  • Human query variation can substantially change retrieved document sets and system rankings for the same information need.
  • LLMs generate diverse query variants with substantial human-query pool overlap but systematic linguistic and retrieval differences.
  • Existing metrics treat rare human-tail concepts and document-salient candidate answer-side concepts as indistinguishable.
  • Synthetic queries, documents, and relevance labels can shift evaluation outcomes and system-ranking correlations relative to human-based benchmarks.
  • The paper isolates candidate answer-side concept intrusion as a query-side mechanism underlying distributional misalignment.

3 Concept-Provenance Framework

The framework treats concepts as normalized semantic units and assigns them to topic-specific provenance zones relative to a strict pre-search boundary. HCIR measures the share of non-generic query concepts assigned to the candidate answer-side zone, with validation and sensitivity controls.

  • 3.1 Concepts and Provenance Zones: A concept is a normalized semantic unit extracted comparably from queries, backstories, or documents.
  • 3.1 Concepts and Provenance Zones: Zone assignment follows Z0 > Z1 > Z2 > G > Z3auto, giving observed backstory or human evidence priority over generic heuristics.
  • 3.1 Concepts and Provenance Zones: Z3auto provides scalable candidate-intrusion detection, while manual validation produces Z3val to assess false positives from matching and extraction errors.
  • 3.2 Intrusion Measurement: HCIR is the proportion of a query's non-generic concepts that fall in Z3auto, including unassigned concepts in the denominator.
  • 3.2 Intrusion Measurement: Worker-level prevalence uses a 10% threshold for Z1 versus Z2, while strict Z3auto absence requires zero workers using the concept.
  • 3.2 Intrusion Measurement: The mad-cow example contrasts rare human-attested Z2 concepts with Z3auto concepts absent from the backstory and human queries but salient in relevant documents.
  • 3.2 Intrusion Measurement: Exact lemmatized matching improves precision at the cost of recall, motivating fuzzy and embedding-based sensitivity checks.
  • 3.2 Intrusion Measurement: Because human absence defines Z3auto, empirical claims concern rates, condition and topic variation, and retrieval effects rather than human-versus-LLM superiority.

4 Methodology

The study evaluates concept provenance on UQV100 human-query data and LLM-generated variants spanning multiple models and prompt conditions. Two extraction pipelines and shared filtering support reproducible cross-pipeline analysis.

  • All thresholds were preregistered, and sensitivity analysis systematically varies each parameter.
  • UQV100 contains 100 TREC Web Track topics with backstories and approximately 50 crowd-sourced initial queries per topic.
  • LLM queries use five conditions: three backstory-only prompts and two oracle controls exposing relevant-document evidence.
  • Eight LLMs generated 20 candidate queries per topic-condition-model cell, yielding 77,004 analyzed queries after deduplication retries.
  • Two complementary extraction pipelines process queries, backstories, and relevant documents under shared lowercase, lemmatization, and punctuation-removal normalization.
  • A shared generic filter excludes concepts using task lists, corpus IDF, or general-language frequency before the Z3auto path.
  • Token-level HCIR is used for cross-pipeline comparison, while concept-level HCIR supports within-pipeline analyses.

4.4 Zone Assignment

Zone assignment operationalizes concept provenance through fixed thresholds, document salience, human annotation, and retrieval-system evaluation. The protocol uses two extraction pipelines and prespecified statistical controls to assess robustness.

  • Zone Assignment: Labels follow a priority order with prespecified thresholds for human centrality, answer-side absence, document salience, and generic-term exclusion.Sensitivity variants alter prevalence, absence, TF-IDF cutoff, IDF, and matching criteria.
  • Zone Assignment: Document-salient concepts require top-k TF-IDF ranking across at least two relevant documents, with k=200 as the primary threshold.Specific, moderate, and generic IDF tiers determine whether concepts can reach the candidate answer-side zone.
  • Zone Assignment: Human validation uses three annotators who classify concept origins from only the backstory, query, and concept, with majority vote and adjudication.Reported precision distinguishes strict requires-research labels from relaxed labels including personal experience.
  • Zone Assignment: Retrieval evaluation uses BM25, BM25+RM3, query likelihood, and a cross-encoder, emphasizing pool overlap, judged ratio, and ranking correlation under incomplete qrels.Reported effectiveness metrics include nDCG@10, Recall@1K, bpref, and RBP.
  • Zone Assignment: Topic-level averages support paired bootstrap tests and mixed-effects regressions controlling for condition, query length, topic, and model effects.Holm correction and Cohen’s d quantify corrected comparisons and effect sizes.

5 Experiment 1: Concept Provenance

Across UQV100, candidate answer-side concepts are common enough to appear across nearly all topics, vary by prompt condition, and remain stable across extraction pipelines and thresholds. Human validation and sensitivity analyses support the distinction while showing that matching choices affect prevalence.

  • 5.1 Findings: 7.40% of non-generic query concepts are Z3auto, occurring in 97 of 100 topics across 51,363 query–concept pairs.The multi-signal filter excludes non-discriminative terms before candidate answer-side classification.
  • 5.1 Findings: 5.60% < 6.28% < 6.84% < 8.02% < 10.16% gives the Z3auto condition gradient from high-knowledge through boundary, generic, oracle-text, and oracle-terms prompts.Relative to generic, high-knowledge has d=−0.12 and boundary d=−0.04 for concept-HCIR.
  • 5.1 Findings: Topic explains approximately 41% of query-level concept-HCIR variance, while adding model as a fixed effect contributes ΔR2 < 0.001.The topic random-intercept model reports ICC = 0.41.
  • 5.1 Findings: High-knowledge prompts lower concept-HCIR but raise HCI-Presence to 41.5% versus 33.5% for generic prompts.The difference reflects denominator dilution from longer high-knowledge queries, averaging 17.9 versus 12.2 tokens.
  • 5.1 Findings: Condition effects survive length controls, with boundary β=−0.006 and high-knowledge β=−0.010 versus generic, while model adds ΔR2 < 0.001.The length coefficient is β=−0.0005, consistent with dilution rather than positive confounding.
  • 5.2 Robustness: Token-level HCIR shows perfect condition-ranking agreement across pipelines, Spearman ρ=1.0 over five means.The pipelines differ in absolute values but preserve the boundary ≪ generic-range ≪ oracle ordering.
  • 5.2 Robustness: Tightening k from 200 to 10 reduces HCIR from 6.2% to 1.9%, while relaxing absence to ≤1 worker adds approximately 13% more Z3auto concepts.The condition structure and cross-pipeline agreement remain stable across ten parameter combinations.
  • 5.2 Robustness: BCC reduces the Z3auto pool by 10.7% and mean HCIR by 15%, from 0.0740 to 0.0630, most for boundary-prompt queries.Strict matching remains primary, while BCC is retained as a compositional refinement.

6 Experiment 2: Evaluation Shift

LLM queries produce substantial evaluation shifts relative to human queries, especially in judgment coverage and ranking overlap, and these shifts persist across systems and length controls. Concept-level HCIR explains less than 2% of aggregate variance after accounting for condition and query length.

  • 6 Experiment 2: Evaluation Shift: RBOmin ranges from 0.23 for boundary to 0.10 for high-knowledge queries, below the conservative 0.5 threshold.LLM queries therefore retrieve from substantially different corpus regions than human queries.
  • 6 Experiment 2: Evaluation Shift: Adding HCIR raises explained nDCG@10 variance from 0.9% to 2.1%, an incremental R2 of 1.2%, after condition and length controls.HCIR coefficients are statistically significant for nDCG@10 and judged ratio despite the small variance contribution.
  • 6 Experiment 2: Evaluation Shift: Bpref reverses the human–LLM direction relative to nDCG@10, with a 173% gap reversal for high-knowledge and 206% for boundary queries.The pattern reflects incomplete judgment coverage and is stronger for oracle conditions.
  • 6 Experiment 2: Evaluation Shift: Length-matched comparisons preserve the judged-ratio deficit, indicating that evaluation shift tracks vocabulary divergence rather than query specificity alone.The condition gradient also survives mixed-effects length controls.
  • 6 Experiment 2: Evaluation Shift: Across BM25, BM25+RM3, query likelihood, and cross-encoder reranking, condition gradients retain their rank order and HCIR incremental R2 remains below 2%.Boundary prompts consistently outperform other LLM conditions on judged ratio and nDCG@10.

7 Experiment 3: Diagnostic Intervention

Controlled deletion and injection probes show that Z3auto concepts carry localized retrieval signal beyond matched human-range or random terms. These effects remain diagnostically meaningful even though aggregate evaluation variance is negligible.

  • 7 Experiment 3: Diagnostic Intervention: Deleting Z3auto concepts lowers nDCG@10 by 0.066 points relative to originals, with d=−0.47 and larger effects for Recall@1K and Judged@10.Z3auto deletion underperforms random deletion by d=−0.34, while IDF-matched replacement partially recovers performance with d=−0.21.
  • 7 Experiment 3: Diagnostic Intervention: Z3auto injection lowers nDCG@10 with d=−0.12 and Judged@10 from 0.664 to 0.576 with d=−0.36.Z2 and IDF-matched Z1/Z2 injections produce near-zero nDCG@10 effects, isolating the Z3auto-specific penalty.
  • 7 Experiment 3: Diagnostic Intervention: Queries with at least two Z3auto concepts show an nDCG@10 decline of Δ=−0.092, nearly double the Δ=−0.048 decline for one concept.Both groups converge near 0.116 after deletion, while query-level HCIR correlates with degradation at Spearman ρ=−0.34.
  • 7 Experiment 3: Diagnostic Intervention: Localized deletion effects of d=−0.47 coexist with aggregate R2 < 2% because prevalence is limited, effects are bidirectional, and pool coverage confounds evaluation.The resulting construct value is diagnostic rather than predictive of system-level evaluation distortion.
  • 7 Experiment 3: Diagnostic Intervention: The Z3auto-specific tier has relaxed precision 78.2%, and the deletion hierarchy Z3auto > random > replacement holds across three lexical retrieval systems.Cross-encoder reranking was not applied to these diagnostic probes.

8 Experiment 4: Boundary Compliance

Prompt-based mitigation reduces candidate answer-side intrusion but does not eliminate it, whereas post-generation provenance-constrained selection achieves near-complete boundary compliance while exposing utility trade-offs.

  • 8.1 Prompt-Based Mitigation: 20.7% HCI-Presence under boundary-constrained prompts is lower than the generic baseline’s 33.5%, but intrusion remains.High-knowledge prompts reduce concept-HCIR more strongly yet increase HCI-Presence through longer queries.
  • 8.1 Prompt-Based Mitigation: Topic explains approximately 67% of concept-HCIR variance, while condition and model each explain less than 1%.Prompt engineering therefore reduces, but does not eliminate, intrusion under the tested conditions.
  • 8.2 Concept-Provenance-Constrained Selection: 99 of 100 topics achieve Z3auto = 0 under Pipeline A selection, reducing mean HCIR from 6.23% to 0.06%.Pipeline B independently achieves 100% zero-intrusion coverage, with cross-pipeline agreement on 99 of 100 topics.
  • 8.2 Concept-Provenance-Constrained Selection: A human-central-rate quality floor preserves 99% zero-intrusion coverage while improving HCR from 0.468 to 0.612.It also recovers most utility loss, raising nDCG@10 from 0.157 to 0.174 and judged@10 from 0.577 to 0.627 relative to unconstrained selection.
  • 8.2 Concept-Provenance-Constrained Selection: 94.9% of 930 targeted re-generations achieve Z3auto = 0, and every resistant topic yields at least one zero-intrusion candidate.Combined with selection, this extends coverage to all 40 previously resistant topics, including UQV100.052.
  • 8.2 Concept-Provenance-Constrained Selection: Over-generation, provenance-constrained selection, and targeted re-generation nearly eliminate intrusion, but selection changes the query distribution and does not fully solve query simulation.Figure 4 compares mean concept-HCIR and HCR while showing how coverage rises with modest over-generation.

9 Discussion

Concept provenance detects localized retrieval effects and distinguishes knowledge intrusion from deployment intrusion, but its aggregate predictive power is limited and its practical use has scope constraints.

  • Diagnostic value: Deletion of Z3auto concepts yields nDCG@10 d = -0.47 versus d = -0.34 for random-word deletion and d = -0.21 for IDF-matched replacement.Despite this localized effect, Z3auto density explains less than 2% of incremental aggregate metric variance.
  • Intrusion mechanisms: 45.5% of validated Z3auto cases are knowledge intrusion, while 45.0% are deployment intrusion.The two mechanisms show that boundary violations include both unavailable knowledge and knowledge users might not deploy initially.
  • Intrusion mechanisms: Relaxed precision is 68.2%, whereas strict precision is 45.5%, reflecting the inclusion of both intrusion mechanisms in the broader measure.A positive bpref response to injection, d = +0.11, is consistent with retrieving relevant but unjudged documents.
  • Annotation limitations: LLM annotators downgrade 16–32% of human-confirmed requires-research items, with LLM–human agreement of κ≈0.36.High LLM–LLM agreement does not remove this systematic calibration error.
  • Operational implications: The quality-floor threshold was fixed at the pool median and not sensitivity-tested, and provenance filtering changes the ecological query distribution.The authors therefore frame selection as an evaluation-construction compliance mechanism rather than proof of full natural-human-query reproduction.

10 Limitations and Conclusion

The study finds that candidate answer-side concepts are a boundary-compliance problem with localized retrieval effects but limited aggregate predictive power. Its conclusions are qualified by construct-validity, data, methodological, and scope constraints, while constrained selection nearly eliminates detected intrusion under tested conditions.

  • Limitations: Provenance zones are relative to the observed UQV100 human-query distribution, so additional human sampling could reclassify some Z3auto concepts as human-tail variation.The claims concern LLM Z3auto rates, their variation, and retrieval effects under tested conditions, not human-versus-LLM superiority.
  • Limitations: 68.2% relaxed precision captures both intrusion mechanisms, while strict Z3auto precision is 45.5% and therefore treated as a high-recall diagnostic.The high-specificity subset reaches 78.2%.
  • Limitations: The evaluation uses one collection, UQV100, although robustness checks span 8 models, 5 conditions, multiple retrieval systems, dual pipelines, and threshold sensitivity.Oracle conditions and diagnostic probes were not evaluated under neural re-scoring, so the study is not a standalone dense-retrieval experiment.
  • Limitations: Both extraction pipelines share TF-IDF-based salience, so their token-level HCIR convergence and manual annotation mitigate but cannot rule out shared errors.Knowledge-projection bias also precludes substituting LLM annotators for human judgment.
  • Limitations: The framework covers initial query formulation only, while deletion and injection probes are artificial interventions rather than naturalistic queries.Findings may not generalize to all current or future architectures or to query reformulation and session-level behavior.
  • Conclusion: Concept provenance is a boundary-compliance diagnostic rather than an evaluation-shift predictor because answer-side concepts have localized retrieval effects but little aggregate predictive power.The conclusion distinguishes detecting information-access violations from predicting overall evaluation changes.
  • Conclusion: Prompting reduces but does not eliminate intrusion, whereas constrained selection nearly eliminates detected intrusion under the tested candidate pool.A quality floor recovers most lost utility after selection.

GenAI Usage Disclosure

The authors disclose limited GenAI assistance for manuscript proofreading, rephrasing, and clarity improvements. They report no GenAI use for code or data generation and affirm author verification and scientific integrity.

  • ChatGPT assisted with proofreading, rephrasing technical sentences, and improving manuscript clarity.
  • The authors report that no GenAI tools generated code or generated or augmented the research data.
  • The authors state that all substantive content was written and verified by them and that GenAI use did not compromise originality or scientific integrity.
Loading 2608.25245v1…