PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring GEO Visibility: Prompt Corpora Define the Answer Market

Olivier Martinez

arXiv:2609.06811v1cs.IRcs.CL

TL;DR

A GEO score’s meaning depends on its prompt corpus, weights, and scoring rules, so a citation rate alone does not specify the situations or scope being measured. The paper treats corpus, formulation policies, and scoring apparatus as an active measurement instrument and proposes score sets compatible with explicit weight constraints. Across selected Amazon Retail, ELI5, and ORCAS strata, activation ranges from 39.66% at weights (0.2, 0.6, 0.2) to 70.54% at (0.2, 0.2, 0.6), conditional on this selection.

  • Problem

    A GEO score’s meaning depends on its prompt corpus, weights, and scoring rules, so a citation rate alone does not specify the situations or scope being measured.

    Source: §1 · p. 2

  • Method

    The paper treats corpus, formulation policies, and scoring apparatus as an active measurement instrument and proposes score sets compatible with explicit weight constraints.

    Source: §1 · p. 2, §4.4 · p. 4

  • Results

    Across selected Amazon Retail, ELI5, and ORCAS strata, activation ranges from 39.66% at weights (0.2, 0.6, 0.2) to 70.54% at (0.2, 0.2, 0.6), conditional on this selection.

    Source: §7.5 · p. 8

  • Takeaways & Limitations

    The paper defines a source-removal contrast for conditional attribution, distinct from effects in production.

    Source: §1 · p. 2

    The comparative empirical usefulness of the documentation protocol remains to be evaluated.

    Source: §1 · p. 2

Abstract

from arXiv · show

GEO (generative engine optimization) visibility scores aggregate source appearances, citations, or brand mentions in generated answers. The prompt corpus selects the situations evaluated, while weights determine their relative importance. Together they define an "answer market" that need not represent actual user demand. Prompt wording can alter retrieval, competing sources, and generated answers. Scoring then requires identifying the appearances, citations, or mentions of interest. If a language model performs this task, its instruction can change the score assigned to an unchanged answer. Our critical survey examines how these choices help define what a GEO score measures. It draws on research into whether indicators measure the intended phenomenon, total survey error, and information retrieval evaluation. The framework specifies situation annotation, prompt formulations, execution conditions, weights, and scoring rules. When weights are unknown or remain to be chosen, the framework reports sets of admissible scores. It distinguishes values compatible with data and assumptions about a target population (partial identification) from variation across weighting conventions (normative sensitivity). A citation alone does not establish a source's contribution. The article defines a comparison of answers generated with and without a source in a controlled documentary context, distinct from an intervention on the full engine with competing sources. The framework is supported by reproducible calculations. No new experiments are reported; its general empirical validity remains to be assessed.

A Critical Survey of Validity, Partial Identification, and Source Attribution

GEO visibility scores depend on an evaluation instrument: prompts can shape generated answers, while corpus, weights, and scoring rules help define what the score measures. This critical survey proposes a framework for documenting those choices and illustrates their consequences with published aggregates, without reporting new experiments.

A Critical Survey of Validity, Partial Identification, and Source Attribution

  • Different weighting schemes yield 39.7% and 70.5% generative-answer activation on selected subcorpora, while reweighting can reverse two systems’ ranking when they excel in different situation categories.These calculations concern answer activation, not brand visibility.Source: A Critical Survey of Validity, Partial… · p. 1

1 Introduction

  • A stated citation rate is insufficient without specifying the request types, engine, period, and weights, because these choices define the score’s scope.The survey also examines how prompts can affect retrieval and how an LLM judge’s instruction can change scoring of the same output.Source: §1 · p. 2
  • A prompt can shape an answer, but corpus, weights, and scoring rules also help define the quantity a GEO score represents.The survey treats the corpus, formulation policies, and scoring apparatus as an active measurement instrument, distinguishing choices that define the target from those that elicit behavior or transform outputs into observations.Source: §1 · p. 2

2 Scope, Selection, and Traceability of the Survey

  • The survey selects references through title and keyword searches and backward citation tracing, with inclusion based on relevance to measurement decisions rather than result direction.Its 44-reference register records publication status, versions, consulted passages, and each source’s role; the demonstrator recalculates published aggregates rather than reproducing experiments.Source: A Critical Survey of Validity, Partial… · p. 1, §2 · p. 2
  • The literature selection is neither exhaustive nor independently coded twice, so unknown inclusion probabilities preclude estimating literature-wide finding frequencies or averages of heterogeneous effects.The survey’s comparative empirical usefulness remains to be evaluated.Source: §2 · p. 2, §1 · p. 2

3 From Construct to Active Instrument

The section separates the phenomenon being measured from its operationalization and calculation, then treats prompts as interacting parts of an evaluation instrument. It distinguishes benchmark, strategic, and usage aims and connects generative evaluation to information-retrieval concerns about which results become observable.

3.1 Four Distinct Objects

  • The construct is the phenomenon of interest, operationalization translates it into observable situations and criteria, the estimand specifies the target quantity, and the estimator is the calculation rule.A string match establishes an exact mention but not, by itself, a favorable recommendation or informational contribution.Source: §3.1 · p. 3
  • Changing weights changes the target when the panel and weights define the conventional population, whereas changing a sample from a population defined in advance changes the estimate, not automatically the estimand.A formulation change can reveal measurement error at a constant construct or genuinely change the task; these possibilities must be distinguished before comparison.Source: §3.1 · p. 3

3.2 Five Functions of Prompts in a Protocol

  • Prompts can specify tasks, select situations, realize needs linguistically, condition interactions, and impose output forms, but these functions are not independent.Requesting sources can change retrieval, the answer, and the citation denominator simultaneously; weights and brand detection are also instrument choices, though not prompt text.Source: §3.2 · p. 3

3.3 Three Aims and a Precedent in Information Retrieval

  • A controlled benchmark targets fixed-case comparisons, a strategic panel applies normative priorities, and a usage estimate targets frequencies in a defined population, period, and interface.These aims are not deficient versions of one another, but become misleading when presented as estimates of a different aim.Source: §3.3 · p. 3
  • Information-retrieval test collections show how selecting documents for judgment can affect measurement when judgments are incomplete, a parallel to selection into observability in generative engines.The parallel does not establish that every pool biases every ranking or that a citation is equivalent to a relevance judgment; generative prompts can additionally affect that selection.Source: §3.3 · p. 3

4 Defining the Target, Then the Compatible Values

The framework defines GEO measurement through annotated situations, prompt and execution policies, weights, and scoring rules. It distinguishes ranking variation under alternative weights from score sets compatible with data and assumptions.

4.1 Annotation and the System Boundary

  • The protocol treats situations as annotated operational units, not observable latent needs, and assigns prompts, execution conditions, and scoring rules distinct roles.The system boundary depends on the intervention: retrieved documents may be downstream mediators for a full-engine prompt intervention, but fixed context in a documentary protocol.Source: §4.1 · pp. 3–4

4.2 Origins of Weights and the Finite Estimator

  • Weight origins include survey inclusion and calibration, documented log frequencies, or normative priorities; uniform formulation weights do not estimate how humans formulate needs.Hierarchical weighting prevents paraphrase multiplication from increasing a family’s weight, while averaging all file rows implicitly weights families by size.Source: §4.2 · p. 4

4.3 Ranking Reversal: A Conditional Possibility

  • In the fictional example, changing λ from 0.2 to 0.8 reverses the ordering: A scores 0.78 versus B’s 0.58, then 0.42 versus 0.52.Reversal is conditional on profiles crossing between strata; common nonnegative weights cannot reverse the ordering if A dominates B in every stratum.Source: §4.3 · p. 4, Table 2 · p. 4

4.4 Admissible Sets and Partial Identification

  • Admissible weight classes yield score sets: these are sensitivity envelopes across legitimate priorities or identified sets when they exactly encode masses compatible with observations and assumptions.For a nonempty compact convex class, extrema are attained and linear constraints yield two linear programs; the class must be justified before results are inspected. Sampling uncertainty is separate, and an unobserved cell retains [0, 1] bounds absent assumptions.Source: §4.4 · pp. 4–5

4.5 Transport and Temporal Comparison

  • Temporal decomposition separates response changes on the initial population from composition changes at final responses, but its reference ordering is descriptive, not uniquely causal.Changing variant policy, environment, or judge enters the response function; changing annotation can alter the situation distribution and conditional means, requiring a common situation space or explicit correspondence.Source: §4.5 · p. 5

5 What the Literature Shows about Formulations

Research on prompt formulations shows that evaluation outcomes depend on both the tested variations and scoring rules. The section argues that these choices must be distinguished and specified for GEO visibility measurement.

5 What the Literature Shows about Formulations

  • Formulation-policy effects in language-model benchmark studies do not automatically transfer to citation or mention rates, and initial prompts and multi-turn trajectories define different answer-opportunity targets.Multi-turn research motivates separating those targets but does not supply the distribution of real purchasing journeys.Source: §5 · p. 5

5.1 Genuine but Conditional Empirical Sensitivity

  • Sclar et al. report substantial differences and altered comparisons under format changes, while the tested variant grammar is not a distribution of human formats [Sclar et al., 2024].Mizrahi et al. compare mean and best performance across 20 models and 39 tasks with 6.5 million instances, but synthetic paraphrases are not a usage sample [Mizrahi et al., 2024].Source: §5.1 · p. 5
  • PromptEval estimates score distributions over prompts, while ReliableEval studies stochastic evaluation; neither supplies a target distribution of human requests [Maia Polo et al., 2024b; Lior et al., 2025].Source: §5.1 · p. 5
  • Researchers must distinguish target-population variation, invariance tests, and performance optimization; tuning on the same observations used for reporting creates selection optimism.A tuned score measures a system plus its adaptation procedure, and unequal adaptation budgets can confound system comparisons; comparable budgets do not fix test reuse.Source: §5.1 · p. 5

5.2 Scoring Can Produce Part of the Sensitivity

  • Scoring methods can account for some observed variation: accepting semantically correct answers reduced benchmark dispersion, but an LLM judge is not an error-free reference [Hua et al., 2025].When an LLM scores outputs, its instructions form a second prompt-sensitive layer, so both answers and their transformation into scores warrant validation.Source: §5.2 · pp. 5–6

5.3 Not Every Response Variation Is a Defect

  • Reformulations are equivalent relative to a task and judging rule; added constraints can legitimately change recommendations, so expected invariance must be distinguished from desirable directional change [Ribeiro et al., 2020].Jack et al. report lower recommendation similarity across variants than identical reruns, but enriched variants also change the need, limiting the inference to robustness under those transformations [Jack et al., 2026].Source: §5.3 · p. 6

6 Building a Corpus: Provenance, Coverage, and Interaction

Corpus provenance, interaction structure, and language or location conditions shape which answer opportunities are represented, so a corpus must fit the inference being made. The section shows that real or synthetic prompts, larger samples, and translated wording do not by themselves establish representativeness.

6.1 Real, Plausible, and Representative

  • Real conversations and privileged platform access still reflect specific selection mechanisms and require explicit population boundaries.WildChat and LMSYS-Chat-1M make varied uses observable, while platform-data access improves observation of a defined population without removing the need to specify its limits.Source: §6.1 · p. 6
  • A synthetic corpus can provide controlled coverage and rare cases, but its size and diversity do not establish that its frequencies represent buyers.The cited PersonaGen-1M validation does not establish correspondence to a buyer population.Source: §6.1 · p. 6
  • More observations cannot necessarily repair selection bias; without non-English observations or additional assumptions, English-request reweighting cannot estimate behavior in other languages.If part of the target population has no chance of collection, the positivity condition is violated; an empty sample cell alone does not prove that structural absence.Source: §6.1 · p. 6

6.2 The Need Is Not Always Specified in the First Turn

  • First-answer citation rates, per-turn rates, and conversation-level citation probabilities use different denominators, and replacing a trajectory with one exhaustive prompt changes the opportunities measured.A source absent from the first answer may be cited after a follow-up.Source: §6.2 · p. 6
  • Turning a hesitant conversation into an exhaustive initial question measures resolution of an already specified need, which can isolate a capability but should be declared.This may remove assistant work such as clarifying, requesting a constraint, or correcting an interpretation.Source: §6.2 · p. 6
  • Complete-interaction evaluation requires a user policy for next messages; reactive simulators require behavioral validation and a stopping rule, while fixed replay can produce incoherent exchanges.Search Arena uses volunteer participants and encourages web-search requests, and neither it nor the cited studies supplies a distribution of purchasing journeys [Miroyan et al., 2026; Laban et al., 2026].Source: §6.2 · p. 6

6.3 Language, Location, and Assumed Knowledge

  • The 234-run exploratory pilot found provider differences under some conditions, but its small set of needs and short collection window do not support a universal linguistic-market claim [Żatuchin, 2026].A control category did not reproduce the same effect.Source: §6.3 · p. 6
  • Translating prompts alone may not preserve relevant local conditions, and expert-generated requests may presume terms users are trying to discover; assess equivalence by need and available knowledge.Textual fluency alone is not the standard for equivalence.Source: §6.3 · pp. 6–7

7 Generative Visibility: Intervention, Competition, and Denominators

Visibility evidence depends on whether sources are supplied or retrieved and on what outcome is scored. The section separates controlled source-removal comparisons from production interventions and treats aggregate calculations as bounded illustrations rather than market estimates.

7.1 From Supplied Context to Endogenous Retrieval

  • A document absent from supplied context ordinarily cannot be cited from it, but discovery, retrieval, selection, contribution, citation, and brand mention are distinct outcomes.Attribution precedents test source support and citation faithfulness; the proposed contrast specifies the source intervention, context policy, and substantive answer score under competition.Source: §7.1 · p. 7

7.2 An Explicit Source Counterfactual

  • A controlled source-removal contrast holds the retrieved list fixed, while production interventions rerun retrieval and reranking under competing publishers’ treatments.The source-removal estimate depends on context policy, including slot handling, compression, and copies; production comparisons require comparable prompts, engine settings, scoring, and initial documentary state.Source: §7.2 · p. 7

7.3 Recognition and Discovery: Bounded Illustrations

  • Named-product and discovery prompts are different tasks, and their observed contrast is not a pure paraphrase effect or a market estimate.The illustrations are not independent replications establishing a language-by-market law or human frequencies; the sources share part of their provenance.Source: §7.3 · p. 8

7.4 What the Compared Evidence Supports

  • Compared studies change the observed variable, unit, context, activation, and success criterion, so large corpora do not make heterogeneous designs repeated confirmations.The comparison’s rightmost column defines what each result can support.Source: §7.4 · p. 8

7.5 A Secondary Demonstrator Using Published Aggregates

  • Grossman’s 65.6% activation rate is for the full corpus, which includes ORCAS; the selected three-subcorpus panel is not a neutral or typical-corpus estimate.The panel reference and its weighting scenarios apply only to the selected support.Source: §7.5 · p. 8
  • With ORCAS fixed at 20%, reallocating weight between Amazon and ELI5 yields a sensitivity envelope of [0.3966; 0.7054], versus [0.174; 0.946] without further three-stratum constraints.These are normative sensitivity envelopes using published rates, not identified sets for an unknown usage population or confidence intervals; their magnitude depends on the selected strata.Source: §7.5 · p. 8
  • Activation rates alone cannot determine conditional mention rates, so the per-query presence calculations use an illustrative 40% conditional mention assumption.The reported 15.864% and 28.216% presence-per-query values hold only under that assumption.Source: §7.5 · p. 8
  • The reconstruction treats query records—not independent needs—as its unit, and rounded source rates yield a remainder compatibility envelope of [76.7%; 77.2%].The five named categories are assumed to be disjoint subsets of 11,500 records; individual responses and separate residual rates remain unavailable, precluding arbitrary full-benchmark reweighting.Source: §7.5 · p. 9

8 Uncertainty and a Computable Protocol

The section treats uncertainty as arising from weights, dependencies, and assumptions about generalization, then sets out a protocol that makes scores and their assumptions inspectable. It extends documentation practices to connect the estimated quantity with the intervention.

8.1 Weights, Dependencies, and Efficient Evaluation

  • The hierarchical variance account distinguishes repetition, reformulation, and sampling, but its design-effect expression is not universal across dependencies, temporal shocks, heteroskedasticity, or estimated weights.For a fixed panel, variation among needs is descriptive unless an explicit population or model makes it sampling error; treating paraphrases as independent situations creates pseudoreplication.Source: §8.1 · pp. 9–10
  • Efficient benchmark evaluation can reduce required observations under generalization assumptions, but it does not identify the distribution of human needs.Estimation efficiency on a benchmark and external validity of its target must be assessed separately.Source: §8.1 · p. 10

8.2 A Protocol with Verifiable Outputs

  • A verifiable protocol links each score to a declared target, observable states, scoring rule, and admissible weights, while retaining records to inspect execution, coding, and composition changes.Missing collections can make deletion estimate a quantity conditional on observation; when missingness is known and the score bounded, worst-case bounds are preferable to unjustified zero imputation.Source: §8.2 · p. 10
  • The implementation records separate retrieval, citation, and brand-mention fields, and an observed citation with an inaccessible source has unassessable—not automatically zero—support.These measures do not establish the causal contribution of a source.Source: §8.2 · p. 10

8.3 An Extension of Datasheets, Not a Competing Checklist

  • The documentation card extends datasheets by linking them precisely to the estimated quantity and intervention, while fixed-core, refreshed-panel, and experimental results remain separate.A fixed core holds one instrument component constant; it does not neutralize documentary change, personalization, or updates.Source: §8.3 · p. 10

9 Discussion and Limitations

Prompts help define the evaluated situations and elicit system behavior, while scoring instructions can alter how unchanged answers are evaluated. The discussion distinguishes what the examples establish from what remains empirically unvalidated and bounded by the survey’s evidence.

9.1 Why Prompts Belong to the Measurement Instrument

  • Prompts define which situations and weights constitute an answer market, rather than necessarily representing an external population of needs.A change in the corpus therefore changes what can be concluded from the score.Source: §9.1 · p. 10
  • Prompt wording assigns different tasks and can alter queries, retrieved documents, and competing sources, so supplied-document evaluations do not automatically transfer to full-engine visibility.These are distinct protocols: one studies answers within a fixed context; the other includes document selection.Source: §9.1 · p. 10
  • An LLM judge’s instruction can change the evaluation of an unchanged answer, so situation selection, answer generation, and scoring need separate documentation.Table 5 links seven implementation steps to four decisions; the steps may serve more than one decision.Source: §9.1 · p. 10, Table 5 · p. 11
  • Rankings are robust only when they hold throughout the selected admissible class under common weights; instability establishes neither system equality nor inability to choose for a particular use.The comparison must specify which choices determine the ordering and tie it to the intended use.Source: §9.1 · p. 11

9.2 What the Developments Establish and What Remains Open

  • The examples establish a conditional ranking reversal and the arithmetic magnitude of reweighting published aggregates, but do not estimate prompt effects on a human population.The novelty is connecting and operationalizing existing problems for generative visibility measurement, not inventing partial identification, weighted variance, or counterfactuals.Source: §9.2 · p. 11
  • Empirical validation remains open: compare corpus construction with observed populations, test annotations and judges, and examine transport beyond the corpus and full-pipeline interventions.The survey is selective and predominantly English-language; convergence in vocabulary is not independent replication, and annotated situations do not reveal true latent needs.Source: §9.2 · p. 11, §9.3 · pp. 11–12

9.3 Limitations of the Evidence Base and Aggregates

  • Published rates describe particular collections, and rounding, query dependencies, and unavailable individual data preclude valid frequentist intervals from the reported aggregates.Visibility bounds without mentions are wide because information is missing; a poorly justified admissible class can make results artificially robust or undetermined.Source: §9.3 · p. 12
  • Exposure, mentions, and attribution do not directly measure attention, trust, purchasing, or persuasion; those outcomes require human units, observations, and separate designs.A documented judge can still be inadequate, and documenting the instrument does not identify these effects by proxy.Source: §9.3 · p. 12

10 Conclusion

The conclusion frames GEO visibility as measurement over protocol-defined answer opportunities, with prompts affecting both situation coverage and system behavior. It proposes four decisions for making scores interpretable, while emphasizing that empirical usefulness remains untested.

10 Conclusion

  • A GEO score summarizes source or brand presence under a protocol, and its prompt corpus and weights delimit the answer market being evaluated.Prompts can alter retrieval and competition, while an LLM judge’s instruction can affect scoring of an unchanged answer.Source: §10 · p. 12
  • The examples show that changing weights can reverse rankings when system profiles cross strata, while published-aggregate calculations change reported quantities through panel composition and denominator conventions.The demonstrator concerns generative-answer activation, not observed publisher visibility; score differences must be interpreted in light of their construction conditions.Source: §10 · p. 12
  • D1 specifies the target, units, annotation, formulations, execution, and scoring so the score’s scope can be assessed and its calculation reproduced.Denominators, missing observations, and any judge and instruction are part of that definition.Source: §10 · p. 12
  • D2 justifies covered situations and weights, reporting admissible values while distinguishing normative sensitivity from population-based identification and sampling uncertainty.D3 defines the source intervention: a controlled removal contrast differs from full-engine effects involving retrieval, reranking, and competitors, which a temporal change alone cannot disentangle.Source: §10 · p. 12
  • D4 compares systems under the same weights and makes dependencies visible; the framework’s empirical usefulness and GEO optimization effectiveness remain unvalidated.Future work should compare corpora with observed usage, evaluate annotations and judges, and test conclusions in other situations and suitable experimental designs.Source: §10 · p. 12

AI Assistance Statement

The manuscript reports AI assistance for writing support, translation, and file preparation, while assigning validation and publication responsibility to the human author.

AI Assistance Statement

  • Codex assisted with writing, translation, and file preparation; source and argument validation and publication responsibility remain with the human author.The assistance is not independent validation.Source: AI Assistance Statement · p. 13

Availability and Reproducibility

The version includes materials and calculation programs for reproducibility, but reports no new experimental collection from users or generative systems.

Availability and Reproducibility

A Corpus Card for This Survey’s Documentary Corpus

The survey applies a corpus card to its own documentary selection, while distinguishing what was consulted from what was not audited.

A Corpus Card for This Survey’s Documentary Corpus

  • The corpus card treats each scientific work, grouping its versions, as the unit and assigns no numerical score or prevalence estimator.Bibliographic population weights are therefore irrelevant, though coverage choices influence the arguments available.Source: A Corpus Card for This Survey’s… · p. 13
  • Table 8 is the corpus card for the survey’s documentary selection.Source: Table 8 · p. 13
  • The register records sections, tables, or methods consulted for references beyond the readings requiring immediate notice in Table 9.“Accessible text” does not mean all experiments were audited.Source: A Corpus Card for This Survey’s… · p. 13, Table 9 · p. 13
  • The main text of Tamer’s survey and all of Section 3 of Jacobs and Wallach were read in full.Section 2 and the reading register specify reading versions, methods, and limits.Source: A Corpus Card for This Survey’s… · p. 13
Loading 2609.06811v1…