PaperlayerView exact revision on arXiv

Source-linked AI summary

What Drives Citations in Production Large Language Models? An Observational Multi-Method Study of Two Million AI Citations Across Ten Thousand Web Pages

Ben Moore, Liam Dunne

arXiv:2609.35077v1cs.AI

TL;DR

Evidence for page-level features that predict production LLM citation frequency has been limited, while practitioner advice has often drawn on SEO patterns and small case studies. The authors analyze production citation data using domain-fixed-effects regression and a nine-method consensus framework, finding prompt-content alignment to be the dominant page-level predictor.

  • Problem

    Page-level predictors of LLM citation frequency are poorly characterised, and existing GEO prescriptions largely rely on SEO-derived patterns and small studies without confound control.

    Source: §1 · p. 1

  • Method

    The authors analyze production citation data with domain-fixed-effects regression and a nine-method consensus framework.

    Source: §3.3 · p. 3, §4.1 · p. 3, §8 · p. 9

  • Results

    β = +0.37 (95% CI [+0.33, +0.41], q ≈10−73) for prompt-content alignment, the dominant page-level predictor; pooled AEO signals reverse or collapse after domain fixed effects.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Within this B2B SaaS study, alignment is the page-level lever with material within-domain effect, and the consensus framework is presented as reusable for observational research.

    Source: §8 · p. 9

    Because interventions were not randomized, estimates are within-domain partial associations rather than treatment effects, and unmeasured confounding cannot be ruled out.

    Source: §7 · p. 9

Abstract

from arXiv · show

Production large language models retrieve and cite web pages alongside generated answers, yet the page-level features that predict citation frequency remain poorly characterised. We present an observational study of approximately 2 million LLM citations from four commercial engines (ChatGPT, Claude, Google AI, Gemini) over six months, joined to 10,000 crawled pages from nineteen B2B SaaS workspaces. Sixty-plus features are tested using a nine-method consensus framework combining mixed-effects regression with domain fixed effects, FDR correction, stability-selection Lasso, double machine learning, generalised additive models, and temporal hold-out replication. Four findings survive all checks. First, prompt-content alignment (Jaccard overlap between page tokens and the full workspace prompt corpus, including non-citing prompts) is the dominant page-level predictor (beta = +0.37, 95% CI [+0.33, +0.41], q ~ 10^-73). Second, the standard AEO checklist (FAQ blocks, structured data, Core Web Vitals) shows positive effects in pooled data that reverse or collapse to zero once domain fixed effects are applied: Simpson's paradox with practical consequences for the AEO literature. Third, domain-level AI authority exceeds the strongest non-alignment page-level feature by a factor of six in mean absolute SHAP value. We release the analytic pipeline as a methodological contribution.

1 Introduction

Practitioner advice on increasing LLM citations has largely relied on SEO-derived prescriptions and small case studies, while large-scale, confound-controlled evidence has been lacking. This study examines production citations and contributes a multi-engine account, a robust alignment finding, and a reusable consensus framework.

1 Introduction

  • AEO prescriptions such as FAQ blocks, structured data, and Core Web Vitals are largely pattern-matched from SEO playbooks and supported by small case studies lacking confound control.Source: §1 · p. 1
  • Prior LLM-citation studies use synthetic prompts, single engines, or small page samples, while empirical GEO studies use pooled estimates without domain-level fixed effects.Source: §1 · p. 1
  • The study analyzes approximately two million citations from four engines over six months and feature-engineers 10,000 cited pages using a nine-method framework.Source: §1 · p. 1

Contributions

  • The study covers four production engines across nineteen B2B SaaS workspaces, addressing the scale and scope of prior evidence.The authors describe it as the first multi-engine, large-N observational study of LLM citation drivers in production data.Source: Contributions · p. 1
  • Prompt-content alignment is the dominant page-level predictor, with β = +0.37 and an effect that survives every robustness check.It is operationalised against the full workspace prompt corpus, including non-citing prompts, to avoid circularity.Source: Contributions · p. 1
  • Median citation age, brand share-of-voice, and third-party platform distributions vary by an order of magnitude across engines.Source: Contributions · p. 1
  • The authors release a nine-method consensus protocol for declaring effects real in observational LLM-citation data, alongside the analytic pipeline.Source: Contributions · p. 2

2 Related work

Prior work evaluates citation attribution and verifiability or tests GEO interventions, but the literature spans different questions and study designs. This paper positions its contribution as a larger, confound-controlled observational analysis of production LLM citation behavior.

2.1 Retrieval and citation in LLMs

  • Attribution research asks whether citations support generated claims, whereas this study asks which page-level features predict citation frequency across cited pages.Source: §2.1 · p. 2

2.2 Empirical GEO literature

  • Empirical GEO studies examine interventions and citation patterns, but the described work includes estimates without domain-level fixed effects and studies of platform-specific variation.The authors distinguish their contribution by its larger corpus, domain-level fixed effects, and nine-method consensus protocol.Source: §2.2 · p. 2

2.3 Observational analysis of blackbox systems

  • Observational analysis of black-box systems has precedent in information retrieval; this work extends that approach to LLM citation behavior using formal multivariate methods.Source: §2.3 · p. 2

3 Data

The study joins production citation records from four engines with a sampled, feature-engineered corpus of cited pages. Its main analyses focus on pages whose features practitioners can modify, and the dataset is not released.

3.1 Citation corpus

  • The citation corpus contains approximately 2.1 million prompt, engine, cited-URL, and position records from four production engines, collected over about six months.Prompts were drafted for nineteen B2B SaaS workspaces and covered top-, mid-, and bottom-of-funnel intent.Source: §3.1 · p. 2

3.2 Page corpus and feature engineering

  • The page corpus includes 10,042 distinct cited URLs, sampled from URLs cited at least twice and capped by domain and bucket to limit overrepresentation.Source: §3.2 · p. 2
  • The 60-plus page features span alignment, structure, recency, infrastructure, and page type; lexical Jaccard overlap uses the full workspace prompt corpus, including non-citing prompts.The authors also report 95.6% URL-level coverage for mobile real-user metrics.Source: §3.2 · pp. 2–3

3.3 Join, bucket classification, and target

  • URLs are grouped as own-brand, competitor-brand, true third-party, or hybrid; headline analyses use the 4,015-page brand-controlled subset.The alignment effect is positive on both own-brand and competitor-brand pages, but substantially larger on own-brand pages, where practitioners directly control content.Source: §3.3 · p. 3

3.4 Anonymisation

  • Client identities are anonymised, and the dataset itself is not released, although the analytic pipeline is.Source: §3.4 · p. 3

4 Methods

The methods combine regression and complementary robustness checks to screen more than 60 features for effects that meet a formal consensus protocol. Domain fixed effects, sensitivity analyses, and temporal replication address distinct threats to inference.

4 Methods

  • The authors use nine methods addressing different inference threats, with four additional methods providing triangulation checks reported in the appendix.An effect enters the main results only if it satisfies the formal consensus protocol.Source: §4 · p. 3

4.1 Mixed-effects regression with domain fixed effects

  • The mixed-effects regression includes domain fixed effects, standardised predictors, and page-type indicators, using OLS with HC1-robust standard errors on 4,015 pages across 297 domains.Source: §4.1 · p. 3

4.2 FDR correction

  • Benjamini-Hochberg correction controls the false-discovery rate at q < 0.05 across the full predictor family.Source: §4.2 · p. 3

4.3 Stability-selection Lasso

  • Stability-selection Lasso uses 200 bootstrap samples with L1 regularisation and a headline selection-probability threshold of 1.0.Source: §4.3 · p. 3

4.4 Double machine learning

  • Double machine learning estimates an orthogonalised partial effect through cross-fitted residualisation, using out-of-fold gradient-boosted regressors.Source: §4.4 · p. 3

4.5 Factor analysis on speed metrics; 4.6 Generalised additive models; 4.7 SHAP feature importance

  • Speed metrics are reduced to latent factors to address collinearity; GAM splines assess nonlinearity, while SHAP values capture interactions missed by the linear model.Source: §4.5 · p. 3, §4.6 · p. 3, §4.7 · p. 4

4.8 Sensitivity analysis; 4.9 Leave-one-domain-out replication

  • Sensitivity checks refit the headline model across five subsets and flag sign or magnitude flips; leave-one-domain-out replication drops each of the eight top domains in turn.Source: §4.8 · p. 4, §4.9 · p. 4

4.10 Temporal hold-out replication; 4.11 Consensus protocol

  • Temporal hold-out keeps URLs fixed while partitioning citations by date, and the consensus protocol requires significance, selection stability, a nonzero DML effect, a suitable GAM shape, and preserved signs.The hold-out uses training data before 1 April 2026 and a hold-out period in April 2026.Source: §4.10 · p. 4, §4.11 · p. 4

5 Results

Across five results subsections, prompt-content alignment is the strongest page-level predictor, while domain-level differences and engine-specific citation patterns qualify the role of on-page features. Domain controls substantially weaken or reverse several pooled associations.

5.1 Prompt-content alignment is the dominant predictor

  • Prompt-content alignment had a standardized coefficient β = +0.37 (95% CI [+0.33, +0.41], FDR-corrected q ≈ 10^-73), corresponding to approximately 0.37 log2 unit higher citation count per standard-deviation increase.It was selected in 200/200 stability-selection bootstraps; independently refit temporal hold-out estimates were +0.61 for six-month training and +0.34 for the one-month hold, both positive with 95% confidence intervals strictly above zero.Source: §5.1 · p. 4
  • Median best-matching-paragraph depth is 0.36; engines preferentially cite paragraphs in the top third of pages.The test against a uniform depth of 0.5 rejects at p < 10^-20.Source: §5.1 · p. 4, Figure 3 · p. 5

5.2 On-page signals are real but small, with substantial confounding

  • FAQ and TLDR/BLUF blocks have small positive coefficients, but Core Web Vitals show no significant effect after domain control.Pooled speed associations collapse toward zero or flip sign after domain residualisation, consistent with Simpson’s paradox.Source: §5.2 · pp. 4–5, Figure 1 · p. 5

5.3 AI engines exhibit substantial heterogeneity

  • Citation age and brand-controlled URL share vary substantially by engine, while third-party LinkedIn citations are concentrated in Google AI Overviews.Claude reaches 60% cumulative citation share within six months, compared with twelve for ChatGPT; Google AI Overviews account for 96.6% of third-party LinkedIn citations.Source: §5.3 · pp. 5–6, Figure 4 · p. 6, Figure 5 · p. 6

5.4 Page format has substantial residual effect

  • Pricing pages have the largest positive residual page-type coefficient (β = +0.39, q < 10^-3), while listicle-review pages have a small negative residual.Binary on-page signals become more predictive deeper in the funnel.Source: §5.4 · p. 6, Figure 6 · p. 6, Figure 7 · p. 6

5.5 Domain authority dominates page-level features

  • Domain authority and prompt-content alignment rank among the largest SHAP contributors; domain authority’s mean absolute SHAP is 0.381 versus 0.060 for the strongest non-alignment page feature.Only approximately 14% of variance remains within domains; the top ten third-party domains account for under 20% of citations, and roughly 1,000 are needed to cover 80%.Source: §5.5 · pp. 6–7, Figure 8 · p. 7

6 Discussion

The discussion argues that domain controls overturn much of the standard AEO case, while prompt-content alignment remains a substantial within-domain lever; it also qualifies causal interpretation and variance decomposition.

6.1 Implications for the AEO literature

  • After domain control, standard structural AEO signals do not support expected citation gains, while prompt-content alignment retains a substantial within-domain effect.The authors recommend harvesting buyer language and reflecting it in page content rather than relying on structural fixes.Source: §6.1 · p. 8

6.2 Implications for empirical study of production LLMs

  • Production-LLM observational studies should use within-domain or within-publisher fixed effects and combine methods because each method has characteristic failure modes.The proposed consensus protocol is conservative by design.Source: §6.2 · p. 8

6.3 Interpreting the R2 decomposition

  • ∆R2 = 0.130 is the page-level increment beyond domain-only R2 = 0.320, and alignment carries the dominant share of that increment.Domain authority reflects accumulated strategy and activity, so the variance split does not partition causal levers.Source: §6.3 · pp. 8–9

6.4 What the alignment finding does and does not say

  • β = +0.37 is a within-domain partial association, not a randomized treatment effect; the analysis does not identify effect direction.The authors suggest natural experiments or controlled prompt manipulations as ways to test the causal claim directly.Source: §6.4 · p. 9

7 Limitations and threats to validity

The estimates are observational and specific to a limited set of workspaces and engines, with additional constraints from prompt-based similarity measures and feature extraction.

Observational design

  • DML controls measured confounders, but unmeasured factors such as content quality and editorial investment cannot be ruled out, especially for weaker findings.The alignment finding survives observable checks, while weaker findings carry greater residual risk.Source: §7 · p. 9

Scope

  • The sample covers nineteen B2B SaaS workspaces on one platform and four engines over six months, so estimates need re-validation in other domains and engines.Engine pipelines may also change across quarters.Source: §7 · p. 9

Alignment metric

  • Title, introduction, and paragraph semantic similarities use citing prompts and should be read as upper bounds on true partial associations.The headline lexical alignment metric instead uses the full workspace prompt corpus, including non-citing prompts.Source: §7 · p. 9

Feature limitations

  • Schema results cover third-party pages only, while 4.4% of PageSpeed coverage is imputed at the median and Jaccard misses semantic alignment beyond unigrams and bigrams.Schema extraction failed on a nontrivial fraction of brand-controlled pages because of CMS-specific JSON-LD rendering.Source: §7 · p. 9

8 Conclusion

The study identifies prompt-content alignment as the dominant page-level predictor of citation frequency, while AEO checklist associations weaken or reverse under domain controls. The authors present a reusable observational protocol and call for causal and broader-domain follow-up.

8 Conclusion

  • β = +0.37 (95% CI [+0.33, +0.41], q ≈10^-73): prompt-content alignment is the dominant page-level predictor and survives the nine-method consensus protocol.The protocol includes FDR-controlled mixed-effects regression, stability-selection Lasso, double machine learning, generalised additive models, and temporal hold-out replication.Source: §8 · p. 9
  • Pooled AEO checklist associations reverse direction or collapse to zero with domain fixed effects, consistent with confounding by domain authority.The conclusion identifies this as a Simpson’s paradox with practical consequences for the AEO literature.Source: §8 · p. 9
  • The nine-method consensus framework is presented as reusable beyond the B2B SaaS domain, while the study calls for causal identification and tests across other content domains.Proposed follow-up includes controlled prompt manipulation, rolling capture windows, and richer semantic alignment metrics.Source: §8 · p. 9

A Full coefficient table

  • Table 1 reports every mixed-effects model coefficient, including non-significant and negatively signed predictors, with confidence intervals, p-values, and FDR-corrected q-values.The table includes coefficients omitted from Figure 1 for visual clarity.Source: A Full coefficient table · p. 11

B Additional robustness output

Prompt-content alignment remains positive across the reported subset and leave-one-domain-out sensitivity checks, and is the only predictor selected in every stability-selection bootstrap.

B Additional robustness output

  • +0.31 to +0.77: alignment stays positive across all five subset definitions, with every confidence interval strictly positive.Across eight leave-one-domain-out fits, its estimate ranges from +0.475 to +0.550, with maximum deviation 0.07; only alignment achieves ˆΠ = 1.0 across B = 200 bootstraps.Source: §B.1 · p. 11

C.1 Temporal hold-out replication

The temporal hold-out keeps the URL set constant and partitions citations by date to assess coefficient stability independently of exposure time. Alignment remains positive in both periods, but training-period predictions transfer poorly to the hold-out.

C.1 Temporal hold-out replication

  • April 2026 citations form the hold target, while earlier citations form the training target, using the same URLs in both partitions.This construction isolates coefficient stability from exposure-time confounding.Source: §C.1 · p. 11
  • +0.605 to +0.342: the alignment coefficient attenuates from training to hold, with both confidence intervals strictly positive.The reported out-of-sample squared Pearson correlation applying training coefficients to the hold target is r2 = 0.018.Source: §C.1 · p. 11
  • R2 = 0.500 training and R2 = 0.418 hold refit, but cross-period absolute predictions do not transfer, consistent with engine-pipeline and prompt-mix variation.The passage identifies dynamics not captured by page-level features as consistent with the weak transfer.Source: §C.1 · p. 11

C.2 Baseline R2 comparators

Domain-level features explain more in-sample variation than page-level features, and the full feature set adds a limited increment. The authors describe that increment as the ceiling on marginal page-level predictive value.

C.2 Baseline R2 comparators

  • The remaining unexplained variance, 1 −0.450, is attributed to engine-pipeline dynamics and unmeasured content-quality signals.Alignment carries the largest share of the domain-only to full-feature increment.Source: §C.2 · p. 11
  • ∆R2 = 0.130: the full feature set improves on the domain-only model, setting the upper bound on marginal page-level predictive value.In-sample R2 is 0.320 for domain-only and 0.450 for the full feature set; word count alone has R2 = 0.004.Source: §C.2 · p. 11, Table 6 · p. 13
Loading 2609.35077v1…