PaperlayerView exact revision on arXiv

Source-linked AI summary

GEO-Flag: Detecting and Measuring GEO-Optimized Web Content

Junjie Chu, Ye Leng, Mingjie Li, Yun Shen, Xinyue Shen, Yang Zhang

arXiv:2608.16824v1cs.LGcs.CRcs.IR

TL;DR

GEO-optimized pages can gain disproportionate visibility while making weak or false information appear supported, yet systematic detection tools remain limited. The paper benchmarks GEO detection, introduces intervention-paired training and citation auditing, and estimates 8.90% GEO prevalence in released search results, reaching 16.36% among pages modified in 2026.

  • Problem

    Systematic tools for identifying GEO-optimized webpages remain limited despite risks to source visibility and provenance in generative search.

  • Method

    The paper introduces GEOFlagBench, Intervention-Paired Training, and a GEO-gated Agent for detecting GEO pages and auditing citation URL provenance.

  • Results

    Strong aggregate performance masked method-level and authorship-related weaknesses, while IPT improved ModernBERT F1 from 0.862 to 0.944 and worst-group accuracy from 0.725 to 0.883.

  • Takeaways & Limitations

    The study establishes a foundation for systematically detecting, auditing, and measuring GEO in real-world search ecosystems.

  • Takeaways & Limitations

    GEO detection does not indicate that a webpage contains false, misleading, or harmful information, so the system is intended for transparency and risk analysis rather than automatic removal.

Abstract

from arXiv · show

Generative Engine Optimization (GEO) modifies web content to increase its likelihood of being selected and cited by generative search engines. This can give strategically optimized pages visibility disproportionate to their authority or relevance and even make weak or false information appear well supported. Unlike conventional search, generative search synthesizes information into direct answers rather than presenting competing sources, which can further amplify these risks, as assessing source provenance and authority requires additional user interaction. Despite these concerns, systematic methods for detecting GEO-optimized webpages remain underexplored. We introduce \texttt{GEOFlagBench}, a benchmark of 3,200 webpages spanning 400 queries, four domains, and eight GEO optimizer families, and use it to systematically evaluate existing GEO detection methods. Although the strongest baseline achieves an aggregate F1 of 0.880, method-level and authorship-conditioned evaluations reveal substantial weaknesses and potential reliance on authorship-related shortcuts. We therefore propose \emph{Intervention-Paired Training} (IPT), which supervises detector responses to GEO interventions and non-GEO AI polishing; on ModernBERT, IPT improves F1 from 0.862 to 0.944 and worst-group accuracy from 0.725 to 0.883. We develop a GEO-gated Agent system for auditing the Source Tier and verifiability of Citation URLs in detected GEO pages. Finally, we deploy the complete pipeline on released Google Search and Gemini-grounded retrieval results for 1,000 real-user queries. Across 10,095 available pages, we estimate an overall GEO prevalence of 8.90\%, reaching 16.36\% among pages modified in 2026. Our results establish a foundation for systematically detecting, auditing, and measuring GEO in real-world search ecosystems.

1 Introduction

GEO can amplify strategically optimized webpages while making weak or false information appear well supported, yet systematic detection tools remain underdeveloped. This paper introduces a benchmark, intervention-based detector training, citation auditing, and empirical prevalence estimates for GEO in real-world retrieval results.

  • Motivation: GEO can give optimized pages visibility disproportionate to their authority or relevance and make weak or false information appear well supported.A false claim can be reinforced when one strategically created page cites another.
  • Detection benchmark: GEOFlagBench systematically evaluates GEO detection methods and reveals substantial method-level weaknesses and potential reliance on authorship-related shortcuts despite strong aggregate performance.The benchmark addresses the underexplored problem of determining whether webpages were deliberately optimized for generative search.
  • Intervention-Paired Training: IPT supervises detector responses to GEO interventions and non-GEO AI polishing, improving ModernBERT F1 from 0.862 to 0.944.Positive pairs require higher GEO scores after GEO interventions, while zero pairs preserve scores after non-GEO AI polishing.
  • Citation auditing: The GEO-gated Agent audits Citation URL Source Tier and verifiability by combining detection, deterministic parsing and retrieval, and constrained Agent assessment.Deterministic rules derive Citation URL Verifiability after Source Tier assignment.
  • Real-world prevalence: 8.90% is the overall estimated GEO prevalence in released Google Search and Gemini-grounded retrieval results, reaching 16.36% among pages modified in 2026.Because live pages lack GEO ground-truth labels and the pipeline is imperfect, these findings are empirical estimates rather than definitive measurements.

2 GEO Flagging with Existing Detectors

Existing GEO detectors can achieve strong aggregate performance, but their effectiveness varies across optimization methods and authorship conditions. These gaps indicate that some detectors may rely partly on human–AI authorship cues rather than GEO-specific signals.

  • Detection Task: GEO detection is a binary document-only task: predicting whether a webpage was modified with GEO techniques, without access to its original version, prompt, or provenance metadata.A positive prediction means resemblance to pages modified by the studied GEO techniques, not malicious intent, factual inaccuracy, or successful search-engine manipulation.
  • Benchmark: GEOFlagBench contains 3,200 documents from 400 queries across Health, Finance, Technology, and Travel, including 1,200 non-GEO controls and 2,000 GEO documents from eight optimizer families.Non-GEO controls comprise human-written, AI-polished, and AI-generated documents; GEO documents use human-written and AI-generated seeds, excluding AI-polished controls.
  • Aggregate Results: 0.880 F1 is achieved by Word TF-IDF with logistic regression, while Character TF-IDF and Pangram reach 0.878 and 0.876, respectively.The results show that competitive flagging is possible with fine-tuned encoders, lexical features, and a proprietary AI-text detector.
  • Method-Level Results: 0.308 and 0.423 are the AutoGEO-Light recalls of the two highest-F1 TF-IDF classifiers, showing that aggregate performance can conceal weak detection of sparse GEO edits.PMA is detected reliably by many methods, whereas AutoGEO-Light may dilute GEO signals because much of the source text remains unchanged.
  • Authorship-Conditioned Results: 0.375 worst-group accuracy and 0.558 ∆FPR accompany Word TF-IDF’s 0.880 overall F1, while Pangram combines 0.876 F1 with 0.192 worst-group accuracy and 0.767 ∆FPR.Large authorship-conditioned gaps are consistent with shortcut reliance, although the observational analysis does not establish detectors’ internal decision rules.
  • Authorship-Conditioned Results: Near-zero authorship gaps can arise from predicting nearly every document as GEO, producing near-perfect GEO-subset recall but worst-group accuracy close to zero.Thus, apparent stability from constant predictions does not demonstrate robust GEO detection.

3 Toward More Reliable GEO Detection

Existing GEO detectors can fail to generalize across optimization methods and authorship conditions, motivating Intervention-Paired Training (IPT), which targets GEO-specific changes while discounting non-GEO AI polishing. IPT substantially improves binary detection and also supports attribution across seven GEO classes.

  • Motivation: Existing detectors generalize poorly across some GEO methods and show differing error rates between human- and AI-authored documents.These patterns suggest reliance on authorship-related shortcuts rather than GEO-specific signals.
  • Intervention-Paired Training: IPT pairs original pages with GEO-intervened and AI-polished versions, requiring GEO interventions to raise scores while polishing leaves them approximately unchanged.Positive pairs identify changes that should increase GEO scores, whereas zero pairs identify LLM-induced changes that should not affect them.
  • Binary Detection Results: IPT improves ModernBERT accuracy from 0.839 to 0.931 and F1 from 0.862 to 0.944.Its shortcut diagnostic also improves, with WGA rising from 0.725 to 0.883.
  • Binary Detection Results: ModernBERT-GRL reaches 0.876 accuracy and 0.897 F1, while improving WGA to 0.783 and reducing ∆FPR and ∆TPR to 0.171 and 0.069.The gains are smaller than those reported for IPT.
  • Multiclass Attribution: IPT achieves at least 0.804 precision for every attribution class, including 0.985 for PMA and 0.984 for RAID, while Non-GEO recall remains 0.950.The seven-class task distinguishes Non-GEO, AutoGEO, GEO Strategy Pool, PMA, RAID, Meta-Optimization, and Human-based GEO.

4 A GEO-Gated Agent System for Assessing Citation URL Verifiability

The GEO-gated Agent audits citation URLs by combining GEO detection, deterministic preprocessing and retrieval, source-tier assignment, and rule-based verifiability aggregation. On the test set, the gate achieved 93.95% accuracy and 93.94% macro F1, while GLM 5.2 achieved the strongest downstream performance across Source Tier, Citation URL Verifiability, and sentence-level verifiability.

  • Metric design: Citation URL Verifiability combines Source Tier with retrieval state, treating source authority and accessibility as separate factors.The audit does not assess factual correctness or whether a citation semantically supports its claim.
  • Reliability: 84.16% agreement and Cohen’s κ=0.751 were obtained for Citation URL Verifiability, while sentence-level best verifiability reached 84.90% agreement and κ=0.760.These values result from applying the same retrieval and aggregation rules separately to primary annotators’ Source Tier assignments.
  • System architecture: The four-module pipeline gates citation audits to predicted GEO pages, extracts sentence–citation URL pairs, labels access status and source tier, and deterministically aggregates verifiability.Pages predicted as NON-GEO are skipped; pages without citation URLs are returned as NO_CITATION, and sentence-level output uses the highest verifiability among citations.
  • GEO Gate Performance: 93.95% accuracy and 93.94% macro F1 were achieved by the GEO gate, with 98.81% GEO precision and 88.97% GEO recall.The gate correctly identified 250 of 281 gold-labeled GEO pages, produced three false positives, and missed 31 GEO pages.
  • GEO Gate Performance: 76.53% of citation occurrences and 76.19% of cited sentences in the gold-labeled GEO citation set were exposed to the Agent through correct gating.Correctly gated pages included 106 of 129 citation-containing gold-labeled GEO pages, exposing 1,653 of 2,160 citation occurrences and 1,104 of 1,449 cited sentences.
  • Downstream performance: GLM 5.2 performed best across all three outputs, achieving 84.02% macro F1 for Source Tier, 83.20% for Citation URL Verifiability, and 83.26% for sentence-level verifiability.Its corresponding accuracies were 84.75%, 83.00%, and 83.15%, respectively.

5 Empirically Estimating GEO Prevalence in Real-World Search Results

Auditing 10,095 unique pages from released Google Search and Gemini-grounded results for 1,000 historical real-user queries, the pipeline estimates 8.90% GEO prevalence overall. GEO is more prevalent in Gemini and on recently modified pages, while Citation URLs on detected GEO pages are predominantly low-verifiability C3 sources.

  • Query Sampling and Retrieval Channels: 1,000 historical ORCAS queries were sampled proportionally across intent and query-length strata, yielding 4,999 eligible queries after deduplication.The audit retained all 1,000 Google Search queries and used 979 Gemini queries with released source links.
  • GEO Prevalence: 8.90% of 10,095 unique pages were detected as GEO, including 8.14% of Google Search pages and 9.09% of Gemini pages.The page-level Gemini–Google difference was 0.95 percentage points, while query-balanced macro rates were 7.91% and 9.34%, respectively.
  • Modification-Time Variation: 16.36% was the estimated GEO prevalence for 2026 pages, up from 7.02% in 2024 and 12.80% in 2025.In 2026, estimates reached 13.52% for Google Search and 18.20% for Gemini, indicating higher prevalence among recently modified pages.
  • Domain-Level Variation: 20.37% of Amazon pages and 4.38% of YouTube pages exhibited estimated GEO, while several large informational and health-related domains had no detected GEO pages.The comparison included domains with more than 50 successfully retrieved usable pages in the unique-page union.
  • URL Source Tier and Verifiability: 68.84% of Citation URLs were assigned to C3 sources and 69.34% received LOW verifiability, with publisher accountability driving most LOW labels.Gemini-returned GEO pages contained 8.80 Citation occurrences per detected GEO page versus 4.90 for Google Search, and had a 73.82% versus 45.14% C3 share.

6 Related Work

Prior work characterizes generative search as retrieving external sources to produce direct answers, while documenting citation errors and instability in retrieved sources relative to conventional search and across repeated executions.

  • Generative Search: Generative search retrieves external sources and synthesizes them into a direct answer to the user’s query.
  • Generative Search: Prior studies find substantial citation and attribution errors in whether generated content is supported by its cited sources.
  • Generative Search: Generative search may retrieve substantially different source sets than conventional web search and vary across repeated executions.

7 Discussion and Limitations

GEO detection concerns optimization for generative search, not whether content was human- or AI-authored. GEO flagging is not evidence that a webpage is false, misleading, harmful, or malicious, and should support transparency and risk analysis.

  • GEO Detection Is Distinct from AI-Text Detection: GEO detection is distinct from AI-text detection because it measures generative-search optimization rather than authorship.Both human- and AI-authored pages can be GEO-optimized, while AI-generated or AI-polished pages can remain non-GEO.
  • GEO Detection Is Distinct from AI-Text Detection: Authorship-dependent detectors may perform strongly while detecting authorship cues instead of GEO optimization.Future benchmarks should control for authorship and include paired interventions, sparse modifications, and human-executed GEO strategies.
  • Use and Interpretation of GEO Flagging: GEO flagging does not imply that a webpage contains false, misleading, harmful, or malicious information.Publishers may use GEO techniques legitimately to improve content structure or visibility in generative search.

8 Conclusion · A Supplementary Details for Building GEOFlagBench · A.1 Domain Selection and Query Collection

The paper presents GEOFlagBench as a systematic benchmark for detecting and auditing GEO-optimized content, while addressing detector weaknesses through IPT and discussing ethical safeguards. The benchmark spans four heterogeneous domains and uses filtered existing queries supplemented by generated informational queries.

  • 8 Conclusion: The study introduces GEOFlagBench to support systematic detection and auditing of GEO-optimized content across diverse domains and optimization strategies.The benchmark is used to evaluate existing GEO detection methods.
  • 8 Conclusion: Strong aggregate detection performance can mask method-level weaknesses and reliance on authorship-related shortcuts.This motivates explicitly evaluating robustness beyond aggregate performance.
  • 8 Conclusion: IPT supervises detector responses to GEO and non-GEO transformations, improving overall detection performance and robustness across authorship groups.The intervention-based supervision directly addresses the limitations identified in existing methods.
  • Ethical Considerations: The study uses publicly accessible, synthetically generated, or modified webpages and does not recruit human subjects or infer sensitive attributes about authors or users.Data collection is limited to content needed to study GEO detection and citation behavior.
  • Ethical Considerations: The benchmark and detectors could offer limited feedback for making GEO content harder to detect, so the paper emphasizes measurement and detection rather than evasion guidance.GEO examples rely on existing or explicitly defined optimization strategies rather than vulnerabilities in deployed systems.
  • Ethical Considerations: Citation analysis assesses source accountability and verifiability rather than claim truthfulness or author reputation, with results reported primarily in aggregate.This reporting approach is intended to reduce unfair characterization of individual webpages or publishers.
  • A.1 Domain Selection and Query Collection: The benchmark covers four domains—Health, Finance, Technology, and Travel—to represent heterogeneous information environments where GEO effectiveness may differ.Health and Finance are consequential settings because inaccurate information may affect users’ health or financial well-being.
  • A.1 Domain Selection and Query Collection: 164 GEO-Bench queries remain after filtering, comprising 58 Health, 26 Finance, 66 Technology, and 14 Travel queries.The retained queries exclude sensitive cases and are mapped to the four benchmark domains using GEO-Bench annotations.

A.2 Non-GEO Data Construction · A.3 GEO Data Construction

The benchmark’s non-GEO data combines provenance-controlled human-written pages with AI-polished and AI-generated controls, using temporal filtering and one seed page per query. Its GEO portion comprises eight optimizer families representing distinct optimization approaches.

  • A.2 Non-GEO Data Construction: Human-written non-GEO candidates come from Wikipedia and the Brave Search API, with Brave retrieval restricted to predefined institutional, scientific, news, and professional publishers.The restriction prioritizes identifiable provenance, stable content, and traceable metadata while reducing anonymous or unstable pages.
  • A.2 Non-GEO Data Construction: The candidate pool spans 14 source domains, with pages restricted to versions available before October 1, 2022.Wikipedia uses its latest pre-cutoff revision, while other publishers are retrieved through Brave Search with the same publication cutoff.
  • A.2 Non-GEO Data Construction: 1,026 candidate pages are retrieved across 400 queries, and temporal auditing reduces them to 871 eligible pages.Main content is extracted with Trafilatura while preserving titles, lists, tables, and hyperlinks; retained pages must have dateModified before October 2022 and no audited post-cutoff modification.
  • A.2 Non-GEO Data Construction: 400 seed webpages are selected, one per query, prioritizing non-Wikipedia pages when multiple eligible sources exist and covering all 14 source domains.For each human-written non-GEO page, the benchmark also constructs an AI-polished counterpart that preserves its content while polishing language.
  • A.2 Non-GEO Data Construction: The benchmark composition is organized by document provenance and optimization, with separate tabulations of provenance and optimizer-family seed composition.These compositions are reported in Tables 13 and 14.
  • A.2 Non-GEO Data Construction: AI-generated non-GEO controls are produced from domain summaries and evenly assigned across the same eight polishing models without overlap.Prompts request natural, coherent articles of comparable length, contain no GEO strategies or optimization objectives, and preserve supplied special markers where appropriate.
  • A.3 GEO Data Construction: Eight GEO optimizer families form the GEO portion of the benchmark, each representing a distinct approach to GEO optimization.The construction details for these families are listed in Table 12.

A.4 Dataset Summary · B Supplementary Details for Improved GEO Detection · B.1 Gradient Reversal for Authorship Invariance

GEOFlagBench contains 3,200 documents spanning GEO and non-GEO content, while the proposed Gradient Reversal Layer suppresses original-authorship information during GEO detection. The method preserves GEO-relevant features, uses adversarial authorship prediction during training, and requires only the GEO prediction at inference.

  • A.4 Dataset Summary: 3,200 documents comprise GEOFlagBench, covering eight GEO optimizer families and three non-GEO subtypes.The dataset includes 2,000 GEO documents and 1,200 non-GEO documents.
  • A.4 Dataset Summary: 2,000 GEO documents include 1,050 human-written and 950 AI-generated seeds.The eight GEO optimizer families account for all 2,000 GEO documents.
  • A.4 Dataset Summary: 1,200 non-GEO documents comprise 400 human-written, 400 AI-polished, and 400 AI-generated seeds.These three groups form the non-GEO subtypes in the benchmark.
  • B.1 Gradient Reversal for Authorship Invariance: The Gradient Reversal Layer suppresses original-authorship information in the learned representation to reduce authorship dependence.ModernBERT produces a shared representation, with separate GEO classification and adversarial authorship heads.
  • B.1 Gradient Reversal for Authorship Invariance: Original authorship identifies the underlying content source, labeling AI-polished pages from human seeds as human and pages from AI seeds as AI.The adversarial objective targets human–AI authorship rather than whether an LLM edited the page.
  • B.1 Gradient Reversal for Authorship Invariance: During backpropagation, the GRL multiplies authorship gradients reaching the encoder by −λ while acting as the identity during the forward pass.The GEO head and encoder minimize LGEO, whereas the authorship head minimizes Lauth.
  • B.1 Gradient Reversal for Authorship Invariance: The encoder preserves GEO-useful information and suppresses authorship-predictive information, using a progressively increased reversal strength during query-disjoint training.This avoids imposing a strong adversarial signal before the authorship head becomes informative.
  • B.1 Gradient Reversal for Authorship Invariance: λmax = 1 and an 8,192-token context are used for fine-tuning, while inference requires neither the authorship head nor the GRL.The detector takes one page and returns only its GEO prediction.

B.2 Ablation Studies for IPT … C.1 Supplementary Settings

The supplementary sections report IPT’s sensitivity to the loss weight β, provide class-wise precision and recall listings, and clarify the units, deduplication, and overlap accounting used in GEO prevalence estimation.

  • B.2 Ablation Studies for IPT: β = 1 achieves the best F1, accuracy, and worst-group accuracy across β ∈ {0.5,1,2,4}.IPT is relatively stable across the tested β values.
  • B.2 Ablation Studies for IPT: β = 0.5 yields the smallest ∆FPR but lower overall performance.The ablation identifies a trade-off between minimizing ∆FPR and maximizing the other reported metrics.
  • B.3 Supplementary Results for GEO Attribution: Tables 16 and 17 list precision and recall values for all classes.The supplementary results cover both precision and recall across the complete class set.
  • B.2 Ablation Studies for IPT: Table 15 evaluates IPT’s loss-weight sensitivity using F1, accuracy, worst-group accuracy, and ∆FPR.Higher F1, accuracy, and worst-group accuracy are preferred, while lower ∆FPR is preferred.
  • C.1 Supplementary Settings: A Citation occurrence counts each extracted Citation URL use, whereas each distinct Citation URL is counted once after global deduplication by normalized target URL.Repeated citations on a page and citations by different pages create separate occurrences before URL-level deduplication.
  • C.1 Supplementary Settings: 539 distinct Citation URLs overlap between the two channel-specific target sets.The target-level overlap is distinct from occurrence-level overlap induced by pages shared between retrieval channels.
  • C.1 Supplementary Settings: 771 Citation occurrences arise from pages shared between the two retrieval channels, compared with 539 overlapping distinct Citation URLs.Of the 539 overlapping URLs, 526 occur on shared pages and 13 are cited by channel-exclusive pages from both channels.

C.2 Supplementary Results

The supplementary results report the exact values underlying Figure 1 in Table 19.

  • Table 19 reports the exact values underlying Figure 1.

D Supplementary Details for Citation URL Metrics

The supplementary audit defines URL Source Tier independently of retrieval and citation support, then deterministically derives Citation URL Verifiability from source tier and retrieval state. The audit pipeline evaluates extracted Citation URLs from detector-positive pages rather than the original search-returned Page URLs.

  • Metric Definitions: URL Source Tier characterizes publisher provenance independently of URL retrieval status and whether the destination supports a cited sentence.The frozen publisher-level rubric is specified in Table 20.
  • Audit Pipeline: Detector-positive pages yield extracted Citation URLs for auditing, while original Page URLs are the URLs returned by search channels.The workflow fetches Page URLs, detects GEO, extracts Citation URLs, and audits those extracted URLs for source tier and verifiability.
  • Audit Outputs: The audit reports both occurrence-level and distinct-URL-level Citation counts.These counts are presented in Table 18.
  • Metric Definitions: Citation URL Verifiability is derived deterministically from URL Source Tier and retrieval state.The complete mapping is specified in Table 21.
Loading 2608.16824v1…