PaperlayerView exact revision on arXiv

Source-linked AI summary

Scoring Without the Engine: Validating a Deterministic, Manipulation-Resistant Content Score for Generative Engines, End to End

Elisha Bajemon, Andre-Louis Rochet

arXiv:2609.07559v1cs.AI

TL;DR

The paper asks how to validate a cheap deterministic proxy for an expensive, rate-limited, non-stationary generative oracle. It uses adversarial gates, bounded skylines, and current-oracle causal re-measurement to validate a GEO content score. Historical anchors fail on modern engines, but the gate-enforced score limits adversarial inflation and is positioned as a quality filter rather than a citation predictor.

  • Problem

    Answer-side evaluation is causally strongest but expensive, rate-limited, non-stationary, difficult to reproduce, and costly to scale.

  • Method

    The protocol selects a deterministic score with falsification gates, bounds query-agnostic signal, re-measures external causal anchors, and fits candidates before held-out confirmation.

  • Results

    None of three strongest 2023 GEO levers moved citation on ten modern engine families, while the selected score limited calibrated-lever amplification to at most 6 points.

  • Takeaways & Limitations

    The surviving artifact is a manipulation-resistant quality filter, not a query-specific citation predictor, and its validation depends on gate-enforced behavior after historical anchors expire.

  • Takeaways & Limitations

    The score’s features have scope constraints including degenerate structural features on plain-text corpora, ASCII-only tokenization, and English-only rarity queries.

Abstract

from arXiv · show

How do you validate a cheap, deterministic proxy for an oracle that is expensive, rate-limited, and non-stationary? We present a protocol built on adversarial falsification gates (negative control, dose response, bounded amplification, duplication penalty, length neutrality) that define and select the proxy, fitted on a training split and confirmed held-out; around them it bounds what the proxy can never resolve, and re-measures external causal evidence on the current oracle rather than assuming it. We demonstrate it end to end on Generative Engine Optimization, where the proxy is a deterministic content score, and one step fails on that domain exactly as the protocol is built to detect: re-measuring the only published causal anchors (2023 effect sizes) on ten modern engine families shows their levers move citation on none, so the anchors are an expired external check; recalibrating to the near-zero modern vector strips the score of its lever-responsive components. What survives is the gate-enforced response surface. The gates buy a measured property: on a 500-source benchmark of adversarial edits, amplifying the score's calibrated levers gains an attacker at most 6 points, and decreases with dose; single-lever amplification is provably bounded, while the cap and cross-lever sub-additivity are empirical findings consistent with it. On detection, web-spam baselines dominate and out-of-distribution attacks evade the score, so the deployable filter layers it over them. A query-conditioned skyline bounds the score's citation signal (within-query Spearman 0.11), repositioning query-agnostic scores as quality filters rather than citation predictors. A query-leakage bug in our first ranking evaluation and a failed confidence flag are disclosed and corrected; every number reproduces offline from released artifacts at zero marginal API cost.

1 Introduction

The paper proposes a validation protocol for cheap deterministic proxies of expensive generative oracles and demonstrates it on GEO. Its results separate a manipulation-resistant quality score from query-conditioned citation prediction, while re-measurement shows historical causal anchors no longer transfer to modern engines.

  • Answer-side GEO evaluation is causally strongest but paid, rate-limited, non-stationary, difficult to reproduce, and increasingly costly with each site evaluated.
  • The protocol selects a deterministic proxy through adversarial falsification gates, bounds its query-conditioned ceiling, checks external causal evidence, and re-measures that evidence on the current oracle.
  • The query-agnostic artifact is a deterministic content score used as an auditable quality filter, while a query-seeing predictor addresses source citation for specific questions.
  • At most 6 points can be gained by amplifying calibrated levers, with gains shrinking under stronger attacks and not compounding across levers.
  • On unseen queries, a flexible content-only ranker reaches ρ = 0.10 versus 0.12 for the fixed score, whereas query conditioning roughly triples the measured signal.
  • None of three strongest 2023 GEO levers moved citation on ten modern engine families, so modern re-measurement yields an expired causal check and near-zero anchor vector.
  • The paper discloses and corrects feature-coverage, negative-control, and ranking-leakage bugs, while a dispersion-based confidence flag fails held-out confirmation.

2 Related work

The paper situates its contribution in evaluation-and-measurement research, deterministic text-quality and web-spam detection, and GEO. It distinguishes its causal, adversarially gated, per-weight-identifiable validation object from observationally validated citation scores.

  • The central research object is validating a deterministic proxy for a generative oracle under Goodhart risk, with GEO serving as the demonstration domain.
  • Generative engine optimization: Prior GEO work established causal effects for content edits, while later studies measured visibility answer-side or examined adversarial manipulation and ineffective tactics.
  • Generative engine optimization: The paper differs from observational citation-score work by calibrating against published causal effects, applying adversarial gates, and reporting per-weight identifiability.
  • Deterministic text-quality measurement: Deterministic readability, document-quality, web-spam, and corpus-filtering methods provide the lineage for the paper’s closed-form features and negative-control falsification gate.
  • The paper states that no prior work combines causal-effect calibration, adversarial selection gates, and per-weight identifiability for a deterministic content score.

3 Problem statement and assumptions

The paper separates query-agnostic score construction from query-conditioned citation prediction, formalizing each with different evidence requirements. Calibration is constrained by adversarial gates and assumptions about historical anchors, volume control, and determinism, while query-blindness imposes a ceiling on citation ranking.

  • Problem 1: The query-agnostic problem seeks an offline, auditable score aligned with published causal visibility gains and robust to single-lever amplification.The score is not required to predict which source an engine cites for a particular query.
  • Problem 2: Query-conditioned citation prediction instead ranks candidate-source visibility from the query, candidates, and engine family using held-out queries and engine families.
  • Query-blindness ceiling: Query-blindness imposes a ceiling because a score must order each source pair identically even when visibility varies across queries.The ceiling becomes nontrivial when pairwise visibility probabilities lie strictly between zero and one.
  • Scope: The paper tests whether a content-only model is near the best attainable within its function class, but this does not estimate the theoretical ceiling itself.
  • Calibration assumptions: Calibration is underdetermined by five anchors and eleven weights, so the paper characterizes admissible weights and uses falsification gates to select aggregation families.The quantile family is rejected for violating dose response despite performing best on one secondary transfer objective.
  • Calibration assumptions: The calibration assumptions include historical anchor validity, transport to deterministic edits, volume control, and deterministic computation under a pinned environment.The paper explicitly tests current-engine transfer rather than assuming historical anchors remain valid.

4 The scoring algorithm

The score combines eleven deterministic text features through a concave aggregation, freshness blending, and caps. Its design addresses stuffing and degenerate inputs, while feature redundancy and historical naming artifacts constrain interpretation.

  • Sub-components: The score uses eleven deterministic sub-components computed offline from text with no LLM, learned embeddings, or network access at scoring time.The implementation is deterministic under pinned libraries and seeds.
  • Sub-components: Information_density and shannon_entropy double-count one signal, with a joint pre-transform weight of approximately 0.42 and Pearson correlation 0.94 on GEO-Bench.A merge refit confirms that treating them as one construct does not materially change the split.
  • Sub-components: Two historical feature names misdescribe their formulas, while three block-structural features remain at degenerate constants for most plain-text sources.These properties motivate excluding the parse-failure proxy from the ranking study.
  • Aggregation: The concave transform reduces marginal returns as a single feature approaches its ceiling, limiting quote or statistic stuffing while rewarding balanced improvements.Its derivative scales as g′(u) ∝u^-1/2.
  • Aggregation: For a single-lever edit, concavity proves that dose-k amplification is at most k times the dose-1 response; the square-root transform attains equality from the floor before capping.
  • Aggregation: The selected square-root family passes the dose-8 saturation gate, with reported per-lever ratios from 0.55 for statistics to 0.96 for quotation and a maximum of 0.96.The bound follows from the transform family; the gate selects the family rather than creating the bound.
  • Aggregation: The square-root family empirically outperforms the raw-linear production family on dose-response and adversarial gates at equal anchor alignment.
  • Pipeline: The page-level score blends content with freshness and applies word-count caps of 35, 50, and 65 points below 100, 200, and 300 words; naturalness and brand trust remain separate.Domain rankings use a shrinkage estimate with prior 50 and strength 5.

5 Constraints first: gates, then anchors

The paper makes falsification gates the feasibility constraints that select the score, while causal anchors serve as a fitting aid and external check. It also diagnoses weaknesses in prior observational calibration and specifies volume-controlled anchor realizations and a held-out confirmation protocol.

  • Constraints first: gates, then anchors: The score is defined by falsification gates, while published causal anchors enter as an external check rather than determining the response surface.The gates test negative control, dose response, bounded amplification, duplication, and length neutrality; modern-anchor evidence is used to assess whether historic causal effects remain valid.
  • Constraints first: gates, then anchors: The prior observational calibration used n = 39 pages and collapses from in-sample Spearman 0.90 to about 0.20 under leave-one-out cross-validation.The strongest per-feature correlations were measured at n ≤20, while the headline composite correlation had Fisher CI95 [0.42, 0.80].
  • Anchors and volume control: Table 1 assigns dedicated feature carriers to Quotation, Statistics, and Cite-Sources, while Technical Terms has no dedicated carrier and is the weakest realized anchor.Dominant held-out responses include quotable_density +47.6 points, statistic_density +80.7, and citation_f1 +82.1.
  • Anchors and volume control: Volume-controlled edits prevent length-sensitive features from inflating every intervention and provide the basis for measuring score responses.Each intervention is paired with a content-neutral filler edit of the same added length.
  • Calibration as constrained alignment: Calibration fits aggregation families on training data, applies gates for selection, and uses the held-out split only for confirmation.Weights use regularized cosine alignment to the anchors, with nested selection of λ on an inner training split.
  • Validity criteria (gates): The five gates test negative control, non-declining dose response, bounded saturation, duplication resistance, and length neutrality; cross-lingual transfer remains a secondary comparison.Cross-lingual transfer is reported for every strategy but is excluded from the gate count and cannot replace feasibility constraints.

6 Strategy comparison

The gates, rather than anchor alignment alone, select the concave strategy: it is the only fitted family passing all five criteria, with held-out confirmation matching training-split selection. Intervention audits also exposed and corrected a negative-control bug, while identifiability analysis separates evidence-supported weights from underdetermined ones.

  • Strategy selection: The selected concave family is the only fitted strategy passing all five gates; alignment alone would select raw-linear candidates that fail dose response.The selected strategy’s worst dose step is −0.36, while alternatives fail at −0.69, −0.74, −1.21, and −2.27.
  • Strategy selection: Training-split recomputation selects the same sqrt family at 5/5 gates, while held-out values serve only as confirmation.The training-split Pearson value is 0.946, and the per-gate pass/fail pattern is identical across splits for fitted families.
  • Intervention testing: French transfer exposed a coverage gap in quotation features, which was fixed by adding attributed quoted-speech patterns in both languages.Before the fix, the French quotation edit produced −1.9 points because added quotes increased sentence count without matching.
  • Intervention testing: A 500-source benchmark shows that standard web-spam baselines handle degradation detection, while the score’s role is bounded inflatability under adversarial edits.The gates define intervention testing as a zero-cost unit-test harness for content features.
  • Intervention testing: A corrected negative-control edit changed the stuffing analysis: the earlier result tested letter noise, so all downstream analyses were regenerated with real keyword stuffing.Under the corrected edit, the quantile family’s negative control is −10.3.
  • Weight identifiability: Per-weight refits divide the score into pinned anchor weights, entropy weights forced nonzero by gates, and weakly identified weights admitted over broad intervals.The unregularized fit concentrates mass on four features, improves training alignment, and loses most of the stuffing response, which the gates veto.

7 The score as a manipulation filter: a gaming-detection benchmark

The benchmark positions the deterministic score as a bounded-inflation quality measure, not a general spam detector. Standard web-spam baselines detect degradation attacks, while the score limits gains from amplifying its calibrated levers.

  • Benchmark: The benchmark uses 500 clean sources per class, length-matched neutral controls, in-distribution attacks, and a 5% false-positive detection threshold.AUC is reported against dose-matched neutral filler so comparisons remain volume-controlled.
  • Bounded amplification: The best single-lever amplification gains +6.1 points at dose 1 and +3.5 points at dose 8, while the three-lever combination reaches +4.5 at dose 8.Statistics declines from +4.6 to +0.1, and cite-sources from +3.9 to +0.5.
  • Bounded amplification: The score’s single-lever bound is the empirical counterpart of a provable concave-transform bound, whereas the cap and cross-lever sub-additivity are measured properties.The shipped concave weighting owns this behavior; the prior linear weighting is not uniformly capped.
  • Scope: A released snapshot cannot determine how bounded score changes affect leaderboard position because that depends on domain clustering near the attacked domain.Measuring this requires a frozen production leaderboard outside the released GEO-Bench artifacts.
  • Detection division of labor: Web-spam baselines dominate detection: token repetition reaches AUC 1.000 on heavy stuffing, shingling reaches 0.988, and the score reaches 0.851.On pure duplication, shingling reaches 0.685 while the score is near chance at 0.529.
  • Positioning: The deployable configuration layers repetition and near-duplicate baselines for degradation-spam detection over the score for bounded quality filtering, editorial control, and regression testing.The score detects none of the tested out-of-distribution attack families and should not be called a manipulation detector.

8 Outcome validation with modern answer engines

Outcome validation finds modest but robust citation alignment for the score, while modern tests show that the strongest published GEO interventions no longer transfer. Recalibrating to the measured modern anchors degenerates, leaving the gate-enforced response surface rather than historical causal alignment.

  • Protocol: The outcome protocol compares unmodified, quotation-edited, and length-matched neutral-filler sources using paired PAWC-like citation differences.It runs under fixed prompts and controlled engine settings across modern answer engines.
  • Observational validation: The fixed score’s pooled within-query Spearman is 0.1142 across 777 records, rejecting the no-signal null but remaining modest for ranking five topically relevant sources.The prior observational weighting reaches 0.137, and the score is positioned as a secondary validity signal rather than the headline result.
  • Calibration: Freshness increases the pooled score from 0.114 to 0.132, a paired within-query difference of +0.018, but its causal status remains unresolved.The blend is retained because of observational evidence and continuity with the deployed configuration.
  • Causal transfer: The strongest published 2023 GEO interventions—Quotation, Statistics, and Cite-Sources—move citation in the published direction on none of the fully covered modern-engine tests.A pooled cite-sources effect is nominally negative, and the authors make no strong causal-transfer claim.
  • Causal transfer: The controlled test regime uses five curated sources, a fixed citation-format prompt, and a PAWC-like share, so non-transfer could partly reflect the simplified frame rather than the engines.The constraint-set validity of the score’s gates does not depend on this causal claim.
  • Recalibration: The modern pooled anchor vector is near zero, and recalibration selects maximum regularization, leaves only 4/5 held-out gates, and makes alignment claims untestable at the 1/3! permutation floor.The unregularized variant removes the lever-carrying features, showing why expired anchors should not determine current calibration.

9 The query-conditioned ceiling, in brief

The query-agnostic score has a measurable but bounded citation signal, and query conditioning recovers substantially more signal. A leakage bug initially overstated fitted-model performance until query-disjoint evaluation corrected it.

  • Content-only ceiling: No fitted content-only model beats the fixed auditable score on unseen queries, consistent with a query-agnostic ceiling.The study treats 229 unique queries as a design constraint and does not claim fitted-model superiority at this scale.
  • Query conditioning: An open-weights cross-encoder reaches 0.388 without fitting, while the full fitted model reaches 0.348 ± 0.013, roughly tripling the measured signal through query conditioning.The comparison quantifies the gain from modeling the query explicitly.
  • Leakage audit: A query-blind probe reached 0.48 under leaked folds but fell to 0.065 under query-disjoint folds, revealing memorization of recurring per-query outcomes rather than learned relevance.The corrected ten-arm extension reports 0.104 under query-disjoint folds.

10 Threats to validity

The paper’s validity claims are bounded by calibration, feature, engine, attack-surface, and sample-size constraints. These boundaries limit interpretation of causal transfer, detection coverage, ranking superiority, and generalization beyond the controlled artifacts.

  • Calibration: Calibrating and gating on the same interventions risks teaching to the test, mitigated by nested selection, ridge anchoring, held-out splits, unseen adversarial variants, and a transfer corpus.The aggressive fit is rejected when its alignment gains sacrifice the gates.
  • Features and corpus: Entropy saturates at 100 on 84% of base sources, while block-structural feature weights are unidentified on plain-text GEO-Bench and tokenization is ASCII-only.These constraints weaken interpretation of absolute feature levels and limit feature generality.
  • Outcome scale and engines: The ten-engine outcome protocol is a controlled five-source, fixed-prompt PAWC-share proxy rather than open-web competition with retrieval and deduplication.Hosted snapshots may drift, and the proprietary replication is single-run because temperature and seed are unavailable.
  • Attack surface: The gaming benchmark cannot exhaustively cover attacks, and the score detects none of the tested out-of-distribution families.This limits any interpretation of the score as comprehensive manipulation detection.
  • Ranking study: The ranking study has only 229 unique queries, so the paper claims no superiority over the best single relevance signal and notes that the freshness increment is observational, not causal.Residual dataset artifacts may remain despite removal of three disclosed leaks.

11 Reproducibility

The reproducibility package makes the analyses independently rerunnable from released artifacts, while distinguishing exact offline reproduction from potentially drifting fresh engine recollection.

  • The public package includes data, analysis code, reproduce.sh, and a consistency audit for reproducing the reported results.The release is hosted at the paper’s stated GitHub repository.
  • Exact reproduction uses released outcome records, feature captures, caches, frozen weights, result JSONs, and no engine access.Fresh recollection from hosted engines remains possible at zero marginal cost, but snapshots may drift.

12 Conclusion

The conclusion scopes the score as a bounded, auditable quality measure layered over standard detection baselines, not a replacement for live causal evaluation or per-query citation prediction.

  • Live engines remain the causal ground truth; the paper contributes a validation protocol and its demonstrated outputs.The protocol produces a deterministic, auditable content score with deliberately scoped claims.
  • Spam-style degradation detection belongs to standard baselines, while the score contributes bounded, auditable quality measurement and the deployable filter layers both.The paper reports that baselines dominate detection and out-of-distribution attacks evade the score.
  • The protocol applies beyond GEO by anchoring proxies to causal evidence, testing them adversarially, bounding them with a conditioned skyline, and re-measuring the anchors.The stated scope is any deterministic proxy standing in for an expensive generative oracle.

A Formal specification of the eleven sub-components

The appendix specifies the scorer’s eleven bounded components, tokenization conventions, fallback behavior, caps, and calibration robustness, while documenting implementation artifacts that limit interpretation.

  • A Formal specification of the eleven sub-components: The scorer is implemented in the released regex_only path, with page text x, clamping, [0, 100] feature ranges, and documented degenerate defaults.The appendix states that unavailable quantities receive documented fallback values rather than causing failure.
  • A.1 Tokenization conventions: Information-theoretic features use lower-cased ASCII word tokens of length at least 3 and a fixed bilingual English–French stop list.Accented characters terminate tokens under this convention, with cross-lingual consequences noted elsewhere.
  • A.1 Tokenization conventions: The information-density feature combines Shannon entropy, lexical diversity, and semantic sophistication with weights 0.70, 0.20, and 0.10.Its formula is fID(x) = 0.70 fshannon(x) + 0.20 Slex(x) + 0.10 Ssem(x).
  • A.1 Tokenization conventions: The entropy and information-density artifacts share approximately 0.417 of pre-transform weight and correlate 0.94, so treating all eleven weights as independent constructs misleads.The paper retains both as separate released artifacts while disclosing this double counting.
  • A.1 Tokenization conventions: Lexical diversity combines MTLD, Herdan’s C, an inverted Maas index, sentence-length variation, and register consistency; sophistication uses rarity, syllables, long-token share, title concepts, and affixes.French sophistication scoring is limited because Zipf lookup uses the English word list, while no title concepts forces Ssem = 50.
  • A.1 Tokenization conventions: Quotable-density scoring uses patterned definitional, enumeration, and attributed-quotation sentences, but q ≤10 creates a long-text ceiling r ≤10/ns.Texts with fewer than three sentences return 0, and very long pages cannot reach the target density regardless of content.
  • A.1 Tokenization conventions: The title–paragraph example gives d = 0.969 and T0 = 11.4 for a technical match versus d = 1.0 and T0 = 7.1 for an unrelated paragraph, showing weak discrimination.The composite assigns this term only 1/17 of its weight.

D Weight identifiability: full analysis

Weight identifiability is structurally underdetermined by alignment alone, so the paper uses gates, prior information, and profile analyses to distinguish sharply identified carriers, control mass, and weakly identified remainder.

  • With five anchors and eleven weights, no per-weight optimality proof is possible because the system is structurally underdetermined.The paper asks reviewers to distinguish aggregate support from unjustified coordinate-level claims.
  • Alignment is constant along the response-matrix null space, whose dimension is at least six, so gates or prior information are required for per-weight statements.This is Proposition 2’s formal consequence for d = 11 and rank at most 5.
  • The identifiability analysis combines leave-one-feature-out refits with profile refits that fix one weight across a grid and refit the remainder on training data.Admissibility is evaluated using held-out criteria.
  • The resulting plateau has three regimes: sharply identified weights have narrow admissible windows, while weakly identified weights remain broadly admissible.The released values are therefore justified by different kinds of evidence rather than one universal optimization principle.
  • Quotable_density is the sole sharply identified carrier: removing it drops alignment from 0.949 to 0.688, and its admissible interval is [0.12, 0.16].Its released weight 0.124 is pinned by the anchor data rather than treated as a tuned preference.
  • Citation_f1 is capped above at 0.02 because its +82-point anchor response would otherwise dominate intervention ordering and misalign the ensemble.The released vector passes the gates jointly; coordinate intervals are slices, not a product box.
  • The entropy family has little anchor alignment but supplies the negative-control response to keyword stuffing, making its retained mass serve a different evidentiary role.Its split across correlated artifact names is reported as immaterial to the control function.
  • Five features admit weights in [0, 0.50], while alignment-only fitting improves training cosine to 0.957 but worsens held-out Pearson to 0.920 and weakens the negative control.The paper therefore retains some weights from the prior production vector and rejects technical-terminology density to avoid teaching to the test.

E The query-conditioned citation predictor: full study

The study separates query-agnostic content scoring from query-conditioned citation prediction, showing that content-only models do not beat the fixed score while query information recovers substantially more signal. Strict query-disjoint evaluation reduces apparent transfer and exposes leakage-driven conclusions, supporting quality-filter rather than citation-prediction use for the fixed score.

  • Study design: Citation is evaluated as a within-query source-ranking problem, so the study compares fixed, query-blind, and query-conditioned models under multiple fold constructions.The feature stack includes deterministic content features, optional cached open-weight models, ridge, pairwise ranking, GBM, and LambdaMART.
  • Leakage audit: Fold leakage allowed the same queries across engine arms, inflating flexible-model results and reversing the apparent dispensability of model-based features.The corrected protocol folds over unique queries rather than engine-query records.
  • Content-only prediction: ρ = 0.104 for the query-blind LambdaMART versus 0.119 for the fixed score, while ridge reaches 0.075, so no fitted content-only model beats the auditable baseline on unseen queries.The query-blind LambdaMART also has hit@1 0.24 versus 0.25 for the fixed score.
  • Query-conditioned prediction: ρ ≈ 0.39 is reached with query conditioning versus 0.119 without it; the strongest single signal is the open-weights cross-encoder at ρ = 0.388.The regex-only ranker reaches 0.222 and recovers 0.10 of the 0.119 →0.388 span, while fitted models cluster at 0.36 to 0.38.
  • Engine transfer: Strict leave-one-engine-out transfer averages 0.337, with engine-and-query holdout required for the defensible estimate; naive engine-only holdout is inflated by shared queries.Strict values range from 0.25 for gpt-oss to 0.43 for gpt-5.4-mini, while the naive regime has mean 0.626.
  • Exploratory analyses: Query-tag patterns are unstable across dataset extensions: the archival-versus-volatile gradient flattens, and the fact-versus-opinion contrast remains noise.At ten arms, law and government is 0.35 ± 0.04 versus health 0.35 ± 0.05, while fact versus opinion is 0.36 ± 0.02 versus 0.30 ± 0.04.
  • Deployment implication: The query-agnostic score is positioned for audits, leaderboards, editorial control, and regression testing, whereas citation prediction requires query-conditioned models.The fixed score’s supported claims are its gate-enforced response surface, historical alignment, and gaming resistance rather than per-query prediction.
  • Feature diagnostics: The feature diagnostics document ceiling saturation, mechanically redundant entropy features, and a quotable-density weight of 0.124 after the coverage correction.Re-evaluation of the corrected capture passes all five gates with held-out anchor Pearson 0.970.
Loading 2609.07559v1…