PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting What Answer Engines Cite: Open Decision Models, Rerankers and a Distilled 4B Student on Ten Engines under Controlled Retrieval

Elisha Bajemon, Andre-Louis Rochet

arXiv:2610.38090v1cs.IRcs.AI

TL;DR

Generative engine optimization needs scorers that predict which candidate pages answer engines cite and how much they contribute. This paper compares rerankers, decision models, and distilled students on GEO-Bench queries answered under controlled retrieval. The engines agree at 0.704, while Jev and two open 27B models are the strongest scorers, and the 4B student is not distinguishable from them on 229 queries.

  • Problem

    Directly measuring which candidate pages answer engines cite and how much they contribute is paid, slow, and non-deterministic, motivating prediction before querying engines.

    Source: §1 · p. 1

  • Method

    The paper evaluates pretrained rerankers, hosted and open typed decision models, and distilled Qwen3-Reranker students on GEO-Bench queries whose five candidate sources are supplied to ten engines.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Engine agreement is 0.704; Jev and the open 27B models score 0.575–0.578, while the 4B student scores 0.542–0.549 and is not distinguishable from them.

    Source: §6 · pp. 4–5

  • Takeaways & Limitations

    Under controlled retrieval, decision models comparing candidates together are the strongest predictors, and the released 4B student is not distinguishable from them on 229 queries.

    Source: §9 · p. 8

    The benchmark measures citations among fixed sources supplied in prompts and does not establish transfer to engines that retrieve pages from the live web.

    Source: §7 · p. 7

Abstract

from arXiv · show

Generative engine optimization needs a scorer that ranks candidate pages by the share of an answer they will receive as citations. We measure pretrained rerankers, a hosted typed decision model (Jev), eight open decision models and distilled students on GEO-Bench queries answered by ten engines that receive the same candidate sources in their prompt. Where several engines answer the same query, they agree with one another at a within-group Spearman correlation of 0.704. Jev and two open 27B decision models, used without training, are the strongest scorers at 0.575 to 0.578. A Qwen3-Reranker-4B fine-tuned with low-rank adapters on teacher labels for other GEO-Bench queries, released as GEO Citation Ranker 4B, reaches 0.542 to 0.549 across three seeds, and its differences with the 27B models and with Jev are not distinguishable from zero on 229 queries. Most of the remaining distance between the student and Jev disappears when Jev scores one passage at a time, as the student does. Students trained on teacher labels and on observed engine citations reach the same level, and labels from Jev alone match the two-teacher target at 0.6B. We release the model, the benchmark, the predictions of every reported scorer and the code.

1 Introduction

Generative engine optimization asks which candidate pages answer engines cite and how much they contribute. This paper evaluates scorers against engine citations and reports their agreement and performance on a shared benchmark.

1 Introduction

  • GEO seeks to predict which candidate pages an answer engine will use and how much of its answer they receive as citations.Source: §1 · p. 1
  • Because direct engine measurements are paid, slow, and non-deterministic, practitioners use models to predict answers before querying an engine.Source: §1 · p. 1
  • The paper evaluates scorer configurations against engine agreement and citation labels, including open 27B decision models and distilled Qwen3-Reranker students.It also examines how candidate presentation and teacher choice affect student performance.Source: §1 · p. 2

2 Related work

Prior GEO work studies content changes and citation behavior, while related work also covers reranking, distillation, and judging. This paper applies typed decision models to comparative citation scoring.

Generative engine optimization

  • GEO research introduced GEO-Bench and measured citation gains from content edits, while later studies examine conversational SEO, source competition, citation influence, and citation failures.Bajemon and Rochet (2026) found query-agnostic content scores weakly predict citation on ten engines; this paper uses answers collected for that study.Source: §2 · p. 2

Citation in generated answers; LLM rerankers and distillation

Judges

  • LLM judges can be sensitive to candidate order, while typed decision models previously addressed binary criteria; this paper applies them to graded comparative evaluation (Wang et al., 2023).Source: §2 · p. 2

3 Problem statement

The task ranks five prompted candidate sources by their citation share in an engine’s answer. The benchmark scores predictions against citation-share labels and uses cross-engine agreement as a reference.

3 Problem statement

  • A scorer sees the query and five candidate source texts, but not the engine answer, and ranks the sources by their citation shares.Source: §3 · pp. 2–3
  • Citation share is the normalized, position-adjusted word-count share of answer sentences citing each source (Aggarwal et al., 2024; Bajemon and Rochet, 2026).Source: §3 · p. 3
  • The benchmark score is mean within-group Spearman correlation between predicted and labeled source rankings, excluding groups whose five labels are all equal.Source: §3 · p. 3
  • Cross-engine agreement scores each engine’s labels against the other engines’ mean labels for the same query, providing a reference rather than an upper bound.It is defined only for queries answered by at least two engines.Source: §3 · p. 3

4 Benchmark and scorers

The benchmark uses GEO-Bench queries with fixed candidate sources supplied to answer engines. It compares pretrained rerankers, typed decision models, and students trained on teacher labels.

4 Benchmark and scorers

  • The evaluation covers 229 GEO-Bench queries and 1,220 non-degenerate groups across ten engines, each given the same five sources without retrieving documents.Sources are represented by their first 320 words; the setting measures citation under controlled retrieval.Source: §4 · p. 3
  • Pretrained NIM, bge-reranker-v2-m3, and Qwen3-Reranker models are scored with default inference code and no training on the same 320-word passages.Source: §4 · p. 3
  • Typed decision models assign each passage an expected score from a six-level citation question, using listwise or pointwise prompts and 250-word passage truncation.The comparison includes hosted Jev and eight open models, with shared questions and no prompt tuning.Source: §4 · p. 3
  • Qwen3-Reranker students at 0.6B, 4B, and 8B are trained on 757 five-source groups labeled by Jev and an LLM judge, with the 4B seed-42 model released as GEO Citation Ranker 4B.Training uses pairwise logistic loss and rank-16 adapters; the 4B run takes about two hours on one RTX 3090.Source: §4 · p. 4

5 Evaluation protocol

The evaluation protocol resamples by query and compares scorers on paired groups. It also checks whether shared pages between training and evaluation materially change student scores.

5 Evaluation protocol

  • 95% percentile bootstraps resample all groups for a query together; paired scorer differences are indistinguishable when their 95% interval covers zero.Equivalence within 0.05 requires the 90% interval of the difference to lie inside that margin under two one-sided tests.Source: §5 · p. 4
  • Top-1 accuracy is reported as a second metric, alongside the benchmark scoring procedure.The passage describes it as the share of groups where the scorer’s top source is a most-cited source, but the definition is truncated.Source: §5 · p. 4
  • Because the 77 queries answered by all ten engines contribute most groups, the protocol also reports mean scores per engine and per query.Source: §5 · p. 4
  • Removing 57 groups containing passages with shared first 40 words changes any student score by at most 0.006 and lowers the 4B student by 0.003.Fifteen evaluation passages shared their first 40 words with a training source; observed-citation students were evaluated by five-fold cross-validation, with each group scored by a model that had not seen its query.Source: §5 · p. 4

6 Results

GEO Citation Ranker 4B reaches 0.542–0.549 across three seeds, and its differences from Jev and the open 27B models are not distinguishable from zero. Engine agreement is higher at 0.704, while scoring Jev one passage at a time brings it closer to the student.

6 Results

  • 0.704 is the within-group Spearman agreement among engines on 1141 groups with multiple engines, exceeding Jev by 0.126 and the 4B student by 0.159.Adding source position changes the student by +0.011 [−0.002, +0.025] and Jev by −0.013 [−0.023, −0.005], so position does not explain the scorer-to-agreement distance.Source: §6 · p. 4, Table 1 · p. 5
  • 0.578 and 0.575 are the scores for untrained JEV-27B and pplx-decider-27B, against 0.575 for hosted Jev.Their paired differences with Jev are +0.003 [−0.034, +0.036] and 0.000 [−0.028, +0.031].Source: §6 · p. 4, Table 1 · p. 5
  • 0.542, 0.546 and 0.549 are the three-seed scores for GEO Citation Ranker 4B; its differences from Jev and the 27B models cover zero.Fine-tuning adds 0.090 [0.059, 0.121] over the same reranker without training.Source: §6 · p. 5, Table 1 · p. 5
  • −0.046 [−0.087, −0.009] is Jev’s score change from listwise to pointwise scoring; under pointwise scoring, its difference from the student is +0.012 [−0.026, +0.056].The student scores one passage per call by construction; Table 2 compares the decision models under both protocols.Source: §6 · p. 6, Table 2 · p. 7
  • 0.467, 0.444 and 0.457 are the 0.6B student scores trained on Jev-only, judge-only and mean labels, respectively; no pairwise difference excludes zero.At 4B, 189 training queries yield 0.537 against 0.542 with all 757; a student trained on observed citations reaches 0.541.Source: §6 · p. 6, Table 3 · p. 7
  • 0.085 [0.051, 0.119] is the 4B student’s advantage over the 0.6B student; the 8B student is not above the 4B at seed 42.The 8B seeds differ by 0.026 [0.008, 0.046], whereas the three 4B seeds do not differ.Source: Table 1 · p. 5, §6 · p. 6

7 Threats to validity

The measurements use controlled retrieval, so they test which provided source an engine cites rather than whether it retrieves a page. Their scope also depends on the teachers, inputs, benchmark, and collection period.

7 Threats to validity

  • Controlled retrieval removes retrieval noise from labels but does not establish transfer to engines that search the web.Testing that transfer would require answers collected from live search products.Source: §7 · p. 7
  • Decision models see 250 words per passage, compared with 320 for engines and students, a difference that can only work against the models.The student targets combine Jev and one LLM judge using one rubric; at 0.6B, Jev labels alone train a student as well as the two-teacher target.Source: §7 · p. 7
  • The benchmark is English, draws on one source dataset, uses five candidates per query, and includes four engines from one vendor.The decision-model state and questions were written once and not tuned; engines change, and the labels describe answers collected in 2026.Source: §7 · p. 7

8 Reproducibility

The release includes the 4B model, benchmark data, scorer predictions, and code, enabling offline reproduction of reported analyses. Teacher labels are not released, and the authors disclose employment by a generative-engine-visibility services provider.

8 Reproducibility

  • The release includes the 4B model and adapters for two other 4B seeds and the 0.6B student, plus the benchmark and analysis code.The benchmark contains 1,145 evaluation passages, 6,345 citation labels, fold assignments, predictions for 36 scorer configurations, and 757 unlabeled training groups.Source: §8 · p. 8
  • The tables and figure can be regenerated offline at no API cost, while retraining requires labeling groups with Jev for about 0.10 USD.The released code recomputes correlations, paired differences, engine agreement, baselines, accuracy, means, and overlap checks.Source: §8 · p. 8
  • Teacher labels are not released, although the LLM-judge rubric is provided in Appendix B.Source: §8 · p. 8
  • The authors are employed by TW3 Partners, which offers generative engine visibility services under the Citead name.Source: §8 · p. 8

9 Conclusion

Under controlled retrieval across ten engines, decision models comparing candidates together are the strongest predictors of citations. The released 4B student is not distinguishable from the 27B models on 229 queries, and the shared benchmark supports comparison of new scorers against engine agreement.

9 Conclusion

  • Decision models comparing candidates together, hosted or open, are the strongest predictors of citations across ten engines under controlled retrieval.Source: §9 · p. 8
  • Under controlled retrieval across ten engines, GEO Citation Ranker 4B, distilled from teacher labels, was not distinguishable from the strongest joint-comparison decision models on 229 queries; labels cost a few cents.Source: §9 · p. 8
  • The engines agree with one another well above every measured scorer, leaving room for scorers that compare candidates jointly.Source: §9 · p. 8
  • New scorers can be evaluated on the same groups against engine agreement and the same intervals.Source: §9 · p. 8

A Engines

Engine-level results vary across engines, while the 4B student remains above the NIM reranker on every engine. Table 4 describes engine coverage and scorer correlations alongside citation-share sparsity.

A Engines

  • Table 4 reports queries answered, non-degenerate groups, zero-share prevalence, and mean within-group Spearman for four scorers per engine.Source: Table 4 · p. 11
  • The 4B student ranges from 0.446 on llama-3.3-70b-instruct to 0.607 on gemma-4-31b-it, and stays above the NIM reranker on every engine.Source: A Engines · p. 11
  • With equal weight per engine, the 4B student scores 0.534, Jev 0.572, and JEV-27B 0.576.Source: A Engines · p. 11
  • gpt-oss-120b assigns zero citation share to 67% of sources, leaving 44 non-degenerate groups out of 80 queries.Source: A Engines · p. 11

B Prompts and rubrics

The prompts ask scorers to estimate each passage’s relative citation share for a query. Decision models, LLM judges, and students use distinct scoring formats and input setups.

B Prompts and rubrics

  • Decision models compare five passages truncated to 250 words, assigning citation-share levels from 0 (not cited) to 5 (dominant); a pointwise protocol scores one passage.The rubric prioritizes relevance, then citable facts, statistics, quotability, apparent authority, and self-containment, without rewarding length.Source: B Prompts and rubrics · p. 11
  • LLM judges receive five passages in seeded random order and score each from 0 to 100 for expected citation share relative to the others.The rubric prioritizes relevance and citable information, then figures and dates, quotability, authority, and self-containment; it also accounts for redundancy and ignores order.Source: B Prompts and rubrics · p. 11
  • Student inputs pair a query with one passage, and the score is the log-odds of the “yes” token against the “no” token.The prompt asks whether the passage would be cited by a generative answer engine writing an answer to the query.Source: B Prompts and rubrics · p. 11

Related papers

Loading 2610.38090v1…