PaperlayerView exact revision on arXiv

Source-linked AI summary

Octen-WebGR: Scaling Generative Retrieval to 200 Million Web Pages

Guangwei Xu, Wenkai Zhang, Yawen Liu, Dongxu Zhang, Xijie Xu, Kuan Zou

arXiv:2610.35256v1cs.IR

TL;DR

Navigational search must rank a single target page first, but click tables cannot handle unseen queries and dense retrievers can rank another page above the target. Octen-WebGR trains a language model to generate identifiers for URLs, and it outperforms the click-enhanced dense baseline overall while the two approaches generalise on complementary query and page cases.

  • Problem

    Click tables cannot answer unseen query formulations, while dense retrievers often rank a different URL above the navigational target.

    Source: §1 · p. 2

  • Method

    Octen-WebGR trains a language model on query–URL clicks to generate two-token semantic identifiers mapped through a table to real URLs.

    Source: §1 · p. 2

  • Results

    Octen-WebGR-4B ranks the target first for 59.6% of navigational traffic, versus 46.5% for a click-enhanced dense retriever; the approaches generalise on complementary query and page cases.

    Source: §1 · p. 2, §13 · p. 19

  • Takeaways & Limitations

    Generative retrieval can serve as a navigational index for 201 million web pages, with embeddings remaining better for pages reached by few queries.

    Source: §13 · p. 19

    New queries make up a third of navigational traffic, and Octen-WebGR-4B ranks their targets first 10.8% of the time; more than half of those targets are outside its universe.

    Source: §12 · p. 18

Abstract

from arXiv · show

Navigational search aims to find a single target page among hundreds of millions of web pages. Click tables work well for queries seen before but cannot generalise to new formulations, while embedding retrievers generalise across content but often rank navigational targets poorly. We introduce Octen-WebGR, a generative retrieval framework that stores web pages in the parameters of a language model: given a query, the model generates a two-token semantic identifier constrained to a table of real URLs. We develop Octen-WebGR-4B, which indexes 201 million web pages, and evaluate it on real navigational traffic from the month after training. Generative retrieval is most effective on new ways of asking for familiar pages: for unseen queries targeting pages with substantial interaction history, it ranks the correct page first 30.9% of the time, against 17.2% for a dense retriever trained on the same clicks (21.4% with a click prior). Embedding retrieval keeps an advantage on rarely visited pages, so the two approaches generalise on complementary sides, the query and the page. Over all navigational traffic, Octen-WebGR reaches 59.6% top-1 accuracy, against 46.5% for a click-enhanced dense retriever and up to 37.5% for frontier LLMs answering from memory, which cost at least 40 times as much per query. The 4B model runs on a single GPU with 10 GB of memory, compared with 412 GB for exact dense retrieval. Scaling experiments with up to 32 million URLs show that, when every query-page pair is seen equally often, the model size needed grows sublinearly with index size, and that retrieval is constrained more by page exposure than by model capacity. We release Octen-WebGR-4B and Octen-WebGR-0.6B with their identifier tables and inference code.

1 Introduction

Navigational search must rank one destination page first, but click tables cannot handle unseen queries and dense retrievers often rank a different page above the target. Octen-WebGR trains a language model to generate identifiers for real URLs, and its results show stronger overall ranking and complementary generalisation across queries and pages.

1 Introduction

  • A click table has no answer for unseen queries, which account for a third of navigational clicks, while dense retrieval often puts another URL above the target.For almost a third of navigational traffic, dense retrieval has the target in its first ten results but ranks another URL first, often another page of the same site.Source: §1 · p. 2
  • Web-scale generative retrieval must assign unique identifiers across hundreds of millions of pages, retain what it learns during training, and learn pages reached by few queries.The paper presents Octen-WebGR as the first generative retriever over hundreds of millions of individual web pages.Source: §1 · p. 2
  • Octen-WebGR assigns each URL a unique two-token semantic identifier and trains on query–URL clicks so a language model can generate the clicked page’s identifier.A table maps generated identifiers back to URLs; the 4B model indexes 201,231,792 URLs, and the 0.6B model indexes 31.3 million.Source: §1 · p. 2
  • 59.6% of navigational traffic has its target ranked first by Octen-WebGR-4B, against 46.5% for a click-enhanced dense retriever and at most 37.5% for frontier LLMs.Going from 2 million to 201 million URLs costs Octen-WebGR 4.7 hit@1 points, versus 22.2 for dense retrieval.Source: §1 · p. 2
  • For new queries targeting pages with many training queries, GR ranks the target first 30.9% of the time, versus 17.2% for dense retrieval and 21.4% with a click prior; embeddings do better on rarely clicked pages.The approaches generalise on complementary sides: GR on new ways of asking, embeddings on pages with little click history.Source: §1 · p. 2

2 Preliminaries

Dense retrieval compares query and document vectors, while generative retrieval produces document identifiers from learned samples. Semantic identifiers group similar documents by a shared prefix, and constrained decoding ensures outputs correspond to valid identifiers.

Dense retrieval

  • Dense retrieval encodes queries and documents as vectors and returns documents with the most similar vectors, but exact search costs memory and time proportional to the index size.As the index grows, documents can come close to a query by chance; prior work found dense retriever accuracy falls faster than sparse retrieval accuracy.Source: §2 · p. 3

Generative retrieval

  • Generative retrieval learns document identifiers from query–document samples and produces an identifier one token at a time, with corpus knowledge stored in model weights.Learning new documents can erode what the model knew about older ones.Source: §2 · p. 3

Semantic identifiers

  • A semantic identifier is a short code sequence whose shared prefix groups similar documents, but ordinary residual quantisation can assign duplicate identifiers.Octen-WebGR’s method for ensuring unique identifiers is described in Section 4.1.Source: §2 · p. 3

Constrained decoding

  • Constrained decoding uses a trie of valid identifiers so each generated token continues a valid identifier and every output names an existing document.Removing an identifier from the trie removes its document from every possible output.Source: §2 · p. 3

3 Task and Data

The paper evaluates retrieval on navigational clicks, using a target defined by click concentration and three URL universes with different coverage. Training and evaluation use proprietary click data split across time.

3.1 Task definition

  • A query is navigational when its most-clicked target receives at least 10 clicks and at least half of its clicks; hit@k is the probability that this target appears among the top k results for a randomly drawn navigational click.Clicks are sampled in proportion to traffic, and a target outside a system’s universe counts as a miss.Source: §3.1 · pp. 3–4

3.2 Click data

  • The training signal is a proprietary query–URL click dataset with aggregated click counts and no user identifiers; training uses data through August 2026, July traffic is for development, and September traffic is for evaluation.Queries are normalized by collapsing whitespace and lower-casing; URLs are lower-cased and stripped of the scheme, leading www., and trailing slash.Source: §3.2 · p. 4

3.3 URL universes

  • The main universe U201M contains 201,231,792 URLs and covers 79.2% of September navigational traffic, compared with 74.9% for U100M; U31M contains 31.3 million URLs.U201M is drawn from 349 million clicked URLs, and the training data restricted to it contain 1.13 billion query–URL pairs over 479 million queries.Source: §3.3 · p. 4

4 Method

Octen-WebGR fine-tunes decoder-only language models to generate two-token identifiers for URLs from queries. Its method builds unique semantic identifiers, trains on weighted click pairs, constrains decoding to valid identifiers, and adapts identifiers when the URL universe grows.

4.1 Semantic identifiers

  • Each URL receives a two-token semantic identifier, with each code drawn from 0 to 65,535; similar URLs share a first code and unique pairs distinguish individual URLs.The codes quantise URL embeddings, and a capacity-constrained assignment ensures unique pairs; 63.5% keep their nearest residual centroid and 96.2% receive one of their eight nearest.Source: §4.1 · pp. 4–5

4.2 Training samples and objective

  • Training samples pair a query with a clicked URL’s two identifier tokens, weighted so clicks shape URL choice within a query and query sampling grows with the square root of query clicks.A navigational floor gives dominant pairs a minimum number of samples, while document-indexing samples map URL text to the page identifier; URL text includes the URL and up to five popular queries.Source: §4.2 · pp. 5–6

4.3 Inference

  • At inference, the model uses width-20 beam search over the identifier-table trie, allowing only codes that continue valid identifiers; lookup maps the 20 generated identifiers to existing URLs.The query is lower-cased, whitespace-normalised, and truncated to 32 tokens; each identifier is scored by the sum of the log-probabilities of its two codes.Source: §4.3 · p. 6

4.4 Extending the universe

  • When extending from U100M to U201M, existing codes are kept and each first-level group is capped at 16,384 URLs so crowded groups spill into a nearby group.Despite the cap, 91.9% of new URLs receive their nearest first-level centroid; U31M shares the U201M identifiers.Source: §4.4 · p. 6

5 Training

The section details training recipes for Octen-WebGR-4B and 0.6B and defines exposures and cumulative learning rate for later forgetting analysis. It also describes the training rounds and schedules used for the released models.

5.1 Optimisation

  • Octen-WebGR-4B uses AdamW and a warmup–stable–decay schedule; each 3.1-second step processes about 16,000 samples across eight H200 GPUs.Samples are packed into two 16,384-token rows per GPU, for 262,144 tokens per step; query–URL pairs are randomly permuted within each round.Source: §5.1 · p. 6

5.2 Exposures and the decay window

  • An exposure is one occurrence of a query–URL pair, and its cumulative learning rate is the area under the learning-rate curve that follows it.This quantity is large for exposures early in a run and small for exposures inside the decay window; Section 8 shows that it governs forgetting.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Training rounds of Octen-WebGR-4B

  • Octen-WebGR-4B starts with URL-string pre-training, then receives one SID round on U100M and four on U201M, totaling 147.6K steps and 39B tokens.The SID training used 1,019 H200 GPU-hours, or 127 hours on eight GPUs; rounds 3 and 4 place frequent navigational queries near the end of training.Source: §5.3 · pp. 6–7

5.4 Learning-rate schedule

  • Re-warming to 10−4 temporarily reduced hit@1 on TRAFFIC-JUL by 4.6 points at round 2 and 3.6 points at round 3.The round-4 constant-rate control ended 0.34 points of hit@1 below the released arm.Source: §5.4 · p. 7

5.5 Training Octen-WebGR-0.6B

  • Octen-WebGR-0.6B is trained on U31M alone, without URL-string pre-training or document-indexing samples, using a simpler warmup-and-linear-decay recipe.It runs for 43.9K steps—four passes over its samples—in 8.9 hours on eight H200 GPUs.Source: §5.5 · p. 7

6 Evaluation on Search Traffic

On September navigational traffic, Octen-WebGR ranks the target first more often than the evaluated baselines, with its gains concentrated on new queries for familiar pages. Held-out tests and combinations with embedding retrieval show complementary strengths across familiar and cold targets.

6.1 Evaluation data

  • TRAFFIC-SEP samples 50,000 September clicks with replacement from queries with at least 10 clicks, weights by clicks, and is entirely after training; TRAFFIC-JUL is the development set used to choose section thresholds and weights.TRAFFIC-SEP contains 39,150 distinct queries, 40.4% navigational; hit@1 and hit@10 are reported on navigational traffic.Source: §6.1 · pp. 7–8

6.2 Baselines

  • The comparison includes an exact dense retriever trained through linear maps, BM25 over the same URL texts, click-table lookups and lookup-first combinations, plus closed-book LLMs and public encoders on specified samples or subsets.The click prior reranks dense retrieval’s top 100 using URL training clicks; the click-table lookup is a reference for seen queries.Source: §6.2 · pp. 8–9

6.3 Main results

  • 59.6% hit@1 for Octen-WebGR-4B beats 46.5% for dense retrieval with a click prior, 24.8% for dense retrieval, and 9.1% for BM25 on TRAFFIC-SEP.At hit@10, GR reaches 68.7% against dense retrieval’s 55.6%.Source: §6.3 · p. 9, Table 2 · p. 10
  • 30.9% hit@1 for GR on new queries with familiar targets exceeds dense retrieval’s 17.2% and its 21.4% with a click prior; on cold targets, dense retrieval and BM25 lead GR, 24.3% and 25.5% versus 14.3%.On seen pairs, GR reaches 88.2% against the click-table lookup’s 93.4%.Source: §6.3 · p. 9, Table 2 · p. 10

6.4 Held-out queries

  • 32.0% hit@1 for GR on HELD-NAV exceeds dense retrieval’s 20.6%, its 20.8% with a click prior, and BM25’s 16.5%; GR leads on familiar targets but trails dense retrieval and BM25 on cold targets.The test excludes queries appearing in URL texts and in July data used for pre-training or maps; controls found no effect on GR from uncleaned texts.Source: §6.4 · p. 9, Table 3 · p. 10

6.5 Combining GR with dense retrieval

  • 63.1% hit@1 for the two-threshold router behind the lookup improves on the lookup-first GR combination’s 62.5%, while retaining GR’s familiar-target lead and recovering most of dense retrieval’s cold-target performance.The thresholds use GR’s first-URL training-pair count and beam probability, and were chosen on TRAFFIC-JUL.Source: §6.5 · pp. 10–11

6.6 Closed-book LLMs

  • 59.8% hit@1 for GR on the LLM comparison sample exceeds GPT-6 Astra’s 37.5% and Claude Opus 5.5’s 36.8% when answering from memory.The frontier models identify the host for 52.4% and 52.7%, respectively, still below GR’s exact-URL result.Source: §6.6 · p. 11

6.7 A public benchmark

  • 16.2% hit@1 for GR on ORCAS-Nav exceeds dense retrieval’s 11.8%, but trails dense retrieval with a click prior at 21.0%.ORCAS-Nav weights rarely clicked targets more heavily than traffic does.Source: §6.7 · p. 11

7 Scaling the Number of URLs and the Model

The section tests how retrieval changes with the candidate URL universe and model size, then examines whether matching universe size to model size helps.

7.2 Model size at fixed data

  • At 6,000 steps, 8B leads 4B by only 0.77 hit@1 points on TRAFFIC-SEP; all four model sizes score 4.7%–6.6% on new queries.The fixed-step comparison ties model size to how often each pair is seen.Source: §7.2 · p. 11

7.1 Number of candidate URLs

  • From 2M to 201M URLs, GR hit@1 falls 4.7 points (64.2% to 59.6%), versus 22.2 for dense retrieval (47.0% to 24.8%).With the click prior, dense retrieval falls 10.0 points (56.5% to 46.5%), still 5.3 points more than GR.Source: §7.1 · p. 11, Figure 6 · p. 12

7.3 Model size at a fixed number of exposures

  • On 8M URLs, NLL reduction per pair rises from 1.5 nats at 0.6B to 5.5 at 14B, and 4B on 32M matches 1.7B on 8M at 37.9% and 37.7% hit@1.The fitted rule equates four times as many URLs with about 2.5 times the parameters; the comparison reflects denser first-level SID groups.Source: §7.3 · p. 12

7.4 Matching the universe to the model

  • A smaller universe helps only when models are trained and decoded there: U31M raises hit@1 by 10.2, 8.9, and 7.6 points for 0.6B, 1.7B, and 4B.Restricting U201M-trained models to U31M changes accuracy by at most half a point; the U31M models retain only part of their gains when decoded over U201M.Source: §7.4 · p. 12
  • Octen-WebGR-0.6B on U31M reaches 56.5% hit@1, 3.1 points below the 4B model and level with it when the target lies in its universe (81.9% versus 82.3%).The same recipe on U201M reaches 51.5%; the best universe grows with training length and, where larger models learn more, model size.Source: §7.4 · p. 12

8 What Limits a Model: Exposure versus Capacity

The section asks whether retrieval is limited by model capacity or by exposure. Across the measured range, the evidence points to exposure and to when and how query–URL pairs are trained.

8.1 Forgetting rather than crowding out

  • With equal training after two exposures, moving from 8M to 32M URLs changes first-level gain by +0.12 nats pooled, despite raw changes of −0.09 to −0.67.The first-level gains plotted against subsequent cumulative learning rate nearly follow one curve across URL subsets.Source: §8.1 · p. 13, Figure 7 · p. 13

8.2 The decay window

  • Pairs are retained best when exposed late at a sufficiently high learning rate: in round 1, half are forgotten by Λ ≈0.42 and forgotten above about 0.56.For curated round-3 pairs, end-point hit@1 is 45.3% for those drawn inside the decay window and 30.0% for those drawn before it; the comparison shows no difference in round 4.Source: §8.2 · p. 13

8.3 Batch size

  • Forgetting tracks learning rate summed over steps, while proportionally scaling learning rate with batch size lets each exposure teach more.Survival of first-level gain is 0.67–0.72 at 0.6B and 0.65–0.66 at 4B; doubling batch raises 4B hit@1 from 37.9% to 43.5%.Source: §8.3 · p. 14

8.4 Storage efficiency

  • Octen-WebGR-4B stores 6.9 bits per pair, or 0.66 bits per parameter, far from known ceilings but not enough to rule out capacity limits.The estimate credits 20,000 single-answer pairs, about 420 million of 1.13 billion training pairs, with their NLL reduction.Source: §8.4 · p. 14

8.5 More exposures per pair

  • On trained pairs, 16 exposures let the 0.6B model reach 67.4% hit@1, above the 14B model’s 64.3% after two exposures.The 0.6B model uses about a quarter of the FLOPs; its stored bits per parameter rise from 0.19 at two exposures to 0.69 at 16.Source: §8.5 · p. 14

8.6 Repeated versus spread exposures

  • On held-out queries, spreading 16 samples across distinct page queries raises hit@1 from 18.6% to 30.6%; repeated samples fit the same two queries.The gain is mostly on held-out queries similar to a training query; for queries similar to none, hit@1 rises from 6.8% to 14.0%.Source: §8.6 · pp. 14–15, Figure 9 · p. 15

8.7 Retrieval of unfamiliar pages

  • New pages have a median of 2 training pairs versus 98 for old pages, and query-pair samples teach far more than URL-only samples.For new pages, document-indexing samples raise hit@1 by 8.5 points in a randomized comparison; URL-only samples teach 0.35 versus 3.93 nats per pair for two query pairs.Source: §8.7 · p. 15

9 Towards Larger Universes

The section turns the exposure findings into a training recipe and estimates the requirements for a billion-URL universe.

9.1 Training recipe

  • Expose key query–URL pairs late at at least about one-quarter peak learning rate, scale learning rate with the largest feasible batch, avoid re-warming, provide new-page queries, and evaluate decayed checkpoints.These five rules are drawn from the section’s measurements.Source: §9.1 · pp. 15–16

9.2 Projected limits

  • About 13B parameters could index a billion URLs as well as the model indexes 201M, if training delivers exposures; the estimate assumes as many training pairs per URL as U201M.The parameter estimate extrapolates the fixed-exposure scaling rule from a single point; the exposure budget is an upper bound measured at hit@10.Source: §9.2 · p. 16
  • Two 65,536-code tokens hold about 1.07 billion URLs under the 16,384-URL group cap, leaving a billion-URL universe at 93% capacity.In practice, clustering may require a higher cap; ten billion URLs would need a third token or larger codebooks.Source: §9.2 · p. 16

10 Deployment and Release

The section describes the released models, their serving requirements and artifacts, and the scope of screening and privacy safeguards.

10.1 Released models

  • Octen-WebGR-4B is Qwen3-4B-Base trained on U201M; Octen-WebGR-0.6B is Qwen3-0.6B-Base trained on U31M, and both released tables are screened.The released tables omit 2.6% of U201M URLs and 1.1% of U31M URLs; evaluation uses the full universes.Source: §10.1 · p. 16

10.2 Serving

  • Octen-WebGR-4B uses 8.7 GB for weights and identifier table, versus 412 GB for an exact dense index; GR and dense retrieval need similar per-query compute.The 4B model answers a single query in 56 ms on an H200; GR does not replace embeddings for informational search.Source: §10.2 · p. 16

10.3 Released artifacts

  • The release includes 4B and 0.6B weights, screened SID-to-URL tables, and GPU/CPU inference code, but not training data or training and identifier-building code.Weights and tables use CC BY-NC 4.0; inference code uses Apache License 2.0.Source: §10.3 · pp. 16–17

10.4 Screening of the released tables

  • Screening removes targets for 1.2% of TRAFFIC-SEP navigational traffic, lowering 4B hit@1 from 59.6% to 58.7%.The tables were not checked against child sexual abuse material lists or screened for personal pages; both are left to takedown.Source: §10.4 · p. 17

10.5 Privacy and intended use

  • The models reflect clicks through August 2026 and ranking biases, and should not be used to infer anything about individual users.URL removals are available by request and published as versioned table diffs.Source: §10.5 · p. 17

11 Related Work

Prior generative-retrieval work had not demonstrated retrieval over hundreds of millions of individual web pages. Octen-WebGR extends that target scale, while related work also studies generative systems for other large target sets and how retrieval scales or forgets.

Generative retrieval

  • At MS MARCO scale, effectiveness fell as the corpus grew, and an 11B model was slightly worse than a 3B model; later systems matched or exceeded several dense retrievers.The authors caution that different encoders, synthetic queries, and identifiers prevent attributing the contrast with their results.Source: §11 · pp. 17–18

Generative systems at scale

  • Earlier generative retrieval reached 8.8M MS MARCO passages, while systems with 100M–1B targets addressed products, ads, videos, or whole sites rather than individual web pages.The authors position Octen-WebGR, to their knowledge, as the first generative retriever at this scale for individual pages.Source: §11 · p. 18

Scaling

  • Prior scaling evidence differs by setting: generative recommenders saturate early on small fixed-data catalogues, while industrial systems report continued gains.Dense retrievers improve with model size but degrade as the index grows.Source: §11 · p. 18

Memorisation and forgetting

  • Prior language-model studies link memorisation to size and repetition, and forgetting to training order or learning-rate dynamics; GR studies also examine document interference and mitigation.This paper reports that GR forgetting tracks cumulative learning rate after exposure, regardless of batch size.Source: §11 · p. 18

12 Limitations and Future Work

The measurements are bounded by target coverage, click-derived labels, baseline choices, and limits in the held-out and scaling experiments. The authors identify untested approaches and broader experiments needed to clarify these boundaries.

New queries and pages

  • More than half of new-query targets are outside the universe, and GR trails dense retrieval and BM25 on targets with few training queries.GR answers new queries correctly at rank one 10.8% of the time; indexing samples from page titles or content were not tested.Source: §12 · p. 18

Value next to a click table

  • For 3.9% of traffic whose target changed since the click table was built, every lookup-first combination ranks the new target first for under 1%.The proposed learned router or reranker is untested, although the union of the lists reaches 72.6%.Source: §12 · p. 18

Baselines

  • The dense baseline trains two linear maps on a frozen encoder using about 2% of GR’s training compute, while BM25 is untuned.A fully fine-tuned dense encoder or a retriever over page content might narrow the reported gaps.Source: §12 · p. 18

Scope of the scaling results

  • The scaling grid stops at 32M URLs and two exposures per pair, and the matched comparison equalises only training after a pair.Only the 0.6B model received up to 16 exposures; capacity limits beyond the tested conditions are not excluded, and spread exposures were tested mainly on old pages.Source: §12 · p. 18

Held-out queries and ablations

  • HELD-NAV queries come from the training period and may resemble training queries more than truly new ones; excluding July data also leaves few head queries.Identifiers and document-indexing samples came from uncleaned texts, and the authors cannot exclude a small effect without retraining.Source: §12 · p. 18

Benchmarks and reproducibility

  • The click data are proprietary, the recipe has not been run on a public click corpus, and each configuration has a single run.The only public evaluation uses ORCAS-Nav clicks six years older than the authors’ data.Source: §12 · p. 19

Future work

  • The authors identify page-title or content indexing, more exposures per decay window, and tests beyond 32M URLs as priorities for future work.Beyond about a billion URLs, identifiers need a third token or larger codebooks.Source: §12 · p. 19

13 Conclusion

Octen-WebGR uses a model and identifier table to index 201 million pages; its strength is new queries for familiar pages, while embeddings remain stronger on rarely reached pages. The appendix adds evaluation details across query weighting, target familiarity, held-out queries, encoders, baselines, near misses, and serving costs.

A.2 Weighting every query once

  • GR remains 7.0 points ahead of dense retrieval in hit@1 when each navigational query is weighted once, but the 0.6B model falls below dense retrieval per query.This weighting gives rare queries more influence than click-weighted hit@k.Source: §A.2 · p. 25, Table 6 · p. 25

A.3 New queries by the familiarity of the target

  • GR’s hit@1 rises from 6.4% with one target training pair to 35.3% with more than 50, while dense retrieval and BM25 peak at 4–10 pairs and then fall.Table 7 groups new traffic by the target URL’s training-pair count.Source: §A.3 · p. 25, Table 7 · p. 25

A.4 Construction of HELD-NAV

  • On HELD-NAV, cleaning URL texts changes dense retrieval’s hit@1 from 35.7% to 20.6% and BM25’s from 45.3% to 16.5%.The withheld query was often among the target URL’s indexed keywords, motivating its removal before evaluation.Source: §A.4 · p. 26, Table 9 · p. 27
  • After excluding July-data overlap, HELD-NAV contains 12,967 queries; for queries absent from target keywords, GR ranks first for 54.0% versus 19.2% for dense retrieval.The authors report no sign that identifier changes from uncleaned texts inflated GR’s result, though they cannot exclude a smaller effect.Source: §A.4 · p. 26

A.5 Dense encoders

  • Zero-shot public encoders reach only 16–18% hit@1 on TRAFFIC-SEP’s 32M-URL subset, while trained Octen maps reach 34.9%, 36.3%, and 37.9%.The encoder sizes are 0.6B, 4B, and 8B; on HELD-NAV, zero-shot encoders perform better on torso and tail than on head queries.Source: §A.5 · p. 27, Table 10 · p. 27

A.6 Dense baseline

  • The dense baseline freezes Octen-Embedding-4B and trains only two 1,024-dimensional linear maps, totaling 2.1M parameters, on navigational click pairs.Its final maps average seven runs, and training including mining took about 23 H200 GPU-hours.Source: §A.6 · p. 27

A.7 Near misses of dense retrieval

  • When dense retrieval ranks the target second to tenth, GR ranks it first for 81.5% of that traffic.Table 11 categorizes the dense retriever’s first result by its relationship to the target URL.Source: Table 11 · p. 28, §A.7 · p. 28

A.8 Serving cost

  • Serving-cost measurements compare GR decoding with dense query encoding plus exact search, using measured H200 throughput and stated assumptions for query batches and pricing.LLM costs are lower bounds because reasoning tokens are excluded; self-hosted costs count measured compute only.Source: Table 12 · p. 28

B Closed-Book Evaluation of LLMs

The paper evaluates frontier LLMs as closed-book navigational search systems by asking them to return ranked URLs from memory. Their site-level accuracy remains below Octen-WebGR’s exact-URL result, though site-level matching is a more permissive criterion.

B Closed-Book Evaluation of LLMs

  • Six frontier models each produced ten ranked URLs for 5,000 traffic-sampled queries, answering from memory without web search or page fetching.Queries were split into 20 batches of 250, each handled in a separate session.Source: B Closed-Book Evaluation of LLMs · p. 29
  • Among the six LLMs, Astra and Opus are within noise of each other, with a paired 95% confidence interval of −0.1 to +1.6 hit@1 points.Restricting comparison to queries all six answered changes no system difference by more than 0.6 points.Source: B Closed-Book Evaluation of LLMs · pp. 29–30
  • On the same 5,000 navigational queries, GPT-6 Astra reaches 52.4% when a URL on the target’s host counts as correct, below GR’s 59.8% exact-URL hit@1.Only 38–61% of LLM-generated URLs appeared among the 349M clicked URLs in the pool.Source: B Closed-Book Evaluation of LLMs · p. 29, Table 13 · p. 29

C Public Benchmark: ORCAS-Nav

On the public ORCAS-Nav benchmark, generative retrieval leads dense retrieval at hit@1, though a click prior reverses the comparison. The scaling analyses examine candidate-universe size, model capacity, training exposure, and query coverage.

C Public Benchmark: ORCAS-Nav

  • 16.2% hit@1 for GR versus 11.8% for dense retrieval; with a click prior, dense retrieval reaches 21.0% hit@1 and 39.9% hit@10.The benchmark samples 20,000 navigational queries from public Bing clicks; its targets largely predate the training data.Source: C Public Benchmark: ORCAS-Nav · p. 30, Table 14 · p. 30

D.1 Number of candidate URLs; D.2 Model size at fixed data

  • At step 6,000, the 4B model ranks first 37.2% of pairs seen once and 68.7% of pairs seen twice, on subsets retaining 291,453 navigational targets and host pages.The candidate universe grows by adding other URLs; the fixed target set helps isolate the effect of candidate count.Source: §D.2 · p. 30, §D.1 · p. 30, Table 15 · p. 31

D.3 The fixed-exposure grid; D.4 Smaller universes

  • Fourfold more URLs require about 2.5 times the parameters by hit@1 fits, while a smaller universe yields the 0.6B model 2.7 of its 10.2 hit@1-point gain.The fourfold scaling estimate is 2.61 or 2.49 times parameters depending on whether logit hit@1 or hit@1 is fitted; the smaller-universe comparison also reflects sample mix.Source: §D.3 · p. 31, §D.4 · p. 31, Table 16 · p. 31

E.1 What the end point retains; E.2 Forgetting in the grid

  • Retention depends on when a pair was seen: early single-exposure pairs retained 88.2% hit@1, while grid analyses found later cumulative learning rate reduced first-level gains.Matched-exposure comparisons show only 0.00 to +0.15 nats change from 8M to 32M URLs; the runs do not establish that candidate count alone causes the residual.Source: §E.1 · p. 32, Table 18 · p. 32, §E.2 · p. 33

E.3 Batch size; E.4 Bits per parameter

  • With learning rate scaled, larger batches raised trained-pair hit@1 from 20.5% to 22.9% for 0.6B and from 37.9% to 43.5% for 4B; the estimate credits pairs only for training-added information.The bits-per-pair estimate compares trained SID likelihood with a starting point that already performs the task, rather than charging all URL-naming bits to training.Source: §E.3 · p. 33, §E.4 · p. 33

E.5 More exposures per pair

  • On held-out queries, 16 exposures did not improve the 0.6B model over its starting point, while moving from 0.6B to 14B at two exposures added 4.1 ± 1.3 hit@1 points.The exposure ladder establishes that no capacity limit lies below 0.69 bits per parameter, not where the limit lies.Source: §E.5 · p. 34, Table 20 · p. 34

E.6 Spread versus repeated exposures

  • Spreading 16 samples across distinct queries raised held-out hit@1 by 12.0 ± 1.7 points over repeating two queries, while repeated exposure left other-query likelihood unchanged at the first SID level.The spread arm also gained 7.9 ± 4.8 points on held-out queries sharing no words with its training queries.Source: §E.6 · pp. 34–35, Table 21 · p. 35

E.7 New pages

  • Training-pair exposure explains much of the old–new page gap: among pairs drawn once, old pages reach 44.2% hit@1 and new pages 10.1%.Document-indexing samples were associated with 45.4% versus 36.8% hit@1 for new pages, while URL-only samples taught substantially less than query pairs.Source: §E.7 · p. 35

Related papers

Loading 2610.35256v1…