Source-linked AI summary
Repeated Queries Exhaust an LLM's Brand Recommendations but Not Its Sources
Dmitrij Żatuchin
TL;DR
Repeated identical buying questions leave the completeness of brand recommendations unresolved, and the paper measures how brand and cited-domain sets accumulate across runs. Using rarefaction, Chao2, and open extraction across two studies, it finds that retrieval closes brand lists while non-retrieval lists and citation maps continue expanding.
Problem
The paper asks how quickly distinct brands and cited domains accumulate in repeated answers, because sampling reports implicitly claim coverage of what a model would say.
Method
The studies treat runs as sampling units, apply exact rarefaction and Chao2, and compare fixed-roster with open extraction across retrieval and non-retrieval engines.
Results
Retrieval closed the brand list at a median of 8 organizations, whereas 86–92% of non-retrieval cells still added organizations at run 15; cited domains continued accumulating at every tested horizon.
Takeaways & Limitations
Grounded brand lists are comparatively cheap to finish, while ungrounded brand lists and citation maps remain open-ended at practical sampling budgets.
Takeaways & Limitations
Study 1 could not separate model behavior from its fixed-roster breadth design and web-search depth design; the author also reports a commercial conflict of interest.
Abstract
from arXiv · showhide
Whether repeated identical buying questions exhaust a language model's brand recommendations depends on retrieval. Across 300 question-engine cells (50 questions, six engines, 15 runs each, open extraction over 1,470 adjudicated organizations), the five engines answering without web search were still adding never-seen brands at run 15 in 86-92% of cells, with median repertoires of 15-31 organizations; the one retrieval-enabled engine closed its list (median 8 organizations, 64% of cells still adding), matching four earlier deep cells where web-search runs saturated by run ten. Cited-domain accumulation keeps rising at every horizon tested: four deep cells were still adding domains at run 24 with 59-84% of the Chao2 lower-bound estimate observed, and 44% of the retrieval engine's breadth cells were still adding domains at run 15. A single run shows 62-77% of the five-run brand set, and across engines the median question draws 38 organizations, of which a median of 15 appear in exactly one engine. Estimators are exact rarefaction and Chao2 richness; a parallel fixed-roster extraction reproduces flat curves on identical responses, so roster-bounded tracking manufactures plateaus that open extraction removes.
1 Accumulation in repeated answers
The paper frames repeated runs as samples from changing sets of brands or cited domains, asking how quickly distinct items accumulate. It applies rarefaction and Chao2 to measure observed growth while distinguishing recurrence from exhaustive coverage.
- The study asks how quickly distinct brands and cited domains accumulate across repeated answers to identical commercial questions.
- Runs are sampling units, while brands and cited domains are treated as species in an accumulation analysis.
- Exact rarefaction estimates expected distinct items after k of n runs by averaging over run subsets.
- The final rarefaction increment is A(n) − A(n −1) = Q1/n, where Q1 counts items seen in exactly one run.
- Q1 = 0 means every observed item recurred, not that no unobserved item exists.
2 Study 1: roster curves flatten; deep cells saturate under web search
Study 1 found that roster-bounded brand curves could flatten while open checks exposed the limits of that conclusion. Deep web-search cells approached brand saturation, but cited domains continued accumulating.
- 250 brand-free buying questions were asked five times each to three engines, with extraction bounded by a 50-brand alias roster.
- On the fixed roster, the median cell named 5 brands, one run showed 80% of the five-run set, and 49% of complete cells had a singleton brand.
- Four 24-run web-search cells reached Q1 = 0 with extended rosters, reaching 91–100% of the observed set by run ten and repertoires of 7–11 organizations.
- Cited domains kept accumulating at run 24, with Q1 values of 18, 6, 41, and 12 and 59–84% of the Chao2 estimate observed.
- Study 1 could not determine whether the brand plateau reflected model behavior or the fixed-roster and web-search designs.
3 Study 2: six engines, open extraction, and the retrieval split
Study 2 used open extraction across six engines and showed a retrieval split: non-retrieval engines kept adding organizations, while the retrieval-enabled engine closed its brand list. Cross-engine results also showed that individual engines omit many organizations and that cited domains remain less settled.
- 4,500 fresh responses covered 50 questions, six engines, and 15 runs per question-engine cell, with open extraction yielding 1,470 organizations.
- A parallel fixed-roster extraction reproduced flat breadth curves with median 4–5 brands per cell, attributing the earlier plateau to the roster.
- 86–92% of cells on the five non-retrieval engines still added a never-seen organization at run 15, with median repertoires of 15–31 organizations.
- The retrieval-enabled engine’s median cell named 8 organizations, reached 100% of its Chao2 estimate by run 15, and was the only engine with a majority of cells stopping additions.
- The median question drew 38 organizations across engines, including a median of 15 organizations appearing in exactly one engine.
- 44% of retrieval-engine breadth cells were still adding cited domains at run 15, with a median of 21 domains per cell.
4 Related work
Prior work established that repeated LLM queries can vary and that repetition within a prompt can change behavior. This study instead measures how distinct brands and cited domains accumulate across repeated calls.
- Repeated-query non-determinism and within-prompt repetition effects were established before this accumulation study.
- The study measures set growth across calls rather than mean stability or accuracy changes within a single prompt.
- Citation instability in generative search motivates quantifying how much of the citation record a sampling budget observes.
5 Conclusion
Whether repeated queries exhaust brand recommendations depends on deployment, with retrieval distinguishing short, quickly saturating lists from open-ended ungrounded lists. Citation maps remain open-ended at practical budgets, while recommendation frequencies require separate measurement.
- Retrieval is the switch determining whether repeated brand recommendations saturate.
- Grounded brand lists are cheap to finish, whereas ungrounded brand lists and citation maps remain open-ended at practical budgets.
- Ungrounded engines produced repertoires three to six times larger than the grounded engine’s while still adding organizations at run fifteen.
- Recommendation frequencies are not settled by these repetition budgets and require their own design.
Declarations
The author discloses a commercial interest in AI-visibility measurement and provides public data, code, and provenance information for the studies.
- The author is CEO of Rankfor.AI, which sells AI-visibility measurement including repeated-query brand audits.
- The Study 1 corpus and Study 2 materials, including raw responses and analysis code, are publicly deposited.
- Study 1 reanalyzes the author’s deposits, whereas Study 2 was collected for this note.