PaperlayerView exact revision on arXiv

Source-linked AI summary

System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Dmitrij Żatuchin

arXiv:2610.00253v1cs.IRcs.CLcs.LG

TL;DR

This paper asks whether aggregated brand-recommendation profiles describe the deployed system or instead reflect query domain and collection harness. It tests response-level attribution and aggregate profiles on one corpus of stored answers. Single responses identify endpoints and transfer across tested domains, while aggregate brand behaviour does not.

  • Problem

    AI-visibility audits summarize brand recommendations into per-system profiles, but whether those profiles describe the system rather than query domain or collection harness is uncertain.

    Source: §1 · p. 2

  • Method

    The paper analyzes stored responses from five deployed endpoints at both the single-response and model-by-domain-by-condition aggregate levels.

    Source: §1 · p. 3

  • Results

    97.84% accuracy identifies endpoints from 800-character prefixes, while aggregate profiles reach 66.53% within the collection window and get 4 of 32 cross-domain units right; single responses retain 89.92% balanced accuracy.

    Source: §7 · p. 20

  • Takeaways & Limitations

    Answer shape carries endpoint identity across the tested query domains, but aggregated brand behaviour does not, bounding AI-visibility measurements based on those aggregates.

    Source: §7 · p. 20

    The design is not crossed: no system appears in every domain, so system label is partly determined by domain.

    Source: §6.4 · p. 19

Abstract

from arXiv · show

Audits of AI visibility summarise the brand recommendations of deployed language models into per-system profiles. We test whether such a profile describes the system on one corpus of 6,475 stored responses (6,324 analysable) collected between December 2025 and February 2026 from five deployed endpoints across gift-recommendation, corporate-reputation and category-ownership queries. The collection harness cut many answers short: 83.1% of Gemini 3 Flash answers in category ownership end mid-sentence under a 1,024-token output cap. With every answer cut to its first 800 characters, a character n-gram classifier cross-validated by prompt attributes one response to GPT-5.2, Gemini 3 Flash, Gemini 3 Flash with search, Grok or Perplexity sonar-pro with 97.84% accuracy (5,028 responses, 383 prompts, majority class 31.5%, 30 split seeds). Length alone falls to the majority rate, 24 formatting statistics reach 95.79%, and masking brand names and capitalised tokens leaves 97.72%. Held-out query conditions keep 97.43% weighted by size and 88.0% unweighted; in a retrieval-grounded arm that changes the harness, no Grok answer is attributed to Grok (0/120). Aggregated into 50 model-by-domain-by-condition units, twelve behavioural features separate four systems at 66.53% under grouped cross-validation, against a label-permutation null with mean 33.71% and 95th percentile 46.0%. Across domains the aggregate profile fails: a forest trained on category-ownership units assigns all 22 gift units to the wrong system, consistent with a reversal in brand volume (8.41 against 0.94 brands per response in gifts, 3.01 against 3.91 in category ownership), while single responses transfer at 89.92% balanced accuracy. The surface form of an answer carries the system across the query domains tested; aggregated brand behaviour does not, and the uncrossed design cannot separate the system from the domain or the harness.

1 Introduction

This paper tests whether per-system brand-recommendation profiles identify the system that produced an answer, rather than reflecting query domain or collection harness. Single responses identify endpoints accurately, but aggregate brand profiles vary across domains.

1 Introduction

  • 95.79% accuracy from 24 formatting statistics shows that answer layout carries most response-level attribution signal.The collection harness also truncates answers, so surface features can reflect the harness as well as the endpoint.Source: §1 · p. 2
  • 97.84% accuracy identifies five endpoints from single 800-character response prefixes; 12-feature aggregate profiles reach 66.53% within the collection window but fail across domains.The aggregate classifier assigns every gift-domain unit to the wrong system when moved from category ownership.Source: §1 · p. 2
  • The study asks how accurately single responses identify endpoints, what carries that signal, how it transfers across domains, and where aggregate profiles fail.Its corpus also documents collection defects and revisits values from an earlier version against raw files.Source: §1 · p. 3

2 Related Work

Prior work studies both chosen-prompt and passive output fingerprinting, deployed-system drift and brand bias, retrieval pipelines, and methods for small grouped datasets. This paper applies passive attribution to brand recommendations and contrasts response-level results with aggregate profiles.

2.1 Which Model Wrote This

  • Passive fingerprinting classifies stored outputs, whereas active methods choose inputs; this study uses consumer-style prompts collected for other purposes.Earlier passive work reports cross-domain fingerprints and high accuracy on deployed chat systems, while the present work adds brand recommendations and aggregate-profile comparisons.Source: §2.1 · pp. 3–4

2.2 Behaviour of Deployed Systems and Its Drift

  • Prior work documents deployed-system drift and brand bias, framing model outputs as statistical behaviour rather than truth claims.The cited work also reports inter-system differences in gift-recommendation brand sets.Source: §2.2 · p. 4

2.3 Retrieval-Augmented Services Are Pipelines

  • Retrieval-augmented services are pipelines; retrieval, citation policy, or re-ranking may distinguish them from parametric endpoints.The study therefore labels results at the deployed-system level.Source: §2.3 · p. 4

2.4 Small Samples, Grouped Data, Calibration

  • Grouped cross-validation, permutation tests, bootstrap intervals, and calibration metrics address leakage, pseudo-replication, and uncertainty in small grouped analyses.The aggregate analysis uses random forests and PCA descriptively at 50 units.Source: §2.4 · p. 4

3 Corpus

The corpus combines three previously collected query domains and records collection defects alongside its analysable responses. Unequal truncation, empty outputs, differing repetitions and temperatures, and a retrieval arm with source requests qualify comparisons across systems.

3.1 What Was Collected

  • 6,475 responses from 121 files form the corpus; after setting aside empty and colliding-index rows, 6,324 responses remain analysable.The collection spans three query domains and was re-read from raw response files.Source: §3.1 · pp. 4–5, Table 1 · p. 5

Empty responses.

  • 136 zero-length GPT-5.2 category-ownership responses are excluded by default, and their concentration on long-answer prompts makes them missing not at random.They include 64 healthtech responses, leaving the GPT-5.2 healthtech unit based on 186 of 250 responses.Source: Empty responses · p. 5

Rows with a colliding iteration index.

  • 15 distinct GPT-5.2 reputation generations sharing reused iteration keys are dropped by keeping the first row per key.Keeping them changes their aggregate unit by 0.010 in mean brand count and 0.011 in abstinence.Source: Rows with a colliding iteration index · p. 5

3.2 What the Collection Protocol Actually Was

  • Recorded repetitions vary by subset—40 for Christmas gifts, 10 for Valentine’s, 5 for category ownership, and usually 1 for reputation—and temperature is recorded only for reputation and category ownership.The gift subset stores no sampling parameters.Source: §3.2 · p. 6
  • The gift-subset Grok version was not logged, so the Grok label may pool versions.The only Grok version string in a collection script is grok-3 in the grounded arm.Source: §3.2 · p. 6

Answers are cut short in three subsets.

  • 83.1% of category-ownership Gemini 3 Flash answers stop mid-sentence under the 1,024-token cap, compared with 2.2% for GPT-5.2 and 0.5% for Perplexity.Across the headline task, 1,107 of 5,028 responses are incomplete, including 1,065 Gemini 3 Flash responses.Source: Answers are cut short in three subsets · p. 6

Completion caps differ between subsets.

  • Completion caps differ between subsets for GPT-5.2: 1,024 tokens in category ownership versus 16,384 in reputation.The empty GPT-5.2 outputs and truncated Gemini answers are described as two faces of the same cap.Source: Completion caps differ between subsets · p. 6

One arm was asked for its sources.

4 Method

The study analyzes attribution at the stored-response level and system profiles at the model-by-domain-by-condition level, using separate classifiers and grouped validation. It also tests controls, uncertainty, calibration, and transfer between query domains.

4.1 Two Units of Observation

  • The response unit is one stored answer, while the aggregate unit is a system-by-domain-by-condition cell summarizing 10–648 responses; 50 aggregate units receive equal classifier weight.The aggregate task has four labels; Grok and Perplexity occur in only one domain each, and the grounded Gemini arm is not classified at the aggregate level.Source: §4.1 · p. 7

4.2 Response-Level Model

  • Character n-grams from 2–4 characters are TF-IDF weighted and classified with multinomial logistic regression; cross-validation holds out whole prompts.The representation captures punctuation, list markers, and casing as well as words.Source: §4.2 · p. 7

4.3 Behavioural Feature Vector

  • The 12 aggregate features form four families, covering mean brand count, brand-count variability and abstention; brand-frequency concentration; sentiment; response consistency; citation absence; and cross-query brand diversity.Citation absence is exactly 1.0 in 37 of 50 units, and positivity rate is exactly 1.0 in 40.Source: §4.3 · p. 7, Brand volume · p. 8, Concentration · p. 8

Brand volume.

  • Brand volume uses mean unique-brand count, its coefficient of variation, and abstinence; concentration uses Gini, normalized entropy, and top-brand share.Sentiment comprises VADER mean, variance, and positivity; behavioural features include repetition consistency, citation absence, and cross-query brand diversity.Source: Brand volume · p. 8, Concentration · p. 8, Sentiment and tone · p. 8

4.4 Aggregate-Level Model and Protocol

  • A class-balanced 500-tree random forest is evaluated with grouped folds that keep domain-condition pairs together; leave-one-group-out and an RBF SVM provide stricter and baseline comparisons.The forest seed is fixed for the headline analysis; an ungrouped protocol is also run to compare against the earlier analysis.Source: §4.4 · pp. 8–9

4.5 Uncertainty and Calibration

  • Uncertainty is assessed with group-preserving permutation tests and group-level bootstrap intervals, while calibration uses 10-bin expected calibration error on out-of-fold predictions.The response-level permutation null uses a lighter character n-gram SVM surrogate; calibration is measured within the training distribution, on held-out conditions, and across domains.Source: §4.5 · p. 9

4.6 Controls

  • Controls compare character-count-only logistic regression, random forest, and gradient boosting with a 24-statistic formatting random forest, mask brand-dictionary spans, uppercase-starting tokens, URLs, and digit runs, and hold out whole collection conditions.Source: §4.6 · p. 9

4.7 Cross-Domain Transfer

  • Cross-domain transfer trains in one domain and tests in another; aggregate tests use shared brand-volume and concentration features, while response tests use balanced accuracy for the two systems present in both domains.Table 2 reports response-level attribution on stored text and 800-character prefixes, excluding the parametric reputation row because its Gemini arm is truncated in storage.Source: §4.7 · p. 9, Table 2 · p. 10

5 Results

Single responses identify deployed endpoints with high accuracy, while aggregate brand-behaviour profiles are less reliable and fail to transfer across query domains. Results also show that truncation, collection conditions, and the harness shape what these measurements capture.

5.1 Attribution at the Response Level

  • 97.84% five-way accuracy on 800-character prefixes identifies the endpoint across 5,028 responses, versus a 31.5% majority baseline.Accuracy is 98.14% on stored text and 97.86% on 3,921 complete responses; the within-prompt permutation null has mean 33.43%.Source: §5.1 · p. 10

Held-out conditions.

  • 97.43% weighted and 88.0% unweighted accuracy persist across 16 held-out conditions that vary queries while keeping the collection scripts.In the sole held-out condition that changes the harness, no Grok answer is attributed to Grok (0/120); the Grok version was not logged, so version differences may also matter.Source: Held-out conditions · pp. 10–11, Table 3 · p. 11

Calibration.

  • Response-level expected calibration error is 0.0317 on prefixes inside the training distribution and 0.0419 on held-out conditions, while the aggregate forest reaches 0.4754 across domains.The response-level calibration claim is limited to use within one harness and one set of query domains.Source: Calibration · p. 11, Figure 1 · p. 12

5.2 What Carries the Signal

  • 95.79% accuracy from 24 formatting statistics approaches the 97.84% n-gram result, while length alone falls to the 31.5% majority rate on 800-character prefixes.Masking brand names and capitalized tokens changes n-gram accuracy by less than 0.2 points; formatting statistics also measure the harness, not only the system.Source: §5.2 · pp. 11–12, Table 4 · p. 12

5.3 Empty Responses as a Sensitivity

  • 100.0% response-level category-task accuracy on prefixes persists when affected prompts are dropped or empty responses are treated as observed answers.GPT-5.2’s aggregate abstinence changes with the empty-response policy, but the brand-volume ordering relative to Gemini 3 Flash does not.Source: §5.3 · pp. 12–13, Figure 2 · p. 13

5.4 Aggregated Behavioural Profiles

  • 66.53% grouped accuracy separates four systems from aggregate profiles, above a within-group permutation null with mean 33.71% and 95th percentile 46.0%.The score is 64.0% under leave-one-group-out; removing sentiment features gives 67.47%, while ungrouped folds score 53.5% and category ownership falls from 80.0% to 46.4%.Source: §5.4 · p. 13, Grouped scores above ungrouped · p. 14

What the forest uses.

  • Response consistency has the largest held-out permutation importance, but its values vary by domain and it measures the harness as well as the system.The earlier 68.0% headline came from an ungrouped split at seed 42 on the earlier feature matrix; the projection of the rebuilt 50-unit matrix is descriptive.Source: What the forest uses · p. 14, The earlier headline · p. 14, Figure 3 · p. 15

5.5 Cross-Domain Transfer at Both Units

  • A forest trained on category-ownership aggregate units assigns all 22 gift units to the wrong system, while response-level transfer reaches 89.92% balanced accuracy on prefixes.Brand-volume ordering reverses: Gemini 3 Flash versus GPT-5.2 averages 8.41 versus 0.94 brands per gift response, and 3.01 versus 3.91 in category ownership; the reversal is consistent with transfer failure but is not shown to cause it.Source: Table 5 · p. 16, §5.5 · p. 16

6 Discussion

Single responses identify deployed endpoints across the query domains tested, whereas aggregated brand behaviour varies with domain. The discussion bounds these findings by the collection design and identifies a crossed study that could separate endpoint from harness.

6.1 The Unit of Observation Decides the Answer

  • 97.84% response-level accuracy contrasts with 66.53% aggregate accuracy; single responses retain 89.92% balanced accuracy across domains, while aggregate profiles invert.The aggregate classifier’s brand-volume ordering reverses between gift and category-ownership queries.Source: §6.1 · pp. 17–18

6.2 Endpoint, Harness and Domain

  • The controls do not separate endpoint from harness, so attribution supports a stable endpoint–harness signature rather than an endpoint-only signature.The only harness-changing holdout attributes all Perplexity answers correctly and no Grok answers correctly.Source: §6.2 · p. 18

6.3 Three Applications, Bounded

  • Calibration is supported within a known harness and its query domains, but the measured calibration does not hold outside them; the corpus does not test drift over time.The reported cross-domain expected calibration error is 0.096.Source: §6.3 · p. 18
  • Brand-behaviour profiles vary with query domain and did not transfer between the two tested domains; the surface signature is editable, limiting provenance claims.The cross-harness evidence is one condition with mixed results.Source: §6.3 · p. 18

6.4 Limitations

  • The design is not crossed: no system appears in every domain, and Grok and Perplexity labels are each tied to particular domains.The data cover one collection window, with no second window to test temporal change.Source: §6.4 · p. 19
  • Three subsets contain truncated answers; the 800-character prefix removes stopping-point information but cannot restore content lost before storage.The affected truncation also enters aggregate brand features for Gemini 3 Flash in category ownership.Source: §6.4 · p. 19
  • The aggregate classifier uses 50 units in 17 groups, while sentiment features mainly proxy response length and lack human-coder validation.The brand instrument has precision 0.827 and recall at most 0.43 on a 50-answer sample; the gift response-level figures rest on 13 prompt groups.Source: §6.4 · p. 19

6.5 A Design That Would Settle the Confound

  • A proposed crossed collection uses multiple domains and windows, a shared instruction set and completion cap, and two harnesses applied to the same endpoints and prompts.The design also calls for logged version strings and an open-weight model with frozen weights as an architectural control.Source: §6.5 · p. 19

7 Conclusion

The paper compares endpoint attribution at the response level with brand-behaviour profiles at the aggregate level. Responses transfer across tested query domains, but aggregate brand behaviour does not, and the design cannot isolate the system from domain or harness.

7 Conclusion

  • 97.84% response-level accuracy on 5,028 responses falls to 66.53% for aggregate classification, which gets 4 of 32 cross-domain units right.The aggregate comparison is against a structure-preserving permutation null with a 95th percentile of 46.0%; single responses retain 89.92% balanced accuracy on the same two-system question.Source: §7 · p. 20
  • The surface form of endpoint answers carries identity across the tested query domains, while aggregated brand behaviour does not.The design cannot establish how much of that behaviour belongs to the system rather than the domain or harness.Source: §7 · p. 20

Appendix A Per-System Detail, Aggregate Classifier

The appendix reports per-system aggregate-classifier detail, specifies the scope of related tables, and documents corrections to earlier analyses. Its supplied passages include a confusion pattern and explain that the headline aggregate value changed when the feature matrix was rebuilt.

Appendix A Per-System Detail, Aggregate Classifier

  • Table A1 reports per-system precision, recall and F1 for the four-system aggregate forest, pooled across grouped cross-validation seeds.Its caption specifies support in aggregated units, but the supplied passage gives no cell values.Source: Table A1 · p. 20
  • 67.3% of Perplexity units are predicted as Gemini, while 32.4% of Grok units are predicted as GPT-5.2, compared with 8.8% in the reverse direction.Source: Appendix A · p. 20
  • Table B2’s 50-unit feature matrix mixes system and domain because Grok occurs only in gifts and Perplexity only in category ownership.Source: Table B2 · p. 21

Appendix C Response-Level Transfer, Per Pair

  • Table C3 covers Gemini 3 Flash versus GPT-5.2 response-level cross-domain transfer, reporting balanced accuracy and expected calibration error for one fit per pair and variant.The masked variant replaces dictionary brand spans, capitalised tokens, URLs and digits.Source: Table C3 · p. 21

D.1 Why the Numbers Changed

  • The corrected analysis restores 360 grounded-reputation responses previously skipped by a loader type gate; the aggregate headline changed because the feature matrix was rebuilt.On the earlier matrix, the grouped protocol gives 68.13%; the headline correction is 66.53%.Source: §D.1 · p. 21, 136 empty responses · p. 22

D.2 Values That Changed

The correction appendix contrasts earlier and corrected values and gives selected reasons for changes. The supplied passages include response-level attribution and aggregate-classifier results, alongside several methodological corrections and limitations.

D.2 Values That Changed

  • The correction table is described as pairing earlier and corrected values, while the supplied fragments also note that the Grok version string is unlogged.The collection window is given as 2025-12-23 to 2026-02-24 per-row timestamps.Source: Table D4 · p. 22, §D.2 · p. 23
  • The aggregate null’s 95th percentile is 46.0%, while two of four labels occur in only one domain.The supplied fragment gives the 25% chance reference and a permutation-null mean of 33.71%.Source: §D.2 · p. 22
  • Macro F1 for four aggregate systems is 0.589, and gift-domain accuracy is 0.63 pooled over 30 grouped seeds.Source: §D.2 · p. 22
  • The corrected category-ownership value is 80.00% with p = 0.001; the passage attributes the null update to 1,000 within-group shuffles.Source: §D.2 · p. 23
  • Response consistency and brand-count coefficient of variation are reported as 0.504 and 0.524, computed per prompt across repetitions.Source: §D.2 · p. 23
  • 97.84% response-level attribution is reported on 5,028 responses using 800-character prefixes.Source: §D.2 · p. 23

D.3 Claims Withdrawn

The earlier version’s cross-window replication, sentiment reliability, source-opacity, transfer, missingness, feature-importance, collection-protocol, and availability claims are withdrawn or corrected. Rebuilt evidence changes or invalidates several reported results.

Sentiment reliability

  • Cohen’s κ = 0.72 is withdrawn: the pipeline used TextBlob, and no agreement against human coders has been measured.The claimed VADER-versus-judge result has no supporting code, sample, or result file.Source: §D.3 · pp. 23–24

Cross-window replication

  • The claimed +0.18 multilingual sentiment replication lacks a supporting result file; the transformer gives +0.0895, while the English-only VADER comparison does not reproduce the February gap under truncation control.The controlled February gap falls from +0.511 to +0.043; the compared systems also differ from those in the original study.Source: §D.3 · p. 24

Study 2 sentiment

  • The reported sentiment gap and positivity rates are invalid because Gemini answers are truncated in storage; with lengths matched, the gap is +0.043.The published values were 0.83 against 0.32 for mean sentiment and 96.4% against 51.5% for positivity.Source: §D.3 · p. 24

Source opacity

  • The source-opacity figures and foundation-versus-RAG comparison are withdrawn because neither reported percentage is computable from the files, whose computable values reverse the ordering.The withdrawn percentages were 70.2% for parametric endpoints and roughly 5% for Perplexity.Source: §D.3 · p. 24

Per-system transfer

  • The reported per-system transfer split is not interpretable: two of four pairs have only two units and one predicts a single label.The appendix reported 73.6% for GPT-5.2 and 35.0% for Gemini.Source: §D.3 · p. 24

Missingness indicators

  • The missingness-indicator design would add constant columns because the four-system feature matrix contains no missing cells.The described matrix pairs each feature with a missingness indicator.Source: §D.3 · p. 24

Feature importance

  • The feature-importance methods do not agree on the top three; the earlier claim is explicitly contradicted by Section 5.4.Source: §D.3 · p. 24

Protocol and availability

  • The stored rows contradict the earlier statements about prompt repetitions, Grok’s version string, and collection-window endpoints; both earlier availability statements are also replaced.Source: §D.3 · p. 24

D.4 What Holds

Five earlier-version quantities reproduce from the stored data or analysis, including the aggregate total and category-ownership accuracy. The permutation p-value also reproduces, although the earlier code did not run that test.

What holds

  • Five earlier-version quantities reproduce, including 1,311 parametric reputation rows, 50 aggregate units, 80.0% category-ownership accuracy, 41.6% versus 42.41% leading-brand agreement, and a 68.0% grouped-protocol headline.The permutation p-value of 0.001 also reproduces, from a test the earlier code did not run.Source: §D.4 · p. 24

Data Availability.

Some response files and analysis artifacts are available from the author on request, while reputation responses are withheld because prompts name clients. The analysis scripts are also available on request; the declarations disclose funding, commercial interests, and automated labeling.

Data Availability.

  • Of 6,475 stored responses, 574 gift responses match a public Zenodo dataset; other specified files are available from the author, while reputation responses are withheld because prompts name clients.The response table and feature matrix are released without reputation-response text; a Zenodo deposit is planned.Source: Data Availability · p. 25

Code Availability.

  • The numbered Python analysis scripts are available from the author on request and will be deposited with the data; source-corpus collection used the open-source Dice Roller package.Source: Code Availability · p. 25

Declarations

  • Rankfor.AI paid API costs, and its founder declares a commercial interest; sentiment and brand-dictionary checks were labeled by an LLM without human raters.The author states that the company is named in the reputation prompts and that the studied attribution methods are relevant to its business.Source: Declarations · p. 25

Related papers

Loading 2610.00253v1…