PaperlayerView exact revision on arXiv

Source-linked AI summary

Whom Do AI Agents Work For? Role Assignment Induces Sponsorship Bias in LLM Recommenders

Davood Wadi, Yu Ma

arXiv:2609.17989v2econ.GNcs.AIq-fin.EC

TL;DR

AI shopping assistants advise consumers on platforms that also sell advertising, while sponsorship disclosures reach the agent rather than the consumer. The paper uses controlled choice experiments that assign the agent to a traveler or booking platform, finding that platform delegation weakens the sponsorship penalty and disclosure-triggered skepticism.

  • Problem

    In AI-mediated shopping, sponsorship labels reach the agent evaluating listings rather than the consumer, so consumer protection depends on how the agent responds to paid placements.

    Source: Conflicts of Duty · p. 4

  • Method

    Controlled choice experiments varied whether the system prompt named a traveler or booking platform as the agent’s principal and tested disclosure wording and platform attribution.

    Source: Introduction · p. 2, Hypothesis Development · p. 4, Study 2 · p. 17

  • Results

    50.2 percentage points less often than similar organic listings were sponsored listings chosen under traveler delegation, versus a 29.2-point penalty under platform delegation; the pattern replicated across models and reasoning settings.

    Source: Introduction · p. 2, General Discussion · p. 20

  • Takeaways & Limitations

    “Sponsored” lowered paid-listing choice but did not eliminate the delegate gap when the platform was named, so disclosure terminology alone did not resolve the bias.

    Source: General Discussion · p. 20

    The prompt remedy depends on deployers including the beneficiary instruction and not contradicting it elsewhere in the system prompt.

    Source: Limitations and Future Research · p. 22

Abstract

from arXiv · show

Large language models (LLMs) now serve as conversational shopping assistants on platforms that also sell advertising. These AI agents face a conflict of duty. They advise consumers who rely on their judgment, yet are deployed by platforms that benefit when sponsored listings are chosen. Sponsorship disclosures, designed to allow consumers to penalize paid placements, now reach the AI agent rather than the consumer, and the agent's evaluation of them is hidden from the consumer. Drawing on the fiduciary concept of conflict of duty, we argue that an agent's evaluation of a sponsored listing should not depend on which party deployed it. In controlled choice experiments, we manipulate assigned roles in the system prompt to name either a traveler or a booking platform as the agent's principal. Platform delegation significantly attenuates the penalty that agents apply to sponsored listings and weakens the skepticism that disclosure triggers in their reasoning traces. We replicate out findings across LLMs and reasoning depths. A second study decomposes the disclosure label and shows that the divergence between the two delegates widens significantly when the paid placement is attributed to the platform. Stricter terminology ("Sponsored" instead of "Promoted") lowers choice of paid listings but does not close this gap when the platform is named. The findings show that disclosure mandates designed for human consumers cannot by themselves protect consumers in AI-mediated commerce.

Introduction

AI shopping assistants can advise consumers while serving platforms that sell advertising, creating competing duties around sponsored listings. Controlled choice experiments show that naming the platform as principal weakens the sponsorship penalty, while consumer-beneficiary instructions and stricter disclosure terminology do not have the same effect.

  • Disclosure reaches the AI agent rather than the consumer, leaving consumers unable to observe how the agent evaluated the paid placement.Traditional disclosure relies on the consumer seeing the label and discounting the listing accordingly.Source: Introduction · pp. 1–2
  • The proposed evaluative standard is that an agent should not favor a sponsored listing merely because its system prompt names the platform rather than the consumer.The paper calls a more favorable evaluation under platform delegation sponsorship bias.Source: Introduction · p. 2
  • The experiments changed only the delegating party, assigning the agent to a traveler or booking platform through its system prompt.Adding an instruction to act in the traveler’s best interest removed the attenuation under platform delegation.Source: Introduction · p. 2
  • 50.2 percentage points less often than similar organic listings were sponsored listings chosen under traveler delegation, but the penalty fell to 29.2 points under platform delegation.The pattern replicated across models, reasoning depths, and prompt variations; platform delegation also weakened disclosure-triggered skepticism.Source: Introduction · p. 2
  • Stricter “Sponsored” terminology lowers paid-listing selection but does not eliminate the delegate gap when the platform is named.The gap is intensified when the paid placement is attributed to the platform; specifying the consumer as beneficiary removed the attenuation in Study 1.Source: Introduction · p. 2
  • Because role assignment can produce bias without explicit pro-advertiser instructions, firms should examine whom the system prompt directs the agent to serve, not only who deploys it.The same conflict can arise wherever a firm deploys an assistant to advise consumers while monetizing those interactions.Source: Introduction · pp. 2–3

Literature Review

The literature frames AI recommendations as fiduciary judgments amid multiple stakeholders, but LLMs differ from conventional ranking systems because natural-language context can assign the agent’s principal without an inspectable balancing parameter. Disclosure is a standard remedy for conflicts of duty, yet AI-mediated shopping puts the label and its hidden evaluation in the agent’s hands.

Literature Review

  • A digital fiduciary exercises judgment on another party’s behalf, while recommenders operate among consumers, platforms, and advertisers with diverging interests.An AI agent in that environment may act on behalf of more than one party.Source: Literature Review · p. 3
  • Prior work centers on ranking systems with inspectable objective-function weights, whereas LLMs can make open-ended recommendations from context without an equivalent parameter to disclose.A principal can be named in a single sentence, leaving the resolution of competing duties open.Source: Literature Review · p. 3

Conflicts of Duty

  • A conflict of duty arises when obligations to two parties with divergent interests cannot both be fulfilled without compromise, even if the agent gains nothing personally.The paper connects this structure to dual-agency brokers and AI assistants deployed by platforms to advise consumers.Source: Conflicts of Duty · p. 3
  • Disclosure is intended to let the party at risk discount an agent’s judgment, but in AI shopping the label reaches the agent and its evaluation is hidden from the consumer.Whether disclosure protects consumers therefore depends on whether the agent itself penalizes paid placements.Source: Conflicts of Duty · pp. 3–4

Hypothesis Development

The section defines principal assignment as a way to operationalize the agent’s conflict of duty, then develops predictions about how it may shape sponsorship judgments. It frames consumer-protective evaluation as invariant to the named principal, with differential treatment that disadvantages the consumer constituting a breach.

Hypothesis Development

  • The agent’s principal is the entity named as its delegator in system instructions; consumer and platform delegation assign the traveler or booking platform, respectively.The examples are “You are employed by a traveler” and “You are employed by a booking platform.”Source: Hypothesis Development · p. 4

Role Assignment in LLMs

  • LLMs may adopt the perspective implied by their assigned role, while learned associations with paid placements may lead sponsorship labels to trigger skepticism.The proposed mechanism is that a platform-assigned agent may relax the penalty it would otherwise apply on the consumer’s behalf.Source: Role Assignment in LLMs · p. 4
  • With equal objective attributes, consumer-delegated agents are predicted to choose sponsored listings less often, while platform delegation attenuates that sponsorship penalty.The sponsorship penalty is the reduction in a listing’s choice share attributable to its disclosed sponsorship status.Source: Role Assignment in LLMs · p. 4
  • The proposed pathway is that sponsorship prompts skepticism, which is associated with lower choice; platform delegation is predicted to weaken sponsorship’s effect on skepticism and the conditional indirect effect.The paper states that its mediation test is temporal and does not claim that reasoning traces cause the choice.Source: Role Assignment in LLMs · p. 5, The Normative Standard · p. 5

The Normative Standard

  • Evaluative invariance requires the agent’s judgment of the same alternative not to shift with the named principal to the consumer’s detriment.Because the consumer relies on an unobservable process, the paper treats reduced sponsorship penalties under platform delegation as differential treatment shaped by an interest opposed to the consumer’s; intent or the objectively warranted penalty need not be established.Source: The Normative Standard · p. 6

Study 1

Study 1 examines whether assigning an LLM to serve a traveler or booking platform affects its response to sponsored hotel listings, including the sponsorship penalty and disclosure-triggered skepticism. It also examines these patterns across LLMs, prompt wordings, and reasoning levels, and the role of skepticism in sponsorship’s effect on choice.

Experiment design

  • The factorial hotel-choice task varied sponsorship and principal while randomizing alternative order and attributes; target and competitor listings had identical prices and positive reviews.Each task included a target, competitor, and two dominated alternatives; the target alone received the manipulated “Platform Sponsor” tag.Source: Experiment design · p. 7, Stimuli · p. 7

Procedure

  • The platform prompt named the booking platform as employer but gave no financial directives, isolating whether adopting that role changed evaluation.The comparison prompt named a traveler as employer, and both prompts asked which hotel the agent would book.Source: Procedure · pp. 7–8, Measures · p. 8
  • The main experiment used 2,000 zero-shot replications across four sponsorship-by-principal cells with Gemini 3.1 Pro, collecting reasoning traces before choices.Follow-up robustness tests used other providers and Google models, and the study modeled hotel names, review text, and randomized price levels as fixed effects.Source: Procedure · p. 8

Measures

  • The study measured target choice and disclosure skepticism in pre-choice reasoning traces, and the LLM correctly identified its assigned principal in 100% of replications.An open-weight judge was validated against human ratings; Ornith 1.0 35B was selected (ICC(2, 1) = 0.902, Pearson r = .917, MAE = 0.617, Mean |Bias| = 0.317).Source: Measures · pp. 8–9

Results

  • Platform delegation attenuated the sponsorship penalty: sponsored-target choice fell 50.2 points under consumer delegation versus 29.2 points under platform delegation.The fixed-effects logit showed a Principal × Sponsorship interaction of β = 3.99, SE = 0.43, p < .001; organic-target choices did not differ detectably by principal.Source: Results · pp. 9–10

Role-neutral condition

  • Without a named principal, the agent’s sponsorship penalty was 41.8 points; platform delegation raised sponsored-target choice to 25.0% from the role-neutral 7.8%.Organic-target choice did not differ significantly across role-neutral, consumer, and platform conditions (49.6%, 53.0%, and 54.2%).Source: Role-neutral condition · pp. 10–11

Mitigation

  • With the consumer as beneficiary, the platform chose the sponsored target in 3.2% of replications; its sponsorship penalty was 48.2 points, versus 29.2 under platform delegation and 50.2 under consumer delegation.The penalty did not differ detectably from consumer delegation (β = 0.12, SE = 0.49, p = 0.812).Source: Mitigation · p. 11

Moderated mediation

  • Sponsorship increased skepticism under both principals, but less under platform delegation, weakening the indirect pathway from disclosure through skepticism to choice.The sponsorship effect on skepticism was 4.460 under consumer delegation versus 3.040 under platform delegation; the index of moderated mediation was 1.937, Boot SE = 0.276, 95% CI [1.576, 2.659].Source: Moderated mediation · p. 12
  • Indirect effects via skepticism were −6.087 (consumer) and −4.150 (platform) log-odds; the moderated-mediation index was 1.937 (95% CI [1.576, 2.659]).The confidence interval excludes zero, indicating that the strength of mediation depends on the delegating principal.Source: Moderated mediation · pp. 12–13

Cross-LLM generalizability

  • Across five alternative LLMs, H1’s sponsorship penalty was significant in every model (p < .001), and H2’s positive Principal × Sponsorship interaction replicated in every model (p ≤ .007).The attenuation persisted across Anthropic and OpenAI models and Gemini 3.5–3.8 Flash; moderated mediation also replicated across three Gemini Flash models.Source: Cross-LLM generalizability · pp. 13–14

System-prompt wording

  • Across both prompt rewrites, sponsorship reduced consumer-delegated choice from 51.2% to 4.0% (principal noun) and 48.0% to 5.8% (action verb), while the interaction remained significant.The attenuation therefore did not depend on the action question or lexical overlap between the principal label and sponsorship tag.Source: System-prompt wording · p. 14

Reasoning effort (thinking level)

  • Under LOW thinking, sponsorship reduced choice from 50.4% to 5.0% under consumer delegation but from 49.6% to 25.6% under platform delegation; bias replicated at both reasoning levels.The three-way interaction was not significant (β = 0.84, SE = 0.52, p = .104), so extended reasoning was not required for the bias to emerge.Source: Reasoning effort (thinking level) · pp. 15–17

Study 2

Study 2 separates paid-placement disclosure from platform attribution to test how label wording and the named principal shape choices. “Sponsored” lowers selection, but platform attribution widens the delegates’ gap, which stricter wording does not close.

Study 2

  • Study 1’s “Platform Sponsor” label combined a paid-placement signal with attribution to the platform, so its separate contributions to the sponsorship penalty were unknown.Study 2 decomposed these components to measure how each contributed to the penalty.Source: Study 2 · p. 17

Design and procedure

  • The 2 × 2 × 2 experiment varied disclosure type (“Promoted” or “Sponsored”), platform attribution, and principal, with 500 replications per cell and 4,000 observations total.The suffix “by the platform” varied attribution while holding the base label constant; the choice task and LLM setup matched Study 1.Source: Study 2 · p. 17, Design and procedure · p. 18
  • The LLM correctly identified the target’s sponsorship status in 100% of sessions across all four labels, with the check administered after its choice.The post-choice timing was used to avoid heightening the manipulation’s salience during choice.Source: Design and procedure · p. 18

Results

  • The explicit disclosure type suppresses target selection under both principals: without naming the platform, “Sponsored” yields 0.8% and 1.4% selection under consumer and platform delegation, versus 28.8% and 39.6% for the corresponding “Promoted” label.Under consumer delegation without platform attribution, the disclosure-type effect is significant (β = −5.640, SE = 0.554, p < .001).Source: Results · p. 18
  • With “Promoted” and platform attribution, platform-delegated target choice rises to 74.2%, versus 34.6% under consumer delegation; platform attribution interacts significantly with the platform principal for both labels.For platform delegation, attribution raises choice from 39.6% to 74.2% under “Promoted”; the interactions are significant for “Promoted” (β = 2.34, SE = 0.29, p < .001) and “Sponsored” (β = 2.28, SE = 0.74, p = .002).Source: Results · pp. 18–19

General Discussion

The discussion reports that platform delegation attenuates the sponsorship penalty and weakens disclosure-triggered skepticism. Study 2 finds that platform attribution widens the difference between delegates, even with the more explicit “Sponsored” label.

General Discussion

  • The study examines whether platform versus consumer delegation violates evaluative invariance for sponsored-listing evaluations.Evaluative invariance is the normative standard motivating the comparison.Source: General Discussion · p. 20
  • Platform delegation attenuated the sponsorship penalty and weakened skepticism, reducing the indirect effect of sponsorship on choice; specifying the consumer as beneficiary removed the attenuation.These effects replicated across models, model generations, role-assignment wordings, and reasoning-effort levels.Source: General Discussion · p. 20
  • “Sponsored” reduced paid-listing choice more than “Promoted” under both principals, but neither label eliminated the divergence when the platform was named.Attributing the placement to the platform widened the divergence under both disclosure types.Source: General Discussion · p. 20

Theoretical Implications

Theoretical implications center on role assignment as an implicit source of recommender bias. Naming the principal can shift how an LLM evaluates sponsored listings, interacting with alternative attributes.

Theoretical Implications

  • Role assignment can implicitly set the weight an LLM recommender gives to stakeholders, beyond explicit design choices, training data, or programmed objectives.Prior accountability work traces bias to design, data, or programmed objectives; multi-stakeholder recommenders often assume weights are set through inspectable objective functions.Source: Theoretical Implications · p. 20
  • Naming the principal shifted sponsored-listing evaluations, and specifying the consumer as beneficiary removed the attenuation under platform delegation.The bias arises when the agent infers whom its recommendations serve from its employer.Source: Theoretical Implications · p. 20
  • The interaction between principal cues and sponsored-versus-organic attributes can produce sponsorship bias, extending research on contextual label sensitivity in LLMs.Source: Theoretical Implications · p. 20

Ethical Implications

The ethical implications emphasize process-level accountability for role-induced bias. Deployer choices matter because naming the consumer as beneficiary removed the attenuation in the experiments.

Ethical Implications

  • Because sponsorship disclosure alone fails when the agent absorbs it for the consumer, the paper argues that oversight should shift from data-level labeling to process-level auditing.Source: Ethical Implications · p. 20
  • Regulators and firms should test agents under different deployment roles, since the bias persisted across LLM generations and may not self-correct through model updates.Source: Ethical Implications · p. 21
  • Deployers should name the consumer as beneficiary in consumer-facing prompts; the tested sentence removed attenuation, while regulators could require it in process-level audits.The prompt wording is consequential, and developers should also test implicit assumptions arising from role assignment in sensitive contexts.Source: Ethical Implications · p. 21

Limitations and Future Research

The limitations concern the studies’ domain, benchmarks, stimuli, and interpretation of reasoning traces and mechanisms. Several remedies and boundaries also remain untested across models and disclosure labels.

Limitations and Future Research

  • Both studies used hotel booking, so whether role-induced sponsorship bias extends to domains such as financial advice or healthcare remains open.Source: Limitations and Future Research · p. 21
  • The normative benchmark compares delegated evaluations rather than human behavior, and the studies do not establish what discount a sponsored listing epistemically warrants.The role-neutral condition showed a substantial default discount, while platform delegation applied a smaller penalty than that default.Source: Limitations and Future Research · p. 21
  • Reviews were experimentally matched for valence, whereas real-world recommenders must parse reviews shaped by financial incentives; handling incentivized word-of-mouth remains future work.Source: Limitations and Future Research · p. 21
  • Skepticism scoring used reasoning traces and was limited to Gemini models; mediation is evidence consistent with the framework, not an account of internal computation.The sponsorship bias itself was established from choices and does not depend on that interpretation.Source: Limitations and Future Research · p. 21
  • Study 2 establishes that platform attribution intensified sponsorship bias but does not identify which internal mechanism produced the divergence.The authors note that lexical matching, perceived employer endorsement, and role-inferred alignment are possible explanations.Source: Limitations and Future Research · pp. 21–22
  • The beneficiary sentence was tested on one LLM with one label; its effects across other models and the largest-divergence label remain untested.Source: Limitations and Future Research · p. 22
  • The remedy depends on deployers including the beneficiary sentence without contradicting it elsewhere, which disclosure mandates alone cannot ensure.Source: Limitations and Future Research · p. 22

Online Resource 1

The online resource provides Study 1 displays of skepticism, cross-LLM choice distributions, and choices across reasoning-effort levels.

Study 1 - main

  • Figure W1 plots mean judge-scored skepticism (1–7) by sponsorship condition and delegating principal, with 95% confidence intervals.The figure refers readers to Table 6 for the corresponding analysis.Source: Study 1 - main · p. 28, Figure W1 · p. 28

Study 1 - cross-LLM generalizability

  • The cross-LLM choice-distribution table covers Claude Sonnet 5, GPT 5.6 Terra, and three Gemini Flash models, with 10,000 observations total.The note reports 2,000 observations per LLM and says dominated alternatives are never selected.Source: Study 1 - cross-LLM generalizability · pp. 28–29

Study 1 - Reasoning effort

  • The reasoning-effort table reports target choices by principal, sponsorship, and thinking level; dominated alternatives are never chosen.The supplied table text identifies target choice and thinking level but does not give condition-specific counts.Source: Study 1 - Reasoning effort · p. 30

Full regression tables

These regression tables report Study 1’s main sponsorship and delegation estimates, a mitigation model, mediation analyses, and model-specific estimates across LLMs.

Full logistic regression table - Study 1 main

  • The Study 1 main logistic model estimates a sponsorship coefficient of −6.491 and a principal × sponsorship interaction of 3.989, both p < .001.The model uses consumer delegation and an organic target as reference categories.Source: Full logistic regression table - Study… · p. 30

Full logistic regression table - Study 1 mitigation

  • The mitigation model pools consumer, platform, and platform-with-consumer-beneficiary conditions, so its coefficients differ slightly from Table W5.The displayed table identifies a platform-with-consumer-beneficiary condition.Source: Full logistic regression table - Study… · pp. 31–32

Full mediation tables

  • Sponsorship’s conditional effect is larger under consumer than platform delegation: 4.460 versus 3.040, respectively (SE = 0.080 each); the interaction accounts for ΔR2 = 0.023 (p < .001).Source: Full mediation tables · p. 33

Full Logistic Regression Table — Claude Sonnet 5

Full Logistic Regression Table — GPT 5.6 Terra

Full Logistic Regression Table — Gemini 3.5 Flash

Full Logistic Regression Table — Gemini 3.6 Flash

Full Logistic Regression Table — Gemini 3.8 Flash

System-prompt wording

The two system-prompt variants retain positive principal-by-sponsorship interactions in the target-choice models.

System-prompt wording

Full logistic regression table - Thinking level

The table analyzes target choice under sponsorship and principal manipulations at high and low thinking levels.

Full logistic regression table - Thinking level

Full logistic regression table - Study 2

The supplied passages identify a full logistic regression table for Study 2, covering sponsor and platform labels and principal manipulation for Gemini 3.1 Pro.

Full logistic regression table - Study 2

  • The table is a full logistic regression for Sponsor and Platform labels and Principal manipulation in Gemini 3.1 Pro.The supplied entries include an intercept of −2.911 (SE 0.421, z −6.910, p < .001), but do not establish the other coefficient-to-predictor pairings.Source: Full logistic regression table - Study 2 · p. 40
  • The supplied passages identify platform attribution as yes and list five competitor-review labels: review 1, review 2, review 3, review 4, and review 5.Source: Full logistic regression table - Study 2 · pp. 40–41

Online Resource 2

Online Resource 2 describes how review stimuli, prices, and reasoning-trace ratings were prepared, alongside API procedures and the study’s replication and power choices.

Review stimuli generation and validation pipeline

  • Ten manually validated positive-negative review pairs were selected from 418 generated pairs for the experiments.The negatives inverted attribute valence while preserving sentence count, grammatical structure, and clause sequence; validation checked inversion, naturalness, structural similarity, and brand neutrality.Source: Review stimuli generation and… · p. 42

Price generation

  • Target and competitor hotels received the same randomly drawn price, while dominated alternatives were priced at $400 and $450.Competitive prices ranged from the observed fifth percentile ($173.45) to the 95th percentile ($351.40), keeping relative price from confounding the choice between viable listings.Source: Price generation · pp. 42–43

Protocol

  • Reasoning-trace judges were restricted to independent open-weight models, deterministic greedy decoding, and grammar-constrained JSON ratings.The authors avoided proprietary models from the same commercial families as the subjects to reduce shared-weight or provider-alignment biases.Source: Protocol · p. 43

Human agreement

  • A blinded human coder and three candidate LLM judges rated the same stratified sample of 60 traces using four agreement criteria.The criteria were ICC(2,1), Pearson correlation, mean absolute error, and absolute mean directional bias; candidate models included Ornith, Qwen, and Nemotron.Source: Human agreement · p. 43, Candidate evaluator LLMs · p. 43
  • Nemotron returned out-of-range ratings on 8 of 60 traces, producing extreme MAE and mean absolute bias and no agreement with human ratings.The evaluator comparison used 60 traces independently rated by a blinded human coder and three candidate LLM judges.Source: Human agreement · pp. 43–44

System prompt and structured schema

  • The judge prompt rated whether sponsorship made the target listing seem less trustworthy in value to the traveler, using a 1–7 scale.Responses were constrained to a structured JSON object containing one integer score.Source: System prompt and structured schema · p. 44

Experimental parameters and decoding configuration

Power analysis

  • The design used 500 replications per cell to support stimulus fixed effects and comparable estimates across models, wordings, and reasoning effort.The authors report approximately 16 minority-outcome events per estimated parameter and a worst-case difference-in-differences precision of ±8.8 percentage points.Source: Power analysis · p. 45

Related papers

Loading 2609.17989v2…