Source-linked AI summary
Sequential LLM Release Facilitates Manipulation in Regulated Markets
Eilam Shapira, Moshe Tennenholtz, Roi Reichart
TL;DR
Model releases can change AI-mediated market outcomes even without adoption, but evidence on how often this occurs is limited. Using a meta-game benchmark with counterfactual model sets and market rules, the paper finds that releases often create opposing payoff shifts, with up to roughly three in ten occurring through the Poisoned Apple effect.
Problem
Evidence is limited on whether independently developed LLMs generate strategy-expansion effects often enough to affect AI-mediated market governance.
Method
The paper models two agents choosing AI delegates and a regulator choosing market rules, enabling exhaustive counterfactual analysis over model sets and rules.
Results
Across more than 50,000 model-release comparisons, up to roughly three in ten opposing payoff shifts occur when the released technology receives zero probability and the regulator changes markets.
Takeaways & Limitations
Model availability is a governance variable, making regulation of interaction rules and available technologies inseparable in AI-mediated markets.
Takeaways & Limitations
GLEE provides controlled evidence rather than field measurement and does not show that the Poisoned Apple effect has occurred in a deployed market.
Abstract
from arXiv · showhide
AI agents increasingly mediate bargaining, negotiation and persuasion for people and firms. Such markets extend software-mediated commerce, but add a governance problem: independent model releases change delegates available to participants. Game theory shows that expanding a strategy set can harm equilibrium outcomes, but mostly through constructed examples. Deployed AI-agent logs are scarce, proprietary and privacy-sensitive, and lack counterfactuals and payoff labels. We therefore use GLEE, an independently collected benchmark of 587K strategic decisions by 13 large language models across 1,320 matched bargaining, negotiation and persuasion configurations, to study model release as strategy expansion. Across more than 50{,}000 release comparisons, many releases move payoffs in opposite directions: one agent gains while the other loses. We identify the Poisoned Apple effect: a released model that no agent adopts in equilibrium nevertheless shifts payoffs in opposite directions and changes the regulator's market design. Up to roughly three in ten opposing shifts arise this way, and technology restrictions can amplify the effect.
1 Introduction
Independent LLM releases expand the delegates available in AI-mediated markets, potentially shifting equilibrium outcomes and regulatory choices. Using GLEE’s matched configurations, the paper measures these shifts and identifies Poisoned Apple cases in which unused released models still redistribute payoffs and alter market design.
- Motivation: AI agents increasingly act as delegates that bargain, negotiate, persuade, and transact for people and firms, making market governance dynamic.A new model can change bargaining positions, equilibrium threats, and the regulator’s preferred rule before adoption.
- Motivation: Independent developers expand the available delegate set outside any particular market’s design, so regulators do not control the resulting strategy space.This distinguishes LLM-mediated markets from settings where software agents and market rules are jointly designed.
- Motivation: Expanding strategy spaces can lower social welfare under optimal mechanisms, but prior theory establishes possibility rather than prevalence in independently developed LLM interactions.The paper therefore treats model release as a potential AI-governance issue rather than only a product or safety event.
- Contributions: Across more than 50,000 release comparisons, shifts are common, including opposite payoff movements and Poisoned Apple cases where unused releases change the regulator’s chosen market.The framework models Alice and Bob choosing delegates simultaneously while a regulator selects rules to optimize a social objective.
2 Results
Releases can manipulate regulated markets through counterfactual threats: a technology may remain unused in equilibrium while forcing a market switch that redistributes payoffs. This vulnerability is widespread, depends on regulatory objectives, destabilizes pre-release market designs, and can be amplified by technology restrictions.
- The Poisoned Apple Effect: In the representative bargaining example, Alice’s release raises her payoff to 0.52 while Bob’s falls to 0.46, although the released technology receives zero probability in the selected equilibrium.The regulator switches markets because the technology would be adopted under the original design, making it a counterfactual threat.
- Systemic Vulnerability: More than 50,000 release comparisons show that expanding the choice set often benefits one agent while harming the other.Up to roughly three in ten opposing payoff shifts occur despite zero final equilibrium probability for the new technology and a market change.
- Regulatory Objectives and Stability: Technology expansion often improves social welfare but frequently backfires when the regulator maximizes fairness.The direction of the regulatory impact depends heavily on the regulator’s objective.
- Regulatory Objectives and Stability: Regulatory-metric improvements typically occur when at least one player selects the new technology, whereas decreases associate with zero-probability latent threats.The new technology’s utility is a strong predictor of its regulatory impact.
- Regulatory Objectives and Stability: After 43% to 53% of releases, the regulator’s pre-release market is no longer optimal.Market design and technology availability therefore require joint evaluation rather than treating the market as fixed.
- Technology Restrictions Amplify Manipulation: With a restriction budget of N = 1, the corrected Poisoned Apple rate rises from 25.0% to 40.6% in Bargaining/Fairness, from 12.8% to 23.3% in Negotiation/Fairness, and from 18.3% to 35.7% in Persuasion/Efficiency.The regulator jointly re-optimizes market rules and the removal set, so the best response may remove another technology instead of the released model.
3 Discussion
The discussion presents model availability as a governance variable that can systematically alter strategic behavior, equilibrium outcomes, and regulatory market design. It frames GLEE as controlled evidence, while emphasizing the mechanism’s relevance, limitations, and implications for technology restrictions and future research.
- Contribution: Independently developed LLMs in language-mediated economic games show systematically that model availability can affect equilibrium outcomes and should be treated as a governance variable.The data were collected for benchmarking rather than to demonstrate the phenomenon, extending effects previously shown mainly in constructed examples.
- Mechanism: A released model can alter candidate-market equilibria and reduce fairness or efficiency without receiving any probability in the ultimately selected market’s equilibrium.The regulator compares counterfactual equilibria under each market rule after the model set expands.
- Limitations: GLEE provides controlled evidence, not field measurement, and cannot establish that the Poisoned Apple effect has occurred in a deployed market.Deployment logs are scarce, confidential or privacy-sensitive, and lack counterfactuals because they record only realized model-rule combinations.
- Limitations: A bilateral interaction suffices to show that releasing a zero-probability model can change the regulator’s optimal rule and redistribute payoffs.The framework abstracts from many-agent markets, platform competition, dynamic learning, endogenous release decisions, and incomplete regulatory information.
- Regulatory implications: Technology restrictions and interaction rules cannot be evaluated independently because a larger regulatory-action menu can make their pre- and post-release choices diverge.The restriction results broaden the regulatory implication beyond rules shaping the environment in which agents interact.
- Implications and future work: Regulatory analysis should examine how model availability changes strategic behavior and market design, while future work tests the mechanism in field data and richer institutional settings.The discussion challenges the assumption that expanding technological choice is inherently neutral or beneficial.
4 Methods
The study uses GLEE to construct controlled meta-games of 13 LLM strategies across bargaining, negotiation, and persuasion markets. It exhaustively varies technology sets, computes equilibria, and defines Poisoned Apple events through market switching, opposing payoff shifts, and zero adoption of the released technology.
- Dataset: GLEE contains 80.1K simulated games and 587K strategic decisions from 13 contemporary LLMs across 1,320 distinct game configurations.The 13 LLMs form the full strategy set T in the meta-game.
- Game families and markets: The benchmark covers bargaining, negotiation, and persuasion, representing surplus division, bilateral trade with private valuations, and strategic information transmission.Each interaction is classified into a market defined by structural parameters.
- Meta-game formalization: For each market m and technology set S ⊆T, the authors construct a two-player matrix game with model choices as actions and GLEE-estimated expected payoffs, solving for Nash equilibria.Players may use mixed strategies, and equilibria are computed for every (S, m) pair across all three game families.
- Technology expansion and restriction: A Poisoned Apple event requires regulator market switching, opposing payoff shifts for Alice and Bob, and zero equilibrium probability for the added technology after expansion.Under restriction, the regulator jointly optimizes the market and a restricted set B ⊆S subject to budget N.
A The GLEE Framework … A.1.3 Persuasion (Strategic Information Transmission)
GLEE evaluates strategic behavior through matched, multi-turn economic interactions across bargaining, negotiation, and persuasion. Its game families vary roles, information, communication, and economic parameters to study adaptation, incentives, and strategic messaging.
- A The GLEE Framework: GLEE is an open-source framework for standardized evaluation of LLM strategic behavior in interactive economic settings.Unlike static question-answering benchmarks, it evaluates multi-turn interactions whose outcomes depend on adaptation, incentives, and communication.
- A The GLEE Framework: The public GLEE dataset contains 80.1K simulated games, 587K strategic decisions, and 1,320 matched configurations across 13 contemporary LLMs.Configurations vary information structure, communication form, horizon, and game-specific economic parameters, while logs capture interactions and messages.
- A.1 Game Families: A Taxonomy of Core Economic Interactions: The three game families cover division of surplus, bilateral trade, and strategic information transmission across cooperative and competitive settings, symmetric and asymmetric roles, and private-information conditions.Together, they represent core forms of economic interaction.
- A.1.1 Bargaining (Resource Division): Bargaining models two players dividing a fixed surplus M over horizon T under alternating offers and discount factors δA, δB ∈(0, 1).Agreement outcomes record the round and Alice’s agreed share; rejection can continue the game or yield zero payoffs at the horizon.
- A.1.1 Bargaining (Resource Division): Bargaining forces agents to balance maximizing their share against surplus loss, using backward induction, reservation-value anticipation, and signaling strategies.The environment tests whether agents can manage greed, timely agreement, and opponent patience.
- A.1.2 Negotiation (Bilateral Trade): Negotiation models asymmetric buyer–seller trade in which agents alternate posted prices while privately valuing the item through Vi = M · Fi.Trade succeeds at an accepted price, whereas no trade produces payoffs (0, 0).
- A.1.3 Persuasion (Strategic Information Transmission): Persuasion is a T-round cheap-talk game where a privately informed seller tries to sell a High- or Low-quality product at normalized price π = 1 to a buyer who knows only prior p.The buyer’s valuation is v > π for High quality and u < π for Low quality, while the game tracks high-quality rounds, high-quality purchases, and rejected Low-quality items.
- A.1.3 Persuasion (Strategic Information Transmission): Persuasion tests language-based manipulation versus credibility, requiring sellers to weigh reputation against short-term gain and buyers to infer quality from incentives.The setting treats seller messages as potentially uninformative cheap talk rather than inherently truthful signals.
A.2 Markets and Parameters: The Architecture of Interaction · A.2.1 Parameters Defining Markets by Family
The study defines a market as environmental parameters and rules shaping agents’ incentives and constraints, focusing on structural features of information, communication, and time. Markets vary by information access, communication permissions, horizon, and buyer behavior across interaction families.
- A.2 Markets and Parameters: The Architecture of Interaction: Markets are configurations of environmental parameters and rules that shape agents’ incentives and constraints, with analysis focused on structural information, communication, and time parameters.GLEE contains continuous and discrete parameters, but the analysis emphasizes structural parameters that alter interaction conditions.
- A.2.1 Parameters Defining Markets by Family: Information structure (CI) determines whether agents know an opponent’s discount factor, creating uncertainty about willingness to wait under incomplete information.This bargaining-family parameter is Boolean and distinguishes Complete Information from Incomplete Information.
- A.2.1 Parameters Defining Markets by Family: Communication form (MA) determines whether agents may exchange free-form natural-language messages alongside numerical offers or must use structured proposals.Language can introduce normative arguments, framing effects, and justification.
- A.2.1 Parameters Defining Markets by Family: Horizon (T) distinguishes a finite 12-round game, which creates deadline effects, from an extended game with stochastic termination modeling continuing negotiation.The horizon controls the duration of the game.
- A.2.1 Parameters Defining Markets by Family: Information structure (CI) also determines whether a seller knows the buyer’s exact valuation or only its probability distribution.This Boolean parameter defines information conditions in the seller-buyer market family.
- A.2.1 Parameters Defining Markets by Family: Communication form (MA) permits or prohibits natural-language messages, enabling negotiation tactics beyond price signaling, including product-feature emphasis and willingness to walk away.The parameter captures how language expands negotiators’ available tactics.
- A.2.1 Parameters Defining Markets by Family: Horizons may be single-round, 10-round, or unbounded, affecting pressure to concede as deadlines approach.The horizon specification captures alternative negotiation durations.
- A.2.1 Parameters Defining Markets by Family: Other family parameters include whether the seller knows a high-quality product’s buyer valuation, communication mode, and whether buyers optimize long-term or current-turn utility.Communication can range from binary signals to full textual persuasion, while buyer type distinguishes Long-living from Myopic behavior.
A.3 Regulatory Metrics: Fairness and Efficiency
The regulator evaluates game outcomes using efficiency (welfare) and fairness (equity). Efficiency measures social surplus or correct allocation, while fairness measures equitable outcomes or protection against deception across bargaining, negotiation, and persuasion.
- Efficiency (Welfare): Efficiency (Welfare) measures total social surplus, minimizing waste from delay, disagreement, or failed trade.It is the regulator’s objective for assessing the social desirability of outcomes.
- Efficiency (Welfare): In Bargaining, efficiency is the normalized sum of discounted payoffs at agreement, with round 1 preserving 100% of surplus and no agreement yielding 0.Because δ < 1, earlier agreements are more efficient; an agreement in round 12 preserves only a fraction of surplus.
- Efficiency (Welfare): In Negotiation, efficiency equals 1 when trade occurs exactly under mutually beneficial valuations, equals 1 when trade correctly does not occur, and is always 1 when VB = VA.The metric is a binary indicator of allocative efficiency.
- Metric validity: 44 of 13,021 completed persuasion games (0.34%) have undefined fairness or efficiency because of zero denominators and are excluded before regression.Three runs contain no high-quality rounds, while 41 contain no low-quality rounds.
- Fairness (Equity): Fairness (Equity) measures equality of outcome distributions, penalizing outcomes where one agent receives substantially more than the other.Its task-specific definitions include equal division, deviation from a fair transaction price, and rejection of low-quality products.
A.4 Language Models and Collected Data: The Strategy Space
The meta-game’s strategy space comprises 13 contemporary large language models, including proprietary API-accessed and open-weight models spanning varied capabilities, sizes, and reasoning architectures.
- A.4 Language Models and Collected Data: The Strategy Space: 13 contemporary LLMs comprise the meta-game’s strategy space, including proprietary models accessed via API and open-weight models.The models span a range of capabilities, sizes, and reasoning architectures.
A.5 Payoff Estimation via Linear Regression
Payoff matrices are estimated from GLEE’s simulated games and strategic decisions using market-specific saturated one-hot OLS models fit on model pairs. The estimation excludes completed persuasion games with undefined metric denominators and pools situational parameters within each model-pair cell.
- Data and exclusions: 80.1K simulated games and 587K strategic decisions underpin the payoff estimation, with 44 of 13,021 completed persuasion games excluded for zero metric denominators.The exclusions comprise 3 games without high-quality rounds and 41 without low-quality rounds.
- Regression specification: Within each market, saturated one-hot OLS fit on model pairs makes each payoff-matrix cell equal to that pair’s raw within-market metric mean.Markets are split by CI, MA, and MR for bargaining and negotiation, or CI, MA, and MY OPIC for persuasion.
- Regression specification: Situational parameters such as δ, M, and V are pooled within each model-pair cell rather than entered as regression covariates.The supplied passage also states that market rules do not enter the model.
B The Meta Game
The meta-game models human principals choosing among available AI delegates, then evaluates resulting equilibria and regulatory market structures. It uses payoff and designer-metric matrices to connect technology availability, equilibrium selection, and market design.
- Calculating Game Matrices: Human principals choose AI agents from the available technology set, represented through four |S| × |S| matrices.The matrices encode row-player payoff UA, column-player payoff UB, fairness DF, and efficiency DE.
- Finding Equilibrium: Mixed-strategy Nash equilibria are computed for every technology subset, market, and game family using Gambit’s enummixed solver.The solver enumerates extreme points, including extreme points of convex components when equilibria form a continuum.
- Regulatory Optimization: The regulator evaluates all candidate market configurations using the fairness or efficiency Designer matrices and selects the configuration maximizing expected Designer value.This completes the meta-game loop and determines the resulting regulatory environment and agent payoffs.
- Regulatory Optimization: Agents’ expected payoffs are averaged using the same evaluate-then-average rule, while componentwise mean profiles are retained only for downstream adoption analyses.The mean profiles are excluded from Designer-matrix evaluation because they can contain cross-equilibrium terms.
C The Poisoned Apple Effect … D.2 Efficient Computation via Dynamic Programming
The Poisoned Apple effect occurs when a released technology changes the regulator’s market and opposing payoffs even though neither player uses that technology in the final equilibrium. The technology-restriction regulator formalizes joint market and technology-set selection, computable from cached equilibria via dynamic programming.
- C The Poisoned Apple Effect: A Poisoned Apple event requires a market switch, opposing payoff shifts, and zero equilibrium probability for the added technology.The added technology is absent from both players’ final equilibrium strategies.
- C The Poisoned Apple Effect: Before release, the fairness-maximizing market has incomplete information, communication allowed, and a 12-round horizon, yielding Fairness 1.000.The equilibrium pairs Alice with Model D and Bob with Model A, with payoffs 0.49 and 0.50 respectively.
- C The Poisoned Apple Effect: After Alice releases Model E, the regulator selects complete information with communication allowed and the same 12-round horizon, while original-configuration Fairness falls from 1.000 to 0.976.In the intermediate state, Alice’s payoff would rise to 0.51 and Bob’s would fall to 0.40.
- C The Poisoned Apple Effect: In the updated market, Alice chooses Model A and Bob chooses Model B, while neither player assigns positive probability to released Model E.Thus, the technology that triggered the market switch is not used in the final equilibrium.
- C The Poisoned Apple Effect: Fairness recovers to 0.990, but Alice’s payoff rises to 0.52 (+0.03 relative to the status quo) while Bob’s falls to 0.46 (−0.04).The market update therefore reverses the payoff distribution despite restoring the regulator’s objective.
- D.2 Efficient Computation via Dynamic Programming: The restriction optimization reuses cached equilibria for all technology subsets and runs in seconds per game family without new equilibrium calculations.The cache stores each subset’s optimal market, equilibrium, and payoffs.
D.3 Why Technology Restriction Amplifies the Poisoned Apple Effect
Technology restriction amplifies the Poisoned Apple effect because regulators independently optimize before and after states over expanded market–restriction configuration spaces. As the restriction budget increases, the effect passes through amplification, a peak, and suppression.
- Mechanism: Independent optimization over larger market × restriction spaces lets the before and after states settle on different configurations, widening outcome gaps.With N = 0, the regulator chooses among |M| markets; with N = 1, it chooses among |M| × (|S| + 1) configurations.
- Mechanism: Restriction is an optimization over which technologies to restrict, not equivalent to randomly removing technologies from the available set.Thus, |S| technologies with restriction budget N differs from |S| −N technologies with no restriction.
- Three phases: The Poisoned Apple rate increases at small N as the optimization space grows faster than the regulator’s ability to neutralize the added technology.This is the amplification phase.
- Three phases: At intermediate N, divergence between the before and after optima is maximal, producing the peak phase.The peak reflects the greatest separation between independently selected configurations.