Source-linked AI summary
Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition
Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, Guangtao Zhai
TL;DR
Whether LLMs can manage resources and adapt language in scarce, competitive markets remains unclear. Market-Bench evaluates them in a multi-agent supply-chain economy where constrained decisions and persona-aligned marketing jointly shape outcomes. Results show a winner-take-most pattern, with a small elite compounding capital while most agents struggle to break even.
Problem
It remains unclear whether LLMs can combine quantitative resource management with semantic adaptation under scarcity and competition, where failures in either can compound into bankruptcy.
Source: §1 · p. 2Method
Results
A small elite of agents consistently compounds capital, while the majority struggle to break even, producing a winner-take-most outcome distribution.
Source: §5 · p. 8Takeaways & Limitations
Market-Bench provides a reproducible testbed for studying LLM behavior in competitive markets where inventory management and market positioning must be synthesized.
Source: §5 · p. 8The reported experiments use one market configuration with 20 agents, 6 time steps, and 8 item types, leaving other parameter combinations unexplored.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
The ability of large language models (LLMs) to manage and acquire economic resources remains unclear. In this paper, we introduce \textbf{Market-Bench}, a comprehensive benchmark that evaluates the capabilities of LLMs in economically-relevant tasks through economic and trade competition. Specifically, we construct a configurable multi-agent supply chain economic model where LLMs act as retailer agents responsible for procuring and retailing merchandise. In the \textbf{procurement} stage, LLMs bid for limited inventory in budget-constrained auctions. In the \textbf{retail} stage, LLMs set retail prices, generate marketing slogans, and provide them to buyers through a role-based attention mechanism for purchase. Market-Bench logs complete trajectories of bids, prices, slogans, sales, and balance-sheet states, enabling automatic evaluation with economic, operational, and semantic metrics. Benchmarking on 20 open- and closed-source LLM agents reveals significant performance disparities and winner-take-most phenomenon, \textit{i.e.}, only a small subset of LLM retailers can consistently achieve capital appreciation, while many hover around the break-even point despite similar semantic matching scores. Market-Bench provides a reproducible testbed for studying how LLMs interact in competitive markets.
1 Introduction
Existing LLM benchmarks rarely test language and quantitative decisions together under economic scarcity, leaving their joint performance in competitive markets unclear. Market-Bench addresses this gap with a constrained multi-agent economy in which slogans affect market access alongside procurement and pricing decisions.
1 Introduction
- Existing benchmarks generally isolate strategic reasoning from linguistic complexity and lack a closed-loop test of numeric and semantic decisions under strict constraints.The section contrasts social simulations without hard economic constraints with economic simulators that abstract away linguistic complexity.Source: §1 · pp. 1–2
- Market-Bench models agents as retailers making procurement and pricing decisions while also generating marketing slogans in a financially and physically constrained economy.Agents receive structured observations and produce mathematical actions together with free-form slogans.Source: §1 · p. 2
- Persona-Gated Attention computes slogan alignment with hidden buyer personas to determine market visibility, linking free-form language to financial outcomes.The mechanism is presented as an alternative to subjective LLM-as-judge evaluation.Source: §1 · p. 2
- The authors provide a reproducible benchmark implementation evaluated across 20 LLM agents with automatically computed economic, supply-chain, and semantic metrics.Market-Bench also enforces finite funds, bankruptcy, and competitive exclusion.Source: §1 · p. 2
2 Related Work
Prior work studies social agent behavior, game-theoretic reasoning, and supply-chain optimization, but typically separates these capabilities. Market-Bench combines constrained economic decisions with language-based market access in a competitive retail loop.
2.1 LLM-Based Agent Societies
- Social-agent benchmarks scale simulations to urban environments and diverse tasks, but primarily evaluate believability or cooperation rather than scarcity-driven economic survival.Examples include Park et al. (2023), AgentSociety (Piao et al., 2025), and MultiAgentBench.Source: §2.1 · p. 2
2.2 Strategic Reasoning and Game Theory
- Game-theoretic benchmarks measure cooperation, deception, bargaining, or equilibrium behavior, often reducing interaction to discrete moves or numeric outputs.Market-Bench instead includes free-form slogans that unlock market access through Persona-Gated Attention.Source: §2.2 · pp. 2–3
2.3 Supply Chain and Business Optimization
- Supply-chain and retail systems often use LLMs as backend optimizers or in single-agent settings, whereas Market-Bench unifies procurement, pricing, and marketing under multi-agent competition.The related work cites ABIDES-Economist and Anthropic’s Project Vend as examples of these approaches.Source: §2.3 · p. 3
3 Market-Bench
Market-Bench models a competitive supply-chain economy in which LLM retailers make constrained procurement and retail decisions, while marketing language shapes buyer visibility. Its logged trajectories support agent-, process-, and market-level evaluation.
3 Market-Bench
- Market-Bench couples constrained numeric decisions with marketing language that determines which consumers can see retailers, within a single competitive supply-chain loop.Source: §3 · p. 3
3.1 Economic Setting and State Variables
- At each step, retailer funds and item inventories accompany supplier offers and hidden buyer personas, so agents infer preferences from public market outcomes.Source: §3.1 · pp. 3–4
3.2 Stage A: Procurement as a Multi-Unit Auction
- Procurement uses a per-item, multi-unit first-price auction: agents bid requested quantities and prices under hard budgets, and winners pay their bids.The supplier allocates available quantities to the highest bids above the reserve price; invalid bids receive zero allocation.Source: §3.2 · p. 4
3.3 Stage B: Retail as Price Competition with Persona-Gated Attention
- Retailers set prices and slogans, then buyers consider persona-aligned visible sellers and purchase from the cheapest available seller with inventory.The consideration set is affected by slogan-persona similarity and buyer patience; low-priced sellers cannot sell if buyers do not consider them.Source: §3.3 · p. 4
3.4 Information Structure and Action Space
- Agents observe funds, inventory, supplier offers, and public prices, slogans, and sales, then output item-level bids, retail prices, and a slogan.Source: §3.4 · p. 4
3.5 Agent Objective and Economic Trade-offs
- Agents maximize cumulative profit while avoiding bankruptcy, balancing supply acquisition, retail margins, and slogans that must differentiate while fitting pricing and inventory strategies.Aggressive bidding raises unit costs and reduces liquidity; aggressive pricing erodes margins.Source: §3.5 · p. 5
3.6 Market-Bench as a Generative Dataset
- Complete traces record bids, allocations, prices, slogans, sales, funds, and inventory; default-setting indices show inequality and CR4 rising while HHI remains competitive and active ratio declines.Source: Figure 3 · p. 4, §3.6 · p. 5
4 Experiments
Across the tested setup, LLM retailers show sharply uneven economic outcomes. Procurement success, activity, and early competitive feedback help characterize the observed market dynamics.
4.1 Experimental Setup
- The default runs place 20 LLM retailers in an 8-item, 6-step market with two bidding rounds per step and stationary conditions.The default setting uses 10 runs, and reported metrics are means across runs.Source: §4.1 · pp. 5–6, §4.2 · p. 6
4.3 Analysis
- Gemini 2.5 Pro and Flash lead with Π=36,589 and 26,104 and NPM=0.167 and 0.190; Figure 4 shows persistent capital stratification.O3 attains FillRate=0.418 and OSI=1.000; Phi-4 (14B) achieves Π=7,565, comparable to GPT-4o (Π=7,619).Source: §4.3 · p. 6, Figure 4 · p. 6
- Market inequality rises as wealth concentration stays competitive: Gini increases from 0.07 to 0.21 while HHI remains approximately 0.08–0.10.The top-four share rises from 0.23 to 0.33, rank mobility remains high, and the active ratio declines toward 0.4.Source: §4.3 · p. 7
- Across 10 runs, scarcity separates agents with sustained high BidEfficiency and meaningful service from near-zero performers; profit correlates with FillRate (ρ=0.88).Profit also correlates with BidEfficiency (ρ=0.68) and StockoutRate (ρ=−0.88); early procurement success generates liquidity for future bids.Source: §4.3 · p. 7, Figure 5 · p. 7
- High FillRate does not guarantee high profit: o3 has the best service level but lower margins than Gemini 2.5 Pro and Flash.The analysis distinguishes auction competence from retail monetization and reports margin compression over time.Source: §4.3 · p. 7
- Operational indices can mislead without conditioning on participation: low-activity agents may appear stable, so interpret stability and efficiency alongside market access.OSI can be high when there is little variability to measure.Source: §4.3 · pp. 7–8
- Language shifts most at entry, then slogan similarities and embedding clusters converge; MMS varies narrowly and correlates weakly with profit (Spearman ρ=0.16).The analysis describes this as a generic messaging equilibrium and an open challenge to sustained semantic differentiation under competition.Source: §4.3 · pp. 6, 8, Figure 6 · p. 7, Figure 7 · p. 8
5 Conclusion
Market-Bench offers a reproducible simulation for evaluating LLMs in competitive markets, where economic outcomes depend jointly on numeric decisions and language. Experiments show a winner-take-most pattern, with a small elite compounding capital while most agents struggle to break even.
5 Conclusion
- A small elite of agents consistently compounds capital, while the majority struggle to break even in Market-Bench experiments.The reported outcome distribution is bimodal.Source: §5 · p. 8
- Market-Bench logs strategic behaviors in a reproducible simulation where numeric decisions and language jointly determine economic outcomes.The benchmark couples budget-constrained procurement with downstream price competition and semantic visibility.Source: §5 · p. 8
- The synthesis of inventory management with market positioning remains a distinct frontier of difficulty for LLMs.Source: §5 · p. 8
Limitations
The experiments use a single market configuration and a fixed, memoryless buyer model, while several parameter combinations and richer buyer dynamics remain unexplored. The appendix describes the simulation loop, bid settlement, and prompts used for procurement and retail.
Limitations
- The experiments use one configuration: 20 agents, 6 time steps, and 8 item types; the parameter space was not exhaustively explored.Future work could vary conditions such as supply relative to demand.Source: Limitations · p. 9
- Buyers use a synthetically generated persona mix with fixed weights and make single-item purchases without remembering prior interactions.The full buyer-heterogeneity space was not explored.Source: Limitations · p. 9
- Proposed extensions include stochastic supply, cross-market arbitrage, longer episodes, and non-English markets.These directions would extend evaluation beyond the current benchmark scope.Source: Limitations · p. 9
A Simulation Algorithm
- Each episode repeats procurement and retail stages across T steps, then records trajectories and final metrics.The loop also applies holding costs and checks for bankruptcy.Source: A Simulation Algorithm · p. 11
A.1 Bid Settlement Procedure
- Available quantities are allocated greedily to the highest bids above the reserve price, with random tie-breaking.Source: §A.1 · p. 12
B.1 Procurement Stage Prompt
- The procurement prompt requires valid JSON bids that stay within funds and meet or exceed base prices.It also restricts bids to offered items and requires non-negative integer quantities and prices.Source: §B.1 · p. 12
B.2 Retail Stage Prompt
- The retail prompt asks agents to set prices and generate a short slogan based on inferred personas, which are not revealed.Agents are instructed to infer personas from market history; slogans are limited to 25 words.Source: §B.2 · p. 12
C Evaluation Metric Definitions
Market-Bench evaluates agents with nine per-agent metrics across economic, operational, and cognitive dimensions.
C Evaluation Metric Definitions
- Market-Bench computes 9 metrics for each agent.Source: C Evaluation Metric Definitions · p. 13
- The metrics span economic, operational, and cognitive dimensions.Source: C Evaluation Metric Definitions · p. 13
- Table 2 presents results using these metrics.Source: C Evaluation Metric Definitions · p. 13
C.1 Economic Metrics
The economic metrics cover profitability, cumulative profit, risk-adjusted returns, and inventory efficiency.
C.1 Economic Metrics
- Net Profit Margin measures profitability relative to revenue, using per-step profit defined as revenue minus goods costs and holding costs.Source: §C.1 · p. 13
- Cumulative Profit totals an agent’s profit across all steps.Source: §C.1 · p. 13
- Risk-Adjusted Return is a Sharpe-like ratio of mean per-step profit to its volatility.Source: §C.1 · p. 13
- Inventory Efficiency Index measures goods sold relative to total throughput and ranges from 0 to 1, with higher values better.The passage identifies total units sold and average inventory in units.Source: §C.1 · p. 13
C.2 Operational Metrics
The operational metrics characterize inventory fulfillment, procurement efficiency, and order stability, while the semantic metric measures slogan alignment with buyer personas.
C.2 Operational Metrics
- Stockout Rate measures directed purchase attempts that fail because inventory is zero, while Fill Rate measures successfully fulfilled directed demand in units.Source: §C.2 · p. 13
- Bid Efficiency combines procurement success and cost efficiency; its described quantities include won and bid quantities, base value of won goods, and actual spend.Higher is better.Source: §C.2 · p. 13
- Order Stability Index measures how well order variability matches sales variability using coefficient of variation, and ranges from 0 to 1, with higher values better.Source: §C.2 · p. 13
C.3 Cognitive/Semantic Metric
- Mean Match Score is the average cosine similarity between agent slogans and buyer personas; higher is better.Source: §C.3 · p. 13
D Metric Replacement Rationale
The metric replacements address numerical instability in traditional supply-chain measures, and the benchmark also specifies persona-gated buyer attention and default buyer personas.
D Metric Replacement Rationale
- Two traditional supply-chain metrics were replaced with bounded alternatives because of numerical instability.Source: D Metric Replacement Rationale · p. 13
D.1 IEI
- Inventory Turnover Ratio becomes unbounded as average inventory approaches zero, with observed values from 0 to 5.4×10^12; the Inventory Efficiency Index reformulates it.Source: §D.1 · p. 13
D.2 OSI
- The Bullwhip ratio explodes when sales variance is near zero, with observed values from 0.27 to 7.3 × 10^12; the Order Stability Index uses coefficient of variation.Source: §D.2 · p. 14
E Persona-Gated Attention Mechanism
- Persona-gated attention weights buyer consideration of an agent using slogan–persona similarity, slogan sensitivity, semantic temperature, and Qwen3-Embedding-8B embeddings.Source: E Persona-Gated Attention Mechanism · p. 14
F Buyer Persona Configuration
- Table 3 presents the default buyer persona distribution, where λ denotes slogan sensitivity weight.Source: F Buyer Persona Configuration · p. 14, Table 3 · p. 14
G Experimental Configuration
The experimental configuration, supplier offers, and evaluated LLM cohort are documented in Tables 4–6.
G Experimental Configuration
- Table 4 summarizes the key simulation parameters in the default experimental configuration.Source: G Experimental Configuration · p. 14, Table 4 · p. 14
H Supplier Offer Structure
- Table 5 details the supplier offer structure used in the 20-agent experiments.Source: H Supplier Offer Structure · p. 14, Table 5 · p. 14
I Evaluated Models
- The evaluation includes 20 LLM agents spanning open-weight and closed-source models, listed in Table 6.Source: I Evaluated Models · p. 14, Table 6 · p. 14