PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidders

Anand Shah, Kehang Zhu, Yanchen Jiang, Jeffrey G. Wang, Arif K. Dayi, John J. Horton, David C. Parkes

arXiv:2507.09083v2cs.GTcs.AI

TL;DR

The paper asks whether out-of-the-box LLMs preserve human behavior in auctions, where human experimental evidence is costly to extend and published results are not directly comparable across mechanisms. It reconstructs benchmarks and evaluates five models across auction settings; non-reasoning large models preserve key difficulty orderings despite differences in deviation magnitude and direction. The findings support using validated LLM panels to screen mechanism comparisons, not as general substitutes for human participants.

  • Problem

    The paper asks which features of human auction behavior general-purpose LLMs preserve, since human experiments are costly to extend and published results are not comparable across mechanisms.

    Source: §1 · p. 2

  • Method

    The authors reconstruct human benchmarks from published moments and evaluate five out-of-the-box LLMs across auction settings by deviation magnitude, direction, and mechanism ordering.

    Source: §1 · p. 2

  • Results

    The three non-reasoning large models preserve key human difficulty orderings despite differences in deviation magnitude and direction; Kendall’s τb for human and GPT-4o rankings is 0.60.

    Source: §1 · p. 3, §6 · p. 20

  • Takeaways & Limitations

    Where orderings have been validated, an LLM panel may help screen candidate interventions before human experiments establish their effects.

    Source: §6 · p. 21

    Human benchmarks reconstruct published moments rather than use micro data, and the shared induced-value objective leaves variation in bidder preferences and risk attitudes untested.

    Source: §5.3 · p. 20

Abstract

from arXiv · show

Training on vast amounts of human-generated data has motivated growing interest in using large language models (LLMs) to simulate human behavior. We ask which features of human behavior general-purpose models preserve when used out of the box in auctions, where multiple bidders interact under explicit rules and incentives. We evaluate five LLMs across seven laboratory settings against human benchmarks reconstructed from published experiments, with uncertainty bands for the private-value comparisons. Our main focus is on three large models without extended test-time reasoning: GPT-4o, Claude~3.5 Haiku, and Gemini~2.0 Flash. LLM and human deviations from theory differ in magnitude and often in direction: humans overbid in second-price auctions, whereas most models that deviate underbid. Surprisingly, without task-specific fine-tuning or calibration to human bids, the three non-reasoning large models robustly preserve key orderings of auction formats by deviation from theory. First-price auctions are harder than second-price, and ascending clocks reduce deviations relative to sealed bids wherever data are adequate. Kendall's $\tau_b$ between the human and GPT-4o difficulty rankings is $0.60$ and positive in every joint bootstrap draw. The reasoning model bids almost at equilibrium in the observed private-value settings, leaving little variation in errors to compare; the small model's large errors yield an inverted ranking. All five models nevertheless reproduce the stronger first-price winner's curse. Clock framing improves bidding for two of the three non-reasoning large models, and GPT-4o recovers the ordering of last-minute bidding across closing rules in an eBay-style marketplace.

1 Introduction

The paper tests whether out-of-the-box LLMs preserve human auction behavior, distinguishing error magnitude, direction, and mechanism orderings. It finds that non-reasoning large models preserve important orderings despite differences from humans in magnitude and often direction.

1 Introduction

  • The benchmark addresses whether simulated bidders can inform mechanism designers, who need to know how bidders play formats rather than only what theory prescribes.Human laboratory evidence is costly to extend to new designs, motivating the paper’s practical question.Source: §1 · p. 2
  • The framework tests magnitudes, directions, and mechanism orderings, with orderings and comparative-static signs as the most directly useful features for comparing candidate mechanisms.The paper provides a cross-mechanism benchmark from reconstructed human moments and applies it to five models across seven formats.Source: §1 · pp. 2–3
  • The study evaluates five models across seven auction settings against human benchmarks reconstructed from published experiments, using a fixed prompt and no task-specific fine-tuning or calibration.The simulation varies auction mechanism, information structure, and model.Source: §1 · pp. 2–3
  • Kendall’s τb between human and GPT-4o difficulty rankings is 0.60, and first-price auctions are harder than second-price for the three non-reasoning large models, as for humans.Ascending clocks reduce deviations relative to sealed bids wherever data are adequate; the ranking is uninformative for GPT-5-mini and inverted for Llama-3-8B.Source: §1 · p. 3
  • Human and LLM deviations often differ in magnitude and direction: humans overbid in second-price auctions, whereas most models that deviate underbid.GPT-5-mini bids almost at equilibrium in observed private-value settings, while Llama-3-8B makes large errors.Source: §1 · p. 3
  • Clock framing improves bidding for GPT-4o and Claude 3.5 Haiku, while GPT-4o places 63% of decisive bids in the last period under hard close versus 7% under soft close.The closing-rule ordering matches patterns observed on eBay and early Amazon.Source: §1 · p. 3

2 Methods

The methods reconstruct comparable human benchmarks from published moments, define a common deviation measure, and simulate bids across auction settings. They compare models with human behavior by magnitude, direction, and mechanism ordering.

2.1 Human Experiment Data Collection

  • Human benchmarks are reconstructed by fitting a mixture model to published aggregate moments, simulating bids, and computing the same deviation measure used for LLM bids.The reconstructions are not micro data; their uncertainty is carried forward, and comparisons are limited to orderings, directions, and comparative statics rather than levels or hypothesis tests.Source: §2.1 · pp. 4–5

2.2 Unified Deviation Metric across Auctions

  • The scaled mean absolute deviation compares every observed bid with its theoretical benchmark and normalizes absolute error by the expected benchmark bid.The benchmark is determined by each environment’s information and mechanism; the metric is interpretable as a percentage of the expected benchmark bid.Source: §2.2 · p. 6

2.3 LLM Data Generation

  • Each setting varies the value model, auction mechanism, prompt framing, or LLM, while bidders receive assigned values or signals and the same expected-profit objective.Baseline repetitions are independent single-auction instances without feedback across instances.Source: §2.3 · pp. 7–8

2.3.1 Canonical laboratory mechanisms.

  • The laboratory mechanisms include sealed-bid auctions and ascending clocks; a second-price clock framing changes prompt language while preserving the sealed-bid allocation and payment rule.The eBay-style proxy-bidding environment is separate from these canonical laboratory mechanisms.Source: §2.3.1 · pp. 8–9

2.3.2 Simulation protocol.

  • Each repetition draws values or signals, prompts bidders with rules and private information, parses their actions, and computes auction outcomes and profits.Actions are numeric bids in sealed-bid formats, stay-or-exit choices in clocks, or increase-or-hold choices in the eBay environment.Source: §2.3.2 · p. 9

2.4 Three Tiers of Comparison

  • The comparison separates deviation magnitudes, directions, and format orderings, with ordinal agreement as the tier the authors claim can inform design comparisons.Magnitude comparisons are qualitative because human anchors are reconstructed; ordering comparisons use Kendall’s τb and signs of design contrasts.Source: §2.4 · p. 10

2.5 Model Panel

  • The model panel comprises three large non-reasoning models, one reasoning model, and one small open-weights model, evaluated in separate auctions rather than mixed within auctions.GPT-4o is also the reference model for deeper distributional diagnostics and has the largest cells.Source: §2.5 · p. 10

3 Benchmark Experimental Auction Data

The section compares human reconstructed SMAD with five models across seven formats, assessing deviations tierwise by magnitude, direction, and ordering.

3.1 Tier 1: Magnitudes do not match

  • GPT-4o SMAD exceeded humans in sealed bids (27.6% vs. 24.8% first-price; 8.7% vs. 5.6% second-price) but was lower in both clocks.The comparison uses reconstructed human anchors and their uncertainty bands; the other models also often fell outside those bands.Source: Figure 1 · p. 11, §3.1 · p. 12, Table 1 · p. 12

3.2 Tier 2: Directions do not match

  • In first-price IPV, humans and non-reasoning large models predominantly overbid, but in second-price IPV humans predominantly overbid while most models predominantly underbid.Claude 3.5 Haiku is the exception among the large models in second-price IPV, predominantly overbidding like humans.Source: §3.2 · pp. 12–13, Figure 2 · p. 14

3.3 Tier 3: Orderings match for the non-reasoning large models

  • The non-reasoning large models recover the human difficulty ranking; GPT-4o’s Kendall’s τb is 0.60 and remains positive in every joint bootstrap draw.GPT-5-mini’s near-equilibrium private-value bids leave little variation to rank, while Llama-3-8B inverts the human ordering.Source: §3.3 · p. 13, Table 2 · p. 13

3.3.1 Payment rule: first-price is harder than second-price

  • First-price SMAD exceeds second-price SMAD for humans and all three non-reasoning large models, while GPT-5-mini is near equilibrium and Llama-3-8B reverses the ordering.The human comparison is 24.8% vs. 5.6%; model comparisons are GPT-4o 27.6% vs. 8.7%, Claude 24.1% vs. 9.7%, and Gemini 27.1% vs. 3.0%.Source: §3.3.1 · p. 15

3.3.2 Implementation: clock beats sealed bid

  • Where clock data are adequate, the non-reasoning large models reproduce humans’ lower deviations in ascending clocks than in sealed bids.The human open-clock ordering holds in all reconstruction re-draws, while the closed-clock ordering is less secure; several model clock cells have too few bids to inform comparison.Source: §3.3.2 · pp. 15–16

3.3.3 Information structure: common value and competition

  • All model classes and humans show larger deviations in first-price than second-price common-value auctions, preserving the stronger first-price winner’s curse.The comparison is within the common-value setting because its benchmark is profit-based.Source: §3.3.3 · p. 16

4 Application to Design: Interventions and Market Structure

The section tests whether design changes and market-structure orderings observed with human bidders also appear in LLM bidding. Clock framing improves bidding for two model families, while soft-close rules shift decisive bids earlier.

4.1 Clock framing as an intervention: a prediction test

  • The clock-framing test changes the prompt while keeping the allocation and payment rule unchanged.Bidding is summarized by per-bid scaled deviation, where negative values indicate underbids and positive values indicate overbids.Source: §4.1 · p. 17
  • Clock framing lowers SMAD for Claude 3.5 Haiku (8.1 → 7.1%) and GPT-4o (11.8 → 6.2%), but raises it for Gemini 2.0 Flash (5.0 → 9.3%).The human-aligned effect is borne out for models with non-degenerate baseline error, but not for Gemini, whose baseline play is already closest to truthful.Source: Table 4 · p. 17, §4.1 · p. 17

4.2 Market structure: closing rules in an eBay-style marketplace

  • The eBay-style comparison isolates the closing rule in a proxy-bidding environment and measures when the eventual winner last raises her maximum bid.The two treatments are a standard hard close and a soft-close extension rule.Source: §4.2 · p. 18
  • Under hard close, 63% of auctions end with a decisive bid in the last scheduled period; under soft close, 7% do, and the median final-winning-bid period falls to 5.5.The soft-close distribution shifts earlier, matching the qualitative ordering in field evidence; the study does not claim the timing magnitude matches human bidding.Source: §4.2 · pp. 18–19, Figure 3 · p. 19

5 Discussion

The discussion positions LLM bidders as tools for ordinal screening, not substitutes for human experiments that establish magnitudes and directions. It also specifies the evidence and behavioral variation that bound this use.

5.1 What LLM simulation can and cannot deliver for mechanism design

  • For non-reasoning large models, key design comparisons match human orderings, but bid magnitudes and often directions do not, limiting support for predictions such as revenue or efficiency loss.The proposed workflow uses a model panel to rank candidate designs or estimate the sign of a change, then tests surviving designs with human experiments.Source: §5.1 · pp. 19–20
  • The paper uses queried reasoning traces only as diagnostics and relies on them for no claim.Source: §5.1 · p. 20

5.2 Practical advantages and the scope for mechanism design

  • LLM experiments cost under $400 for the reported GPT-4o experiments versus roughly $15,000 for a comparable human study, but simulated effects require separate validation with humans.The authors present the cost difference as an advantage for exploration, not evidence that simulations replace human experiments.Source: §5.2 · p. 20

5.3 Limitations

  • Human benchmarks reconstruct published moments rather than use micro data, so their uncertainty bands are not sampling bands and support only orderings, directions, and comparative statics.Simulated bidders vary only in induced values, leaving behavior with variation in risk attitudes or other dimensions open.Source: §5.3 · p. 20

6 Conclusion

Across the tested auctions, LLMs differ from humans in bid magnitude and often direction, yet three non-reasoning large models preserve key orderings of auction difficulty. The conclusion frames validated orderings as a possible screening aid, with human experiments still needed to establish effects.

6 Conclusion

  • The three non-reasoning large models preserve key orderings of auction difficulty without task-specific fine-tuning or calibration to human bids.The paper reports more than 1,000 LLM auction experiments across five models, benchmarked against reconstructed human laboratory evidence.Source: §6 · p. 20
  • Near-equilibrium private-value bids leave little variation for comparing human difficulty, while the small model’s large errors yield an inverted ranking; both nevertheless preserve the stronger first-price winner’s curse.Source: §6 · p. 21
  • Validated LLM orderings may help screen interventions before human experiments establish their effects, but transfer to voting, matching, or contract design remains empirical.Source: §6 · p. 21

A Details about Human Data Collection

The section reconstructs comparable human bidding deviations from published aggregate statistics, adapting the fitted data-generating process to each study’s reported measures and auction setting.

A.1 Mixture Model Fitting Details

  • Published studies’ reported deviations are mapped to SMAD using fitted bid distributions, with Table 5 listing mixture parameters by source and treatment.The reconstruction preserves each source’s unit of observation and uses Monte Carlo variation for point estimates and uncertainty.Source: A Details about Human Data Collection · pp. 25–26, Table 5 · p. 26

A Details about Human Data Collection

  • For Kagel–Levin’s IPV auctions, regressions are fit separately by bidder count, and bid distributions are reconstructed from reported moments.The procedure assumes uniform values and fits auxiliary parameters to overbidding frequencies.Source: A Details about Human Data Collection · p. 25
  • Clock-auction MAD uses only bidders who exit because the winner’s terminal willingness-to-pay is unobserved.For repeated auctions, the reconstruction follows each source’s convention for selecting post-learning blocks.Source: A Details about Human Data Collection · p. 25
  • For common-value auctions, SMAD compares bids with the equilibrium rule implied by the signal structure and mechanism, using a normalization factor for the large shared valuation component.The equilibrium strategy is applied to bids as in private-value settings.Source: A Details about Human Data Collection · p. 26

A.2 Reconstruction-Uncertainty Bands

The section quantifies uncertainty in reconstructed human benchmarks and describes the prompting and simulation procedures for laboratory and eBay-style auctions.

A.2 Reconstruction-Uncertainty Bands

  • A parametric bootstrap draws 2,000 vectors of fitted mixture parameters from ranges tied to reported statistics, propagates them through each reconstruction, and reports percentile bands.Admissible ranges reflect sampling uncertainty or reporting granularity; some parameters are varied through regression slopes, R2, or published standard errors.Source: §A.2 · p. 27
  • The first-price versus second-price difficulty gap holds in 100% of reconstruction draws, while clock improvements hold in 100% for open clocks and 92.6% for closed clocks.The first-price/second-price SMAD ratio has a 95% band of [3.54, 5.65], centered on a 4.4× point gap; reconstruction bands support orderings, not level comparisons or tests against LLM cells.Source: §A.2 · p. 27, Table 6 · p. 27

B LLM Simulation Procedure

  • Each simulated bidder receives a mechanism scenario, provides a short bidding plan, and returns a structured bid or decision that can be parsed deterministically.The prompts query bidders independently.Source: B LLM Simulation Procedure · pp. 27–28

B.1 Simulation process in sealed-bid auctions

  • Sealed-bid prompts specify the bidder role, auction rules, profit objective, and structured plan/action response; repeated-round prompts additionally provide prior outcomes.The main experiments use single-round auctions, while the multi-round ablation includes a standardized history transcript.Source: §B.1 · p. 28

B.2 Simulation process in Ascending Clock auctions

  • Ascending-clock agents decide whether to stay or exit at each increasing price; open-clock prompts show prior dropouts, unlike closed-clock prompts.Rounds end when one bidder remains or two bidders drop simultaneously, triggering random winner selection.Source: §B.2 · p. 29

B.3 Stylized eBay proxy-bidding environment

  • The stylized eBay environment uses proxy bidding and compares a hard close with an extension rule, abstracting away features such as shipping and Buy-It-Now options.A new leader in the final period triggers an extra period under the extension rule, repeated until a final period has no new bids.Source: §B.3 · pp. 29–30

B.4 Simulation process in eBay auctions

  • eBay bidding is simulated in discrete periods, with agents deciding whether to raise or hold after observing the auction state and prior bidding history.The final period is simultaneous, so agents cannot observe others’ final-period actions before deciding.Source: §B.3 · p. 30, §B.4 · p. 30

C Examining Learning Effects in Repeated Games

The repeated-game analysis finds no detectable drift in LLM bidding deviations, while GPT-4o’s mechanism difficulty ordering remains robust across temperature settings. The closest-format ordering is an exception.

C Examining Learning Effects in Repeated Games

  • FPSB mean SMAD is 0.2523 in the first five rounds and 0.2524 in the last five, with no detectable learning or systematic drift across 15 rounds.SPSB likewise shows no significant early-to-late difference or temporal trend; Figure 4 plots SMAD across rounds.Source: C Examining Learning Effects in… · p. 31, Figure 4 · p. 32

D.1 Varying Temperature for GPT-4o Agent

  • At every temperature, GPT-4o ranks first-price as harder than second-price and has positive Kendall’s τb against the human ranking in 100% of joint bootstrap draws.Across temperatures, SMAD-vector rank correlations are at least +0.90; the SP-APV versus SPSB IPV order matches humans only at T = 0.5, and clock cells at T = 0.1 and 1.0 have few usable bids.Source: §D.1 · p. 32, Figure 5 · p. 33

D.2 Varying the Base Models for LLM bidding

The cross-model comparison finds that model capability tier shapes how well auction-format difficulty orderings match human benchmarks, while sparse clock bids constrain some comparisons.

D.2 Varying the Base Models for LLM bidding

  • Clock comparisons are sparse: only Claude in the open clock and Gemini in the closed clock have more than a handful of usable non-winning bids; remaining cells have n ≤ 8.GPT-5-mini states the dominant strategy instead of completing the clock procedure, so its clock cells enter no statistic.Source: §D.2 · p. 33
  • GPT-5-mini’s private-value sealed-bid SMAD is below 1.4%, while Llama-3-8B exceeds 38% in every private-value format and ranks formats opposite to humans.Claude 3.5 Haiku and Gemini 2.0 Flash reproduce the human ordering; all five models show stronger first-price than second-price deviations in common-value formats.Source: §D.2 · pp. 33–34, Figure 6 · p. 35
  • Deviating-bid directions are reported as underbid/overbid shares outside the ±2% benchmark band, alongside within-band shares and sample sizes.Table 8 marks cells with fewer than 30 usable bids and identifies GPT-5-mini’s clock cells as having no recorded bids.Source: Table 8 · p. 35

E Rank Concordance Details

The non-reasoning large models’ difficulty rankings broadly track reconstructed human rankings, while reasoning and small models do not. Concordance estimates vary in precision and depend on which formats are included.

E Rank Concordance Details

  • GPT-4o’s five-format Kendall’s τb is 0.60, positive in 100% of joint draws, with first-price ranked hardest in every draw.The estimate reflects 8 concordant and 2 discordant pairs; both discordances involve AC-B.Source: E Rank Concordance Details · p. 36
  • Claude 3.5 Haiku has one sealed-format discordance, a near-tie that flips in about 45% of resampling draws.Its interval reaches 1.00.Source: E Rank Concordance Details · p. 36
  • GPT-5-mini’s negative point estimate reflects small differences between near-equilibrium bids, while Llama-3-8B’s ordering is inverted in all but one joint draw.Two GPT-5-mini cells are at 0.08% and 0.20%.Source: E Rank Concordance Details · p. 36
  • Three-format intervals are driven by LLM resampling, and the human-side uncertainty represents reconstruction uncertainty rather than sampling error.LLM bids are treated as i.i.d. within cells; common-value and third-price formats are excluded from the ranking.Source: E Rank Concordance Details · p. 36

F Winner’s Curse: Details by Auction Format and Number of Bidders

LLM bidders exhibit a winner’s curse in common-value first-price auctions, with losses worsening as bidder numbers rise. The curse is weaker in second-price auctions, where losses emerge only under stronger competition.

F Winner’s Curse: Details by Auction Format and Number of Bidders

  • Winning is adverse selection because the highest signal implies a lower conditional common value; neglecting that correction can produce overbids and negative winner profits.Laboratory evidence also finds larger winner losses with more bidders and attenuation when price discovery reveals more information.Source: F Winner’s Curse: Details by Auction… · p. 36
  • In SPSB, winners average 1.30 profit with 4 bidders, approach zero with 5, and incur statistically significant losses only at n ≥6.The winner’s curse is substantially stronger in FPSB than SPSB, matching the qualitative experimental ranking.Source: F Winner’s Curse: Details by Auction… · p. 37
  • Mean winner profit in FPSB falls from −2.54 with 4 bidders to −5.55 with 7, and positive-profit winners fall from 24% to 0%.Winner profits are negative in every FPSB setting; one-sided tests reject nonnegative winner profit in all treatments.Source: Table 9 · p. 37, F Winner’s Curse: Details by Auction… · p. 37

G Robustness of the Clock-Framing Effect: An Additional Model Family

A robustness test extends the clock-framing intervention to Gemma-3-27B, whose baseline second-price bidding leaves room for improvement. Clock framing improves its bidding, consistent with the pattern across model families.

G Robustness of the Clock-Framing Effect: An Additional Model Family

  • Gemma-3-27B is outside the main panel, and its baseline pools two independently run instances of the standard second-price description.The intervention battery used identical rule templates.Source: G Robustness of the Clock-Framing… · p. 37
  • The earlier GPT-4o run differed from the reported templates in its bid bounds and written-plan limit.That earlier prompt capped bids at the bidder’s value and plans at 50 words; the current templates specify $0.1 increments and a 100-word plan limit.Source: G Robustness of the Clock-Framing… · p. 37
  • Clock framing lowers Gemma-3-27B’s SMAD from 26.3% to 13.5% and raises bids within ±2% of value from 6% to 22%.The SMAD change has Mann–Whitney p < 0.001; across four families, framing helps three and fails only for the family closest to truthful baseline play.Source: G Robustness of the Clock-Framing… · p. 37, Table 10 · p. 38, Figure 7 · p. 38

H Additional Results for the eBay-Style Environment

The appendix reports additional eBay-style timing results and documents prompts for the auction settings and intervention. In the eBay-style environment, the soft-close rule shifts final winning bids earlier.

H Additional Results for the eBay-Style Environment

  • The soft-close extension shifts final winning bids substantially earlier than the standard hard close in the eBay-style IPV environment.Figure 8 compares the cumulative distribution of final winning-bid periods under T1 and T2.Source: Figure 8 · p. 39

I Auction Rule Prompts

  • The auction-rule prompt section holds the prompt structure fixed while changing the allocation and payment description across auction formats.The prompts are introduced as following the simulation procedure described in Section B.1.Source: §I.1 · p. 39

I.1 Independent Private Values Prompts

  • The independent-private-value prompts distinguish formats by payment rules, including second-highest payment for SPSB and third-highest payment for TPSB.The FPSB description is also presented in this prompt section.Source: §I.1 · pp. 39–40

I.2 Obviously Strategy-Proof Prompts

  • The strategy-proof-format prompts describe an ascending clock, while AC-B explicitly withholds notice when competitors drop out.The clock begins at 0 and increases in $1 increments.Source: §I.2 · p. 40

I.3 Common Value Auctions

  • Common-value prompts describe correlated bidder values and also include a noisy-perception version in which bidders’ true value is shared.The ascending-clock prompt broadcasts bidder exits as the price rises.Source: §I.3 · pp. 41–42

I.4 Ebay-proxy prompt

  • The eBay proxy prompt has bidders submit maximum bids, with proxy bidding maintaining their position as high bidder only as needed.Under the modified closing rule, a new maximum bid on the final day extends the auction by another day.Source: §I.4 · pp. 42–43

I.5 Intervention Prompt

  • The clock-framing intervention makes the sealed bid an automatic exit price in a subsequent simulated ascending clock.The clock rises in increments, and the auction ends when only one bidder remains.Source: §I.5 · p. 43

Related papers

Loading 2507.09083v2…