PaperlayerView exact revision on arXiv

Source-linked AI summary

PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents

Pavel Kireyev

arXiv:2609.31468v1econ.GNcs.AIcs.CLq-fin.EC

TL;DR

Existing agent benchmarks measure task completion rather than which qualifying option the agent chooses, although that choice reveals its purchasing policy. PriceBench applies a logit choice model to hotel bookings from 28 LLMs. Capability tracks choice consistency rather than preference content, and price–quality preferences vary across models, shifting mean booked nightly price from $247 to $393 on identical tasks.

  • Problem

    Existing agent benchmarks measure whether an agent completes a task, not which admissible option it chooses or the purchasing policy revealed by that choice.

    Source: §1 · p. 1

  • Method

    PriceBench fits a logit choice model to booking choices across 3,600 tasks drawn from 179 real NYC hotel properties to recover LLM price, quality, and brand preferences.

    Source: §1 · p. 1

  • Results

    Capability is associated with choice consistency rather than preference content, and willingness to pay for a one-point review-score gain spans 16×, shifting mean booked nightly price from $247 to $393 on identical tasks.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    What an agent buys is specific to the individual LLM and should be measured rather than assumed; PriceBench releases the tasks, code, and response sets for such audits.

    Source: §1 · p. 2

    The benchmark’s tasks involve NYC hotel rooms, and its methodology is validated only in that domain; headline estimates are conditional on a neutral booking prompt at temperature zero.

    Source: Limitations · p. 8

Abstract

from arXiv · show

LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferences quietly fix what gets bought and what it costs. Hotel booking is a clean instance: a high-volume choice settled on a few comparable attributes, where the pick reveals those preferences. We introduce PriceBench, a diagnostic benchmark that recovers an LLM's price, quality, and brand preferences from its booking choices with a logit choice model, applied to 28 LLMs from 8 providers on 3,600 hotel tasks from 179 real New York City properties. We find that capability is associated with how consistently an LLM chooses, not with what it chooses: more capable LLMs hold stronger, more consistent preferences, while weaker ones either lock onto one position, exploitable by whoever controls listing order, or choose almost indifferently. What those preferences favor varies sharply across providers and even within one family: price sensitivity spans more than an order of magnitude, and the price/quality trade-off moves mean booked nightly price from \$247 to \$393 on identical tasks. What an agent buys must therefore be measured per LLM, not inferred, and we release the tasks, code, and all 28 response sets.

1 Introduction

PriceBench addresses the gap between measuring whether a booking agent completes a task and measuring which qualifying option it chooses. Across 28 LLMs, capability is associated with choice consistency rather than preference content, while price–quality preferences vary substantially across models.

1 Introduction

  • Existing agent benchmarks measure task completion but not which admissible option an agent chooses, leaving its revealed purchasing policy unmeasured.In commerce requests with guardrails such as a budget cap, minimum star rating, or preferred neighborhood, several listings can satisfy the constraints.Source: §1 · p. 1
  • PriceBench presents 3,600 hotel choice tasks from 179 real New York City properties and fits a logit choice model to recover price, quality, and brand preferences.Each task is scored in its original ordering and with options swapped; the model includes an intercept for constant first-slot preference.Source: §1 · p. 1
  • Capability is associated with choice consistency, not preference content: stronger models hold consistent preferences, while weaker models may fix on one position or choose nearly indifferently.Among the 28 models, five are position-locked at temperature zero; among the other 23, willingness to pay for a one-point review-score gain spans 16× and mean booked nightly prices range from $247 to $393 on identical tasks.Source: §1 · pp. 1–2
  • Figure 1 illustrates the split in revealed choices: several models select option A in Task 113, while Gemma3 4B, DeepSeek-R1 7B, and Phi-4 Mini select B.The figure caption identifies GPT-5.4, GPT-5.4 Mini, GPT-4.1 Mini, Claude Haiku 4.5, and Gemma3 27B as choosing A.Source: Figure 1 · p. 2
  • The preferences vary sharply among models: willingness to pay for a one-point review-score gain spans 16×, shifting mean booked nightly price from $247 to $393 on identical tasks.The direction of the price-versus-quality lean is not associated with scale and can differ even among models in the same family.Source: §1 · p. 2
  • The authors release the tasks, estimation code, all 28 response sets, and runners for prompt variants, decoding, and five-option tests.They conclude that an LLM’s purchases should be measured for that individual model rather than assumed.Source: §1 · p. 2

2 Related Work

Prior agent benchmarks emphasize task success, while research on LLM choice has estimated selected economic preferences without treating the LLM’s own purchasing policy as the object of measurement. PriceBench focuses on choices among qualifying options and uses them to recover multiple preferences.

2 Related Work

  • Existing agent benchmarks test navigation, constraint satisfaction, or support success, but do not isolate which qualifying option an agent actually prefers.Prior work has also documented ordering sensitivity, which PriceBench treats as a prerequisite for identifying preferences.Source: §2 · p. 2
  • PriceBench measures the choice itself, complementing success-oriented benchmarks and extending prior LLM-choice work to price, functional form, dollar willingness to pay, and brand preference.The related-work discussion frames the question of whether preferences can be predicted from the LLM’s provider as part of this analysis.Source: §2 · p. 2
  • Discrete-choice research recovers price sensitivity and willingness to pay, while LLM studies have examined biases, dispositions, discount rates, and conjoint responses.Those studies generally treat LLMs as stand-ins for human respondents; PriceBench instead studies the LLM’s own booking policy and reads brand preference from binding choices.Source: §2 · p. 2

3 Benchmark Design

The benchmark uses a frozen pool of real NYC hotel profiles and repeated, randomized presentations to create comparable choices. Binary tasks provide the main analysis, with ternary tasks serving as a format robustness check.

3 Benchmark Design

  • The pool contains 179 real NYC hotel profiles spanning 1–5 stars, 36 neighborhoods, and $45–$1,650 per night, with ten fixed or re-randomized attributes.Four attributes remain fixed; six, including price, room details, and review measures, are re-randomized across appearances.Source: §3 · p. 3
  • The benchmark samples 450 unique pairs presented four times for 1,800 binary tasks and 300 unique triples presented six times for 1,800 ternary tasks.Each task is scored in both option orders, yielding up to 7,200 observations per LLM; pooled estimation cancels pure position bias.Source: §3 · p. 3
  • The study scores 28 LLMs from eight providers, spanning sub-billion-parameter open-weight models to frontier proprietary models, with temperature-zero greedy decoding.Prompts present options as Booking.com-style cards with all ten attributes labeled.Source: §3 · p. 3

4 Estimation

PriceBench estimates attribute preferences from differences between options in a logit choice model, separating constant first-slot tendency from attribute weights. It evaluates alternative price functions and estimates brand premiums while controlling for other attributes.

4 Estimation

  • For binary tasks, the logit model regresses choices on attribute differences, with an intercept absorbing constant first-listed-option tendency; ternary tasks use multinomial logit.Scoring each task in both option orders lets a pure position effect load on the intercept without contaminating attribute weights.Source: §4 · p. 3
  • The price term is tested in linear, logarithmic, and non-parametric decile forms, which encode different patterns of price sensitivity.The non-parametric form leaves the price-response shape free to expose thresholds and price-as-quality effects; pooled price-decile cut-points are reused across LLMs.Source: §4 · p. 3
  • The brand audit adds chain-family indicators to price-decile, star, room-type, cancellation, breakfast, and review controls, estimating each chain’s premium over an otherwise comparable independent hotel.Source: §4 · p. 3

5 Results

Engagement is a gateway to interpretable choices: capability is associated with decisiveness, not preference direction. Among engaged LLMs, price, quality, and brand preferences vary widely, with measurable differences in what gets booked.

5 Results

  • Capability is associated with decisiveness, not policy direction; price–quality leanings and brand preferences vary across LLMs and must be measured individually.Source: §5 · pp. 3–4

5.1 Position engagement is the gateway test

  • Five LLMs are position-locked at temperature zero, picking the first option regardless of attributes; listing-order control therefore determines 88–100% of their bookings.Source: §5.1 · p. 4

5.2 Price sensitivity and functional form

  • βln p ranges from −7.02 for GPT-5.4 to −0.40 for Qwen3 30B-A3B; AIC selects linear for 10 and log for 10 of 21 negative-response LLMs, with one tie.Price-response shapes include linear-like declines, convex and threshold patterns, and an early positive bump; GPT-5.4 Nano reaches +0.62 at decile 2.Source: §5.2 · p. 4, Figure 2 · p. 5

5.3 The price–quality map

  • Willingness to pay spans roughly 16×, from $68 to $1,071, and mean nightly bookings range from $247 to $393 on identical tasks; willingness to pay is uncorrelated with decisiveness.The price–quality ratio removes overall coefficient scale; dollar valuations are conditional on elicitation format, which moves magnitudes while preserving ordering.Source: §5.3 · p. 6

5.4 Brand preferences across LLMs

  • Mean brand-coefficient correlation is +0.079 within providers versus +0.090 across providers (permutation p = 0.48); neighborhood controls reduce all-five-chain preferences from 11 to 7 of 23.Source: §5.4 · p. 6

5.5 LLM valuations against the human record

  • 20 LLMs value a review point at $68–$1,071 (median $269), versus human estimates of roughly $2–$22; this bounds plausibility, not correctness.Source: §5.5 · pp. 6–7

6 Robustness

Robustness tests find that price–quality preference direction and cross-model ordering generally persist across prompt formats, decoding choices, and larger choice sets. Bootstrap and external-capability checks also support the reported patterns.

Prompt format and reasoning

  • Across prompt variants, no LLM’s price–quality lean flips when both coefficients remain estimable, and cross-model ordering stays at least Spearman +0.60; formats shift dollar magnitudes systematically.The persona raises the quality lean for all six tested LLMs, while chain-of-thought lowers it for all five that ran it.Source: §6 · p. 7

Decoding

  • Sampling at temperature 0.7 leaves four of five locked LLMs choosing the first option 89.2–99.3% of the time; the fifth has almost no attribute signal.No locked LLM yields usable preferences under sampling.Source: §6 · p. 7

Larger choice sets

  • Binary-fitted utilities predict five-option choices out of sample at 42–52% hit rates versus 20% chance, while binary and ternary price rankings correlate at Spearman ρ = 0.88–0.90.Position habits also persist in longer lists for Mistral-Nemo 12B and locked Llama 3.1 8B.Source: §6 · p. 7

Statistical stability

  • Cluster-bootstrap mean rank correlations are 0.996 for price sensitivity and 0.991 for willingness to pay; every coefficient claimed at p < 0.001 survives both multiple-testing corrections.Source: §6 · p. 7

External capability axis

  • Across 18 covered engaged LLMs, the public capability index correlates with decisiveness (r = +0.63) and price-coefficient magnitude (r = +0.71), but not willingness to pay (all p > 0.09).Source: §6 · p. 7

7 Conclusion

PriceBench finds that capability tracks decisiveness rather than preference direction, while price–quality balance and brand preferences vary across LLMs. The conclusion is that an agent’s purchasing behavior should be measured for each LLM.

7 Conclusion

  • Across 28 LLMs, capability tracks decisiveness rather than preferences; first-slot locks affect 5/28 (88–100%), while price–quality balance shifts mean booked price by $146 and brand varies (p = 0.48).Source: §7 · p. 7

Limitations

PriceBench’s findings are bounded by its NYC hotel setting, neutral-prompt estimates, and cross-sectional comparisons. Its prompt uses prose cards with equally formatted attributes to reflect production-style booking prompts and avoid formatting-driven attribute weighting.

Limitations

  • Validation covers NYC hotels only; applying the method elsewhere requires a new profile pool, and price sensitivity may differ across product categories.The estimator also assumes attribute variation comparable to that naturally present in hotels.Source: Limitations · p. 8
  • Headline estimates use a neutral booking prompt at temperature zero; robustness checks cover six of 28 LLMs, dollar valuations depend on format, and tool-augmented multi-turn agents remain untested.The tested variants include prompt formats, sampled decoding, and five-option lists.Source: Limitations · p. 8
  • The human reference range draws on studies with different constructs, and PriceBench has no human choices on its tasks, though its released tasks support that comparison.Source: Limitations · p. 8
  • Capability findings are cross-sectional associations across 28 LLMs, while undisclosed provider details and untested within-model variation limit analysis of potential explanations.Training data, alignment, and architecture could jointly determine capability and choice consistency; RLHF stage and context window are not consistently disclosed.Source: Limitations · p. 8
  • Chain mapping groups distinct hotel tiers under one brand dummy; sub-brand or hotel fixed-effects analyses remain possible, while a luxury-tier indicator barely changes chain premia.For example, Marriott combines Ritz-Carlton and Fairfield Inn.Source: Limitations · p. 8

A Prompt template

  • The prompt uses plain-text cards rather than JSON because prose matches production booking prompts and equal attribute formatting avoids imposing weights through markup.The system prompt adds no persona, role-play, or chain-of-thought scaffold.Source: A Prompt template · p. 10

B Triage: per-LLM engagement rates

The appendix details how PriceBench distinguishes position-following from engaged choices, then tests the recovered price, quality, and brand preferences across specifications and capability measures. It finds robust but heterogeneous preferences, with capability associated with decisiveness rather than the direction of the price–quality trade-off.

B Triage: per-LLM engagement rates

  • Five open-weight LLMs are position-locked: their binary first-shown rates are 92.6–100.0%, and their valid outputs follow the displayed slot rather than hotel attributes.The locked models are small members of their family ladders; excluding them from preference estimation is conservative because their rare deviations provide little attribute signal.Source: B Triage: per-LLM engagement rates · pp. 10–11
  • Intermediate first-shown biases also matter: Qwen3 models cluster near 20%, while larger Gemma3 models and Llama 3.2 3B sit at 56–66%.These biases do not invalidate preference estimation because the slot intercept absorbs them, but listing order still shifts choices.Source: B Triage: per-LLM engagement rates · p. 10

C.1 Non-parametric decile curves

  • Across 23 engaged LLMs, decile-based price curves reveal nonlinear shapes, while Gemma3 1B and Phi-2 2.7B show nearly flat responses with pseudo-R2 values of 0.03 and 0.01.For steep responders, decile utilities exceed both parametric curves at cheaper prices and fall only at the priciest decile.Source: §C.1 · p. 11

C.3 Capability scaling

  • Among 15 open-weight engaged LLMs with significant price effects, parameter count correlates with price sensitivity (Pearson r = 0.62), but family exceptions rule out a universal size rule.On the public capability index, decisiveness and price-coefficient magnitude correlate positively, whereas willingness to pay is uncorrelated with the index across tested subsets.Source: §C.3 · pp. 11–12

D Quality coefficients per engaged LLM

  • Within an LLM, relative quality coefficients show which dimensions matter most; across LLMs, column z-scores separate relative preferences from overall coefficient scale.Review score and star rating dominate for the sharpest price-sensitive LLMs, while cancellation or breakfast can rival them for several middle-tier LLMs.Source: D Quality coefficients per engaged LLM · pp. 12, 14

C.2 Binary vs. ternary robustness

  • Binary and ternary price estimates have rank correlations of ρ = 0.88 for log-price and ρ = 0.90 for linear-price, with most ranking movement at the extremes.The middle of the ranking is most stable, and the price–quality map is reproduced under either task block.Source: §C.2 · p. 11, Figure 6 · p. 13

D Quality coefficients per engaged LLM

  • Willingness-to-pay rankings interleave providers and separate same-family models; the log-price spread is 15.6×, while the linear-price specification gives 31.4×.Rankings correlate at 0.974 across specifications, though the exact spread depends on specification.Source: D Quality coefficients per engaged LLM · p. 14

E Brand coefficients per engaged LLM

  • The provider-null result survives every tested specification; dropping each of the five Wyndham properties in turn keeps GPT-5.4’s Wyndham coefficient within [+2.07, +2.38], so the premium is family-wide.Source: E Brand coefficients per engaged LLM · p. 14

F Prompt-format ablation

Prompt-format and decoding checks find that most cross-model preference rankings persist, although prompt variants can shift price–quality lean and measured decisiveness. The appendix also reports brand-coefficient patterns and tests preferences on five-option tasks.

F Prompt-format ablation

  • The four prompt variants are JSON record cards, price-first reordered attributes, a value-conscious-client persona, and chain-of-thought elicitation; Gemma3 27B’s chain-of-thought cell was not run.Table 2 reports per-LLM price and review-score coefficients, willingness to pay, and pseudo-R2, with baselines re-estimated on each variant’s task subset.Source: F Prompt-format ablation · pp. 14–15
  • GPT-5.4’s Wyndham preference is spread across five properties, with leave-one-property-out estimates moving its coefficient by at most 0.21; Claude Haiku 4.5 favors all five chains relatively evenly.Claude Haiku 4.5’s chain coefficients have a minimum of +0.60 and a mean of +0.71.Source: Figure 10 · p. 15
  • Cross-model price-sensitivity and willingness-to-pay rankings remain highly correlated with baseline under JSON (+1.00/+1.00), reordered attributes (+0.94/+0.94), persona (+0.89/+0.89), and chain-of-thought (+0.60/+0.70).The chain-of-thought correlations use five LLMs; the other variants cover six.Source: F Prompt-format ablation · p. 15
  • The persona raises the quality lean of all six tested LLMs, while chain-of-thought lowers it in all five tested LLMs.In both variants, the lean itself moves rather than the price coefficient shifting uniformly.Source: F Prompt-format ablation · p. 15
  • Under JSON, Qwen3 4B and Phi-4 Mini lose decisiveness, while Gemma2 9B, Gemma3 27B, and both GPT-5.4 models do not; the two-versus-four split is tentative.The authors describe a possible association between format robustness and capability as an observation, not a finding.Source: F Prompt-format ablation · p. 15

G Decoding temperature

  • At temperature 0.7, four of five position-locked LLMs remain locked; Qwen3 0.6B enters the engaged band but recovers only a weak price signal (pseudo-R2 0.031).That fit is below the two engaged controls’ pseudo-R2 values of 0.114 and 0.280; sampling adds noise rather than usable preferences for this model.Source: G Decoding temperature · p. 15

H Five-option tasks

  • On 300 five-option tasks, all five tested LLMs beat chance by a wide margin on every metric, while the three engaged LLMs put only 2–10% of picks on the first slot.Boundary Mistral-Nemo 12B and locked Llama 3.1 8B continue following the first slot.Source: H Five-option tasks · p. 15

I Bootstrap intervals and multiplicity

Bootstrap resampling supports the stability of the reported rankings and significance claims, while the human-reference comparison depends on explicit scale conversions and differing choice settings. The authors present the human anchors as plausibility bounds, not target values.

I Bootstrap intervals and multiplicity

  • Resampling 450 unique binary choice pairs across 500 shared cluster-bootstrap draws produced no failed refits and yielded per-LLM intervals for price sensitivity and willingness to pay.Shared draws make per-draw rankings comparable across LLMs.Source: I Bootstrap intervals and multiplicity · pp. 15–16
  • Mean bootstrap rank correlations are 0.996 for price sensitivity and 0.991 for willingness to pay; the 16× willingness-to-pay spread has a 95% interval of [11.7×, 23.7×].Of 190 pairwise willingness-to-pay comparisons, 148 have non-overlapping intervals, so adjacent LLMs are often not separable even though the extremes and overall spread are.Source: I Bootstrap intervals and multiplicity · pp. 16–17
  • Of the 40 coefficients claimed at p < 0.001, none loses significance under Bonferroni or Benjamini–Hochberg correction.Across all 184 tests, 138 survive Bonferroni and 155 survive Benjamini–Hochberg at α = 0.05.Source: I Bootstrap intervals and multiplicity · p. 17

J Human-anchor conversion arithmetic

  • For 20 well-identified engaged LLMs, willingness to pay per review-score point ranges from $68 to $1,071 at a $250 nightly reference, with a median of $269.These values equal 27.4% to 428.6% of the nightly rate, with a median of 107.7%.Source: J Human-anchor conversion arithmetic · p. 17
  • Converting human benchmarks requires halving per-point effects from a 10-point to a 5-point review scale, approximating a 10-point gain from ReviewPro’s 0–100 score, and converting HKD at 7.80 per USD.The ReviewPro conversion is a linear extrapolation and therefore an upper bound; the dollar conversions are the authors’ calculations from published coefficients.Source: J Human-anchor conversion arithmetic · p. 17
  • Human reference values are distributions and differ in construct from PriceBench estimates; humans spend their own money under budgets, whereas LLM agents book with synthetic money.The authors treat the human anchors as bounds on plausibility, not as a definition of a correct value.Source: J Human-anchor conversion arithmetic · p. 17

K Datasheet

PriceBench is a diagnostic benchmark for measuring LLM booking preferences, built from hotel profiles and repeated forced-choice tasks. Its perturbed listing data is intended for research measurement, not current commercial guidance.

Composition

  • The benchmark combines 179 NYC hotel profiles with 3,600 forced-choice tasks, split evenly between binary and ternary choices and presented in both orderings.Profiles span 1–5 stars, 36 neighborhoods, five chains plus independents, and $45–$1,650 per night.Source: K Datasheet · pp. 17–18

Collection and preprocessing

  • Profiles were compiled from public travel-agency listings, while model responses were collected using plain-text option cards with greedy decoding at temperature zero.On each appearance, six attributes are re-randomized within property-specific ranges, including price and review information.Source: K Datasheet · pp. 18–19

Uses

  • PriceBench is intended to measure revealed booking preferences and support discrete-choice analysis, not to recommend bookings or provide current commercial information.Its prices and reviews are perturbed rather than current listing facts.Source: K Datasheet · pp. 17, 19

Distribution

  • The release includes tasks, scoring code, analysis pipeline, response sets, and variant runners; the task and response data are released for research use.The hotel pool is derived from public listings, and users should consult source-platform terms before non-research reuse.Source: K Datasheet · p. 19

Related papers

Loading 2609.31468v1…