Source-linked AI summary
Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
Daniel Robert Kling Alexander, Catherine Louise Kling
TL;DR
Questions about value, choice, and trade-offs often lack correct answers to score against, and human answers are an uncertain substitute. The paper applies stated-preference economics’ validity framework to LLM evaluation and demonstrates it with a water-quality referendum survey. The tests sharply distinguish models, while passing them establishes coherence rather than correctness.
Problem
Many LLM questions have no correct answer key, and matching uncertain human answers is a weak substitute for evaluating them.
Source: §1 · p. 1Method
The paper adapts stated-preference economics’ validity framework for LLM evaluation and demonstrates it with a published water-quality referendum survey administered to six models.
Source: §1 · p. 2Results
The oldest models fail the most basic theoretical test, while the newest pass every scorable theoretical test and diverge under convergent validity.
Source: §5 · p. 6Takeaways & Limitations
Validity tests can locate where models’ answers fail or diverge, but passing them shows coherence rather than correctness.
Source: §5 · p. 6Passing the tests cannot distinguish genuine scenario evaluation from a learned rule of thumb, and withholding the scale’s name does not remove that risk.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
1 Introduction
Many LLM questions lack a correct answer key, and matching uncertain human answers is a weak substitute for evaluating them. The paper proposes stated-preference economics’ validity framework for such cases and demonstrates it with a water-quality survey administered to six models.
1 Introduction
- LLM evaluation can ask whether answers are any good when accuracy is undefined and human responses are uncertain.Researchers and firms also increasingly use LLMs in place of survey respondents.Source: §1 · p. 1
- Stated-preference validity concepts provide a way to judge answers without knowing the true value, including content, criterion, and construct validity, reliability, incentive compatibility, and consequentiality.For valuation, theoretical validity asks whether demand slopes down and value responds to the good’s scope and household income.Source: §1 · p. 2
- The framework applies beyond economic questions, while stated-preference economics adds tests for questions without right answers and concrete behavioral checks such as scope tests.The paper notes that none of the compared approaches uses the full validity framework.Source: §1 · p. 2
- The paper demonstrates the framework using Vossler et al. (2023, PNAS)’s published water-quality referendum instrument with six models from two developers.The instrument is selected because it is well documented and its validity tests are well understood.Source: §1 · p. 2
2 The validity framework for language models
The framework transfers validity concepts to LLM evaluation, with theoretical validity doing much of the work and respondent-stakes concepts posing open empirical questions. An advisory framing makes theoretical validity a normative standard rather than a description of individual preferences.
2 The validity framework for language models
- Theoretical validity transfers unchanged and needs predictions about how responses should move, not a true answer; a larger good should be worth at least as much as a smaller good it contains.Reliability can be tested cheaply through repeated model sampling, with stability across model versions as an additional dimension.Source: §2 · p. 2
- Criterion validity, incentive compatibility, and consequentiality have no direct analogue for LLMs and remain open empirical questions.Table 1 sets out what each validity concept asks of an LLM evaluation.Source: §2 · p. 2, Table 1 · p. 3
- When advising a household, a model that recommends paying less for a policy delivering strictly more makes an error, not merely a preference choice.This advisory framing turns theoretical validity into a normative standard for models acting on a user’s behalf.Source: §2 · p. 3
3 A demonstration: valuing water quality
The demonstration reproduces a published water-quality referendum survey and varies improvement, region, tax, and household income across six models. Willingness to pay is measured using a vote-share threshold and a fitted logit.
3 A demonstration: valuing water quality
- The survey reproduces the original water-quality referendum text and withholds the published scale name to reduce the chance that models recall the study.The instrument surveyed households in three river basins about ecological conditions measured on a six-level scale.Source: §3 · p. 3
- The design crosses nine improvement-region scenarios with ten annual tax levels and three household incomes, using ten independent draws per cell across six models.Improvements include one-level gains and minimum-level thresholds; regions include local, non-local, and full-study areas.Source: §3 · p. 4
- WTP is the first tax level where fewer than half of votes are yes; cases remaining at or above half through $3,000 are censored.A standard logit also estimates median WTP as α/β using penalized likelihood.Source: §3 · p. 4
4 Results
The economic tests reveal distinct price, scope, and income response patterns; convergence across measures is mixed, and version stability is difficult to establish.
4.1 Theoretical validity
- At $3,000, Haiku 4.5 voted yes in all 600 draws and GPT-4o mini in 87%, while Sonnet 5 and Terra fell to 0.00 and 0.08.The figure plots yes-vote share against annual tax; a downward slope is required for a valid valuation.Source: §4.1 · p. 4, Figure 1 · p. 5
- Only one nested-dominance violation occurred, while distance decay held in all 33 scored comparisons; spatial-extent tests separated the models.The nested-dominance violation was Haiku at the lowest income, within noise. Sonnet 5 and Terra passed every spatial-extent comparison; Sonnet 4.5 and GPT-4o priced the full region below the local watershed in five of eight.Source: §4.1 · p. 4
- WTP never fell in 69 scored income comparisons, although many $200,000-income values were censored above $3,000.Source: §4.1 · p. 4
4.2 Convergent validity
- Logit estimates yielded no significant nested-dominance, distance-decay, or income violations, but cross-format agreement was weaker and newest-model income elasticities exceeded typical SP findings.Sonnet 5 and Terra implied income elasticities of 1.0–2.2, while SP studies usually find well below one; the logit and referendum estimates use the same raw data.Source: §4.2 · p. 5
4.3 Reliability
- At ten draws per tax, vote-share standard error near one-half is about 0.15; stability across versions is harder to establish for undated newest models.The newest models are available only under undated names, so a later run under the same name may reach a different model.Source: §4.3 · pp. 5–6
5 Discussion
The framework distinguishes coherent-but-not-necessarily-correct answers and identifies where tested models fail. Applying it beyond economics and testing concepts that do not yet transfer remain next steps.
5 Discussion
- The oldest models fail the most basic theoretical test, while the newest pass every scored theoretical test but diverge in value levels and income sensitivity under convergent validity.A single accuracy score could not make this distinction, and without ground truth there would be no accuracy score to compute.Source: §5 · p. 6
- Passing validity tests indicates coherent answers, not correct ones; a learned rule of thumb could pass without genuine scenario weighting.The tests cannot distinguish such a rule from genuine weighing, and models may have learned expected tests from stated-preference literature; withholding the scale name limits but does not remove this risk.Source: §5 · p. 6
- Criterion validity has no direct counterpart because models do not pay, while incentive compatibility and consequentiality presuppose respondents with something at stake.Whether answers change when advice will be acted on or question wording invites agreement is testable and connects to evaluation awareness and sycophancy.Source: §5 · p. 6
- The evidence covers one instrument and one good; the $3,000 tax ceiling, ten draws per tax, and exploratory first round constrain interpretation.The newest models ran under undated names, and the $75,000 watershed and study-region cells came from different rounds a week apart, so some spatial and income comparisons may not involve the same model. The Yes/No order was not swapped; a yes-bias could shift vote shares or flatten curves for models that vote yes at nearly every price.Source: §5 · p. 6
- The paper concludes that stated-preference lessons apply to LLMs answering value, judgment, and advice questions, with applications beyond economics still to come.Further work should test concepts that do not yet transfer.Source: §5 · p. 6
Appendix
Table A1 reports estimated willingness to pay at $75,000 household income and includes human means for an approximate comparison.
Appendix
- Table A1 estimates willingness to pay at a $75,000 household income in dollars per year for five years.The human comparison means average respondents’ own watersheds, which vary in size.Source: Table A1 · p. 7
- Estimates use α/β from a Firth-penalized logit fitted to 100 votes per cell, with delta-method standard errors in parentheses.A dash denotes a price coefficient that is not significantly positive; a dagger marks an estimate above the $3,000 top bid.Source: Table A1 · p. 7
- Human means come from Vossler et al. (2023), Model 1 (mixed logit), in 2021 dollars, so the comparison is approximate.Source: Table A1 · p. 7