PaperlayerView exact revision on arXiv

Source-linked AI summary

Fairshare Data Pricing via Data Valuation for Large Language Models

Luyang Zhang, Cathy Jiao, Beibei Li, Chenyan Xiong

arXiv:2502.00198v4cs.GTcs.CL

TL;DR

LLM training-data markets are criticized for opaque, unfair pricing that undervalues sellers’ labor. This paper combines game-theoretic market modeling with data-valuation-based fairshare pricing; it reports benefits for both buyers and sellers, supporting long-term market sustainability.

  • Problem

    LLM data markets are criticized for opaque, unfair pricing that undervalues sellers’ labor and fails to reflect data quality or effort.

    Source: §1 · p. 1

  • Method

    The paper models the LLM training-data market using game theory and introduces fairshare pricing, with data-valuation scores guiding buyer and seller decisions.

    Source: §1 · p. 2

  • Results

    Fairshare yields a win-win: sellers earn optimal prices while buyers achieve higher gains at lower cost, supporting long-term market sustainability.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework offers policymakers and regulators actionable insights for promoting fairness and transparency in LLM training-data markets.

    Source: §5 · p. 10

    The framework assumes a non-cooperative Stackelberg market and models fairness behaviorally through seller participation or exit.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Training data is the backbone of large language models (LLMs), yet today's data markets often operate under exploitative pricing -- sourcing data from marginalized groups with little pay or recognition. This paper introduces a theoretical framework for LLM data markets, modeling the strategic interactions between buyers (LLM builders) and sellers (human annotators). We begin with theoretical and empirical analysis showing how exploitative pricing drives high-quality sellers out of the market, degrading data quality and long-term model performance. Then we introduce fairshare, a pricing mechanism grounded in data valuation that quantifies each data's contribution. It aligns incentives by sustaining seller participation and optimizing utility for both buyers and sellers. Theoretically, we show that fairshare yields mutually optimal outcomes: maximizing long-term buyer utility and seller profit while sustaining market participation. Empirically when training open-source LLMs on complex NLP tasks, including math problems, medical diagnosis, and physical reasoning, fairshare boosts seller earnings and ensures a stable supply of high-quality data, while improving buyers' performance-per-dollar and long-term welfare. Our findings offer a concrete path toward fair, transparent, and economically sustainable data markets for LLM.

1 Introduction

The paper frames LLM data procurement as a market where undervalued annotation labor threatens fairness and sustainable data supply. It proposes fairshare, using data valuation to align buyer and seller incentives, and reports theoretical and simulated mutual benefits.

1 Introduction

  • The paper situates its proposal amid low and unfair data prices that undervalue annotation labor, especially in low-wage markets and marginalized communities.It presents fairshare as a response to these market concerns and a guide for sustainable data procurement.Source: §1 · pp. 1–2
  • Fairshare bases prices on data valuation and assumes transparent access to scores, supporting seller participation and overall data quality.Buyers use scores to select datasets under budget constraints, while sellers set prices based on anticipated buyer demand.Source: §1 · p. 2
  • Fairshare yields a theoretical buyer–seller win-win and, in simulations, higher buyer performance per dollar with sustained seller participation and stable data supply.The empirical simulations cover math problems, medical diagnosis, and physical reasoning.Source: §1 · p. 2
  • Across valuation-method variants, fairshare consistently benefits buyers and sellers, and the framework is described as generalizable across models and tasks.The ablation includes BM25, InflIP, and Datainf; the reported robustness also covers utility measures.Source: §1 · p. 2
  • The framework models the LLM training data market with data valuation, market economics, and game theory.Fairshare uses shared valuation scores to guide buyers’ dataset selection under budget constraints and sellers’ pricing decisions.Source: §1 · p. 2

2 Related Work

Prior work studies data-market pricing, data labor, game-theoretic market behavior, and data valuation, but the paper identifies limited integration of these strands for LLM data markets.

2 Related Work

  • Earlier ML data-market frameworks study auctions and personalized pricing, typically targeting profit or efficiency with myopic buyers and fixed seller participation.The cited work largely concerns classic machine-learning models rather than LLMs.Source: §2 · p. 3
  • Research documents exploitative conditions in LLM data labor, but those findings have not been integrated into formal models of LLM data pricing.The related work describes low pay, precarious labor, limited protections, and harmful-content exposure among outsourced workers.Source: §2 · p. 3
  • Stackelberg games and bilevel optimization model strategic decisions in other domains, while repeated-game work examines long-term incentives and fairness perceptions.The paper notes these frameworks had not yet been applied to LLM data markets.Source: §2 · p. 3
  • Data valuation estimates training examples’ contributions to model performance, using approaches such as influence functions and Shapley-based methods.Influence-based approximations can reduce computation, while BM25 provides a model-agnostic lexical-similarity baseline.Source: §2 · p. 3
  • Data valuation supports LLM data selection and other applications, but exact value estimation remains computationally costly at LLM scale.The cited work reports that influence-based estimates correlate meaningfully with oracle values obtained through retraining.Source: §2 · p. 3

3 A Theoretical Framework of LLM Data Market

The framework models LLM data markets as strategic buyer–seller interactions shaped by data valuation, pricing, and participation. It analyzes how exploitative prices undermine long-term market outcomes and characterizes fairshare pricing as a stable, mutually beneficial alternative.

3.1 Data Market Definition

  • The market is modeled as a non-cooperative Stackelberg game: sellers price first, buyers respond, and shared data-valuation scores inform both sides’ decisions.The framework treats seller participation and exit as behavioral fairness, with leader–follower timing reflecting price setting and demand response.Source: §3.1 · pp. 3–4
  • Buyers choose datasets to maximize net utility subject to budget and non-negative-utility constraints, while sellers set prices to maximize anticipated sales minus dataset creation costs.Buyer utility represents downstream model-performance gains, and seller pricing anticipates buyer purchasing decisions.Source: §3.1 · pp. 4–5

3.2 Exploitative Pricing: A Lose-Lose Outcome

  • Under the stated assumptions, exploitative pricing is optimal only over a finite horizon and becomes strictly suboptimal as seller participation declines.The buyer’s objective trades immediate surplus against future participation; exploitative pricing can reduce data quality and availability over time.Source: §3.2 · p. 6

3.3 Fairshare Pricing: A Win-Win for Sellers and Buyers

  • A seller’s optimal price is one of the buyers’ maximum willingness-to-pay values, aligning seller profit with dataset utility and buyer budget.Maximum willingness to pay is bounded by the buyer’s marginal utility and budget surplus.Source: §3.3 · p. 7
  • In the single-buyer, single-seller infinite-horizon model, the seller-optimal fairshare price also maximizes the buyer’s expected cumulative utility and sustains participation.The price is p*_t = min{u_t, b_t}; the threshold time for fairshare to outperform exploitative pricing increases as the discount factor decreases.Source: §3.3 · pp. 7–8
  • In a synthetic simulation with utility 2, budget 1, and discount factor 0.95, the buyer moves from an exploitative price at T = 1 to fairshare by T = 5.The simulation then maintains the fairshare price, which remains 1 under these conditions.Source: §3.3 · p. 7

4 Empirical Analyses: Benefits of Fairshare Pricing

The experiments compare fairshare with alternative pricing and valuation methods across open-source LLM tasks, finding benefits for market welfare and buyer performance-per-cost. The valuation-method ablation also identifies differences in cost–performance balance and valuation accuracy.

4.1 Experimental Setup

  • The experiments model buyers using open-source LLMs and sellers providing human-annotated training data for math, medical diagnosis, and physical reasoning tasks.Models include Llama-3.2-Instruct-1b, Pythia-1b, and Pythia-410m; datasets include MathQA, GSM8K, MedQA, and PIQA.Source: §4.1 · p. 8

4.2 LLM Data Market Experiments

  • The market experiment simulates two buyers and ten sellers over multiple time steps, with high- and low-budget buyers and four pricing methods.The methods are fairshare, reduced-price, random, and exploitative pricing; participation is simulated over 100 steps with discount factor δ = 0.98.Source: §4.2 · p. 9
  • Exploitative pricing gives LLM developers short-term utility gains but drives sellers out, collapsing data supply and hindering model advancement.The reported seller exit occurs over time, leaving even well-funded developers unable to source sufficient data.Source: §4.2 · p. 9
  • Fairshare yields sellers the highest profit over time and maximizes long-term utility for high-budget buyers, while low-budget buyers trade reduced short-term utility for sustained supply.The low-budget buyer’s limited budget constrains its ability to use the increased dataset supply.Source: §4.2 · p. 9

4.3 Ablation Study: Effect of Different Data Valuation Methods

  • Across tasks, BM25, InflIP, and DataInf generally improve buyer performance; InflIP provides strong gains at lower cost than constant, random, and BM25 valuations.Figure 6 compares average price-per-sample cost and model performance versus cost before purchasing and after purchasing 2K and 4K samples.Source: Figure 6 · p. 10, §4.3 · p. 10
  • Oracle–InflIP Spearman correlations are 0.54 for MathQA and 0.42 for PIQA, indicating room to improve valuation accuracy and scalability.The paper frames improved valuation methods as compatible with its flexible fairshare framework.Source: §4.3 · p. 10

5 Conclusion

The conclusion presents fairshare as a data-valuation-based pricing framework for LLM training data, reporting buyer gains at lower cost and optimal seller prices. It identifies extensions to valuation methods, information settings, and data domains as future work.

5 Conclusion

  • Fairshare combines data valuation with pricing for LLM training data, with reported buyer gains at lower cost and optimal seller prices supporting long-term market sustainability.The conclusion describes the outcome as a win-win and links it to social welfare.Source: §5 · p. 10
  • The framework is presented as an actionable source of insights for policymakers and regulators concerned with fairness and transparency in LLM training-data markets.The conclusion also connects fair market access with opportunities for small businesses and startups.Source: §5 · p. 10
  • Future work could add data valuation methods, incomplete-information settings such as Bayesian games, and data domains such as pretraining.The conclusion also names fine-tuning as a data domain for future study.Source: §5 · p. 10

A Influence-based Data Valuation

The appendix describes an influence-based valuation approach, a royalty pricing model, and adaptations for different buyer-outcome mappings. These include linear, discrete, and zero-one outcomes.

A Influence-based Data Valuation

  • InflIP estimates a training sample’s impact on a test sample’s loss by taking a training step and measuring the resulting loss change; its approximation is the dot product of their learning gradients.The passage introduces the setup using model parameters θ, training sample d, test sample d′, and learning rate η.Source: A Influence-based Data Valuation · p. 17

B Royalty model

  • The royalty model replaces full upfront prices with rates proportional to future LLM economic value, targeting buyers uncertain about model outcomes or lacking sufficient cash flow.The model assumes the rate function is non-decreasing in the individual rate, and buyers have a maximum rate they are willing to pay.Source: B Royalty model · p. 17

B.1 Solving for optimal price of the royalty model

  • Under royalty pricing, a buyer considers a dataset when adding it raises net utility and remains within its rate budget; the seller’s optimal rate reflects buyer willingness to pay.The optimal rate is characterized by the minimum of marginal utility and the buyer’s budget surplus, with the rate function also affecting total payment.Source: §B.1 · p. 18

C Applications for Real-Life Scenarios

  • The framework allows different mappings from data valuation scores to buyer utility, including linear, discrete, and zero-one outcomes.The seller’s pricing problem remains unchanged across these mappings.Source: C Applications for Real-Life Scenarios · p. 19

C.1 Linear Outcome

  • For a linear outcome, buyer utility is an affine function of data valuation, with a bias term for factors independent of the valuation score.The buyer’s net utility subtracts the price of the selected data collection.Source: §C.1 · p. 19

C.2 Discrete Outcome

  • For a discrete outcome, valuation-score ranges map to category rewards, with higher score ranges assigned higher rewards.The example is an LLM benchmark challenge in which performance ranges can affect leaderboard rankings.Source: §C.2 · p. 19

C.3 Zero-One Outcome

  • For zero-one outcomes, the buyer models a small positive reward with probability equal to valuation and a large negative reward for undesirable rare events.The medical example concerns the risk of incorrect information causing misdiagnosis or unsuitable emergency protocols.Source: §C.3 · p. 19

C.4 Multiple tasks

For multiple tasks, the buyer’s utility is modeled as an affine combination of task utilities, which are mapped from task-specific data valuation scores.

C.4 Multiple tasks

  • The multiple-task buyer combines task utilities using a coefficient vector and an independent offset, with each task utility based on its valuation score.The model assumes each task uses one of the outcome categories described earlier.Source: §C.4 · p. 20

D Lemmas and Proofs

The proofs characterize the costs of exploitative pricing, seller pricing under fairshare, and conditions for buyer utility. They also connect gradient-based valuation to the appendix’s data valuation experiments.

D Lemmas and Proofs

  • Exploitative pricing is optimal for buyer cumulative utility only over a finite horizon, after which it is strictly suboptimal.The proofs also establish that exploitative pricing steadily reduces participation, with long-run participation loss bounded below by 1−e^-S.Source: D Lemmas and Proofs · p. 20
  • A seller’s optimal dataset price is characterized by buyers’ maximum willingness to pay.A buyer purchases only when the dataset improves net utility and the purchase remains within budget.Source: D Lemmas and Proofs · p. 21
  • The ideal price p*_t = min{u_t, b_t} maximizes buyer cumulative net utility over an infinite horizon.In the single-buyer, single-seller case, a price above p*_t prevents purchase, while a lower price fails to maximize seller profit.Source: D Lemmas and Proofs · p. 21
  • Lemma 4 defines the threshold period when fairshare pricing yields higher cumulative utility than exploitative pricing.The supplied proof passage does not establish a consistent direction for how this threshold changes with δ.Source: D Lemmas and Proofs · p. 22
  • A gradient step on training sample d approximates its effect on another sample’s loss through the samples’ gradient dot product.The approximation scales with learning rate η, with higher-order terms omitted.Source: D Lemmas and Proofs · p. 22

E.1 Data Valuation Experiments

  • E.1 scores market samples by similarity to 200 representative demonstrations sampled for each dataset.Buyers train with LoRA for three epochs on equal-sized samples from their purchases; evaluation uses 5-shot MathQA and 4-shot MedQA test sets.Source: §E.1 · p. 23

E.2 Data Pricing Experiments

The pricing experiments vary buyer budgets and discount factors, while the market algorithm makes purchases conditional on utility and budget before sellers select prices.

E.2 Data Pricing Experiments

  • The pricing simulation uses high- and low-budget buyers whose budgets are sampled from 95–100% and 90–95% of the total utilities of ten datasets.Seller arrival order is shuffled each period, and sellers price their datasets sequentially.Source: §E.2 · p. 23
  • For the high-budget buyer, δ values of 0.999, 0.99, and 0.98 yield fairshare thresholds t* of 32, 38, and 44, respectively.These values are reported as consistent with the theoretical analysis.Source: §E.2 · p. 23
  • Figure 7 compares buyers’ cumulative utilities over time for δ = 0.980, 0.990, and 0.999.The passage specifies the plotted variables and discount factors but gives no outcome comparison.Source: Figure 7 · p. 23

F.1 Algorithms

  • The market algorithm lets a buyer purchase a dataset only if it improves net utility and fits the buyer’s budget.Sellers evaluate candidate prices against buyer decisions, compute net profit, and select a fixed dataset price.Source: §F.1 · p. 24

F.2 Data Valuation Oracle Experiments

The oracle experiments compare InflIP valuation with a one-step-training oracle in both valuation agreement and resulting model performance. Performance differences vary with agreement and the amount of purchased data.

F.2 Data Valuation Oracle Experiments

  • InflIP estimates a sample’s influence on another example, while the Oracle score measures the loss change after one gradient step on that sample.The Oracle score is defined as L(d′; θ) − L(d′; θ̂), with θ̂ = θ − η∇L(d; θ) [71].Source: §F.2 · pp. 24–25
  • When Oracle and InflIP valuation agreement is decent, their final model performance is close.Figure 8 places valuation correlation comparisons in the left columns and performance comparisons in the right columns.Source: Figure 8 · p. 24, §F.2 · p. 25
  • When valuation agreement is lower, the initial performance gap between Oracle and InflIP becomes smaller as more data is purchased.The paper reports that the resulting final performance can still be similar in this case.Source: §F.2 · p. 25

F.3 Additional Experimental Results

The appendix shows additional pricing and valuation experiments across models, tasks, and buyer budgets, then describes the dataset tables and the paper’s stated societal motivation.

F.3 Additional Experimental Results

  • The valuation experiments compare model performance and costs using constant, random, BM25, and InflIP purchasing methods on math, medical, and physical reasoning data.The plotted models are Pythia-1b and Pythia-410m.Source: Figure 9 · p. 25
  • Additional experiments track buyer utility, seller profit, and seller counts under four pricing schemes across three models and MedQA, MathQA, and PIQA.A separate figure tracks purchased dataset counts for high- and low-budget buyers across the three tasks and models.Source: Figure 10 · p. 26, Figure 11 · p. 27, Figure 12 · p. 28

F.4 Datasets

  • The appendix includes dataset split and demonstration tables for MathQA, GSM8K, MedQA, and PIQA, including examples used for MathQA and MedQA few-shot evaluation.The demonstrations were randomly selected from their respective training sets.Source: Table 1 · p. 31, Table 2 · p. 32

G Limitations and Impact

  • The paper frames its contribution as addressing underpayment of annotators and compensation disconnected from the economic value of their contributions to LLMs.It presents fair compensation from LLM developers to data annotators as an ethical and societal concern.Source: G Limitations and Impact · p. 33

G.1 Limitations

The limitations section identifies larger-scale simulations, human buyer–seller simulations, and additional market dynamics as future work. The impact statement emphasizes benefits for annotators and LLM developers, while the checklist says the paper’s claims are based on its experiments, analysis, and findings.

G.1 Limitations

  • Future work could test the framework with more datasets and models, human buyers and sellers, and market dynamics such as incomplete information.These adjustments are described as beyond the paper’s scope.Source: §G.1 · p. 33

G.2 Impact Statement

  • The authors state that fair compensation and sustained annotator participation support an equitable data ecosystem and a steady supply of diverse, high-quality datasets.The impact statement also says the framework maximizes LLM developers’ long-term utilities and welfare.Source: §G.2 · p. 33

NeurIPS Paper Checklist

  • The checklist says the abstract and introduction are based on the paper’s experiments, analysis, and main findings.The authors report that their theoretical and empirical results support the proposed framework.Source: NeurIPS Paper Checklist · p. 34

2. Limitations

The checklist reports that experiments focus on qualitative or directional insights rather than formal statistical claims. It also records disclosures on proofs, experimental reproducibility, code and data access, and human-subject research.

Theory Assumptions and Proofs

  • The authors report that assumptions and proofs are discussed in the main paper and appendix.The checklist marks the theoretical assumptions-and-proofs item Yes.Source: §2 · pp. 34–35

Experimental Result Reproducibility

  • The paper reports dataset, split, model, and training-procedure or hyperparameter details, and says experimental resources are open source.The checklist marks disclosure sufficient for reproducing the main experimental results Yes.Source: §2 · p. 35

Open access to data and code

  • The authors say code and data will be openly available on GitHub.The checklist question asks whether open access and instructions are provided for reproducing the main results.Source: §2 · p. 36

Experiment Statistical Significance

  • The experiments focus on qualitative or directional insights, not formal statistical claims, and the checklist marks reporting of statistical significance information as No.The checklist question concerns suitably defined error bars or other statistical-significance information.Source: §2 · p. 36

Crowdsourcing and Research with Human Subjects

  • The authors state that they conducted no research with human subjects.The crowdsourcing and human-subjects checklist item is marked NA.Source: §2 · p. 39
Loading 2502.00198v4…