Source-linked AI summary
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Kawin Ethayarajh, Dan Jurafsky
TL;DR
Performance-based leaderboards can reward accuracy while overlooking model qualities valued by practitioners, making them imperfect proxies for practitioner preferences. The paper compares leaderboard and practitioner utility, argues that their priorities can diverge, and recommends transparency that lets users estimate utility and re-rank models.
Problem
Performance-focused leaderboards may not reflect practitioner preferences for qualities such as fairness, efficiency, and robustness.
Source: §1 · p. 1, §3 · p. 2, §3.1 · p. 3Method
Results
The paper argues that leaderboard utility can diverge from practitioner utility because rank-based evaluation may not reflect model costs or other practitioner-valued qualities.
Source: §1 · p. 1, §3 · p. 2, §3.1 · p. 3Takeaways & Limitations
Leaderboards should require practical statistics so users can estimate each model’s utility and re-rank models to fit their preferences.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -- in their current form -- can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).
1 Introduction
Performance-based leaderboards have helped drive more accurate NLP models, but can overlook qualities the community values, including fairness, efficiency, and robustness. The paper uses utility to compare leaderboard and practitioner priorities, identifying design limitations and advocating practical-statistic transparency.
1 Introduction
- Leaderboards can reward models that practitioners find impractical, while overlooking fairness, energy efficiency, and robustness.The paper identifies three mismatches: rank may not reflect every accuracy gain, prediction costs are treated as zero, and robustness benefits may leave leaderboard utility unchanged.Source: §1 · pp. 1–2
- The paper frames leaderboards and practitioners as model consumers whose benefits from models are represented by utility functions.Both groups value accuracy, but practitioners also consider model size and efficiency, unlike leaderboards.Source: §1 · p. 1
- The paper’s three critiques are not comprehensive; it leaves other leaderboard problems for future work.Source: §1 · p. 2
- Because practitioners have diverse preferences, the paper advocates requiring leaderboards to report practical statistics so users can estimate model utility.Examples include model size and energy efficiency; without a leaderboard requirement, creators have little incentive to report such statistics.Source: §1 · p. 2
2 Utility Functions
The paper models leaderboards and NLP practitioners as consumers to compare their preferences over models. Leaderboard preferences are revealed by rank, while practitioners may value several model properties; the framework assumes shared held-out test data.
2 Utility Functions
- In cardinal utility theory, the amount consumed is mapped to a numerical value that quantifies utility in utils.The paper defines utility as the benefit a consumer receives from a good.Source: §2 · p. 2
- The paper uses “leaderboard” for a performance-based ranking of models or systems on a shared benchmark.Examples include GLUE and one-off SemEval STS challenges.Source: §2 · p. 2
- The framework assumes that all models are evaluated on the same held-out test data.Source: §2 · p. 2
- Leaderboard utility is monotonic in rank: the top-ranked model is preferred to all others, and each lower-ranked model to those below it.The exact shape of the leaderboard utility function remains unknown.Source: §2 · p. 2
- Practitioners derive utility from multiple model properties, such as accuracy, energy efficiency, and latency, and their preferences can differ.Their utility is monotonic non-decreasing in each desideratum, holding other properties constant.Source: §2 · p. 2
3 Utilitarian Critiques
The paper critiques leaderboard utility as a poor proxy for practitioner preferences, while noting that the critiques vary by leaderboard. Its analysis focuses on rank-based accuracy incentives, ignored prediction costs, and robustness and fairness attributes that rankings may not reward.
3 Utilitarian Critiques
- The critiques apply across practitioner utility-function shapes, but not every critique fits every leaderboard.StereoSet’s bias-focused ranking makes the fairness critique less applicable.Source: §3 · pp. 2–3
3.1 Non-Smoothness of Utility
- Leaderboard utility changes with accuracy only when accuracy changes rank, while practitioner utility can rise with any accuracy improvement.The paper characterizes leaderboard utility as a step function and practitioner utility as smooth with respect to accuracy.Source: §3.1 · p. 3, Figure 1 · p. 3
- Rank-based incentives underserve practitioners seeking lightweight models between baseline and SOTA accuracy, leaving limited options and impeding Green AI adoption.The paper gives the SNLI leaderboard as an example: most submitted models are highly parameterized and exceed 85% accuracy.Source: §3.1 · p. 3
3.2 Prediction Cost
- Leaderboards rank prediction performance without accounting for model costs, so an impractical SOTA model can have high leaderboard utility but zero practitioner utility.The paper notes that GPT-3’s size precluded full reproduction by researchers, while API access was restricted; high resource demands particularly burden practitioners with fewer resources.Source: §3.2 · pp. 3–4
- Cheaper models are created despite leaderboard incentives, but reporting prediction costs can help users estimate utility because cost-ignorant rankings may be poor proxies for their preferences.The paper says cost reporting and limits in challenge tasks are not yet mainstream.Source: §3.2 · p. 4
3.3 Robustness
- Rankings based on test performance do not reward robustness or fairness, despite examples of brittle or biased models and efforts to address these issues.The paper cites adversarial-performance certificates, Winogender accuracy in SuperGLUE, and approaches to limit overfitting from repeated submissions.Source: §3.3 · p. 4
4 The Future of Leaderboards
Because practitioners differ in their priorities, the paper recommends transparent reporting and user-specific rankings rather than a single ordering. It also describes ways to estimate utility for groups of users.
4.1 A Leaderboard for Every User
- Leaderboards should require reporting practitioner-relevant metrics, including training time, model size, inference latency, and energy efficiency.The paper argues that voluntary reporting offers no explicit incentive, while leaderboards make these statistics accessible in one place.Source: §4.1 · p. 5
- With access to these statistics, practitioners could estimate model utility, re-rank models, adjust metric weights, and filter out models that miss their criteria.One example is filtering models above a chosen parameter count.Source: §4.1 · p. 5
- Reporting prediction costs could put large institutions and poorly resourced creators on more equal footing and encourage attention to worst-off group performance.The paper suggests this could motivate simpler methods whose ease of use compensates for weaker performance.Source: §4.1 · p. 5
4.2 A Leaderboard for Every Type of User
- Groups with shared goals can be modeled by a single utility function, such as low-latency programmers or researchers prioritizing accuracy.The approach is tractable when group members have many points of consensus, despite within-group variation.Source: §4.2 · p. 5
- A static group-specific leaderboard could estimate utility explicitly by asking users how much they would pay for model properties or improvements.The paper gives accuracy-latency trade-offs in speech translation as an example of eliciting user preferences.Source: §4.2 · p. 5
- An implicit approach would infer preferences from models practitioners use, but usage data may be unavailable and choices may lack complete information.This approach draws on revealed preference theory.Source: §4.2 · p. 5
5 Conclusion
The paper uses a utility framework to examine the divergence between what NLP leaderboards incentivize and what practitioners value. It advocates transparency so users can estimate model utility and create customized rankings.
5 Conclusion
- The paper uses a utility framework to study how leaderboard incentives diverge from practitioner values, extending earlier critiques of NLP leaderboards.The authors describe their work as the first to apply this framework to the divergence.Source: §5 · p. 5
- Because practitioners have diverse preferences, transparent reporting lets each user estimate model utility and re-rank models for a custom leaderboard.The paper proposes reporting statistics of practical concern.Source: §5 · p. 6