PaperlayerView exact revision on arXiv

Source-linked AI summary

The Problem with Metrics is a Fundamental Problem for AI

Rachel Thomas, David Uminsky

arXiv:2002.08512v1cs.CYcs.AI

TL;DR

This paper examines how AI’s reliance on optimizing metrics can lead to manipulation and other harms when metrics stand in for broader goals. It uses real-world cases to illustrate those failures and proposes a framework combining diverse metrics, qualitative accounts, and broader ethical input.

Abstract

from arXiv · show

Optimizing a given metric is a central aspect of most current AI approaches, yet overemphasizing metrics leads to manipulation, gaming, a myopic focus on short-term goals, and other unexpected negative consequences. This poses a fundamental contradiction for AI development. Through a series of real-world case studies, we look at various aspects of where metrics go wrong in practice and aspects of how our online environment and current business practices are exacerbating these failures. Finally, we propose a framework towards mitigating the harms caused by overemphasis of metrics within AI by: (1) using a slate of metrics to get a fuller and more nuanced picture, (2) combining metrics with qualitative accounts, and (3) involving a range of stakeholders, including those who will be most impacted.

Goodhart’s Law and Machine Learning

Metrics are central to machine-learning training because models are optimized against quantitative cost functions. Goodhart’s Law therefore offers a lens for understanding how making a measure the target can undermine its value.

Goodhart’s Law and Machine Learning

  • A machine-learning performance measure must be quantitative, and choices about its definition include how to weigh different errors and partial credit.Organizational metrics and KPIs are also used more broadly for quantified business and software-product measurements.Source: Goodhart’s Law and Machine Learning · p. 2
  • Training seeks neural-network parameters that reduce a cost function containing a performance measure and typically regularization terms.The paper links this explicit optimization target to heightened risks from metric optimization.Source: Goodhart’s Law and Machine Learning · p. 2
  • Goodhart’s Law holds that a measure that becomes a target ceases to be a good measure, a pattern first described in monetary targeting.The paper applies this lens to deep learning because training targets optimization of a cost function.Source: Goodhart’s Law and Machine Learning · p. 2

Previous related work

Prior work frames metric optimization failures through Goodhart’s Law, reward hacking, and alternatives to direct utility maximization. This paper focuses on real-world cases involving human and algorithmic behavior, online environments, and business practices.

Previous related work

  • Manheim and Garrabrant (2019) distinguish regressional, extremal, causal, and adversarial Goodhart failures when a proxy for a goal becomes a target.Their taxonomy also subdivides causal and adversarial failure modes.Source: Previous related work · p. 2
  • Reward hacking describes algorithms finding ways to optimize a reward function that subvert the designer’s intent.The paper distinguishes this focus on internal algorithmic behavior from its interest in systems combining human and algorithmic behavior.Source: Previous related work · pp. 2–3
  • Green (2018) raises the concern that optimizing explanations may yield answers humans find appealing rather than accurate.The paper’s case studies examine how metric failures manifest in practice and how online environments and business practices exacerbate them.Source: Previous related work · pp. 2–3
  • Taylor (2016) proposes q-quantilizers as an alternative to deterministic utility maximization, trading off utility and depending on a well-understood safe action distribution.Slee (2019) argues that pure incentive maximization can be incompatible with optimal outcomes for intended use cases.Source: Previous related work · p. 3

We can’t measure the things that matter

Metrics often serve as proxies for outcomes that matter but are difficult to measure directly. The examples show how a proxy can capture healthcare access or engagement rather than the intended outcome.

We can’t measure the things that matter

  • A model intended to predict stroke risk largely identified people who used healthcare more, because its records captured care access and billing codes rather than stroke events.Insurance, affordability, time, childcare, gender and racial bias, and cultural factors all influence who receives care and diagnoses.Source: most · p. 3
  • Clicks, student test scores, and arrests are used as proxies for content preference, teacher effectiveness, and crime, though these measures are not equivalent to those goals.The paper cautions that many things people care about cannot be measured and that metrics remain proxies.Source: most · p. 3
  • YouTube used hours watched as a proxy for user happiness, and Chaslot said this incentivized conspiracy theories because they kept users watching.The cited explanation is that viewers could be persuaded that other media were lying.Source: most · p. 3

Metrics can, and will, be gamed

When metrics carry too much weight, people and systems can game them, distorting the measured outcome. Cases in healthcare, recommendations, and essay grading illustrate different forms of that problem.

Metrics can, and will, be gamed

  • 90% of patients were officially reported as seen within four hours, versus 69% who reported this in surveys, amid healthcare target gaming.Hospitals reportedly canceled scheduled operations, queued ambulances, and used hallway stretchers as beds to meet wait-time targets.Source: Metrics can, and will, be gamed · p. 3
  • Gaming can take the forms of ratchet effects, threshold effects, and output distortions, including meeting targets at the cost of unmeasured performance.In U.S. education, overemphasis on test scores was followed by scandals involving teachers and principals altering student scores.Source: Metrics can, and will, be gamed · p. 3
  • In a sample of 84,695 YouTube videos, RT was an extreme outlier in how often other channels recommended it, which Chaslot said suggested manipulation of the recommendation algorithm.Although the video had only about 55,000 views, Chaslot said recommendations could gradually elevate Russian views and generate more advertising revenue and reach.Source: Metrics can, and will, be gamed · p. 4
  • Essay graders prioritize sentence length, vocabulary, spelling, and subject-verb agreement but cannot assess difficult-to-quantify qualities such as creativity.The passage says computer-generated gibberish with sophisticated words scored well; students from mainland China scored higher with the algorithms than with expert graders, suggesting possible use of pre-memorized text.Source: Metrics can, and will, be gamed · p. 4

Metrics overemphasize short-term concerns

Short-term measures are easier to quantify, but overemphasizing them can displace attention from long-term concerns, values, trust, and reputation.

Metrics overemphasize short-term concerns

  • Cross-selling pressure and an unethical sales culture led Wells Fargo employees to open 3.5 million fraudulent accounts without customers’ consent.The paper contrasts the short-term cross-selling metric with the longer-term goal of nurturing customer relationships.Source: Metrics overemphasize short-term… · p. 4
  • Overemphasizing metrics shifts attention toward short-term outcomes and away from longer-term values, trust, reputation, and societal and environmental impact.Source: Metrics overemphasize short-term… · p. 4
  • Click-through rates, month-over-month churn, and quarterly earnings are easier to measure than many complex long-term trends.Source: Metrics overemphasize short-term… · p. 4

highly addictive environments

Tech companies use clicks, time spent, and engagement as proxies for user preference, even as these measures are gathered in environments shaped by addictive design and dark patterns. The paper argues that such metrics may reflect the environment’s engineered choices rather than what users would prefer in a healthier one.

highly addictive environments

  • Clicks, time spent, and engagement are used as proxies for user preference in environments engineered to be addictive and laden with dark patterns.Source: highly addictive environments · pp. 4–5
  • 1,818 dark patterns appeared on 1,254 shopping websites, or 11.1% of sites in a study of approximately 11K websites (Mathur, et. al. 2019).More popular websites were more likely to feature dark patterns, including obstruction, misdirection, and misrepresented user actions.Source: highly addictive environments · p. 5
  • The paper compares recommendation systems that autoplay or prioritize content to a cafeteria that automatically offers more food after a child finishes a snack.Source: highly addictive environments · p. 5
  • As users’ selections become normalized, recommended videos can become more extreme, while the metrics may fail to reflect preferences in a healthier or more empowering environment.Source: highly addictive environments · p. 5

A Framework for a Healthier Use of Metrics

The paper proposes keeping metrics in context through multiple measures, qualitative accounts, and involvement of affected stakeholders. It extends stakeholder participation to the development of metrics and guardrails.

A Framework for a Healthier Use of Metrics

  • Use a slate of metrics, qualitative accounts, and input from affected stakeholders to support a fuller and more nuanced use of metrics.The paper cautions against reducing multiple measures to a single score.Source: A Framework for a Healthier Use of… · pp. 5–6
  • For diversity and inclusion assessments, hiring rates alone are limited; the paper also names promotion, ownership, retention, harassment, and underleveling measures.It says these data should still be combined with first-person accounts from workers.Source: A Framework for a Healthier Use of… · p. 5
  • Qualitative information can address phenomena that metrics cannot be known to capture in advance; model and dataset documentation can include intended uses, assumptions, caveats, and ethical considerations.Source: A Framework for a Healthier Use of… · pp. 5–6
  • The paper urges including domain experts and affected stakeholders in developing metrics, not only in understanding model implications.It points to methods for eliciting feedback from underrepresented populations.Source: A Framework for a Healthier Use of… · p. 6
  • Guardrails that restrict users require transparent and just corrective measures shaped by an expanded ethical circle.Source: A Framework for a Healthier Use of… · p. 6

Related papers

Loading 2002.08512v1…