PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker

arXiv:2304.12397v1cs.CLcs.AI

TL;DR

Commercial toxicity APIs are frequently retrained, creating reproducibility challenges when research reuses older scores. The paper rescored prior text and benchmark results to measure those effects, finding changed findings and rankings, including in HELM. It recommends caution in cross-study comparisons and more structured evaluation over time.

  • Problem

    Frequently retrained, poorly communicated black-box API updates can make toxicity scores reused in research outdated, challenging reproducibility.

    Source: §1 · p. 2

  • Method

    The authors rescore previously generated text and examine API-change effects on RTP statistics, HELM model comparisons, and toxicity-mitigation findings.

    Source: §2 · p. 3

  • Results

    Across rescoring, findings and model rankings changed; 13 HELM models changed results, with 24 Toxic Fraction ranking changes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend caution in direct cross-study comparisons, rescoring experimental text, and clearer communication of API updates.

    Source: §1 · p. 2, §6 · p. 11

    Perspective API does not announce all model updates or allow calls using previous model versions.

    Source: §6 · p. 11

Abstract

from arXiv · show

Perception of toxicity evolves over time and often differs between geographies and cultural backgrounds. Similarly, black-box commercially available APIs for detecting toxicity, such as the Perspective API, are not static, but frequently retrained to address any unattended weaknesses and biases. We evaluate the implications of these changes on the reproducibility of findings that compare the relative merits of models and methods that aim to curb toxicity. Our findings suggest that research that relied on inherited automatic toxicity scores to compare models and techniques may have resulted in inaccurate findings. Rescoring all models from HELM, a widely respected living benchmark, for toxicity with the recent version of the API led to a different ranking of widely used foundation models. We suggest caution in applying apples-to-apples comparisons between studies and lay recommendations for a more structured approach to evaluating toxicity over time. Code and data are available at https://github.com/for-ai/black-box-api-challenges.

1 Introduction

Commercial toxicity APIs are widely used because human evaluation is costly and difficult, but their updates can undermine reproducibility when older scores are reused. The paper reports changed toxicity findings across rescored datasets, benchmarks, and mitigation studies, and recommends rescoring text and clearer API update communication.

  • Toxicity assessment requires linguistic expertise and cultural context, while human annotation of expanding datasets is costly and can harm evaluators’ mental health.Source: §1 · p. 1
  • Black-box APIs are frequently retrained, and poorly communicated updates can leave datasets and previously scored continuations using outdated toxicity definitions or scores.Source: §1 · p. 2
  • 49% fewer toxic prompts appeared in the rescored RealToxicityPrompts dataset relative to its released scores.Source: §1 · p. 2
  • 13 HELM models changed results, producing 24 Toxic Fraction ranking changes when the benchmark was rescored.The comparison covers the same models evaluated at different times.Source: §1 · p. 2
  • Mitigation benchmark results published as recently as a few months before the study were affected by rescoring with a newer Perspective API version.This creates reproducibility challenges for papers that inherit scores to compare techniques.Source: §1 · p. 2
  • The authors recommend rescoring experimental text and clearer communication of changes to commercial black-box APIs.Source: §1 · p. 2

2 Methodology

The study rescores previously generated text to examine how Perspective API changes affect reproducibility across datasets, a living benchmark, and toxicity-mitigation research. It describes the RTP data and model-evaluation protocol used for these comparisons.

2 Methodology

  • Figure 1 compares HELM rankings before and after rescoring for Toxic Fraction and Expected Maximum Toxicity; rankings change for both metrics.Source: Figure 1 · p. 3
  • The study examines reproducibility of values, findings, and conclusions across RTP statistics, living-benchmark model comparisons, and toxicity-mitigation findings.Source: §2 · pp. 2–3

2.1 RealToxicityPrompts (RTP)

  • RTP contains 100K OpenWebText sentences sampled equally across four toxicity ranges, then split into prompts and continuations for rescoring.A sequence is deemed toxic when its Perspective API Toxicity score exceeds 0.53.Source: §2.1 · pp. 3–4

2.2 Evaluating Model Toxicity

  • Model toxicity is evaluated by generating continuations from prompts, scoring them with Perspective API, and separating results by prompt toxicity.Source: §2.2 · p. 4
  • The evaluation reports Expected Maximum Toxicity, Toxicity Probability, and Toxic Fraction across published-score, rescored-generation, and fully rescored scenarios.Expected Maximum Toxicity and Toxicity Probability aggregate over 25 generations; Toxic Fraction is the fraction of generated instances classified as toxic.Source: §2.2 · p. 4

3 Results

Rescoring historical Perspective API outputs changed RTP toxicity distributions and altered model rankings and toxicity-mitigation results. These shifts challenge like-for-like comparisons across studies using inherited scores.

3.1 RealToxicityPrompts Distribution Changes

  • 49%: rescoring reduced toxic RTP prompts from about 22K to around 11K, while average toxicity scores fell 34%.The published dataset was scored before September 2020 and was rescored in February 2023.Source: §3.1 · p. 4, Table 1 · p. 5
  • From 25K to around 5K: the 0.75–1.0 toxicity bin shrank, while the least-toxic bin grew from 25K to 48K sequences.The published dataset had 25K samples in each bin; these counts cover joint prompts and continuations.Source: Table 2 · p. 5, §3.1 · p. 5
  • 0.189 and 0.153: ‘threat’ and ‘severe toxicity’ had the largest distribution shifts; the ‘toxicity’ distance was 0.097.The study measured differences between published and current attribute distributions using Wasserstein distances.Source: Figure 2 · p. 5, §3.1 · pp. 5–6

3.2 Impact of API Changes on Rankings of Model Risk

  • Rescoring generations alone lowered metrics for toxic and non-toxic prompts, but rescoring prompts too raised them, especially for toxic prompts.The three scenarios use the same prompt and generation texts, but the generations-only scenario mixes toxicity definitions for prompts and generations.Source: Figure 3 · p. 6, §3.2 · p. 6

3.2.1 Impact on Living Benchmarks

  • 13 models’ Toxic Fraction values changed, producing 24 rank changes; openai_text-curie-001 rose 12 places, from 35th to 23rd.The table reports 24 Toxic Fraction rank changes although only 13 models’ metric values changed.Source: §3.2.1 · p. 7, Table 3 · p. 8

3.3 Impact on API Changes on Reproducibility of Research Contributions

  • 33.2% to 23.6%: UDDIA’s EMT fell after rescoring, while UDDIA and DAPT shifted in opposite perceived-toxicity directions.The models were evaluated on 10K non-toxic prompts; UDDIA was perceived as more toxic and DAPT as less toxic than at release.Source: Figure 4 · p. 8, §3.3 · p. 9

4 Recommendations

The recommendations aim to improve reproducibility and confidence in large-scale evaluations using black-box APIs. They call for consistent API versioning, release of scoring materials, rescoring when comparing studies, and controls in living benchmarks.

4 Recommendations

  • Effective recommendations also require community collaboration and awareness of evaluation limitations.Source: §4 · p. 9
  • API maintainers should version models and consistently notify users of updates.The Perspective API announces some changes through a Google group, but says it cannot notify users of every model update and scores may change without notice.Source: §4 · p. 9
  • Authors should release model generations, toxicity scores, and code when possible, and record the date each model was scored.Source: §4 · p. 9
  • When comparing new mitigation techniques with prior work, rescore open-sourced generations and treat unreleased generations as having outdated, unsafe-to-compare scores.Source: §4 · p. 9
  • For living benchmarks, rescore a control set whenever a model is added and rescore previous models if its toxicity metrics change.If access restrictions prevent rescoring, note the results are outdated or remove them from that benchmark version.Source: §4 · p. 9

5 Related Work

Related work frames reproducibility as spanning conclusions, findings, and measured values, while noting that toxicity evaluation involves both human and automated trade-offs. It also reviews categories of language-model toxicity mitigation techniques.

5 Related Work

  • Undisclosed, uncontrollable black-box API versions can make measured values and findings incomparable, bias conclusions, and prevent repeatable results across time.The paper adopts three reproducibility dimensions: conclusions, findings, and measured or calculated values.Source: §5 · p. 10
  • Human toxicity annotation is costly, difficult to scale, and can harm annotators, while automated tools are low-cost but may score minority-community language higher and vary across paraphrases.Annotation quality also varies with annotator expertise and whether annotators belong to the targeted group; even experts can produce biased annotations.Source: §5 · p. 10
  • The reviewed mitigation methods include decoding-time distribution control, pretraining-data filtering, and domain-adaptive fine-tuning.Examples include UDDIA, DExperts, PPLM, and GeDi, which use different mechanisms to reduce toxic generation.Source: §5 · p. 10

6 Conclusion

The conclusion warns that mixing outdated and fresh toxicity scores prevents fair comparisons over time and can bias conclusions. It urges caution because the Perspective API neither announces all updates nor allows calls to earlier versions.

6 Conclusion

  • Using outdated and fresh scores together prevents fair comparisons of toxicity mitigation techniques over time and can lead authors to biased conclusions.Source: §6 · p. 11
  • Changes in the Perspective API affected published UDDIA results and the living HELM benchmark, which has added models and benchmarked them at different times.Source: §6 · p. 11
  • Because Perspective does not announce all updates or allow calls using previous versions, the authors urge caution when directly comparing their results with other work.The study depended on publicly released continuations and author collaboration.Source: §6 · p. 11

A Perspective API

The Perspective API uses machine-learning models to estimate the perceived impact of text on a conversation. This paper focuses on its toxicity attribute, which is returned as a probability score.

A Perspective API

  • Given a comment or other text, the Perspective API predicts its perceived impact on a conversation.Source: A Perspective API · p. 14
  • The API returns probability scores for attributes such as toxicity, insult, and profanity.Source: A Perspective API · p. 14
  • The paper focuses on toxicity, defined as a rude, disrespectful, or unreasonable comment likely to make someone leave a discussion.Source: A Perspective API · p. 14

B Qualitative evaluation of RTP prompts

Rescoring RTP prompts changed toxicity labels in both directions, with far more Toxic-to-Non-Toxic shifts. The largest score changes were associated with different kinds of language in each direction.

B Qualitative evaluation of RTP prompts

C UDDIA Rescored Results

Normalized UDDIA results show that toxicity-metric variation across API versions was not consistent across models. The reported pattern differs between EMT and TP.

C UDDIA Rescored Results

  • Toxicity metrics’ rates of variation across Perspective API versions were not consistent across models.Figure 6 shows normalized results, with line slopes indicating variation.Source: Figure 6 · p. 17
  • For EMT, UDDIA was perceived as more toxic and DAPT as less toxic than when released, while other baseline models were constant.Source: Figure 6 · p. 17
  • For TP, DAPT’s perceived toxicity variation was more pronounced than that of other models.Source: Figure 6 · p. 17

D HELM

The HELM analysis compares its v0.2.2 toxicity results with rescored generations and with rescored generations and prompts using the April 2023 Perspective API.

D HELM

  • Table 5 presents HELM v0.2.2 results alongside rescored versions of the same generations.Source: D HELM · p. 17
  • The rescoring compares both Rescored Generations (RG) and Rescored Generations and Prompts (RGP).Source: D HELM · p. 17
  • The benchmarked model names were scraped from HELM’s website and used to download prompts, continuations, and metric results.Source: D HELM · p. 17

D.1 Toxicity Measurement Details

HELM’s toxicity measurements differ from the regular RTP protocol, so its results are not comparable with research that closely followed RTP. The section identifies differences in generation, scoring, and toxic-fraction calculation.

D.1 Toxicity Measurement Details

  • HELM results are not comparable to research that closely followed RTP because their toxicity measurement protocols differ.Source: §D.1 · p. 17
  • HELM generates 5 continuations per prompt, compared with 25 in the RTP framework.Source: §D.1 · p. 17
  • HELM’s default benchmark API requests use top-p and top-k values of 1, and some models may generate five continuations that are most likely the same.Source: §D.1 · p. 17
  • HELM counts textually identical toxic completions as one unique instance but divides the toxic fraction by five total instances.Source: §D.1 · p. 17
  • RTP uses Perspective’s summaryScore for the whole sequence, whereas HELM uses the first spanScore for API-detected relevant sections.Source: §D.1 · p. 18
Loading 2304.12397v1…