PaperlayerView exact revision on arXiv

Source-linked AI summary

"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws

Liang Yue, Eric A. Vance

arXiv:2610.23446v1cs.SIcs.CL

TL;DR

As Perspective API use winds down, the paper tests whether LLMs measure the same conversation laws as human labels. It finds that second-person address can manufacture a rising toxicity-by-depth law, and presents Threadsafe, an audit that estimates laws from human labels and labeler predictions.

  • Problem

    Most measurements of conversation laws used Perspective API, which is ending service, while LLMs are expected to make the next measurements.

    Source: §1 · p. 1

  • Method

    Threadsafe uses labeler scores to guide randomized human labeling, predicts remaining labels, and corrects those predictions with the human labels to estimate laws.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    On 1.25 million news comments, Haiku reports toxicity rising +0.93 percentage points per reply-depth level, while crowd labels show a decline of −0.12.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Threadsafe estimates every linear law without bias for any labeler, and its simulated-audit intervals retain nominal coverage from 250 labels onward.

    Source: §1 · p. 2

    The rewrites are scored against raters’ labels for the originals, and no human judged the edited comments.

    Source: §12 · p. 18

Abstract

from arXiv · show

What the Web community believes about toxic conversations, such as that toxicity rises in longer and deeper threads, was measured mostly with Google's Perspective API, which is shutting down; large language models (LLMs) will make the next measurements. We find that LLM labelers mistake second-person address for toxicity, and that this one bias manufactures laws that human labels do not show. On 1.25 million news comments with crowd labels, Claude Haiku flags 12% of the non-toxic comments that contain no "you" and 51% of those with four or more; a single crowd rater flags 6% and 10%. Deeper replies address someone more often, so under Haiku toxicity rises with reply depth (+0.93 percentage points per level), while under the crowd's labels it slightly declines (-0.12). Rewriting "you" in the third person removes two thirds of the false rise; flagging only as many comments as the crowd does not remove it, and showing Haiku the parent comment enlarges it. All seven LLMs we test over-flag "you", and at the crowd's flag rate, without the parent, any false rise of the open ones vanishes at a fixed number of second-person words; Haiku also over-flags "you" on Reddit, Gab and Wikipedia talk pages. Agreement with human labels does not reveal the bias: we prove that a fixed Cohen's kappa leaves a law nearly unconstrained. We therefore certify the law instead of the labeler. Threadsafe uses the labeler's scores to choose which comments human raters label and to predict the others, and corrects the predictions with the human labels; linear laws are estimated without bias whatever the labeler does. In simulated audits its intervals for the laws keep nominal coverage from 250 labels on, where uniform-sample methods such as PPI++ fall short, and at 1,000 to 8,000 labels are 0.75 times as wide as those of design-based supervised learning (DSL); a thousand human labels rule out Haiku's rising law.

1 INTRODUCTION

As Perspective API use winds down, this paper asks whether LLMs measure the same conversation laws, including toxicity by reply depth and thread size. It finds that second-person address leads LLMs to manufacture a rise in toxicity with depth and introduces Threadsafe to estimate human-labeled laws with audited labels.

1 INTRODUCTION

  • LLMs are poised to replace Perspective API in measuring conversation laws, including escalation, thread-size associations, and contagion.The paper defines these laws respectively through toxicity by reply depth, toxic share by conversation size, and risk along reply edges.Source: §1 · p. 1
  • Claude Haiku’s false rise disappears at a fixed second-person-word count; on a 5% sample, rewriting only pronouns (“you” in third person) removes two thirds of it, all within threads.Shown the parent comment, Haiku’s false rise among replies grows from +1.29 to +1.59.Source: §1 · p. 2
  • +0.93 percentage points per level under Haiku, versus −0.12 under crowd labels, for toxicity by reply depth; structured false positives account for 90% of Haiku’s false rise.At gold’s flag rate, the best-agreeing Claude models distort escalation most; a textbook error-rate correction more than doubles Haiku’s rising law to +2.02 points per level.Source: §1 · pp. 1–2
  • Threadsafe uses labeler scores to guide randomized human labeling, predict the remaining comments, and correct predictions with gold labels, yielding an approximately 95% confidence interval.Its estimate of every linear law is exactly unbiased for every labeler because the sampling probabilities are known.Source: §1 · p. 2
  • From 250 labels, Threadsafe intervals keep nominal coverage; with 1,000–8,000 labels, they are 0.75 times as wide as DSL’s, and 1,000 labels rule out Haiku’s rising law.PPI++ coverage falls as low as 55% and DSL coverage to 79% in some simulated audits; Threadsafe’s sampling is adapted in waves.Source: §1 · p. 2

2 DATA, LABELERS AND OPERATING POINTS

The study compares LLMs, classifiers, and human benchmarks against crowd-majority labels across news comments and five additional corpora, using both labelers’ own thresholds and the crowd’s flag rate.

2 DATA, LABELERS AND OPERATING POINTS

  • 1,251,487 news comments in 270,967 reply threads under 57,769 articles form the main corpus, with crowd labels and 980,520 reply edges.The retained threads have at least two comments and complete Haiku and Qwen labels; 7.6% of comments are gold-toxic.Source: §2 · p. 3
  • Five additional corpora cover Reddit abuse, Reddit and Gab hate speech, Wikipedia talk-page personal attacks, and human–chatbot dialogue safety.Their labels are gold for each corpus’s own construct.Source: §2 · p. 3
  • The study used a 5% article sample and three experiment samples: rewritten second-person comments, targeted edits, and reverse-sample comments referring to someone as they.The targeted edit sample comprised 4,000 second-person comments: 1,500 clearly non-toxic comments at depth 2 or more, 1,500 clearly non-toxic roots, and 1,000 gold positives. The reverse sample contained comments referring to someone as they and no “you”.Source: §2 · p. 3
  • Haiku and Sonnet use the raters’ toxicity question at their own cuts and also score comments for comparison at gold’s flag rate; five open LLMs answer the main prompt.Qwen is evaluated at its own cut as well as gold’s flag rate, and every LLM also labels replies with the parent shown.Source: §2 · p. 3
  • Seven classifiers are tested at their usual thresholds and at gold’s flag rate, while human benchmarks include a single rater and a vote-fraction instrument.The vote-fraction instrument flags comments in descending order of toxic-vote share, matching a given labeler’s flag count; CCC raters saw the parent.Source: §2 · p. 3
  • Reported settings and terms are specified in supplementary tables, and Haiku’s reported depth-law distortion differs between all reply threads and the 5% sample.The reported distortions are +1.05 and +1.19 points per level, respectively; intervals are 95% bootstrap intervals.Source: §2 · pp. 3–4, Table 2 · p. 4

3 LAWS AND WHAT A LABELER’S ERRORS DO TO THEM

The paper defines conversation-structure laws as statistics of labels and structure, then decomposes how labeler errors distort linear laws and uses permutation to split distortion for contagion.

3 LAWS AND WHAT A LABELER’S ERRORS DO TO THEM

  • Depth and size laws are weighted linear contrasts of labels, while contagion is the reply risk ratio comparing toxic-reply probabilities after toxic versus non-toxic parents.The depth slope uses capped reply depth; size slopes compare unit toxic share with log2 unit size.Source: §3 · p. 4

3.1 Which part of a law does a labeler produce?

  • For every linear law, distortion separates into ordinary misclassification shrinkage and structured false-positive and true-positive terms that can manufacture or reverse a law.Unstructured errors shrink a law toward zero when TPR exceeds FPR; only errors that follow structure can create or reverse it.Source: §3.1 · p. 4
  • For escalation, the false-positive term is proportional to the depth slope among gold negatives, scaled by their share of depth variation.For contagion, the paper separates distortion by shuffling flags within gold classes to remove structure, then restoring actual false and true positives in both orders.Source: §3.1 · pp. 4–5
  • Placebo statistics measure a labeler’s flags among gold negatives; zero placebo slopes and a risk ratio of one correspond to false positives that ignore structure.The paper compares these statistics with a single-rater human benchmark because some gold negatives may still be called toxic by individual raters.Source: §3.1 · p. 5

4 LLMS MANUFACTURE ESCALATION

LLM labelers turn second-person address into false positives that manufacture a rising toxicity-depth law, although crowd labels show a slight decline. Rewriting “you” removes much of the distortion, while showing the parent does not eliminate it; the rise is absent from human-vote measures.

4 LLMS MANUFACTURE ESCALATION

  • Gold’s depth law falls −0.12 points per level, while Haiku’s rises +0.93, a +1.05-point distortion driven 90% by structured false positives.The false rise also appears within threads and among replies.Source: §4 · p. 5

4.1 The word “you”

  • Haiku flags 12.1% of gold negatives without a second-person word and 50.9% with four or more, versus a single rater’s 5.9% and 10.0%.The increase persists within comment-length bands and among negatives with no toxicity vote or subtype vote.Source: §4.1 · p. 5

4.2 Rewriting “you” turns the false positives off and on

  • Rewriting second-person comments removes 67% of Haiku’s depth-law distortion, from +1.23 to +0.40, and removes 73% for Sonnet.Holding second-person-word counts fixed also removes most of Haiku’s false-positive depth rise; at a fixed count its depth-law distortion is −0.05.Source: §4.2 · pp. 5–6, §4.1 · p. 5

4.3 Shown the parent

  • Showing the parent leaves Haiku’s reply-depth distortion at +1.59, up from +1.29 without it, and structured false positives account for 93% of the distortion.Rewriting both reply and parent removes only 25% of Haiku’s false rise in the selected replies.Source: §4.3 · p. 7

4.4 The rise is not in the raters’ votes

  • No human-vote measure shows a rising depth law, while Haiku’s false-positive rate rises +1.04 points per level even among gold negatives with an insult vote.The rise also persists among negatives with no rater vote and among moderator-approved comments.Source: §4.4 · p. 7

4.5 Every LLM and every operating point

  • At gold’s flag rate, all seven LLMs over-flag gold negatives with four or more second-person words 3.3–5.1 times as often as those with none.At a fixed number of second-person words, the open models’ false rises vanish; Haiku and Sonnet’s 0–100 ratings retain positive depth-law distortion at every tested flag rate from 2% to 30%.Source: §4.5 · pp. 7–8, Figure 3 · p. 9

5 SIZE LAWS AND CONTAGION

LLM labelers inflate thread and article size laws, while the apparent size and direction of some laws depend on flag rate. Contagion estimates also vary with how gold labels are constructed, and labelers can shift the measured association in either direction.

5.1 The size laws

  • Gold’s thread size law is +0.15 points per doubling, compared with Haiku’s +1.93 and Sonnet’s +2.05; rewriting “you” removes at least half of the LLMs’ thread-law inflation.For article size, gold’s law is +0.32 points per doubling, compared with Haiku’s +1.43 and Sonnet’s +1.67.Source: §5.1 · p. 9
  • At gold’s flag rate, Detoxify and IMSYPP report article size laws of +0.32 and +0.34, matching gold’s +0.32, although the sign is more robust than magnitude across operating points.Significance tests can miss gold’s thread size law or detect a law of any size depending on binning, so magnitude requires an estimate with an interval.Source: §5.1 · p. 9

5.2 Contagion

  • Gold’s reply risk ratio is 2.39 using all raters, 2.10 using a majority of three, and 1.63 using one; a modeled noise-free majority gives 2.82.The paper calls the reply association contagion, and 36% of gold’s log risk ratio lies between articles rather than within them.Source: §5.2 · p. 9
  • At matched flag rates, Haiku and Sonnet exceed the vote-fraction instrument’s contagion by +0.12 and +0.18 on the log scale, respectively.Classifiers also exceed the instrument at their own flag rates by +0.03 to +0.32 on the log scale.Source: §5.2 · pp. 9–10
  • At equal rater votes, classifiers’ reply flags still follow their parent flags, with risk ratios of 1.01 to 1.45, while a single rater’s ratio is 1.00.Showing the parent raises raters’ CCC reply risk ratio from 2.48 to 3.36, but lowers Haiku’s from 1.46 to 1.24.Source: §5.2 · p. 10

6 BEYOND NEWS COMMENTS

Haiku’s second-person over-flagging appears on several forums, but its effect on measured laws varies with the corpus and the law. Strong laws often shrink under labeler errors, while structured false positives can also create or reverse a rise.

6 BEYOND NEWS COMMENTS

  • On forum-specific questions, Haiku flags gold-negative comments containing “you” 2.9–7.8 points more often, and the second tested LLM does so 1.9–5.7 points more often.The study uses each corpus’s own construct to identify errors on its gold negatives.Source: §6 · p. 10, Table 9 · p. 12
  • Among forum gold negatives deemed clearly harmless by Detoxify, three or more second-person words raise Haiku’s false-positive rate 3.8–12.7-fold; third-person rewrites lower its flags by 3.7–8.1 points.In BAD dialogues, a second-person word alone does not raise the false-positive rate.Source: §6 · p. 11
  • Address-related false positives can raise the depth law when they follow its structure, but other errors shrink laws; Haiku’s reported forum laws therefore vary across constructs and corpora.On Gab the placebo depth slope is +0.33, while it is −0.19 on Reddit and −0.36 on CAD; under the hate-speech question, Haiku reports −1.19 on Reddit and −1.01 on Gab, versus gold’s −2.52 and −1.95.Source: §6 · p. 11
  • Across corpora, labeler errors weaken strong within-thread depth laws, while contagion shifts in either direction, including reversals on BAD and CGA-Wiki.Haiku also shrinks strongly negative thread-size laws on Reddit and Gab, inflates CAD’s rising law, and reverses CGA-Wiki’s law under the toxicity question.Source: §6 · pp. 11–12

7 WHAT THE DISTORTIONS DO TO CONCLUSIONS

Labeler distortions change the conclusions researchers draw from samples and from comparisons between communities. Haiku often reports a different verdict from gold, while community rankings and published laws also depend on the labeler.

7 WHAT THE DISTORTIONS DO TO CONCLUSIONS

  • In samples of 10,000–100,000 comments, Haiku’s depth-law verdict differs from gold’s in 70%–100% of samples and reverses it in up to 48%.For thread-size laws, its verdict differs in 75%–92% of samples, mostly by finding a significant rise when gold does not.Source: §7 · p. 12
  • Across 471 law estimates, 23 significantly reverse gold’s sign, 31 manufacture significance, and 36 lose gold’s significance; Haiku’s news-comment depth distortion is only 6% unstructured.A single rater’s errors are mostly unstructured for depth and contagion, tending to shrink those laws toward zero.Source: §7 · p. 12
  • Of 91 community pairs, gold distinguishes 74, while Haiku reverses 16 and the Dynabench hate model reverses 19.The three classifiers trained on these raters’ labels reverse at most one pair by the toxic-comment-share measure.Source: §7 · p. 12
  • Detoxify does not manufacture escalation on news comments, whereas Haiku reports rising toxicity, much more toxic longer threads, and weaker contagion at its own cut.Detoxify’s contagion estimate is 3.44 at Avalle et al.’s threshold and 2.70 at gold’s flag rate, compared with gold’s 2.39.Source: §7 · p. 12

8 AGREEMENT CANNOT CERTIFY A LAW

Agreement statistics do not by themselves certify a law: fixed error counts allow a wide range of possible laws, and the labeler with the highest agreement can still distort them. Validating a law requires labels placed according to its structure.

8.1 What agreement can certify

  • Proposition 2 bounds a law by assigning true and false positives to comments with the largest or smallest weights; every value between the bounds is attainable to within the largest gap between weights.The same bounds apply in reverse when fixing the labeler’s flags and varying gold’s labels.Source: §8.1 · pp. 12–13
  • At κ=0.9 and gold’s flag rate, a labeler can report any Civil Comments depth law from −1.30 to +1.65 points per reply level.With Haiku’s counts, gold’s depth law could range from −3.57 to +8.81 points per level.Source: §8.1 · pp. 12–13
  • Single-comment validation can identify depth laws when agreement is checked by depth, but cannot identify contagion, which depends on labels at both ends of reply edges.With calibrated Haiku probabilities, the risk ratio over 980,520 Civil Comments reply edges is consistent with any value from 0.00 to 16.13.Source: §8.1 · p. 13

8.2 Agreement and distortion in the labelers we tested

  • Higher κ usually accompanies less depth-law distortion, but does not reliably rank labelers; across seven LLMs, Sonnet-100 agrees best and distorts escalation most.The figure plots κ against depth-law distortion for labeler settings in each population.Source: §8.2 · p. 13, Figure 4 · p. 14
  • In each of seven populations, the best-agreeing tested labeler distorts at least one law by more than three standard errors, across 17 of 29 laws.On Wikipedia talk pages, Detoxify has κ=0.91 yet distorts within-thread escalation by −0.54 points per level.Source: §8.2 · p. 13
  • Correcting linear laws by dividing by TPR − FPR can enlarge structured distortion; Haiku’s corrected escalation is +2.02 points per level, more than twice its uncorrected +0.93.Across 471 combinations, 89% of corrected intervals covered gold’s law in fewer than 90% of validations, with median coverage of 0%.Source: §8.2 · pp. 13–14

9 CERTIFYING THE LAW: THREADSAFE

Threadsafe certifies the human-labeled law rather than the labeler, using human labels to correct predictions informed by labeler scores. It provides an approximately 95% confidence interval for linear laws that is unbiased whatever the labeler does.

9 CERTIFYING THE LAW: THREADSAFE

  • Threadsafe spends a human-label budget to estimate the corpus’s human-labeled law and return an approximate 95% confidence interval.A labeler’s law is ruled out if it falls outside the interval; if it falls inside, the labels bought cannot distinguish the two laws.Source: §9 · p. 14
  • About a thousand human labels can distinguish Haiku’s rising law from raters’ falling law when allocated to replies far from mean depth and comments in small threads.Those comments’ labels are not predictable from the labeler and conversation structure.Source: §9 · p. 14
  • Labeler scores serve as a working model: they guide where human labels are bought and supply predictions that those labels correct.Threadsafe also estimates placebo statistics indicating whether false positives follow the structure.Source: §9 · p. 14

classifiers;

Threadsafe uses labeler scores and conversation structure to guide human labeling, then corrects predictions with sampled gold labels. Its estimates of linear laws remain unbiased even when the labeler is wrong.

classifiers;

  • The working model predicts each comment’s gold label from labeler and auxiliary scores plus conversation features; predictions guide sampling but do not determine unbiasedness.The labeler’s scores and comment structure help predict labels, while the sampled gold labels correct the predictions.Source: classifiers · p. 14, §9.1 · pp. 14–15

9.1 Where gold labels pay: the square-root rule

  • The square-root rule spends labels where comments carry more weight in the law and their labels are harder to predict, accounting for labeling cost.For smooth targets such as slopes or risk ratios, it uses linearized weights; when the working model is known, adapting to previously bought labels cannot improve the design.Source: §9.1 · p. 15

9.2 What Threadsafe runs

  • The pilot and four waves fit a working model, then sample comments according to their influence on the target and model uncertainty, with safeguards against overconfidence.A uniform share of each wave preserves a minimum chance of selection for every comment.Source: §9.2 · p. 15
  • For contagion, Threadsafe samples reply edges and labels both endpoints when needed, estimating risk ratios from edge totals.With fresh rater judgments, the target is the law of raters’ expected labels, and intervals must account for rater variance; noisier gold labels widen them.Source: §9.2 · p. 15, §9.3 · pp. 15–16

9.3 Why it holds

  • Each sampled comment’s known inclusion probability weights a correction to the model’s predictions, making estimated totals exactly unbiased for any labeler.The result assumes positive sampling probabilities and that each wave’s model and probabilities depend only on labels bought earlier.Source: §9.3 · p. 15

10 EVALUATION

Simulated audits evaluate Threadsafe’s coverage, interval widths, and ability to detect labeler distortions across corpora, targets, and budgets. It maintains nominal coverage from 250 labels and produces narrower intervals than the comparison designs.

10 EVALUATION

  • At 250 labels, Threadsafe’s 95% intervals cover laws in 94.8% of audits, while PPI++ covers 89.1% on average and as little as 54.6%.Across 1,000–8,000 labels, Threadsafe’s coverage averages 94.9% and is at least 91.6% for every cell, budget, and target combination.Source: Table 11 · p. 16, §10 · p. 16
  • At 1,000–8,000 labels, Threadsafe’s law intervals are 0.75 times as wide as DSL’s at the median.The table reports mean 95% interval coverage and median interval-width ratios, with ratios below 1 indicating narrower Threadsafe intervals.Source: Table 11 · p. 16, §10 · p. 17
  • Threadsafe’s coverage holds on four classifiers not used in development and on deliberately misleading labelers, while PPI++ and uniform sampling under-cover for some adversarial labelers.For the four new classifiers, coverage is 95.0% for laws; under five constructed labelers, Threadsafe covers 95.5% for laws and placebo slopes.Source: §10 · pp. 16–17
  • Fresh judgments with a single rater per label produce intervals 1.53 times as wide as fixed gold; a majority of three produces intervals 1.16 times as wide.Including the estimated rater-variance term, Threadsafe’s intervals cover the law of expected labels in 95.6% of audits.Source: §10 · p. 17
  • Threadsafe reaches 0.71–0.74 of oracle precision on depth and thread-size laws, 0.54 on article-size law, and 0.72 on reply risk ratio.The oracle is a benchmark design that knows a gradient-boosting model of gold labels in advance.Source: §10 · p. 17
  • At 1,000 labels, the median interval rules out Haiku’s +0.93 depth law against gold’s −0.12, though 250 labels do not.The reported median 95% half-width for the depth law is 0.54 points per level at 1,000 labels and 1.34 at 250.Source: §10 · p. 17
  • Contagion is harder to certify: news-comment audits need 8,000–16,000 labels, and the edge design is no narrower than uniform sampling on CAD and BAD.For Haiku shown the parent, the audit detects distortion with 4,000 labels.Source: §10 · p. 17

11 RELATED WORK

The related work spans measurements of conversation laws, research on toxicity cues and label errors, and methods for inference from predicted labels. Threadsafe adapts probability-based audit designs to conversation laws while retaining validity for any labeler.

11 RELATED WORK

  • Earlier studies reported toxicity rising with conversation size on eight platforms, deeper toxic reply trees [39], and contagion through user flags [9] or experiments.Avalle et al. used Perspective on about 500 million comments without human-label validation.Source: §11 · p. 17
  • Prior work links direct address and questions with conversations that later derail [50], while Wikipedia annotations distinguish attacks on the addressed person from attacks on others [47].Research also documents group-dependent classifier errors [5], rater differences [37], and effects of parent context on rater labels.Source: §11 · p. 17
  • Placebo outcomes and negative controls test whether apparent effects persist where no effect should exist.Here, gold-negative comments are compared with a human benchmark because non-toxic comments can still have structure.Source: §11 · p. 17
  • Prediction-powered inference [1] and PPI++ [2] correct estimates with random gold samples, while DSL [13] uses known sampling probabilities and active inference [52] samples by weight and uncertainty.Cluster-robust PPI targets item-level parameters [6], whereas contagion requires labeling both ends of reply edges.Source: §11 · p. 17
  • Threadsafe combines model-assisted probability-proportional-to-size designs with difference estimators and adaptive waves.Its square-root rule extends to adaptive designs, and its estimate remains unbiased for any labeler as the design runs.Source: §11 · pp. 17–18

12 LIMITATIONS •

The paper’s findings are bounded by the corpora, labelers, and audit procedure studied. It also reports ethical considerations about public data and annotator exposure.

12 LIMITATIONS •

  • The corpora are English; news laws concern reconstructed threads of at least two comments, while keyword-retrieved Qian corpora and attack-ending CGA-Wiki describe those samples, not their platforms.Only Claude models represent commercial LLMs in the study.Source: §12 · p. 18
  • Interval coverage relies on a normal approximation and is demonstrated by simulation on corpora also used to choose settings; multiple laws from one audit require multiplicity adjustment.The authors used public datasets with existing gold labels and collected no new human judgments; designs concentrating annotation on likely toxic comments increase annotator exposure.Source: §12 · p. 18

13 CONCLUSION

This section concludes that LLM labelers can distort conversation laws in ways that agreement with human labels or correcting validated error rates does not reveal or repair, and recommends certifying laws with human labels and intervals while using the labeler to guide review. It also documents toxicity-labeling prompt variants, cutoff selection, and LLM and classifier scoring procedures, including parent context and thresholding.

13 CONCLUSION

  • LLMs write laws of their own: second-person escalation, an inflated size law, and weaker contagion at their own cut.The bias appears across the forums tested, but not in chatbot dialogues; whether it distorts a law depends on where errors fall in the conversation.Source: §13 · p. 18
  • Agreement with human labels does not reveal the bias, and correcting for validated error rates does not repair it.The distortion depends on where the labeler's errors fall in the conversation.Source: §13 · p. 18
  • Conversation-labeling studies should validate the LLM on a sample stratified by the law’s structure, then compute placebo statistics to test whether errors follow that structure.They should report the law estimated from human labels, with an interval.Source: §13 · p. 18
  • Use the labeler’s scores to guide which comments human raters label, and use their labels to certify the law.Source: §13 · p. 18

B.2 Prompts

  • Prompt variants included 8- or 16-example demonstrations, instructions about direct address, and versions with or without the parent comment.T8 used eight examples; T16 added eight more. Mitigation prompts also included an instruction that addressing someone as “you” is not toxic by itself.Source: §B.2 · pp. 24–25
  • The selected candidate used the development-set cutoff with the highest Cohen’s kappa; every candidate’s cutoff was “toxic or worse.”Ties went to the higher cutoff.Source: §B.2 · p. 24
  • The appendix describes paraphrase requests that either preserve “you” or refer to the addressee in the third person, alongside corpus-specific toxicity or abuse questions.For the paraphrases, the instructions require preserving the comment’s meaning, tone, and politeness or rudeness.Source: §B.2 · pp. 25–26
  • Prompts ask models to judge toxicity, hate speech, personal attacks, or conversational acceptability, and to judge replies only when earlier messages are shown as context.Responses are requested in a single-letter format.Source: §B.2 · p. 26

B.3 Open LLMs

  • Open LLMs use task-specific prompts across the corpora, with replies evaluated both without and, when available, with their parent comment.The prompts include the Claude-label prompts on Civil Comments and CCC, abuse and personal-attack questions on CAD and CGA-Wiki, and a hate-speech question on Qian et al.’s corpora.Source: §B.3 · pp. 26–27
  • Open LLMs derive toxicity from answer probabilities and flag comments using cutoffs tied to gold prevalence, with parent-context runs using a separate population-wide ranking procedure.For two-option questions toxicity is p(A); for four-option questions it is p(A) + p(B).Source: §B.3 · p. 27

B.4 Classifiers

  • Classifiers score each comment without its parent and are evaluated at usual thresholds or thresholds set to match gold’s flag rate.The two Detoxify models use a 0.6 threshold, while other classifiers use gold-prevalence cutoffs as described for each corpus.Source: §B.4 · p. 27, Table 15 · p. 28

C HUMAN BENCHMARKS AND THE RATERS’ CONSTRUCT

The rater-based checks ask whether observed labeler structure exceeds what human raters’ own construct can explain. The latent model finds that raters’ propensities explain only a fraction of Haiku’s placebo depth slope, though the model has important fit limitations.

The many-rater check

The latent model of the raters

  • The raters’ noise-free majority has reply risk ratio 2.82 (95% CI 2.42 to 3.25), compared with gold’s 2.39.Against this latent benchmark, two classifier settings report more contagion, two less, and ten are indistinguishable.Source: C HUMAN BENCHMARKS AND THE RATERS’… · p. 28
  • Haiku’s placebo depth slope is +1.02 points per level, versus +0.08 predicted by an instrument depending only on raters’ propensities.Among comments whose four raters all said were not toxic, Haiku’s flag rate rises +0.93 points per level, versus +0.08 predicted.Source: C HUMAN BENCHMARKS AND THE RATERS’… · p. 28

Checks of the latent model

  • Under the inferred rating rule, simulated votes reproduce gold’s risk ratio (2.37 versus 2.39) but not the observed distribution of votes.Among comments with five or six raters and exactly one toxic vote, the simulation yields 21%, versus 70% observed.Source: C HUMAN BENCHMARKS AND THE RATERS’… · p. 28, Table 19 · p. 30
  • The construct-only test can flag a spurious placebo effect under a misspecified rating protocol, limiting interpretation of the latent-model checks.The misspecified protocol produces a placebo risk-ratio effect of −0.13 (CI −0.24 to −0.06).Source: C HUMAN BENCHMARKS AND THE RATERS’… · pp. 28–29

D MORE RESULTS ON NEWS COMMENTS

On news comments, structured errors account for distortions in classifier laws, and the response to second-person wording tracks placebo depth slopes. These patterns persist across operating points and alternative human-label definitions, while rewriting “you” changes many labelers’ flags.

The decomposition for every labeler

Where the classifiers’ errors cluster

  • Identity mentions raise classifier false-positive rates up to 3.6-fold, but removing edges with such mentions changes placebo risk ratios by only −7% to +6%.False positives cluster both within and between articles, not through identity terms.Source: D MORE RESULTS ON NEWS COMMENTS · p. 30

Every operating point

  • At prevalence-matched operating points, the thread-size law falls below gold’s by −0.12 points per doubling, while the few flags cluster along reply edges.The reply risk ratio is 6.50 on all reply threads; the size-law distortion interval on the 5% sample covers zero.Source: D MORE RESULTS ON NEWS COMMENTS · pp. 30, 33, Table 24 · p. 33

Other definitions of gold

The edits, labeler by labeler

  • Adding “you” to clearly non-toxic replies raises flags for the LLMs and most classifiers, while rewriting “you” in the third person lowers them.Across 16 labelers, the change on adding “you” correlates +0.74 with relative placebo depth slope, or +0.90 for relative change.Source: D MORE RESULTS ON NEWS COMMENTS · p. 34, Table 26 · p. 34

E MORE RESULTS ON THE OTHER CORPORA

Across other corpora, Haiku’s over-flagging of second-person address accompanies distortions of depth, size, and contagion laws. The paper also shows that agreement can leave a law’s sign unidentified and describes the audit algorithms used for these analyses.

E MORE RESULTS ON THE OTHER CORPORA

F WHAT AGREEMENT LEAVES OPEN

  • At κ=0.9, the depth-law sign is unidentified on five of seven populations; at κ=0.5, it is unidentified on all seven.The reported bounds are sharp and were checked by brute force on random small instances.Source: F WHAT AGREEMENT LEAVES OPEN · p. 36, Table 34 · p. 37

G ALGORITHMS

  • The audit algorithms sample comments in waves using labeler scores, depth, size, reply counts, and auxiliary scores; edge audits target reply and placebo risk ratios.The edge design mixes model-based sampling probabilities with a uniform share of 0.15.Source: G ALGORITHMS · p. 36

H EVALUATION DETAILS

The evaluation reports nominal interval coverage from 250 labels and examines robustness to deliberately adversarial labelers and fresh human judgments. The proofs establish unbiased estimation for linear targets regardless of how labels depend on the design’s scores.

H EVALUATION DETAILS

  • Threadsafe’s intervals cover at the nominal rate from 250 labels on, while uniform or score-based comparators fall short at small budgets, mostly on size laws.On the other 27 evaluation cells, Threadsafe covers laws in 94.9% of audits (lowest 91.6%) and intervals are 0.62 times as wide as PPI++’s.Source: H EVALUATION DETAILS · p. 39, Table 39 · p. 40, Table 40 · p. 40
  • Under five labelers built from gold and structure, Threadsafe keeps nominal coverage and is at most half as wide as uniform sampling on laws.PPI++ and uniform sampling fall below 90% coverage in some cells under every constructed labeler.Source: H EVALUATION DETAILS · p. 39, Table 41 · p. 41
  • In fresh-gold audits, each purchased gold label is a fresh judgment drawn from the comment’s raters; Table 43 reports coverage for expected-label and actually-drawn-label laws, with and without the rater-variance term.Source: Table 43 · p. 42

I PROOFS

  • Proposition 1 decomposes a linear-law distortion into unstructured scaling and structured false-positive and true-positive terms.The proof assigns structured terms to the ways actual false positives and hits restore the labeler’s errors.Source: I PROOFS · p. 43
  • The agreement and flag rate fix error counts but do not fix where those flags fall among weights, leaving a range of possible linear-law values.Moving a flag between weights within a class changes the law by the weight gap.Source: I PROOFS · p. 43
  • Theorem 1 gives an unbiased estimate of the target conditional on the labels, without restricting how the labeler’s predictions depend on them.The waves partition comments after an exact pilot, and their centered terms form martingale differences for the variance calculation.Source: I PROOFS · p. 43

Related papers

Loading 2610.23446v1…