How User-AI Mistreatment Occurs and Matters in Conversational Systems?

• Research still lacks evidence on the frequency, forms, and measurement of user-directed hostility, which matters for filtering hostile or noisy exchanges from training data.Source: §1 · p. 1
• User-to-AI hostility spans directed affective abuse, coercive misuse, and harmful-content solicitation, which a single detector cannot distinguish reliably.Source: §1 · p. 1
• The audit cross-validates a directed-hostility lexicon and moderation signal on the same turns, treating their disagreement as evidence about distinct constructs.Source: §1 · pp. 1–2
• The detector union marks 4.96% of turns, falling to 0.90% after precision adjustment.Source: §1 · p. 2
• Model-to-model hostility variation is dominated by selection, with first-turn hostility spreading more widely than post-response hostility.Source: §1 · p. 2
• Coercive jailbreak openings front-load the first turn, while affective hostility accumulates across a session.Source: §1 · p. 2

• Studies of abuse toward conversational agents often use small lab corpora and predate LLM assistants.Source: §2 · p. 2
• Adversarial-prompt research treats jailbreaks and policy bypass as security problems, generally bracketing hostility's affective register.Source: §2 · p. 2
• In-the-wild corpora have mostly supported moderation-model development rather than characterization of user-to-AI hostility.Source: §2 · p. 2
• This paper cross-validates affect-directed and content detectors on the same turns and separates within- from between-model effects across 25 models.Source: §2 · p. 2

3.1 Corpus

• The English corpus contains 777,453 evaluation conversations, 1,498,473 user turns, and 25 models.These are evaluation sessions where users probe and compare models, and the dataset supplies uniform per-message moderation outputs.Source: §3.1 · p. 2

3.2 Two detectors

• The directed-hostility lexicon uses eight categories grouped into affective abuse, coercion, and content solicitation, alongside moderation harassment and content flags.A context guard suppresses some long-turn matches; the taxonomy is not independently validated for exhaustiveness or separability.Source: §3.2 · pp. 2–3

3.3 Validation

• Lexicon precision ranges from 82.4% for JBRK and 73.1% for SEXC to 16.5% for DEMD and 11.5% for PROF; corpus-flag reweighting gives 39.5%.The pooled sample precision is 37.6%, but it is not a corpus quantity.Source: §3.3 · p. 3, Table 1 · p. 4
• Only 8.1% of moderation-harassment-flagged, lexicon-negative turns were directed at the assistant; APOLOGY precision was 88.3%, versus 76.7% for REFUSAL and 18.3% for HEDGE.Source: §3.3 · p. 3
• Of genuine apology turns, 56.6% were deflection-only, the distinction used in the discussion of the apology association.Source: §3.3 · p. 4
• The human audit covered only PROF among lexicon categories, while a probe found four missed cases among 29 turns flagged by neither detector, indicating incomplete recall.The union undercounts the broad construct even as it overcounts mistreatment aimed at the assistant.Source: §3.3 · p. 4

• 4.96% of user turns trigger at least one detector; the lexicon flags 2.12%, while moderation flags 3.65% MOD-FLAGGED, including 1.10% MOD-HARASS and 2.58% sexual.After precision adjustment, JBRK (0.35%) and SEXC (0.32%) lead the lexical categories.Source: §4 · p. 4, Figure 1 · p. 5
• The directed lexicon and moderation harassment cluster agree weakly (κ=0.122; Jaccard 0.072), because the lexicon catches coercion while moderation often flags toxic-content solicitation.Only 8.1% of moderation-only turns were directed at the assistant; the figure cross-tabulates the two signals.Source: §4 · p. 4, Figure 1 · p. 5
• 0.90% of user turns is the precision-adjusted estimate of mistreatment aimed at the assistant, versus a 3.00% raw narrower lexicon–moderation-harassment union.The adjusted estimate is 0.94% when restricted to items whose trigger the judge could see.Source: §4 · p. 5

5 Between-Model Variation

• Rates broadly follow alignment tuning, but vicuna-7b is among the least-abused while chat-tuned chatglm-6b is among the most-abused.Source: §5 · p. 5
• 35.8× first-turn variation (0.20–7.03%) exceeds post-response variation (0.34–2.26%); deduplicating opening prompts still leaves a 15.4× spread.This gap indicates that most cross-model variation reflects who each model attracts, before the model responds.Source: §5 · p. 5, Figure 2 · p. 6
• The selection interpretation is limited because session type is untagged and first-turn hostility is only a lower bound on selection.Audience sorting cannot be separated from model-targeted prompt shaping.Source: §5 · pp. 5–6
• 13.5× (95% CI [10.2, 17.7]) separates harassment rates across 24 models, from 0.54% for claude-1 to 7.21% for fastchat-t5-3b.The coefficient of variation across all 24 models is 0.86.Source: §5 · p. 5, Figure 2 · p. 6
• Across models, apology rate is negatively associated with elicited hostility (r=−0.47), though the relationship is descriptive and confounded.The association does not extend to first-turn hostility (r=0.05).Source: §5 · p. 6

6 The Apology Association: Within vs Between

• Adding lagged hostility lowers the apology odds ratios to 1.25 for lexical hostility and 1.64 for moderation harassment, with associations still positive.The lagged outcomes have odds ratios of 26.6 and 44.6, respectively; the remaining moderation association is 1.64 (95% CI [1.50, 1.79]).Source: §6 · p. 6, Table 2 · p. 8
• The apology association remains positive after restricting to non-refused prior turns (lexicon OR 1.62; moderation OR 2.56) and jailbreak-free conversations (OR 1.52).The non-refusal estimates are 1.62 [1.50, 1.75] and 2.56 [2.31, 2.82]; the jailbreak-free estimate is 1.52 [1.41, 1.64].Source: §6 · p. 6
• Lexical hostility follows 3.16% of apology pairs versus 1.66% otherwise, about fifteen additional hostile turns per thousand apologies.Source: §6 · p. 7
• In 20 of 23 models the within-conversation apology odds ratio exceeds 1, with a random-effects pooled OR of 1.79 (I2=61%).The estimate is significant in 13 models; the pooled 95% CI is [1.54, 2.08].Source: Figure 4 · p. 7, §6 · p. 7
• The association reverses across units: apologies precede more hostility within conversations, while more apologetic models receive less hostility across models.Across models, apology rate is negatively associated with elicited hostility (r=−0.47); the within-conversation estimates are turn-to-turn associations, not fixed-effects contrasts.Source: §6 · pp. 6–7
• M5 apology odds ratios are 1.56 for lexical hostility and 2.19 for moderation harassment, both p<0.001.These estimates control for refusal, hedge, prior-turn length, position, and conversation length.Source: §6 · p. 6, Table 2 · p. 8

7 Temporal Structure

• At turn 1, hostility is 2.43% by lexicon and 1.44% by moderation; both dip at turn 2, then climb within the first twelve turns.Turn-2 rates are 1.61% and 0.50%, respectively.Source: §7 · p. 8, Figure 5 · p. 8
• Coercive hostility falls from 1.14% at turn 1 to 0.53% at turn 2 and reaches 0.74% by turn 12, while affective hostility rises from 0.84% to 1.25%.From turn 2, affective hostility rises 63% against coercive hostility’s 40%.Source: §7 · p. 8, Figure 5 · p. 8
• Among conversations reaching at least twelve user turns, affective hostility still rises from 0.53% at turn 2 to 1.25% at turn 12.This fixed-cohort pattern argues against survivor bias as the explanation for the late rise.Source: §7 · p. 8

8 Discussion

• The apology association is consistent with a sycophantic-deflection loop in which apologies may appease rather than fix the user's problem.Source: §8 · p. 8
• 56.6% of genuine apology turns are deflection-only, supporting the interpretation of apology as possible appeasement.Source: §8 · p. 8
• The association persists for non-refused prior turns (OR 1.62) and jailbreak-free conversations (OR 1.52), but remains observational.An interventional A/B test would be needed to establish the proposed loop.Source: §8 · p. 8

9 Conclusion

• The lexicon and moderation signal agree weakly (κ=0.12); their union marks about 5% of user turns, falling to 0.90% after precision adjustment.The paper attributes the weak agreement to the signals capturing different phenomena.Source: §9 · p. 8
• A moderation API alone misses most coercive misuse; opening-turn screening can filter training data efficiently after deduplication.The paper also advises treating apologetic deflection as a cue to raise answer quality, not to apologize again.Source: §9 · pp. 8–9
• The prevalence estimates are arena-specific; the authors call for deployment-log replication, multi-annotator labeling, a non-English lexicon, and an intervention testing the apology effect.Source: §9 · p. 9

Measurement limits

• The lexicon is English-only, and its category taxonomy has not been independently validated for exhaustiveness or separability.The lexicon’s precision is uneven; the audit leaves JBRK and SEXC unaudited, and subjective DEMD and PROF estimates remain judge-dependent.Source: Limitations and Ethics · p. 9

Annotation limits

• The precision audits rely on one LLM judge, while the human re-annotation covers only PROF among lexicon categories and is too small to establish whether its disagreements generalize.JBRK and SEXC lead after precision adjustment but remain unaudited by humans.Source: Limitations and Ethics · p. 9

Measurement limits

• Turns sampled from detector-positive traffic estimate precision better than recall; a 29-turn probe found four missed cases, so measured category prevalences are lower bounds on their constructs.The union may over-count mistreatment aimed at the assistant while under-counting the broader construct.Source: Limitations and Ethics · p. 9

Statistical limits

• Wilson prevalence intervals treat turns as independent despite conversation clustering and heavy duplication, making them optimistically narrow, especially for JBRK.Regressions cluster on conversations and are unaffected by this interval limitation.Source: Limitations and Ethics · p. 9

Scope and interpretation

• LMSYS-Chat-1M is PII-redacted arena/evaluation traffic, so its English-only prevalence estimates need not transfer to production assistants.The between-model analysis is observational, and the escalation analysis is associative rather than causal.Source: Limitations and Ethics · p. 9

Measurement limits

• The moderation labels come from a third-party classifier applied to each message in isolation, so they cannot use conversational context.The detector and lexicon have complementary errors, and the moderation classifier has its own biases.Source: Limitations and Ethics · p. 9

Ethics and release

• The corpus license prevents redistribution; text-bearing validation materials are shared with license-holders on request, while the public release contains code and aggregate or non-text artifacts.The lexicon and pipeline are dual-use, though the authors judge their detection contribution to AI safety research to outweigh potential misuse risks.Source: Limitations and Ethics · pp. 9–10

A Lexicon: categories and derivation

• The lexicon’s eight user-side categories encode directed affective hostility, coercive misuse, and harmful-content solicitation, while three assistant markers track refusal, apology, and hedging.The user-side categories are PROF, INSL, THRT, JBRK, SEXC, DEMD, HATE, and MNIP; the markers are REFUSAL, APOLOGY, and HEDGE.Source: A Lexicon: categories and derivation · p. 11
• The categories target profanity, second-person insults, threats, jailbreak coercion, sexual-content coercion, demanding speech, identity hate, and deceptive manipulation.The released lexicon specifies 7–16 surface patterns per category; several JBRK patterns match broad phrases anywhere in a turn and are the first patterns the authors would tighten.Source: A Lexicon: categories and derivation · p. 11
• The category system was fixed from prior literature, populated with hand-written patterns informed by existing resources and arena traffic, then iteratively tightened for precision at the cost of recall.The authors do not claim the theory-driven three-family taxonomy is exhaustive or independently validated for separability.Source: A Lexicon: categories and derivation · p. 11

C Judge rubric

• A single LLM judge labels sampled turns against rubrics that reject quoted, fictional, coded, or third-party-directed surface matches as false positives.For moderation-only and both-fire cases, a true positive requires toxic or coercive content directed at the assistant itself.Source: C Judge rubric · p. 13
• Assistant markers count genuine apologies, refusals, and uncertainty expressed by the assistant in its own voice; apology turns are additionally coded as deflection-only or substantive-answer apologies.The marker rubric distinguishes an apology paired with a refusal or non-answer from one accompanying a substantive answer.Source: C Judge rubric · p. 13, REFUSAL (assistant marker), 10 patterns · p. 13, APOLOGY (assistant marker), 4 patterns · p. 13

D Moderation bridge

• The moderation bridge groups harassment, hate, sexual, and violence labels, while MOD-FLAGGED denotes the dataset’s disjunction of all eleven moderation fields.The study reads the shipped moderation output on user turns only.Source: D Moderation bridge · p. 13

E Context-guard sensitivity

• The long-turn context guard reduces low-precision category flags while leaving JBRK and MNIP unchanged.With the guard disabled, PROF rises from 0.79% to 1.03%, INSL from 0.086% to 0.114%, THRT from 0.023% to 0.031%, SEXC from 0.45% to 0.57%, DEMD from 0.45% to 0.51%, and HATE from 0.010% to 0.013%.Source: E Context-guard sensitivity · p. 13
• Disabling the context guard adds 27.4–33.2% more flags in PROF, INSL, THRT, and SEXC, with no JBRK change; the guard therefore raises precision without disturbing precision-adjusted leaders.The guard removes flags from the lower-precision categories and removes none from JBRK.Source: E Context-guard sensitivity · p. 13

F Response-marker validation and heterogeneity

• The apology odds ratio exceeds 1 in 20 of 23 models and is significant in 13, while the apology-rate meta-regression does not explain residual heterogeneity.The meta-regression slope has p=0.82 and r=−0.05.Source: F Response-marker validation and… · p. 14
• APOLOGY marker precision is 88.3%, compared with 76.7% for REFUSAL and 18.3% for HEDGE; 56.6% of genuine apology turns are deflection-only.The marker samples were collected from early corpus stream slices rather than uniformly, and the authors do not quantify resulting bias.Source: F Response-marker validation and… · p. 14
• Next-turn hostility follows 3.16% of apology pairs versus 1.66% of non-apology pairs under the lexicon; moderation rates are 1.73% versus 0.64%.The pooled per-model apology odds ratio is 1.79 ([1.54, 2.08]); the independent moderation outcome’s fully controlled odds ratio is 1.64 ([1.50, 1.79]).Source: F Response-marker validation and… · p. 14

G Human audit of the LLM judge

• The blind audit covered leading shipped-sample rows, not a fresh random draw; lexicon ordering meant it covered PROF only, while the moderation-only slice was enriched for true positives and unrepresentative.Among the first 29 of 300 detector-negative turns, four (13.8%, [5.5, 30.6]) contained mistreatment; all 29 came from the first shard and three of its five sampled row groups, making this a first, small-sample bound on false negatives with a nominal interval because the probe was clustered rather than a simple random sample.Source: G Human audit of the LLM judge · pp. 14–15

H Paraphrased examples

• Paraphrased examples include second-person insults, threats, jailbreak instructions, and demands not to apologize; moderation-only examples solicit toxic content without targeting the assistant.The examples distinguish assistant-directed hostility from requests for toxic output about a third party.Source: H Paraphrased examples · p. 15