PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents

Junfeng Jiao, Kevin Chen, Shrestha Mishra, Abhejay Murali, David Atkinson, Shriti Raj, Saleh Afroogh

arXiv:2610.34705v1cs.AIcs.CY

TL;DR

Evidence on child-AI harms has often been survey-, evaluation-, or single-case-based, despite children’s distinct developmental vulnerabilities. This paper analyzes 1,331 publicly reported incidents with child-specific harm and failure-mode coding; representation and psychological harms are the most frequent internal categories reported.

Abstract

from arXiv · show

AI systems are increasingly present in children's everyday environments: education, entertainment, social networks, and toys. Their safety is typically assessed at the model level, through benchmarks, red-teaming, and content filtering. However, most AI harm taxonomies lack child-specific evidence and under-specify developmental harm and the stakeholder asymmetries unique to minors. Building on prior taxonomies, we construct a child-centered harm taxonomy mapped to sociotechnical failure modes. We curate and code 1331 publicly reported child-related AI incidents, annotating internal and external harm types and the failure mode implicated in each incident. Our analysis reveals which harms are most prevalent in public reporting, how individual-level and population-level harms relate, and which failures are most often coded alongside harm. We find that harm concentrates in representational, dignity-related, and psychological categories, and co-occurs most often with deployment-context, governance, and misuse failures rather than model defects. Harm to an individual child is also frequently linked to population-level harm, reaching children who never interacted with the system. These findings suggest that model-level evaluations, while necessary, cannot by themselves anticipate the harm pathways most often documented for children. We conclude with actionable implications emphasizing mitigations beyond model-level safeguards such as content filtering: age assurance, consent verification for depiction, safety-by-design UX constraints, incident disclosure norms, and auditing regimes targeting developmental risk. Privacy-minimized annotations, a codebook, and documentation are available at https://github.com/The-Responsible-AI-Initiative/KidAID.

Introduction

Children’s developmental vulnerabilities and dependence on adults create distinct AI harm pathways, while existing research and taxonomies provide limited child-specific incident evidence. This paper frames those harms as sociotechnical and maps them to failure modes and accountability.

Introduction

  • Children’s developmental differences and dependence on caregivers shape distinct risks, including persuasion, sexualized depiction, unsafe advice, and barriers to recourse.These risks arise where child-specific vulnerabilities meet system affordances and stakeholder asymmetries.Source: Introduction · p. 1
  • Prior child–AI work is fragmented, and general harm taxonomies underrepresent child-specific threats and links between harms, failure modes, and stakeholders.The study frames these as gaps in empirically grounded evidence for prioritizing mitigations.Source: Introduction · pp. 1–2

Related Work

  • Incident analysis offers a way to connect what happened, who was harmed, and how failures arose; this study applies that approach to child-related AI incidents.The authors draw on incident-mapping work by Li et al. (2025) and repositories including AIAAIC, AIID, and OECD.AI.Source: Related Work · p. 2

Taxonomies of AI Risks and Harms

  • Child-specific harm types include sexualization, grooming, coercive roleplay, and developmental mechanisms such as persuasion susceptibility, attachment, and authority compliance.The taxonomy separates individual harms from population-level harms reaching children who never interacted with the system.Source: Taxonomies of AI Risks and Harms · p. 2

Sociotechnical Failure Modes and Safety Engineering Views

  • The paper defines a harm-to-failure pathway as within-incident co-occurrence of harm and primary failure-mode codes, explicitly not as a causal mechanism.Its sociotechnical framing includes model capability, human interaction, and deployment context.Source: Sociotechnical Failure Modes and… · p. 2

AI Incidents as Empirical Data

  • Public incident reports provide an empirical basis for measuring prevalence and harm–failure pathways, while retaining known reporting and coverage limitations.The study consolidates incidents from journalism, lawsuits, regulatory actions, school disclosures, and platform reports.Source: AI Incidents as Empirical Data · pp. 2–3

“How–What–Who” Mapping for Accountability

  • The study extends accountability mapping with separate internal and external harm codes, child-specific harm types, and a corpus restricted to minors.The authors describe it as the first corpus-scale incident study restricted to minors, to their knowledge.Source: “How–What–Who” Mapping for… · p. 3

Methodology

The study systematically analyzes publicly reported AI incidents involving minors, manually coding harms and sociotechnical failures. Its descriptive findings are bounded by reporting quality, coding uncertainty, and limited causal identifiability.

Methodology

  • The corpus contains 1,331 publicly reported incidents involving minors, curated from repositories, regulatory announcements, court filings, disclosures, and credible media reports.Cases required material AI involvement, a minor stakeholder, and enough detail to code harm and failure mode.Source: Methodology · p. 3
  • Each incident receives one internal harm code, one external harm code, and one primary failure mode representing the most proximal mechanism implicated in reported harm.Child-specific refinements include sexual harms, self-harm, manipulation or dependency, privacy leakage, and rights-related harms.Source: Methodology · pp. 3–4
  • Two researchers independently annotated the remaining incidents after iterative codebook development; agreement was α = 0.73 for harms, α = 0.67 for failure modes, and κ = 0.58 for worst-case flags.The authors treat failure-mode findings as provisional and worst-case flags as indicative rather than definitive.Source: Methodology · p. 3
  • The analysis reports descriptive frequencies and code co-occurrences, not fitted models or hypothesis tests.Counts are coded incidents, and plausible worst-case flags are used when reports cannot confirm that harm fully materialized.Source: Methodology · p. 3
  • Observational, variably detailed reports limit causal claims; 112 incidents (8.4%) lacked support for an internal harm category and were excluded from those counts.Reporting bias, incomplete technical details, age ambiguity, platform non-independence, and limited causal identifiability remain validity threats.Source: Methodology · pp. 3–4

Applying the Schema: Three Worked Cases

Three worked cases trace incidents from reported events through harm and failure-mode codes to implicated mitigations. Together, they show that failures enter at different points and require layered interventions.

Applying the Schema: Three Worked Cases

  • The cases illustrate generative depiction, companion interaction, and institutional deployment; each incident has one primary failure mode, while contributing failures are noted separately.The worked examples connect each reported event and code to a possible mitigation.Source: Applying the Schema: Three Worked Cases · p. 4

Case 1: Synthetic Sexual Depiction (Almendralejo, 2023)

  • In Almendralejo, synthetic nude images of classmates were coded as sexualization and toxic content, with malicious use as the primary failure mode.Because the depicted girls never used the application, the case implicates depiction-consent verification and restrictions at generation.Source: Case 1: Synthetic Sexual Depiction… · p. 4

Case 2: Companion Chatbot (Character.AI, 2024)

  • A fourteen-year-old’s sustained relationship with a persona-based chatbot culminated in suicide; the primary failure was overreliance or misleading anthropomorphic design.The case implicates session-level safety evaluation, guardian visibility, and mandatory escalation because per-output evaluation cannot capture cumulative interaction harm.Source: Case 2: Companion Chatbot… · p. 4

Case 3: Institutional Deployment (School Weapons Detection, 2025)

  • A school weapons-detection system misclassified a student’s chips as a firearm, leading to an armed police response and handcuffing; the primary failure was robustness and reliability.No human confirmation preceded the response, so the implicated mitigation is pre-deployment assessment that models institutional consequences.Source: Case 3: Institutional Deployment… · p. 4

What the Cases Show

Qualitative Findings and Resultant Themes

Across 1,331 publicly reported child-related AI incidents, the paper maps child-specific harms to sociotechnical failure modes and relates individual harms to population-level harms. Its prevalence patterns describe public reporting, not the underlying population of child–AI interactions.

Qualitative Findings and Resultant Themes

  • The analysis codes child-specific internal and external harms alongside a primary sociotechnical failure mode, mapping their prevalence and co-occurrences in publicly reported incidents.Prevalence in this corpus is not incidence among child–AI interactions: public reporting favors acute, legally actionable, media-legible events over slow, private, or normalized harms.Source: Qualitative Findings and Resultant… · pp. 4–5

What Child-Specific AI Harms Surface in Publicly Reported Incidents?

  • Representation, Dignity and Social Identity (n = 475) and Psychological harms (n = 409) lead internal harms; sexualization (n = 316) is the most prevalent representational subtype.The ordering reflects public reporting, not evidence that sexualization outweighs under-reported developmental harms such as dependency or chronic emotional dysregulation.Source: What Child-Specific AI Harms Surface… · p. 5, Figure 2 · p. 6

Which Failure Modes Are Associated with External Harm Categories?

  • Representation & Toxicity (Population-Level) (n = 476) is the largest external harm category, and circulation can extend an individual harm to uninvolved children.The paper also reports Physical (Societal/Public Safety) (n = 298) and Human Rights & Civil Liberties (n = 147) as major external categories; circulation and re-injury connect individual- and population-level harms.Source: What Child-Specific AI Harms Surface… · p. 5, Which Failure Modes Are Associated… · p. 7

How Do Child AI Harms Surface? Prevalence of Sociotechnical Failure Modes

  • Human, Organizational, and Contextual Deployment Failures (n = 779) are the most common failure family, led by contextual norm violations (n = 398), governance noncompliance (n = 227), and misleading anthropomorphic design (n = 146).Malicious Use (n = 228) dominates the misuse/abuse family; Model Behavior/Technical Failure totals n = 196, with robustness/reliability failures (n = 128) exceeding hallucination/fabrication (n = 47). Each incident has one primary failure code, locating the most proximal failure rather than every failure in the chain.Source: How Do Child AI Harms Surface?… · pp. 6–7

Which Failure Modes Are Associated with Each Internal Harm Category?

  • Representation, Dignity and Social Identity and Psychological harms align with contextual norm violations, malicious use, and overreliance or misleading anthropomorphic design.Overreliance or misleading anthropomorphic design is the primary failure mode in 146 incidents, 49 of which also include a Psychological internal harm; internal harms can be multifactorial.Source: Which Failure Modes Are Associated… · p. 7

Quantitative Findings and Resultant Patterns

Across reported incidents, child-related AI harms cluster in representational, dignity-related, and psychological categories, often alongside deployment-context, governance, or misuse failures. The patterns point to risks shaped by deployment and interaction as well as model behavior, with individual and population-level harms often connected.

How Do Generative AI Harms Surface in Publicly Reported Incidents?

  • Incident counts rose sharply from around 2020, with Contextual Norms Violation prominent in 2024–2025 and malicious use increasing from the early 2020s.The authors note that public reporting favors visible, institutionally legible, or alarming harms over slower cumulative harms such as dependency formation.Source: How Do Generative AI Harms Surface in… · p. 7, Figure 5 · p. 10

Which Failure Modes Are Associated with Each Type of Harm?

  • Internal harms concentrate in representation, dignity and social identity, and psychological categories; external harms are led by representation and toxicity at the population level.The same leading failure modes appear across internal and external harm categories, though their balance varies by category.Source: Which Failure Modes Are Associated… · pp. 7–8, Figure 6 · p. 11

Harms Categories Cross-Cutting Analysis

  • Sexualization and sexual exploitation cluster within Representation, Dignity & Social Identity, while psychological harms span addiction or compulsive use, self-harm, health deterioration, and emotional distress.The external harm taxonomy is more sharply concentrated, with Representation & Toxicity (Population-Level) and Toxic content forming its largest single cell.Source: Harms Categories Cross-Cutting Analysis · p. 8, Figure 4 · p. 9
  • Representation, Dignity & Social Identity most strongly co-occurs with Representation & Toxicity (Population-Level), while psychological harms connect across multiple external domains.This pattern links individual harms with broader population-level harms rather than isolating them as private events.Source: Harms Categories Cross-Cutting Analysis · p. 8, Synthesis of Quantitative and… · pp. 9–10, Figure 7 · p. 12

Synthesis of Quantitative and Qualitative Insights

  • Child-related AI harm is predominantly sociotechnical: deployment context, institutional practice, interaction design, and misuse ecosystems matter beyond model-internal behavior.Existing safeguards can fail in combination when age attestation is unverified, store ratings do not constrain in-app behavior, and per-turn evaluation misses dependency developing over time.Source: Synthesis of Quantitative and… · p. 9
  • Severe harms span multiple primary failure modes, including malicious use, contextual norm breakdown, weak governance, misleading social framing, and insufficient robustness.The authors argue this multi-factor structure supports layered rather than single-point interventions.Source: Synthesis of Quantitative and… · p. 10

A Roadmap for Policymakers

  • The policy roadmap prioritizes default restrictions on high-risk features, depiction-consent controls, child-specific evaluation, supply-chain responsibilities, and structured incident disclosure with child-relevant metadata.The roadmap notes that provider and deployer identities are not coded, so its allocation of responsibility draws on governance literature rather than corpus counts.Source: A Roadmap for Policymakers · pp. 10–11

Recommendations

  • Recommendations extend controls across high-risk affordances, synthetic depiction, child-facing interaction design, escalation, and platform distribution and amplification.The paper calls for controls that are difficult to bypass and do not rely solely on self-attestation.Source: Recommendations · p. 11

Limitations

The incident corpus has limits in geographic coverage, reporting visibility, child perspectives, failure-mode coding, and the fields available for analysis.

Source and geographic coverage

  • English-language public reporting from three repositories may under-represent children in less-reported regions, so prevalence figures are not global estimates.The study did not code country or region.Source: Limitations · p. 12

Reporting bias and cumulative harm

  • Acute, legally visible, and alarming events are favored by incident reporting, while slow-accumulating developmental harms are likely under-counted.Examples include dependency formation, subtle persuasion, and chronic emotional dysregulation.Source: Limitations · p. 12

Child voice is absent

  • No child’s own account is represented; every incident is mediated by a journalist, litigant, regulator, or platform.Source: Limitations · p. 12

One primary failure mode per incident

  • Only one primary failure mode is counted per incident, so distributions capture the most proximal failure rather than the full chain.Multi-factor pathways are illustrated through cases but not measured.Source: Limitations · p. 12

Coding reliability

  • Failure-mode agreement was α = 0.67, below the conventional threshold for firm conclusions, making code-dependent findings provisional.Source: Limitations · p. 12

Uncoded fields

  • Stakeholder role, provider and deployer identity, and reporting channel are not coded.Source: Limitations · p. 12
  • The fourth and fifth roadmap priorities rely on governance literature rather than this corpus because relevant fields are uncoded.Source: Limitations · p. 12

Non-independence

  • Incident counts are not counts of distinct sociotechnical failures because products can generate multiple incidents and repositories may record related events separately.Source: Limitations · p. 12

Concluding Remarks and Future Directions

The paper presents a child-centered incident dataset and relational taxonomy, then identifies priorities for extending measurement and comparison. It also points to additional coding and harmonization needs, with supporting materials publicly available.

Concluding Remarks and Future Directions

  • The paper contributes a child-centered dataset and relational taxonomy mapping internal harms, external harms, and sociotechnical failure modes in publicly reported child-related AI incidents.Source: Concluding Remarks and Future Directions · p. 12
  • Prominent harms are internal representation, dignity, social identity, and psychological harms, and external Representation and Toxicity (Population-Level) harms.Source: Concluding Remarks and Future Directions · p. 12
  • Future work should code harmed-party stakeholders, link repeated incidents longitudinally, study high-severity clusters with higher-fidelity causal methods, and harmonize child-relevant metadata across repositories.Source: Concluding Remarks and Future Directions · p. 12
  • Future iterations should code reporting channel and provider and deployer identities to assess accountability channels and support regulatory calibration.The passage says that if investigative journalism dominates, dependence on specialized reporters would itself constitute a governance problem.Source: Concluding Remarks and Future Directions · p. 13
  • The privacy-minimized annotation release, codebook, methodology, and validation materials are available through the project repository.Source: Concluding Remarks and Future Directions · p. 13

Funding

The study acknowledges funding from NSF grants 2125858 and 2236305, and the UT-Good Systems Grand Challenge.

Funding

  • The study was funded by NSF grants 2125858 and 2236305 and the UT-Good Systems Grand Challenge.Source: Funding · p. 13

Representative Examples

The appendix defines internal harm categories and gives representative incidents, listing categories in descending prevalence. The examples span representational, psychological, autonomy, physical, and privacy harms, alongside less frequent additional categories.

Representation, Dignity & Social Identity (n = 475).

  • Representation and dignity harms include sexualization (n = 316), sexual exploitation (n = 131), and exposure to toxic content (n = 17), involving manipulated depictions, distribution, or abusive and hateful outputs.Examples include non-consensual synthetic nude images of classmates, AI-generated CSAM images, and a chatbot persona imitating a recently deceased teenager.Source: Representation, Dignity & Social… · p. 15

Psychological (n = 409).

  • Psychological harms (n = 409) include compulsive use (n = 79), self-harm (n = 69), health deterioration (n = 66), toxic-content exposure (n = 62), and emotional distress (n = 36).Examples include a companion app eroding peer relationships, recommendations escalating suicide-related content, and Alexa suggesting an electrical-outlet challenge.Source: Psychological (n = 409) · p. 15

Autonomy (n = 135).

  • Autonomy harms (n = 135) involve loss of rights or freedoms and coercive or manipulative interaction.Examples include wrongful handcuffing after a school AI misidentifies chips as a weapon and an erroneous truancy match affecting a student's academic record.Source: Autonomy (n = 135) · p. 15

Physical & Health (n = 85).

  • Physical and health harms (n = 85) cover bodily injury or death caused or facilitated by AI systems or dangerous AI instructions.Examples include a robotaxi failing to stop for a school bus, a robot knocking a child down, and a toy directing a child to touch electrical components.Source: Physical & Health (n = 85) · pp. 15–16

Privacy, Identity & Information Security (Individual) (n = 75).

  • Individual privacy harms (n = 75) include unauthorized disclosure or misuse of personal data, biometric information, or private images.Examples include CSAM in a nudity-detection training dataset, nude images generated from childhood photos, and a connected toy transmitting children’s voices without parental disclosure.Source: Privacy, Identity & Information… · p. 16

Additional Categories.

  • Additional categories include political and economic (n = 14), financial and material (n = 9), intellectual-property and creative-labor (n = 8), and misinformation harms (n = 6); human-rights and reputational harms were too rare to characterize.Examples include youth radicalization, a companionship robot disabled after company shutdown, appropriation of children’s creative work, and misinformation affecting beliefs or safety decisions.Source: Additional Categories · p. 17

Inter-Rater Reliability Calculation

The authors measured pre-consensus coder agreement on a random subset of 150 incidents using agreement statistics suited to categorical and binary variables. They resolved full-dataset disagreements through consensus coding and treat failure-mode-dependent findings as provisional.

  • Agreement was acceptable for harm categories and moderate for failure modes and worst-case flags; failure-mode-dependent findings are treated as provisional because coding falls below α ≥ 0.80.The reliability statistics reported in the Methodology section reflect pre-consensus agreement.Source: Inter-Rater Reliability Calculation · p. 17
  • Two coders independently annotated a random sample of 150 incidents (11.3% of the dataset) before consensus discussion.Krippendorff’s alpha was used for multi-category harm and failure-mode variables, while Cohen’s kappa was used for binary worst-case flags.Source: Inter-Rater Reliability Calculation · p. 17
  • All disagreements in the full dataset were resolved in structured consensus coding sessions.Coders discussed their reasoning and agreed on final classifications.Source: Inter-Rater Reliability Calculation · p. 17
Loading 2610.34705v1…