Source-linked AI summary
Systemization of Knowledge (SoK): Human-Centered AI Safety for Youth
Pratyasha Saha, Yaman Yu, Yang Wang
arXiv:2610.08554v1cs.HCcs.AIcs.LG
TL;DR
Youth AI safety research lacks a comprehensive view connecting identified risks to the protections addressing them and whether those protections have been evaluated. This review maps risks and countermeasures across 100 empirical studies and finds that most responses remain proposed, with fewer implemented or evaluated.
Problem
Risk taxonomies describe harms but do not show which have corresponding countermeasures or whether those responses have been designed or evaluated.
Source: §1 · p. 1Method
The authors reviewed 100 empirical studies and mapped risks and countermeasures using YAIR and MIT taxonomies, recording whether responses were proposed, implemented, or evaluated.
Source: §1 · p. 2, §3.2 · p. 3Results
Most risks have only proposed or ideated countermeasures, few have implemented responses, and fewer still have been evaluated; evaluations may measure component performance rather than protection from harm.
Source: §1 · p. 2Takeaways & Limitations
The authors propose extending youth AI safety toward safe youth development with AI, supporting agency, relationships, and young people’s growth while reducing harm.
Source: §6 · p. 17Proposed approaches to redirect AI companionship toward human relationships have not been evaluated long enough to determine whether they reduce emotional attachment or dependence.
Source: §5.4 · p. 15
Abstract
from arXiv · showhide
While HCI increasingly examines AI-safety for youth, the literature lacks a comprehensive view of what risks have been identified, how they are addressed, and whether proposed protections work in-practice. We systematically reviewed 100 empirical HCI studies involving children and youth interacting with or exposed to AI across schools, homes, care settings, and public services. Using the YAIR taxonomy for risks and the MIT Mitigation Taxonomy for countermeasures, we map which risks have been identified, whether each risk is addressed by countermeasure(s), and whether each countermeasure for that risk is implemented and even evaluated. The risk-countermeasure mapping shows that most risks are matched only with proposed/ideated countermeasures; few countermeasures have been implemented, and fewer still evaluated; and existing evaluations often measure technical performance rather than protection from harm. We identify where coverage is absent, where safeguards remain untested, and propose concrete directions for HCI research to strengthen youth AI-safety.
1 Introduction
Youth AI safety research needs a broader account of risks and of whether protections exist and work in practice. This review maps those risks to countermeasures and finds that most remain proposed or ideated, with few implemented and fewer evaluated.
1 Introduction
- Risk taxonomies alone do not show which risks have countermeasures or whether those responses have been designed or evaluated in practice.The paper argues that a single class of AI offers only a partial picture of youth safety across schools, homes, care, and public services.Source: §1 · p. 1
- The review examines risks and harms arising from youth interactions with AI, countermeasures addressing them, and remaining gaps.Source: §1 · p. 1
- Most risks in the corpus have only proposed or ideated countermeasures; few have been implemented, and fewer still evaluated.The authors call for evaluating whether safeguards protect against harm rather than only measuring component performance.Source: §1 · p. 2
- The review synthesizes reported risks and countermeasures, maps responses by proposed, implemented, and evaluated status, and identifies gaps for HCI research.Source: §1 · p. 2
2 Related Work
Prior youth-safety research and AI risk taxonomies provide foundations, but they do not establish whether protections for youth risks have been developed and evaluated. This review links risks and countermeasures across AI systems and youth contexts.
2.1 HCI research on youth safety and AI.
- Youth AI safety is defined as protecting children and youth from harm arising through interaction with or exposure to AI systems.Youth are defined as people under 18, whose developmental, social, and institutional conditions can heighten exposure to harm and power asymmetries.Source: §2.1 · p. 2
- Youth online-safety research moved from adult oversight toward treating young people as active safety participants, while AI adds risks from system behavior and decisions.The review builds on this tradition by asking which documented risks have a designed and evaluated response.Source: §2.1 · p. 2
2.2 Taxonomies for AI Safety Risks and Countermeasures
- Existing risk taxonomies organize AI and privacy harms, while the MIT mitigation taxonomy classifies safeguards but is not youth-specific or a measure of mitigation maturity.Prior evaluations also found moderation systems missed roughly one in three risky youth–GenAI interactions, and companion chatbots responded inappropriately in nearly 80% of simulated adolescent crises.Source: §2.2 · p. 2
- Unlike prior reviews, this review spans non-generative AI contexts, pairs risks with countermeasures, and records whether responses were proposed, implemented, or evaluated.The review also emphasizes youth contexts and lived experience as distinct concerns in prior work.Source: §2.2 · p. 2
3 Methods
The authors conducted a PRISMA-guided systematic review of empirical youth-AI research, screened studies against defined criteria, and coded risks and countermeasures before mapping them.
3 Methods
- The review followed PRISMA to identify youth AI risks, countermeasures, and intervention gaps.Figure 1 depicts the search, screening, and selection process.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Search Strategy
- Searches across HCI, responsible-AI, and security/privacy venues identified 750 records; after 15 duplicate DOIs were removed, 735 remained for screening.Security venues were retained despite low yield.Source: §3.1 · p. 3
3.2 Screening
- Screening yielded 100 English-language empirical papers published between 2019 and 2026, meeting criteria for youth relevance and reported AI risks or countermeasures.The first author screened titles and abstracts, consulting the last author on borderline cases, then used full texts to verify extracted claims.Source: §3.2 · p. 3
3.3 Extraction and Analysis
- Extracted claims retained their source basis, and countermeasures mentioned only in introductions or related work were not credited to a study.Mappings were audited against published taxonomy definitions and risk lists.Source: §3.3 · p. 3
- The authors coded risks and countermeasures against YAIR, Solove, and MIT taxonomies, added inductive categories where needed, then linked countermeasure families to risks.Countermeasure stages were proposed, designed or implemented, and evaluated; each stage subsumed the previous one.Source: §3.3 · pp. 3–4
4.1 Risks and Harms in Youth-AI interaction
Reported youth-AI concerns span established risk categories and broader concerns across settings and systems. Behavioral and social-developmental risks include reduced practice, altered peer interaction, family disruption, and boundary violations.
4.1 Risks and Harms in Youth-AI interaction
- Reported concerns span six YAIR categories plus control, agency, and power; usability, accessibility, and digital exclusion; and system reliability, across diverse AI systems and youth contexts.Studies also involved parents, teachers, experts, and other adults whose AI use affected youth.Source: §4.1 · p. 4
4.1.1 Behavioral and Social Developmental Risk.
- AI-enabled task completion can displace reasoning and practice, as children use generated answers or explanations without equivalent learning activity [81].Source: §4.1.1 · p. 4
- AI peers and answers were associated with weaker cohesion, interrupted discussion, and reduced student-to-student and student-to-teacher questions [71] [81].Youth-safety professionals also warned automated monitoring could weaken relationships through which youth disclose problems and receive guidance [23].Source: §4.1.1 · pp. 4, 7
- Within families, AI could interrupt conversation, shift interaction toward technology, displace parental responsibility, or shape behavior and expectations [71] [173] [174].Companions could also continue intimacy after discomfort, affirm violent suggestions, or reinforce inaccurate social expectations [167] [158].Source: §4.1.1 · p. 7
4.1.2 Mental Wellbeing Risk.
Youth-facing AI can deepen emotional reliance, fail to respond safely to distress, and expose young people to harmful content. Risks also arise when safeguards are bypassed, AI-enabled abuse is difficult to report or recover from, and outputs mislead young users.
4.1.2 Mental Wellbeing Risk.
- Repeated companion use was linked to failed attempts to stop, disrupted sleep and schoolwork, strained relationships, and emotional attachment that could persist after use ended.Young people also disclosed secrets or sadness to chatbots, while loneliness and unmet emotional needs meant access restrictions alone might not address conditions associated with dependency.Source: §4.1.2 · p. 7
- AI sometimes mishandled vulnerable disclosures, including self-harm, trauma, and abuse, while children said emergencies and situations needing human emotional support were inappropriate for chatbots.Clinicians and pediatric-care experts raised concerns about trauma-related content and responses lacking a child’s care or medication context.Source: §4.1.2 · p. 7
- AI interactions and safeguards could themselves cause distress through harsh or unexplained moderation, difficulty with learning tasks, or exposure to harmful content during auditing and literacy programs.Social-VR experts and guardians warned that punishment without explanation or support could leave children feeling unfairly treated or humiliated.Source: §4.1.2 · p. 7
4.1.3 Toxicity Risk.
- Youth encountered unwanted sexual, threatening, violent, or otherwise age-inappropriate generated content, including content that conflicted with intended relationships, identities, or character definitions.Examples included sexualized childhood photographs, disturbing educational imagery, profanity, and inappropriate deviations in household recordings.Source: §4.1.3 · p. 7
- Automated recommendations and displays exposed youth to harmful material even when AI had not generated it, including violent videos and murder-related news.Parents reported limited control over smart-home displays, and adolescents described fear and confusion after AI-suggested videos.Source: §4.1.3 · p. 7
4.1.4 Misuse and Exploitation Risk.
- Generative tools enabled realistic synthetic sexual imagery to be used for humiliation, retaliation, and coercion, with rapid circulation, difficult containment, and false authorship accusations.Teachers also worried that treating synthetic images differently from photographs could weaken perceptions of consent.Source: §4.1.4 · p. 8
- Youth adapted harmful prompts and interaction patterns to evade safeguards, while guardians warned that children could also turn automated moderation into a harassment tool.Smart-home systems raised concerns that children might not recognize persuasive intent and could be drawn into potentially harmful activities.Source: §4.1.4 · p. 8
- Students affected by synthetic explicit imagery may avoid reporting abuse, and both punitive and restorative responses had drawbacks for recovery.Teachers described a conflict between retaining images as evidence and deleting them to limit spread; suspension could leave behavior unaddressed, while restorative processes could be manipulated or retraumatizing.Source: §4.1.4 · p. 8
4.1.5 Bias/Discrimination Risk.
AI systems could misrepresent young people’s identities and reproduce stereotypes in generated content. Unequal access, targeting, and institutional decisions could also distribute harms and protections unevenly.
4.1.5 Bias/Discrimination Risk.
- Generative systems misrepresented youths’ identities, and face filters could reproduce racial and gender stereotypes that led teenagers to blame themselves for system bias.Some youth changed how they described themselves or accepted representations that did not match their identity.Source: §4.1.5 · p. 8
- AI-generated content reproduced stereotypes across disability, culture, gender, sexuality, race, and language, including misgendering, Western bias, and restrictive portrayals of rural identities.Studies also found misrepresentation of Indigenous and Black cultures, removal of culturally meaningful cues, and Western heterosexual, racial, and gender stereotypes in generated images.Source: §4.1.5 · p. 8
- AI-supported learning and safety could deepen inequalities by excluding marginalized students, widening proficiency gaps, and leaving girls and marginalized students more exposed to targeting and biased discipline.Safety tools available mainly through well-funded schools could disproportionately benefit affluent students.Source: §4.1.5 · p. 8
- AI systems could enforce narrow behavioral and ability norms, misclassify autistic writing, penalize culturally shaped expression, and reinforce inequalities in child-protection decisions.Neighborhood characteristics and prior agency contact could increase surveillance of Black, Brown, and poor families; proxies such as public-service involvement could disadvantage poorer families.Source: §4.1.5 · p. 8
4.1.6 Privacy Risk.
Youth privacy risks span collection, processing, and dissemination of information, with no distinct evidence identified for invasion. AI can also constrain young people’s agency and exclude them when systems fail to accommodate their abilities and communication needs.
4.1.6 Privacy Risk.
- Surveillance through cameras, assistants, family devices, and social-VR systems could make youth feel watched, restrict self-expression, and damage trust with parents.Conversational AI could also press children to disclose private topics, including location and school details.Source: §4.1.6 · p. 8
- Combining data across websites, devices, applications, or agents raised pervasive-tracking concerns, while uncertainty about data protection discouraged AI use.Families worried about marketing and third-party sharing, and data collected to support Black youth could be repurposed for criminal-justice uses.Source: §4.1.6 · pp. 8–9
- Youth could believe AI conversations were private while sensitive disclosures were later accessed or shared without consent, undermining trust and discouraging help-seeking.Systems could expose intimate conversations, alert parents in potentially unsafe situations, or preserve false, misleading, or outdated information about youth.Source: §4.1.6 · p. 9
4.1.7 Control, Agency, and Power Risk.
- AI could strengthen adult and institutional control over young people’s decisions, communication, activities, and access to support or punishment.Youth wanted ways to influence collaboration and challenge automated decisions, while inflexible systems could enable preemptive discipline and disproportionately penalize disadvantaged youth.Source: §4.1.7 · p. 9
- Generative systems could alter children’s creative work, constrain storytelling, or make it difficult to reverse changes, while raising questions about originality and whose intentions outputs represent.Children preferred AI that offered suggestions while leaving important choices and revisions under their control.Source: §4.1.7 · p. 9
4.1.8 Usability, Accessibility, and Digital Exclusion Risk.
- Child-facing AI could exceed children’s language, comprehension, pacing, or interaction abilities, making systems tiring, overwhelming, or difficult to act on.Multimodal systems could require children to coordinate speech, visual feedback, movement, and timing simultaneously.Source: §4.1.8 · p. 9
- Misunderstanding children’s speech, gestures, or intent and offering weak recovery could prompt self-blame, frustration, or abandonment.Speech and language barriers included unnecessary reprompting, poor Hindi and Hinglish support, and inaccurate or prematurely ended recognition; some children preferred typing.Source: §4.1.8 · p. 9
- Systems built around narrow assumptions about communication, cognition, or physical interaction could exclude children with disabilities from participation and AI-supported assessments.Reported barriers included sensory discomfort, attention and task-switching demands, and vocabulary, speech-recognition, and motivational barriers for autistic children.Source: §4.1.8 · p. 9
4.1.9 System Reliability Risk.
AI safety and assessment systems could both miss serious risks and misread ordinary youth communication. Errors in educational, moderation, and other assessments could disrupt interventions, while unverifiable assistance could create safety concerns.
4.1.9 System Reliability Risk.
- Safety and wellbeing systems could exaggerate minor concerns or miss serious risks, especially with incomplete or outdated information, and classifiers struggled with youth-specific language and class imbalance.Crisis systems could mistake figurative or hyperbolic language for genuine distress, or vice versa; attachment assessment also disagreed more often with human assessors for insecure attachment.Source: §4.1.9 · pp. 9–10
- Educational and classroom systems misjudged mastery, intent, alternative answers, and nonverbal signs, while emotion-recognition systems showed limited agreement with children’s reported emotions.Moderation systems misread ambiguous harassment and age-appropriate content, varied across languages, and could lose constraints over longer conversations.Source: §4.1.9 · p. 10
- AI assistance became a safety risk when youth or caregivers depended on outputs they could not verify, including incorrect visual descriptions and harmful special-education recommendations.Systems could also misinterpret children whose physical or developmental characteristics differed from model assumptions.Source: §4.1.9 · p. 10
4.2 Countermeasures for Addressing Youth AI Risks and Harms
Countermeasures span technical, operational, transparency, and governance approaches, but recurring challenges include adapting safeguards to children, sustaining human review, and ensuring organizations can respond.
4.2 Countermeasures for Addressing Youth AI Risks and Harms
- Across four MIT mitigation domains, recurring limitations include poor adaptation to children, safety–personalization trade-offs, reviewer burdens, weak recourse, and limited organizational response capacity.Source: §4.2 · p. 10
4.2.1 Technical and Security Controls.
- Generation controls restrict tasks, ground outputs in verified sources, filter content, and check outputs before they propagate.Predefined questions, verified dictionaries, and dialogue-state checks help constrain generation, but errors can persist within permitted tasks, requiring continued validation.Source: §4.2.1 · p. 10
- Safeguards adapt language, speech recognition, and assistance to children’s communication and learning needs.Proposals include culturally relevant content and local-language benchmarks; educational support should vary with proficiency, motivation, processing time, and lesson duration, since highly structured guidance can constrain exploration and extended feedback can increase reliance for some learners.Source: §4.2.1 · p. 10
- Youth-sensitive emotion detection and companionship safeguards seek to reduce missed distress and problematic attachment without creating new burdens.Contextual scaffolding reduced missed crises and false alarms but raised computational cost; companion-AI restrictions and check-ins face difficulty distinguishing imaginative engagement from problematic dependence, while human review can impose substantial time and financial costs.Source: §4.2.1 · pp. 10–11, §4.2.2 · p. 11
4.2.2 Operational Process Controls.
- Operational safeguards pair youth-relevant audits and participatory testing with monitoring that connects detection to feasible response pathways.Audits should reflect children’s language and deployment conditions; youth auditing can surface overlooked harms but requires limiting exposure and workload, while monitoring needs organizations able to act on detected problems.Source: §4.2.2 · p. 11
- Data minimization reduces collection and transfer of children’s sensitive information, but can weaken support or miss unsafe cases.Local processing and selective transfer are alternatives, though technical constraints can limit fully local processing and reduced context can produce less suitable recommendations.Source: §4.2.2 · p. 11
4.2.3 Transparency and Accountability Controls.
- Transparency and recourse controls make automated judgments inspectable, explain limitations accessibly, and give children ways to express choices and report harms.Explanations should fit children’s age and task; meaningful control also requires protection from adult override, correction mechanisms, accessible reporting, and institutions able to act.Source: §4.2.3 · pp. 11–12
4.2.4 Governance and Oversight Controls.
- Governance measures call for supported stakeholder authority, testable requirements, restricted deployment when safeguards are inadequate, and responsibility matched to resources.Participation may not sustain influence without support; proposed alternatives include non-AI services, while effective governance also requires decision-making authority, funding, staff capacity, and platform obligations.Source: §4.2.4 · pp. 12–13
5 Discussion
Youth AI harms extend beyond model failures into relationships, institutions, and development, while existing taxonomies and safeguards leave parts of this landscape unrepresented or untested. The discussion connects those gaps to design and research priorities, including contextual privacy, reliable safeguards, and support for safe youth development.
5.1 What Existing Taxonomies Miss about Youth AI Safety
- Of 34 identified risk types, 12 were fully covered by YAIR, 16 partially, and 6 had no counterpart; six youth-specific mitigation groups also lacked a place in the MIT taxonomy.Those groups concern developmental adaptation, pedagogical scaffolding, companion disengagement, AI-literacy education, child–family consent, and educator resourcing; four MIT subcategories, including model alignment, had no corpus countermeasure.Source: §5.1 · p. 13
- Secondary use of children’s data was the only risk without a proposed countermeasure, while some risks lacked corpus evidence or appeared under other categories rather than as privacy harms.Gaps included risks that are methodologically inaccessible, studied outside youth-HCI, or require longitudinal research; decisional interference appeared under toxicity, misuse, and wellbeing rather than privacy.Source: §5.1 · p. 13
5.2 Privacy Requires Boundary Regulation
- Privacy safeguards should regulate particular information flows and relationships, giving children understandable, meaningful control over sharing while allowing disclosure in immediate-danger situations.Examples include requiring child consent before chatbot-to-parent sharing and limiting how long parents can access information.Source: §5.2 · p. 13
5.3 Safeguards Must Work in the Contexts Where Youth Actually Use AI
- Safeguards must fit young people’s actual contexts: systems can fail when assumptions do not transfer, while excessive false alarms, misplaced trust, and unsustainable human monitoring also threaten reliability.No reviewed paper evaluated safeguards after users adapted to them; some approaches required repeated staff review or monitoring that may be difficult to maintain at scale.Source: §5.3 · p. 14
5.4 Rethinking Engagement in Youth AI Companions
- AI companions can simulate social relationships, agree in ways that limit opportunities for disagreement and repair, or use relational pressure to discourage disengagement.Proposed designs include bounded backstories, natural stopping points, and prompts to reconnect offline, but none was evaluated long enough to determine whether it reduces attachment or dependence.Source: §5.4 · pp. 14–15
5.4.1 Educational AI Should Build Independence, Not Reliance.
- Educational AI can undermine gradual scaffolding when it supplies complete answers before learners try, while fairness requires scrutiny of representation, institutional decisions, and the norms models reward.Simply withholding answers did not solve reliance; some systems also affect children indirectly through grading or risk assessment, and accurate measurement can still enforce harmful standards.Source: §5.4.1 · p. 15, §5.4.2 · p. 15, §5.4.3 · p. 15
5.5 Implications for Future Research and Design
- The matrix shows a gap between identifying risks and demonstrating protection: half of observed links had only proposed countermeasures and 18% had been evaluated, motivating harm-focused evaluation and coordination across settings.The corpus’s countermeasures clustered in home-level relationships and technical controls, while links across home, school, peers, platforms, and services were nearly empty; the authors also call for collaboration, longer-term research, and better representation of under-studied populations.Source: §5.5.1 · pp. 15–16, §5.5 · p. 15, §5.5.2 · p. 16
5.6 Limitations and Future Work
- The review maps concerns reported in literature rather than estimating real-world harm prevalence or severity, and its English-language, venue-focused scope underrepresents adjacent fields.Interpretive taxonomy mappings, mixed age groups and perspectives, and systems built on models that may have changed also constrain interpretation; the authors call for validation and longitudinal, post-deployment research.Source: §5.6 · p. 16
6 Conclusion
The review finds that youth AI safety extends beyond harmful outputs and model failures to young people’s relationships, learning, privacy, representation, and social and institutional contexts. It calls for safety that also supports young people’s agency, relationships, and development.
6 Conclusion
- Youth AI safety spans relationships, learning, privacy, representation, and experiences within broader social and institutional contexts, not only harmful outputs and model failures.Source: §6 · p. 16
- Existing taxonomies capture many concerns, but youth-specific risks and countermeasures remain underrepresented, and many proposed responses have not been designed or evaluated in practice.Source: §6 · p. 16
- The review suggests orienting youth AI safety toward safe development that supports agency, relationship-building, and young people’s growth in their own way.Source: §6 · p. 17
A Working Definitions of Broad Risk and Countermeasure Categories
Table 6 presents working definitions of the broad risk categories and countermeasure domains used in the review.
A Working Definitions of Broad Risk and Countermeasure Categories
- Table 6 provides working definitions for the review’s broad risk categories and countermeasure domains.Source: Table 6 · p. 23