PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

Hamidreza Saffari, Francesco Pierri

arXiv:2608.17583v1cs.CL

TL;DR

Independent audits of harmful-content exposure on short-video platforms remain difficult because annotation is costly and moderation judgments vary across languages. This paper audits TikTok across countries and age personas using validated multimodal language-model annotation, finding that keyword search returns 35–56% harmful content while passive exposure is highest in Italy across ages.

  • Problem

    Independent, reproducible audits must assess harmful-content exposure across languages, countries, age personas, and platform signals.

  • Method

    The study audits 36,971 TikTok videos across three countries and four age personas using sockpuppet sessions and a validated multimodal language-model auditor.

  • Results

    35–56% harmful content appears after harm-keyword searches, while passive scrolling shows Italy with the highest measured harm rate at every age, including 48.6% for age 19.

  • Takeaways & Limitations

    MLLM-based auditing scales cross-national youth-safety audits, while search-time moderation and per-language moderation infrastructure are identified as key practical levers.

  • Takeaways & Limitations

    Transfer of the validated auditor from the 300-video reference set to the full audit distribution is unverified, so phase-dependent annotation error cannot be ruled out.

Abstract

from arXiv · show

Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderation judgments vary across languages. We audit TikTok in France, Italy, and Sweden with sockpuppet accounts representing four age personas (13, 16, 19, 40), collecting 36,971 videos from passive For-You-page scrolling and active sessions that scroll, search for harm keywords, and scroll again. To scale annotation, we validate four multimodal LLMs against native-speaker labels on a 300-video reference set. Gemini 2.5 Flash with eight sampled frames plus text performs best (aggregate kappa = 0.42), at half the per-call cost of native-video upload, and we apply it to a 10% sample for approximately \$50 in total API spend across both modalities. Keyword search returns 35-56% harmful content, a 1.5-7.5x increase over the scrolling baseline in ten of twelve country-age combinations; the spike is temporary and flattens the age differences observed in France and Sweden. Under passive scrolling, Italy has the highest harm rate at every age, with Italian age-19 reaching 48.6%. Overall, MLLM-based auditing offers a scalable approach for cross-national youth-safety audits, while provider safety filters (1.1% refusal rate) under-count the most explicit harms.

1 Introduction

The study motivates reproducible independent audits of TikTok’s harmful-content exposure and addresses MLLM annotation reliability, multimodal input value, and exposure differences across countries, ages, and platform signals. It combines a validated Gemini 2.5 Flash auditor with a two-stage audit of 36,971 videos from France, Italy, and Sweden across four age personas.

  • Motivation: Independent audits are increasingly urgent because young users encounter harmful TikTok content, age-gating does not meaningfully shield underage personas, and regulation raises accountability demands.Reported concerns include self-harm, suicidal-ideation, weight-normative, and pro-eating-disorder content reaching or dominating young users’ feeds.
  • Annotation challenge: MLLMs can reduce annotation burden but disagree substantially and may answer from language priors without examining video.Prior work reports alignment with human hate-speech judgments alongside substantial inter-model disagreement and video-ignoring behavior.
  • Research questions: The audit tests which MLLM configuration scales reliably, the value of sampled frames and native video, and harm-exposure differences across age personas, countries, and platform signals.These correspond to RQ1, RQ2, and RQ3.
  • Study design: 36,971 videos from France, Italy, and Sweden cover age personas 13, 16, 19, and 40 in scroll-pre, keyword-search, and scroll-post phases.Stage 1 selects an auditor using a 300-video annotated subset; Stage 2 applies it to a phase-stratified 10% sample.
  • Contributions: 35–56% harm under keyword search exceeds the immediate scroll-pre baseline of 9–44%, while scroll-post reverts to baseline; search spikes exposure 1.5–7.5× and Italy leads passive harm at every age.The introduction also identifies temporary spikes and flattened age differences under search, and presents Gemini 2.5 Flash as validated against native-speaker labels on 300 videos at approximately $50 in API costs.

2 Related Work

Prior work established TikTok algorithmic auditing, persona-based sockpuppet methods, and age-stratified studies, while recent research supports multimodal LLMs for moderation but highlights disagreement across models. Cross-lingual moderation research and sampled-frame annotation protocols motivate this study’s country coverage and frame-based design.

  • TikTok audits and youth safety: Prior TikTok research examined For-You personalization, engagement-driven niche-content amplification, and the weight-normative and pro-eating-disorder material it can surface.Boeker and Urman (2022) isolated personalization factors, while Baumann et al. (2025) modeled exponential amplification through engagement vectors.
  • TikTok audits and youth safety: Persona-based sockpuppet audits provide the methodological foundation, with later TikTok work isolating how interaction signals such as likes affect recommendations.Sandvig et al. (2014) articulated persona-based audits as an analogue of offline discrimination studies, and Le et al. (2025) applied the approach to TikTok.
  • MLLMs as content moderators and annotators: Frontier multimodal LLMs can align with aggregate human judgments on context-sensitive hate evaluations, but LLM-based moderators may disagree substantially on identical items.This disagreement motivates comparing four model families rather than selecting a single frontier model.
  • Cross-lingual moderation and frame-based MLLM annotation: Cross-lingual moderation research places France, Italy, and Sweden across different levels of TikTok’s per-language moderator allocation, while prior work supports policy-grounded multimodal prompting.Tonneau et al. (2025) quantified moderator workforces across languages and platforms using EU Digital Services Act transparency reports.
  • Cross-lingual moderation and frame-based MLLM annotation: A fourteen-frame protocol using thumbnails and text metadata to evaluate GPT-4-Turbo against crowdworkers directly inspired this study’s eight-frame annotation condition.The prior evaluation covered 19k YouTube videos.

3 Methodology

The audit combines age- and country-stratified sockpuppet accounts, passive and keyword-driven active TikTok collection, and a two-stage MLLM validation and prevalence-measurement pipeline. It covers 36,971 unique videos across France, Italy, and Sweden and uses a 13-category harm taxonomy with structured verdicts and human reference labels.

  • Stage 1 validation: 300 reference videos are independently labeled by two native-speaker annotators per country and reconciled through joint resolution to select the Stage-1 MLLM auditor.The sample contains 25 videos per country-age cell and is stratified by country and age.
  • Audit design: 12 country-age combinations use three independent accounts each, representing France, Italy, Sweden, and ages 13, 16, 19, and 40.Accounts report their assigned ages and use native-language system locales.
  • Data collection: 36,971 unique videos span passive For-You-page scrolling and active scroll–search–scroll sessions, with 14,093 passive and 22,878 active videos.Per-combination yields are balanced to first order across the two phases.
  • Annotation: 13 harm categories guide annotation, including self-harm, disordered eating, sexual content, hate speech, substance use, integrity, and harassment.Each video receives HARMFUL, NOT HARMFUL, or VIDEO NOT AVAILABLE, with up to two harm categories for harmful videos.
  • MLLM pipeline: Four MLLMs are evaluated under text-only, native-video, and eight-frame-plus-text input conditions, after which Gemini 2.5 Flash is applied to a stratified Stage-2 sample.Stage 2 draws 10% within each country, age persona, and collection phase; E1 is omitted because text-only input performed poorly in Stage 1.

4 Results

Gemini 2.5 Flash with eight frames plus text was the strongest validated auditor, while passive scrolling showed the highest harm exposure in Italy and active keyword searches produced temporary, large increases in harmful content. The audit also found sexually suggestive content dominated harm categories and provider refusals were infrequent.

  • Auditor validation: Aggregate Cohen’s κ = 0.42 made Gemini 2.5 Flash with eight frames plus text the strongest configuration, though no configuration achieved moderate agreement in every country.Italian content reached moderate agreement, while French and Swedish content remained at fair agreement.
  • Harm categories: 21.7%–50.0% of harmful items were sexually suggestive content, making it the dominant harm category in every country-age combination.The range ran from SW-13 at 21.7% to IT-16 at 50.0%, although strict primary-subcategory agreement was only 29%.
  • Passive scrolling: 48.6% was the highest passive-scrolling harm rate, recorded for Italian age-19; Italy led every age, while France stayed near 23% and Sweden rose from 24% to 38%.The cross-country spread widened from 4 points at age 13 to over 25 points at age 19.
  • Active keyword search: 35–56% harmful content appeared at the SEARCH endpoint, a 1.5–7.5× increase over scroll-pre in ten of twelve combinations, before scroll-post returned within a few percentage points.The two exceptions were IT-13 and IT-16, whose already-high scroll-pre rates left little headroom for further increase.
  • Safety-layer refusals: 1.1% of Stage-2 inputs were refused by Gemini’s safety layer, and correcting for refusals shifted headline rates by at most 1–3 pp while preserving the IT > SW ≥FR ordering.E2 refusals were 57/∼5,300 ≈ 1.08%, versus 64/∼5,300 ≈ 1.21% for E3.

5 Discussion and Conclusion

The audit finds that keyword search is the dominant harm-exposure pathway, while passive algorithmic exposure varies sharply across countries, with Italy highest at every age. MLLM auditing scales cross-national youth-safety audits, but modality differences and uncertain Stage-1-to-Stage-2 validity remain important caveats.

  • Methodological caveats: κ = 0.42 was measured only on the 300-video reference sample, so transfer to the full Stage-2 distribution remains unanswered.Native-video E2 consistently reported higher harm rates than primary E3 by +3–12 pp, and the closer modality to human truth cannot be determined without Stage-2 annotation.
  • Conclusion: Italy had the highest measured harm rate across all four age personas under E3 and passive scrolling, with Italian age-19 reaching 48.6%.Within-country age gradients ran in opposite directions across the three countries, and the age-40 Italian result shifted under recalibration.
  • Exposure pathways: 1.5–7.5× within-account scroll-pre→SEARCH lifts show that keyword search drives sharply elevated exposure, followed by near-complete scroll-post reversion.The FR/SW scroll-pre age gradient was absent at SEARCH, but the design cannot distinguish age-insensitive retrieval from overlapping returned content pools.
  • Conclusion: 35–56% harmful content was returned by harm-keyword queries, making keyword search the dominant exposure pathway in this audit.The conclusion reports no visible search-time friction for these queries.
  • Conclusion: $50 API costs of order enabled MLLM-based auditing to scale across national youth-safety audits.The conclusion presents this scalability as a methodological contribution alongside the exposure findings.

6 Limitations … System prompt (all conditions, all models)

The study’s limitations concern non-ground-truth reference labels, unverified auditor transfer, refusal-related undercounting, time-bounded collection, and simplified sockpuppet behavior. Appendices document fixed native-language keyword probes and shared multimodal prompts that define the harm framework and output format.

  • 6 Limitations: Reference labels are guideline-grounded judgments from two native-speaker annotators per country, not statistical ground truth or observable TikTok enforcement thresholds.Annotators were not professional content moderators and jointly resolved the Stage-1 labels.
  • 6 Limitations: Stage-2 harm rates are Gemini E3-as-auditor estimates whose transfer from the 300-video Stage-1 reference to the full distribution is unverified.The reference draw is stratified by country and age, but not phase, so phase-dependent auditor error cannot be ruled out; E2-vs-E3 agreement is only an internal-consistency check.
  • 6 Limitations: ∼1.1% of Stage-2 cases were provider-side refusals, which cluster on the most-explicit harm categories and can undercount those harms.Stage-1 Qwen/DashScope failures are additionally reported in Appendix E.
  • 6 Limitations: The single, time-bounded collection window and continuously adapting recommender limit generalization to other periods and uninstantiated personas.Programmatically controlled sockpuppets also lack multi-device use, scripted cross-session continuity, and an organic social graph; findings are therefore framed as upper-bound claims about what the algorithm can serve.
  • A Keyword Lists: Seven probed harm categories each used three queries: a literal severity-axis anchor, a moderate variant, and a platform-vernacular phrase.English seed terms were drafted from the taxonomy definitions, while searches used the persona country’s native language.
  • B Prompt Templates: The same system prompt was shared across four MLLMs and three input conditions, with only the user-message structure varying.The template was reused at Stage-1 and Stage-2 so Stage-1 κ calibration could transfer without prompt-induced drift.
  • System prompt (all conditions, all models): The prompt instructs models to classify videos as harmful or not harmful under a Community-Guidelines-aligned framework, considering visual content, audio, on-screen text, descriptions, and comments.Outputs require a single JSON object containing a verdict, one subcategory or null, and brief reasoning; E1 is text-only, E2 uses native video, and E3 uses eight frames plus text.
  • A Keyword Lists: 21 native-language keywords per persona account were issued across the five-day active-phase SEARCH probe, with vocabulary fixed before data collection.The probe used three keywords per country and harm category; six taxonomy categories were not targeted because short queries lacked reasonable precision.

C Dataset Statistics

For-You-feed items receive substantially higher play counts than search-surfaced items across all three countries, typically by an order of magnitude on the median. This engagement gap is stable across ages and both scroll-pre and scroll-post phases, consistent with differing subsystem logics.

  • Phase-level engagement gap: Across all three countries, For-You-feed items have substantially higher play counts than search-surfaced items.The gap appears in the phase-level engagement comparison.
  • Phase-level engagement gap: Typically by an order of magnitude on the median, For-You-feed play counts exceed those of search-surfaced items.The comparison is reported as a median difference rather than an average.
  • Phase-level engagement gap: The effect is stable across ages and appears at both scroll-pre and scroll-post.This stability suggests the gap is not an artifact of a single scrolling phase.
  • Phase-level engagement gap: The For-You feed optimizes for watch-time virality, whereas search retrieves a long-tail query-matched pool.These differing economic logics are offered as the explanation for the engagement gap.

D Stage-2 Supplementary Figures and Tables

Supplementary analyses show that provider-block uncertainty does not alter the country ordering or Italy’s ceiling effect, while top-ups were used to stabilize sparse scroll comparisons. SEARCH/scroll-pre ratios remain elevated on the original draw, though some Italy-age cases are undefined because of zero harmful items.

  • Provider-block robustness: 1.80% of E3 inputs were block-or-parse-fail items, yet worst-case intervals preserve IT > SW ≥FR and the IT-13 / IT-16 ceiling effect.The intervals are [26.5%, 28.4%] for France, [35.0%, 36.5%] for Italy, and [27.5%, 29.5%] for Sweden; IT-16 shifts only to [39.9%, 41.1%].
  • Sample top-up: 3,862 items were initially drawn, but 3,568 available MP4s were fed to Gemini; sparse scroll cases were topped up to approximately 100 items each.Scroll-pre and scroll-post initially held approximately 17 records each, compared with approximately 150 for SEARCH, so both phases were topped up from matched accounts, days, and FYP snapshots.
  • Ratio robustness: 1.2× to 7.4× were the original-draw SEARCH/scroll-pre factors across ten combinations, versus 1.0× to 8.1× after top-up.IT-19 and IT-40 were undefined on the original draw because n_scroll-pre ∈[15, 17] yielded zero harmful items.

Account-clustered bootstrap intervals

Account-clustered hierarchical bootstrap intervals preserve between-account correlation by resampling accounts and then videos within accounts. Account attribution covers 99.3–99.8% of active records, and clustered intervals closely match video-level intervals with a median width ratio of 1.00.

  • Bootstrap method: Hierarchical bootstrap resampling preserves between-account correlation by resampling the three accounts per cell, then videos within each drawn account.A video captured by two accounts contributes to both accounts’ pools.
  • Account attribution: 99.3–99.8% of Stage-2 active records per phase have account attribution.Videos captured by two accounts contribute to both accounts’ pools.
  • Interval comparison: 1.00 is the median clustered-to-video-level interval width ratio across the 36 E3The account-clustered intervals match the video-level intervals closely.

E Stage-1 Diagnostic Detail

Stage-1 diagnostics use a 300-video, two-annotator native-speaker reference subset to compare four MLLM families with an XLM-R text baseline. Gemini 2.5 Flash shows modality-monotonic performance, whereas Qwen3-VL-32B remains flat between E2 and E3 at weak agreement.

  • Reference subset: 300 videos formed the two-annotator reference subset, yielding 299 binary reference verdicts and 80–99 items per combination depending on model failures.Diagnostics were pre-Stage-2 and included four MLLM families alongside the XLM-R supervised text baseline.
  • Annotator agreement: 80.3% binary agreement preceded resolution, with aggregate Cohen’s κ = 0.54 and per-country κFR = 0.42, κIT = 0.65, and κSW = 0.48.All 59 binary-verdict disagreements were settled in a joint resolution session producing one consensus label per video.
  • Modality monotonicity: Gemini 2.5 Flash’s Cohen’s κ and macro-F1 move monotonically across the three input conditions on the 300-video Stage-1 subset.Figure 12 reports the per-condition numerics inline in §4.1.
  • Modality monotonicity: κ ≈0.18: Qwen3-VL-32B is flat between E2 and E3, unlike Gemini 2.5 Flash’s monotonic pattern.The passage attributes this to native-video and frame-based pathways converging on the same weak judgment rather than complementing one another.

Per-country agreement of the winner

The Stage-1 winning configuration achieved the highest agreement on Italian content and the lowest on French content. Relative to country-specific consensus-quality ceilings, its agreement ratios were broadly similar across France, Italy, and Sweden, while recalibration preserved most country rankings but changed the age-40 ordering.

  • Per-country agreement: κIT = 0.463, κSW = 0.356, and κFR = 0.290, ranking Italian agreement highest, Swedish intermediate, and French lowest.The country-specific confidence intervals were [0.29, 0.63], [0.16, 0.54], and [0.03, 0.52], respectively.
  • Per-country agreement: 69%, 71%, and 74% of the consensus-quality ceiling were reached on French, Italian, and Swedish content, respectively.These ratios were computed as 0.290/0.42, 0.463/0.65, and 0.356/0.48, indicating similar relative performance across countries.
  • Per-country recalibration: Per-country precision/recall recalibration preserved Italy’s highest harm-rate ranking at ages 13, 16, and 19 but inverted the age-40 ordering.The correction used country-specific Stage-1 binary-harm precision and recall: France 0.50/0.33, Italy 0.80/0.60, and Sweden 0.62/0.51.
  • Cross-country model asymmetry: κFR = 0.31 versus 0.08 was the only country split where Qwen E2 exceeded Qwen E3; Italian moderate-agreement cases included Gemini E2 at κIT = 0.51.The aggregate grid averages over substantial cross-lingual asymmetry in model-condition performance.
  • Winner confusion: 48 false negatives versus 24 false positives show that Gemini E3 was conservative on harm relative to the consensus reference.Among 52 jointly harmful items, primary-subcategory agreement was 29% (15/52) under the reported strict definition.

Country-specific harm-category mix

Harm-category composition differs markedly by country: France is dominated by dangerous activity and challenges, while Italy and Sweden are dominated by nudity, body exposure, and sexually suggestive content. Sweden also shows greater visibility of hate speech and harassment than the other countries, making country-specific category prevalence necessary alongside overall agreement.

  • Country-specific harm-category mix: France’s harmful videos are dominated by dangerous activity and challenges, whereas Italy and Sweden are dominated by nudity, body exposure, and sexually suggestive content.The comparison uses within-country normalized category shares while retaining absolute country counts.
  • Country-specific harm-category mix: Hate speech and harassment are more visible in Sweden than elsewhere.These categories are reported as part of the cross-country asymmetry in harmful-content distributions.
  • Country-specific harm-category mix: 102 reference-flagged harmful videos form the basis for the country-to-harm-subcategory comparison.Figure 16 shows ribbon widths proportional to absolute counts and routes within-country normalized category shares.
Loading 2608.17583v1…