PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Framework for Measuring Creator Communication on Short-Form Video

Julie R. Ancis, Michael Fire, Omer Tarshish

arXiv:2610.33043v1cs.SIcs.CLcs.CY

TL;DR

Short-form video carries meaning across speech, imagery, and on-screen text, while common detection approaches do not measure creators’ broader repertoires. This paper applies a modular, five-stage multimodal framework to creator archives; its coding reaches expert-comparable agreement overall, though agreement varies by indicator.

  • Problem

    Measuring creator repertoires is difficult because single-label classifiers miss multiple themes and strategies, while short-form video communicates across modes.

    Source: §I · p. 2, §II-A · p. 3

  • Method

    A five-stage framework combines metadata, transcripts, and frame descriptions, applies fixed and open-ended LLM coding, maps labels to categories, and aggregates prevalence equally across creators.

    Source: §I · p. 2, §III · p. 4

  • Results

    The framework reached trained-annotator agreement at scale; κ = 0.42 between annotators, versus 0.37 and 0.43 for model–annotator agreement, with news commentators and entertainment creators emerging in the analysis.

    Source: §VI · p. 12, §VII · p. 14

  • Takeaways & Limitations

    The framework provides a way to compare creator communication across complete archives and over time, including what influencers say and how they say it.

    Source: §VII · p. 14

    Agreement is not uniform across indicators, so validation strength varies by measure.

    Source: §VI · p. 12

Abstract

from arXiv · show

Short-form video combines speech, imagery, and on-screen text, creating challenges for systematic content analysis. We present a five-stage multimodal framework for measuring creators' themes and communication strategies at scale. The framework integrates platform metadata, speech transcripts, and vision-language descriptions of sampled frames, then applies large language models to generate predefined indicators and open-ended labels. An audited keyword scheme consolidates thematic labels, while creator-weighted aggregation gives each account equal weight regardless of posting volume. We demonstrate the framework on 9,852 TikTok videos from sixteen creators, retaining 7,311 videos for analysis across seven themes and seven communication indicators. Automated profiles reveal substantial differences in creators' use of education, humor, and responses to current events. Validation against two independent expert annotators yielded a mean framework-expert agreement of Cohen's $\kappa = 0.40$ across the six binary indicators, close to the $\kappa = 0.42$ the two experts achieved with each other, though agreement varied substantially across individual indicators. The framework provides a modular approach to comparative analysis of creator communication, combining scalable annotation with transparent aggregation and construct-specific validation.

I. Introduction

Short-form video’s multiple modes make creator communication difficult to measure at scale, while single-label detection misses creators’ broader repertoires. The paper introduces a five-stage framework and demonstrates it on advocacy creators, with automated coding showing expert-comparable agreement overall but variation by indicator.

I. Introduction

  • The motivating gap is that manual coding is costly, single-label classifiers miss creators’ topic and strategy repertoires, and short videos convey meaning across several modes.Captions or transcripts alone recover only part of a video’s message.Source: §I · p. 1
  • The framework combines metadata, speech transcripts, and visual descriptions with fixed indicators and open-ended thematic coding, then aggregates prevalence equally across creators.Its five stages collect archives, enrich videos, classify content, construct a taxonomy, and normalize and aggregate creator-level measures.Source: §I · p. 2
  • Applied to sixteen creators countering antisemitism, the study asks which themes and strategies they use and how much model coding can be trusted.The framework takes target accounts, indicator questions and response scales, and master categories as study inputs.Source: §I · p. 2
  • κ = 0.40 mean model–expert agreement across six binary indicators, versus κ = 0.42 between experts; agreement varied across indicators.For humor or sarcasm, agreement with one expert was 0.70 versus 0.71 between experts; for the October 7 category, agreement with one expert was 0.75 across 92% of videos.Source: §I · p. 2
  • Across 7,311 retained videos, the study reports substantial variation in creators’ themes and strategies, including educational content, responses to current events, and humor.Education was the most common communication strategy; two recurring profiles emphasized teaching and news tracking or entertainment and humor.Source: §I · p. 2

II. Related Works

Related work motivates measuring multimodal creator repertoires rather than predicting a single label, and pairing fixed questions with open-ended coding. Influencer credibility and counterspeech research also inform the framework’s unit of analysis and constructs.

II-A Computational analysis of multimodal social

  • Multimodal analysis captures how text, images, video, and audio combine, while single-label classifiers do not characterize many creator practices across a population.Prior work reports that combining visual and textual cues improves accuracy over text-only methods.Source: §II-A · pp. 2–3
  • Zero-shot LLM coding has performed well on some labeling tasks [23], but benchmark results find it behind fine-tuned classifiers on taxonomic labeling.The paper’s design pairs a closed pass for comparability with an open pass for inductive description, at some cost in accuracy.Source: §II-A · p. 3

II-B Influencer narratives: the demonstration case

  • Influencer research links repeated personal interaction with transferred credibility and trust, and connects creators with shifts in attitudes, advocacy, and political action.The literature describes creators as potential interventions against online hate.Source: §II-A · p. 3
  • Counterspeech challenges harmful narratives through factual correction, hypocrisy exposure, warnings, emotional appeals, denunciation, or humor.Evidence cited here associates exposure to counterspeech with greater bystander willingness to speak up, while perpetrator response depends partly on the speaker [41].Source: §II-A · p. 3
  • Antisemitism appears in multiple national contexts and rose after October 7, 2023; the open question is how influencers might counter hate using reach and credibility.The cited material reports that 73% of American Jews encountered antisemitism online.Source: §II-A · p. 3

II-A Computational analysis of multimodal social

  • The framework reports prevalence per creator because credibility and visibility shape narrative reach [41], and combines closed comparisons with open thematic discovery.Its fixed questions cover part of the counterspeech repertoire, while leaving elements such as hypocrisy exposure and warnings of consequences outside scope; transcript and frame descriptions are motivated by evidence that visual and textual cues together outperform text alone.Source: §II-A · p. 3

III. Methods

The methods convert each creator’s video archive into comparable measures through five sequential, modular stages. Fixed indicators support cross-creator comparison, open labels add thematic depth, and audited taxonomy mapping and creator-weighted aggregation make those outputs countable and comparable.

III. Methods

  • The five-stage framework is modular: speech-to-text, vision-language, and classification models can each be replaced without changing the analytical logic.Each stage passes a named product to the next, from archive and metadata to creator-weighted prevalence.Source: §III · p. 4, Figure 1 · p. 5
  • Collection retrieves public videos and metadata, removes duplicates by platform identifier, and enrichment combines a metadata header, speech transcript, and sampled-frame descriptions into one document.Frame descriptions use structured prompts and chronological timestamps; carry-forward summaries help maintain consistency across frames.Source: §III · p. 4
  • The closed pass applies fixed, literature-based indicators for comparable coding, while the open pass produces short free-form theme and strategy labels for inductive discovery.The open pass avoids restricting outputs to a predefined label set; the two passes provide complementary fixed-dimension comparison and thematic depth.Source: §III · pp. 4–5
  • Taxonomy construction pools and ranks labels, combines deductive categories with recurrent inductive ones, then maps labels to master categories through audited keyword stems.An iterative audit adds missed stems and constrains overly broad ones; the resulting map is frozen and assignments remain traceable to their triggering stem.Source: §III · p. 5
  • Category assignment is non-exclusive, so a video can map to multiple master categories; videos with only out-of-domain labels are excluded from the analytic corpus.The analytic corpus retains videos whose labels map to at least one substantive category.Source: §III · p. 5
  • Each video is reduced to binary category presence, each creator’s prevalence is calculated, and the primary corpus statistic averages those creator proportions equally.This prevents prolific accounts from dominating; creator-level dispersion and video-weighted pooled prevalence are also reported, with the latter serving as a robustness check.Source: §III · p. 6
  • The same normalization applies to closed indicators; ordinal indicators use mean scores, and creator-level descriptive statistics accompany the normalized measures.The validation compares blinded, independent human coding and model coding using raw agreement and Cohen’s κ for binary items, plus within-one-point agreement and Spearman correlation for emotional intensity.Source: §III · p. 6

IV. Experiments

The framework was demonstrated on public TikTok archives from 16 advocacy and counter-antisemitism creators, with seven stages instantiated for the study and outputs checked against expert annotations and reduced-document variants. The experiments describe account selection, corpus construction, measurement, aggregation, and validation.

Account selection; Collected corpus

  • Sixteen purposively selected accounts yielded 9,852 videos from April 3, 2018 to June 4, 2025, after seven of 23 candidates failed archive-access screening.The accounts were selected to span content domains, advocacy styles, and creator backgrounds.Source: §IV · pp. 6–7

Collected corpus

  • The archive contained 155 hours of footage, with a median video length of 43 seconds and 3,277 videos posted in 2024.Median plays were 28,400 versus a mean of 270,342; median engagement rates ranged from 3.4% to 12.0% across accounts.Source: §IV · p. 7

Pipeline instantiation

  • The instantiated pipeline transcribed audio with Whisper large-v3, described sampled frames with Qwen2.5-VL 7B, and classified unified documents with GPT-5.1.Comments were excluded; long documents were capped by subsampling frame descriptions, and classification used structured JSON output.Source: §IV · p. 7
  • Seven retained closed-question items comprised six binary indicators with Yes rates of 33.6%–60.1% and one ordinal emotional-intensity item; eleven others were omitted for too few positive cases to compare creators.Source: §IV · p. 7

Aggregation

  • Indicator prevalence counted only Yes responses, treating No and Unclear as absence; emotional intensity was averaged within each creator.Creator-weighted mean prevalence, between-creator standard deviation, and video-weighted pooled rate were reported as CW, SD, and VW.Source: §IV · p. 7

Human validation; Modality ablation

  • Two blind annotators independently coded a creator-stratified sample of 100 videos, and three reduced-document variants tested the contribution of frames, transcripts, and metadata.Thematic annotation tested both open labeling and keyword mapping, while indicator questions matched the model’s response options; one metadata-only noncompletion left 99 videos in all four versions.Source: §IV · pp. 7–8

V. Results

Framework–expert agreement was near expert–expert agreement on average, but varied by indicator and showed systematic upward bias. Creator profiles differed in themes and strategies, while enrichment and weighting affected the resulting estimates.

V. Results

  • Mean κ was 0.37 and 0.43 against the two annotators, bracketing their 0.42; framework κ exceeded expert–expert κ on four of six indicators.Agreement on October 7 / Hamas / Hostages was highest among thematic categories, with κ of 0.75 and 0.70.Source: §V · pp. 8–9, Table III · p. 9
  • The framework systematically overestimated theme presence and emotional intensity, yet creator-level intensity rankings aligned better with annotators than video-level rankings.Spearman correlations were 0.67 and 0.50 for creator-level means, versus 0.55 and 0.34 for individual videos.Source: §V · p. 9
  • Removing both transcript and frame descriptions lowered mean κ from 0.40 to 0.32, a 21% decline; removing frames alone retained nearly all agreement.The clearest losses concerned items depending on what a video says or shows.Source: Table V · p. 9, §V · p. 9
  • Enrichment recovered more themes identified by annotators—88 and 64 correct assignments versus 71 and 53 with metadata alone—but also increased false positives.Because reported thematic prevalence is an upper bound, the authors prioritize the recall gained over the precision cost.Source: §V · pp. 9–10
  • Geopolitics, Jewish Identity, and Israel Advocacy were the leading themes at roughly half the corpus each, while interfaith content was 13.3%.Jewish Identity was broadly distributed, whereas event-driven themes concentrated in news-oriented accounts.Source: Table VI · p. 10, §V · p. 10
  • Six news-driven educator accounts paired education with current-events responsiveness, while entertainment-native creators built advocacy on humor; eitanthegoalie used humor in 99.2% of videos.Creator-level emotional intensity also diverged, from 2.43 for yosephhaddad to 0.92 for motiankari.Source: §V · p. 10
  • Educational content led the binary indicators at 58.3% creator-weighted prevalence, followed by responses to current events at 46.4%.Current-events prevalence was 57.3% video-weighted, indicating that higher-volume creators responded to current events more often.Source: Table VII · p. 10, §V · p. 10
  • Creator weighting reduced current-events prevalence from 57.3% to 46.4% and emotional intensity from 1.93 to 1.78, equalizing creators despite sharply different posting volumes.The relevance restriction retained 7,311 of 9,852 videos (74.2%), with retention varying from 23.3% to full retention across creators.Source: §V · pp. 8, 10

VI. Discussion and limitations

Across the sixteen creators, two communication types recur across topics, strategies, and emotional intensity, while creator-level profiles reveal differences obscured by averages. The framework processes complete archives at scale, but agreement and thematic prevalence have important construct- and mapping-specific limits.

VI. Discussion and limitations

  • Closed-question agreement was κ = 0.42 between annotators and κ = 0.37 and 0.43 between the framework and each annotator, while emotional-intensity rankings correlated at ρ = 0.67 and 0.50.Agreement varied by indicator; the framework's open-question agreement was 75.1% and 74.0%, versus 87.8% between annotators.Source: §VI · p. 12
  • Automated analysis covered nearly 10,000 videos and about 155 hours, avoiding an estimated 600 hours of manual coding; creator-level weighting prevents prolific accounts from dominating prevalence estimates.In this corpus, response-to-current-events prevalence fell from 57.3% to 46.4% when creators received equal weight.Source: §VI · pp. 11–12
  • For closed questions, the full multimodal document reached κ = 0.40, versus 0.32 with metadata alone; transcripts and frame descriptions partly compensated for one another when either was absent.Without frame descriptions agreement remained 0.40, and without the transcript it was 0.36; frames may matter more for visual or detailed questions.Source: §VI · p. 12
  • For documents exceeding 20,000 characters, frame descriptions were subsampled to at most 30 frames, so the model saw only part of the frames in long videos.The authors suggest testing frame selection based on scene changes or new information to preserve important content while keeping documents short.Source: §VI · p. 12
  • Six news-driven educators pair high educational content with current-events responsiveness, while entertainment creators use humor; the five highest- and lowest-intensity creators fall into those respective groups.emilyintelaviv leads the educator group at 91.9% educational content and 83.8% responses to current events; eitanthegoalie uses humor in 99.2% of videos.Source: Figure 2 · p. 11, Figure 3 · p. 11, §VI · p. 13
  • After October 7, 2023, the share of videos addressing the seven themes rose from 55.9% to 89.9%; humorous videos received more views than others for 13 of 14 creators, without establishing causality.Emotional intensity also rose after October 7 for each of the fifteen creators who posted both before and after that date.Source: §VI · p. 13
  • The analysis covers sixteen non-randomly selected accounts in one domain and on TikTok, so its findings cannot be generalized to all creators; observed relationships do not explain why creators behave as they do.Topic prevalences are upper bounds because keyword mapping over-assigns, and the 100-video validation set cannot establish accuracy for each creator individually.Source: §VI · pp. 13–14

VII. Conclusions and future work

The paper presents a modular framework for profiling creator communication across complete video archives, combining multimodal inputs with open and closed language-model questions. Its application demonstrates scalable human-level agreement and reveals differences across creators and over time.

VII. Conclusions and future work

  • The framework combines creators’ archives, speech, on-screen content, and metadata, then uses open and closed questions to produce comparable profiles.Source: §VII · p. 14
  • Close to 10,000 videos from sixteen creators were analyzed; the framework reached trained annotators’ agreement level at scale, and modality ablation favored combining modalities.The analysis retained complete creator archives for measurement.Source: §VII · p. 14
  • Profiles distinguished news commentators from entertainment creators and showed sharp post–October 7, 2023 shifts in content and emotional intensity.Source: §VII · p. 14
  • Its design pairs complete archives and a unified multimodal document with open label discovery and consistent closed-question measurement.Source: §VII · p. 14
  • Future work can extend the framework across populations and platforms, use more frames, audio, or other models, and add repeated snapshots with larger validation samples.Source: §VII · p. 14
  • The authors conclude that systematic measurement across archives and over time can illuminate what influencers say, how they say it, and how it changes.Source: §VII · p. 14

Code Availability

The analysis-stage code and links to analyzed videos are slated for public release, but the video corpus itself will not be redistributed.

Code Availability

  • Upon publication, code for Stages 2–5, including prompts, indicator definitions, and the keyword-scheme loader, will be public and model-replaceable.Source: Code Availability · p. 14
  • Links to analyzed videos will be made available upon publication, but the video corpus will not be redistributed.Source: Code Availability · p. 14
  • The corpus is withheld because it is third-party content subject to platform terms, and creator archives can change as videos are edited or withdrawn.Source: Code Availability · p. 14

Appendix A. Vision–language frame-description prompt (Stage 2)

Stage 2 prompts the vision–language model to produce structured frame descriptions rather than default captions, with detailed content and marketing-oriented instructions.

Appendix A. Vision–language frame-description prompt (Stage 2)

  • Each sampled frame is paired with an instruction eliciting a structured natural-language description rather than a default caption.Source: Appendix A · p. 16
  • The frame instruction requests thorough analysis of people, themes, visual elements, visible text and branding, marketing signals, and cultural context.Source: Appendix A · p. 16
  • Prompts include the frame’s index and timestamp; later frames also include recovered person, brand, and setting features to encourage consistency.Generation used temperature 0.05, top-p 0.92, top-k 40, a 700-token limit, and cubic-interpolation resizing.Source: Appendix A · p. 16

Appendix B. Classification prompts and response schema (Stage 3)

Stage 3 applies open- and closed-question passes to the same unified video document, producing creator-topic labels and structured strategy indicators.

Appendix B. Classification prompts and response schema (Stage 3)

  • The open-question pass requests concise, distinct labels for creators’ topics and communication strategies, with each label limited to five words.Source: Appendix B · p. 16
  • The reported closed questions cover storytelling, education, current events, calls to action, humor, strategic hashtags, and emotional intensity.Source: Appendix B · p. 16
  • Eleven additional closed questions include trends, series, statistics, green-screen use, and collaborations; they were omitted from tables because too few videos received Yes.Source: Appendix B · p. 16
  • Strict structured-output schemas require every field and restrict closed responses to specified categories, yielding syntactically valid, complete records.Source: Appendix B · pp. 16–17

Appendix C. Taxonomy construction (Stage 4)

Stage 4 maps open-ended labels to categories with keyword stems, refining the stems through an auditable loop before freezing them for corpus-wide use.

Appendix C. Taxonomy construction (Stage 4)

  • The mapping assigns labels to categories when a category’s keyword stem occurs in the label, using case-insensitive substring matching by default.Stems are truncated base forms that can capture morphological variants; word-boundary anchoring is added selectively during refinement.Source: Appendix C · p. 17
  • The refinement loop assigns labels, records each triggering stem, and inspects assignments ordered by label frequency and grouped by stem.The loop takes the corpus vocabulary and video frequencies as input and produces frozen stem sets.Source: Appendix C · p. 17
  • Systematic misses are corrected by adding stems, while spurious substring matches are constrained with word-boundary anchoring.For example, an art stem can be restricted to the isolated word rather than matching fragments in “start,” “party,” or “smart.”Source: Appendix C · p. 17
  • Refinement stops when another pass makes no material change, after which the category definitions and keyword sets are frozen and applied unchanged across the corpus.Retaining the definitions and triggering stems makes the mapping deterministic and reproducible for the same inputs.Source: Appendix C · p. 17

Appendix D. Normalization and aggregation (Stage 5)

Stage 5 converts mapped labels into per-video category presence, calculates category prevalence within each creator, and reports an equal-weight average across creators.

Appendix D. Normalization and aggregation (Stage 5)

  • The aggregation starts with videos indexed by creator and mapped categories, then processes each creator-category pair.Source: Appendix D · p. 17
  • Binary presence is used for each video and category to remove redundancy among labels.The procedure then computes a category prevalence for each creator from that creator’s videos.Source: Appendix D · p. 17
  • The primary statistic averages each creator’s category prevalence across the K creators, giving each account equal weight.The framework also reports the standard deviation of creator-level prevalence across creators.Source: Appendix D · p. 17
Loading 2610.33043v1…