Source-linked AI summary
Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Zhixi Zhu, Kristina Gligoric
TL;DR
Cost-saving heuristics for VLM video annotation have not been comprehensively evaluated for both annotation quality and downstream inference validity. This paper tests these choices on sentiment and topic classification, finding that accuracy can diverge from inferential validity and that a compact grid can approach full-video understanding at a fraction of the token cost.
Problem
It remains unclear which cost-saving video-annotation heuristics preserve annotation quality and downstream conclusions, since prior evaluations have not comprehensively tested their interacting choices and inferential validity.
Source: §1 · p. 1, §2.2 · p. 3Method
The study systematically evaluates frame sampling, image-grid composition, and modality choices on MELD sentiment and TikTok topic classification, assessing accuracy, downstream inferential validity, and token cost.
Source: §1 · p. 1Results
Accuracy and downstream validity can diverge, modality value depends on task, and a 2 × 8 grid approaches Gemini full-video understanding (κ within .05) at 14.8% of per-video token cost.
Source: §1 · pp. 1–2, §5 · p. 8Takeaways & Limitations
A small pilot comparing the best unimodal and multimodal annotations can help assess whether added modalities are worth their cost in the studied tasks.
Source: §5 · p. 8The study covers two datasets of 900 videos each with constrained label spaces, so generalization to larger corpora, longer videos, and finer-grained or multi-label tasks remains open.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single $2\times8$ image grid built via simple shot-transition detection approaches full-video understanding ($\kappa$ within~.05), at $\sim 15\%$ of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.
1 Introduction
The introduction frames cost-saving VLM annotation as a joint problem of reducing token costs while preserving annotation quality and downstream conclusions. It evaluates frame selection, grid composition, and modality choices, then highlights accuracy–validity divergence, task-dependent modality value, and efficient grid compression.
Research gap
- Researchers lack a comprehensive evaluation of how frame sampling, grid composition, and modality interact in cost-saving video annotation.The introduction also asks whether these heuristics preserve accurate downstream conclusions, not merely annotation quality.Source: §1 · p. 1
Contributions
- The evaluation benchmarks cost-saving heuristics on MELD sentiment, TikTok topic classification, and comparison with full-video understanding.The three contribution areas are benchmarking, quantitative insights, and evidence-based recommendations for choosing compression, modalities, and human-label budgets.Source: §1 · pp. 1–2
Quantitative insights
- On MELD, effects are missed in multiple configurations, and the highest-accuracy configuration reverses the ground-truth coefficient sign; TikTok estimate errors can also be large.These findings show that annotation accuracy and the validity of downstream inference can diverge.Source: §1 · pp. 1–2
- A single 2 × 8 image grid approaches Gemini full-video understanding (κ within .05) at 14.8% of per-video token cost.The grid uses shot-transition detection, and the comparison is at fixed resolution.Source: §1 · p. 2
- Modality value depends on the task: text-only performs strongly on dialogue-driven sentiment, while multimodal grids win when TikTok topic cues are complementary.The introduction contrasts dialogue-driven and topic-classification settings.Source: §1 · p. 2
2 Background
The background distinguishes video-annotation accuracy from the validity of downstream statistical inference. It positions the paper against prior work on compact video inputs, inference repair, and annotation costs, emphasizing that cost-saving multimodal heuristics remain underexamined in CSS applications.
2.1 VLMs
- Prior compact-grid and keyframe-sampling studies primarily evaluate video question-answering correctness, whereas this paper evaluates annotations as inputs to downstream CSS analyses.The background identifies inferential validity, not just direct accuracy, as relevant to these applications.Source: §2.1 · p. 2
2.2 Inferences based on VLM annotations
- PPI and DSL combine a small set of human labels with many model predictions to produce valid confidence intervals, but their reliability for heuristic-compressed video remains open.Prior applications of these repair frameworks have focused mainly on static images and text.Source: §2.2 · pp. 2–3
2.3 Cost in downstream applications
- Although prior work audits token costs, model choice, and compute allocation, the implications of multimodal annotation heuristics for downstream inference are rarely examined.The paper presents its study as an empirical basis for such applications.Source: §2.3 · p. 3
3 Methods
The methods compare frame-sampling, grid, modality, and bias-correction choices across sentiment and topic tasks. The study uses shot-transition sampling and image grids, applies PPI to downstream estimates, and compares LLM configurations with machine-learning and unimodal baselines.
3.1 Evaluation setup
- The recommended evaluation setup combines shot-transition sampling, image grids, and bias correction, with other configurations treated as ablations.The study varies frame sampling, grid composition, modality subsets, and whether PPI correction is applied.Source: §3.1 · p. 3
3.2 Datasets
- The evaluation uses MELD sentiment labels and TikTok creator-assigned hashtag labels, with distinct downstream estimands for each dataset.MELD uses 900 short Friends clips and studies sentiment association with cast role; TikTok uses 18 hashtags and estimates their frequencies.Source: §3.2 · pp. 4–5
3.3 Metrics
- Performance is measured with accuracy and Cohen’s kappa, while per-video costs are estimated from tokenizer counts.Accuracy captures correct predictions; kappa measures agreement beyond chance, with per-class and micro-average accuracy also computed.Source: §3.3 · p. 5
3.4 Baselines
- The baselines include Random Forest classifiers on video embeddings alone or video-plus-transcript embeddings, and LLMs given transcripts or frames alone.The unimodal LLM settings isolate the contribution of each modality.Source: §3.4 · p. 5
3.5 Models
- GPT-4o-mini is the main LLM, with Claude Sonnet 4.6, GPT-5.1, and Grok 4.3 also tested.LLM prompts use temperature 0.3 and specify task context, input format, instructions, labels, and JSON output.Source: §3.5 · p. 5
4 Results
The results compare video-annotation configurations across classification accuracy, downstream inferential validity, and token cost. They show that accuracy alone can mislead, modality value depends on the task, and image grids can reduce cost while retaining strong performance.
4.1 Accuracy and validity diverge
- MELD’s top-accuracy Text-only configuration (.610, κ=.398) still yields a significant Negative coefficient opposite ground truth (β=−.43, p=.038 vs. +.27, p<.001), showing accuracy can mislead inference.The authors conclude that accuracy and validity rankings disagree and that bias correction is required.Source: §4.1 · pp. 5–6, Table 1 · p. 6
4.2 Modality value is task-dependent
- MELD favors Text-only (.610) over multimodal variants, whereas TikTok multimodal grids (.819–.834) beat Text-only (.713) and Frames-only (.712), making modality value task-dependent.The authors attribute the difference to dialogue carrying the sentiment construct on MELD and complementary visual and textual cues for TikTok hashtags.Source: §4.2 · pp. 6–7
4.3 Cost can be decoupled from video length
- On TikTok, grid inputs average 36,596–47,665 tokens per video versus 2,275,411 for 1-FPS sampling, whose cost grows with video length.The table reports grid sampling at roughly 37–48k tokens and about 50× lower average cost than 1-FPS sampling.Source: §4.3 · p. 7, Table 2 · p. 7
- A single ShotT 2×8 grid uses 2,332 Gemini tokens per video versus 15,774 for full-video comprehension (14.8%) while matching aggregate accuracy and agreement (.870, κ=.860).With Gemini held fixed, the grid matches full-video understanding in aggregate; GPT-4o-mini achieves .820 (κ=.806) with the same grid representation.Source: §4.3 · p. 7
- Sampling performance depends on shot density: MELD shows no significant ShotT–uniform difference at matched grid size, while TikTok uniform 3×3 (.834) slightly beats ShotT 3×3 (.827).The authors report that targeting transitions adds little once uniform sampling covers enough shots, and that annotation accuracy is lowest for very short TikTok videos.Source: §4.3 · p. 7
5 Discussion
Accuracy can improve while downstream inference worsens, and the value of each modality depends on the task. The discussion translates these findings into pilot-based choices for annotation configurations, costs, and statistical correction.
5 Discussion
- Accuracy can improve while inference worsens: text-only sentiment exploits cast-role linguistic cues, producing a wrong-sign coefficient that example-averaged accuracy misses.An accuracy-only model-selection procedure can therefore lead to inaccurate downstream inference.Source: §5 · pp. 7–8
- Unimodal sufficiency pilots can guide modality choice: single-channel constructs may not need other modalities, while distributed cues can reward combining them.The authors suggest testing this on a small pilot rather than assuming multimodality is useful.Source: §5 · p. 8
- A grid pays one token cap regardless of resolution or duration, while per-frame input pays per frame, making grid savings grow with duration after roughly 9 seconds.The cost-decoupling result was tested up to twelve minutes; the upper bound remains to be investigated.Source: §5 · p. 8
- PPI correction targets aggregate estimands, not individual predictions; corrected frequencies can move farther from reference, so report intervals and treat width as a bias-correction signal.The passage gives homedecor as an example, moving from −0.7 to −1.8 after correction.Source: §5 · p. 8
5.1 Practical guidelines
- For new tasks, pilot modality sufficiency, compare sampling strategies, and choose the smallest grid with acceptable κ rather than relying on defaults.The recommended procedure is derived from two short-form video tasks, and the authors say specific values should be re-estimated for other label spaces, durations, or genres.Source: §5.1 · pp. 8–9
- Gold-label budgets depend on the estimand: class-frequency estimates stabilized from about 100 labels, while regression coefficients required at least 200 in the reported sensitivity analysis.Apply PPI and report 95% confidence intervals; wide intervals can signal large model bias or an insufficient correction sample.Source: §5.1 · p. 8
Limitations
The evidence is bounded by dataset scale, label design, reference quality, and implementation assumptions. These constraints limit how broadly the reported findings and cost savings can be applied.
Limitations
- The evaluation covers two moderate-scale datasets of 900 videos each with constrained label spaces, and studies single-label rather than multi-label annotation.Whether findings generalize to larger corpora, longer videos, or finer-grained labels remains open.Source: Limitations · p. 9
- Dataset labels are noisy references: TikTok hashtags capture creator intent, not necessarily objective content, so accuracy and PPI validity are relative to those labels.The authors suggest pairing creator-assigned labels with content-grounded gold labels in future work.Source: Limitations · p. 9
- The 14.8% cost result depends on specific models, pricing assumptions, and API image-token policies, while multimodality’s task-dependent value may not generalize across datasets and models.The authors recommend empirical pilot testing rather than defaulting to text-only.Source: Limitations · p. 9
- The study tests a simple shot-transition heuristic; advanced keyframe selection may reduce costs further, but its own inference overhead should be counted.The passages also identify DSL and adaptive human-label sampling as possible extensions beyond PPI.Source: Limitations · p. 9
Ethical considerations
The authors flag risks from cheaper, more scalable annotation and distinguish their intended aggregate research use from individual-level monitoring. Their data practices also limit what is released and under which terms.
Ethical considerations
- Lower-cost annotation could enable surveillance, profiling, or individual tracking; the authors intend the methods for aggregate, population-level CSS analysis, not individual-level inference.They also warn that PPI does not guarantee uniform subgroup error rates and recommend auditing sensitive applications.Source: Ethical considerations · p. 9
- TikTok materials are public videos collected through the official Research API; the release contains video IDs and query lists, not raw media.The passage notes the API’s limited, non-exclusive research license and that MELD is available under GPL-3.0.Source: Ethical considerations · p. 9
A.1 Prompting
The prompting materials describe frame-grid and transcript inputs for MELD sentiment analysis and a grid-plus-transcript prompt for TikTok hashtag classification. The appendix also reports task-specific modality performance.
A.1 Prompting
- For MELD, the prompt pairs a chronological frame grid with the transcript and asks for character visibility before emotion and sentiment labels.If the character is absent, both labels are set to N/A.Source: §A.1 · pp. 11–12
- Table 3 reports task-specific modality performance: text-only leads on MELD, while multimodal grids outperform both unimodal baselines on TikTok.The caption identifies per-class accuracy across configurations for MELD sentiment and TikTok hashtag classification.Source: Table 3 · p. 12
- For TikTok, the prompt uses a chronological frame grid and transcript to choose one hashtag from the supplied list, favoring visuals when they conflict with text.The listed options span 18 hashtags.Source: §A.1 · pp. 12–13
A.2 PPI details
PPI separates configuration selection from final inference: a pilot selects the annotation setup, and a distinct human-labeled correction sample supports inference and uncertainty estimates.
A.2 PPI details
- A random 100-video pilot selects the configuration, while a separate random 100-video correction sample is reserved for PPI and excluded from model or configuration selection.The correction videos are drawn from the remaining data.Source: §A.2 · p. 13
- Figure 3 compares κ and cost across 9-frame 3×3, 16-frame 2×8, and 62-frame Gemini inputs; ShotT 3×3 exceeds κ=.80 at a fraction of the cost.Source: Figure 3 · p. 13
- For TikTok, PPI estimates each hashtag frequency separately; for MELD, it adjusts the full-data predicted-label logistic score using the correction sample’s human-versus-model score difference.Source: §A.2 · p. 13
- Bootstrap intervals resample the correction set while holding full-data predictions fixed, quantifying finite-correction-sample uncertainty for the observed dataset; results average 100 trials.Source: §A.2 · p. 13
A.3 Additional analyses
The additional analyses compare compressed-grid quality and token cost, report how accuracy and κ are presented across configurations, and show per-hashtag PPI intervals against ground truth.
A.3 Additional analyses
- The reported accuracy-and-κ comparisons cover 100 common videos, modality and sampling conditions with bootstrap CIs, and MELD and TikTok results across models.Source: Table 4 · p. 13, Table 5 · p. 14, Table 6 · p. 14
- Figure 4 presents per-hashtag PPI 95% confidence intervals against ground truth.Source: Figure 4 · p. 14, §A.3 · p. 14
- κ is only ~.05 below Gemini for both grids at roughly 14.8% of its token cost; 3×3 already exceeds κ=.80, and moving to 2×8 yields negligible quality gain.The results suggest diminishing returns from investing in more frames or a more expensive model for most CSS annotation tasks.Source: §A.3 · p. 14
A.4 All the models
Results across models show task-dependent modality value, while PPI sensitivity analyses find that larger human-annotation budgets improve coverage and reduce estimate variability.
A.4 All the models
- On MELD, text-only is competitive or best across four models; on TikTok, frames raise κ by 9–13 points over text-only, and sparse 3×3 grids are best where available.Source: §A.4 · p. 14
A.5 Number of human annotations for PPI
- At nh=200, coverage reaches at least 95% for all three classes or topics on both datasets; increasing the human-annotation budget generally reduces estimate variability.Source: Table 8 · p. 15, Table 9 · p. 15, §A.5 · p. 15
- For MELD Negative, increasing nh from 80 to 200 reduces ∆ from +0.95 to +0.16 and the across-draw SD from 3.73 to 0.41.Source: §A.5 · p. 15