PaperlayerView exact revision on arXiv

Source-linked AI summary

AdShot: Benchmarking Multimodal Large Language Models for Video Advertisement Clipping

Wen Xie, Om Rastogi, Sai Siddhartha Vivek Dhir Rangoju, Gijs Overgoor, Yakov Bart, Sarah Ostadabbas

arXiv:2610.33377v1cs.MMcs.CV

TL;DR

MLLMs perform strongly on general video understanding, but their ability to make specialized editorial decisions for ad clipping has not been established, despite the labor-intensive work of producing duration variants. AdShot benchmarks shot selection against professional ad edits and finds model-dependent performance, systematic content biases, and difficulty meeting duration constraints.

  • Problem

    Although MLLMs show strong general video understanding, their ability to perform specialized editorial shot selection for ad clipping remains unexplored, despite the labor-intensive creation of duration variants.

    Source: §1 · p. 1

  • Method

    AdShot benchmarks MLLMs against 823 professionally edited 30-sec/15-sec ad pairs spanning 194 brands and 17 industries, annotated with shot boundaries and content features.

    Source: §1 · p. 2

  • Results

    Architecture matters more than scale, and 9 of 15 models surpass a heuristic random baseline while 6 fall below it.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark reveals systematic preferences for information- and emotion-focused shots over action and imagery, providing diagnostic evidence for evaluating MLLM ad-editing behavior.

    Source: §1 · p. 2, §7 · p. 10

    Each ad pair has a single ground-truth edit, so equally valid alternative shot selections are penalized as incorrect.

    Source: §6 · p. 10

Abstract

from arXiv · show

Advertisers routinely create multiple duration variants of ads to meet marketing budgets and viewer preferences, a labor-intensive and costly process. While multimodal large language models (MLLMs) excel at general video understanding, their ability to perform specialized editorial tasks remains unexplored. We introduce AdShot, a benchmark for evaluating MLLMs on shot selection for ad clipping. It contains 823 professionally edited ad pairs (30-sec sources and 15-sec edits) across 194 brands and 17 industries, annotated with shot-level boundaries and content features (action, emotion, information, imagery). We evaluate 13 state-of-the-art video-only MLLMs and 2 audio-visual MLLMs (7B-72B parameters) on shot selection accuracy (Precision, Recall, F1), duration constraints (15s), and content preservation, including an audio ablation. We find: (1) model architecture matters more than scale, with smaller models often matching or exceeding larger ones; performance is strongest for target ads with 6-10 shots; (2) models exhibit a mean absolute duration error of 5.39s, with most tending to overshoot the 15s target, though they can still substantially reduce human editing effort; (3) all models exhibit systematic content biases, favoring information-rich and emotion-focused shots while under-selecting action and imagery. We publicly release our benchmark to facilitate research at the intersection of multimodal AI and computational advertising.

1 Introduction

Ad clipping requires selecting and ordering shots under tight duration limits while preserving advertising-relevant content, a manual process that remains labor-intensive. AdShot benchmarks this task against professional edits and finds model-dependent performance and systematic content biases.

1 Introduction

  • Creating duration variants requires manual shot selection and rearrangement, making the process a labor-intensive production bottleneck.Source: §1 · p. 1
  • Ad clipping must balance strategic content composition with strict duration constraints rather than simply selecting the most salient moments.Shots may convey product information, elicit emotion, establish imagery, or motivate action.Source: Figure 1 · p. 2, §1 · p. 2
  • AdShot provides 823 professionally edited 30-sec/15-sec ad pairs from 194 brands and 17 industries, with shot boundaries and four high-level content attributes.The attributes are action, emotion, information, and imagery.Source: §1 · p. 2
  • Architecture matters more than model scale, and 9 of 15 evaluated models surpass a heuristic random baseline while 6 fall below it.Audio-visual models nearly match the best video-only model at a third of the parameters.Source: §1 · p. 2
  • Models favor information- and emotion-focused content over action and imagery, a systematic bias relevant to real-world advertising use.Source: §1 · p. 2

2 Related Work

Prior video and advertising benchmarks largely assess comprehension or generation, while AdShot focuses on matching professional editorial decisions. The related work frames ad clipping as a distinct task because it must preserve balanced content composition within strict duration limits.

2.1 Video Understanding Benchmarks

  • Existing ad benchmarks such as VideoAds and AdsQA primarily assess advertisement comprehension through question answering.Source: §2.1 · p. 3

2.2 Video Summarization and Editing with MLLMs

  • Prior video summarization and ad-generation work evaluates representative clips or generation quality, whereas AdShot evaluates alignment with professional edits using paired references.Source: §2.2 · p. 3

2.3 Computational Advertising

  • Ad content composition serves campaign goals: awareness campaigns emphasize emotion and imagery, while direct-response ads prioritize information and action.Source: §2.3 · pp. 3–4

3 The AdShot Benchmark

AdShot formalizes ad clipping as selecting whole shots from a 30-second source to match a professional 15-second edit under a duration constraint. Its paired-ad benchmark combines validated shot matches, broad dataset coverage, and shot-level content labels.

3.1 Ad Selection Task Formulation

  • The task selects a subset of source shots that maximizes overlap with the editor-selected 15-second reference while satisfying a duration constraint.Source: §3.1 · p. 4
  • AdShot measures whole-shot keep/drop selection, leaving reordering, intra-shot trimming, transitions, grading, and audio mixing outside its scope.The authors note that 45 of 50 manually reviewed pairs preserve the original order, but order, flow, and splice quality remain unmeasured.Source: §3.1 · p. 4

3.2 Dataset Construction

  • 30-second and 15-second ads are paired by clustering shots, with same-cluster shots treated as matches and every reference-ad shot required to match a source shot.Source: §3.2 · p. 5
  • 98.2% agreement between automated pair construction and human judgment was found across 401 shots; two reviewers agreed on all shot-level judgments.Source: §3.2 · p. 5
  • The dataset contains 823 ad pairs from 194 brands across 17 industries; source ads average 14 shots and edited ads average 8.Average shot durations are similar across versions, suggesting editors preserve temporal pacing rather than simply truncating content.Source: §3.2 · p. 5

3.3 Shot-Level Content Features

  • Each shot receives 18 binary content-feature annotations grouped into action-, information-, emotion-, and imagery-focused categories.The labels are produced with AdNotator, built on Video-LLaMA 2 and fine-tuned on about 8,000 video ads.Source: §3.3 · p. 5

4 Evaluation Methodology

The evaluation supplies shot-level visual and metadata inputs to MLLMs, then measures their selections against professional edits across accuracy, duration adherence, and content alignment.

4.1 Model Input and Output Format

  • Models receive task instructions, shot order and duration, and sequential shot frames with explicit boundaries, then return selected shot IDs, total duration, and per-selection explanations.The pipeline evaluates these outputs for accuracy and content preservation.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Evaluated Models

  • The evaluation covers 13 video-only and 2 audio-visual MLLMs across eight model families and parameter scales from 7B to 72B.The model selection spans capabilities relevant to shot selection, including temporal reasoning, fine-grained grounding, and audio-visual input.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 Evaluation Metrics

  • Shot-set overlap is evaluated with Precision, Recall, and F1, distinguishing over-selection, missing editor-selected shots, and their balance.Precision reflects shots the model selects but editors exclude; recall reflects shots editors keep but the model omits.Source: §4.3 · p. 7
  • Duration adherence measures error against a 15-second target with a 14–16-second acceptable range, while content alignment compares normalized feature densities for action, information, emotion, and imagery against professional edits.Signed feature-density differences indicate over- or under-selection in each content category.Source: §4.3 · p. 7

5 Experimental Results

Across the evaluated models, shot-selection performance varies by architecture and model, while duration control and preservation of professional edits’ content composition remain challenges. Performance is generally strongest for targets with 6–10 shots, and models systematically favor information- and emotion-focused shots over action and imagery.

5.2 Shot Selection Performance

  • F1 ranges from 0.443 to 0.6763, with Aria (25.3B) highest at 0.676; 9 of 15 models surpass the random-baseline F1 of 0.503, and audio-visual models nearly match Aria at roughly a third of its parameters.Removing audio lowers Qwen2.5-Omni 7B and MiniCPM-o 2.6 8B F1 to 0.585 and 0.644, respectively.Source: Table 4 · p. 8, §5.2 · p. 8
  • Six models fall below the random baseline, with precision at or above 0.597 but recall of 0.366–0.450; all six undershoot the 15s target.The models span 8B to 72B parameters, and the passage frames this as a zero-shot fit challenge for dense, fast-cut ads.Source: §5.2 · pp. 8–9
  • F1 is lowest for targets with fewer than 6 shots and often highest in the 6–10-shot range, where moderate shot counts allow better alignment with professional choices.The comparison covers targets with fewer than 6, 6–10, and more than 10 shots.Source: Table 5 · p. 9, §5.2 · p. 9
  • Higher precision than recall typically accompanies undershooting, while higher recall tends to accompany overshooting; F1 and duration error correlate across models (r = 0.78) but measure separate axes.The passage attributes the correlation to selecting more shots, which raises both overlap and overshoot.Source: §5.2 · p. 9
  • Models systematically over-select emotion- and information-focused shots while under-selecting action and imagery relative to professional edits.The passage reports full results in Appendix C.Source: §5.2 · p. 9

5.3 Duration Constraint Adherence

  • The 13 video-only models have mean duration error of 5.39s (SD = 2.85), and the two audio-visual models average 6.09s; overshooting is slightly more common than undershooting.The errors are large against the 14–16s tolerance, and audio-visual models also overshoot on average.Source: §5.3 · p. 9

6 Limitations and Future Work

AdShot’s findings are bounded by its single approved edit per pair, its 30-to-15-second focus, and uneven industry representation. The authors identify domain-specific fine-tuning and richer frame and audio processing as directions for future work.

Task scope

  • Shot selection is scored, but reordering, intra-shot trimming, transitions, grading, and audio mixing are outside scope; shot order and audio continuity are unmeasured.The authors note that shot order is rarely changed in cutdowns and audio may be independently mixed, but reused audio still requires crossfading and re-timing.Source: §6 · p. 10

Duration coverage

  • The benchmark covers 30-to-15-second adaptations; shorter conversions and micro-ads remain unexplored, although the pipeline extends to them.The covered duration pair is described as the most common in television advertising [MediaRadar, 2023].Source: §6 · p. 10

Ground truth and annotations

  • Each pair has one approved ground-truth edit, so equally valid alternatives are penalized and inter-editor variance remains unmeasured.Content-bias estimates also depend on AdNotator; agreement is lowest for the subjective Emotion and Imagery features, making those estimates most uncertain.Source: §6 · p. 10

Industry imbalance

  • Legal and Government ads exceed 28% of pairs, while Electronics and Apparel have only 2 and 4; the smallest industries are not reliable per-domain findings.The authors report stable F1 across all 17 industries and say the imbalance does not drive headline results.Source: §6 · p. 10

Future work

  • The paired data support domain-specific fine-tuning; proposed further work includes content-adaptive frame selection and content-aware audio processing.The authors identify fine-tuning as the most direct route to closing the gap for sub-baseline models; raw unfiltered audio already yields measurable gains.Source: §6 · p. 10

7 Conclusion

AdShot provides a framework for assessing MLLMs’ advertisement shot-selection decisions against professionally edited ads. Its evaluation treats selection as a constrained editorial task and concludes that current models still fall short of professional standards, with content biases and difficulty meeting temporal constraints.

7 Conclusion

  • AdShot evaluates MLLM shot selection against professional edits and offers diagnostic insight for domain-specific fine-tuning and hybrid approaches.The authors frame this as a case for evaluating strategic editorial decision-making, not only comprehension.Source: §7 · p. 10

A Annotation Questions

  • Table 6 organizes annotation questions by ad-content feature category.The supplied caption identifies the table’s category and question columns but does not specify the questions.Source: Table 6 · p. 14

B System Prompts

  • The system prompt asks models to review shot durations and visual content, select whole shots for an approximately 15-second clip, and explain selections.It requires a 14–16-second total and a JSON response with selected shots, total duration, and explanations.Source: B System Prompts · p. 15

User Prompt Format

  • The user prompt pairs shot-duration metadata with representative frames, with the number of frames determined by the sampling setting.The examples show images listed under shot labels; sampling uses 1, 3, or 5 frames at temporal percentiles.Source: User Prompt Format · p. 16

C Content Alignment Analysis

Models’ shot choices diverge systematically from professional edits, and denser temporal sampling trades off selection quality against duration control. A random-selection baseline provides calibration for interpreting model scores, while its large duration error limits practical usefulness.

C Content Alignment Analysis

  • Models over-select emotion- and information-focused shots and under-select action- and imagery-driven content; information-focused divergence averages 0.08 more features per shot than professional edits.The pattern persists in larger models, including Qwen3-VL-32B-Instruct and LLaVA-Onevision-72B, and can coexist with decent F1.Source: C Content Alignment Analysis · p. 17, Figure 5 · p. 17

Ablation Study on Temporal Sampling

  • Denser 3- or 5-FPS sampling can improve F1 for some models but usually increases mean duration error; effects vary, with MiniCPM 8B showing no benefit or negative effects.LLaVA-OneVision 7B is a mild positive exception, improving MDE while marginally increasing F1; some models could not be evaluated at 5 FPS due to token limits.Source: Ablation Study on Temporal Sampling · p. 17, Table 7 · p. 18, Figure 6 · p. 18

Formulation

  • The heuristic baseline samples a subset size uniformly from 1 to the number of shots, then selects that many distinct shots uniformly at random, using no duration or content information.The variable subset size produces selections at extreme sizes more often than the optimal selection size, contributing to large duration errors.Source: Formulation · p. 19, D Heuristic Baseline · p. 19

Setup and Results

  • Across 823 pairs and 20 random seeds, the heuristic baseline scores F1 0.503, Precision 0.569, Recall 0.541, and MDE 8.18s.Its overshoot rate is 55.3% and undershoot rate is 44.5%, consistent with errors distributed around the 15-second target without duration awareness.Source: Setup and Results · p. 19

D Heuristic Baseline

  • The baseline calibrates chance-level F1 rather than competing with models: 9 of 15 models exceed it and 6 fall below, while matching its F1 does not imply comparable practical usefulness.Its relatively high recall of 0.541 comes with MDE of 8.18s because uniform subset sizes often select many ground-truth shots.Source: Setup and Results · p. 19

E Per-Industry Performance

Qwen3-VL 32B is used to characterize industry-level performance because it combines the benchmark’s best precision with strong F1 and low duration error. Its results are stable across industries with enough examples, while the smallest industries are too sparse to interpret.

E Per-Industry Performance

  • Qwen3-VL 32B achieves the best precision (0.636), F1 of 0.611, and MDE of 3.71, balancing selection accuracy with duration fidelity.Aria 25.3B reaches 0.881 recall but has an MDE of 11.95.Source: E Per-Industry Performance · p. 20
  • For industries other than Electronics (n=2) and Apparel (n=4), F1 ranges from 0.551 to 0.674, with no domain collapsing.The 2–4-pair estimates for the two smallest industries are too small to interpret.Source: E Per-Industry Performance · p. 20, Table 8 · p. 20
  • Retail (n=159), Legal (n=125), and Government (n=108) score F1 0.616, 0.601, and 0.584, respectively, close to the overall average.The authors report that these high-volume industries do not make the aggregate headline numbers an artifact of industry mix.Source: E Per-Industry Performance · p. 20

F Audio Ablation

In controlled same-model comparisons, removing audio lowers F1 for both audio-visual models, primarily through reduced recall; audio raises recall in most industries. Qualitative examples show genuine shot selection, with errors skewing toward over-selection.

F Audio Ablation

  • Removing audio lowers F1 by 0.025 for MiniCPM-o (0.669→0.644) and 0.083 for Qwen2.5-Omni (0.668→0.585), mainly through reduced recall.MDEpos is higher with audio by 0.90s and 2.87s, while removing audio reduces overall MDE.Source: F Audio Ablation · p. 21, Table 9 · p. 21
  • Audio raises Qwen2.5-Omni recall in 16 of 17 industries and F1 in 14; the Food & Beverages and Restaurants F1 changes are under 0.02, while Apparel has only 4 pairs.The gains are consistently recall-driven.Source: Table 10 · p. 21, F Audio Ablation · p. 21

G Qualitative Examples

  • Across three ads from Automotive, Health, and Retail, Qwen3-VL 32B recovers editor-kept anchor shots but more often adds dropped shots than misses kept ones.The examples include both-marker matches, blue-only over-selections, and yellow-only misses; the pattern is consistent with MDEpos 3.31 versus MDEneg 0.39.Source: G Qualitative Examples · p. 22, Figure 7 · p. 22, Figure 8 · p. 23

H Frontier-Model Agreement with AdNotator

A comparison of AdNotator labels with Gemini 3.1 Pro labels on 1,000 AdShot shots finds 81.96% agreement across 18,000 matched decisions. Agreement is lower for the more subjective Emotion and Imagery categories, and does not establish which label set is more correct.

H Frontier-Model Agreement with AdNotator

  • The comparison uses 1,000 randomly sampled AdShot shots, labeled on the same 18 binary questions, yielding 18,000 matched question–shot decisions.The questions cover Action, Information, Emotion, and Imagery.Source: H Frontier-Model Agreement with… · p. 23, Table 11 · p. 24
  • Gemini 3.1 Pro agrees with AdNotator on 81.96% of decisions; agreement is highest for Action (93.96%) and lower for Emotion (76.44%) and Imagery (70.63%).Action features include concrete cues such as a displayed URL, price, or stated offer.Source: H Frontier-Model Agreement with… · pp. 23–24, Table 11 · p. 24
  • The comparison supports that an independent model largely reproduces AdNotator’s labels, but does not show which label set is more correct; the content-bias analysis continues to use AdNotator.Emotion and Imagery category-level bias estimates carry the most uncertainty.Source: H Frontier-Model Agreement with… · pp. 23–24

I Dataset Documentation, Licensing, and Availability

AdShot releases annotations, derived per-shot media, metadata, and evaluation code rather than source videos. The release is scoped to non-commercial research for the media and annotations, with the evaluation code separately under MIT licensing.

I Dataset Documentation, Licensing, and Availability

  • For 823 ad pairs, the release includes video IDs, shot boundaries, ground-truth shot matching, 18 binary content-feature labels per shot, and brand and industry metadata.It also includes the evaluation harness and per-model raw outputs for reproducing the paper’s tables.Source: I Dataset Documentation, Licensing,… · p. 25
  • No source videos or 15-second target-ad media are released; derived RGB keyframes and 16 kHz mono audio are provided for 30-second sources, with keyframes available for 819 of 823 ads.The four missing keyframe sets correspond to sources removed, made private, or age-gated before extraction.Source: I Dataset Documentation, Licensing,… · p. 25
  • Derived media supports reproducibility after source videos become unavailable, while the annotation layer remains usable independently.The paper reports periodic re-validation of video ID availability and publication of attrition rates.Source: I Dataset Documentation, Licensing,… · p. 25
  • Annotations, derived media, and metadata use CC BY-NC 4.0, while evaluation code uses the MIT license; the underlying advertisements remain their owners’ property.Users are responsible for using source videos in accordance with YouTube’s Terms of Service.Source: I Dataset Documentation, Licensing,… · p. 25

Related papers

Loading 2610.33377v1…