Source-linked AI summary
OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer
Boyang Wang, Guangyi Xu, Jiahui Zhang, Zhipeng Tang, Zezhou Cheng
TL;DR
Shot Boundary Detection remains limited by non-interpretable boundaries, missed discontinuities, and weak annotation and benchmark coverage. OmniShotCut jointly predicts shot ranges and intra- and inter-shot relations using a shot-query Transformer with synthetic supervision, and consistently outperforms existing methods across multiple benchmarks and evaluation dimensions.
Problem
Shot Boundary Detection remains limited by non-interpretable boundaries, missed subtle discontinuities, and constrained annotation and benchmark coverage.
Method
OmniShotCut jointly predicts shot ranges with intra-shot and inter-shot relations using a shot-query Transformer and fully synthetic transition supervision.
Results
The model consistently outperforms existing methods across multiple benchmarks and evaluation dimensions.
Takeaways & Limitations
The results support fully synthetic supervision as a scalable and effective paradigm for next-generation shot boundary detection datasets.
Takeaways & Limitations
More sophisticated artistic and semantically dynamic transitions may require modeling beyond the current synthetic parameterization and unavailable industry-scale template collections.
Abstract
from arXiv · showhide
Shot Boundary Detection (SBD) aims to automatically identify shot changes and divide a video into coherent shots. While SBD was widely studied in the literature, existing methods often produce non-interpretable boundaries on transitions, miss subtle yet harmful discontinuities, and rely on noisy, low-diversity annotations and outdated benchmarks. To alleviate these limitations, we propose OmniShotCut to formulate SBD as structured relational prediction, jointly estimating shot ranges with intra-shot relations and inter-shot relations, by a shot query-based dense video Transformer. To avoid imprecise manual labeling, we adopt a fully synthetic transition synthesis pipeline that automatically reproduces major transition families with precise boundaries and parameterized variants. We also introduce OmniShotCutBench, a modern wide-domain benchmark enabling holistic and diagnostic evaluation. Experiments on the benchmarks demonstrate the effectiveness and generality of our method.
1 Introduction
The introduction argues that existing Shot Boundary Detection pipelines remain limited in interpretability, sensitivity to subtle discontinuities, annotation quality, and benchmark scope. OmniShotCut addresses these gaps through relational shot prediction, a unified shot-query Transformer, synthetic supervision, and a modern wide-domain benchmark.
- Problem formulation: Existing models miss subtle sudden jumps that introduce abrupt motion or texture changes and harm downstream tasks.The proposed inter-shot relation classification explicitly captures these relationships.
- Benchmark: OmniShotCutBench is a modern wide-domain benchmark with diverse transition patterns, sudden jumps, relational labels, and confidence scores for annotation uncertainty.It is designed to address noisy annotations, outdated and narrow video sources, and missing sudden-jump coverage.
- Problem formulation: Shot Boundary Detection should enrich each shot with intra-shot and inter-shot relational information beyond simple temporal range prediction.This formulation is intended to provide higher-level structural information and clarify transition semantics for downstream applications.
- Model architecture: A shot query-based Transformer jointly optimizes range prediction and relational classification within shared hidden states.The architecture is designed for unified temporal shot-range prediction and relational understanding, unlike prior 3D CNN-based models.
- Data generation: A fully synthetic pipeline with self-supervised clustering provides precise, diverse transition labels without manual annotation.The clustering groups similar but nonidentical clips, addressing the imprecision and labor of manually localizing gradual-transition boundaries.
2 Related Works
Prior SBD methods operate on dense video sequences and commonly use handcrafted low-level features to detect transitions. SBD also supports temporally coherent video curation, while synthetic-data pipelines address the difficulty of obtaining precisely aligned training annotations.
- Shot Boundary Detection: SBD requires native full-video inputs and frame-level precision, motivating high-density temporal modeling.The passage states that SBD operates without frame downsampling and must localize each boundary precisely.
- Shot Boundary Detection: Traditional methods such as PySceneDetect and Koala-36M primarily use handcrafted low-level features, including color-histogram differences and structural similarity.These feature-based approaches are described as methods for detecting abrupt transitions.
- Downstream Applications: SBD supports internet-scale video curation by segmenting long-form videos into temporally coherent clips for downstream models requiring continuous image sequences.The passage identifies video generation as an example of a downstream task that depends on temporal consistency.
- Synthetic Data: Synthetic-data strategies use programmable forward transformations to construct large-scale labeled datasets when precisely aligned training data is difficult to collect.Related examples include degradation synthesis for low-level vision and scripted synthesis of manipulated images for forensics and editing detection.
3 Method
OmniShotCut formulates shot boundary detection as structured relational prediction, jointly estimating shot ranges with intra-shot and inter-shot relations. Its method combines synthetic transition generation, a shot-query video Transformer, and the OmniShotCutBench benchmark for broad evaluation.
- Problem Formulation: OmniShotCut jointly predicts each shot’s temporal range, intra-shot relation, and relation to the previous shot across eight intra-relation categories.The listed intra-shot categories are General video, Dissolve, Wipe, Push, Slide, Zoom, Fade, and Doorway.
- Synthetic Transition Composition: The synthesis pipeline curates clean, standardized video segments, groups similar sources with DINO-based SSL clustering, and stitches clips using diverse within- and cross-cluster transitions.Clips are filtered by duration, resolution, fps, and aspect ratio, then cropped to a maximum duration of one minute.
- Synthetic Transition Composition: Sudden Jump is explicitly incorporated by estimating motion strength because abrupt manually cropped discontinuities violate fluent-shot assumptions and align with the SBD objective.The supplied passage introduces motion-strength estimation but does not include its complete formulation.
- Shot Query-based Dense Video Transformer: The shot-query video Transformer encodes spatiotemporal frame tokens with 3D positional embeddings, then uses learnable decoder queries and three heads to predict ranges and both relation types.The three heads are a range head, an intra-relation head, and an inter-relation head.
- OmniShotCutBench: OmniShotCutBench evaluates versatile transitions from modern internet videos using confidence-scored range labels and disjoint real edited videos.Its sources span vlog, anime, movie, concert, documentary, monitor recording, game, and sports content, with 114 videos totaling 114 minutes.
4 Experiment
The experiments evaluate OmniShotCut against established shot-boundary detectors on both modern and legacy benchmarks, using range-based and diagnostic relational metrics. Ablations show that discrete boundary classification and structured synthesis choices are important for balanced performance.
- Evaluation setup: OmniShotCut is evaluated on OmniShotCutBench and BBC Planet Earth Documentation using range precision, recall, F1, transition IoU, sudden-jump accuracy, and relation-classification accuracy.Range metrics use a default tolerance of 2 frames, while sudden-jump accuracy requires an exact frame match.
- Main results: PySceneDetect, TransNetV2, and AutoShot reach overall range F1 scores of 0.75–0.82 but achieve only 0.18–0.25 transition IoU.The results indicate that strong range-based scores can coexist with imprecisely localized transition boundaries.
- Ablation studies: Purely random transition-source sampling consistently degrades performance across all evaluated metrics compared with sampling semantically aligned clips from the same SSL-curated cluster.The base setting samples from the same cluster with 75% probability.
- Ablation studies: Removing continuous hard-cut synthesis degrades performance across all metrics, although fewer than 0.005% of purely random samples contain more than five consecutive hard cuts.The base synthesis enforces continuous hard cuts in 25% of synthesized videos.
- Ablation studies: The base architectural configuration gives the most balanced performance; 24 shot queries achieve the best sudden-jump accuracy and range F1, while more encoder layers improve sudden-jump accuracy but harm most other metrics.A training resolution of 96×128 provides the best resolution trade-off.
5 Conclusion
OmniShotCut reformulates shot boundary detection as relational prediction using a shot query-based Transformer, supported by synthetic transition data and a modern benchmark. Future work must address sophisticated artistic and semantically dynamic transitions beyond current synthetic parameterization.
- Conclusion: OmniShotCut reformulates shot boundary detection with explicit intra-shot and inter-shot relations in a shot query-based Transformer.The method jointly models relational structure for shot-boundary detection.
- Conclusion: A fully synthetic pipeline generates diverse transition effects with precise temporal supervision, alongside a modern complex shot-boundary benchmark.This design addresses limitations associated with manual annotation and outdated evaluation resources.
- Limitations and Future Work: Sophisticated artistic and semantically dynamic transitions may require modeling beyond the current synthetic parameterization.Complex cinematic patterns could benefit from large-scale industry-level transition-template collections, which are not publicly available.
Supplementary Material
The supplementary material provides implementation and technical details, qualitative visualizations, and additional information on transition genres, synthesis parameters, model training, and benchmark annotation.
- Supplementary Material: The supplement expands technical and implementation details while adding qualitative visualizations to complement the main manuscript.It covers transition genre types, transition synthesis parameter settings, model training and settings, and benchmark annotation GUI details.
A Full Transition Genre Types
The paper defines a broad taxonomy of editing transitions, spanning common and fine-grained variants organized by transition pattern. Its synthesis pipeline scales to wide-domain internet video, producing 11.9M synthetic transitions from approximately 1.5M curated clips.
- Transition taxonomy: Transitions are classified into categories according to their editing patterns, with the full type visualization presented in Figure 6.The authors note that the main-paper visualization does not exhaust all considered transition types.
- Transition taxonomy: The transition taxonomy covers diverse common and fine-grained variants, including Dissolve, Wipe, Push, Slide, and Zoom transitions.Listed subtypes include Transparent, Cross-Blur, and Ripple Dissolve; seven Wipe variants; two Push variants; and three Slide variants.
- Synthetic corpus: 11.9M transitions are synthesized for training from approximately 1.5M curated video clips collected through a wide-domain, internet-scale curation pipeline.Genres are annotated by Qwen3, while the synthetic corpus reports inter-shot relation and main/sub-transition statistics.
B Transition Synthesis Details
The transition-synthesis pipeline constructs training data through parameterized clip sampling, semantic grouping, transition-specific distributions, and extensive controls over temporal, spatial, intensity, and content factors. Its curation process also specifies source datasets, similarity thresholds, motion-tracking settings, and SSL clustering parameters.
- Source curation: Curation draws mainly from Huggingface and public video datasets, using εsim = 0.9, εdup = 0.05, motion tracking every 3 frames at 256x320, and 27,000 SSL clusters.The clustering uses DINOv3 ViT large.
- Synthetic clip construction: Synthetic training data uses Poisson-sampled clip counts with λ = 7.0 constrained to [1] [28], Gaussian clip durations N(2.8, 1.62) seconds, and 75% same-cluster sampling.Sudden-jump cases crop [40] frames from source videos with motion strength in the [25, 60] range.
- Transition distributions: Transition distributions allocate 35% to hard cuts, while dissolves and wipes receive specified proportions across transparent, blur, ripple, directional, spin, circle, bar, and mosaic variants.The passage specifies 9.4% transparent dissolve, 2.4% cross-blur dissolve, 1.8% ripple dissolve, and multiple wipe allocations.
- Parameter control: The pipeline controls discrete modes, temporal timing and speed, spatial geometry, intensity, and content layout while removing near-duplicate frames near transition phases.Examples include transition direction, edge softness, smoothing type, effect centers, blur curves, zoom parameters, lighting, feather widths, text wrapping, and spacing.
C More Implementation Details
The model is trained by optimizing a weighted sum of three classification losses.
- Training optimizes a weighted sum of three classification losses.
D Benchmark Annotation Details
OmniShotCutBench uses a dedicated annotation tool for precise, efficient frame-level boundary and relation labeling. Its inspection and workflow features help annotators verify subtle transitions and preserve annotation progress.
- Annotation Tool: The annotation tool swiftly locates per-frame boundaries and labels dense transition cases in long videos.It was developed specifically for the OmniShotCut benchmark.
- Annotation Tool: A floating label window, multi-selection, and auto-save support rapid verification, simultaneous labeling, and preservation of annotation progress.The window displays segment type, confidence, or relation labels.
- Inspection Mode: Open-image inspection enables frame-by-frame examination of subtle jumps, dissolves, and fades with associated type, confidence, and relation annotations.Annotators can open frames individually and navigate sequentially to inspect gradual transitions accurately.
E Visual Comparisons
OmniShotCut qualitatively detects fading, dissolve, and sudden-jump transitions more successfully than TransNet V2 and AutoShot. Its clearer transition boundaries are better suited to downstream applications requiring an unambiguous first frame.
- Qualitative comparisons: OmniShotCut successfully detects fading, dissolve, and sudden-jump transitions in the qualitative comparison.Figure 11 compares the models on fading, sudden jump, and dissolve examples.
- Qualitative comparisons: TransNet V2 and AutoShot often select the middle of fading or dissolve effects as the start frame, producing confusing boundaries.These baseline behaviors are described as unfavorable for downstream video-generation applications.
- Downstream implications: Clear first-frame sources support downstream applications such as image-to-video generation.The passage states that downstream video generation requires a clear first-frame source.