PaperlayerView exact revision on arXiv

Source-linked AI summary

PERSIST: Persistent-State Discrimination for Shot Boundary Detection

Tingyu Lin, Christian Stippel, Armin Dadras, Jakob Zenzmaier, Florian Kleber, Wolfgang Aigner, Robert Sablatnig

arXiv:2608.29287v1cs.CV

TL;DR

Shot boundary detection must distinguish genuine shot changes from sharp local discontinuities caused by within-shot events. PERSIST uses latent-state persistence and interpretable gates to make this distinction, matching the strongest public detector while suppressing pseudo-event false positives under stricter training. Its persistence test is weakest for jump cuts and wipes, where latent movement is limited or local cues already suffice.

  • Problem

    Shot boundary detection must distinguish genuine shot changes from sharp local discontinuities caused by shake, motion blur, flicker, occlusion, illumination changes, and archival damage.

  • Method

    PERSIST reformulates SBD as boundary semantic discrimination using a FiLM-conditioned sinusoidal latent state, a dual-rate temporal backbone, and a three-gate persistence discriminator.

  • Results

    PERSIST matches the strongest public detector under stricter training and, at matched true-transition recall, its gate-free ablation produces 63 more pseudo-event false positives on average.

  • Takeaways & Limitations

    The framework makes suppression attributable to per-frame gate triples and separates high-evidence pseudo-events from true boundaries while preserving true-transition recall.

  • Takeaways & Limitations

    The persistence test is weakest for jump cuts and wipes, and cross-domain results are transfer measurements because training uses ClipShots real transitions only.

Abstract

from arXiv · show

Shot boundary detection (SBD) is widely treated as the localisation of local visual discontinuities, yet many false positives such as hand-held shake, illumination flicker, motion blur, occlusion, and damaged archival material produce equally sharp local change without introducing a new shot. We reformulate SBD as boundary semantic discrimination: a frame is favoured as a boundary only when its local change evidence is accompanied by a persistent update of the video's latent temporal state, rather than a transient excursion that returns to the surrounding trend. This persistence test is operationalised with a continuous latent state from a FiLM-conditioned sinusoidal representation network and a structured discriminator that combines three semantic cues, local change, transient impulse, and return-to-trend, into a single interpretable per-frame signal over a dual-rate temporal backbone. The resulting framework, PERSIST, turns every decision into an inspectable one: the persistence criterion is trained into the classifier, its per-frame effect stays readable from the gate triple, and its learned latent state is measurably boundary-discriminative. On a 2,727-video per-subtype diagnostic it removes 33-80% of flash, text-overlay, and archival false positives relative to an identically trained cue detector, and at matched true-transition recall it roughly halves TransNetV2's pseudo-event false positives on that diagnostic and cuts its false positives on ClipShots footage by about a quarter, while preserving recall. It does so while reaching parity with the strongest public detector across online, broadcast, short-form, and historical-archive transfer evaluations, under markedly stricter training: it learns from ClipShots real transitions only, whereas the anchor draws on additional corpora whose transitions are 85% synthetic. Code is available at https://github.com/linty5/PERSIST.

1 Introduction

PERSIST reframes shot boundary detection as semantic discrimination: local visual change is treated as a boundary only when it reflects a persistent latent-state update rather than a transient excursion. Its structured persistence mechanism makes suppression interpretable while targeting false positives without changing standard frame-level SBD output.

  • Motivation: Local discontinuities from shake, motion, blur, occlusion, illumination change, flicker, and archival damage can mimic shot transitions without introducing a new shot.These non-transition events motivate distinguishing true boundaries from pseudo-boundaries.
  • Boundary semantic discrimination: A boundary is favoured only when strong local evidence is accompanied by change that cannot return to the surrounding visual trend.The reformulation requires local change, non-absorption into the surrounding trend, and persistent temporal updating.
  • Framework: PERSIST combines a dual-rate TransNetV2-style backbone, a FiLM-conditioned SIREN latent, and multiplicative change, transient-impulse, and return gates.The three components provide short- and long-range context, a persistence-defined representation, and structured discrimination.
  • Interpretability: Every suppression is attributable to a per-frame gate triple rather than hidden in an opaque score sum.The output remains a standard frame-level boundary score and is directly verifiable with existing SBD evaluation infrastructure.

2 Related Work

Prior SBD work largely detects local discontinuities using handcrafted or learned spatio-temporal cues, while related approaches address pseudo-boundaries through data augmentation or cue fusion. PERSIST instead combines multi-rate temporal context with a continuous latent-state persistence test and evaluates transfer across diverse benchmark domains.

  • Classical and deep shot boundary detection: Classical and deep SBD methods detect discontinuities using handcrafted descriptors or learned spatio-temporal representations over short video segments.Examples include edge-change ratios, colour descriptors, 3D ConvNets, and fully convolutional systems.
  • Pseudo-boundary suppression and structured prediction: Existing pseudo-boundary suppression approaches use hard-negative mining or additive frame-similarity and colour-histogram cue fusion.PERSIST contrasts this with a multiplicative joint gate over a learned continuous latent.
  • Multi-rate and continuous temporal representations: SBD combines short-range localisation with longer-range verification, motivating multi-rate backbones and continuous temporal representations.The related-work discussion connects this need to SlowFast-style separation and TransNetV2-style dilated temporal cells.
  • Benchmarks, domain transfer, and protocol reporting: Modern-video evaluation commonly spans ClipShots, BBC Planet Earth, and AutoShot/SHOT, while historical transfer uses digitised archival film with expert annotations.These benchmark packages cover online, broadcast, short-form, and historical-archive settings.
  • Historical film: Historical film adds damaged reels, scratches, splices, low-quality footage, and long transition patterns that make handcrafted cues fragile.HISTORIAN supplies digitised analogue film with expert cinematographic annotations.

3 Method

PERSIST builds a per-frame boundary score from dual-rate evidence and context, a FiLM-conditioned continuous latent state, and three operational persistence gates. The model turns persistent-boundary semantics into a trainable, interpretable suppression signal rather than a direct inference-time filter.

  • Dual-rate backbone: PERSIST produces per-frame boundary scores from a dual-rate TransNetV2-style backbone, with fast features for local evidence and slow features for trend context.The slow pathway conditions the latent state, while the fast pathway supplies start, end, and base evidence heads.
  • Persistent-boundary criterion: A persistent boundary requires latent separation across positive offsets and stabilization of the post-boundary state in a new regime.A pseudo-boundary fails either condition by not updating persistently or by returning near its pre-event state.
  • Operationalization: The criterion is conceptual: PERSIST estimates the latent state and evaluates differentiable per-frame proxies rather than enforcing the conditions directly.These proxies operationalize the persistent-boundary definition in the learned model.
  • Latent state for persistence testing: The FiLM-conditioned SIREN represents the latent state as a smooth function of normalized time, with exact analytic derivatives used by the dynamics loss.The latent has width K=16 and uses three hidden layers with SIREN frequency ω0=30.
  • Latent conditioning: A projected slow-path signal provides teacher embedding u(t) for latent distillation and the observed-increment signal used by the transient-impulse gate.An auxiliary SIREN classifier is used only during training.
  • Persistence discriminator: The discriminator gates local evidence through change, transient-impulse, and return cues, then combines them multiplicatively into a joint suppression signal.Suppression is conservative: failing any gate leaves evidence close to its unsuppressed value, and each event is attributable to (gc,gt,gr).
  • Persistence discriminator: The multiplicative product primarily trains background suppression, while inference uses only a small evidence adjustment and exposes per-cue attribution.Thus persistence shapes the classifier rather than acting as a selective inference-time filter.
  • Training objective: The training objective combines classification, latent-state, and background-suppression losses.The latent loss distils φ toward u, penalizes temporal variation, and uses analytic derivative energy to make dynamics boundary-discriminative.

4 Experiments

PERSIST is evaluated through literature comparison, controlled attribution, component ablations, pseudo-event suppression, and latent-state analyses. It reaches benchmark parity while selectively reducing false positives through persistence-based discrimination, with limitations on jump cuts and wipes.

  • The experiments assess benchmark positioning, controlled attribution, component importance, pseudo-event suppression, and whether the latent state is boundary-discriminative.
  • Benchmark comparison: A single ClipShots-trained PERSIST matches official TransNetV2 across four datasets, equaling or exceeding it on five of eight cells and trailing only on AutoShot/SHOT beyond 0.003.PERSIST uses ClipShots real transitions only, whereas TransNetV2 uses additional corpora with 85% synthetic transitions.
  • Controlled attribution: PERSIST leads the cue-only baseline on all four datasets under oracle-best F1, with gains of +0.012 on ClipShots, +0.006 on BBC, +0.011 on HISTORIAN-GST12, and +0.007 on AutoShot/SHOT.Under fixed-0.50, gains are positive on ClipShots (+0.065) and BBC (+0.018), while the other two datasets are exceptions.
  • Component ablation: Removing the discriminator adds 63 pseudo-event false positives at matched recall, although its aggregate-F1 effect is mostly within seed spread.Removing all three latent-shaping components stays within 0.003 of the full model on every cell except AutoShot/SHOT at fixed-0.50, where it is 0.020 lower.
  • Pseudo-event suppression: At threshold 0.50, PERSIST reduces cue-only false positives by 35% for flash, 33% for archival, and 80% for text-overlay footage.The counts are flash 329 versus 508, archival 270 versus 404, and text-overlay 80 versus 399; the cue baseline has higher recall, 0.765 versus 0.726.
  • Pseudo-event suppression: At matched recall, TransNetV2 emits 1,459 diagnostic pseudo-event false positives versus PERSIST’s 723, while ClipShots false positives are 2,147 versus 2,665 and 2,794 for the other detectors.The diagnostic comparison uses TransNetV2 threshold 0.22 and recall 0.72; the ClipShots comparison uses matched recall 0.85.
  • Limitations: The persistence test stays within 0.002 F1 of the discriminator-free variant on clean cuts and is strongest on dissolve, but jump cuts and wipes remain limitations.The latent barely moves for jump cuts, while wipes are sharp sliding boundaries that local cues track directly.
  • Latent-state analysis: The latent moves 1.49±0.13× more at boundaries than elsewhere, while true cuts and high-evidence pseudo-events have similar raw evidence but final scores of 0.725 and 0.269.Forcing the joint gate off changes the diagnostic pseudo-event count by only 7, indicating that the trained classifier carries most of the separation while the gate remains an interpretable read-out.

5 Conclusion

PERSIST reformulates shot boundary detection as a persistence test over latent temporal state, combining a dual-rate backbone, FiLM-conditioned sinusoidal latent, and three-gate discriminator.

  • PERSIST treats a boundary as persistent latent-state change rather than local visual discontinuity alone.
  • The framework combines a dual-rate TransNetV2-style backbone, FiLM-conditioned sinusoidal latent φ(t), and three-gate persistence discriminator.

Supplementary Material PERSIST: Persistent-State Discrimination for Shot Boundary Detection

The supplement accompanies the main paper in the same document and defines how its internal references should be interpreted.

  • Supplementary references such as “Table S1” and “Figure S1” refer to material within this supplement.

S1 Data, annotation, and the diagnostic split

The supplement documents evaluation datasets, archival annotation conventions, and a controlled diagnostic split rendered from held-out ClipShots shots with known labels.

  • S1 Data, annotation, and the diagnostic split: The evaluation spans online, broadcast, short-form, and archival video, with cross-domain tests using one ClipShots-trained model without target fine-tuning.
  • S1 Data, annotation, and the diagnostic split: HISTORIAN-GST12 is a focused archival subset containing annotated gradual-transition spans, used to test transfer to damaged archival footage.
  • S1 Data, annotation, and the diagnostic split: Gradual transitions are annotated as shot segments, with categories including dissolve, wipe, flip-over, swish-pan, missing-frame, and fades.
  • S1 Data, annotation, and the diagnostic split: Borderline gradual-transition cases require visual judgment, leaving residual subjectivity between faint transitions and non-transition disturbances.
  • S1 Data, annotation, and the diagnostic split: The diagnostic contains 2,727 rendered clips from held-out ClipShots test shots, combining authentic imagery with fully known true-transition and pseudo-event labels.
  • S1 Data, annotation, and the diagnostic split: Pseudo-event categories include flash, fast pan, text overlay, archival degradation, and scratches, while true-transition categories include cuts, jumps, dissolves, fades, and wipes.

S2 Method and training details

The supplement details PERSIST’s comparison tiers, component ablations, architecture, hyperparameters, and training losses, including the persistence gates’ training role.

  • S2 Method and training details: Tier 1 compares PERSIST with literature models under a unified evaluator despite differences in training data, frameworks, and inference windows.
  • S2 Method and training details: PERSIST trains on ClipShots real transitions only, whereas official TransNetV2 uses a larger mix containing approximately 85% synthetic transitions.
  • S2 Method and training details: The cue-only baseline shares PERSIST’s backbone, optimizer, schedule, seed, and lack of synthetic augmentation, differing in its descriptor-cue stack.
  • S2 Method and training details: The ablation variants remove the persistence discriminator, latent shaping, dynamics auxiliary, or slow pathway while retaining the common training setup.
  • S2 Method and training details: The model processes 27 × 48 frames in T=64 sliding windows with dual-rate stages and a FiLM-conditioned SIREN latent branch.
  • S2 Method and training details: The three gates encode local change, transient impulse, and return-to-trend, with their product primarily driving background-suppression training.

S3 Extended benchmark results and controls

The controls test whether PERSIST’s benchmark parity and pseudo-event suppression are robust to seeds, ensembles, windows, persistence horizons, supervision, and threshold calibration.

  • Multi-seed statistics: Seed variation remains small across test sets, with per-seed standard deviation at most 0.0095 under fixed-0.50 and 0.0083 under oracle-best.Across the four main-paper benchmarks, the maximum standard deviation is 0.0064.
  • Ensembling: The three-seed ensemble reaches or exceeds the official TransNetV2 anchor on four of five test sets and is within 0.002 on the fifth.It closes the AutoShot oracle-best gap at 0.829 versus 0.830 and exceeds the anchor on the synthetic diagnostic, 0.631 versus 0.557.
  • Inference-window control: 100-frame inference does not improve either model over 64 frames, with oracle-best differences at most 0.010 F1.The longer window is slightly worse at the fixed threshold on AutoShot and HISTORIAN-GST12, so the comparison is not confounded by window length.
  • Persistence-horizon sensitivity: Changing return-gate offsets to half or double the default leaves ClipShots oracle-best F1 nearly flat, moving it by at most 0.0035.All tested horizons remain far below the cue baseline’s 1,391 and matched-recall TransNetV2’s 1,459 pseudo-event false positives.
  • Hard-negative supervision: On natural footage, PERSIST removes 518 false positives, compared with 218 for hard-negative training from the same cue baseline.On rendered events, hard negatives perform better because they train on that exact rendering distribution; PERSIST uses no hard negatives.
  • Threshold calibration: PERSIST’s calibration is strongest in-domain and better than the matched cue reproduction across modern benchmarks, but it is not uniformly better than the official anchor.ClipShots ECE is 0.0088 versus 0.0112 for official TransNetV2 and 0.0209 for the cue reproduction.

S4 Mechanism analysis

Mechanism analyses show that persistence improves pseudo-event suppression without sacrificing genuine-cut behavior, while the latent-state and gate diagnostics clarify how the decision is formed and where the study is bounded.

  • Per-subtype real-transition F1: The single-rate variant slightly raises real-transition F1 on several subtypes while also increasing hard-negative false positives.Dropping the slow pathway makes the detector fire more readily, exposing the role of longer-range context.
  • Per-component pseudo-event decomposition: 723 total pseudo-event false positives rise to 771 without the persistence discriminator and 840 without the slow pathway.Both components therefore contribute suppression, although the full per-component decomposition also evaluates dynamics and other ablations.
  • Latent-state analysis: The full model’s boundary-to-non-boundary latent-change ratio is 1.49±0.13, versus 1.31±0.16 without the discriminator and 1.32±0.14 without φ shaping.The table cautions that the margins over shaping ablations lie inside seed spread.
  • Decision behavior: True cuts and high-evidence pseudo-events have near-identical raw evidence, yet the final score separates them by 2.7× while the joint gate fires almost identically.This supports a trained-classifier effect rather than a selective inference-time filter.
  • Leave-one-gate-out: Leave-one-gate-out tests show that gates are not redundant: every gate is load-bearing on natural footage for every paired seed.On rendered diagnostics, the return gate is consistently load-bearing while the other arms are worse on two of three seeds; gate costs are non-additive.
  • Decision atlas: On genuine boundaries the final probability rises with evidence and stays high through the transition, unlike the near-zero pseudo-event score.The gate read-outs fire around both cases, while the final score provides the separation.
  • Limitations: The study is bounded by latent-movement weaknesses on some true boundaries, single-corpus training, an unevaluated alternative slow-pathway family, and reported batched-throughput differences.The authors specifically identify jump cuts and wipes as real-transition subtypes where the mechanism does not help.

S5 Interactive visual-analytics tool

PERSIST EXPLORER links the method’s per-frame outputs to synchronized visual and latent-state views, enabling analysts to inspect boundary and within-shot behavior interactively.

  • Interactive views: Scrubbing the timeline synchronizes a filmstrip, per-frame signal panel, rotatable 3D latent-φ(t) trajectory, and frame inspector.The inspector exposes the three gate read-outs as the analyst moves through time.
  • Within-shot inspection: At a within-shot frame, the change gate reads zero, the suppression band stays closed, and the latent trajectory remains inside its within-shot cluster.This is presented as the quiescent counterpart to the boundary view.
Loading 2608.29287v1…