PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Temporal Abstraction for General Video Understanding

Etienne Casanova, Sevan Brodjian, Pietro Perona

arXiv:2608.27929v1cs.CV

TL;DR

Video understanding tasks need temporal units, but existing approaches often use separate task-specific representations. STITCH extracts adaptive chunks from frozen video-text embeddings once per video and reuses them across tasks, remaining competitive across event detection, moment retrieval, and long-video frame selection while facing limitations from the frozen embedding space.

  • Problem

    Different video tasks often rely on separate task-specific temporal representations, despite all needing to identify meaningful moments in dense video.

  • Method

    STITCH encodes short video windows with a frozen video-text backbone, detects changes in the embedding sequence, and partitions the video into query-independent adaptive chunks reused across tasks.

  • Results

    Across event detection, moment retrieval, and long-video QA, STITCH remains competitive with specialized methods, including 64.6 R@1@0.5 on QVHighlights and gains over uniform sampling with eight input frames.

  • Takeaways & Limitations

    A single reusable chunked timeline can support localization, retrieval, and sparse frame selection without task-specific training.

  • Takeaways & Limitations

    STITCH depends on the frozen embedding space: subtle transitions may be missed, nuisance motion may cause over-segmentation, and one scale cannot capture every temporal organization level.

Abstract

from arXiv · show

Videos are expensive to analyze frame by frame, yet many video understanding tasks depend on knowing where relevant moments occur. A system may need to find when an action changes, locate the segment described by a sentence, or choose a few frames for a vision-language model. Existing methods often solve these problems separately, using task-specific training data or specialized architectures. We study whether a pretrained video-text model can provide enough temporal structure to support several of these tasks at once. We present STITCH, a training-free method that divides a video into semantically meaningful temporal chunks. STITCH embeds short video windows with a frozen video-text backbone and detects changes in the resulting embedding sequence. These chunks are computed once per video and reused across tasks. We evaluate STITCH on generic event boundary detection, language-based moment retrieval, and frame selection for long-video VLM reasoning. Across all three settings, STITCH remains competitive with more specialized methods while requiring no task-specific training, with especially clear gains when only a small number of frames or tokens can be processed. These results suggest that reusable temporal abstraction is a promising direction for general video understanding, allowing dense video streams to be converted once into semantic units that can be localized, retrieved, sampled, or reasoned over by downstream systems.

1 Introduction

Video understanding tasks need meaningful temporal units, but existing systems often build task-specific representations. STITCH instead extracts adaptive, query-independent chunks from frozen video-text embeddings for reuse across boundary detection, retrieval, and frame selection.

  • Boundary detection, moment retrieval, and frame selection all require identifying when meaningful information occurs in dense video.The motivation is shared across event changes, language-matched moments, and sparse evidence selection.
  • Existing systems often learn separate temporal representations for each task, tying localization to task-specific data and machinery.The proposed alternative separates finding useful temporal units from applying them downstream.
  • STITCH uses a frozen video-text backbone and embedding changes to divide videos into adaptive semantic chunks without labels or task-specific training.The chunks are computed independently of downstream queries and reused as a common temporal structure.
  • The same query-independent chunk structure supports event boundary detection, language-based moment retrieval, and frame selection for long-video VLM reasoning.STITCH treats temporal abstraction as a shared interface rather than a task-specific model component.
  • STITCH remains competitive with specialized methods across retrieval and reasoning benchmarks, particularly when downstream systems cannot process video densely.The supplied result passages identify competitiveness and stronger relevance under constrained processing budgets.

2 Related Work

Prior work addresses adaptive segmentation, moment retrieval, long-video frame selection, and unified localization through separate specialized or training-free approaches. These lines of work motivate STITCH’s shared, reusable temporal abstraction interface.

  • Adaptive temporal segmentation and chunking: Adaptive segmentation replaces fixed temporal grids with temporally coherent units for event, scene, action, and long-video processing.KTS Revisited is identified as a close training-free precedent for adaptive sampling.
  • Moment retrieval and training-free temporal grounding: Moment retrieval commonly uses supervised query-conditioned representations, while newer training-free methods score, refine, or filter candidate spans at inference time.The related-work passage places STITCH among training-free temporal-grounding approaches.
  • Long-video VLM frame selection and compression: Long-video VLMs face limited frame and token budgets, motivating adaptive selection, learned querying, redundancy reduction, narrative threading, and training-free heuristics.Uniform sampling can miss short or sparse evidence, which makes temporal selection central.
  • Unified localization frameworks: Unified localization frameworks combine multiple temporal tasks, typically through supervised pretraining or shared localization architectures.The cited examples unify grounding, action localization, segmentation, moment retrieval, and GEBD in different ways.

3 Approach

STITCH encodes short video windows with a frozen video-text model, detects semantic changes in the embedding trajectory, and partitions the video into adaptive chunks. The resulting query-independent structure is computed once and reused by lightweight downstream readouts.

  • Dense window embeddings: STITCH converts a video into a reusable chunk structure by encoding short temporal windows and detecting changes in their embedding sequence.The pipeline uses one frozen video-text backbone and adaptive partitioning.
  • Dense window embeddings: Each window covers a fixed span and is encoded from a small number of uniformly sampled frames using the frozen video-text backbone.Typical window spans are 0.5–2 seconds and typically use four frames, according to the supplied implementation passage.
  • Dense window embeddings: Nearby windows with similar embeddings indicate a shared semantic period, while sharp embedding changes suggest a transition into a new temporal unit.The shared embedding stream supports both chunking and later query-based scoring.
  • Semantic temporal chunking: Kernel change-point detection uses the full embedding trajectory rather than isolated adjacent-similarity thresholds to determine boundaries.This makes boundary decisions depend on local sequence structure.
  • Semantic temporal chunking: A cosine-kernel objective partitions the window sequence into contiguous segments, with a length-dependent penalty discouraging excessive segmentation.The penalty is β(T) = 2 Var(s) log T, and a minimum segment length prevents degenerate fragments.
  • Semantic temporal chunking: Short chunks may be merged with neighboring chunks when local semantic consistency improves without violating a maximum duration constraint.This post-processing reduces fragmentation while preserving content-adaptive structure.
  • Downstream readouts: Final chunks retain their original window-level embeddings instead of being re-encoded, preserving fine-grained evidence for downstream readouts.The same chunk structure is then used for boundaries, query scoring, or frame selection.

4 STITCH Variants

STITCH variants test the change-point solver and per-window frame budget. These experiments characterize whether the default configuration is necessary and how accuracy trades off against computational cost.

  • The default STITCH configuration uses standard kernel change-point detection and four sampled frames per temporal window.Variants alter one of these two components while keeping the rest of the pipeline unchanged.
  • STITCH-PELT replaces the standard solver with Pruned Exact Linear Time change-point detection, providing a linear-time chunking variant under pruning conditions.The primary design choice concerns how boundaries are selected from the embedding stream.
  • STITCH-2F reduces the per-window visual budget from four frames to two while preserving the remainder of the pipeline.This tests whether the method remains effective with cheaper window encoding.

5 Application 1: Event Detection

STITCH uses its adaptive chunk boundaries directly as event-boundary predictions, testing whether semantic changes in frozen video embeddings align with visible action or scene changes.

  • Event boundary detection: STITCH treats the final adaptive chunk boundaries as event boundary predictions without task-specific machinery.It encodes frozen window embeddings, applies cosine-kernel change-point detection, and post-processes the resulting chunks.
  • Event boundary detection: Predicted boundaries land near interpretable changes in visible action on two Kinetics-GEBD clips.

6 Application 2: Moment Retrieval

STITCH reuses query-independent chunks as moment-retrieval proposals, scoring them with frozen video-text embeddings and optionally grouping adjacent high-scoring chunks.

  • Query-independent proposals: STITCH reuses the same query-independent chunk structure as a proposal set for localizing language-described spans.The video-side chunk computation remains unchanged across queries.
  • Chunk scoring: Each chunk is scored by comparing the query embedding with its internal window embeddings using maximum similarity.This score preserves a strongest local match when only part of a chunk aligns with the query.
  • Retrieval: For single moments, STITCH selects or ranks chunks by score and can group adjacent chunks when spans extend across neighboring regions.Optional grouping also supports cases with multiple relevant moments.

7 Application 3: Video QA Understanding with VLMs

For long-video VLM question answering, STITCH uses shared chunks to allocate a limited frame budget through greedy or chunk-aware MMR selection. The main evaluation reports MMR with eight frames and compares it against uniform sampling across benchmarks.

  • Frame-budgeted reasoning: STITCH provides a content-aware chunk structure for allocating the few frames a VLM can inspect in long videos.Uniform sampling can spend frames on redundant or irrelevant regions.
  • Greedy selection: Greedy scores each chunk by its maximum query similarity and sends one representative frame from each selected chunk to the VLM.It selects chunks in decreasing score order using the frozen text and video embeddings.
  • MMR selection: Chunk-aware MMR selects individual windows using query relevance, redundancy penalties, and a penalty for repeatedly representing the same chunk.λ and γ control the relevance–diversity tradeoff and same-chunk penalty.
  • Shared configuration: The MMR variant uses one fixed parameter setting across MLVU, LongVideoBench, and VideoMME rather than tuning separately for each benchmark.The setting is selected once and applied unchanged across the three benchmarks.
  • Cross-task evaluation: STITCH is evaluated across event detection, moment retrieval, and long-video VLM QA with one training-free temporal abstraction front end.The table reports VLM accuracy gains over uniform sampling using LLaVA-OV 7B with eight input frames and the MMR selector.

8 Discussion

STITCH reuses one query-independent temporal chunk structure across event detection, moment retrieval, and long-video frame selection. It remains competitive with specialized methods, with strengths under training-free comparison and sparse visual budgets, while fine-grained boundaries and some QA benchmarks remain challenging.

  • STITCH is strongest against training-free, zero-shot, and unsupervised methods while remaining competitive with many specialized systems.The shared timeline is evaluated across event changes, sub-action boundaries, sentence-level localization, highlight retrieval, and frame-budget allocation.
  • Event detection: 83.9 Avg F1 on Kinetics-GEBD is close to supervised boundary detectors, while 44.8 Avg F1 on TAPOS is below supervised methods.The TAPOS gap indicates that reusable chunks capture broad transitions better than fine procedural sub-actions.
  • Moment retrieval: 64.6 R@1@0.5 on QVHighlights exceeds Moment-DETR and QD-DETR without moment annotations or a query-conditioned temporal head.Stronger supervised systems still perform better through proposal ranking and boundary refinement.
  • Long-video QA: With eight input frames, chunk-aware MMR improves over uniform sampling across MLVU, LongVideoBench, and VideoMME using shared hyperparameters.The largest gains occur on MLVU, while STITCH trails specialized selectors on LongVideoBench and VideoMME.
  • Method variants and limitations: STITCH remains limited by its frozen embedding space: subtle transitions can be missed, nuisance motion can cause over-segmentation, and one scale cannot capture every temporal level.The paper points toward hierarchical chunking, stronger backbones, and streaming variants as responses to these limitations.

9 Conclusion

STITCH converts dense video into reusable semantic chunks that support multiple downstream understanding tasks. This shared timeline separates temporal abstraction from task-specific localization, retrieval, and reasoning.

  • 9 Conclusion: STITCH converts short-window embeddings from a frozen video-text model into reusable semantic chunks for downstream video understanding.The same chunked timeline supports event boundary detection, language-based moment retrieval, and frame selection for long-video VLM reasoning.
  • 9 Conclusion: STITCH remains competitive with many supervised systems and is strongest against training-free, zero-shot, and unsupervised methods.The comparison spans event boundary detection, moment retrieval, and frame selection for long-video reasoning.
  • 9 Conclusion: The chunked timeline supports localization, scoring, sampling, or storage according to each task’s needs.This makes temporal abstraction a practical interface between raw video and downstream reasoning.

A Full Benchmark Results

The evaluation covers event boundary detection, moment retrieval, and frame-budgeted VLM reasoning, while also examining implementation choices for chunking and retrieval.

  • A.1 GEBD: Table 2 evaluates generic event boundary detection on Kinetics-GEBD and TAPOS using F1 scores across relative-distance thresholds.The table distinguishes supervised, unsupervised, zero-shot, and training-free systems.
  • A.2 Moment Retrieval: Tables 3 and 4 report language-based moment retrieval results for multiple STITCH chunk-scoring variants.The variants include max-similarity, mean pooling, GEM, and coherence-weighted mean.
  • A.2 Moment Retrieval: All four moment-retrieval pooling variants perform comparably, indicating limited sensitivity to the specific pooling rule.GEM interpolates between arithmetic mean and coordinate-wise maximum, while coherence weighting acts as a soft medoid.
  • A.3 VLM Frame Selection: The VLM evaluation measures multiple-choice accuracy with eight input frames across MLVU, LongVideoBench, and VideoMME.Both LLaVA-OV and Qwen3-VL are evaluated, with uniform sampling compared against selection methods.
  • Implementation: Chunking uses task-dependent sampling intervals and minimum segment lengths, with penalties and post-processing set separately for the evaluated tasks.The maximum chunk duration is 60 seconds, and parameter search was limited to values chosen on available training splits.

C Compute Resources

STITCH’s resource profile centers on frozen-backbone feature extraction and standardized VLM inference. The broader-impact discussion frames the method as useful for accessible video analysis but requiring safeguards in sensitive settings.

  • C Compute Resources: STITCH runs on a single consumer GPU, primarily for InternVideo2 feature extraction.The reported system uses an NVIDIA RTX 5080 with 16 GB VRAM and 32 GB RAM.
  • C Compute Resources: VLM inference uses hosted endpoints or APIs with temperature 0 and a maximum output length of 32 tokens.These settings match the multiple-choice benchmark format.
  • Evaluation Protocol: Bootstrap confidence bands in Figure 5 use 2,000 query-resampling iterations without a normality assumption.The bands are formed from the 2.5th and 97.5th percentiles of the bootstrap accuracy distribution.
  • Broader Impacts: STITCH may improve efficiency and accessibility by reducing dense processing and task-specific training requirements.The paper identifies video search, assistive navigation, educational retrieval, and long-recording analysis as potential applications.
  • Broader Impacts: The method is not presented as deployment-ready for high-stakes decisions because localization errors and misuse could create harms.The paper calls for privacy safeguards, human oversight, and domain-specific evaluation.
Loading 2608.27929v1…