PaperlayerView exact revision on arXiv

Source-linked AI summary

Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding

Alberto Presta, Michal Byra, Grzegorz Stefański, Karol Szurkowski, Eryk Kołodziejczyk, Krzysztof Arendt

arXiv:2609.31135v1cs.CVcs.AIcs.MM

TL;DR

STVG requires localizing a text-described target in space and time, while many existing approaches rely on costly architectures, complex optimization, or large inference models. Pocket-STVG combines efficient pretrained components in a cascade with query-independent video indexing and alternative temporal decoders. With fewer than 90M parameters, it matches weakly supervised methods on HCSTVG-v2 m vIoU and improves on earlier zero-shot approaches, but remains below weakly supervised methods at strict spatial overlap.

  • Problem

    STVG must localize a text-described target in both space and time, while many approaches rely on computationally expensive architectures, complex optimization, or large models at inference.

    Source: §1 · pp. 1–2

  • Method

    Pocket-STVG combines pretrained temporal and spatial components in a cascade, precomputes video representations independently of the query, and uses either T-Unet or thresholding for temporal localization.

    Source: §1 · p. 2, §3.3.3 · p. 4, §5 · p. 8

  • Results

    With fewer than 90M parameters, P-STVG matches weakly supervised methods in m vIoU on HCSTVG-v2 and improves on earlier zero-shot approaches at a fraction of their memory and computational cost.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Query-independent indexing and a single-forward-pass inference pipeline offer a compact performance–efficiency trade-off for video grounding.

    Source: §1 · p. 2, §5 · p. 8

    P-STVG remains below weakly supervised methods on both benchmarks under strict spatial overlap, and its efficiency was measured on a resource-constrained desktop GPU rather than mobile hardware.

    Source: §5 · p. 8

Abstract

from arXiv · show

Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.

1. Introduction

STVG must localize a queried target in both space and time, but resource-efficient deployment remains challenging because many approaches use complex optimization or large inference models. Pocket-STVG combines efficient pretrained components in a lightweight, query-independent indexing and inference pipeline, with fewer than 90M parameters and competitive performance at lower memory and computational cost.

Motivation

  • STVG localizes a text-specified action both spatially and temporally, requiring the system to combine multimodal information while focusing on relevant targets.The paper notes applications including anomaly detection and robotics.Source: §1 · p. 1
  • Existing weakly supervised and zero-shot approaches can rely on complex iterative optimization or large inference models, challenging offline use on resource-constrained edge devices.The introduction contrasts this with the potential for smaller architectures distilled from image-text foundation models.Source: §1 · pp. 1–2

Contribution

  • P-STVG precomputes query-independent video embeddings, then uses lightweight temporal and spatial decoders for efficient inference across long videos and large collections.Its temporal decoder can use a 1D U-Net or parameter-free thresholding.Source: §1 · p. 2
  • With fewer than 90M parameters, P-STVG achieves competitive performance with weakly supervised and zero-shot methods while substantially reducing memory footprint and computational cost.The paper presents it as a compact performance–efficiency operating point for resource-constrained grounding.Source: §1 · p. 2

2. Related works

Related work spans fully supervised tube prediction, weak supervision with video-level descriptions, and zero-shot grounding without task-specific training. The approaches differ in their annotation needs and in how they connect temporal and spatial reasoning.

2.1. Fully supervised STVG

  • Recent fully supervised methods predict target tubes directly, while early methods first generated candidate tubes; these systems use spatial and temporal labels and often prioritize benchmark performance over size and generalization.Target-aware query modules and semantic bridges between temporal reasoning and spatial decoding are among the approaches described.Source: §2.1 · p. 2

2.2. Weakly supervised STVG

  • Weakly supervised STVG uses video-level descriptions rather than costly frame-level boxes, with methods aligning language and visual regions or connecting frame-level boxes into tubelets.Other approaches use latent alignments with variational EM or reconstruct masked query phrases to refine tube boundaries.Source: §2.2 · pp. 2–3

2.3. Zero-shot STVG

  • Zero-shot STVG seeks spatio-temporal localization without task-specific training or annotated boxes and intervals, using pretrained vision-language models or agentic reasoning frameworks.Reported strategies include latent-variable localization with E- and M-steps and MLLM grounding tokens.Source: §2.3 · p. 3

3. Method

P-STVG defines grounding as temporal interval detection followed by spatial box prediction, organized as query-independent indexing and query-conditioned inference. It offers two temporal-decoding options: adaptive thresholding or a lightweight T-Unet.

3.1. Preliminaries

  • STVG predicts a time-bounded sequence of boxes, and P-STVG divides the task into temporal grounding followed by spatial grounding.The temporal stage identifies the interval; the spatial stage extracts boxes within it.Source: §3.1 · p. 3

3.2. General architecture

  • During indexing, the video is split into chunks and spatial and temporal embeddings are computed; at inference, the query embedding guides interval prediction and bounding-box decoding.The pipeline projects the query embedding for alignment with the spatial decoder, and users can treat temporal and spatial grounding as distinct tasks.Source: §3.1 · p. 3, §3.2 · p. 3

3.3. Temporal video grounding

  • Videos are partitioned into equal-sized, non-overlapping chunks, and two temporal video-decoding techniques are provided for inference.The architecture retrieves image embeddings for the predicted interval before spatial box decoding.Source: §3.3 · pp. 3–4, §3.2 · p. 3

3.3.1. Temporal Video Encoder (TVE)

  • The temporal video encoder uses pretrained Tiny MobileViCLIP to produce video embeddings that are compared with the query embedding during inference.Tiny MobileViCLIP is described as designed for reliability and efficiency, including edge-device deployment.Source: §3.3.1 · p. 4

3.3.2. Temporal Video Decoder with thresholding

  • Adaptive zero-shot thresholding selects chunks whose normalized query–video similarity exceeds a threshold, then fills gaps and retains the longest connected interval.The threshold parameter τ is fixed at 0.5 without evaluation-set tuning; this variant adds no parameters and uses no relevant-dataset training.Source: §3.3.2 · p. 4

3.3.3. Temporal video decoder with Temporal Unet

  • T-Unet uses one-dimensional convolutions in a U-Net-like module to aggregate information across video chunks and model long-range temporal dependencies.It combines similarity vectors and chunk–query features, then thresholds its sigmoid outputs and applies the same gap-filling post-processing.Source: §3.3.3 · pp. 4–5

3.4. Spatial video grounding

Spatial grounding uses image embeddings computed during indexing and MDETR to predict boxes within the temporally localized interval. The section also describes adapting MobileViCLIP’s text encoder for MDETR and training the temporal U-Net with Dice loss.

3.4. Spatial video grounding

  • Given a localized interval, the spatial decoder applies SID to indexed image embeddings to extract bounding boxes for its frames.MDETR supplies the natural-language-conditioned object detection component.Source: §3.4 · p. 5
  • Character-level token offsets let the adapted MobileViCLIP text encoder support MDETR’s soft token prediction, while an attention mask excludes padding embeddings from cross-modal attention.The tokenizer maps each subword to its start and end character indices in the original query.Source: §3.4 · p. 5

3.5.1. MDETR tuning with text encoder alignment

  • MDETR is fine-tuned to align with MobileViCLIP’s text encoder using its original soft-token-prediction and contrastive-alignment strategy, plus distillation from a frozen RoBERTa-based MDETR teacher.The training uses Flickr30k, MS COCO, and Visual Genome; the distillation term is added to the MDETR loss with λ=1.Source: §3.5.1 · pp. 5–6
  • Table 1 compares methods on HCSTVG-v1/v2 using m_tIoU and m_vIoU, and counts only modules exposed to STVG annotations for its STVG-supervised parameter totals.Source: Table 1 · p. 6

3.5.2. T-Unet training

  • T-Unet is trained to extrapolate the temporal interval with Dice loss; training runs for 50 epochs with Adam at 1 × 10^-4, reduced to 5 × 10^-5 after epoch 35.The loss compares shot-level predictions with ground-truth labels.Source: §3.5.2 · p. 6

4. Experiments

Experiments compare Pocket-STVG on HCSTVG-v1/v2 and declarative VidSTG, alongside an efficiency analysis and temporal-decoder studies. Results show dataset-dependent comparisons with weakly supervised methods, while the compact pipeline uses fewer parameters and less inference cost.

4.1. Datasets and Evaluation

  • Evaluation covers HCSTVG-v1/v2 and declarative VidSTG, using m tIoU, m vIoU, and vIoU@R to measure temporal, spatio-temporal, and thresholded overlap.Videos use 20 chunks for HCSTVG and 32 for VidSTG; T-Unet is trained only on temporal labels, without bounding boxes.Source: §4.1 · p. 6, §4.2 · p. 6

4.3.1. Quantitative results

  • On HCSTVG-v2, P-STVG variants reach 22.31 and 22.84 m vIoU versus 20.0 for STPro; on VidSTG, P-STVGUnet reaches 46.57 m tIoU versus CoSPal's 41.10.On VidSTG its m vIoU is 15.70, comparable to weakly supervised approaches at 15.5–16.0.Source: §4.3.1 · p. 7, Table 2 · p. 7
  • On HCSTVG-v1, P-STVGUnet's 18.92 m vIoU exceeds WINNER and STPro but trails TubeRMC and CoSPal; both variants remain below weakly supervised methods on v@.5.On VidSTG, the fully zero-shot variant reaches 14.25 m vIoU, exceeding only RedCircle.Source: §4.3.1 · p. 7
  • With ground-truth temporal intervals on HCSTVG-v2, P-STVG reaches 39.73 m vIoU and 32.85 v@.5, indicating that imprecise temporal intervals account for most end-to-end loss.Its end-to-end results are 22.84 m vIoU and 9.15 v@.5.Source: §4.3.1 · p. 7
  • P-STVG outperforms zero-shot methods RedCircle and ReCLIP, while agentic and MLLM-based approaches achieve higher accuracy using models of several billion parameters.On VidSTG, the full architecture remains below 90M parameters and nearly two orders of magnitude smaller than MLLM-based zero-shot methods.Source: §4.3.1 · p. 7

4.3.2. Efficiency analysis

  • Under the HC-STVG inference protocol, P-STVG has lower peak memory usage, shorter query time, and fewer TFLOPs than the compared methods.The comparison uses an RTX 4060 with 8 GB VRAM; UNLEASH required an H100, so its figures use a different hardware budget.Source: §4.3.2 · p. 7, Table 3 · p. 7

4.4.1. Different T-Unet configuration

  • Across the T-Unet grid search, m tIoU correlates more strongly with depth L (0.62) than with MC (0.11) or CC (0.20).TV D* outperforms TV D3 and TV D4 despite roughly an order of magnitude fewer parameters, and matches TV D5 with one thirteenth of its budget.Source: §4.4.1 · p. 8, Figure 4 · p. 8, Table 4 · p. 8

4.4.2. Different number of chunks

  • Chunk count creates a trade-off: 24 chunks use 49.3 KB per video versus 44 KB for 20, while performance does not improve monotonically.Too few chunks, such as 12, reduce performance through coarser temporal predictions; excessive chunks may add noise and shorten the temporal span used by MobileViCLIP.Source: §4.4.2 · p. 8, Table 5 · p. 8

5. Conclusion

Pocket-STVG combines efficient pre-trained components in a compact cascade with separate indexing and inference stages. It matches weakly supervised methods on HCSTVG-v2 m vIoU and improves on earlier zero-shot approaches, but strict-overlap accuracy and mobile-hardware validation remain limitations.

5. Conclusion

  • Pocket-STVG combines a temporal video encoder and aligned text encoder based on MobileViCLIP, an MDETR-derived spatial encoder-decoder, and temporal localization by thresholding or a lightweight U-Net.Query-independent video representations are precomputed, so inference reduces to a single forward pass.Source: §5 · p. 8
  • With fewer than 90M parameters, P-STVG matches weakly supervised methods on HCSTVG-v2 m vIoU and improves on earlier zero-shot approaches at a fraction of their memory and computational cost.Its vIoU@0.5 remains below weakly supervised methods on both benchmarks, and efficiency was measured on a resource-constrained desktop GPU rather than mobile hardware.Source: §5 · p. 8
  • The authors identify improving temporal precision as the most promising next step, including finer or overlapping chunks and propagating boxes across frames near interval boundaries.They also propose mobile-hardware and longer-video validation as future directions.Source: §5 · p. 8

A. Distillation in MDETR tuning

The Flickr test results compare MDETR fine-tuning configurations for text-encoder alignment, with the teacher network shown as a reference. Fine-tuning from the teacher provides the major gain, while distillation contributes marginally.

A. Distillation in MDETR tuning

B. Qualitative samples

The qualitative examples compare temporal grounding by thresholding and U-Net, while noting that spatial grounding can differ because the methods consider different frames.

B. Qualitative samples

C. Zero-shot quantitative results using different threshold

The zero-shot thresholding evaluation reports temporal results on HCSTVG-v2, with the best performance obtained for thresholds above 0.35.

C. Zero-shot quantitative results using different threshold

D. Estimation of state-of-the-art parameters

Table 8 presents parameter estimates for selected state-of-the-art zero-shot and weak-shot methods; the authors caution that estimates may not be exact but are useful for comparing orders of magnitude.

D. Estimation of state-of-the-art parameters

Loading 2609.31135v1…