Source-linked AI summary
DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding
Zhuo Cao, Bingqing Zhang, Sen Wang, Xue Li
TL;DR
Training-free VTG still faces frame-centric visual representations that miss temporal dependencies and fixed query embeddings that cannot resolve linguistic ambiguity. DSE-VTG combines frame- and clip-level visual evidence through Multi-scale Similarity Fusion with video-specific query refinement through Query-level Test-Time Adaptation. DSE-VTG establishes state-of-the-art training-free performance on standard benchmarks; on Charades-STA it achieves 51.30 mIoU, 5.61 above the strongest prior training-free baseline [19], and shows OOD robustness.
Problem
Training-free VTG still faces frame-centric visual representations that miss temporal dependencies and fixed query embeddings that cannot resolve linguistic ambiguity.
Source: §2.1 · p. 3Method
DSE-VTG combines frame- and clip-level visual evidence through Multi-scale Similarity Fusion with video-specific query refinement through Query-level Test-Time Adaptation.
Source: §1 · p. 2Results
Takeaways & Limitations
Across five benchmarks, DSE-VTG provides a scalable upgrade for similarity-based grounding pipelines and shows strong robustness to distribution shifts.
Source: §5 · p. 9The zero-shot protocol excludes task-specific training or fine-tuning and permits only a disposable per-query offset at test time.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Text-guided Video Temporal Grounding (VTG) aims to localize the relevant segments in an untrimmed video based on text queries, yet collecting dense temporal annotations and training task-specific models remain costly and brittle under distribution shift. Recent training-free VTG approaches mitigate this issue by directly matching pretrained vision-language representations, but they still face two fundamental information bottlenecks: frame-wise visual encoding overlooks temporal dynamics, while fixed query embeddings cannot resolve query ambiguity. To address these issues, we propose DSE-VTG, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both without any task-specific training. On the visual side, Multi-scale Similarity Fusion (MSF) combines frame- and clip-level similarities into a unified, temporally aware similarity profile. On the textual side, Query-level Test-Time Adaptation (Q-TTA) optimizes a lightweight additive offset to adapt the query embedding to the video at test time, without finetuning the backbone or calling external large language models. Extensive experiments on three standard and two OOD benchmarks show that DSE-VTG achieves state-of-the-art performance among training-free methods. On Charades-STA, it improves mIoU over the strongest prior training-free method by 5.61 points. Under distribution shift, DSE-VTG reaches 50.86 mIoU on Charades-CG Novel-Word, surpassing the strongest supervised baseline by 2.76 mIoU. Our code will be released upon acceptance.
1. Introduction
DSE-VTG targets visual and textual representation limits in training-free video temporal grounding with dual-side enhancement. It reports state-of-the-art training-free performance across standard and OOD benchmarks.
1. Introduction
- Training-free VTG uses pretrained vision-language representations for inference-time cross-modal matching, avoiding costly temporal annotations and task-specific training.The introduction also identifies susceptibility to distribution shifts as a limitation of fully supervised approaches.Source: §1 · p. 1
- Existing training-free methods miss temporal dependencies by treating videos as independently sampled frames, while fixed query representations cannot resolve linguistic ambiguity.The introduction illustrates ambiguity with vague references such as “something.”Source: §1 · p. 1
- Multi-scale Similarity Fusion combines frame- and clip-level evidence, while Query-level Test-Time Adaptation refines the query embedding using video context without updating VLM parameters.Clip similarities are projected onto a unified timeline, and Q-TTA uses pseudo-labels from initial similarities to optimize a lightweight offset.Source: §1 · p. 2
- DSE-VTG establishes state-of-the-art performance in the training-free setting; on Charades-STA, it reaches 51.30 mIoU, outperforming the strongest prior training-free baseline [19] by 5.61 mIoU.Source: §1 · p. 2
2. Related Work
Prior training-free VTG work motivates multi-scale temporal modeling and efficient query adaptation as complementary ways to improve alignment. DSE-VTG addresses both visual representation limits and query ambiguity.
2.1. Video Temporal Grounding
- Training-free VTG avoids task-specific training, but prior approaches include LLM-based query decomposition or temporal pooling and coherence clustering to address query and similarity issues.TFVTG decomposes complex queries into sub-events; TAG uses temporal pooling and temporal-coherence clustering.Source: §2.1 · p. 3
- Existing training-free methods often use a single visual temporal resolution, despite video-language models offering frame- and clip-level representations suited to events at different scales.DSE-VTG combines coarse event semantics with fine-grained boundaries.Source: §2.1 · p. 3
2.2. Query Understanding and Semantic Alignment
- Prior VTG methods either simplify queries without adaptation or use costly LLM decomposition, motivating lightweight video-context query refinement without external LLM inference or backbone fine-tuning.Queries can contain vague references and contextual dependencies that challenge direct vision-language alignment.Source: §2.2 · p. 3
3. Method
DSE-VTG performs zero-shot video temporal grounding by combining video-specific query adaptation with frame- and clip-level visual evidence. Its pipeline fuses the resulting similarities and decodes them into temporal boundaries.
3.1. Problem Formulation
- Under the zero-shot protocol, VTG predicts one or more temporal moments in an untrimmed video, with start and end timestamps and a confidence score for each.No component is trained or finetuned on a VTG dataset; the only test-time optimization is a disposable per-query offset.Source: §3.1 · p. 3
3.2. Overview
- The overview connects video and query encoding, query adaptation, similarity fusion, and structured interval optimization to produce temporal boundaries.MSF projects both similarity streams onto a common frame timeline before decoding.Source: §3.2 · p. 3
3.3. Multi-Granularity Feature Extraction
- A video-capable backbone extracts frame-level features and temporally aware clip-level features, preserving fine-grained local semantics alongside broader temporal dynamics.Frames are sampled at 3 fps by default; clips come from sliding temporal windows.Source: §3.3 · p. 3
3.4. Query-level Test-Time Adaptation
- Q-TTA adapts the query embedding with a disposable additive offset optimized from pseudo-labels, while keeping all VLM parameters frozen and discarding the offset after answering.The initial query and frame similarities provide the basis for pseudo-labeling; the offset is optimized with a binary cross-entropy objective.Source: §3.4 · p. 4
3.5. Multi-scale Similarity Fusion
- MSF aligns frame- and clip-level similarity streams on a common frame timeline, then fuses them into a unified profile for interval optimization.Frame scores offer fine temporal resolution but can be noisy, while clip scores add broader temporal semantics but lack boundary precision.Source: §3.5 · p. 5
3.6. Structured Interval Optimization
- Structured interval optimization selects query-relevant intervals from fused scores, first suppressing low-confidence regions and then ranking candidate proposals.It uses adaptive thresholding, a Box–Cox transformation, and timeline partitions from clustered frame features to define candidate boundaries.Source: §3.6 · p. 5
4. Experiments
DSE-VTG is evaluated across standard and out-of-distribution VTG benchmarks, where it reports strong training-free results. Ablations examine its two components, backbone choices, sensitivity, and computational cost.
4.1. Datasets & Evaluation Metrics
- Evaluation covers Charades-STA and ActivityNet Captions in-distribution, QVHighlights, and the Charades-CG and Charades-CD OOD splits, using mIoU, recall, mAP, and related thresholds.The OOD splits include Novel-Composition, Novel-Word, and Test-OOD.Source: §4.1 · pp. 5–6
4.3. Experimental Results
- 58.58 R@0.5 and 51.30 mIoU on Charades-STA improve over the prior training-free baseline [19] by 10.00 and 5.61, respectively; DSE-VTG also leads training-free methods on ActivityNet Captions.Table 1 covers Charades-STA and ActivityNet Captions.Source: Table 1 · p. 6, §4.3 · p. 7
4.2. Implementation Details
- Experiments use Qwen3-VL-Embedding-8B with 3-fps frame features and 4-second clip windows at 2-second stride; default Q-TTA uses 40 steps, AdamW, BCE loss, and early stopping.The MLLM comparison is a controlled reference under a unified protocol, not an estimate of each model's best attainable performance.Source: §4.2 · p. 6
4.3. Experimental Results
- 38.64 mAP on QVHighlights surpasses prior SOTA [50] by 5.49 mAP, with the largest gains under stricter localization criteria.All improvements are reported without task-specific training or query rephrasing.Source: Table 2 · p. 7, §4.3 · p. 7
- On Charades-CG, DSE-VTG reaches 46.46 and 50.86 mIoU on Novel-Composition and Novel-Word, surpassing TAG [19] by 4.42 and 3.26 mIoU; on Charades-CD it reaches 72.21 R@0.3 and 54.84 R@0.5.Charades-CD results exceed TAG by 4.51 and 4.56 at R@0.3 and R@0.5, respectively.Source: Table 4 · p. 7, Table 3 · p. 7, §4.3 · p. 7
4.4. Ablation Studies
- On Charades-STA, Q-TTA and MSF individually raise mIoU from 45.69 to 47.06 and 50.25, while their combination reaches 51.30; on QVHighlights, the full model reaches 38.64 mAP from 35.08.The ablation reports the components' gains as complementary.Source: §4.4 · p. 7, Table 5 · p. 8
- With Qwen3, frame-only and clip-only features achieve 47.06 and 42.82 mIoU, while fusion reaches 51.30 mIoU and raises R@0.7 from 27.69/15.40 to 32.23.The results suggest clip-level context complements frame-level temporal resolution.Source: §4.4 · pp. 7–8, Table 6 · p. 8
- Negative-only Q-TTA reaches 51.00 mIoU and combining positive and negative objectives reaches 51.30; performance scales with backbone capacity.At 40 steps Q-TTA takes approximately 25 ms per query; indexed end-to-end query processing takes 69 ms, and Figure 4 reports robustness to positive/negative frame ratios.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8
5. Conclusion
DSE-VTG combines multi-scale visual modeling with video-specific query refinement to address visual bias and query ambiguity in training-free Video Temporal Grounding. Across five benchmarks, it achieves state-of-the-art training-free performance and robustness to distribution shifts.
5. Conclusion
- DSE-VTG combines Multi-scale Similarity Fusion for frame- and clip-level temporal modeling with Query-level Test-Time Adaptation for video-specific query refinement.The framework addresses image-centric visual bias and static query ambiguity in training-free Video Temporal Grounding.Source: §5 · p. 9
- Across five benchmarks, DSE-VTG achieves state-of-the-art training-free performance and strong robustness to distribution shifts.Source: §5 · p. 9
- DSE-VTG is presented as a simple, scalable upgrade for similarity-based grounding pipelines.Source: §5 · p. 9