Source-linked AI summary
Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
Ye-Chan Kim, Seunghee Choi, SeungJu Cha, Si-Woo Kim, Hwiseon Kim, Hyungee Kim, Dong-Jin Kim
TL;DR
WSDVC must localize and describe events from ordered captions without temporal boundaries, while prior transition-caption synthesis is visually ungrounded and rigid. SBS uses VLM-generated frame narratives to detect meaningful transitions and adapt their temporal placement. On ActivityNet Captions and YouCook2, it achieves state-of-the-art captioning and localization results.
Problem
WSDVC lacks event boundaries, while prior LLM-generated transition captions are not visually grounded and are rigidly applied to every inter-event gap.
Method
SBS uses a VLM to generate frame-level narratives, selectively gate transitions from semantic variation, and localize them using semantic change points with temporal masks.
Results
SBS achieves state-of-the-art performance in both captioning and localization on ActivityNet Captions and YouCook2.
Takeaways & Limitations
Transition supervision should be applied selectively and localized according to visual evidence rather than uniformly at heuristic midpoints.
Takeaways & Limitations
SBS depends on VLM caption quality; generic, repetitive, or inaccurate descriptions can cause missed genuine transitions or spurious detections.
Abstract
from arXiv · showhide
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.
1 Introduction
Weakly-Supervised Dense Video Captioning learns event localization and captioning from ordered event descriptions without temporal boundaries, making language–visual alignment essential. SBS addresses rigid, visually ungrounded transition supervision by using a VLM to detect semantic transitions and adaptively place transition cues.
- Motivation: WSDVC learns localization and captioning from videos paired with temporally ordered descriptions, without annotated event start and end timestamps.Effective alignment between language supervision and visual content is therefore crucial for fine-grained localization.
- Limitations of Prior Work: LLM-based methods synthesize transition captions for every inter-event gap, but lack visual grounding and use rigid heuristic rules.These methods cannot determine reliably whether a gap contains a transition or where that transition occurs.
- Limitations of Prior Work: Assigning transition captions to every gap can add redundant noisy supervision and hallucinated content derived only from surrounding event descriptions.Figure 1 illustrates this failure with a hallucinated “pour a cup of water” description.
- SBS: SBS uses a VLM to generate visually grounded transition cues selectively, only when an informative event occurs, and places them according to video content.This reframes transition augmentation as visually grounded transition event discovery rather than text-only synthesis.
- SBS: SBS measures semantic variation in frame-level narratives and uses an adaptive gate to activate inter-event supervision only for salient changes.The gate threshold is determined by local variation statistics within each inter-event gap.
- SBS: The method refines transition placement by combining the semantic change point with the temporal midpoint instead of using a fixed midpoint alone.This keeps transition events from collapsing onto neighboring event centers while accounting for visual content.
- Results: SBS achieves state-of-the-art performance on ActivityNet and YouCook2 for both captioning and localization.The contribution is validated across both target datasets and both DVC tasks.
2 Related Work
Weakly-Supervised Dense Video Captioning replaces dense temporal annotations with ordered event descriptions, while related approaches learn localization and captioning through reconstruction-based objectives. SAIL extends this line of work with synthesized transition captions between neighboring events.
- Weakly-Supervised Dense Video Captioning: WSDVC methods learn from videos paired with ordered event descriptions rather than dense temporal boundary annotations.This setting motivates methods that jointly address event localization and captioning from weak supervision.
- Weakly-Supervised Dense Video Captioning: Early WSDVC approaches use cycle consistency to localize temporal segments from captions and reconstruct those captions from the segments.This couples localization and captioning through a reconstruction process.
- Weakly-Supervised Dense Video Captioning: ILCACM constructs event-specific visual features with Gaussian masks and learns localization and captioning through reconstruction without a cycle system.Its reconstruction objective implicitly drives both tasks simultaneously.
3 Proposed Method
SBS uses VLM-generated frame narratives to identify genuine inter-event transitions and adapt their temporal supervision. It combines selective gating with content-adaptive centers and widths to align auxiliary captions with visual dynamics.
- Event Representation: SBS represents each event with differentiable Gaussian temporal masks predicted from video features and event queries.The model predicts each event’s center and width with sigmoid-activated linear layers before constructing Gaussian masks.
- Narrative Generation: VLM-generated frame captions form a narrative flow used to search for semantic changes within inter-event gaps.BLIP-2 produces frame-specific captions, and adjacent caption embeddings are compared within each gap.
- Adaptive Inter-Event Selection: SBS uses an adaptive gate to inject transition supervision only when semantic variation exceeds a gap-specific threshold.The threshold is based on the gap’s mean and standard deviation, and the gate opens at g_n ≥ 0.5.
- Adaptive Inter-Event Masks: The adaptive transition center blends the neighboring-events midpoint with the semantic change point rather than using a fixed midpoint alone.The semantic change point is the location with the largest dissimilarity within the gap, while the midpoint remains a temporal regularizer.
- Adaptive Inter-Event Masks: SBS selects the transition-mask width that maximizes cross-modal alignment between masked video content and the VLM transition caption.Candidate widths are evaluated at the refined center using the caption’s CLIP text embedding.
- Training Objective: The gated attraction loss supplements captioning and contrastive losses, weighting transition supervision by transition confidence.SBS applies the attraction loss selectively to accepted intervals rather than uniformly across all gaps.
4 Experiments
SBS is evaluated on two weakly supervised dense video captioning benchmarks across captioning, localization, ablation, qualitative alignment, transition detection, and computational cost. It achieves strong performance while visually grounding and selectively applying inter-event supervision.
- Comparison with State-of-the-Art: SBS achieves the best captioning and localization performance among weakly supervised methods on ActivityNet Captions and YouCook2.On ActivityNet Captions, it obtains CIDEr 36.87 and F1 58.18, with gains attributed to recall 56.13 and precision 60.38.
- Ablation Studies: The ablation study shows that VLM-generated captions, selective inter-event guidance, and adaptive masks each improve performance, with their combination performing best.The components improve captioning and localization by grounding descriptions visually, suppressing noisy guidance, and tailoring mask centers and widths to content.
- Ablation Studies: All tested VLM variants consistently outperform the LLM-based baseline across evaluation metrics, indicating that visual grounding is more important than the particular VLM choice.The tested variants include BLIP-2, InternVL3, Qwen2.5-VL, xGen-MM, and SmolVLM2.
- Ablation Studies: SBS aligns transition captions more closely with meaningful visual changes than prior LLM-based synthesis, which injects captions into every gap and can misalign them temporally.The qualitative comparison highlights both selective transition generation and improved alignment with the underlying frames.
- Ablation Studies: Interpolating the semantic change point improves captioning and localization over SAIL’s fixed midpoint, with α = 0 producing the lowest scores.Figure 4 reports that α > 0 consistently improves performance.
- Ablation Studies: Textual semantic guides outperform raw visual features for adaptive gap selection, while SBS more faithfully represents inter-event regions through higher visual-caption similarity.The inter-event analysis also reports a substantial similarity increase over the LLM-based baseline.
- Ablation Studies: On human-annotated gaps, SBS’s gate substantially outperforms always-inject and random gating for transition identification.The validation set contains 95 gaps, including 36 with transitions and 59 without.
- Ablation Studies: SBS adds negligible computational cost, with training and inference times nearly identical to the baseline and comparable memory consumption.The cost analysis averages measurements over five runs.
5 Conclusion
SBS grounds inter-event supervision in visual evidence, selectively detecting transitions and localizing them according to video content rather than a fixed midpoint. Experiments on ActivityNet Captions and YouCook2 show state-of-the-art performance in captioning and localization.
- SBS selectively applies inter-event information only where the video exhibits a transition, rather than uniformly across all gaps.
- SBS localizes each transition according to actual video content instead of fixing it at a heuristic midpoint.
- A VLM serves as a transition-search tool that grounds both whether to introduce a transition and where to place it in visual evidence.
- SBS achieves state-of-the-art performance in both captioning and localization on ActivityNet Captions and YouCook2.
Limitations
SBS effectiveness is bounded by the quality of its VLM-generated frame captions, which can make the caption-space transition signal unreliable.
- SBS depends on VLM-generated frame captions whose generic, repetitive, or inaccurate descriptions can make transition detection unreliable.This can cause the adaptive gate to miss genuine transitions or open on spurious ones, especially in domains underrepresented in VLM pre-training.
Appendix
The appendix provides additional details and qualitative results supporting the paper's findings.
- The appendix contains additional details and qualitative results supporting the reported findings.
A Additional Implementation Details
The implementation samples frames per dataset, uses specified VLM captioning settings, filters very short inter-event gaps, and keeps VLM-generated narratives offline during training. Offline VLM generation takes comparable time to SAIL while producing better captioning and localization performance.
- ActivityNet Captions and YouCook2 use uniform sampling of 32 and 100 frames per video, respectively.
- BLIP-2 uses “What is happening in this image?”, while InternVL3 and Qwen2.5-VL use one-sentence image descriptions with a 30-token generation limit.
- Gaps containing fewer than 4 frames are discarded because they are unlikely to provide meaningful transitions or reliable statistics.
- VLM-generated narratives are created offline before training and used only to construct selective inter-event supervision, so inference has no VLM dependency.
- SBS requires 1H 46M of offline caption generation versus SAIL's 1H 38M, while yielding better captioning and localization performance.
B Computational Cost
SBS’s VLM-based caption generation has comparable offline cost to SAIL’s LLM-based synthesis while producing better captioning and localization scores.
- 1H 46M vs. 1H 38M, SBS’s VLM-based caption generation requires comparable offline time to SAIL’s LLM-based synthesis on ActivityNet Captions.Captions are generated offline before training, and only their precomputed outputs are used afterward.
- SBS yields clearly better captioning and localization scores than SAIL despite comparable offline caption-generation time.
C More Qualitative Results
Qualitative comparisons on ActivityNet Captions and YouCook2 show that SBS selectively identifies meaningful transitions and temporally aligns captions with the corresponding visual content.
- SBS discards uninformative inter-event gaps and supplies transition guidance only when genuine visual changes occur.Examples include rafting beginning and a jump being performed.
- SBS anchors transition captions at the correct temporal locations, whereas the LLM-based method can mismatch regions through hallucinated content and fixed placement.
- Figures A.1 and A.2 compare SBS’s VLM-based inter-event captions with prior work’s LLM-based captions on ActivityNet Captions and YouCook2.