Source-linked AI summary
End-to-End Dense Video Captioning with Parallel Decoding
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, Ping Luo
TL;DR
Dense video captioning must generate multiple temporally localized captions without the hand-crafted proposal processing used by prior two-stage systems. PDVC formulates the task as parallel set prediction with an event counter and jointly predicts event locations and captions. On ActivityNet Captions and YouCook2, it achieves high-quality and state-of-the-art results, including a reported 22.22% relative BLEU4 improvement with predicted proposals over state-of-the-art scores.
Problem
Prior dense video captioning methods rely on a hand-crafted localize-then-describe pipeline whose proposal generation and selection limit end-to-end interaction between localization and captioning.
Method
PDVC formulates dense video captioning as set prediction, using parallel localization and captioning heads plus an event counter to predict an appropriate event set.
Results
22.22% relative improvement in BLEU4 over state-of-the-art scores is reported for PDVC with predicted proposals and C3D features, alongside improvements in METEOR, CIDEr, and SODA c.
Takeaways & Limitations
PDVC directly produces temporally localized sentence sets while simplifying proposal selection and avoiding non-maximum suppression or recurrent event sequence selection.
Takeaways & Limitations
Caption quality is worse for activities requiring fine-grained action cues or small objects, motivating fine-grained object features and stronger action recognition models.
Abstract
from arXiv · showhide
Dense video captioning aims to generate multiple associated captions with their temporal locations from the video. Previous methods follow a sophisticated "localize-then-describe" scheme, which heavily relies on numerous hand-crafted components. In this paper, we proposed a simple yet effective framework for end-to-end dense video captioning with parallel decoding (PDVC), by formulating the dense caption generation as a set prediction task. In practice, through stacking a newly proposed event counter on the top of a transformer decoder, the PDVC precisely segments the video into a number of event pieces under the holistic understanding of the video content, which effectively increases the coherence and readability of predicted captions. Compared with prior arts, the PDVC has several appealing advantages: (1) Without relying on heuristic non-maximum suppression or a recurrent event sequence selection network to remove redundancy, PDVC directly produces an event set with an appropriate size; (2) In contrast to adopting the two-stage scheme, we feed the enhanced representations of event queries into the localization head and caption head in parallel, making these two sub-tasks deeply interrelated and mutually promoted through the optimization; (3) Without bells and whistles, extensive experiments on ActivityNet Captions and YouCook2 show that PDVC is capable of producing high-quality captioning results, surpassing the state-of-the-art two-stage methods when its localization accuracy is on par with them. Code is available at https://github.com/ttengwang/PDVC.
1. Introduction
Dense video captioning addresses the limitations of single-sentence captioning for long, untrimmed videos by localizing and describing multiple events. PDVC replaces the hand-crafted two-stage pipeline with parallel set prediction, including an event counter, and achieves strong benchmark performance.
- Motivation: Dense video captioning localizes and captions multiple events to provide more detailed, coherent descriptions of long, untrimmed videos.
- Problem: Previous methods depend on proposal quality, anchor design, and post-processing such as non-maximum suppression, limiting interaction between localization and captioning.
- Method: PDVC formulates dense captioning as set prediction and feeds shared intermediate representations into parallel localization and captioning heads.
- Method: An event counter predicts the appropriate event-set size, helping segment videos into event pieces while avoiding missing information and replicated events.
- Results: Experiments on ActivityNet Captions and YouCook2 show state-of-the-art performance, including comparable results with a lightweight vanilla LSTM caption head.
2. Related Work
Related work treats dense video captioning as a multitask problem involving localization and captioning, while prior systems add context modeling, task-interaction mechanisms, or proposal-selection stages. PDVC builds on transformer set prediction to parallelize these functions and reduce reliance on hand-crafted processing.
- Temporal Event Proposals: Temporal event proposal methods use anchor-based or boundary-based strategies to predict event-containing segments in untrimmed videos.
- Dense Video Captioning: Dense video captioning combines event localization with event captioning, with prior work enriching event representations through context, relationships, or multimodal fusion.
- Dense Video Captioning: Earlier methods explored task interaction through language-reward prediction or differential masking, but these mechanisms provided limited coupling between localization and captioning.
- Dense Video Captioning: PDVC uses one-to-one matching between intermediate features and target events, unlike prior many-to-one anchor assignment, to obtain discriminative event representations.
- Caption Coherence: SDVC reduced proposals from 100 to 2.85 on average but required multi-step training and recurrent sequence selection, whereas PDVC parallelizes localization, selection, and captioning end to end.
- Transformer-Based Detection: Transformer-based detectors motivate PDVC’s set-prediction formulation, while deformable attention addresses the slow convergence associated with global attention.
3. Method
PDVC formulates dense video captioning as parallel set prediction: a deformable Transformer produces event queries, while localization, captioning, and event-count heads jointly determine the final event set.
- Set prediction: PDVC directly detects temporally localized captions as an event set, with the set size predicted rather than fixed manually.Each event contains a start time, end time, and caption.
- Transformer decoder: A deformable Transformer captures inter-frame, inter-event, and event-frame interactions and produces event query features from video features.Its deformable attention uses sparse sampling across multi-scale feature maps to improve convergence and representation ability.
- Parallel decoding: Three parallel heads predict event boundaries and confidence scores, generate captions, and estimate an appropriate number of events.The decoder conditions these heads on learnable event-query embeddings and reference points refined across decoding layers.
- Captioning head: The lightweight captioning head feeds each event query into a vanilla LSTM, while deformable soft attention adds language-conditioned access to nearby frame features.Sampling around reference points helps connect linguistic cues with video frames without requiring predicted event boundaries as attention limits.
- Event counter: The event counter globally pools event queries, predicts the event count, and selects the top Nset events with accurate boundaries and good captions.This addresses the readability trade-off in which too many events duplicate captions, whereas too few omit information.
- Training objective: Hungarian matching trains global correspondence between predicted events and ground truths using localization and classification costs, alongside captioning and event-count losses.The final objective sums set-prediction losses across Transformer decoder layers.
4. Experiments
PDVC is evaluated on ActivityNet Captions and YouCook2 across localization, dense captioning, paragraph captioning, efficiency, and ablation settings. The experiments report strong performance, efficient parallel inference, and benefits from caption supervision, deformable operations, event counting, and length modulation.
- Localization performance: PDVC achieves similar or slightly better event localization than SDVC and surpasses MFT on ActivityNet Captions.The comparison attributes this performance to parallel set prediction without hand-crafted proposal designs.
- Dense captioning performance: 22.22%/4.31%/75.87%/63.35% relative improvement over state-of-the-art scores is reported for BLEU4/METEOR/CIDEr/SODA c with predicted proposals on ActivityNet Captions.PDVC with C3D features achieves the best performance on all four listed metrics.
- Dense captioning performance: PDVC achieves state-of-the-art performance with considerable gains over other methods on all metrics on the YouCook2 validation set.The result is reported for dense captioning performance.
- Paragraph captioning performance: With predicted proposals, PDVC has comparable paragraph-captioning performance to previous methods using ground-truth proposals.The authors connect this result to query features covering main video segments and parallel decoding reducing dependence on accurate proposals.
- Efficiency: PDVC is more efficient than the compared two-stage methods because it predicts a few proposal-caption pairs in parallel without dense-to-sparse selection.Sparse sampling in MSDAtt is also cited as an efficiency factor.
- Interaction between Localization & Captioning: Caption supervision gives PDVC the best proposal descriptiveness score among four models despite a slightly lower F1 score.For high-precision proposals, caption supervision yields a considerable boost in caption quality, while deformable components improve temporally sensitive feature extraction and caption optimization.
5. Conclusion
PDVC formulates dense video captioning as set prediction, directly producing temporally localized sentences with parallel localization and captioning heads. Experiments on two benchmark datasets show high-quality captions that surpass state-of-the-art methods.
- PDVC formulates dense video captioning as a set prediction task.
- The framework directly produces temporally localized sentences without dense-to-sparse proposal generation and selection.
- Parallel event-localization and event-captioning heads exploit mutual benefits between the two subtasks.
- Experiments on two benchmark datasets show that PDVC generates high-quality captions and surpasses state-of-the-art methods.
6. Supplementary Materials
The supplementary materials analyze proposal distributions, activity-dependent captioning performance, and qualitative temporal-captioning behavior. They also describe caption-only proposal training and implementation choices for matching and inference.
- More Implementation Details: Caption-only training modifies reference points, deformable-attention sampling, and bipartite matching to learn event proposals without localization supervision.
- More Implementation Details: PDVC light is trained on YouCook2 with captioning loss, and its final-layer reference points serve as predicted proposals during inference.
- Visualization: Each event query describes a location mode, while all queries produce video-wide proposals with low redundancy and distributions similar to ground truth.
- Activity Types: PDVC captioning performance varies across activity types, with stronger METEOR for distinct scene cues or large objects than for fine-grained actions or small objects.
- Visualization: Figure 5 plots normalized proposal center positions against normalized proposal lengths across sampled validation videos, with colored clusters representing event queries.
- Visualization: Qualitative visualizations compare temporally localized captions from MT, PDVC light, and PDVC using predicted or ground-truth proposals.
- Visualization: Figure 6 reports PDVC METEOR scores across 200 activity classes, while Figure 7 marks incorrect and correct caption phases in red and green.