Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Wenzheng Zeng; Siyi Jiao; Chen Gao; Hwee Tou Ng; Mike Zheng Shou

Source-linked AI summary

Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng, Mike Zheng Shou

arXiv:2607.02963v1Jul 3, 2026cs.CVcs.AIcs.MM

TL;DR

Dense video captioning is hindered by the latency of strictly token-by-token decoding, especially as videos become longer and contain more events. PadCaptioner restructures dependencies for event-wise parallel autoregressive decoding and achieves state-of-the-art grounding and captioning with a 3.8× wall-time decoding speedup.

Problem
Strictly token-by-token decoding makes dense video captioning increasingly inefficient as video length and event density grow.
Method
PadCaptioner uses latent event-level planning and event-wise parallel decoding to relax weak cross-event dependencies while preserving sequential coherence within events.
Results
3.8× actual wall-time decoding speedup accompanies state-of-the-art performance on event grounding and captioning across benchmarks.
Takeaways & Limitations
PadCaptioner improves inference efficiency and temporally grounded captioning accuracy while generalizing to other temporally grounded audio-visual understanding tasks.
Takeaways & Limitations
The model may still generate coarse-grained captions for long events, motivating improved information aggregation or representation learning.

Abstract

from arXiv · show

Dense video captioning aims to generate temporally grounded descriptions of video events, benefiting both event-level video understanding and generation. In this domain, autoregressive video large language models have emerged as a prevalent paradigm due to their strong generative and cross-modal modeling capacity. However, generating dense captions under the token-by-token paradigm severely limits inference efficiency and hinders scalability as video length and event density increase. In this work, we propose a parallelized autoregressive framework that not only improves generation efficiency but also enhances temporally grounded captioning performance. Our key insight is to exploit the weak local dependencies across temporally distinct events to restructure the causal dependency graph, thereby enabling lossless parallel generation. Specifically, tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding to preserve local semantic coherence. To realize this insight, we introduce two key components for lossless parallel decoding: (1) a latent global planning mechanism that automatically learns the event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, guiding subsequent dependency restructuring and parallel decoding; and (2) an event-factorized parallel decoding mechanism that effectively balances local focus with global inter-event awareness. Experiments on various benchmarks demonstrate the clear advantage of our approach in both efficiency and performance in omni-modal event grounding and captioning. Project website: https://github.com/showlab/PadCaptioner.

1 Introduction

PadCaptioner restructures weak cross-event dependencies to enable lossless parallel autoregressive decoding while preserving within-event coherence. It achieves state-of-the-art event grounding and captioning with a 3.8× wall-time decoding speedup and generalizes to other temporally grounded omni-modal tasks.

Motivation: Dense Video Captioning jointly localizes densely occurring meaningful events and generates temporally grounded captions, supporting applications from egocentric perception to video search.Unlike conventional video captioning, DVC describes multiple temporally localized events in untrimmed long videos.
Results: PadCaptioner surpasses the previous SOTA [8] in grounded captioning accuracy and achieves a 3.8× actual wall-time decoding speedup on LongVALE [13].The framework achieves SOTA performance on both event grounding and captioning while improving decoding efficiency.
Core insight: The framework exploits weak dependencies across temporally distinct events by decoding them in parallel while retaining sequential decoding within each event.This restructures the causal dependency graph without sacrificing local semantic coherence.
Method: A latent global planning mechanism autoregressively generates compact event tokens that represent temporally coherent events and preserve inter-event causal structure.The event tokens are conditioned on the audio-video input and guide subsequent parallel decoding.
Method: Event-wise decoding decomposes generation into event-conditioned subchains that run in parallel across events while remaining sequential within each subchain.Each subchain is anchored by its corresponding global event token Gi.

2 Related Work

Dense video captioning has progressed from proposal-based and query-based systems to multimodal LLMs, which improve performance and generalization but retain sequential-decoding inefficiency. Related research explores parallel sequential generation, latent reasoning, and temporally grounded multimodal LLMs, motivating latent planning for event-level structure and audio-visual semantics.

Dense video captioning: DVC evolved from proposal-and-caption pipelines to query-based frameworks that jointly localize events and predict captions [22,35,54,58].
Dense video captioning: Video large language models have become the dominant DVC paradigm [42], offering stronger language priors, cross-modal reasoning, performance, and task generalization than traditional task-specific systems.Their strictly sequential decoding process nevertheless causes substantial inference inefficiency.
Parallel prediction in sequential generation: Parallel generation methods include diffusion-based language models [39] [69] and query-based DVC systems such as PDVC, but diffusion methods require iterative refinement and PDVC cannot inherit pretrained MLLM knowledge and vision-language priors.These differences limit practical speedup for diffusion-based methods and performance or generalization for conventional query-based systems.
Latent reasoning and temporally grounded multimodal LLMs: Latent-reasoning methods use intermediate latent tokens for grounded image and video perception and long-horizon reasoning [10] [53], while temporally grounded multimodal LLMs target time-aware, event-grounded video understanding [16] [56] [73].The work applies latent planning to event-level structure and adaptively aggregates audio-visual semantics into latent tokens for expressive decoding.

3 Method

The method restructures Omni-LLM’s strictly autoregressive dependency chain by exploiting weak cross-event dependencies, enabling event-wise parallel decoding while preserving within-event coherence. It introduces latent global planning to infer temporally grounded event structure and provide event-specific anchors for focused parallel generation.

3 Method: Given synchronized audio-visual input, modality-specific encoders produce token sequences that are grouped into temporally aligned multimodal chunks and concatenated with instruction embeddings as the decoder prefix.The standard Omni-LLM decoder then generates tokens autoregressively conditioned on the prefix and all previously generated tokens.
3 Method: The framework restructures token dependencies across temporally distinct events, replacing redundant cross-event serialization with compact global representations while retaining fine-grained interactions within events.This design targets efficient parallel generation without sacrificing temporally grounded caption quality.
Latent Global Planning: Latent global planning autoregressively generates event tokens {G1, …, GK, S}, with K adaptively inferred rather than predefined to capture the video’s intrinsic temporal structure.Each Gi is a condensed semantic representation intended to encode a temporally coherent event.
Latent Global Planning: A similarity-based grounding constraint aligns each global event token with audio-visual features inside its ground-truth event interval and suppresses similarity outside it.The constraint uses binary temporal indicators and normalized dot-product similarity, making Gi reliable anchors with intrinsic localization capability.
Event-Factorized Parallel Decoding: During event-wise parallel decoding, the shared multimodal prefix and generated global event tokens encode inter-event structure, while each Gi conditionally anchors a branch focused on its designated event.The event token’s event-specific semantics are obtained by aggregating relevant audio-visual prefix tokens from its grounded temporal segment.

4 Experiments

PadCaptioner improves dense video captioning performance while substantially accelerating decoding, and generalizes across multiple omni-modal temporally grounded tasks. Ablations attribute these gains to latent planning, event-factorized attention, and adaptive audio-visual semantic aggregation.

LongVALE: PadCaptioner outperforms recent state-of-the-art methods across LongVALE dense-captioning metrics, with gains of at least 6.7% in grounding F1 and 6.1% in captioning Sim.The model is also relatively lightweight compared with methods using 7B-scale models.
Efficiency analysis: Around 3.7× faster total decoding and 3× faster per-token decoding than the previous state of the art are achieved on both LongVALE and ChronusAV.Measurements use actual wall-clock time on a single NVIDIA A6000 GPU and compare against ChronusOmni (7B) [8] and a 3B base model [50].
Generalization: PadCaptioner performs consistently well on LongVALE’s Omni-TVG and Omni-SC tasks and competitively across all six ChronusAV tasks, supporting broad omni-modal temporal grounding and cross-modal reasoning.The ChronusAV tasks emphasize time awareness and cross-modal reasoning [8].
Ablation studies: Parallel decoding delivers about 1.63× overall and 1.66× per-token speedups, while latent planning substantially improves both event grounding and caption quality.Latent planning also provides event-level structure needed for dependency restructuring and parallel decoding; the speed comparison uses Table 6 rows 2 and 3.
Ablation studies: Full event-factorized attention and adaptive audio-visual semantic aggregation improve performance by enabling global event tokens to provide cross-event awareness and expressive event-focused anchors.Removing semantic aggregation causes a substantial performance drop, while standard causal attention also significantly degrades performance under parallel decoding.

5 Conclusion and Limitations · Appendix

The paper concludes that PadCaptioner addresses inefficient sequential decoding by restructuring dependencies around latent event-level planning and parallel decoding. It also identifies limitations in event-level granularity and coarse captioning for long events, motivating finer-grained parallelization and improved information aggregation.

5 Conclusion and Limitations: PadCaptioner restructures unnecessary cross-event token dependencies through latent event-level planning and parallel decoding to address inefficient sequential generation in Video-LLM dense captioning.The framework is designed to balance local decoding with event-level dependency restructuring.
Appendix: The appendix material supplied here contains no additional claims beyond the conclusion and limitations passages.No appendix-specific evidence is present in the provided passages.
5 Conclusion and Limitations: The dependency-graph restructuring is limited to event-level structures, leaving sub-events and spatiotemporal instances as directions for finer-grained parallelizable units with less reliance on human-defined annotations.Future work could learn parallelizable units from finer-grained cues rather than restricting restructuring to event-level structures.
5 Conclusion and Limitations: Despite adaptive semantic aggregation improving grounding and caption quality, the model may still produce coarse-grained captions for long events.This limitation motivates better information aggregation for long-event captioning.

A More Implementation Details

The appendix specifies implementation choices for multimodal frame processing and event grounding, including a general video–audio token similarity formulation. It also describes two inference-time grounding strategies and notes robust threshold behavior and broader task generalization.

Implementation: Frames are capped at 61,250 pixels and proportionally resized when exceeding this resolution, following.
Event grounding: Grounding via feature similarity projects each global event token Gi and the video–audio prefix tokens through a lightweight two-layer MLP before similarity computation.The formulation compares each Gi with all video–audio prefix tokens and aggregates token similarities within the same temporal segment.
Event grounding: Inference supports grounding through textual timestamp generation or feature similarity matching, with training constraints giving each Gi intrinsic foreground-event discrimination.
Event grounding: A segment is included in the grounded interval when its similarity exceeds a fraction of the event token’s maximum, and threshold 0.5 yields satisfactory, stable performance across a reasonable range.The appendix reports this robustness in the LongVALE and ChronusAV threshold studies.
Generalization: The framework extends beyond dense video captioning to broader temporally grounded video-understanding tasks because it jointly models temporal grounding and textual generation.

B More Empirical Analysis · B.1 The Impact of Caption Length

Caption length alone does not determine caption quality: the proposed method outperforms the previous SOTA [2] while producing shorter captions, whereas naive fine-tuning lengthens captions without improving metrics.

B.1 The Impact of Caption Length: Longer captions do not necessarily imply better quality.The section’s overall finding is supported by both the naive fine-tuning and proposed-method comparisons.
B.1 The Impact of Caption Length: The proposed method outperforms the previous SOTA [2] with shorter average captions on ChronusAV: 549 versus 685 tokens.This directly shows that higher caption quality does not require longer outputs.
B.1 The Impact of Caption Length: Naive base-model fine-tuning increases average caption length by 2.5× without improving caption metrics.The comparison excludes the proposed designs and isolates the effect of naive fine-tuning.
B.1 The Impact of Caption Length: Without the proposed designs, naive fine-tuning produces substantially longer captions.Average caption length rises by 2.5× under the naive base-model fine-tuning setup.
B.1 The Impact of Caption Length: The proposed method achieves a lower average caption length than the previous SOTA [2].On ChronusAV, the averages are 549 and 685 tokens, respectively.
B.1 The Impact of Caption Length: Caption-length increases from naive fine-tuning are not accompanied by metric improvements.The average caption length grows by 2.5×, while caption metrics remain unimproved.

B.2 Base-Model Generalization

The proposed components generalize beyond the main paper’s base model to a different base model. On ChronusAV with fair 12K-data training, Table A3 reports consistent improvements, indicating the method is not tied to one base model.

Base-model generalization: The generalization test extends evaluation beyond the main paper’s adopted base model.It examines whether the proposed components transfer to a different base model.
Base-model generalization: The results suggest that the method is not tied to a specific base model.This conclusion follows from the consistent improvements observed with the alternative base model.
Base-model generalization: Consistent improvements on ChronusAV with fair 12K-data training show that the proposed components generalize to a different base model.The comparison is reported in Table A3.

B.3 Degree of Parallelization

Parallelization is driven by the number of predicted event subchains, with more subchains yielding greater speedup without degrading performance.

B.3 Degree of Parallelization: The method predicts 13.2 event subchains on average, which determines its degree of parallelization.
B.3 Degree of Parallelization: Speedup over serial decoding increases with subchain count, following a log-like trend in Fig. A1.
B.3 Degree of Parallelization: Parallel decoding does not degrade performance, as validated in the main paper.

B.4 Training Cost · B.5 Inference Requirement

Under a 12K-sample budget, the method’s modified causal attention increases training memory and time without dedicated kernel optimization. During inference, parallel decoding preserves KV-cache size, with peak GPU memory remaining below 30GB.

B.4 Training Cost: The training-cost analysis uses a 12K-sample training budget.
B.4 Training Cost: Without dedicated kernel-level optimization, the method incurs about 0.3× extra memory usage during training on H200 GPUs.
B.4 Training Cost: Training takes 1.4× longer at the same batch size and 1.9× longer at the maximum feasible batch size on H200 GPUs.
B.4 Training Cost: Modified standard causal attention is not fully compatible with some optimized kernels, including FlashAttention-2 [3].
B.5 Inference Requirement: Parallelized decoding does not increase KV-cache size when total generated caption length remains unchanged.
B.5 Inference Requirement: Additional inference memory mainly comes from transient activations for predicting K tokens simultaneously rather than one token, remaining manageable below 30GB peak GPU memory.

C Qualitative Analysis

Qualitative analyses show that the method improves temporal localization, caption coherence, and audio-aware perception in dense video captioning, while generalizing to multiple temporally grounded omni-modal tasks. Failure analysis identifies imperfect event splitting and grounding errors as key limitations.

Dense video captioning: The method produces more accurate event boundaries and more coherent, informative captions than the compared recent SOTA, especially for audio-related events such as music and speech.The comparison highlights better alignment with semantic transitions and more effective exploitation of acoustic cues.
Omni temporal grounding: On V2T and A2T omni temporal grounding, the method yields more accurate temporal boundaries, whereas the strong baseline often includes irrelevant segments.The advantage remains evident when neighboring frames are visually similar, making visual-only localization difficult.
Cross-modal grounding and captioning: For A2V and V2A cross-modal grounding and captioning, the method reasons over the complementary modality and generates faithful descriptions without hallucination.These tasks require grounding an event from one modality before describing its temporal segment using the other.
Omni segmentation captioning: Omni segmentation captioning evaluates whether the model interprets user-specified temporal intervals and describes the corresponding event across T2V and T2A variants.T2V targets visual descriptions, while T2A targets audio descriptions.
Failure case analysis: Grounding remains imperfect: errors can propagate into captions, and the model may fail to split adjacent events with fine-grained semantic differences.The limitation is especially apparent for temporally close, visually similar events.