PaperlayerView exact revision on arXiv

Source-linked AI summary

Progress-Aware Video Frame Captioning

Zihui Xue, Joungbin An, Xitong Yang, Kristen Grauman

arXiv:2412.02071v2cs.CV

TL;DR

Image captions describe isolated frames and video captions summarize whole clips, leaving action progression across individual frames unaddressed. The paper introduces ProgressCaptioner with the FrameCap dataset and FrameCapEval benchmark. It reports improved caption quality over leading open-source VLMs and practical applications including keyframe selection and video understanding.

  • Problem

    Image captioners treat frames in isolation and video captioning summarizes clips, leaving a need for captions that describe frame-specific action progression.

    Source: §1 · pp. 1–2

  • Method

    ProgressCaptioner interleaves pseudo labeling with frame-pair and full-sequence learning stages, creating FrameCap, while FrameCapEval benchmarks frame-wise caption quality.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    ProgressCaptioner improves caption quality 1.8× to 2.7× over leading open-source VLMs and achieves the highest user-study selection rate, surpassing GPT-4o [2] and Gemini-1.5-Pro [53].

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper demonstrates applications in keyframe selection and enhanced video understanding using temporally fine-grained captions.

    Source: §5 · p. 8, §2.3 · p. 21

    The proprietary-model comparison is not apples-to-apples because those models are much larger and trained with more extensive data.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility.

1. Introduction

Image captioning describes frames in isolation, while video captioning summarizes clips globally; progress-aware frame captioning instead describes each frame while tracking action progression. ProgressCaptioner, FrameCap, and FrameCapEval address this task, for which existing VLMs can miss frame differences or hallucinate progression.

1. Introduction

  • Progress-aware captions could support video understanding, retrieval, and generation, as well as applications such as step-by-step guidance for daily tasks.The paper gives keyframe selection and describing object-state changes during how-to videos as practical applications.Source: §1 · pp. 1–2
  • The paper introduces progress-aware frame captioning to describe each frame while detailing how an action progresses over time.Its fine-grained descriptions convey action stages or procedural steps throughout the sequence.Source: §1 · pp. 1–2
  • Existing VLMs can produce captions that fail to distinguish subtle frame differences or suggest progression absent from the visual frames.The paper terms the latter error “temporal hallucination.”Source: §1 · p. 2, Figure 3 · p. 2
  • ProgressCaptioner interleaves pseudo labeling with two learning stages, and the process creates FrameCap, a dataset of action videos with filtered frame-wise captions.Stage I develops frame-pair captioning, and Stage II extends it to full sequences.Source: §1 · p. 2
  • ProgressCaptioner improves caption quality 1.8× to 2.7× over leading open-source VLMs and achieves the highest user-study selection rate, surpassing GPT-4o [2] and Gemini-1.5-Pro [53].FrameCapEval uses videos from four public action datasets to assess caption quality and benchmark models.Source: §1 · p. 2

2. Related Work

Related work covers image-difference and video captioning, temporal-perception assessment, and neighboring video-description tasks. The paper distinguishes its focus on frame-level action progression from approaches that use static image pairs or produce coarse event-level descriptions.

Image Captioning; Video Captioning

  • Image-difference captioning addresses static image pairs or sets and coarse differences, leaving temporal action progression unexplored.Traditional video captioning instead produces a single description for a clip, while newer work expands duration, detail, uniqueness, temporal narrative, or summarization.Source: §2 · pp. 2–3

Related Tasks; Vision Language Models

  • Storytelling, dense captioning, audio description, and video-paragraph captioning still describe events coarsely rather than breaking actions into frame-level details.VLM advances improve captioning, but hallucination remains a recognized issue.Source: §2 · p. 3

Temporal Perception

  • Temporal-perception benchmarks assess overall summaries or general questions, while OSCaR is limited to three frames and object-state-change videos.The paper describes its approach as using scalable data collection and automatic evaluation beyond OSC videos, with focus on how frames evolve within one event.Source: §2 · p. 3

3. Approach

Progress-aware frame captioning aims to describe each frame accurately while capturing action progression across time. ProgressCaptioner develops this capability through frame-pair training, automatic caption evaluation, and a second stage that extends labeling to full sequences.

3.1. Progress-aware Video Frame Captioning

  • FrameCap supplies frame sequences with corresponding captions, using leading VLMs to create pseudo caption sequences from video sources.The cited sources include HowToChange and COIN.Source: §3.1 · pp. 3–4
  • Frame-pair prompting balances temporal context: full-sequence inputs can produce brief, misaligned captions, while isolated frames lose essential context.A tennis-serve frame, for example, was mistaken for a forehand follow-through when captioned in isolation.Source: §3.1 · p. 4

3.2. ProgressCaptioner

  • ProgressCaptioner starts with adjacent-frame caption pairs, using progression detection and caption matching to assess temporal change and frame-caption alignment.Progression detection uses majority voting across multiple LLM assessments; caption matching tests whether a VLM assigns each caption to its correct frame.Source: §3.2 · p. 5
  • Stage I fine-tunes LLAVA-OV-7B with accepted frame-caption pairs, then uses automatically constructed preference pairs for DPO to address hallucination.Progression detection and caption matching provide the positive and negative examples, avoiding manual preference labeling.Source: §3.2 · p. 5
  • Stage II extends pseudo-labeling to frame sequences with a two-frame sliding window, then combines data from both stages for SFT and DPO.The final model accepts inputs from two to T frames, allowing users to vary temporal context.Source: §3.2 · p. 5

4. Experiments

The experiments benchmark frame-wise caption quality and test whether progress-aware captions support keyframe selection and video understanding. Across these evaluations, ProgressCaptioner shows strong caption quality and downstream performance.

4.1. Benchmarking Video Frame Captioning

  • ProgressCaptioner leads FrameCapEval across in-domain and unseen datasets, outperforming similarly sized open-source VLMs and even much larger proprietary models.Table 1 reports caption-matching and progression-detection accuracy; the results support generalization from HowToChange and COIN to Penn Action and Kinetics.Source: Table 1 · p. 6, §4.1 · p. 7
  • On three action sequences, baseline captions sometimes suggest unsupported progression, while ProgressCaptioner describes frame-specific action progress precisely.The examples include the bowling sequence, where baseline captions incorrectly describe progression in frame 2.Source: Figure 6 · p. 6, §4.1 · p. 7
  • ProgressCaptioner receives a 31.6% average top-1 caption selection rate, 2× to 3.6× the rate of comparably sized best literature models.It also leads all models on top-1 preference; for top-2, GPT-4o scores 50.3% and ProgressCaptioner 47.3%.Source: Figure 7 · p. 6, §4.1 · p. 7

4.2. Applications of Video Frame Captioning

  • Progress-aware captions support selecting distinct keyframes that represent different action stages, including welding and grapefruit squeezing.The selection uses captions as an intermediate representation and an LLM to identify keyframes.Source: §4.2 · p. 7, Figure 8 · p. 7
  • Replacing uniform samples with four caption-selected frames raises top-1 accuracy by +1.7% for both SlowFast and X3D on the Kinetics Temporal subset.The selection uses ProgressCaptioner captions and GPT-4o to choose four frames from the 10-frame clips.Source: §4.2 · p. 8, Figure 9 · p. 8
  • ProgressCaptioner beats LLAVA-OV on zero-shot frame-wise classification and improves NExT-QA (ATP-Hard) results by +3.4% over VideoAgent.On the temporal subset, it gains +4.7% over a comparable LLAVA-OV setup.Source: §4.2 · p. 8, Figure 10 · p. 8

5. Conclusion

The conclusion introduces progress-aware frame-level captioning and presents ProgressCaptioner as an approach to temporal precision, with keyframe selection and video understanding as demonstrated applications.

5. Conclusion

  • ProgressCaptioner improves temporal precision and frame alignment, with demonstrated applications in keyframe selection and video understanding.The conclusion presents progress-aware video frame captioning as a new standard for temporal precision.Source: §5 · p. 8

1. Dataset

The dataset section describes FrameCap training-data preparation and the construction of FrameCapEval to assess temporally fine-grained frame captions.

1.1. FrameCap Training Data

  • FrameCap uses VLM-generated pseudo-labels filtered by progression detection and caption matching to prepare frame-pair and multi-frame training data.The caption-matching evaluator is VILA; its precision on 90 human-answered questions is 0.75, the highest among open-source VLMs, while Gemini-1.5-Pro scores 0.89.Source: §1.1 · p. 13
  • Stage I collects 240K frame-caption pairs for SFT and 21K preference pairs for DPO, while Stage II adds 34K multi-frame SFT sequences and 26K DPO sequences.Training videos come from HowToChange and COIN, with frames sampled at 1 FPS; multi-frame sequences contain 3–6 frames.Source: §1.1 · p. 13

1.2. FrameCapEval Benchmark

  • FrameCapEval uses balanced videos from four action datasets, follows their validation or test splits, and manually removes sequences lacking fine-grained progression.Frames are sampled at 1 FPS and grouped into sequences of 3–6 using CLIP-feature clustering; selected sequences include cases with and without progression.Source: §1.2 · pp. 13–14, Table 4 · p. 14
  • The authors plan to publicly release FrameCap and FrameCapEval as resources for frame-level temporal descriptions and VLM training.The stated aim is to help advance temporal precision in video understanding.Source: §1.2 · p. 14

2. Experiments

The experiments define automatic and human evaluations and prompts for assessing frame-wise caption quality and action progression. Results include qualitative caption comparisons, ablations, and a keyframe-selection application.

2.1. Experimental setup

  • Progression detection uses balanced accuracy for binary labels; caption matching uses sequence-level accuracy on manually validated T-frame action-progression sequences, while 15 English-fluent graduate students’ study reports average selection rates per model.The study used 85 randomly sampled frame sequences totaling 364 frames.Source: §2.1 · pp. 14–15

2.2. Prompt used

  • The prompts distinguish any visual change for training from action progression for evaluation, while generated captions must be frame-specific and describe action progression.The caption-generation prompt was selected through preliminary experiments, manually reviewed, and used for both pseudo-labeling and VLM evaluation.Source: §2.2 · pp. 15–16

2.3. Results

  • Stage-II sequence training, adding COIN to HowToChange training data, and DPO each improve performance in the HowToChange ablation.The passage reports lower progression detection for Stage I than Stage II, a particularly large caption-matching gain from adding COIN, and performance declines without DPO.Source: §2.3 · p. 16, Table 5 · p. 17
  • ProgressCaptioner consistently produces more temporally precise, progress-aware captions than the four leading VLM baselines in the qualitative comparisons.The comparisons mark baseline inaccuracies in red and progression-aware caption details in blue.Source: §2.3 · p. 17, Figure 14 · p. 17, Figure 15 · p. 18
  • ProgressCaptioner selects distinct action stages where V2Xum includes duplicate keyframes or misses subtle stages.The examples include three stages of slicing oranges and all four stages of cutting a sausage; the selected set size adapts to sequence content.Source: §2.3 · p. 21, Figure 18 · p. 21
  • Caption-matching and progression-detection evaluation trends remain consistent across different LLM and VLM evaluators.Figure 19 presents results with Gemini and GPT as VLM evaluators and Llama and GPT as LLM evaluators.Source: §2.3 · p. 21, Figure 19 · p. 21
Loading 2412.02071v2…