PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Video-Moment Retrieval and Step-Captioning

Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oğuz, Yasher Mehdad, Mohit Bansal

arXiv:2303.16406v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Prior video retrieval, moment retrieval, summarization, and captioning work generally treated these related tasks separately, despite their shared role in searching video corpora. HIREST combines them in a dataset and benchmark with task-specific and joint baselines; fine-tuning improves performance across all four tasks, while substantial room for improvement remains.

  • Problem

    Prior work generally addressed video retrieval, moment retrieval, summarization, and captioning separately, rather than in a joint setup for searching video corpora and generating summaries.

    Source: §1 · p. 1

  • Method

    HIREST pairs instructional-video queries with relevant moments and step annotations, and benchmarks video retrieval, moment retrieval, moment segmentation, and step captioning with task-specific and joint baselines.

    Source: §1 · pp. 1–2

  • Results

    Fine-tuning models on HIREST improves performance across all four benchmark tasks, though substantial room for improvement remains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The manually annotated step captions can serve as a source for training and testing step-by-step reasoning in large multimodal language models.

    Source: §1 · p. 2

    Baseline performance remains below upper-bound accuracies, leaving substantial room for future improvement.

    Source: §5 · p. 7

Abstract

from arXiv · show

There is growing interest in searching for information from large video corpora. Prior works have studied relevant tasks, such as text-based video retrieval, moment retrieval, video summarization, and video captioning in isolation, without an end-to-end setup that can jointly search from video corpora and generate summaries. Such an end-to-end setup would allow for many interesting applications, e.g., a text-based search that finds a relevant video from a video corpus, extracts the most relevant moment from that video, and segments the moment into important steps with captions. To address this, we present the HiREST (HIerarchical REtrieval and STep-captioning) dataset and propose a new benchmark that covers hierarchical information retrieval and visual/textual stepwise summarization from an instructional video corpus. HiREST consists of 3.4K text-video pairs from an instructional video dataset, where 1.1K videos have annotations of moment spans relevant to text query and breakdown of each moment into key instruction steps with caption and timestamps (totaling 8.6K step captions). Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks. In moment segmentation, models break down a video moment into instruction steps and identify start-end boundaries. In step captioning, models generate a textual summary for each step. We also present starting point task-specific and end-to-end joint baseline models for our new benchmark. While the baseline models show some promising results, there still exists large room for future improvement by the community. Project website: https://hirest-cvpr2023.github.io

1. Introduction

The paper introduces HIREST as a hierarchical benchmark for text-based video information retrieval and stepwise summarization, combining four tasks that prior work generally addressed separately. It provides dataset annotations and baseline models, which improve with fine-tuning but leave room for further progress.

1. Introduction

  • HIREST contains 3.4K text-video pairs, with moment and step annotations for 1.1K videos totaling 8.6K steps.Each annotated step has a timestamp and a manually curated English caption.Source: §1 · p. 1
  • The benchmark links video retrieval, moment retrieval, moment segmentation, and step captioning in one hierarchy.The tasks move from finding a relevant video to locating its moment, segmenting that moment into steps, and generating captions for them.Source: §1 · p. 2, Figure 1 · p. 2
  • Fine-tuning models on HIREST improves performance across all four tasks, although substantial room for improvement remains.The paper also provides a joint baseline for moment retrieval, segmentation, and step captioning.Source: §1 · p. 2

2. Related Work

Prior work developed video retrieval, moment retrieval, query-focused summarization, and captioning largely as separate tasks. HIREST combines these functions and contributes diverse, manually written step annotations for instructional videos.

2.1. Text-based Information Retrieval from Video

  • The paper combines video retrieval, moment retrieval, query-focused video summarization, and stepwise captioning so users can search at multiple levels.Earlier systems generally specialized in a single task.Source: §2.1 · pp. 2–3

2.2. Instructional Video Datasets

  • HIREST provides manually written step captions for diverse instructional videos, unlike datasets limited to one domain or predefined steps.HowTo100M provides diverse instructional videos but lacks specific step-by-step annotations; COIN and CrossTask use predefined steps.Source: §2.2 · p. 3

3. HIREST: Hierarchical Retrieval and Step-Captioning Dataset

HIREST is a dataset of diverse instructional videos with query-relevant moments and human-written step captions. Its two-stage annotation process supports a hierarchy of video and moment retrieval, step segmentation, and step captioning.

3. HIREST: Hierarchical Retrieval and Step-Captioning Dataset

  • HIREST is a video dataset with 3.4K text-video pairs, 1.8K moments, and 8.6K human-written step-caption annotations, covering diverse open-domain instructional videos with many annotations per video.Its step captions are manually written by human annotators, cover diverse domains, and use a large vocabulary.Source: §3 · p. 3, Table 1 · p. 4

3.1. Dataset Collection

  • Crowdworkers first verify query-video relevance and trim relevant moments, then segment moments into steps and write captions of up to seven words.A video is considered clippable when its relevant moment is less than 75% of the original video length.Source: §3.1 · p. 3

3.2. Dataset Analysis

  • Of 3.4K videos, 1.8K have clippable moments and 1.1K are annotated with steps, averaging 7.6 steps each and totaling 8.6K.The captions average 4.42 words and include 633 unique starting verbs and 3,382 unique words; the most frequent categories are Hobbies and Crafts, Food and Entertaining, and Home and Garden.Source: §3.2 · p. 3

3.3. Hierarchical Tasks Enabled by HIREST

  • The benchmark defines video retrieval among relevant and distractor videos, moment retrieval, step-boundary segmentation, and instructional caption generation.The retrieval test set includes 1.4K query-paired videos and 2.8K distractor videos.Source: §3.3 · pp. 4–5

4. Experiments

The experiments evaluate task-specific baselines and a joint model across four hierarchical video tasks, using task-appropriate retrieval, span, and caption metrics.

4.1. Task-specific Models

  • Video retrieval baselines compare text-to-image and text-to-video models, while moment retrieval tests CLIP boundary heuristics and BMT event proposals.CLIP and EVA-CLIP average frame embeddings; Frozen-in-Time uses 4 frames and MIL-NCE uses 32. The moment-retrieval heuristics either threshold similarity drops or select 8 frames on either side of the top-scoring frame.Source: §4.1 · p. 5
  • For segmentation, the task-specific baselines use adjacent-frame SSIM changes or BMT proposals; captioning uses BMT or SwinBERT on step-bounded video features.The captioning models receive ground-truth step boundaries; SwinBERT uses a YouCook2 checkpoint and 32 frames per step.Source: §4.1 · p. 5

4.2. Joint Model

  • The joint model adapts pretrained visual, speech, and text encoders with a multimodal encoder and text decoder, then trains across tasks in a round-robin multitask setup.It predicts moment boundaries in parallel and step boundaries autoregressively with masking; its decoder generates a caption independently for each step.Source: §4.2 · pp. 5–6

4.3. Metrics

  • Evaluation uses Recall@k for video retrieval, IoU-thresholded recall or precision for span tasks, and text-overlap, embedding, and entailment metrics for step captions.Caption metrics include CIDEr, METEOR, SPICE, BERTScore, CLIPScore, and an entailment score.Source: §4.3 · pp. 6–7

5. Results and Discussions

Results show promising baselines but a remaining gap to upper-bound accuracies. Performance varies across retrieval, segmentation, and captioning, and audio removal hurts most tasks.

5. Results and Discussions

  • The baseline models remain below upper-bound accuracies, leaving substantial room for improvement.The authors describe the initial results as promising while noting the remaining performance gap.Source: §5 · p. 7

Video Retrieval

  • EVA-CLIP outperforms the other video-retrieval models by a large margin, and its features are used for the three downstream tasks.Increasing input frames raises recall until 20 frames; finetuning CLIP on HIREST makes little difference.Source: §5 · p. 7

Moment Retrieval

  • The joint model exceeds finetuned BMT on moment-retrieval R@0.5, while finetuned BMT scores higher on R@0.7.Among zero-shot CLIP heuristics, the 8-frames-left/right method outperforms the similarity-drop method for both CLIP and EVA-CLIP.Source: §5 · p. 7

Moment Segmentation

  • The joint model performs better than BMT on both moment-segmentation recall and precision.Zero-shot SSIM can outperform zero-shot BMT, while finetuning improves BMT on recall metrics.Source: §5 · p. 7

Step Captioning

  • The joint model achieves similar CIDEr and sentence-level embedding scores to SwinBERT but significantly higher entailment; finetuning improves N-gram and entailment scores, not BERTScore or CLIPScore.Zero-shot BMT and SwinBERT perform poorly on N-gram and entailment metrics, with captions longer than HIREST step captions.Source: §5 · pp. 7–8

Audio Ablation

  • Removing audio significantly lowers both models’ moment-retrieval and moment-segmentation scores, and lowers the joint model’s step-captioning score.BMT step-captioning scores show little difference without audio.Source: §5 · p. 8

Visualization of Hierarchical Model Pipelining

  • In the butter-biscuit example, predicted boundaries and captions correspond semantically with annotations, but retrieval misses the final dough-cutting and cooking portion.The authors note that an instructor standing up makes those frames visually different from the preceding batter-making process.Source: §5 · p. 8

6. Conclusion

The conclusion presents HIREST as a benchmark for hierarchical retrieval and summarization in instructional videos, with human-written step annotations and baseline models.

6. Conclusion

  • HIREST covers video retrieval, moment retrieval, moment segmentation, and step captioning, including diverse, human-written instruction steps with timestamps.The authors present task-specific and end-to-end joint baselines as starting points for the benchmark.Source: §6 · p. 8
  • The authors hope the dataset and benchmark will foster multimodal work on holistic video retrieval, summarization, and step-by-step reasoning.Source: §6 · p. 8
  • The paper reports dataset analysis and experiments with baseline models for each task as part of the benchmark contribution.Source: §6 · p. 8

A. Data Annotation Details

Annotation uses qualified crowdworkers and a staged interface for matching videos to queries and trimming relevant moments, with separate qualifications for each stage.

A. Data Annotation Details

  • Workers needed a 95% approval rate, at least 1,000 completed tasks, and a qualification test; 72 workers passed both stage tests.Workers were required to come from an English-speaking country because the queries and collected captions were in English.Source: A. Data Annotation Details · p. 10
  • Base pay was $0.20 for stage 1 and $0.45 for stage 2, with bonuses and task-based pay increases for high-quality work.Typical earnings could reach $0.27 per stage-1 task and $0.81 per stage-2 task.Source: A. Data Annotation Details · pp. 10–11
  • Stage 1 workers judge whether a video answers a query and, if so, trim it to the relevant portion using a sliding bar.Workers had to pass a stage-specific qualification test that included judging query relevance and identifying a relevant moment.Source: A. Data Annotation Details · p. 10, Figure 7 · p. 11

B. CLIP-based Moment Retrieval Method

The CLIP-based moment retrieval heuristics locate a moment around the frame with the highest text-frame cosine similarity, using either a similarity threshold or a fixed frame window.

B. CLIP-based Moment Retrieval Method

  • The method sets moment boundaries around the highest-scoring text-frame pair using either a similarity-drop threshold or a fixed window of eight frames on each side.The fixed window totals 17 frames, including the highest-scoring frame.Source: B. CLIP-based Moment Retrieval Method · p. 11

C. Model Performance Analysis in Video Categories and Duration

The analysis reports retrieval, segmentation, and captioning performance across video categories and duration groups. Results vary by category and duration, with several stated advantages depending on the model and evaluation group.

Evaluation setup

  • The test-split analysis groups results by video category and video or moment duration; its category distribution counts prompts and videos separately because some prompts pair with multiple videos.Table 7 reports the prompt and video category distributions.Source: C. Model Performance Analysis in Video… · p. 11, Table 7 · p. 12

Video retrieval

Moment retrieval

Moment segmentation

Step captioning

  • SwinBERT performs best in Food and Entertaining, likely because it was pretrained on YouCook2.The category-specific step-captioning results are reported on the HIREST test split.Source: C. Model Performance Analysis in Video… · p. 13
  • Captioning metrics show no significant differences across step durations for N-gram and sentence-level metrics, while entailment improves with longer steps for finetuned SwinBERT and slightly worsens for the joint model.The reported metrics include CIDEr, BERTScore, and CLIPScore.Source: C. Model Performance Analysis in Video… · p. 13

D. Evaluation Details

Moment segmentation evaluation postprocesses BMT's proposed segments against the ground-truth moment before assessing the resulting steps.

D. Evaluation Details

  • For segmentation evaluation, segments outside the ground-truth moment are removed, overlapping segments are suppressed with NMS, and gaps are counted as separate steps.BMT can generate up to 100 proposed segments, including segments outside the ground-truth moment, overlaps, and gaps.Source: D. Evaluation Details · p. 13
  • The step-captioning tables report results by video category and by step duration on the HIREST test split.Both tables identify finetuning as FT.Source: Table 13 · p. 14, Table 14 · p. 15
Loading 2303.16406v1…