PaperlayerView exact revision on arXiv

Source-linked AI summary

AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq-Neng Hwang, Saining Xie, Christopher D. Manning

arXiv:2410.03051v4cs.CV

TL;DR

Video detailed captioning lacks sufficiently detailed benchmarks and evaluation methods. AuroraCap uses a large multimodal model with token merging, while VDC and VDCscore provide a detailed benchmark and divide-and-conquer evaluation. The paper reports competitive performance across image, video captioning, and video question answering, with token merging reducing overhead with little performance loss.

  • Problem

    Video detailed captioning is limited by insufficient detailed caption data and evaluation methods, while existing benchmarks often provide only short descriptions.

  • Method

    AuroraCap adapts a large multimodal model to video with token merging, and the paper introduces the VDC benchmark and VDCscore for detailed-caption evaluation.

  • Results

    AuroraCap achieves competitive results across image captioning, video captioning, and video question answering, while token merging reduces computational overhead without compromising performance.

  • Takeaways & Limitations

    VDC provides a detailed video-captioning benchmark, and VDCscore uses a divide-and-conquer strategy to evaluate long captions through multiple short question-answer pairs.

  • Takeaways & Limitations

    AuroraCap currently performs poorly on OCR because its training data contains limited OCR-related samples.

Abstract

from arXiv · show

Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.

1. Introduction

Video detailed captioning requires comprehensive, temporally coherent descriptions, but progress is constrained by limited detailed data and inadequate evaluation. AuroraCap uses a simple multimodal design with token merging, alongside the VDC benchmark and VDCscore metric, to improve efficient captioning and assessment.

  • Method: AuroraCap adapts a LLaVA-like multimodal model to video without additional temporal parameters, using high-quality video-text instruction data for fine-tuning.
  • Method: Token merging reduces visual-token overhead with marginal performance loss, supporting AuroraCap’s efficient processing of image and video inputs.The strategy combines similar tokens and reduces the number passed through the model.
  • Benchmark: VDC addresses shortcomings of existing benchmarks by providing over one thousand high-quality, structured video-caption pairs with substantially longer and more detailed descriptions.The benchmark uses dense frames, multiple caption aspects, and manual quality inspection.
  • Evaluation: VDCscore evaluates detailed captions by addressing the limitations of word-frequency metrics and direct LLM judgments for semantically rich descriptions.Its divide-and-conquer design is introduced to improve evaluation of detailed captions.
  • Results: AuroraCap achieves competitive results across image captioning, video captioning, and video question answering benchmarks, including strong performance in image and video evaluation settings.The paper evaluates image captioning with CIDEr, BLEU, METEOR, and ROUGE-L, and video question answering across multiple datasets.

3. VDC: A Video Detailed Captioning Benchmark

VDC is designed as a detailed video-captioning benchmark with diverse, structured, human-reviewed descriptions and an LLM-assisted metric for evaluating their quality. Experiments compare AuroraCap with existing models and metrics, while token-merging studies examine efficiency and performance.

  • 3.1.2. Structured Detailed Captions Construction Pipeline: VDC combines diverse video sources with structured captions covering multiple perspectives, extending caption length and detail beyond prior benchmarks.The construction pipeline uses hierarchical prompting, dense frames, and manual review to correct hallucinations and omitted visual elements.
  • 3.1.3. Comparison on Numerical Statistics: Over 87% of VDC videos span 10K–12K frames, while detailed structured captions average over 500 words.Only 13% of videos are shorter than 10 seconds, and 1% extend beyond 60 seconds.
  • 3.2.1. VDCscore: Evaluating Detailed Captions with LLMs: VDCscore evaluates long-caption similarity by decomposing ground-truth captions into question-answer pairs, generating answers from predictions, and averaging correctness and quality scores.The pipeline uses Llama-3.1-8B and constrains generated answers to concise sentences or phrases to reduce length-related bias.
  • 3.2.2. Benchmarking Video Detailed Captioning: AuroraCap achieves superior video detailed-captioning performance while using significantly fewer visual tokens than other models.Table 6 compares AuroraCap with LLM-based baselines, while Table 7 reports VDCscore across structured-caption categories.
  • 3.2.2. Benchmarking Video Detailed Captioning: VDCscore correlates better with human Elo rankings than VDD and ROUGE, supporting its use for detailed-caption evaluation.The benchmark includes comparisons with rule-based and model-based metrics.
  • 3.2.2. Benchmarking Video Detailed Captioning: Most models retain more than 80% of peak performance when only 0.2 of visual tokens are kept, although AuroraCap’s temporal setting can make the optimal ratio intermediate.The ablation varies the token-merging ratio and measures performance across the resulting curve.

4. Conclusion

The paper introduces AuroraCap, VDC, and VDCscore for efficient video detailed captioning and evaluation. Across video and image captioning benchmarks, AuroraCap achieves competitive results, while the authors identify remaining trade-offs and captioning weaknesses.

  • 4. Conclusion: AuroraCap uses token merging to reduce computational overhead without compromising performance, while VDC and VDCscore support detailed-caption benchmarking and evaluation.The paper reports competitive results across video and image captioning benchmarks and validates token merging through ablations.
  • 4. Conclusion: The current model performs poorly in the trade-off between performance and input-token scale, with further room for improvement in camera handling and detailed captioning.These limitations are stated as remaining areas for improvement in the conclusion.

Ethics Statement

The paper uses public datasets and acknowledges privacy, bias, and misuse risks in automatic caption generation. It also documents implementation and evaluation details to support reproducibility.

  • Publicly available datasets are used, with stated compliance with privacy regulations.
  • The authors acknowledge potential gender- and race-related biases in automatic caption generation and report efforts to evaluate and minimize them.
  • The paper recognizes misuse risks, including the generation of misleading captions, while committing to further improvements.
  • Architecture, training procedures, hyperparameters, benchmark settings, and prompt templates are documented in supplementary sections and linked repositories.

Supplementary Material

The supplementary material reviews related video captioning and multimodal-model work, reports long-video results, and provides additional token-merging analysis. It also examines how token merging affects longer video sequences and n-gram evaluation against human rankings.

  • Large Multimodal Models for Video: LLaVA-like models can be adapted to video without additional temporal-modeling parameters, although video sequences create computational and length-related overhead.The review describes video as sequences of frames and surveys parameterized approaches for reducing visual-token costs.
  • Related Work: Existing video benchmarks include question-answering datasets and short-caption datasets, while no benchmark specifically evaluates detailed video captioning.The supplementary review highlights MovieChat-1K, CinePile, and other benchmark designs before positioning VDC as a missing evaluation resource.
  • S2. More Results on Long-form Video Benchmarks: AuroraCap achieves comparable performance on MovieChat-1K and Egoschema without long-video training or specialized design.Its token-merging curve on Egoschema demonstrates stability and effectiveness for longer video sequences.
  • S5. Token Merging Visualization: Token merging maintains stable Egoschema performance across visual-token kept ratios, supporting its use for longer video sequences.The supplementary material visualizes the token-merging ablation curve and identifies best performance at a kept ratio of 0.6 to 0.7.
  • S3. N-gram Based Evaluation Results: N-gram evaluation scores are compared with human Elo rankings on VDC.The supplementary figure presents the correlation analysis without reporting correlation values in the supplied passage.

S3. N-gram Based Evaluation Results

This section compares n-gram evaluation with VDC examples and visualizes token merging across images. The examples contrast short benchmark captions with VDC’s substantially longer detailed descriptions, while the token-merging formula specifies how many tokens are merged per transformer layer.

  • N-gram Evaluation: VDC’s n-gram evaluation is assessed through correlations with human Elo rankings, with a ROUGE-based answer-matching table also provided.The supplied passages identify the comparison and table scope but do not provide correlation or ROUGE values.
  • Benchmark Examples: VDC uses a 618-word detailed caption, compared with 4-word MSR-VTT and 13-word VATEX examples.The supplied examples illustrate the caption-length distinction underlying VDC’s detailed-captioning focus.
  • Token Merging Visualization: The number of tokens merged in each ViT transformer layer is computed from image dimensions, patch size, layer count, and the kept-token ratio.The merging is evenly distributed across transformer layers.
  • Token Merging Visualization: Token merging reduces image representations from 490 tokens to 6 tokens in the visualization examples.The examples cover COCO, VG, SA-1B, and OCR-VQA, with intermediate representations of 154 and 18 tokens.

S6. Ablation Studies

The ablations show that token merging substantially reduces visual tokens while preserving much of the performance, whereas slowfast inference offers limited or negative gains in some settings. The section also compares efficiency and training strategies.

  • Token merging: Token merging reduces visual-token counts while maintaining minimal performance drops across image captioning, visual question answering, video captioning, and video question answering.Figures S9 and S10 report the cross-task ablations and mark retention thresholds associated with 90% and 80% performance.
  • Efficiency: Inference efficiency is evaluated using time per video question-answering pair across settings with or without token merging and SGLang.The supplied passages state that reducing visual tokens and using SGLang produce excellent inference times, without giving the plotted values.
  • Slowfast inference: Slowfast inference does not consistently improve performance and can worsen degradation as visual-token retention decreases, particularly for video captioning.The corresponding ablation table reports marginal gains or drops with additional computing cost.
  • Training strategy: The ablations compare four language-stage training strategies while keeping datasets and hyperparameters consistent, with Setting C selected as the final strategy.The settings vary end-to-end training, token retention, first-frame treatment, and long-video data usage.

S7. Detailed Training Settings

The detailed training settings specify AuroraCap’s backbone initialization, image and video preprocessing, token-merging schedule, and training data composition. The supplementary tables document hyperparameters and datasets for the pretraining, vision, and language stages.

  • Backbone and Initialization: AuroraCap-7B initializes with CLIP ViT-H and Vicuna-1.5-7B backbones.The supplied implementation passage identifies the specific model families used for visual and language initialization.
  • Data Preprocessing: Video frames are sampled uniformly at 2 FPS, resized to a short side of 378, and processed with bilinear position-embedding interpolation rather than center cropping.Token merging uses a constant schedule for each transformer block.
  • Training Configuration: The supplementary material provides dedicated tables for AuroraCap’s hyperparameters and stage-specific training datasets.Tables S4–S7 cover hyperparameters and datasets for pretraining, vision, and language stages.
  • Training Data: Training data include image, video-instruction, and language-only datasets, totaling 15.0M examples with video captioning and instruction datasets duplicated twice.The dataset summaries are documented across the pretraining, vision, and language-stage tables.

S8. Evaluation Benchmarks and Settings

The evaluation uses structured prompts and fixed decoding settings across benchmarks, while additional analysis examines CLIP and VDCscore against human ratings. VDCscore correlates more strongly with human judgments as the number of QA pairs increases.

  • Evaluation prompts separately target camera motion, background, main-object attributes and movements, and detailed video description.The settings use a maximum of 1,024 tokens for VDC evaluation.
  • All benchmark evaluations use temperature 0, top-p 0, and one beam according to the evaluation-settings summary.
  • Figure S15 compares CLIP scores and VDCscore variants using different QA-pair counts against human Elo rankings.
  • VDCscore reaches a Pearson correlation above 0.86 with human ratings when computed using 20 QA pairs.More QA pairs produce more robust evaluation results, whereas CLIP correlates only 0.39 with human ratings.

S10. Limitations

The appendix reports benchmark-specific evaluation details and notes limitations in AuroraCap’s current OCR capability. It also documents prompt-based caption and QA-pair generation procedures used in the evaluation pipeline.

  • AuroraCap performs poorly on OCR-related samples because its training data contains limited OCR examples.The limitation is reported in comparison with LLaVA models.
  • The VQA comparison uses the same LLM for all models because VQA performance depends heavily on the LLM.
  • Structured captions are generated from multiple aspects, including short, background, main-object, camera, and detailed descriptions.The prompt asks the detailed caption to expand the reference caption without introducing unsupported entities or relationships.
  • VDCscore generation uses Llama-3.1-8B to create 20 short, open-ended question-answer pairs from detailed descriptions.The generated answers are constrained to concise sentences or phrases for subsequent evaluation.
  • The QA-pair prompt covers camera style, transitions, lighting, atmosphere, and camera movement through open-ended questions.

S13. More Statistics Information of VDC

VDC includes detailed structured-caption statistics and QA-pair distributions, and the evaluation compares model outputs through blinded pairwise human preferences converted into Elo ratings. The proposed VDC metric is reported as closest to human evaluations.

  • Figures S16–S18 summarize caption word diversity, QA-pair proportions, and question-type distributions in VDC.
  • VDC generates 96,902 question-answer pairs, averaging 18.87 pairs per detailed caption.
  • The structured captions distribute QA pairs similarly across sections, while VDCscore uses open-ended questions across environment, scene, character, and object content.
  • Human Elo ratings come from blinded pairwise comparisons between captions generated by two models for the same video.A frontend randomly selects the caption pair, records the evaluator’s preference, and calculates Elo values.
  • The proposed VDC metric has the closest Pearson correlation to human Elo evaluations among the compared metrics.

S16. Case Study

The case study examines AuroraCap across varied videos and reports detailed captions addressing camera motion, backgrounds, and main objects with less hallucination. Supplementary examples illustrate short, detailed, background, camera, and main-object caption outputs.

  • AuroraCap provides detailed captions about camera motion, background, and main objects with less hallucination across varied videos.
  • The supplementary examples include short captions describing people at a table and a drummer in a recording studio.
  • Detailed-caption examples describe a woman on a beach and a man lying against a tree with appearance, setting, and posture details.
  • Background-caption examples characterize a race track through its surface, barriers, surrounding vegetation, distant hills, and cloudy weather.
  • Camera-caption examples describe steady or static viewpoints, framing, camera movement, and the resulting coverage of subjects and surroundings.
  • A main-object example follows a black car navigating a damaged street around obstacles and debris.

S17. Predicted Answer Extraction Prompt Template

The appendix specifies LLM-assisted extraction of answers from generated detailed captions and evaluation of those answers against correct responses.

  • S17. Predicted Answer Extraction Prompt Template: The extraction prompt instructs an LLM to answer questions using only the video or image description.It requires short, accurate answers without additional explanation.
  • S17. Predicted Answer Extraction Prompt Template: The extraction example converts a detailed scene description into the answer that the cyclist’s hands interact with bicycle handlebars.
  • S18. Correctness Evaluation Prompt Template: The evaluation prompt compares each predicted answer with its correct answer for meaningful semantic agreement.Synonyms and paraphrases are treated as valid matches.
  • S18. Correctness Evaluation Prompt Template: The evaluator returns both a binary correctness judgment and an integer score from 0 to 5.A score of 5 represents the highest meaningful match.
  • S18. Correctness Evaluation Prompt Template: The correctness example treats “Carpeted surface” as a prediction to compare against the correct answer “carpet.”
  • S18. Correctness Evaluation Prompt Template: The required evaluation output is a Python dictionary string with `pred` and `score` keys and no additional text.
Loading 2410.03051v4…