AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

2.1. Architecture

• AuroraCap adapts a LLaVA-like model to video without additional parameters for temporal modeling and uses frame-wise token merging to retain only 10%–20% of visual tokens with marginal performance drop.Similar tokens are gradually combined using bipartite soft matching, reducing input-token overhead and facilitating higher-resolution or longer video inputs.Source: §1 · p. 2, §2.1 · pp. 3–4

1. Introduction

• VDC provides over one thousand high-quality video-caption pairs with detailed descriptions of world knowledge, object attributes, camera movements, and event timing.The captions are generated with GPT-4o assistance and hierarchical prompts, use dense-frame extraction to preserve information and temporal consistency, and undergo manual quality inspection; the paper presents VDC as the first benchmark focused on detailed video captioning.Source: §1 · pp. 2–3
• VDCscore evaluates detailed captions by decomposing ground-truth captions into concise question-answer pairs and assessing answers generated from predicted captions.The paper motivates this divide-and-conquer metric by noting that conventional short-caption metrics and existing LLM-based evaluation struggle with detailed captions.Source: §1 · p. 3

2.2. Training Recipe

• AuroraCap uses three training stages: connector-only pretraining, vision-stage training with the ViT unfrozen and LLM frozen, then end-to-end language-stage training.Pretraining uses 1.3M image-caption pairs; the final stage mixes image and video captions and instructions, duplicates video-caption datasets, and removes video data for the image-based variant.Source: §2.2 · p. 4

2.3. Evaluation

• Image-captioning evaluation uses the image-based AuroraCap in zero-shot settings on Flickr, NoCaps, and COCO-Cap, whose single-sentence captions only partially reflect model performance.The paper evaluates with CIDEr, BLEU-4, BLEU-1, METEOR, and ROUGE-L; the rest of its reported performance refers to video-based AuroraCap.Source: §2.3 · p. 5
• Video-captioning evaluation compares AuroraCap on MSR-VTT and VATEX using CIDEr, BLEU-4, BLEU-1, METEOR, and ROUGE-L, although these benchmarks contain only one-sentence captions.The paper also notes that existing video benchmarks often use short descriptions, motivating VDC’s focus on detailed captions.Source: §1 · p. 2, §2.3 · p. 5
• AuroraCap achieves competitive video question-answering performance on ActivityNet-QA and iVQA, but more modest results on MSVD-QA and MSRVTT-QA.The authors suggest that prompting the model to generate a comprehensive video caption can produce the correct answer in some failures, possibly because training disrupted instruction-following.Source: §2.3 · pp. 5–6

3.1.1. Video Collection and Processing

• VDC balances videos from Panda-70M [20], Ego4D [40], Mixkit, Pixabay, and Pexels, using varied sources to reduce homogeneity.The videos are not in AuroraCap’s training data; the collection also uses clip splitting and dense frame extraction.Source: §3.1.1 · p. 6, Table 5 · p. 7

3.1.3. Comparison on Numerical Statistics

• Over 87% of VDC videos span 10K–12K frames, 13% last under 10 seconds, and detailed captions average over 500 words.Only 1% of videos extend beyond 60 seconds; Figure 1 shows video-duration statistics and Figure 2 shows structured-caption lengths.Source: Figure 1 · p. 7, Figure 2 · p. 7, §3.1.3 · p. 8

3.1.2. Structured Detailed Captions Construction Pipeline

• VDC’s construction pipeline creates short, main-object, background, camera, and detailed captions, then manually reviews them to correct hallucinations and add omitted visual elements.GPT-4o generates captions from dense video frames in two rounds: an initial structured-caption generation and a detailed-caption integration step guided to avoid new entities or relations. Existing benchmarks often provide only one less-structured caption [19] [61].Source: §3.1.2 · pp. 6–8

3.2.1. VDCscore: Evaluating Detailed Captions with LLMs

• VDCscore evaluates predicted against ground-truth detailed captions by decomposing long captions into short question-answer pairs and averaging pairwise scores.Llama-3.1-8B generates standardized ground-truth pairs, answers their questions from predictions, and scores answer correctness and quality; identical captions receive 100% accuracy. Direct LLM judging can struggle with detailed captions and assign disproportionately low scores.Source: §3.2.1 · pp. 8–10

3.2.2. Benchmarking Video Detailed Captioning

• AuroraCap achieves higher VDCscore than other models with fewer visual tokens, and VDCscore correlates better with human Elo judgments than VDD and ROUGE-L.Visual-token count represents efficiency because inference-time comparisons across architectures and deployment settings were considered unfair.Source: §3.2.2 · pp. 10, 12, Figure 4 · p. 11, Figure 5 · p. 12
• Most models retain over 80% of performance with only 0.2 of visual tokens kept, while AuroraCap’s optimum may occur at a middle token-retention ratio.The study examines token-merging ratios; AuroraCap’s spatial merging interacts with temporal features, adding complexity to identifying the optimal ratio.Source: Figure 6 · p. 12, §3.2.2 · pp. 12–13

4. Conclusion

• AuroraCap achieves competitive results across video and image captioning benchmarks, outperforming state-of-the-art models on some tasks.The paper also reports that ablation studies validate token merging and other aspects of the model.Source: §4 · p. 13
• Token merging reduces AuroraCap’s computational overhead without compromising performance.The conclusion notes that the model still performs poorly in the trade-off between performance and input-token scale.Source: §4 · p. 13
• The paper introduces VDC to evaluate comprehensive, coherent video descriptions and VDCscore, an LLM-assisted metric using a divide-and-conquer strategy.Source: §4 · p. 13
• The conclusion identifies camera handling and detailed captioning as areas where AuroraCap still needs improvement.Source: §4 · p. 13

Ethics Statement

• The authors say their publicly available datasets comply with privacy regulations and that they checked the training data for safeguards against misuse.Source: Ethics Statement · p. 14
• They recognize potential biases in automatic captions, particularly around gender, race, and other characteristics, and report measures to evaluate and minimize them.Source: Ethics Statement · p. 14
• The authors acknowledge the risk of generating misleading captions and commit to further improvements in mitigating bias.Source: Ethics Statement · p. 14

Reproducibility Statement

• Implementation details, benchmark settings, generation and evaluation prompts, Elo calculations, hardware requirements, and code, weights, benchmarks, and training data are documented or slated for release.Model architecture, training procedures, and hyperparameters are in Section S7; the material says code and model and data artifacts will be released with the paper.Source: Reproducibility Statement · p. 14

Supplementary Material

• The supplement covers related work, long-video results, token merging, ablations, training and evaluation settings, benchmark and metric statistics, prompts, Elo calculations, limitations, and case studies.These topics are listed across supplementary Sections S1–S18.Source: Supplementary Material · p. 29

S1. Related Works

• Most existing video-captioning benchmarks use short captions, whereas this work targets detailed descriptions of short clips without scene changes or camera switches.Video captioning also requires describing temporal content such as actions and camera or object movement.Source: S1. Related Works · p. 29
• Prior work finds vanilla LLaVA-like models can perform strongly on video [77] [136] [141], and argues against strong temporal merging when sampled frames have low similarity.The passage notes that frame-to-frame similarity is low at 1 FPS except in static scenes, and can be even lower with keyframes.Source: S1. Related Works · pp. 29–30
• Existing video benchmarks do not evaluate detailed captioning, and VDCscore uses ground-truth-derived question-answer pairs to evaluate caption accuracy.The passage distinguishes VDCscore from LLaVA-Hound-DPO [159], which uses a similar divide-and-conquer strategy for a preference dataset.Source: S1. Related Works · p. 30

S2. More Results on Long-form Video Benchmarks

• AuroraCap achieves comparable performance on MovieChat-1K [114] and Egoschema [88] without long-video training or special design.The reported comparisons cover both benchmarks.Source: Figure S1 · p. 30, Figure S2 · p. 30, S2. More Results on Long-form Video… · p. 30
• The token-merging curve on Egoschema is stable for longer sequences, with best performance at a visual token kept ratio of 0.6 to 0.7.The paper describes this curve as demonstrating the stability and effectiveness of token merging on longer video sequences.Source: S2. More Results on Long-form Video… · p. 30, Figure S3 · p. 31

S3. N-gram Based Evaluation Results

• The VDCscore variant evaluates final question-answer-prediction triplets with n-gram matching, with ROUGE answer-matching results in Table S1 and correlations shown in Figure S4.The supplied passage does not report the table values or correlation results.Source: S3. N-gram Based Evaluation Results · p. 31, Table S1 · p. 31

S4. Benchmark Comparison

• The benchmark comparison presents examples from MSR-VTT, VATEX, and VDC, with the VDC example shown as a detailed caption.The captions shown are “Teams are playing soccer” and a description of a woman demonstrating how to remove a pumpkin’s insides for MSR-VTT and VATEX, respectively.Source: S4. Benchmark Comparison · p. 32, Figure S6 · p. 32, Figure S7 · p. 32
• MSR-VTT [140] and VATEX [133] examples use 4- and 13-word captions, while the sampled VDC caption from Ego4D [40] is labeled 618 words.The VDC example describes moss-covered machinery, shifting camera views, and people handling components in a workspace.Source: S4. Benchmark Comparison · pp. 32–33, Figure S5 · p. 32

S5. Token Merging Visualization

• Token merging is distributed evenly across ViT transformer layers; its token count depends on image width and height, patch size, layer count, and the retention ratio.The formula itself is not included in the supplied passages.Source: S5. Token Merging Visualization · p. 33
• Figure S8 presents frame-wise token-merging visualizations for COCO [78], VG [57], SA-1B, and OCR-VQA, represented by 490, 154, 18, and 6 tokens, respectively.Source: S5. Token Merging Visualization · p. 33, Figure S8 · p. 34
• Across the shown token counts, the examples answer that jersey number 45 is visible and that the white car is left of the red car.The 6-token car caption places the white car closer to the foreground and the red car farther back near the intersection.Source: S5. Token Merging Visualization · p. 35

Token merging

• Most models maintain >80% performance at Rvtk=0.4; video benchmarks often perform best at low retention, with some improving below 0.1, indicating greater video-input redundancy.Retention needs vary by task: COCO-Cap CIDEr stays above 90% at Rvtk=0.3, whereas GQA accuracy reaches 90% at Rvtk=0.8.Source: S6. Ablation Studies · pp. 35, 38, Figure S9 · p. 36, Figure S10 · p. 37

Efficiency

• Reducing visual tokens and using SGLang yield faster inference across seven video-understanding datasets than retaining full visual tokens or omitting SGLang.Each input used 378 × 378 video resolution and 8 sampled frames on a single H100 GPU.Source: S6. Ablation Studies · p. 38, Figure S11 · p. 38

Slowfast inference

• Slowfast inference brings only marginal gains on video question answering and can reduce video-captioning performance, despite its additional computing cost.Across tasks, including VDC, full first-frame tokens do not consistently help and can worsen performance degradation as the retention ratio falls, particularly for video captioning.Source: S6. Ablation Studies · pp. 38, 41, Table S2 · p. 39, Figure S12 · p. 39

Training strategy and backbone selection

• Setting C—end-to-end training at Rvtk=0.1—was selected; Rvtk=1.0 raised training time, while LLM-only training reduced performance.Comparisons used the same datasets and hyperparameters; backbone selection was guided by loss over the final ten iterations of original LLaVA alignment pretraining.Source: S6. Ablation Studies · pp. 41–42, Figure S14 · p. 41, Table S3 · p. 42

• AuroraCap-7B is initialized from CLIP ViT-H and Vicuna-1.5-7B, identified as DFN5B-CLIP-ViT-H-14-378 and vicuna-7b-v1.5-16k.The model names are linked to their Hugging Face entries in the footnote.Source: S7. Detailed Training Settings · p. 42
• Images and video frames are resized to short-side 378 with aspect ratio preserved; video frames are sampled uniformly at 2 FPS, and token merging uses a constant schedule per block.Training samples use bilinear position-embedding interpolation instead of center cropping.Source: S7. Detailed Training Settings · p. 42
• Hyperparameters for both training stages are reported in Table S4.Source: S7. Detailed Training Settings · p. 42, Table S4 · p. 42
• Dataset summaries cover AuroraCap’s pretraining, vision, and language stages; vision-stage classification, reasoning, VQA, and generation use M3IT-processed data.Source: Table S6 · p. 43, Table S5 · p. 43, Table S7 · p. 43

S8. Evaluation Benchmarks and Settings

• For VDC evaluation, the maximum token count is 1,024; across benchmarks, temperature and top p are 0 and the number of beams is 1.Source: S8. Evaluation Benchmarks and Settings · p. 44, Table S8 · p. 44
• Evaluation prompts ask about camera changes, the background, the main subject and its movements, and a detailed description of the video.Source: S8. Evaluation Benchmarks and Settings · p. 44

S9. Additional Evaluation Metrics Analysis

• VDCscore’s Pearson correlation with human judgments exceeds 0.86 using 20 QA pairs, and more pairs produce more robust results.Figure S15 plots the correlation of different QA-pair counts with human Elo ranking on VDC.Source: Figure S15 · p. 44, S9. Additional Evaluation Metrics… · p. 45
• CLIP score correlates only 0.39 with human ratings, possibly because its text encoder misses fine-grained errors in lengthy captions.The authors note that a long caption may contain only one or two such errors.Source: S9. Additional Evaluation Metrics… · pp. 44–45, Figure S15 · p. 44

S10. Limitations

• AuroraCap is evaluated against LLM-based state-of-the-art methods on visual question-answering benchmarks in a zero-shot setting.The paper states that it uses the same LLM for fair comparison because VQA performance heavily depends on the LLM.Source: S10. Limitations · p. 45, Table S9 · p. 45
• AuroraCap currently performs poorly on OCR, which the authors attribute to the limited number of OCR-related samples in its training data.Source: S10. Limitations · p. 45, Table S10 · p. 45

S11. VDC Generation Prompt Template

• AuroraCap’s VDC-generation prompt asks an LLM to produce five captions: short, background, main-object, camera, and whole-video detailed captions.The captions cover complementary aspects of the video, from concise action and mood to subjects, setting, and camera work.Source: S11. VDC Generation Prompt Template · p. 46
• The reference caption prompt requires at least 300 words describing visible actions, surroundings, sensory details, camera techniques, and emotional atmosphere without frame-by-frame narration.It requests a Python dictionary string as the response format.Source: S11. VDC Generation Prompt Template · p. 46
• The detailed-caption prompt expands on the four other captions and forbids introducing entities or relationships absent from them.The resulting caption is manually checked and refined.Source: S11. VDC Generation Prompt Template · p. 47

S12. Question-answer Pairs Generation Prompt Template of VDCscore

• For VDCscore, Llama-3.1-8B generates question-answer pairs from structured ground-truth captions for subsequent evaluation.The prompt targets 20 pairs per description.Source: S12. Question-answer Pairs Generation… · p. 47
• The pair-generation instructions require open-ended questions about the main objects and actions, with each answer given as a short sentence or phrase.Questions may begin with prompts such as What, Who, Where, When, Why, or How.Source: S12. Question-answer Pairs Generation… · p. 47
• The prompt requires the pairs as a Python list of question-and-answer tuples, followed by manual checking and refinement.It specifies that no other output text or explanation should be included.Source: S12. Question-answer Pairs Generation… · p. 47

S13. More Statistics Information of VDC

• VDC’s structured captions show diverse word distributions, and Table S11 presents the distribution of question categories used for VDCscore.Figure S16 is a word cloud of the structured captions.Source: S13. More Statistics Information of VDC · p. 48, Figure S16 · p. 48, Table S11 · p. 48

S14. More Statistics Information of VDCscore

• VDCscore uses 96,902 question-answer pairs for VDC, averaging 18.87 pairs per detailed caption.The pairs are used to evaluate detailed captions.Source: S14. More Statistics Information of… · p. 48
• VDCscore questions are open-ended and cover environment and scene, characters and objects, attributes and relations, and intent and outcomes.These categories address setting and atmosphere, entities, their properties and interrelations, and interpretive questions about methods, purposes, reasons, and outcomes.Source: S14. More Statistics Information of… · p. 48
• Structured-caption sections contain similar numbers of question-answer pairs, while question types vary by caption: character and object questions dominate main-object captions, and camera questions are more prominent in camera captions.Environment and scene questions are more prominent in background captions.Source: S14. More Statistics Information of… · p. 48, Figure S17 · p. 49, Figure S18 · p. 49

S15. Calculation of Elo Ranking

• Human evaluators chose between anonymized captions from two randomly selected models for the same video, and Elo values were calculated using configured parameters.The authors collected each participating model’s captions for every video; Table S12 summarizes the Elo parameter settings.Source: S15. Calculation of Elo Ranking · p. 49, Figure S19 · p. 49, Table S12 · p. 50
• The proposed VDC metric is closest to human evaluations among the metrics compared with human Elo values.The authors report calculating Pearson correlations between metrics and human Elo values.Source: S15. Calculation of Elo Ranking · pp. 49–50

S16. Case Study

• AuroraCap provides detailed video captions about camera motion, backgrounds, and main objects with less hallucination.The case study illustrates this capability across a variety of videos.Source: S16. Case Study · p. 50

Example Caption

• The examples range from short descriptions of people at a table and a man playing drums to detailed scene captions.The detailed examples describe a woman looking toward the ocean and a man lying by a tree trunk.Source: Example Caption · pp. 51–52
• Captions describe a race-track background and camera views of a woman in a library and children playing in a backyard.The library caption describes camera movement across bookshelves, while the backyard caption describes a mostly static, eye-level view.Source: Example Caption · p. 52
• A main-object caption follows a black car driving through a debris-strewn street and navigating around a parked blue car.It also describes the car’s central position and the slightly elevated camera angle.Source: Example Caption · p. 52

S17. Predicted Answer Extraction Prompt Template

• For question-answer pairs based on the ground-truth caption, Llama-3.1-8B uses a designed prompt template to extract predicted answers based on the generated caption.Source: S17. Predicted Answer Extraction… · p. 53
• The extraction prompt requires answers to rely only on the detailed description and be a short sentence or phrase.It instructs the model to read the description carefully and provide no extra output or explanation.Source: S17. Predicted Answer Extraction… · p. 53
• In the example, the model answers that the cyclist’s hands are interacting with the bicycle handlebars.The prompt pairs a detailed video description with a question about the cyclist’s hands.Source: S17. Predicted Answer Extraction… · p. 53

S18. Correctness Evaluation Prompt Template

• Following [87], Llama-3.1-8B evaluates predicted answers against the question and correct answer, returning a True/False judgment and a relative score from 0 to 5.Source: S18. Correctness Evaluation Prompt… · p. 53
• The evaluation prompt treats synonyms or paraphrases as valid when they meaningfully match the correct answer.It asks the model to compare the predicted and correct answers for meaningful agreement.Source: S18. Correctness Evaluation Prompt… · p. 53
• The evaluation response must be a Python dictionary string with a yes/no prediction and an integer score from 0 to 5.The example accepts “Carpeted surface” for “carpet” and returns yes with a score of 5.Source: S18. Correctness Evaluation Prompt… · pp. 53–54