ARGUS: Hallucination and Omission Evaluation in Video-LLMs

1. Introduction

• VideoLLMs’ verification ability does not necessarily transfer to freeform dense captioning, a capability important for assistance to users with perceptual disabilities.The authors argue for evaluating freeform generation rather than relying solely on question answering.Source: §1 · p. 1
• Hallucination and omission costs are correlated in Figure 1, although most models exhibit more omissions than hallucinations.Marker size indicates average caption length, and Gemini-2.0-Flash achieves the best performance.Source: Figure 1 · p. 1
• ARGUS measures both hallucinations and omissions because a caption can avoid falsehoods by saying nothing, making accuracy alone inadequate.The authors frame the two measures as assessing both accuracy and completeness.Source: §1 · pp. 1–2
• ARGUS compares generated-caption sentences with human-caption sentences to identify inaccurate content, incorrect temporal relationships, and omitted important statements.The framework uses entailment analysis for hallucination judgments and checks whether the model caption contains important human-caption statements.Source: §1 · p. 2
• The authors posit that more accurate and comprehensive dense captions could bolster downstream tasks such as VideoQA and reasoning.They associate this possibility with acquiring a more holistic understanding of video content.Source: §1 · p. 2

2. Related Work

• VideoHallucer [58] uses adversarial binary questions, EventHallusion [22] extends QA to binary and open-ended questions on short single-event videos, and VIDHAL [12] uses caption reordering.These approaches assess verification rather than generating open-ended captions.Source: §2 · p. 2
• ARGUS is described as the only dedicated dense-captioning benchmark, with finer-grained evaluation and no loss in video count or length relative to other benchmarks.Table 1 compares video-hallucination benchmarks; its caption defines “Evals Video” as average evaluations per video, while tasks can differ.Source: §2 · p. 2, Table 1 · p. 2
• QA-based benchmarks can miss hallucinations because predefined questions cover a narrow error range and may not require visual understanding.The authors report that GPT-4o answered 32.52% of basic hallucination-related question pairs correctly on a VideoHalluCer subset using text only.Source: §2 · pp. 2–3
• QA approaches focused on isolated short events do not account for complex interactions and temporal dependencies across multiple events.The authors illustrate this boundary with four sequential events, for which a binary setting offers a 50% chance of guessing correctly.Source: §2 · p. 3

3. ARGUS: Freeform Captioning Benchmark

• The dynamic program assigns fabricated events to content or temporal error to prevent false matches from inflating temporal penalties.This protects the alignment of otherwise-correct temporal statements.Source: §3 · p. 3

3.1. Sentence-Level Entailment

• ARGUS matches generated-caption sentences to human-caption evidence and uses an LLM judge to classify entailment, sentence type, and supporting evidence.Dynamic actions receive temporal-order checks; summaries and visual descriptions are checked for entailment without strict ordering.Source: §3.1 · p. 3, §3 · p. 3

3.2.1. Base Cost Matrix

• The base cost is 1 for non-entailed targets; entailed summaries and visual descriptions cost 0, while dynamic actions cost 0 only at their supporting source sentence.A mismatched evidence sentence for an entailed dynamic action receives cost 1.Source: §3.2.1 · pp. 3–4

3.2.2. Dynamic Programming Formulation

• ARGUS uses dynamic programming to minimize alignment costs while accounting for temporal-order violations among dynamic actions.Its alignment history supports counting inversions; λ controls the penalty, and the described setup uses λ = 0.1.Source: §3.2.2 · pp. 4–5
• ArgusCost-H normalizes observed hallucination cost by the video-model pair's maximum possible cost to support comparisons across captions of different lengths.The maximum accounts for base penalties and the worst-case ordering penalty among entailed dynamic actions.Source: §3.2.2 · p. 5
• ArgusCost-O reverses source and target captions, measuring human-caption sentences that the generated caption does not entail.A non-entailed human sentence indicates information omitted by the model.Source: §3.2.2 · p. 5

4.1. ArgusBench: Curation and Statistics

• ArgusBench contains 500 videos; captions average 477 words per video and 24.4 words per second, with balanced sentence types and approximately 19 sentence-level evaluations per video.The videos come from existing video-understanding datasets, text-to-video datasets, and publicly available sources.Source: §4.1 · p. 5, Figure 5 · p. 6

4. Benchmarking Video-LLMs

• The evaluation covers 23 Video-LLMs—18 open-source and five proprietary—and samples 16 frames per video unless specified otherwise.Models with predefined frame-sampling strategies use those; otherwise, sampling is uniform.Source: §4.2 · p. 5

4.2. Evaluations & Insights

• ArgusCost-H was 41% for Gemini-2.0-Flash (lowest), approximately 44% for GPT-4o and GPT-4o-mini, 45% for LLaVA-Next-Video (DPO), and 80% for InternVL2 (4B).Gemini-2.0-Pro scored 48%, while LLaVA-Next-Video without DPO scored 59%.Source: Figure 6 · p. 6, §4.2 · p. 6
• Summary errors contribute little, and as ArgusCost-H falls, dynamic-action errors decrease faster than visual-description errors, leaving stronger models more reliable on temporal events than visual details.Proprietary models also produce some summary errors involving abstract interpretations of motives, atmosphere, and mood.Source: §4.2 · pp. 6–7
• ArgusCost-H and ArgusCost-O correlate at 0.65; most models omit more than they hallucinate, while caption length correlates 0.09 with omission and 0.32 with hallucination.Both costs have a mild positive correlation (approximately 0.25) with clip duration.Source: §4.2 · p. 7
• Within-family scaling generally lowers both costs, but InternVL2 reverses before returning to its 1B-model level; SmolVLM2 scaling from 256M to 2B improves hallucination by over 15% and omission by 5%.Qwen 2.5-L improves hallucination by 10% and omission by 8%.Source: §4.2 · p. 7, Figure 7 · p. 7
• With more frames, Gemini models and SmolVLM2 improve both costs, while MiniCPM-V-2.6 and Qwen2.5-VL show fluctuating hallucination costs; omission improves for all models.From 2 to 64 frames, ArgusCost-H improves by 10.02% for Gemini-2.0-Flash and 12.38% for SmolVLM2, on a 50-video subset.Source: Figure 7 · p. 7, §4.2 · p. 8

4.3. ARGUS Sensitivity Analysis

• Sensitivity checks found prompt-related standard errors of 1–3, judge-ranking correlations r ≥0.92, and 91.26% average human agreement with LLM verdicts.Inter-prompt error was slightly higher than intra-prompt error.Source: §4.3 · pp. 8–9, Table 2 · p. 8, Figure 8 · p. 8

5. Discussion & Conclusion

• ARGUS quantifies hallucinations and omissions in dense video captions, and its experiments find substantial hallucinated content even among top VideoLLMs, with trends consistent across samplings, prompts, and judges.The authors identify mitigating hallucinations and improving dense-caption accuracy as future research needs.Source: §5 · p. 9

A. The Limitations of Current QA Approaches - Extended

• 32.52% of text-only GPT-4o answers to basic hallucination-related question pairs were correct, including 61.33% in one subcategory, against 25% chance.The test used a randomly selected subset of Video-HalluCer questions without visual context, illustrating that QA scores can reflect general world knowledge.Source: A. The Limitations of Current QA… · p. 12
• Correctly verifying a claim need not translate to accurate free-form generation, as illustrated by a model verifying one chameleon but captioning two.The paper tested this distinction by turning each generated caption line from 50 randomly selected videos into a verification prompt for the same model.Source: A. The Limitations of Current QA… · p. 12
• r = −0.48 was the Pearson correlation between verification accuracy and hallucination cost across 50 videos, a moderate negative association.The study reformulated caption lines as verification prompts for the same model and compared its responses with ground-truth entailment labels.Source: A. The Limitations of Current QA… · p. 12, Figure 9 · p. 13
• Predefined QA questions cover a narrow error range, leaving other hallucinations undetected until models generate free-form captions.QA also typically isolates short events rather than representing complex interactions and temporal dependencies among multiple events.Source: A. The Limitations of Current QA… · pp. 12–13

B. Extended Related Work

• ARGUS adapts textual NLI by comparing generated caption sentences with human-caption sentences to quantify inaccurate content, visual descriptions, temporal relations, and omissions.For omission evaluation, the human caption serves as the hypothesis and the generated caption as the premise.Source: B. Extended Related Work · p. 13
• Dense video captioning focuses on annotating important visual and temporal details; earlier systems often relied on narrow datasets, while recent approaches use large multimodal models.The related-work discussion identifies surveillance, video retrieval, and educational accessibility among its applications.Source: B. Extended Related Work · pp. 13–14

• Two NVIDIA A40 GPUs with 48GB memory each and two NVIDIA A100 GPUs with 82GB memory each were used for open-source model experiments.Source: C. Additional Evaluation Details · p. 14
• Open-source models were available through HuggingFace, while proprietary models were accessed through their respective providers’ APIs.Source: C. Additional Evaluation Details · p. 14
• All evaluations of proprietary models were conducted in February 2025.Source: C. Additional Evaluation Details · p. 14

• Table 3 lists model names and corresponding checkpoints, with open-source models available via HuggingFace and proprietary models via providers.Source: Table 3 · p. 14
• Intra-prompt standard errors were 1.63 for InternVL2 and 2.86 for Qwen, which the authors say did not substantially affect conclusions.The analysis sampled multiple captions per model using the default prompt and averaged hallucination cost across three runs.Source: D. Additional Sensitivity Analysis · pp. 14–15
• Hallucination-cost trends showed minimal variation across three additional prompts compared with the default prompt.The additional prompts asked for a thorough explanation, a summary of key elements and events, or a scene-by-scene walkthrough.Source: D. Additional Sensitivity Analysis · p. 15, Figure 10 · p. 15
• r ≥ 0.92 was the Pearson ranking correlation across judge models, with GPT-4o correlations ranging from 0.92 to 0.97 against four alternatives.The alternatives were DeepSeek-R1, DeepSeek-V3, LLaMa-3.3, and Qwen-2.5; the authors report strong agreement across judge models.Source: D. Additional Sensitivity Analysis · p. 15
• Temperature increases showed no clear cost trend across models; SmolVLM2 costs decreased consistently, while MiniCPM-V-2.6 and Qwen2.5-VL oscillated.Source: D. Additional Sensitivity Analysis · p. 15

• An automated NLI evaluator may flag a visually present detail omitted from the human caption, such as a background flower, as a hallucination.The paper contrasts this with “he is running fast,” which its evaluator generally accepts when the human caption says “he is running.”Source: E. Human Study Details · pp. 15–16
• 91.26% was the average per-clip agreement rate between human reviewers and the LLM evaluator across a study with 26 graduate student volunteers.Each participant reviewed at least three clips and assessed 3.54 clips on average.Source: E. Human Study Details · p. 16
• Visual-description sentences labeled hallucinations by the LLM but judged entailments by humans had the highest disagreement rate.Source: E. Human Study Details · p. 16
• The authors characterize the evaluation as conservative: false positives can slightly over-penalize accurate details missing from human captions.They say most disagreements skew toward false-positive hallucination labels and that lines judged entailments are highly likely to be grounded in video content.Source: E. Human Study Details · p. 16

F. Additional Benchmark Details

• 12% of the dataset comes from Ego4D, 25% from Panda-70M, and 63% from free stock platforms, with miscellaneous sources adding further temporal events.VDC captions were initially drafted by GPT-4o and then manually verified and corrected by the authors.Source: F. Additional Benchmark Details · pp. 16–17
• GPT-4o judge self-bias was examined with other state-of-the-art LLM judges, whose rankings had extremely high Pearson correlation with GPT-4o’s.Source: F. Additional Benchmark Details · p. 18

G. Additional Evaluation Ablations

• About two-thirds of hallucination errors are undetermined and one-third are contradictions across models and sizes.As models improve, the undetermined proportion rises, suggesting that improvements primarily come from reducing contradiction errors.Source: G. Additional Evaluation Ablations · p. 18
• Temporal penalties contribute only 1% to 4% of total cost, reaching 4% for LLaVa-Video and InternVL2 (8B).Penalties are near zero for most models; the authors suggest weaker models often fail to generate dynamic actions, while stronger models preserve event order for actions they get right.Source: G. Additional Evaluation Ablations · p. 18, Figure 15 · p. 19
• Longer videos cluster in higher ArgusCost-H and ArgusCost-O regions, and their captions tend to be denser.Figure 16 represents duration by color and caption size by marker size.Source: G. Additional Evaluation Ablations · pp. 18–19, Figure 16 · p. 19
• Increasing frames consistently improves ArgusCost-O for all models, while ArgusCost-H falls consistently for Gemini models and SmolVLM2 but fluctuates for MiniCPM-V and Qwen2.5-VL.From 2 to 64 frames, ArgusCost-O improves by 4.45% for Gemini-1.5-Flash and 9.69% for Gemini-2.0-Flash.Source: Figure 17 · p. 20, G. Additional Evaluation Ablations · p. 20

H. Relationship With Video Characteristics

• The examined video characteristics show no strong or consistent correlation with ArgusCost-H; model-wise heatmaps show the strongest negative correlation with LAION aesthetic score.The authors describe this analysis as initial and plan to explore more comprehensive features, including motion dynamics.Source: H. Relationship With Video… · p. 20

I.1. Extended Discussion on Type and Verdict Categorization

• ARGUS separates summary, visual-description, and dynamic-action sentences, applies order penalties only to dynamic actions, and uses NLI entailment, contradiction, and undetermined verdicts.Source: §I.1 · pp. 20–22

I.2. Computational Costs

• GPT-4o NLI evaluation costs $0.01216 per video, an upper bound because base instruction tokens can be cached.The authors note that strong open-source models correlate highly with GPT-4o’s judgments and could potentially be used instead.Source: §I.2 · p. 22

I.3. Prompt

• The prompt treats the source caption as premise and each target line as a hypothesis, assigning a sentence type, evidence, reasoning, and an entailment, contradiction, or undetermined verdict.Source: §I.3 · pp. 22–23

J.1. Figure-1 Details.

• The Sunfeast example contrasts a detailed account of the boy and woman preparing pasta with a model caption that frames them as mother and son and emphasizes the product’s convenience.The detailed account includes sequential cooking actions and recipe-card details; the model caption also mentions a woman in a car.Source: §J.1 · pp. 24–25

J.2. Figure-2 Details.

• The chameleon descriptions differ: one follows a single animal changing from reddish to pink to green, while another describes red and green chameleons as separate animals.Source: §J.2 · pp. 25–26