PaperlayerView exact revision on arXiv

Source-linked AI summary

VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis

Shubhashis Roy Dipta, Tz-Ying Wu, Subarna Tripathi

arXiv:2509.16538v3cs.CVcs.CL

TL;DR

Reference-free video-caption evaluation remains underexplored, and existing metrics have limitations in factual assessment, context handling, and interpretability. VC-Inspector uses controllable factual-error captions for training a lightweight open-source multimodal evaluator that predicts quality and explains factual errors. It is reported to correlate highly with human judgments across video and image-caption benchmarks, while its focus and synthetic training data leave scope limitations.

  • Problem

    Reference-free evaluation of in-the-wild video captions remains underexplored, and existing metrics face limitations in context handling, scoring consistency, reproducibility, and video-specific modeling.

    Source: §1 · p. 1

  • Method

    VC-Inspector is a lightweight, open-source multimodal evaluator trained with captions containing controllable factual errors to predict quality and explain factual judgments.

    Source: §1 · p. 2

  • Results

    VC-Inspector achieves high correlation with human judgments, outperforming reference-free methods and rivaling reference-based metrics across video and image-caption benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its factual explanations are intended to make quality assessments interpretable and guide caption improvement.

    Source: §1 · p. 2

    VC-Inspector primarily targets object and action correctness; attributes, spatial relationships, and fine-grained temporal ordering are not explicitly modeled, and training partly relies on synthetic captions and pseudoscores.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We propose VC-Inspector, a lightweight, open-source large multimodal model (LMM) for reference-free evaluation of video captions, with a focus on factual accuracy. Unlike existing metrics that suffer from limited context handling, weak factuality assessment, or reliance on proprietary services, VC-Inspector offers a reproducible and fact-aware alternative that aligns closely with human judgments. To enable robust training and interpretable evaluation, we introduce a systematic framework for generating captions with controllable factual errors, paired with graded quality scores and explanatory annotations. Experiments demonstrate that VC-Inspector achieves state-of-the-art correlation with human judgments, generalizing across diverse domains (e.g., VATEX-Eval, Flickr8K-Expert, and Flickr8K-CF benchmarks) and revealing the potential for caption improvement. Project page is available at https://dipta007.github.io/VC-Inspector

1 Introduction

The introduction frames reference-free video-caption evaluation as underexplored and argues that existing approaches struggle with factual accuracy, context, and interpretability. VC-Inspector pairs factual assessment with explanations and controllable training data, and is reported to align strongly with human judgments across caption benchmarks.

1 Introduction

  • Reference-free evaluation for in-the-wild video captions remains underexplored, while existing metrics have context, scoring-scale, reproducibility, and video-specificity limitations.Embedding-based metrics can be context-limited and difficult to interpret; proprietary-model approaches depend heavily on prompt engineering.Source: §1 · p. 1
  • The proposed scoring goal is to lower a caption’s score in proportion to factual errors, because objects and actions are critical to video understanding.The introduction illustrates this with captions that differ in whether the subject is a girl or a woman.Source: §1 · pp. 1–2
  • A controllable generation framework creates video captions with factual errors and graded quality, yielding 44K ActivityNet-FG-It instances for instruction tuning.The source captions come from ActivityNet-Captions.Source: §1 · p. 2
  • VC-Inspector jointly predicts caption quality and explains factual errors, aiming to make judgments interpretable and guide caption improvement.The method is built on a lightweight, open-source large multimodal model.Source: §1 · p. 2
  • VC-Inspector achieves high correlation with human judgments, outperforming reference-free methods and rivaling reference-based metrics across video and image-caption benchmarks.The evaluation also reports strong performance on object-error hallucination benchmarks.Source: §1 · p. 2

2 Related Works

Prior caption metrics range from text-only reference comparisons to image-aware and reference-free approaches, but video evaluation remains challenging. The review highlights limits in temporal modeling, factual assessment, and interpretability, while Figure 2 depicts a graded-caption data pipeline used for instruction tuning.

2 Related Works

  • Text-only reference metrics capture syntactic or semantic similarity but depend on human-written captions, can struggle with paraphrases, and ignore visual content.The review contrasts rule-based metrics, SPICE, embedding-based BERTScore, and LLM-as-a-Judge CLAIR.Source: §2 · p. 2
  • Image-aware methods align captions with visual input, but some still require references and image-focused approaches do not model video temporal dynamics.VIFIDEL is limited to discrete object categories, while VLM-based metrics offer richer semantic alignment but remain constrained by image-centric settings.Source: §2 · pp. 2–3
  • Video reference-free metrics such as EMScore, PAC-S, and FactVC have limits in context, corruption granularity, and explanations, while proprietary-model approaches raise scalability and reproducibility concerns.The review contrasts prior binary corruption distinctions with VC-Inspector’s varying caption quality and interpretable explanations.Source: §2 · p. 3
  • Figure 2 depicts a pipeline that generates synthetic captions with diverse quality scores and explanations, then uses the dataset to instruction-tune VC-Inspector.The figure presents generation on the left and instruction tuning on the right.Source: Figure 2 · p. 3

3 Methodology

VC-Inspector is a reference-free video-caption evaluator that targets factual correctness using a lightweight open-source LMM. Its method synthesizes captions with controlled factual errors, graded scores, and explanations, then trains the evaluator to return scores and explanatory text.

3.1 Overview

  • VC-Inspector evaluates video-caption pairs without references, targeting factual correctness with a lightweight open-source LMM and interpretable scores plus explanations.The framework focuses on object and action inaccuracies and is designed to address the need for targeted supervision of pretrained LMMs.Source: §3.1 · pp. 3–4, §3.2 · p. 5

3.2 Data Generation

  • An LLM extracts objects and actions, replaces sampled elements with distinct, plausible alternatives, and generates factual-error explanations for pseudo-captions.The sampled replacements define the altered set; the resulting caption and explanation provide supervision for training.Source: §3.2 · p. 4
  • The deterministic score subtracts the fraction of changed objects and actions from 1, then discretizes the result to a 1–5 scale.More incorrect elements yield lower scores; discretization preserves ordinal relationships and error severity.Source: §3.2 · pp. 4–5
  • Generation yields 374K pseudo-captions, balanced to about 218K across five score categories, then subsampled to 44K examples with 8.8K per label.The 44K subset is ActivityNet-FG-It, used for instruction tuning while preserving category balance and diversity.Source: §3.2 · p. 5

3.3 Training of VC-Inspector

  • VC-Inspector fine-tunes the LLM component of Qwen2.5-VL while freezing its video encoder and visual-language projector, predicting a score and explanation.Explanations use information about altered objects and actions collected during data generation.Source: §3.3 · p. 5, §3.2 · p. 5

4 Experiments

Experiments evaluate VC-Inspector’s training setup, consistency, human correlation, hallucination sensitivity, data-synthesis choices, explanations, and runtime. Across the supplied results, it performs strongly against evaluation baselines and its explanation supervision supports caption refinement.

4.1 Experimental Settings

  • Training uses 44K ActivityNet-FG-It video-caption pairs with quality scores and explanations, and evaluation reports Kendall’s τb and Spearman’s ρ.The experiments assess reliability, human correlation, targeted factual-error sensitivity, and component contributions.Source: §4.1 · p. 5

4.2 Baselines

  • The baseline comparison includes language-based, image-augmented, and video-based metrics, with reference-free evaluation as the primary focus.ViCLIPScore is formed by adapting CLIPScore to ViCLIP, and the vanilla Qwen2.5-VL model is also compared.Source: §4.2 · p. 6

4.4 Correlation with Human Judgments

  • VC-Inspector outperforms reference-free metrics on VATEX-Eval, achieves the highest correlation on YouCook2-Eval, and remains best-performing under the reference-free setting on Flickr8K benchmarks.On VATEX-Eval it also surpasses most reference-based metrics; on Flickr8K it outperforms several reference-based metrics.Source: Table 2 · p. 6, §4.4 · pp. 7–8, Table 3 · p. 7

4.3 Consistency and Reliability of Quality Estimates

  • On synthetic ActivityNet-FG-Eval and YouCook2-FG-Eval, VC-Inspector outperforms baselines at differentiating incorrect captions and generalizes beyond ActivityNet training.The evaluation datasets contain pseudo-captions with varying factual errors; Table 1 reports correlations with ground-truth annotations.Source: §4.3 · p. 6, Table 1 · p. 6

4.5 Sensitivity to Hallucinations

  • VC-Inspector performs strongly on image- and video-based hallucination benchmarks that test detection of captions with a single object error.The reported results indicate sensitivity to targeted factual errors across video lengths and visual contexts.Source: §4.5 · p. 8, Table 5 · p. 8

4.6 Ablation on Data Synthesis Strategies

  • Changing both objects and actions in training captions yields better human-rating alignment than changing either factual element alone.All tested variants align strongly with human ratings relative to EMScore.Source: §4.6 · p. 8, Table 6 · p. 8

4.7 Analysis of Explanations

  • On VATEX-Eval, training with pseudo-explanations improves model performance over training only on scores; for caption refinement, VC-Inspector feedback consistently beats no feedback and off-the-shelf Qwen2.5-VL-7B feedback on reported object-semantics and event dimensions.Source: §4.7 · pp. 8–9, Table 7 · p. 9

4.8 Computational Efficiency

  • VC-Inspector-3B averages 0.30 seconds per clip on one A100 GPU, compared with 0.42 seconds for EMScore and 0.34 seconds for ViCLIPScore.Runtime is measured over 2,000 videos, each uniformly sampled into 16 frames.Source: §4.8 · p. 9

5 Conclusion

VC-Inspector is presented as a fact-aware, reference-free evaluator for video captions, trained using captions with controllable factual errors. Across multiple domains, it aligns highly with human judgments and detects factual errors.

5 Conclusion

  • VC-Inspector is a lightweight, open-source multimodal evaluator trained on captions with systematically generated, controllable factual errors.The conclusion describes it as a reference-free tool for assessing video-caption factual accuracy.Source: §5 · p. 9
  • Experiments across multiple domains show high alignment with human judgments and stronger factual-error detection than existing metrics.The conclusion also identifies explanation ablations as supporting quality estimation and caption refinement.Source: §5 · p. 9
  • The authors characterize VC-Inspector as a practical tool for assessing factual accuracy in real-world video-captioning scenarios.They describe its versatility and interpretability as part of that practical scope.Source: §5 · p. 9

Limitations

VC-Inspector focuses on object and action correctness, while several other factual dimensions are not explicitly modeled. Its synthetic training data and multimodal backbone also impose scope and computational boundaries.

Limitations

  • Attributes, spatial relationships, and fine-grained temporal ordering are not explicitly modeled, despite the focus on object and action correctness.The paper identifies these other aspects as directions for future work.Source: Limitations · p. 10
  • Synthetic captions and pseudoscores may not fully capture the diversity of real-world captioning errors.The authors note that the scores are deterministic and validated through correlation with human judgments.Source: Limitations · p. 10
  • Dependence on a multimodal backbone makes VC-Inspector more computationally costly than purely embedding-based metrics.The paper nevertheless describes it as substantially lighter than proprietary LLM-based evaluators.Source: Limitations · p. 10

A Qualitative Results

Qualitative examples show VC-Inspector assigning scores that track human judgments while explaining factual inaccuracies in candidate captions. Examples with progressively more errors illustrate its severity-sensitive scoring.

A Qualitative Results

  • Examples from ActivityNet-FG-Eval and VATEX-Eval show quality assessments consistent with ground-truth scores and explanations of factual errors.The figure marks factual errors in red.Source: A Qualitative Results · p. 12, Figure 5 · p. 13
  • The explanations pinpoint incorrect objects or actions in candidate captions, with scores assigned accordingly.Source: A Qualitative Results · p. 12
  • For examples with progressively more factual errors, VC-Inspector detects the incorrect elements and gives scores reflecting error severity.The examples are from ActivityNet-FG-Eval.Source: A Qualitative Results · p. 12

B.1 Evaluation of Explanation Quality using Language Models

VC-Inspector explanations are evaluated against pseudo-ground-truth explanations on two synthetic datasets. The reported results show close alignment, while an additional visualization illustrates identification of incorrect objects and actions.

B.1 Evaluation of Explanation Quality using Language Models

  • VC-Inspector’s explanations and quality assessments align closely with pseudo-ground-truth explanations on two synthetic datasets.VATEX-Eval lacks ground-truth explanations; Table 9 reports BERT score and LLM Score for this experiment.Source: §B.1 · p. 13, Table 9 · p. 13
  • BERT score measures semantic similarity to pseudo-ground-truth explanations, while LLM Score uses a Llama-3-8B-Instruct judge and rescales predicted-explanation scores relative to ground truth.Source: §B.1 · p. 13

B.2 Complete Results for Caption Refinement

The complete caption-refinement results provide F1 scores for all 12 CAPability dimensions, while the reported experiment focuses on objects and actions. A figure shows an initial and refined caption produced with VC-Inspector guidance.

B.2 Complete Results for Caption Refinement

  • F1 scores are provided for all 12 CAPability dimensions, with the experiment focusing on objects and actions as the work’s primary focus.Table 10 contains the complete dimension results.Source: §B.2 · p. 13
  • Figure 7 visualizes an initial and refined caption produced under VC-Inspector guidance.Source: §B.2 · p. 13

C Training Hyperparameters

The appendix reports instruction-fine-tuning hyperparameters and gives the prompts used to generate factual-error examples, train and evaluate explanations, and refine captions. It also discloses the authors’ use of AI assistance and illustrates a caption-refinement example.

C Training Hyperparameters

D Prompts

  • Prompt D.6 requests a 1–5 caption-quality score and explanation for training; D.7 evaluates explanations against ground truth, and D.8 refines captions using evaluations and video frames.Source: D Prompts · pp. 14–15
  • Prompts D.1 and D.2 extract object and action lists from captions for use in the prompt set.Source: D Prompts · p. 14
  • Prompts D.3–D.5 generate distinct co-hyponym objects or alternate actions and substitute them into captions to create factual errors.Source: D Prompts · pp. 14–15

E Use of AI Assistance

  • The authors used Cursor for coding and ChatGPT for proofreading and language refinement, with original author-written content as input; a refinement example corrects object errors and raises CAPability from 3/5 to 5/5.Source: E Use of AI Assistance · p. 15, Figure 7 · p. 16
Loading 2509.16538v3…