PaperlayerView exact revision on arXiv

Source-linked AI summary

RefCaptioner: Multi-Reference Image-Grounded Video Captioning

Tengfei Liu, Yang Shi, Yuran Wang, Xiaohan Zhang, Yuqing Wen, Yuqi Tang, Qixun Wang, Zhuoran Zhang, Xuanyu Zhu, Weihong Lin, Xinlei Yu, Yujie Wei, Xinwei Long, Fengxiang Wang, Xinlong Chen, Yue Ding, Jialu Chen, Haotian Wang, Yuanxing Zhang

arXiv:2607.28509v1cs.CV

TL;DR

Existing video captioning models do not explicitly link local video elements to multiple reference images, limiting multi-reference grounding. RefCaptioner introduces a two-stage post-training framework and achieves the best overall performance among open-source models while remaining strong on general captioning benchmarks.

  • Problem

    Existing video captioning models describe videos but do not explicitly identify correspondences between video content and multiple reference images.

  • Method

    RefCaptioner combines mixed-data supervised fine-tuning with Hierarchical Coverage-Discounted GRPO to optimize factual captioning and phrase-level reference grounding.

  • Results

    RefCaptioner achieves the best overall performance among open-source models, with best or tied-best results on every reported grounding metric while remaining competitive on general captioning benchmarks.

  • Takeaways & Limitations

    Explicit multi-reference grounding can be incorporated into video captioning without compromising general description quality.

Abstract

from arXiv · show

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.

1. Introduction

The paper introduces multi-reference image-grounded video captioning, requiring factual video descriptions with phrase-level grounding to relevant reference images while rejecting distractors. RefCaptioner addresses this task through mixed-data SFT and HCD-GRPO, evaluated with MRVBench.

  • Task: Multi-reference image-grounded video captioning generates clear video captions while placing each reference-image tag after its corresponding visual phrase.Unlike traditional video captions, the task explicitly identifies relations between video content and multiple reference images.
  • Challenges: The task requires ambiguous reference selection, phrase-level reference binding, distractor rejection, and cross-reference consistency beyond standard video captioning.Models must select supported images among similar, partially relevant, redundant, or unrelated candidates and bind each selected image to the correct phrase.
  • Method: RefCaptioner uses a two-stage post-training framework combining balanced mixed-data SFT with Hierarchical Coverage-Discounted GRPO for factual captions and phrase-level reference grounding.SFT teaches reference selection and tag binding while preserving factuality, coverage, and detail; HCD-GRPO jointly optimizes captioning and grounding.
  • Evaluation: MRVBench comprises 462 real-world and AI-generated videos, 3, 831 reference images, and 2, 172 question–answer pairs for factuality evaluation.It evaluates caption factuality, reference selection, phrase-level binding, distractor rejection, and cross-reference consistency.
  • Results: RefCaptioner achieves the best overall performance among open-source models while remaining competitive on general video captioning benchmarks.This result is reported alongside human evaluation confirming annotator preference and more source-faithful video reconstruction.

2. Related Work

Related work has progressed from short video summaries toward detailed, temporally precise, and long-form descriptions. Benchmarking has likewise expanded from lexical-overlap evaluation toward factual correctness and fine-grained visual content.

  • Video Captioning: Video captioning evolved from short clip summaries to detailed descriptions with richer temporal understanding through shared video–text representations and end-to-end video Transformers.Early methods include VideoBERT and UniVL, while SwinBERT enabled end-to-end caption generation with video Transformers.
  • Video Captioning: Multimodal captioners such as Tarsier and AuroraCap describe subjects, attributes, actions, scenes, and camera changes.These models extend caption content beyond concise clip summaries toward richer visual and cinematic descriptions.
  • Video Captioning: Recent systems target extended or fine-grained videos: Video ReCap and LongCaptioning produce hierarchical or long-form descriptions, while ProgressCaptioner emphasizes frame-level temporal precision.These advances improve the richness and temporal fidelity of video captions.
  • Video Caption Benchmarks: Early benchmarks MSR-VTT and VATEX established large-scale open-domain and multilingual testbeds, commonly using lexical-overlap metrics such as BLEU and CIDEr.These benchmarks were designed primarily for concise captions.
  • Video Caption Benchmarks: Recent benchmarks emphasize factual correctness and fine-grained content because lexical-overlap metrics are less suitable for long, free-form descriptions.DREAM-1K and VDC evaluate verifiable visual facts, while VCapsBench and VidCapBench are identified as more recent benchmarks in the supplied passage.

3. Problem Formulation

Multi-reference video captioning generates a detailed, faithful video caption with explicit local grounding tags for semantically corresponding reference images. The task evaluates observable semantic correspondence between references and video content rather than causal generation provenance.

  • Problem Formulation: The task takes a video V and candidate reference images I = {I1, . . . , IN} as input.The reference set contains N candidate images.
  • Problem Formulation: The goal is to generate a detailed and faithful caption Y with explicit local grounding tags <Image i>.Grounding tags are attached to locally corresponding phrases in the caption.
  • Problem Formulation: Each image tag should be placed immediately after a phrase that semantically matches the referenced image.This is the stated accuracy requirement for a valid output.

4. Method

RefCaptioner uses two post-training stages: mixed-data supervised fine-tuning followed by Hierarchical Coverage-Discounted GRPO (HCD-GRPO). Its rewards jointly assess factual video description, valid reference binding, distractor rejection, and consistency across multiple references.

  • Two-Stage Post-Training: RefCaptioner fine-tunes Qwen3-VL-8B-Instruct first on equal-ratio multi-reference and general-caption data, then applies HCD-GRPO for factual captioning and grounding.The multi-reference data teaches relevant-reference selection and phrase-level tag placement, while general captions preserve detailed video description ability.
  • Hierarchical Coverage-Discounted GRPO: HCD-GRPO samples candidate captions, normalizes rewards within groups, and updates only the language model while freezing the visual encoder and vision–language merger.Its two branches separately evaluate caption correctness and completeness, and reference selection, binding, and grouping; coverage rewards are discounted by observable errors.
  • Caption Reward: A keypoint bank and video-QA bank score factual coverage and errors, with structured judges assigning values 0, 0.5, or 1 for graded coverage and factual correctness.Incorrect content discounts factual coverage, while unanswered questions receive no additional error penalty because omissions are captured by keypoint coverage.
  • Reference Reward: Valid-reference coverage increases only when references are emitted and correctly bound to corresponding phrases, preventing incorrectly bound tags from earning positive reward.The design also discourages avoiding difficult binding decisions by emitting few or no image tags.
  • Reference Reward: Distractor-Aware Evidence Suppression penalizes captions that use references whose content does not appear in the video.The distractor penalty activates only for samples containing distractors, while consistency reward activates only when an entity has multiple references.
  • Reference Reward: Entity-level consistency is maximized when all references for an entity are grouped after its corresponding phrase, and decreases for omissions, unrelated tags, or split views.An LLM judge extracts tags immediately following each entity phrase and compares them with the annotated tag set without merging tags across separate phrases.

5. MRVBench

MRVBench evaluates multi-reference video captioning through factuality, reference selection, and phrase-level reference binding. It is paired with a 20,000-video, 171,354-reference-image corpus and includes complex reference mappings and injected distractors.

  • Benchmark scope: MRVBench jointly evaluates video factuality, reference selection, and phrase-level reference binding.Its metrics cover fact coverage, question-answering support, reference-image precision and recall, and phrase-level tag binding.
  • Data construction: 20,000 videos and 171,354 reference images form the authorized-source training corpus.The benchmark and corpus are constructed through MLLM-assisted annotation.
  • Benchmark composition: Approximately 60% of samples contain complex mappings where multiple references correspond to the same subject or visual unit.These mappings test multi-reference association within the benchmark.
  • Benchmark composition: Approximately 40% of samples, about 185, contain injected distractors.Distractor-containing samples test whether systems reject irrelevant reference images.

6. Experiments

RefCaptioner delivers strong standard video-captioning performance while achieving its clearest gains in multi-reference grounding, robustness, and downstream video reconstruction. Ablations show that HCD-GRPO’s reward components provide complementary supervision for factuality, distractor rejection, and subject recall.

  • Standard video-caption evaluation: RefCaptioner achieves the best overall performance among evaluated open-source models, with highest open-source KP-Cov and VQA-Cov on standard video-caption evaluation.Its KP-Cov is comparable to proprietary baselines, and it maintains factual video understanding with more complete descriptions.
  • Multi-reference grounding evaluation: RefCaptioner achieves the best or tied-best result on every reported grounding metric among open-source models and surpasses both proprietary baselines on Ref-Tag-R, Subj-R, and Subj-F1.It exceeds at least one proprietary baseline on Ref-Tag-P, Eff-Bind, and Dist-Rej, indicating valid reference selection and binding.
  • Caption refinement comparison: RefCaptioner performs best on Ref-Tag-R, Ref-Bind, Subj-R, and Subj-F1 against every two-stage CaptionRefine baseline, while Qwen3-VL-32B-Instruct with CaptionRefine achieves highest KP-Cov.The comparison uses the same 462 MRVBench samples, contrasting one-pass grounded generation with post-hoc revision.
  • General video-captioning ability: RefCaptioner achieves the highest accuracy across all five VDC dimensions, with 5.66–6.08-point gains over Qwen3-VL-8B in background, main-object, and detailed description.This demonstrates that reference-oriented post-training preserves general video-captioning ability rather than specializing only in image-tag insertion.
  • Reference-grounding robustness: 0.769 is RefCaptioner’s robustness score on the 13+ reference group, exceeding GPT-5.4 at 0.703 and Qwen3.6-35B-A3B at 0.398; it leads across all groups.The score multiplies reference recall, binding accuracy, and distractor rejection, penalizing weakness in any component.
  • Downstream utility and ablations: RefCaptioner-based reconstructions receive more Good than Bad judgments against every baseline, while HCD-GRPO’s full model achieves the best VQA, Dist-Rej, and Subj-R among ablations.Removing the factual-caption reward most reduces VQA, DAES most affects distractor rejection, and CRSC consistently decreases subject recall when removed.

7. Conclusion

RefCaptioner addresses multi-reference image-grounded video captioning by combining mixed-data SFT with HCD-GRPO, and introduces MRVBench for evaluating factuality and multi-reference grounding. It achieves the best overall performance among open-source models while remaining strong on general captioning benchmarks, with human evaluation validating its caption quality.

  • Contributions: RefCaptioner combines mixed-data SFT with HCD-GRPO to optimize factual captioning and reliable reference grounding.The framework targets multi-reference image-grounded video captioning.
  • Contributions: MRVBench evaluates video factuality and multi-reference grounding on real-world and AI-generated videos.
  • Results: RefCaptioner achieves the best overall performance among open-source models while remaining strong on general captioning benchmarks.
  • Human Evaluation: Human evaluation validates RefCaptioner’s caption quality.

Supplementary Material · A. MRVBench Construction and Evaluation · A.1. Benchmark Overview

MRVBench is a multi-reference video-captioning benchmark with 462 test samples spanning real and AI-generated videos. Each sample pairs a video with tagged candidate references, a grounded golden caption, reference annotations, and frozen evaluation banks.

  • A.1. Benchmark Overview: 462 test samples comprise 185 AIGC videos and 277 real videos.The benchmark covers both AI-generated and real-world video content.
  • A.1. Benchmark Overview: 3,831 candidate reference images are included across the test set.Candidate references include both useful positive references and distractors.
  • A.1. Benchmark Overview: Each sample contains one video and multiple tagged candidate reference images.The candidate set is designed to support multi-reference selection and distractor handling.
  • A.1. Benchmark Overview: Each sample provides a reference-grounded golden caption for evaluating grounded video description.The golden caption is paired with the video and candidate references.
  • A.1. Benchmark Overview: Per-reference keep/delete annotations identify which candidate references should be retained or discarded.These annotations distinguish useful references from distractors.
  • A.1. Benchmark Overview: Same-entity reference-group annotations support evaluation of consistency across references depicting the same entity.The benchmark explicitly records reference-group relationships.
  • A.1. Benchmark Overview: Frozen key-point and video-QA banks provide fixed resources for benchmark evaluation.Both evaluation banks are frozen for consistent assessment.

A.2. Data Sources and Release Scope … B.1. Base Model and Supervised Fine-Tuning

The paper withholds its lawfully collected training corpus because redistribution rights vary, while releasing MRVBench data and annotations permitted by applicable licenses. It defines structured reference-grounding annotations and evaluation procedures, then fine-tunes Qwen3-VL-8B-Instruct with parameter-efficient LoRA adaptations.

  • A.2. Data Sources and Release Scope: The training corpus will not be publicly released because source redistribution rights differ, but MRVBench test data and annotations permitted by applicable licenses will be released.The corpus was collected from authorized sources under appropriate usage rights.
  • A.3. Annotation Structure: Positive references are assigned to concrete visual units, grouped only for the same entity across viewpoints or appearances, and paired with grounded phrases and visual evidence.Semantically related but distinct entities are not merged.
  • A.3. Annotation Structure: The frozen key-point bank covers six dimensions across 5,846 key points from 462 samples, while the QA bank contains 2,172 questions with reference answers and evidence dimensions.The dimensions are subject, appearance, action, background, camera, and style.
  • A.4. Metric Definitions: MRVBench evaluates key-point coverage, video QA, reference selection, phrase-level binding, distractor rejection, false references, and subject consistency using defined pooled or macro-averaged metrics.Reference selection uses unique-tag matching, while binding scores assess whether each phrase’s visual referent matches its referenced image.
  • A.5. Evaluation Protocol: All models receive the same instruction, ordered references, and video, with Gemini-3.1-Pro judging structured text-only outputs for key-point coverage, video QA, binding, and subject consistency.Final scores are computed programmatically from the judge’s structured outputs.
  • A.6. Qualitative Example: The qualitative MRVBench example uses 20 candidate references and shows captions placing image tags after corresponding visual phrases, grouping same-element references, and omitting unsupported candidates.The example is generated by RefCaptioner.
  • B.1. Base Model and Supervised Fine-Tuning: RefCaptioner uses Qwen3-VL-8B-Instruct with frozen vision components and LoRA adapters on language-model attention and feed-forward projections for supervised fine-tuning.The LoRA configuration uses rank 64, alpha 128, and dropout 0.05; SFT runs for three epochs on 8× NVIDIA H800 GPUs and selects checkpoints by validation loss.

B.2. GRPO Post-Training … B.5. Repeated Runs and Statistical Significance

The paper details GRPO post-training, a reward combining caption quality with multi-reference grounding, and shared visual preprocessing constraints. It also reports consistent, statistically significant gains over the base model across three repeated runs.

  • B.2. GRPO Post-Training: GRPO starts from the SFT checkpoint, freezes the visual encoder, optimizes the language model without LoRA, and uses 12,838 training examples plus 126 validation examples.Samples exceeding 16,000 processed prompt tokens are excluded before training.
  • B.2. GRPO Post-Training: Training uses 32× NVIDIA H800 GPUs across four nodes, sampling 96 captions per update with a 1×10^-6 actor learning rate and 0.005 KL coefficient.Each update samples six responses for 16 prompts, caps responses at 512 tokens, and selects the final policy by validation performance.
  • B.3. Reward Configuration: The GRPO reward combines factual-caption quality with multi-reference grounding quality through caption key-point coverage and reference-binding criteria.The reference branch penalizes incorrect local binding, distractor use, and inconsistent grouping of references depicting the same entity.
  • B.3. Reward Configuration: λ_qa = 0.5, and the caption and reference reward branches are combined with equal weights.The caption branch discounts video-QA errors using the specified error-discount coefficient.
  • B.3. Reward Configuration: Malformed, invented, non-local, or prohibited tags use τ_struct = 0.2 and receive a capped reward instead of the raw reward.Without a structural violation, the reward remains R = Rraw.
  • B.4. Visual Preprocessing: SFT and GRPO sample videos at 2 fps, limit frames to 602,112 pixels and reference images to 401,408 pixels, and remove overlong multimodal samples rather than truncating them.These constraints define the shared visual preprocessing setup for both stages.
  • B.5. Repeated Runs and Statistical Significance: Across three independent repeated runs, RefCaptioner consistently outperforms the base model, with the improvement reported as statistically significant.The construction seed is fixed at 20260710.

C. General Video-Captioning Evaluation … F. Human Evaluation Details

The paper evaluates general captioning under standardized protocols, specifies strict multi-reference captioning and judging procedures, and tests downstream reconstruction with fixed generation settings. The supplied passages also detail qualitative reconstruction comparisons, but provide no substantive human-evaluation details.

  • C.1. Evaluation Setup: All baseline models use identical video inputs, benchmark instructions, and deterministic decoding, with benchmark-specific sampling, generation, and evaluation procedures.This establishes a consistent comparison across captioning methods.
  • C.2. VDC: VDC evaluation samples videos at 1 fps with up to eight frames and 2,048 new tokens, then applies its two-stage question-based protocol over 1,027 videos.Gemini-3.1-Pro judges generated-caption answers against reference answers using the benchmark’s aggregation procedure.
  • C.3. VCapsBench: VCapsBench requests captions of at least 200 words, samples up to 64 frames at 2 fps, allows 1,024 new tokens, and evaluates against a video-specific QA bank.Gemini-3.1-Pro and the benchmark’s answer mapping and aggregation are applied consistently to all methods.
  • D.1. Training Input: Training inputs place an ordered list of tagged reference images before one video, while allowing distractors that need not appear in the output.The task requires a fluent caption describing visible content and locally binding usable reference tags to grounded visual phrases.
  • D.1. Training Input: The formatting rules require tags immediately after concrete noun phrases, permit stacked tags only for the same visual unit, and require ungrounded distractor tags to be omitted.Captions must avoid unsupported names, relationships, causes, dialogue, audio, and story details.
  • D.2. Evaluation Prompts: Reference-grounded outputs are judged separately for key-point coverage, video QA, phrase–tag binding, and subject consistency using structured labels, scores, or predicted tag sets.The protocols penalize unsupported content, incorrect referents, missing tags, and intruder tags.
  • E.1. Reconstruction Protocol: Downstream reconstruction provides only the reference images cited by each caption, removes image tokens, and uses fixed Wan2.1-VACE-14B generation settings across caption sources.Outputs contain 81 frames at 1280 × 720 or 720 × 1280, using seed 0, 50 sampling steps, and guidance scale 5.0.
  • E.2. Qualitative Comparisons: Qualitative comparisons use Wan 2.1 VACE and Seedance 2.0, keeping generation settings identical within each backend while conditioning on each captioner’s caption and selected reference subset.This compares end-to-end reconstruction conditioning under matched backend settings.

F.1. Human Preference Study · SOURCE VIDEO · F.2. Reconstruction Evaluation

Human evaluators preferred RefCaptioner’s captions and found their reference-to-phrase bindings accurate across real and AIGC videos. Caption-conditioned reconstruction comparisons further evaluated the combined utility of each model’s captions and selected reference-image subsets under controlled generation settings.

  • F.1. Human Preference Study: Three experts ranked anonymized captions from five models on 40 MRVBench examples spanning 20 AIGC and 20 real videos.Evaluators viewed the source video, candidate references, and generated captions with randomized presentation order.
  • F.1. Human Preference Study: Evaluators independently ranked captions for overall preference and naturalness, with lower mean rank indicating stronger preference.Rank 1 denoted the most preferred output and rank 5 the least preferred; mean ranks were computed across samples and evaluators.
  • SOURCE VIDEO: Reconstruction comparisons used captions and selected reference-image subsets from each captioner at common timestamps for AIGC samples 000069, 000066, and 000070.Wan 2.1 VACE generated the reconstructions for samples 000069, 000066, and 000070.
  • F.1. Human Preference Study: Reference-level judgments separately measured whether each image was bound to the correct local phrase in each caption.Binding accuracy aggregated independent judgments across samples and evaluators.
  • F.1. Human Preference Study: RefCaptioner’s human preference reflects caption structure and expression in addition to correct image insertion.Its training organizes captions around subjects, appearance, actions, background, visual style, and camera work, binding components to appropriate references rather than repeatedly reusing one image.
  • SOURCE VIDEO: Additional Seedance 2.0 comparisons covered AIGC samples 000072 and 000181, including identity, clothing, setting, and action details.The comparisons used each captioner’s caption and selected reference-image subset under the same generation settings.
  • F.2. Reconstruction Evaluation: Three experts compared RefCaptioner-conditioned reconstructions against each baseline using anonymized pairwise Good/Same/Bad judgments.Source videos and two reconstructions were shown with randomized left–right order and hidden model identities.
  • F.2. Reconstruction Evaluation: The reconstruction evaluation measured combined downstream utility because each captioner supplied both generated text and its cited reference-image subset.The reconstruction backend, generation settings, and random seed were held fixed across captioners.
Loading 2607.28509v1…