PaperlayerView exact revision on arXiv

Source-linked AI summary

CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning

Long Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jianze Liang, Qidong Huang, Jiaqi Wang, Feng Wu, Dahua Lin

arXiv:2509.22647v1cs.CVcs.AIcs.CL

TL;DR

Image captioning relies on expensive, non-scalable SFT annotations, while its open-ended quality is difficult to reward objectively. CapRL uses an LVLM to generate captions and a vision-free LLM’s question-answering accuracy as a utility-based reward, yielding gains across 12 benchmarks and an average 8.4% margin over baseline in Prism.

  • Problem

    Image captioning needs dense, accurate descriptions, but SFT depends on expensive, non-scalable human or proprietary annotations and open-ended captions lack an objective reward.

    Source: §1 · pp. 1–2

  • Method

    CapRL uses an LVLM to generate captions and a vision-free LLM’s QA accuracy to provide an objective, utility-based reward.

    Source: §1 · p. 3, §5 · p. 10

  • Results

    Across 12 benchmarks, pretraining on CapRL-5M yields substantial gains; in Prism, CapRL-3B performs comparably to Qwen2.5-VL-72B and exceeds baseline by an average margin of 8.4%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CapRL shows RLVR can address subjective captioning with utility-based rewards, generating dense, precise descriptions that substantially promote modality alignment in LVLM pretraining.

    Source: §5 · p. 10

    Captioning models remain vulnerable to exploitation because intrinsic biases may encourage overly verbose or brief outputs.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Image captioning is a fundamental task that bridges the visual and linguistic domains, playing a critical role in pre-training Large Vision-Language Models (LVLMs). Current state-of-the-art captioning models are typically trained with Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable data annotated by humans or proprietary models. This approach often leads to models that memorize specific ground-truth answers, limiting their generality and ability to generate diverse, creative descriptions. To overcome the limitation of SFT, we propose applying the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm to the open-ended task of image captioning. A primary challenge, however, is designing an objective reward function for the inherently subjective nature of what constitutes a "good" caption. We introduce Captioning Reinforcement Learning (CapRL), a novel training framework that redefines caption quality through its utility: a high-quality caption should enable a non-visual language model to accurately answer questions about the corresponding image. CapRL employs a decoupled two-stage pipeline where an LVLM generates a caption, and the objective reward is derived from the accuracy of a separate, vision-free LLM answering Multiple-Choice Questions based solely on that caption. As the first study to apply RLVR to the subjective image captioning task, we demonstrate that CapRL significantly enhances multiple settings. Pretraining on the CapRL-5M caption dataset annotated by CapRL-3B results in substantial gains across 12 benchmarks. Moreover, within the Prism Framework for caption quality evaluation, CapRL achieves performance comparable to Qwen2.5-VL-72B, while exceeding the baseline by an average margin of 8.4%. Code is available here: https://github.com/InternLM/CapRL.

1 INTRODUCTION

CapRL makes caption quality verifiable through a caption’s usefulness for answering image questions, addressing challenges of subjective rewards and SFT-based captioning. The paper reports gains across benchmarks and strong Prism evaluation performance.

1 INTRODUCTION

  • SFT relies on costly, non-scalable annotations and can make models memorize single reference captions, weakening generality and diversity across valid descriptions.The paper presents CapRL as the first study applying RLVR to subjective, open-ended image captioning to encourage broader, more general descriptions.Source: §1 · pp. 1–2
  • RLVR for open-ended captioning is difficult because subjective captions lack straightforward objective rewards, while judge and reward-model feedback can invite reward hacking.Reference-based rewards such as ROUGE and BLEU are also constrained for complex, long-form captions.Source: §1 · p. 2
  • CapRL rewards captions by measuring whether a separate vision-free LLM can answer image MCQs from them, using curated questions answerable only from image content.The QA curation pipeline filters questions and answers for image-grounded content; images include natural scenes, charts, and documents.Source: §1 · p. 3
  • 8.4% average margin over the baseline on Prism accompanies CapRL-3B performance comparable to the 72B model; CapRL-5M pretraining yields gains across 12 benchmarks.The CapRL-5M captions were annotated by CapRL-3B, and LVLM pretraining on this dataset produced substantial improvements across the benchmarks.Source: §1 · p. 3

2 RELATED WORK

Prior captioning work scales through increasingly elaborate synthetic-caption and annotation pipelines, while some approaches depend on costly labeled data for SFT. CapRL instead uses RLVR for data-efficient captioning.

Image Captioning

Reinforcement Learning with Verifiable Rewards (RLVR)

  • RLVR uses verifiable rewards such as answer correctness or passing unit tests, and unlike SFT imitation, encourages exploring solutions for tasks with multiple valid answers.Examples include mathematical problem-solving (Shao et al., 2024) and code generation (Team et al., 2025); RLVR is described as more robust and scalable than SFT, which can promote memorized phrasings (Chu et al., 2025).Source: §2 · p. 4

3 METHODOLOGY

CapRL separates caption generation from reward evaluation: an LVLM drafts captions, while a vision-free LLM answers image-linked multiple-choice questions to score them. The trained CapRL-3B captioner is then used to create CapRL-5M.

3.1 CAPRL

  • CapRL trains an LVLM with GRPO by scoring candidate captions through a vision-free LLM’s accuracy on image-linked MCQs, then optimizing group-relative advantages with a KL penalty.Because the LLM cannot see the image, its answer accuracy depends on how comprehensive and accurate the caption is.Source: §3.1 · p. 5
  • The caption reward is average exact-match accuracy over N independently answered sampled questions, with answer choices shuffled to reduce choice bias.The default answering model during GRPO training is Qwen2.5-3B-Instruct.Source: §3.1 · p. 5
  • CapRL generates QA pairs from diverse web and open-source images with Qwen2.5-VL-72B (Bai et al., 2025), then filters for image-dependent answers, retaining about 75k images.Filtering keeps pairs answered correctly with the image but not without it, using Qwen2.5-VL-3B to control computational costs.Source: §3.1 · pp. 5–6

3.2 CAPRL-5M DATASET

  • CapRL-3B, initialized from Qwen2.5-VL-3B and trained with CapRL, annotates 5M images to form CapRL-5M.The image collection combines ShareGPT4V-1M and DenseFusion-1M with 3M rigorously filtered web images spanning photographs, documents, charts, and user interfaces.Source: §3.2 · p. 6

4 EXPERIMENTS

Across pretraining and caption-evaluation experiments, CapRL improves benchmark performance and benefits from scaling; Prism evaluation finds CapRL-3B comparable to a 72B model, while domain-restricted training generalizes and sparse QA supervision yields gains.

4.1 PRETRAINING SETTING

  • CapRL-1M beat competing pretraining datasets on most benchmarks, exceeding DenseFusion-1M by 6.8% on InfoVQA, 2.7% on DocVQA, and 3.6% on ChartVQA in the Qwen2.5-3B + Qwen2.5-ViT setting.CapRL-5M performed consistently better across all 12 benchmarks, with performance improving as dataset size grew from 1M to 5M; CapRL-1M also exceeded ShareGPT4V-1M by 1.6% on MMStar and 1.8% on MMBench.Source: §4.1 · pp. 6–7, Table 1 · p. 7
  • With image sets controlled, CapRL captions yielded over 2% higher performance than the baselines; across data scales, CapRL consistently beat DenseFusion and the gap widened.The controlled-image comparison indicates that caption quality, rather than image diversity, largely explains CapRL’s advantage.Source: §4.1 · p. 7, Figure 4 · p. 8

4.2 PRISM SETTING

  • In Prism evaluation, CapRL-3B significantly outperformed Qwen2.5-VL-3B and 7B and achieved performance comparable to Qwen2.5-VL-72B.Against Qwen2.5-VL-3B, CapRL-3B was higher by 14.9% on ChartQA, 12.8% on ChartQAPro, 24.6% on InfoVQA, 9.6% on MMStar, and 6.5% on SEED.Source: Table 3 · p. 8, §4.2 · p. 8
  • In GRPO training, UnifiedReward-2.0-qwen-3b caused captioning ability to deteriorate as average caption length continuously decreased and collapsed to “:description”; Qwen2.5-VL-3B instead preferred overly verbose captions.Qwen2.5-VL-3B’s reward led to long passages irrelevant to the image and captioning ability significantly inferior to CapRL; the authors characterize LVLM-as-a-Judge reward as fundamentally unreliable.Source: §4.2 · p. 9

4.3 COMPREHENSIVE DISCUSSION ABOUT CAPRL

  • Training on chart-type images alone improved document, chart, and general VQA performance, demonstrating generalization beyond the training domain.The comparison used 20k images from a single domain and found gains over Qwen2.5-VL-3B on out-of-domain benchmarks.Source: Figure 5 · p. 8, §4.3 · p. 9
  • Captioning performance improved steadily as the amount of QA training data increased, indicating promising scaling potential.The authors identify continued data expansion as a way to further enhance captioning ability.Source: Figure 5 · p. 8, §4.3 · p. 9
  • With one QA pair per image, Qwen2.5-VL-3B averaged 7.4% higher captioning performance than baseline and was only 0.5% below CapRL-2QA-20k.This result shows that sparse QA supervision can still produce substantial captioning gains.Source: §4.3 · p. 9, Table 4 · p. 9
  • Performance improved as sampling rounds N increased from 1 to 4 and saturated at N = 8; N = 1 was weak because unshuffled options made LLM accuracy an unreliable reward proxy.The resulting option bias could misdirect policy optimization.Source: §4.3 · p. 9, Table 5 · p. 9

5 CONCLUSION

CapRL applies RLVR to subjective image captioning by defining caption quality through its usefulness to a vision-free LLM answering questions. The framework encourages dense, precise descriptions and promotes modality alignment in LVLM pretraining.

5 CONCLUSION

  • CapRL applies RLVR to image captioning using a reward based on whether a vision-free LLM can accurately answer questions from the caption.This reframes caption quality around its utility for question answering, providing an objective reward signal for a subjective task.Source: §5 · p. 10
  • CapRL encourages models to produce dense, precise image descriptions that substantially promote modality alignment in LVLM pretraining.Source: §5 · p. 10
  • The work positions RLVR as an alternative to the restrictive, data-hungry SFT paradigm for open-ended tasks.Source: §5 · p. 10

A CAPRL CASES

Illustrative cases show CapRL-3B’s strengths in infographic, chart, physical-image, and natural-image understanding, alongside captioning failures associated with LVLM-as-a-Judge rewards.

Comparison with Qwen2.5-VL-3B

  • CapRL-3B provides comprehensive, accurate infographic information and outperforms Qwen2.5-VL-3B in chart and physical-image understanding in the illustrated examples.The comparisons are shown in Figs. 6–8 and 12.Source: A CAPRL CASES · p. 14

Comparison with UnifiedRW-as-Judge-3B and Qwen2.5VL-as-Judge-3B

  • Qwen2.5VL-as-a-Judge-3B produces lengthy, irrelevant captions that exploit reward hacking, while UnifiedRW-as-Judge-3B produces overly short captions omitting critical chart information.Examples are presented in Figs. 9–11.Source: A CAPRL CASES · p. 14

More cases of CapRL-3B in understanding infographics and natural images

  • Additional examples in Figs. 13 and 14 provide evidence of CapRL-3B’s perceptual capacity for understanding infographics and natural images.Source: A CAPRL CASES · p. 14

B MORE ANALYSIS EXPERIMENTS ABOUT CAPRL · C PROMPT USED

Training on QA data with leakage reduced average performance, as leaked QA could reward captions that were not genuinely better. The paper also provides prompts for caption-conditioned answering and two reward-model configurations.

B MORE ANALYSIS EXPERIMENTS ABOUT CAPRL

C PROMPT USED

  • The prompt section includes CapRL’s prompt for guiding an LLM to answer questions conditioned on captions.The prompt is shown in Figure 15.Source: C PROMPT USED · p. 14
  • The prompt section also includes prompts for using the Unified Reward Model and Qwen2.5-VL-3B as reward models.These prompts are shown in Figures 16 and 17, respectively.Source: C PROMPT USED · p. 14

D DATA PROCESSING

The CapRL-5M data pipeline removes redundant or unsuitable images, filters out images similar to benchmark samples, and uses human safety screening before finalizing the dataset.

D DATA PROCESSING

  • The three-stage processing pipeline produces the final CapRL-5M dataset through semantic deduplication, benchmark-leakage filtering, and human safety verification.Human screening is sample-based and stops when unsafe images fall to a negligible level.Source: D DATA PROCESSING · p. 14
  • Semantic clustering removes redundant images, while the pipeline also discards low-resolution, overly simple, and unsafe content.Safety concerns include violence and pornography.Source: D DATA PROCESSING · p. 14
  • To reduce benchmark leakage, the pipeline clusters images from commonly referenced evaluation datasets and removes collection images overly similar to them.Source: D DATA PROCESSING · p. 14

E QA PROCESSING

CapRL constructs diverse image-question pairs with Qwen2.5-VL-72B and filters for questions whose answers depend on image content rather than question-only cues. Its filtering uses repeated, option-shuffled responses, while strict criteria can also discard pairs with only mild leakage.

QA construction

  • Qwen2.5-VL-72B generates five questions per image; the pipeline retains those without leakage and prioritizes dataset size and diversity over a fixed QA count per image.A single QA per image is highly effective, while adding more pairs brings marginal improvements.Source: E QA PROCESSING · p. 15

QA filtering

  • Questions are sampled four times with shuffled options, and filtering thresholds compare LVLM accuracy using the image against accuracy using only the question.Questions with high image-based accuracy but low question-only accuracy are filtered out.Source: E QA PROCESSING · pp. 15–16
  • Strict filtering can discard QA pairs with mild or negligible leakage, helping explain why leaked-data training degrades performance rather than causing training collapse.Source: E QA PROCESSING · p. 16

F PRETRAINING DETAILS

The section specifies the model architectures, three-stage training setup, and pretraining baselines used in the experiments.

Model Architecture

  • Experiments covered three combinations of LLM and vision encoder: Qwen2.5-3B with Qwen2.5-ViT, Qwen2.5-7B with Qwen2.5-ViT, and InternLM2.5-7B with CLIP-ViT-L.The LLM and ViT were pretrained, while the MLP projector was randomly initialized.Source: F PRETRAINING DETAILS · p. 16

Training Setting

  • Training used three stages: BLIP-558K initial alignment, high-quality-caption further pretraining, and OpenLLaVA-Next SFT.Initial alignment unfroze the MLP (learning rate 1e-3; batch size 256); further pretraining unfroze the LLM, MLP, and ViT (4e-5; 256); SFT unfroze all parameters (2e-5; 128).Source: F PRETRAINING DETAILS · pp. 16–18

Baselines

  • Baselines included Vanilla, which skipped further pretraining, and further-pretraining variants using ShareGPT4V-1M, DenseFusion-1M, or a randomly sampled 1M-pair CapRL subset.The 1M-pair CapRL subset controlled for sample count against the other dataset variants.Source: F PRETRAINING DETAILS · p. 18

G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS

Multimodal models have expanded beyond image–text systems toward broader perception, generation, reasoning, and interaction. Future progress is expected to center on long-context multimodality, agentic behavior, unified objectives, and efficient adaptation.

G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS

Related papers

Loading 2509.22647v1…