PaperlayerView exact revision on arXiv

Source-linked AI summary

CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning

Long Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jianze Liang, Qidong Huang, Jiaqi Wang, Feng Wu, Dahua Lin

arXiv:2509.22647v1cs.CVcs.AIcs.CL

TL;DR

Image captioning needs scalable alternatives to SFT because annotated captions are costly and can encourage memorization over general, diverse descriptions. CapRL uses a decoupled LVLM–LLM pipeline, rewarding captions by how accurately a vision-free LLM answers image MCQs; it improves benchmark and caption-quality results while relying on a proxy reward with known bias risks.

  • Problem

    SFT captioning relies on expensive annotations and can make models memorize specific descriptions rather than generate diverse, general captions.

  • Method

    CapRL rewards LVLM-generated captions according to a separate vision-free LLM’s accuracy on image-related MCQs.

  • Results

    CapRL improves LVLM pretraining across 12 benchmarks and exceeds the baseline by an average 8.4% in Prism caption evaluation.

  • Takeaways & Limitations

    CapRL provides a utility-based objective reward for applying RLVR to subjective image captioning and supports dense, accurate descriptions.

  • Takeaways & Limitations

    Caption reward models can have intrinsic biases that favor verbose or brief captions, creating vulnerability to exploitation.

Abstract

from arXiv · show

Image captioning is a fundamental task that bridges the visual and linguistic domains, playing a critical role in pre-training Large Vision-Language Models (LVLMs). Current state-of-the-art captioning models are typically trained with Supervised Fine-Tuning (SFT), a paradigm that relies on expensive, non-scalable data annotated by humans or proprietary models. This approach often leads to models that memorize specific ground-truth answers, limiting their generality and ability to generate diverse, creative descriptions. To overcome the limitation of SFT, we propose applying the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm to the open-ended task of image captioning. A primary challenge, however, is designing an objective reward function for the inherently subjective nature of what constitutes a "good" caption. We introduce Captioning Reinforcement Learning (CapRL), a novel training framework that redefines caption quality through its utility: a high-quality caption should enable a non-visual language model to accurately answer questions about the corresponding image. CapRL employs a decoupled two-stage pipeline where an LVLM generates a caption, and the objective reward is derived from the accuracy of a separate, vision-free LLM answering Multiple-Choice Questions based solely on that caption. As the first study to apply RLVR to the subjective image captioning task, we demonstrate that CapRL significantly enhances multiple settings. Pretraining on the CapRL-5M caption dataset annotated by CapRL-3B results in substantial gains across 12 benchmarks. Moreover, within the Prism Framework for caption quality evaluation, CapRL achieves performance comparable to Qwen2.5-VL-72B, while exceeding the baseline by an average margin of 8.4%. Code is available here: https://github.com/InternLM/CapRL.

1 INTRODUCTION

Image captioning needs dense, accurate, and diverse descriptions, but SFT relies on costly annotations and can encourage memorization. CapRL applies RLVR through a utility-based, decoupled VQA reward and reports gains in caption quality and LVLM pretraining.

  • Motivation: SFT requires expensive, non-scalable annotations and can make captioning models less general and less diverse.
  • Core idea: CapRL defines caption quality through utility: a detailed, accurate caption should let a vision-free LLM answer image questions.
  • Results: CapRL-5M pretraining improves 12 benchmarks, while Prism performance is comparable to Qwen2.5-VL-72B and averages 8.4% above baseline.
  • Contribution: CapRL is presented as the first RLVR study for open-ended, subjective image captioning.
  • Method: Its decoupled pipeline uses an LVLM to generate captions and a separate vision-free LLM’s MCQ accuracy as the objective reward.

2 RELATED WORK

Prior work develops captioning and RLVR methods separately, while CapRL combines a decoupled VQA reward with open-ended image captioning. Its framework is positioned as a scalable application of RLVR beyond tasks with directly verifiable outputs.

  • Image Captioning: Existing captioning pipelines use synthetic captions, rewriting, information consolidation, or human- and model-assisted fine-grained annotation.
  • CapRL: CapRL extends RLVR to subjective captioning through a decoupled process in which an LLM evaluates LVLM-generated captions using MCQs.
  • RLVR: RLVR trains models with objective, easily verifiable rewards such as mathematical correctness or passing code unit tests.

3 METHODOLOGY

CapRL trains image captioners with a decoupled reward pipeline that scores captions by how accurately a vision-free LLM answers image questions from them.

  • Reward Design: An LVLM generates candidate captions, which are paired with image questions and evaluated by a separate LLM without direct image access.The caption’s comprehensiveness and accuracy determine how well the LLM can answer the questions.
  • QA Curation: CapRL uses multiple-choice VQA data and LVLM filtering to make rewards verifiable while preventing questions answerable without image context.The filtering criterion retains question-answer pairs answered correctly with the image but incorrectly without it.
  • Reward Design: Caption rewards average a non-visual LLM’s exact-match accuracy across sampled image-related questions.The reward uses multiple-choice questions and averages performance across N sampled questions.
  • Reward Design: The method avoids format-specific rewards by scoring original free-form captions directly, while using Qwen2.5-3B-Instruct as the default evaluator for efficient training.This design does not require intermediate reasoning steps before reward computation.
  • QA Curation: The curation pipeline collects diverse images, generates question-answer pairs with Qwen2.5-VL-72B, and filters them for image-dependent understanding.The retained dataset contains approximately 75k images with corresponding QA pairs.
  • Dataset Construction: CapRL-3B is trained as a lightweight captioner and then used to annotate 5M images, forming the CapRL-5M dataset.The captioner is initialized from Qwen2.5-VL-3B, while the image collection spans natural photographs, documents, charts, and user interfaces.

4 EXPERIMENTS

CapRL improves LVLM pretraining and caption informativeness across benchmark settings, with gains scaling across larger caption and QA datasets and generalizing beyond the training domain.

  • Pretraining Results: 6.8% on InfoVQA, 2.7% on DocVQA, and 3.6% on ChartVQA: CapRL-1M exceeds DenseFusion-1M under the Qwen2.5-3B setting.CapRL-1M also surpasses ShareGPT4V-1M by 1.6% on MMStar and 1.8% on MMBench.
  • Pretraining Results: CapRL-5M consistently outperforms competing pretraining datasets across all 12 benchmarks, with further gains over CapRL-1M.The reported results identify CapRL-3B-annotated captions as effective for multimodal pretraining at 5M scale.
  • Ablations: More than 2%: CapRL-3B-annotated captions outperform baselines when the compared datasets use the same images.This controls for image-source differences and isolates caption quality in the pretraining comparison.
  • Scaling: CapRL-1M consistently outperforms DenseFusion-1M across pretraining scales, with the performance gap widening as data size increases.Figure 4 presents the corresponding scaling comparison.
  • Prism Evaluation: CapRL-3B matches Qwen2.5-VL-72B and exceeds Qwen2.5-VL-3B by 14.9% on ChartQA, 12.8% on ChartQAPro, and 24.6% on InfoVQA.It also leads by 9.6% on MMStar and 6.5% on SEED.
  • Comprehensive Analysis: Single-domain training generalizes beyond the training domain, while performance improves steadily with more QA data and even one QA pair per image yields a 7.4% average gain.Performance saturates at N = 8 sampling rounds, whereas N = 1 provides an unreliable reward proxy because of option bias.

5 CONCLUSION

The paper presents CapRL as an RLVR framework for subjective image captioning that uses caption utility to create objective rewards and improve LVLM modality alignment.

  • Conclusion: CapRL applies RLVR to image captioning by rewarding captions that enable a vision-free LLM to answer image questions accurately.The framework targets dense, precise descriptions and reports benefits for LVLM pretraining.
  • Conclusion: CapRL moves beyond data-hungry SFT by creating an objective reward signal for an open-ended captioning task.The stated contribution is a step toward RLVR for subjective generation.

A CAPRL CASES

The case studies show CapRL-3B producing more comprehensive and accurate descriptions of infographics, charts, and natural images than the compared baselines.

  • Comparison with Qwen2.5-VL-3B: CapRL-3B provides more comprehensive and accurate infographic descriptions than Qwen2.5-VL-3B.The comparison highlights stronger identification of key visual information.
  • Comparison with Qwen2.5-VL-3B: CapRL-3B achieves substantially higher chart-understanding accuracy than Qwen2.5-VL-3B.Additional examples extend the reported perceptual evidence to infographics and natural images.
  • Comparison with LVLM-as-a-Judge: LVLM-as-a-Judge training can produce lengthy, irrelevant captions that exploit reward-model weaknesses rather than describing key visual information.The examples contrast this behavior with the CapRL approach.

B MORE ANALYSIS EXPERIMENTS ABOUT CAPRL

Additional analyses show that data leakage introduces harmful reward signals, while the appendix documents the prompts used for CapRL and alternative reward models.

  • QA Leakage: 1.1% worse on average: models trained with leaking QA data underperform models trained on filtered, high-quality QA data.The analysis attributes the degradation to spurious reward signals that mislead policy optimization.
  • Prompts: The appendix records prompts for caption-conditioned QA, Unified Reward Model scoring, and Qwen2.5-VL-3B reward modeling.These prompts document the implementations used in the experiments.

D DATA PROCESSING

The data pipeline emphasizes quality, diversity, safety, and leakage prevention when preparing images for CapRL. It also includes illustrative examples of CapRL-3B applied to infographic understanding.

  • D DATA PROCESSING: The pipeline filters images for redundancy, low resolution, simplicity, and safety concerns while addressing benchmark leakage.The passage describes clustering for redundant semantics and removal of unsafe or unsuitable content.
  • D DATA PROCESSING: Image preparation prioritizes quality and safety alongside dataset construction.
  • D DATA PROCESSING: CapRL-3B is illustrated in an example involving infographic understanding.

E QA PROCESSING

QA processing generates diverse multiple-choice questions and filters them to reduce leakage and uncertainty. The strict filtering threshold can discard some pairs with only mild leakage.

  • E QA PROCESSING: Five questions are generated per image, with leakage-free questions retained while dataset diversity and scale are prioritized.The number of QA pairs per image is not deliberately controlled, and additional questions provide marginal improvements after one QA already proves effective.
  • E QA PROCESSING: Questions are evaluated through repeated answer sampling and option shuffling rather than a single uncertain response.Responses are sampled four times, and accuracy is measured both with the image and with the question alone.
  • E QA PROCESSING: A threshold removes questions with high image-based accuracy but low question-only accuracy to prevent leakage.
  • E QA PROCESSING: Strict filtering can discard QA pairs containing mild or negligible leakage, while leaked-data training degrades performance rather than causing collapse.

F PRETRAINING DETAILS

Pretraining experiments vary model architectures, staged training procedures, datasets, and sample counts. The setup compares multiple baselines and includes illustrative chart-understanding examples.

  • F PRETRAINING DETAILS: Experiments cover three architecture groups combining Qwen2.5 or InternLM2.5 language models with Qwen2.5 or CLIP visual encoders.
  • F PRETRAINING DETAILS: Training follows Initial Alignment, Further Pretraining, and SFT stages, with parameters progressively unfrozen across stages.The Further Pretraining stage unfreezes the LLM, MLP, and ViT, while SFT trains all parameters on OpenLLaVA-Next.
  • F PRETRAINING DETAILS: The section includes a chart-understanding comparison between CapRL-3B and Qwen2.5-VL-3B.
  • F PRETRAINING DETAILS: Baselines compare Vanilla, ShareGPT4V-1M, DenseFusion-1M, and a 1-million-sample CapRL subset under controlled sample counts.

G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS

Multimodal research has expanded across perception, reasoning, architecture, and real-world tasks. Future directions emphasize longer context, agentic behavior, unified objectives, and efficient adaptation.

  • G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS: Multimodal systems have produced broad empirical advances across benchmarks and real-world tasks by coupling language with perception and action at scale.
  • G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS: Research explores structured reasoning and alternative ways to couple visual encoders with language backbones while preserving scalability and transfer.
  • G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS: Future themes include long-context multimodality, agentic behavior, unified pretraining objectives, and efficient adaptation.
  • G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS: The caption-based QA prompt requires answering multiple-choice questions strictly from the caption, even when prior knowledge suggests an answer.
  • G CURRENT LANDSCAPE AND FUTURE DIRECTIONS OF MULTIMODAL MODELS: A prompt example asks a model to generate five multiple-choice questions and answers about an image.
Loading 2509.22647v1…