Source-linked AI summary
Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning
Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng, Ziqi Miao, Gege Shi, Qiuyu Zhao, Jinsong Lan, Xiaoyong Zhu, Bo Zheng
TL;DR
Image-captioning RL struggles to make LVLMs explore novel reasoning strategies, limiting caption refinement. Re3Cap uses multimodal retrieval through CRS and CQA to identify hallucinations and omissions without extra annotations, and experiments show superiority over prior methods including SFT.
Problem
RL-based LVLM captioning has limited exploration of novel reasoning strategies, while generated captions remain vulnerable to hallucinations and omitted visual details.
Method
Re3Cap uses multimodal retrieval with CRS and CQA to preserve supported content, detect hallucinations and omissions, and guide refinement without additional annotations.
Results
Re3Cap consistently outperforms previous SFT approaches in image captioning and achieves the best performance when CRS and CQA are combined.
Takeaways & Limitations
Retrieval discrepancies provide an informative signal for assessing caption quality and help LVLMs generate previously unexplored caption candidates.
Takeaways & Limitations
Re3Cap depends on the quality and scale of its retrieval set; small datasets may cause relevant results to be missed and reduce the method to simple RL.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re$^3$Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re$^3$Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.
1 Introduction
Existing image-captioning methods often produce hallucinations and omit fine-grained visual details, while RL-based LVLM training poorly explores novel reasoning strategies. Re3Cap uses multimodal retrieval with CRS and CQA to refine captions without additional annotations, improving performance beyond prior methods and SFT.
- Image captioning remains difficult because generated captions are prone to hallucinations and often miss fine-grained visual details.
- RL methods commonly optimize rewards over pregenerated candidates, limiting exploration of novel reasoning strategies and unexplored captions.
- Multimodal retrieval exposes caption misalignment because discrepancies between image-based and caption-based retrieval descriptions indicate hallucinations.
- Re3Cap combines CRS and CQA to preserve consistently supported caption elements, identify hallucinations and omissions, and guide refinement without additional annotations.
- Re3Cap guides LVLMs toward previously unexplored caption candidates, and experiments report improvements across LVLMs over state-of-the-art methods, including SFT.
2 Related Work
Image captioning research spans supervised, retrieval-augmented, self-supervised, controllable, and reinforcement-learning approaches. Re3Cap’s retrieval-based strategy analyzes semantic overlap and retrieval discrepancies to guide caption refinement.
- Image captioning is a fundamental computer-vision task and a bridge between visual and linguistic modalities.
- Encoder–decoder captioners generate text autoregressively from visual representations, while retrieval-augmented methods additionally condition on relevant external texts.
- SFT, self-supervised, and controllable approaches improve caption accuracy or detail but can depend on expensive, large-scale image–caption datasets.
- RL captioning methods optimize task-specific rewards, including image–caption correlation, cycle consistency, self-retrieval, and self-correction objectives.
- Re3Cap analyzes semantic elements consistent across visually similar images and uses retrieval discrepancies to guide LVLMs toward retaining key content and refining captions.
3 Method
Re3Cap uses retrieval-guided signals to identify semantic content, hallucinations, and omissions in captions, then trains LVLMs to generate refined descriptions. Its inference policy ultimately depends only on the image, without retrieval or k-core analysis.
- Caption Refinement Suggester: CRS suggests the retained semantic elements so LVLMs incorporate them into more accurate refined captions.The retained elements correspond to content consistently appearing across descriptions of visually similar images.
- Caption Refinement Suggester: CRS retrieves visually similar image-text pairs, constructs a sentence-similarity graph, and uses k-core analysis to retain semantically consistent descriptions.The retrieval graph connects sentence nodes when their SBERT similarity exceeds a threshold, allowing long-tail descriptions to be filtered.
- Caption Quality Assessor: CQA compares image- and caption-based retrieval results through graph structures to characterize shared content, hallucinations, and omissions.It reformulates caption quality assessment as textual discrepancy analysis without requiring additional annotations.
- Reinforcement Learning Optimization: During training, Re3Cap samples initial captions, generates retrieval-based guidance, prompts refined captions, and scores each refined caption with a reward.The policy is optimized after removing initial captions and guidance from its input, creating an off-policy setting addressed with a decoupled proximal policy.
- Overview: Re3Cap combines CRS and CQA to identify correct elements, hallucinations, and omitted semantic content for caption refinement.CRS and CQA derive guidance from image-conditioned and caption-conditioned retrieval, respectively.
- Reinforcement Learning Optimization: At inference, the optimized policy is conditioned solely on the image, eliminating the need for retrieval or k-core analysis.The initial caption and guidance are used during rollout but removed from the policy input during optimization.
4 Experiment
Experiments evaluate Re3Cap’s reasoning capability, component contributions, and performance across LVLMs, benchmarks, reward functions, and state-of-the-art baselines. Results show stronger exploration and consistent captioning gains, including improvements over GRPO and SFT-based methods.
- Experimental Settings: Experiments span reasoning analysis, multiple LVLMs, COCO-LN500 and DOCCI500, GRPO reward functions, state-of-the-art comparisons, and component ablations.The evaluation also includes training-free and SFT-based methods.
- Reasoning Capability Analysis: At k = 1, CIM-based reinforcement learning performs strongly, but its growth slows as k increases and eventually falls below the base model.This pattern suggests narrowed output diversity under conventional RL.
- Reasoning Capability Analysis: Re3Cap benefits from larger sampling budgets without additional training, surpasses the base model as k increases, and generates more diverse, previously unexplored caption candidates.The results indicate that Re3Cap expands the base model’s capability boundary.
- Reinforcement Learning on Base Model: 8.64% improvement in Relations QA on COCO-LN500 and 7.57% on DOCCI500 over GRPO are achieved by Re3Cap, averaged across base LVLMs and reward functions.With CLIP rewards, gains average 4.39% in Objects F1, 2.44% in Attributes F1, and 6.74% in Relations QA across both benchmarks.
- SOTA Comparison: Re3Cap outperforms SFT-based methods by 4.08% in Relations QA on COCO-LN500 and 4.86% on DOCCI500, averaged across reward functions.It also gains 1.47% in Objects F1 and 1.21% in Attributes F1 on COCO-LN500, using single-stage RL rather than a two-stage SFT-plus-RL pipeline.
- Ablation Study: CRS improves Objects F1, Attributes F1, and Relations QA by 1.03%, 0.76%, and 2.19%, while CQA improves them by 1.34%, 0.82%, and 3.98%, respectively.Combining the complementary CRS and CQA components achieves the best performance.
5 Conclusion
Re3Cap is a reinforcement learning framework that expands caption candidates through retrieval-guided reasoning and consistently outperforms prior SFT approaches in image captioning.
- Re3Cap uses retrieval-based reasoning to guide LVLMs toward previously unexplored caption candidates during reinforcement learning.The framework uses discrepancies between retrieval results to assess caption quality and refine generation behavior.
- Re3Cap consistently achieves superior image-captioning performance compared with strong Supervised Fine-Tuning baselines.
- The framework is presented as a perspective for enhancing LVLM reasoning in reinforcement learning and caption generation.
Limitations
Re3Cap’s effectiveness depends on the quality and scale of its retrieval set, with small datasets reducing the availability of relevant retrieval results.
- Re3Cap’s effectiveness depends on the quality and scale of the retrieval set.
- When the retrieval dataset is too small, many captions fail to retrieve relevant results.
- Under this condition, the method may degenerate into a simple reinforcement learning approach.
- Increasing the retrieval set’s scale and diversity is expected to improve the approach’s robustness.
Ethical Considerations
The experiments use public image-caption datasets and benchmarks whose biases and imperfections may affect model behavior and evaluation outcomes; no additional method-specific ethical risks were identified.
- Experiments are conducted on publicly available image-caption datasets and benchmarks.
- Existing datasets and evaluation protocols may contain social, annotation, and sampling biases that affect model behavior and evaluation outcomes.
- The authors identify no additional ethical risks introduced specifically by Re3Cap beyond risks associated with multimodal training, retrieval, and evaluation on public datasets.
B Additional Experiments
Additional experiments with InternVL3-8B and Qwen3-VL-8B show that Re3Cap consistently outperforms GRPO across multiple reward functions, especially for relation reasoning.
- Re3Cap consistently outperforms GRPO across multiple reward functions on InternVL3-8B and Qwen3-VL-8B.
- 6.59% is Re3Cap’s average improvement over GRPO in Relations QA on COCO-LN500 for InternVL3-8B across multiple reward functions.
- 5.12% is Re3Cap’s average improvement over GRPO in Relations QA on DOCCI500 for InternVL3-8B across multiple reward functions.
B.2 Robustness of the Choice of Hyperparameters
Re3Cap remains robust across the tested retrieval-number and similarity-threshold settings. Performance changes only slightly as K and τ vary.
- K varied from 3 to 11, while τ varied from 0.5 to 0.9 in the hyperparameter study.The experiments evaluated both retrieval number and graph-construction similarity threshold.
- Across K values, variations in Objects F1, Attributes F1, and Relation QA remained within 0.53%, 0.74%, and 1.44%, respectively.
- Across τ values, variations in Objects F1, Attributes F1, and Relation QA remained within 1.32%, 1.48%, and 1.38%, respectively.
- The results demonstrate that Re3Cap is robust to the choice of these hyperparameters.
B.3 Robustness across Diverse Encoders
The encoder study evaluates alternative image and text encoders and finds only minor performance variation. These results indicate robustness across diverse encoder choices.
- The study compares DINOv3 ViT-L/16 and OpenCLIP ViT-H/14 as image encoders.
- It compares SBERT with MiniLM-base and MPNet-base backbones as text encoders.
- Performance varies only slightly across the evaluated encoder configurations.
B.4 Analysis of Computational Overhead
Under the reported settings, Re3Cap requires modestly more training time than GRPO while maintaining comparable memory overhead. The supplied material reports a 12.5% training-time increase alongside significant performance gains.
- 216 GPU hours were required for Re3Cap to converge, compared with 192 GPU hours for GRPO under the same settings.
- 12.5% more training time corresponds to Re3Cap's additional 24 GPU hours over GRPO.
- Re3Cap's memory overhead is comparable to that of GRPO.