RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

Tzu-Heng Huang; Sirajul Salekin; Javier Movellan; Frederic Sala; Manjot Bilkhu

Source-linked AI summary

RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

Tzu-Heng Huang, Sirajul Salekin, Javier Movellan, Frederic Sala, Manjot Bilkhu

arXiv:2603.09160v1Mar 10, 2026cs.CVcs.AIcs.LG

TL;DR

Dense image captioning needs scalable supervision, but manual annotation is expensive, synthetic-caption distillation can limit diversity and generalization, and open-ended quality lacks deterministic verification. RubiCap addresses this bottleneck by turning diverse VLM consensus and policy deficiencies into sample-specific, structured rubric rewards. It achieves leading captioning results, strong word efficiency, better capability retention than supervised distillation, and stronger pretrained VLMs from compact-model captions than from proprietary-model captions.

Problem
Dense captioning requires expensive expert annotations, while supervised distillation of synthetic captions can limit diversity and generalization and open-ended quality lacks deterministic verification.
Method
RubiCap uses diverse VLM candidate captions and an LLM rubric writer to create sample-specific, interpretable criteria for structured reinforcement-learning rewards.
Results
RubiCap achieves the highest win rates across compared methods, matches 32B-scale CaptionQA performance at 7B, preserves pretrained capabilities better than supervised distillation, and improves pretraining utility.
Takeaways & Limitations
Compact RubiCap captioners can provide high-quality training data, with RubiCap-3B captions producing stronger pretrained VLMs than GPT-4V captions.
Takeaways & Limitations
At 3B and 2B scales, the Reference-Likert reward eventually induces self-praising reward hacking and yields low PixMoCap win rates.

Abstract

from arXiv · show

Dense image captioning is critical for cross-modal alignment in vision-language pretraining and text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthetic captioning via strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these limitations, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers -- a luxury not available in open-ended captioning. We address this bottleneck with RubiCap, a novel RL framework that derives fine-grained, sample-specific reward signals from LLM-written rubrics. RubiCap first assembles a diverse committee of candidate captions, then employs an LLM rubric writer to extract consensus strengths and diagnose deficiencies in the current policy. These insights are converted into explicit evaluation criteria, enabling an LLM judge to decompose holistic quality assessment and replace coarse scalar rewards with structured, multi-faceted evaluations. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it demonstrates superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using the compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.

1 Introduction

Dense image captioning needs scalable, fine-grained supervision, but conventional synthetic-caption distillation and existing RL rewards struggle with diversity, generalization, and open-ended verification. RubiCap addresses this gap with sample-specific rubrics and achieves strong results across caption quality, efficiency, capability retention, and pretraining utility.

Dense captioning requires expert-level visual perception and precise language grounding, making manual annotation prohibitively expensive at frontier-model scale.
SFT on synthetic captions can collapse linguistic diversity and induce severe catastrophic forgetting.
Open-ended caption quality is subjective and context-dependent, leaving RL without a deterministic verifier.
RubiCap uses diverse VLM candidate captions, LLM-written deficiency analyses, and interpretable rubrics to produce structured, multi-faceted rewards.The rubrics assess dimensions such as object presence, attribute correctness, spatial reasoning, and hallucination.
RubiCap-7B achieves the highest win rates across compared methods, while RubiCap-3B and 7B show strong word efficiency against larger models.The 7B model matches 32B-scale frontiers on CaptionQA, and the 3B model outperforms 7B base models.
RubiCap-trained models preserve pretrained capabilities more effectively than supervised distillation, and RubiCap-3B captions yield stronger pretrained VLMs than GPT-4V captions.

2 Related Works

Prior dense-captioning work moved from concise scene summaries toward localized, fine-grained descriptions, while RL methods have mainly succeeded where correctness is automatically verifiable. RubiCap targets open-ended captioning by replacing limited multiple-choice coverage with consensus-grounded rubrics.

Dense Image Captioning: Dense captioning extends scene-level summaries by requiring global description and localization of multiple salient regions.The fine-grained annotation burden motivates synthetic alternatives.
Reinforcement Learning in Visual Language Models: RL has advanced in domains with automatically verifiable correctness, but open-ended dense captioning lacks ground-truth verification.
Reinforcement Learning in Visual Language Models: CapRL uses multiple-choice questions as proxy rewards, but failures absent from the option set go unpenalized.
Reinforcement Learning in Visual Language Models: RubiCap’s diverse VLM committee produces open-ended criteria capable of surfacing failures that fixed choice sets cannot anticipate.

3 Framework

RubiCap builds image-conditioned rubrics from diverse teacher captions and targeted student-deficiency analysis, then uses an LLM judge and GRPO to optimize captioning with structured rewards.

3.1 Setup and Notation: RubiCap uses a committee of diverse VLMs to generate teacher captions that cover more visual elements than a single reference.The framework collects teacher captions for each image before rubric synthesis.
3.2 Automated Rubric Synthesis: The rubric writer compares student captions with teacher consensus, retaining consensus elements supported by at least ⌈K/2⌉ teachers and diagnosing discriminative omissions or errors.This suppresses single-teacher noise and targets only deficiencies the student failed to capture or misrepresented.
3.2 Automated Rubric Synthesis: Each diagnosed deficiency becomes a human-readable binary criterion with severity weights of 1.0, 2.0, or 3.0 for minor, important, and critical failures.Every criterion must support an unambiguous pass/fail judgment.
3.2 Automated Rubric Synthesis: The resulting rubrics are image-conditioned and discriminative, checking gaps such as object presence, attribute correctness, spatial reasoning, and hallucination.They adapt to the image and the student’s current failure modes rather than applying a fixed generic checklist.
3.3 Rubric-Guided Reinforcement Learning: An LLM judge converts criterion satisfaction into binary scores, which are combined into a normalized severity-weighted reward measuring addressed quality gaps.The judge replaces coarse scalar evaluation with structured, multi-dimensional signals.
3.3 Rubric-Guided Reinforcement Learning: GRPO samples multiple student rollouts, estimates each rollout’s advantage relative to the group, and updates the student policy using importance ratios and clipping.Rollouts outperforming their peers are rewarded, while those below the group mean are penalized.

4 Experiments

Across dense-captioning experiments, RubiCap consistently improves caption quality, self-improvement, knowledge preservation, word efficiency, and downstream pretraining utility over supervised, RL, human, and proprietary baselines.

4.1 Comparison with Supervised Distillation and Other RL Objectives: RubiCap achieves the highest win rates against base models across model sizes and datasets, including +20.8% on PixMoCap and +14.4% on DenseFusion for RubiCap-7B.These gains exceed SFT variants, RL baselines, and synthetic-caption distillation methods.
4.2 Comparison with Human Annotations and Proprietary Models: RubiCap surpasses human-expert annotations and GPT-4V-enriched outputs, improving RubiCap-7B win rates over the base model by 13.4% on PixMoCap and 8.4% on DenseFusion.RubiCap-generated captions win more than half of pairwise comparisons against both reference types.
4.2 Comparison with Human Annotations and Proprietary Models: RubiCap-7B earns the highest rank-1 proportion, lowest hallucination penalty, and strongest accuracy while matching the 72B model on completeness and clarity.The blind ranking comparison includes 72B and 32B frontier models.
4.3 Knowledge Preservation: RubiCap preserves pretrained capabilities better than SFT-based models across 10 VLM benchmarks, which show the largest degradation and weaker generalization.The results indicate reduced catastrophic forgetting relative to supervised distillation.
4.4 Effectiveness beyond Rubric-Augmented SFT: RubiCap outperforms rubric-augmented SFT even with identical rubrics, reaching 68.6% versus 64.0% at 3B and 70.8% at 7B.At 7B, RubiCap also exceeds rubric-augmented SFT by 6.2 percentage points against human-expert refined captions.
4.5 Word Efficiency: Under a 100-word limit, RubiCap-7B improves CaptionQA by +12.01% over Qwen2.5-VL-7B-Instruct, while RubiCap-3B improves by +9.53% over its 3B counterpart.RubiCap-3B and RubiCap-2B also surpass the larger 7B base model under identical constraints.
4.5 Word Efficiency: RubiCap-7B outperforms the 32B model under 100–300 word limits and matches it under 400–600 word budgets, providing 32B-level caption quality at 7B inference cost.The gains are attributed to prioritizing salient, information-dense content through multiple evaluation dimensions.
4.6 Pretraining Utility: RubiCap-annotated captions improve downstream VLM pretraining by 3.42% relative to GPT-4V on average across 9 benchmarks, including with compact 3B models.RubiCap-3B annotations yield stronger pretrained VLMs than captions from the proprietary GPT-4V system.

5 Conclusion

RubiCap addresses the verification bottleneck in RL-based dense captioning with synthetic, sample-specific rubrics and demonstrates strong performance across caption quality, efficiency, and downstream pretraining.

RubiCap addresses the verification bottleneck in RL-based dense captioning by deriving synthetic, sample-specific rubrics as fine-grained reward signals.
RubiCap achieves the highest win rates in GPT-4.1-judged comparisons against all baselines and surpasses human-expert annotations and GPT-4V-augmented outputs at 7B scale.
RubiCap-7B obtains the highest proportion of rank-1 assignments in blind ranking, outperforming 72B and 32B frontier models.
Under fixed word budgets, RubiCap produces more information-dense captions, enabling a 7B model to match 32B-scale performance at lower inference cost.
A compact RubiCap-3B captioner yields stronger pretrained VLMs than those trained on proprietary GPT-4V captions.

A Appendix

The appendix documents RubiCap’s rubric-synthesis and judging prompts, including consensus-based criteria, weighted binary checks, and structured reward evaluation, alongside baseline prompts.

The appendix presents prompts for rubric synthesis, LLM judging, baseline reward methods, and caption-quality assessment.
Rubric items require clear pass/fail rules, severity weights, and justifications tied to teacher consensus for interpretable scoring.
The rubric writer uses diverse teacher outputs and weak-model captions to identify consensus, evaluate quality dimensions, and create targeted criteria only for unmet weaknesses.
The LLM judge evaluates each generated caption against one rubric criterion and returns a binary satisfaction score with reasoning.
Baseline VLM-as-a-judge prompts assign holistic caption-quality scores from 0 to 10, with Reference-Likert additionally supplying a reference caption.

D Evaluation Prompts

The evaluation prompts compare captions through randomized GPT-4.1 pairwise judgments and blind multidimensional rankings that emphasize factual precision, informativeness, clarity, completeness, and hallucination penalties.

CapArena computes pairwise win-lose rates with GPT-4.1 using randomized caption assignments and detailed winner justifications.
Blind ranking evaluates RubiCap, Qwen2.5-VL models at 72B, 32B, and 7B, and PixMoCap expert captions across five caption sources.
Pairwise evaluation prioritizes precision and informativeness while ignoring writing style, phrasing, caption length, and grammatical variation.
The ranking evaluator scores accuracy, completeness, clarity, and hallucination penalty on integer scales from 0 to 10.
Overall score is computed as (accuracy + completeness + clarity) / 3.0 - hallucination_penalty x 1.5, with captions ranked by descending total score.

E Algorithm Overview

Algorithm 1 summarizes RubiCap as a two-stage framework: automated rubric synthesis followed by rubric-guided reinforcement learning.

Algorithm 1 unifies RubiCap’s two stages: automated rubric synthesis and rubric-guided reinforcement learning.

F Training Details

RubiCap constructs sample-specific rubrics from teacher consensus and student deficiencies, then uses rubric-based LLM judgments to train captioning policies with GRPO. Experiments use fixed datasets, model scales, rollout settings, and SFT baselines.

Data and compute: RubiCap samples 50,000 training images and holds out 500 evaluation images from each dense captioning dataset.The datasets are PixMoCap and DenseFusion, with the remainder used for training.
Rubric synthesis: The automated rubric pipeline extracts teacher consensus, diagnoses student deficiencies, and formulates binary criteria with severity weights.Criteria are generated per image from teacher captions and the current student caption.
Rubric-guided RL: During reinforcement learning, the student generates N rollouts and an LLM judge assigns binary satisfaction scores for every rubric criterion.The judge evaluates each rollout against the synthesized rubric set.
Optimization: GRPO training uses a 1 × 10−5 learning rate, cosine scheduling, 0.01 warmup, 1,024-token completions, N = 4 generations, and 1 epoch.All RL baselines use the same training configuration except for reward design.
Baselines: The SFT baseline trains for 1 epoch with the same 1 × 10−5 learning rate and uses expert-refined, GPT-4V-augmented, or frontier-model-distilled captions.The supervision sources include PixMoCap, DenseFusion, and Qwen2.5-VL-72B-Instruct.

G Captioning Results

RubiCap is evaluated with preference-based, reference-based, reference-free, and text-overlap metrics across model scales. It consistently outperforms RL baselines, with the largest reported gains at 7B and strong results at 3B.

Overall results: RubiCap consistently outperforms all RL baselines across both datasets and all model scales, with the largest gains at 7B.The reported evaluation covers Tables 2, 3, and 4.
CapArena: 70.80% CapArena win rate against the base model (+20.80) is reported for RubiCap at 7B.The same results passage reports average gains of +6.05 on PixMoCap and +4.80 on DenseFusion.
Scale comparison: At 3B, RubiCap maintains gains of +4.62 and +2.68 on average while CapRL-3B reaches a SPECS score of 0.00 on both datasets.The passage also reports Reference-Likert failures at 3B and 2B, with averages of 7.33 and 11.08.

H VLM Benchmark Results

The study supplements captioning evaluation with VLM benchmarks at the 3B and 2B scales to measure retention of pretrained capabilities after fine-tuning. Results are reported in separate tables for each scale.

Reported results: The supplementary VLM results are reported separately for RubiCap-3B and RubiCap-2B.Table 5 contains the 3B results and Table 6 contains the 2B results.
Evaluation setup: VLM benchmark evaluation measures how well fine-tuned models retain the pretrained capabilities of their base models.All fine-tuned models are evaluated on 9 VLM benchmarks using VLMEvalKit.

I Qualitative Examples

Qualitative comparisons examine RubiCap-7B against both its base model and a larger 72B frontier model. The examples use GPT-4.1 judgments and cover PixMoCap and DenseFusion variants.

Comparison setup: Qualitative examples compare RubiCap-7B with Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-72B-Instruct on samples from both datasets.A GPT-4.1 judge selects the preferred caption and displays its reasoning in each caption block.
PixMoCap examples: Figure 7 presents a qualitative comparison to the 72B frontier using RubiCap-7B-PixMoCap.The figure caption identifies the PixMoCap variant.
DenseFusion examples: Figures 8 and 9 show DenseFusion comparisons against the 72B frontier and the base model, respectively.Both figures use RubiCap-7B-DenseFusion.