PaperlayerView exact revision on arXiv

Source-linked AI summary

BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Shaokai Ye, Vasileios Saveris, Yihao Qian, Jiaming Hu, Elmira Amirloo, Peter Grasch

arXiv:2605.07394v1cs.CVcs.AI

TL;DR

Existing captioning-RL methods often optimize a narrow notion of caption quality, creating trade-offs among utility, reference coverage, and linguistic quality. BalCapRL jointly optimizes these dimensions with continuous-reward c-GDPO and length-conditional reward masking. Across multiple base models and benchmarks, it consistently improves caption quality, including gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena.

  • Problem

    Existing captioning-RL methods often optimize a narrow notion of caption quality, while gains on one dimension can regress or moderately improve others.

  • Method

    BalCapRL jointly optimizes utility-aware correctness, reference coverage, and linguistic quality using c-GDPO and length-conditional reward masking.

  • Results

    +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena were achieved by BalCapRL over the baseline with LLaVA-1.5-7B, while improvements were consistent across models and three benchmark views.

  • Takeaways & Limitations

    The results support balancing multiple caption-quality dimensions rather than optimizing toward a single benchmark or view.

  • Takeaways & Limitations

    The reward design may undervalue reasonable but non-pointable inferences, with preservation depending indirectly on reference-caption quality and coverage.

Abstract

from arXiv · show

Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that improve downstream question answering while harming fluency, whereas arena-style objectives can favor fluent but generic descriptions with limited usefulness. To address this, we propose a more balanced RL framework that jointly optimizes utility-aware correctness, reference coverage, and linguistic quality. In order to effectively optimize the resulting continuous multi-objective reward formulation, we apply GDPO-style reward-decoupled normalization to continuous-valued captioning rewards and show that it improves performance over vanilla GRPO. Additionally, we introduce length-conditional reward masking, yielding a more suitable length penalty for captioning. Across LLaVA-1.5-7B and Qwen2.5-VL 3B and 7B base models, our method consistently improves caption quality, with peak gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across different models.

1 Introduction

Recent MLLM advances and captioning-focused reinforcement learning aim to produce more detailed captions, but existing methods often optimize only one dimension of caption quality. BalCapRL addresses these trade-offs with a balanced objective spanning utility-aware correctness, reference coverage, and linguistic quality.

  • MLLM advances have enabled increasingly open-ended and detailed image captions.
  • Captioning-RL has increasingly gained popularity as a way to maximize the captioning capabilities of modern MLLMs.
  • Existing captioning-RL methods often optimize a narrow notion of caption quality, causing gains on one dimension to regress or only moderately improve others.
  • The main evaluation views are downstream utility, reference-based correctness and completeness, and arena-style preference judgments.
  • BalCapRL jointly optimizes utility-aware correctness, reference-coverage completeness, and linguistic quality to provide a more balanced captioning-RL framework.
  • BalCapRL consistently improves caption quality across benchmarks and models representing all three evaluation views, outperforming prior methods in almost all settings.

2 Method

BalCapRL decomposes captions into atomic assertions and jointly rewards verifiable, pointable correctness, reference coverage, and linguistic quality. It optimizes these continuous rewards with c-GDPO and uses length-conditional masking to avoid undesirable caption-length behavior.

  • Reward design: Generated and reference captions are decomposed into atomic assertions or reference units for reward computation.
  • Reward design: Precision rewards assertions only when they are visually verifiable and refer to visually pointable elements.
  • Reward design: Recall measures how much key information from the reference caption is mentioned or reasonably inferable from generated assertions.
  • Reward design: The linguistic reward averages normalized assessments of clarity, fluency, and coherency.
  • Policy Optimization: Vanilla GRPO can make distinct continuous reward trade-offs indistinguishable because it sums multiple rewards before group normalization.
  • Policy Optimization: c-GDPO normalizes each reward separately before weighted aggregation, preserving finer distinctions among reward combinations and producing more expressive training signals.
  • Policy Optimization: Length-conditional reward masking gates the linguistic reward using the predicted-to-reference length ratio rather than imposing a one-sided penalty.

3 Results

BalCapRL improves captioning across multiple quality views and model settings, while ablations show that balanced rewards, c-GDPO, pointability, reference quality, and length control each shape its behavior.

  • Main results: BalCapRL improves DCScore, CaptionQA, and CapArena across LLaVA-1.5-7B and QwenVL2.5 models, outperforming prior methods in almost all settings.The evaluation represents correctness-and-completeness, utility, and arena-style preferences.
  • Main results: 13.6 DCScore, 9.0 CaptionQA, and 29.0 CapArena points separate BalCapRL from the LLaVA-1.5-7B baseline.BalCapRL also exceeds FEEDQUILL by 2.1 points on DCScore.
  • Main results: BalCapRL outperforms CapRL-3B on DCScore and CapArena, while CapRL-3B scores higher on CaptionQA and produces captions roughly 3× longer than its base policy.The CapRL comparison associates excessive length with degraded fluency and a 16.6-point CapArena regression versus its base model.
  • Main results: BalCapRL outperforms RubiCap-3B on all metrics and significantly exceeds RubiCap-7B across evaluated benchmarks when using the same QwenVL2.5-7B base model.It approaches RubiCap-7B on DCScore and CaptionQA in the 3B comparison.
  • General vision benchmarks: BalCapRL has no notable regression on ten tested general vision benchmarks while improving several, unlike SFT and some prior RL models.The comparison notes regressions for SFT, RubiCap, and CapRL, including CapRL losses in TextVQA and DocVQA.
  • Ablation studies: Removing c-GDPO causes substantial degradation across benchmarks, supporting separate normalization for multiple rewards with distinct dynamics.Vanilla GRPO sums the three rewards before group normalization, whereas c-GDPO normalizes each reward separately.
  • Ablation studies: Removing precision preserves the large CapArena gain but lowers DCScore, whereas removing linguistic reward raises CaptionQA and DCScore but substantially lowers CapArena.These variants expose trade-offs among reward dimensions.
  • Ablation studies: Removing pointability hurts performance and permits fluent meta-commentary that lacks downstream utility, while replacing GPT-5-mini references with original captions causes consistent benchmark drops.The reference-caption ablation suggests higher-quality references provide a more effective recall signal.

4 Related work

Prior captioning-RL and evaluation methods emphasize different, partial views of caption quality. Related approaches span utility, correctness-and-completeness, and arena-style preference perspectives, motivating their combination.

  • Captioning-RL methods increasingly use reinforcement learning to improve MLLM image-captioning performance.
  • Utility metrics assess downstream text-only question answering, while correctness-and-completeness metrics assess caption faithfulness and coverage.
  • Arena metrics evaluate caption quality through pairwise competition and preference judgments.
  • Optimizing any single perspective is insufficient because correctness-and-coverage objectives, utility training, and arena judgments encourage different weaknesses.These include repetitive descriptions, hallucinated or overly long captions, and fluent but generic captions.
  • BalCapRL combines the three perspectives to reduce biases introduced by evaluating or optimizing caption quality separately.

5 Limitations

The framework does not explicitly model every aspect of caption quality. In particular, it may undervalue reasonable non-pointable inferences and incurs latency and cost trade-offs from using an MLLM judge.

  • Some caption-quality aspects are absent from the reward design, especially plausible world-knowledge inferences that are not directly visually pointable.Their preservation depends indirectly on the recall objective and the quality and coverage of reference captions.
  • Using an MLLM-as-judge simplifies the pipeline relative to FEEDQUILL but introduces latency and cost trade-offs.

A.1 Implementation Details

The implementation builds c-GDPO on verl, uses fixed rollout, optimization, and training settings, and evaluates captions with GPT-based decomposition and verification.

  • The codebase implements c-GDPO on top of verl.
  • Training uses 8 rollouts, a 5e-6 learning rate, batch size 256, cosine scheduling, and 1 epoch.
  • Main results use reward weights wpre = 0.1, wrec = 0.3, wling = 0.3, τl = 0.5, and τu = 2.
  • The roughly 90K-pair ShareGPT4V dataset uses GPT-4V captions and compares them with GPT-5-mini recaptions.
  • GPT-4o-mini decomposes on-policy captions into atomic assertions and labels their correctness, pointability, and reference coverage.
  • Removing KL divergence and applying dual-clip improved training stability, while token-sum-sequence-mean further improved empirical performance.

A.2 Training Prompt

The training prompt extracts atomic assertions, verifies them using pointability and visual truth, checks reference coverage, and scores linguistic quality before returning structured JSON.

  • The prompt extracts all shortest-possible factual claims from synthetic and ground-truth captions, including meta-commentary and subjective statements.
  • An assertion is verified only when it is pointable or evidentially justified and actually true in the image.The prompt marks assertions failing either condition as is_verified: false.
  • The pointability test accepts physical objects, visible attributes, spatial locations, and visible text, while rejecting abstractions, effects, judgments, emotions, relationships, and intentions.
  • Visual verification checks assertions against the ground-truth caption or image evidence and rejects hallucinated or factually incorrect claims.
  • Recall evaluates whether the synthetic caption covers ground-truth content without requiring exact wording.
  • Linguistic quality is scored independently for clarity, fluency, and coherency on a 1-to-10 scale.The prompt penalizes excessive length, redundancy, convoluted syntax, unnatural phrasing, and disjointed structure.
  • The evaluator returns assertion verification, coverage, linguistic scores, and explanations in a prescribed JSON-only format.

A.3 Full Proof

The proof shows that vanilla GRPO collapses continuous reward vectors to their weighted aggregate before normalization, discarding reward-trade-off information. Reward-decoupled normalization instead preserves per-dimension deviations, so equal aggregate rewards need not produce equal optimization signals.

  • Vanilla GRPO: Reward vectors on the same weighted-sum hyperplane are indistinguishable to vanilla GRPO.The optimizer therefore discards orthogonal reward-trade-off information.
  • Vanilla GRPO: Vanilla GRPO’s normalized advantage depends on a reward vector only through its aggregated weighted sum.This creates a many-to-one mapping from reward vectors to scalar normalized advantages.
  • Reward-decoupled normalization: c-GDPO normalizes each reward dimension separately before aggregating the normalized deviations.Each dimension’s normalization depends on its own coordinate and competing-rollout statistics.
  • Reward-decoupled normalization: Equal aggregate reward does not generally imply equal decoupled advantage because independently normalized reward coordinates can contribute differently.This difference is generically nonzero when at least two weighted reward dimensions have nondegenerate variation.
  • Implication: Reward-decoupled normalization retains finer-grained optimization information for continuous multi-reward trade-offs.Unlike vanilla GRPO, it does not collapse all reward trade-offs sharing an aggregate reward into one update signal.

A.4 Evaluation Prompt (b-CapScore)

The b-CapScore evaluation prompt verifies caption assertions using a two-part test: claims must be pointable or visually justified and must be true in the image. It permits ordinary visual inference but rejects hallucination and unsupported speculation.

  • Verification rubric: Each synthetic assertion is verified only when both pointability or justified visual evidence and image truth are satisfied.The rubric requires a person to point to the claim or to visible evidence supporting it through ordinary world knowledge.
  • Verification rubric: Directly visible facts such as objects, colors, positions, and readable signs pass the pointability test.Examples include a car, a red door, a silver hubcap, a left-side position, and a sign reading STOP.
  • Verification rubric: Visually justified inferences may use ordinary world knowledge when clear image evidence supports them.Examples include inferring wet roads from puddles or a queue from people aligned toward an entrance.
  • Failure cases: Unsupported speculation includes mind-reading, storytelling, symbolism, and weak social interpretations without strong visible evidence.The prompt explicitly rejects claims about intentions, emotions, family relationships, reunions, or luxury when the image does not strongly support them.
  • Truth checking: The image is the primary source of truth, while the ground-truth caption may assist but cannot by itself invalidate a visually supported claim.Assertions that are hallucinated, factually incorrect, or too ambiguous should not be verified.

A.5 A Balanced Captioning Metric (b-CapScore)

b-CapScore is a balanced captioning metric combining pointability-aware precision, reference coverage, and linguistic quality through a harmonic mean. It is designed to evaluate multiple caption-quality dimensions without relying on arena-style pairwise comparisons.

  • Metric definition: b-CapScore is defined as the harmonic mean of pointability-aware precision, reference coverage, and linguistic quality.The metric reuses images and human reference captions from DCScore as its data sources.
  • Metric definition: The harmonic mean penalizes imbalance across correctness, coverage, and linguistic quality.This supports a single balanced quality score while retaining the three component dimensions.
  • Metric components: Pointability-aware precision counts an atomic assertion as correct only when it is visually verified and pointable or supported by pointable visual evidence.The verification rubric is the same one used during training.
  • Metric components: b-CapScore evaluates reference coverage and linguistic quality alongside correctness.Reference coverage measures key information from the reference caption, while linguistic quality includes clarity, fluency, and coherency.
  • Human alignment: Model rankings from b-CapScore use average reference-based caption scores rather than arena-style pairwise comparisons.Table 5 compares its model-level Spearman alignment with CapArena’s arena-derived rankings.

A.6 Reward Weight Ablation Studies

The reward-weight ablation examines how changing the precision weight affects performance, while the appendix also specifies a length penalty based on deviation from an acceptable range. The supplied qualitative examples illustrate caption outputs but do not state comparative outcomes.

  • Reward-weight ablation: Setting wprec = 0 produces the worst overall performance in the reward-weight ablation.This result comes from varying wprec while keeping the other two reward weights fixed.
  • Reward-weight ablation: For non-zero wprec values, increasing the precision weight introduces trade-offs across benchmarks.The supplied passage does not specify the direction or magnitude of each benchmark trade-off.
  • Qualitative examples: The section includes additional qualitative examples involving pumpkins and statues with birds, but the supplied passages provide no stated comparison or outcome.These examples are descriptive rather than quantitative ablation findings.
  • Length penalty: The length penalty measures deviation from the acceptable range [τl, τu] and subtracts a linear penalty from the normalized advantage.λlen controls penalty strength, while A denotes the normalized advantage and à the penalized advantage used for optimization.
Loading 2605.07394v1…