PaperlayerView exact revision on arXiv

Source-linked AI summary

ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison

Tianle Li, Xuyang Shen, Yan Ma, Rongxin Guo, Shaoxiang Chen, Jiacheng Chen, Haochen Wang, Hongyang Tang, Yucong Zhou, Yu Cheng

arXiv:2605.20278v2cs.LGcs.AIcs.CV

TL;DR

Long-form caption RL struggles to reward individual visual claims while balancing factuality and coverage. ClaimDiff-RL uses image-verified actor–reference claim differences as reward units, yielding more controllable tradeoffs and preserving or improving downstream capability across evaluations.

  • Problem

    Long-form captioning needs rewards that balance avoiding unsupported visual claims with covering salient image content, beyond holistic or reference-similarity objectives.

  • Method

    ClaimDiff-RL verifies actor–reference visual differences against the image, assigns typed errors, and composes their statistics into scalar RL rewards.

  • Results

    Across diagnostic, public captioning, and VQA evaluations, ClaimDiff-RL improves hallucination–missing-fact tradeoffs, preserves or improves capability, and surpasses Gemini-3-Pro-Preview on selected fine-grained dimensions.

  • Takeaways & Limitations

    Typed, verifiable claim differences provide a more controllable and diagnosable reward unit for long-form caption RL.

  • Takeaways & Limitations

    ClaimDiff-RL depends on strong multimodal judges, whose individual claim-level judgments can remain noisy despite aggregate reliability.

Abstract

from arXiv · show

Long-form image captioning exposes a reward granularity problem in RL: captions are judged as whole sequences, while the important errors occur at the level of individual visual claims. A good dense caption should be both faithful and informative, avoiding hallucination without omitting salient details. Yet pairwise preferences, reference-based metrics, and holistic scalar rewards compress these local errors into a single sequence-level signal, obscuring the tradeoff between factuality and coverage. We introduce ClaimDiff-RL, a framework that uses reference-conditioned atomic claim differences as the reward unit for caption RL. Given an image, an actor caption, and a reference caption, a multimodal judge enumerates visually grounded differences, verifies each difference against the image, assigns open-vocabulary error types and severity levels, and produces per-difference statistics for reward composition. This makes hallucinated claims and omitted salient facts separately measurable and tunable. Experiments show that holistic scalar rewards can reduce hallucination by increasing missing facts, while ClaimDiff-RL exposes this faithfulness and coverage tradeoff and enables more balanced operating points. On a 160-image human-labeled diagnostic benchmark, public captioning benchmarks, and VQA benchmarks, ClaimDiff-RL improves the hallucination--missing-fact balance, preserves general capability, and even surpasses Gemini-3-Pro-Preview on several fine-grained Capability dimensions such as object counting, spatial relations, and scene recognition. These results suggest that typed, verifiable claim differences are an effective reward unit for fine-grained and diagnosable caption RL.

1 Introduction

Long-form caption RL needs rewards that distinguish local visual errors because dense captions must balance faithfulness with coverage. ClaimDiff-RL addresses this by verifying typed actor–reference claim differences against the image, enabling controllable hallucination–coverage tradeoffs while preserving captioning and VQA capability.

  • Motivation: Dense captions contain many local visual claims, making sequence-level rewards too coarse for distinguishing errors in objects, attributes, counts, relations, text, identities, and scene details.The introduction frames this as a reward granularity problem for open-ended generation.
  • Motivation: A good dense caption must avoid unsupported visual claims while still covering salient image content.The desired objective is explicitly both faithful and informative.
  • Problem: Holistic scalar judging obscures whether a caption’s score reflects hallucinated claims, omitted salient facts, or other quality differences.Pairwise preferences, RLHF-style methods, and LLM- or MLLM-based evaluators generally score complete outputs or produce scalar judgments.
  • Method: ClaimDiff-RL changes the reward unit from holistic caption scores to image-verified claim differences while retaining standard scalar-reward optimization.A multimodal judge compares actor and reference captions, verifies each difference against the image, assigns side-specific typed errors, and composes their statistics into rewards.
  • Results: Relative and actor-only reward compositions expose different operating points on the faithfulness–coverage frontier, reducing hallucination without simply increasing omissions.The framework is reported to provide more controllable tradeoffs while preserving or improving captioning and VQA capability.

2 Related Work

Prior captioning work spans scalable reference-based evaluation, increasingly fine-grained diagnosis of local visual claims, and decomposed reward construction for reinforcement learning. ClaimDiff-RL is situated within the decomposed-reward direction while differing from fixed-schema parsing.

  • Automatic metrics for image captioning: Reference-based metrics including BLEU, METEOR, CIDEr, and SPICE provide scalable evaluation and optimization signals but poorly fit long-form dense captioning.Dense captions can vary validly in wording, order, length, and detail, making reference-based matching less suitable.
  • Fine-grained diagnosis of caption quality: Recent evaluation treats caption quality as local visual claims, with benchmarks measuring unsupported content and attribute- or question-based capabilities.Examples include CHAIR, POPE, HallusionBench, MMHal-Bench, DLC-Bench, GAR-Bench, Capability, and CaptionQA.
  • Reward construction for caption RL: Caption reinforcement learning progressed from self-critical optimization of CIDEr toward verifiable scalar and decomposed rewards based on downstream QA or parsed visual entities.CapRL uses downstream QA utility, while SC-Captioner decomposes rewards from parsed object, attribute, and relation sets; ClaimDiff-RL replaces fixed-schema parsing.

3 Method: Claim-Difference Rewards for Caption RL

ClaimDiff-RL makes typed, image-verified actor–reference claim differences the basis for scalar caption rewards. It supports relative and actor-only compositions, severity weighting, ambiguity control, and standard group-normalized RL optimization.

  • Judge and claim differences: The judge contrasts actor and reference captions to propose visual differences, verifies each against the image, and assigns side-specific typed errors.The reference serves as a proposal mechanism rather than exhaustive ground truth; image verification determines which claims are supported.
  • Reward composition: The relative reward compares actor- and reference-side weighted errors, whereas the actor-only reward penalizes only actor-side errors on discovered differences.Relative rewards favor the actor when it has fewer or less severe errors than the reference; actor-only rewards avoid benefiting from reference-side errors.
  • Judge and claim differences: Each difference records a visual aspect, both captions’ claims, an image-grounded judgment, and side-specific error descriptions with optional severity.Error types are open-vocabulary and can express compound categories such as color_hallucination, count_mismatch, and detail_omission.
  • Reward composition: Severity-weighted statistics assign larger penalties to more severe errors, allowing hallucinations or wrong counts to outweigh minor style or wording errors.Severity labels map through a monotone weighting scheme before reward normalization.
  • Reward composition: Zero-difference samples receive a neutral reward rather than maximum reward, preventing short or non-committal captions from becoming trivially high-reward examples.The method also applies a lightweight penalty for repeated ambiguity phrases while allowing a length-dependent free quota.
  • RL optimization: The composed scalar reward is used with group-normalized rewards from multiple sampled captions and can plug into standard scalar-reward RL pipelines.The method is not tied to a specific RL objective.

4 Experiments

Experiments evaluate whether CLAIMDIFF-RL improves the hallucination–coverage tradeoff while preserving public captioning and general multimodal ability. Results show that claim-difference rewards provide more controllable tradeoffs than holistic scalar rewards, with strong capability retention and tunable severity effects.

  • Hallucination–missing-fact tradeoff: Holistic rewards suppress hallucination but substantially increase missing facts, whereas CLAIMDIFF-RL provides more controllable faithfulness–coverage operating points.Direct scalar rewards can achieve high reward by producing shorter, conservative captions; CLAIMDIFF-RL maintains longer responses and exposes the tradeoff.
  • Public captioning capability: CLAIMDIFF-RL relative improves average public-captioning F1 from 69.5 for SFT to 71.5, while improving object category, object number, spatial relation, scene, and camera angle.It matches SFT on OCR, indicating that reward optimization can preserve or improve detailed captioning capability.
  • Public captioning capability: The actor-only variant remains near SFT at 69.3 average F1, while holistic scalar rewards degrade captioning capability, especially without a reference.The actor-only model improves number, color, spatial relation, and camera angle but drops on OCR and style.
  • General VQA capability: SFT lowers average VQA performance from 66.90 for Qwen3VL-32B-Instruct to 58.23, while RL partially mitigates this degradation.This evaluates whether caption-side optimization preserves broader multimodal understanding beyond captioning.
  • Severity weighting: Severity weights w = (1, 1.25, 1.6) reduce hallucination from 2.18 to 1.60 and MEDC from 0.92 to 0.52, while increasing missing facts to 0.76.The stronger setting w = (1, 1.5, 2) further lowers hallucination to 1.32 but increases missing facts to 0.92.
  • Judge reliability and consistency: Gemini achieves 87% per-claim accuracy in an audit of approximately 100 samples containing about 300 claim-level labels.This supports the reliability of typed claim-level judgments for aggregate diagnostic evaluation.

5 Conclusion

CLAIMDIFF-RL uses image-verified actor–reference claim differences as fine-grained reward units, separating hallucinations, missing facts, and correct extra details to diagnose faithfulness–coverage tradeoffs. Across diagnostic, captioning, and VQA evaluations, it preserves or improves downstream capability and surpasses Gemini-3-Pro-Preview on selected fine-grained dimensions.

  • Framework: CLAIMDIFF-RL derives scalar rewards from image-verified actor–reference claim differences rather than holistic caption scores.The framework separates hallucinated claims, missing facts, and correct extra details before composing rewards.
  • Empirical results: Across diagnostic, public captioning, and VQA evaluations, claim-difference rewards preserve or improve downstream capability versus SFT and holistic scalar optimization.
  • Empirical results: On Capability, CLAIMDIFF-RL surpasses Gemini-3-Pro-Preview on selected dimensions including object counting, spatial relations, and scene recognition.The result supports typed, verifiable claim differences as an effective reward unit for diagnosable multimodal RL.

A Limitations · B Broader Impact · C Training and Implementation Details

The framework’s limitations center on judge and reference dependence, reward hacking, evaluation breadth, and computational cost, while its broader impact includes accessibility benefits alongside risks from persuasive errors. Training uses a controlled GRPO setup with fixed rollout and optimization choices across reward variants.

  • A Limitations: ClaimDiff-RL depends on strong multimodal judges whose individual claim-level judgments can remain noisy despite aggregate reliability.Errors may stem from difficult visual evidence, OCR ambiguity, small objects, or subjective style descriptions.
  • A Limitations: Reference-conditioned comparison improves consistency but can miss visual aspects omitted by the reference or inherit its unusual details and style.The reference caption serves as a comparison anchor that narrows the judge’s search space.
  • A Limitations: Decomposed rewards do not eliminate reward hacking, including exploiting judge preferences, repeating safe details, hedging, or optimizing for prompt style.Judge ensembles, adversarial audits, stronger uncertainty handling, and periodic evaluations are proposed defenses.
  • A Limitations: The 160-image diagnostic benchmark enables controlled analysis but may underrepresent rare concepts, specialized OCR, medical imagery, scientific imagery, and culturally specific entities.Public captioning and VQA benchmarks complement the diagnostic evaluation, but broader coverage remains necessary.
  • A Limitations: ClaimDiff-RL costs more than direct scalar scoring because judges enumerate differences and assign side-specific typed errors.This added cost may limit scaling across very large training sets or many rollout samples.
  • B Broader Impact: ClaimDiff-RL may benefit accessibility, image retrieval, education, and dataset curation, but detailed incorrect captions can appear more persuasive in high-stakes domains.The framework also remains dependent on automatic judges and reference captions that may contain biases or omissions.
  • C Training and Implementation Details: RL training uses GRPO on 10K images from the SFT pool, with 4 × 8 H100 GPUs, global batch size 32, eight rollouts per prompt, and a 1 × 10−6 learning rate.Advantages use a within-group mean baseline, cosine decay is applied, and the vision tower remains frozen while the LLM and projection layers update.
  • C Training and Implementation Details: All reward variants share data, initialization, rollout, and optimization settings; step 200 is selected because performance degrades with further training.This controlled setup makes reward design the primary experimental variable.

C.1 Ambiguity penalty implementation … E.4 Deterministic mapping to hallucination and missing facts

The paper specifies an ambiguity-aware reward safeguard, complete claim-difference and holistic reward prompts, and a fixed diagnostic pipeline that separates visually unsupported claims from omitted salient facts. The diagnostic judge compares reference and candidate captions, verifies candidate claims against the image, and applies deterministic hallucination and missing-fact rules.

  • C.1 Ambiguity penalty implementation: C.1 Ambiguity penalty implementation: Applied after reward composition and before advantage normalization, the multiplicative ambiguity penalty reweights rollouts without changing the CLAIMDIFF-RL reward family.The fixed decay coefficient is c = 0.1, and the monotonic penalty does not invert claim-level reward gradients.
  • C.1 Ambiguity penalty implementation: C.1 Ambiguity penalty implementation: The penalty detects hedges, approximators, visual disjunctions, and negated-certainty phrases in actor captions.Matches are counted from fixed regex patterns, including constructions such as “possibly,” “about,” “X or Y,” and “not clearly.”
  • C.1 Ambiguity penalty implementation: C.1 Ambiguity penalty implementation: A length-proportional quota allows one free hedge per 90 words, typically permitting 1–3 hedges before penalization.Only max(0, namb − nfree) excess matches enter the exponent.
  • D.1 ClaimDiff-RL reward prompt: D Reward Prompts: The claim-difference judge compares an actor caption with a Gemini-3-Pro reference, enumerates concrete differences, verifies support from the image, and assigns typed errors and severity levels.The prompt prioritizes hallucination or incorrect claims over omission or incomplete information, while using fine-grained error types such as count_mismatch and spatial_relation_error.
  • D.2 Holistic scalar reward prompts: D Reward Prompts: Holistic baselines score captions from 0–10 and convert the result to SCORE/10, with variants using or omitting a reference caption.Both prompts assess factual correctness, salient-content coverage, attributes, counts, spatial relations, OCR/text, identities, hallucination avoidance, and clarity.
  • E.1 Image set and references: E.1 Image set and references: The diagnostic benchmark fixes N=160 images with human-written captions and measures hallucinated visual claims alongside missing salient facts.The same balanced set is used across checkpoints, making differences attributable to model behavior rather than evaluation-set sampling.
  • E.2 Candidate caption generation prompt: E.2 Candidate caption generation prompt: Each model produces overview and detailed English descriptions, with Detailed_Description used as the candidate caption for evaluation.The prompt emphasizes visually observable subjects, quantities, colors, spatial relations, OCR/text, scene details, style, camera information, and other salient elements.
  • E.3 Vision judge and prompt schema: E.3 Vision judge and prompt schema: GEMINI-3-PRO-PREVIEW detects contradiction, extra_info, and missing_fact differences, then verifies candidate claims against the image.The two-stage design accepts image-supported details absent from the reference, counts unsupported candidate details as hallucinations, and tags differences as natural or design.

E.5 Metrics

The metrics separately measure hallucinated candidate claims and missing facts, using benchmark-level means over 160 images. Their joint behavior exposes the faithfulness–coverage tradeoff, while discouraging both verbose unsupported and overly short captions.

  • Hall counts candidate-side claims identified as hallucinations, while Miss counts missing-fact items.
  • Benchmark-level metrics are computed as simple means over N=160 images.
  • Hall and Miss jointly reveal the faithfulness–coverage tradeoff: reducing Hall by saying less typically increases Miss.Figure 3 reports both quantities across training steps.
  • Hall is not length-normalized, penalizing verbose captions with more unsupported claims, while Miss prevents overly short captions from appearing artificially strong.

E.6 Why this benchmark is needed

This diagnostic benchmark is needed because public captioning and VQA evaluations do not separately measure hallucinated unsupported details and omitted salient facts. It isolates this tradeoff, showing holistic scalar rewards can reduce hallucination while increasing missing facts, whereas CLAIMDIFF-RL offers more controllable operating points.

  • Benchmark motivation: Public captioning and VQA benchmarks assess capability and multimodal understanding but do not separately measure hallucination and omission in captions.Captioning evaluations cover object, number, colour, spatial relation, scene, camera angle, OCR, and style; VQA evaluates broader multimodal understanding.
  • Benchmark motivation: The diagnostic benchmark isolates the faithfulness–coverage tradeoff between hallucinating unsupported details and omitting salient facts.It is designed specifically to distinguish these two caption-specific failure modes.
  • Reward tradeoff: Holistic scalar rewards can reduce hallucination aggressively but often increase missing facts, indicating conservative under-captioning.This demonstrates that reducing one failure mode can worsen coverage.
  • Reward tradeoff: CLAIMDIFF-RL exposes more controllable operating points: relative reward prioritizes coverage, while actor-only reward is more hallucination-averse.Relative reward tends to keep missing facts low; actor-only reward reduces actor-side hallucination without a large missing-fact increase.

F Judge Reliability and Consistency

Claim-level judgments are reasonably reliable and become more consistent across judges when conditioned on a reference caption. Agreement is higher for the SFT baseline than for RL-trained models, suggesting RL leaves subtler errors that warrant aggregate and multiple diagnostic metrics.

  • Human audit: 87% per-claim accuracy was achieved by Gemini-3-Pro-preview in a human expert audit covering approximately 300 hallucination and missing-fact labels.Three human experts audited approximately 100 samples, verifying each label against the image and ground-truth caption.
  • Cross-judge consistency: Reference-conditioned judging increased GPT–Gemini hallucination correlation from 0.377 to 0.537 and missing-fact correlation from 0.284 to 0.334.The reference caption provides shared visual comparison axes, while correctness remains decided against the image.
  • Per-model consistency: The SFT baseline reached ρhall=0.651 with reference and 0.419 without reference, while RL-trained models showed lower judge agreement.The reported pattern suggests evaluation difficulty contributes to agreement: SFT errors are more obvious, whereas RL errors are subtler.
  • Takeaways: The reliability and consistency analyses motivate reporting aggregate trends and multiple diagnostic metrics instead of relying on single-sample judgments.This conclusion follows from the observed human-audit reliability, stronger reference-conditioned agreement, and lower agreement on RL-trained models.

G Ambiguity Penalty Ablation

The ambiguity-penalty ablation tests CLAIMDIFF-RL relative with default severity weights and shows that removing the penalty increases hedging ambiguity. This supports the penalty’s role as a targeted safeguard rather than a separate reward objective.

  • The ablation evaluates CLAIMDIFF-RL relative using default severity weights w = (1, 1.25, 1.6).
  • Ambiguity rate is computed as matched ambiguity or hedging expressions normalized by generated words.
  • Removing the ambiguity penalty increases ambiguity rate from 0.89% to 1.75%, confirming its targeted safeguard against hedging.

H Code and Data Availability

An anonymized supplementary repository provides the materials needed to reproduce the reported evaluations and judge-based pipeline. After acceptance, the authors plan to release the code, data, and trained checkpoints publicly.

  • Code and Data Availability: The anonymized repository includes evaluation scripts, reward parsing code, prompt templates, diagnostic benchmark annotations, reported model outputs, and judge-evaluation instructions.These materials support reproducing the paper’s tables, figures, and judge-based evaluation pipeline.
  • Code and Data Availability: After acceptance, the authors plan to publicly release the code, data, and trained checkpoints.
Loading 2605.20278v2…