PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models

Xingming Long, Jie Zhang, Yuecong Min, Shiguang Shan, Xilin Chen

arXiv:2609.35294v1cs.CV

TL;DR

On-policy hallucination mitigation can reduce hallucinations by saying less when sparse rewards and response-level credit leave valid claims unrewarded or penalized. The paper pairs DOPA’s dense object labels with SCAPO’s subsentence-level credit assignment. It reports faithful, informative descriptions and gains on discriminative evaluation, while noting vocabulary coverage and annotation-cost boundaries.

  • Problem

    On-policy optimization can reduce hallucinations by making fewer valid claims when sparse annotations and response-level advantages impede precise error correction.

    Source: §1 · p. 2

  • Method

    The framework pairs DOPA’s exhaustive presence-and-absence annotations across an expanded vocabulary with SCAPO’s independent subsentence-level credit assignment.

    Source: §1 · p. 2

  • Results

    The framework overcomes the faithful-informative trade-off across hallucination benchmarks and transfers generative gains to downstream discriminative tasks.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Faithful, informative descriptions can also serve as effective auxiliary context for downstream discriminative tasks.

    Source: §6 · p. 10

    The predefined vocabulary leaves concepts outside V without explicit evaluation and limited training signal, while dense annotations add offline construction and verification cost.

    Source: G LIMITATIONS · pp. 21–22

Abstract

from arXiv · show

Object hallucination remains a major challenge for large vision-language models. While off-policy preference optimization proves to be an effective solution, on-policy reinforcement learning provides a more promising direction as it directly targets a model's current failure modes. However, we find that without fine-grained reward formulation and allocation, on-policy optimization often falls into an easy shortcut: reducing hallucinations merely by saying less---making fewer valid claims. To comprehensively resolve this, we propose a fine-grained alignment framework that couples dense reward signals at the data level with precise credit assignment at the algorithmic level. Specifically, we first construct the Dense Object Presence and Absence (DOPA) dataset to address sparse annotations that prevent valid object claims from being verified and rewarded. DOPA exhaustively annotates the deterministic presence and absence of every concept across an expanded vocabulary, significantly increasing the density of reliable reward signals during on-policy rollouts. Second, we propose Subsentence-level Credit Assignment for on-Policy Optimization (SCAPO) to prevent response-level shared advantages from allowing local hallucinations to compromise all other valid outputs within the same response. By assigning credit to each subsentence independently based on its object claims, SCAPO can precisely reinforce faithful generations and penalize hallucinations. Furthermore, we leverage the resulting faithful image descriptions as auxiliary context to transfer generative gains to discriminative tasks. Experiments demonstrate that our method produces highly informative, faithful descriptions in generative tasks while yielding clear performance gains on discriminative evaluation.

1 INTRODUCTION

On-policy hallucination mitigation can reduce unsupported claims by also suppressing valid ones when rewards are sparse and credit is assigned across whole responses. The paper combines dense object annotations with subsentence-level credit assignment, and reports descriptions that are both faithful and informative.

1 INTRODUCTION

  • On-policy optimization can reduce hallucinations by making fewer valid claims when sparse annotations leave claims unrewarded and response-level advantages let local hallucinations compromise valid outputs.This encourages global conservatism rather than precise error correction.Source: §1 · p. 2
  • DOPA exhaustively annotates concept presence and absence across an expanded vocabulary, providing dense, reliable reward signals for object claims during on-policy rollouts.This addresses sparse annotations that otherwise leave valid claims unverified and unrewarded.Source: §1 · p. 2
  • SCAPO assigns credit independently to each subsentence based on its claims, preventing local hallucinations from suppressing other valid outputs.It addresses the response-level shared-advantage bottleneck.Source: §1 · p. 3
  • The framework overcomes the “saying less” shortcut, producing descriptions that are both faithful and informative.It also reports that these descriptions provide useful auxiliary context for discriminative tasks.Source: §1 · p. 3

2 RELATED WORK

Related work spans preference optimization, inference-time mitigation, and training-based approaches. The paper motivates finer-grained on-policy credit assignment and combines it with dense object-existence annotations.

2.1 RLHF AND PREFERENCE OPTIMIZATION

  • GRPO-style methods typically assign response-level rewards, which can be limiting when hallucinations form only a small portion of an otherwise faithful response.The paper proposes selectively penalizing hallucinations while reinforcing correct content.Source: §2.1 · p. 3

2.2 INFERENCE-TIME HALLUCINATION MITIGATION

2.3 TRAINING-BASED HALLUCINATION MITIGATION

  • The paper presents DOPA as a reusable, high-density reward oracle and SCAPO as subsentence-level credit assignment for on-policy rollouts, addressing sparse supervision and costly judge-based evaluation.The cited work describes on-policy training as targeting evolving policy failure modes (Yu et al., 2025b; Yang et al., 2025).Source: §2.3 · p. 3

3 CONSTRUCTION OF DOPA DATASET

DOPA is constructed to provide dense and reliable rewards for on-policy object claims. Its pipeline builds a rollout-informed vocabulary, assigns explicit presence or absence labels, and refines annotations through edge-case review and manual verification.

3 CONSTRUCTION OF DOPA DATASET

  • DOPA is built on 5,000 MS COCO training images disjoint from evaluation images, using vocabulary construction, coarse annotation, edge-case refinement, and manual verification.The pipeline ends with a high-fidelity object-existence matrix E∗.Source: §3 · p. 4, Figure 2 · p. 4

3.1 VOCABULARY CONSTRUCTION

  • The rollout-derived vocabulary retains canonical labels based on frequency, visual verifiability, and semantic clarity, with aliases for synonyms and morphological variations.It contains 160 canonical labels and 894 aliases.Source: §3.1 · p. 4

3.2 COARSE ANNOTATION

  • DOPA defines a closed-set label space in which every image-label pair is explicitly marked present or absent, removing the unknown state for in-vocabulary claims.A judge model creates the initial matrix using visual evidence and strict criteria for occlusion and commonsense inference.Source: §3.2 · p. 4

3.3 EDGE-CASE REFINEMENT & MANUAL VERIFICATION

  • High-risk edge cases are re-annotated with label-specific criteria, then checked through repeated random samples of 30 image-label pairs per label until no errors appear in a new sample.The finalized matrix serves as a reusable reward oracle that requires deterministic lookups rather than repeated judge-model calls.Source: §3.3 · pp. 4–5

4 SCAPO METHOD

SCAPO assigns object-claim-based rewards to individual subsentences instead of relying on response-level shared advantages. It also uses generated grounded descriptions as context for discriminative answers.

4.1 SUBSENTENCE-LEVEL REWARD

  • Each subsentence is scored using counts of newly covered supported objects, repeated supported objects, and hallucinated objects.Source: §4.1 · p. 5
  • Hallucinations trigger penalties; hallucination-free subsentences earn rewards for novel claims and marginally for repetition, while object-free subsentences incur regularization.Source: §4.1 · pp. 5–6

4.2 SUBSENTENCE-LEVEL ON-POLICY RL TRAINING

  • SCAPO broadcasts each subsentence’s reward as the advantage for its tokens, then optimizes a clipped objective with KL regularization toward a fixed reference policy.Source: §4.2 · p. 6
  • Unlike PPO-based RLHF, SCAPO needs no auxiliary value model and does not propagate advantage signals across subsentence boundaries.Source: §4.2 · p. 6

4.3 DESCRIPTION-AUGMENTED DISCRIMINATIVE INFERENCE

  • For discriminative queries, the optimized model first generates a grounded description, then conditions its answer on both that description and the original image.Source: §4.3 · p. 6

5 EXPERIMENTS

SCAPO achieves strong results across generative and discriminative evaluations. The experiments attribute gains to dense DOPA annotations and subsentence-level credit assignment, which preserve informative outputs while controlling hallucinations.

5.1 EXPERIMENTAL SETUP

  • Across MS COCO/AMBER generation and PhD/AMBER discrimination, methods use Qwen2.5-VL-7B; Cap. Score harmonically combines 1-Hal. Rate and Cover Rate.The evaluated methods include inference-time, off-policy, and on-policy approaches.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2 MAIN RESULTS

  • 82.8% Cap. Score on AMBER and 79.6% on MS COCO accompany SCAPO’s 87.4% AMBER discriminative accuracy and best results across four PhD tasks.SCAPO outperforms alternatives across generative and discriminative settings.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 EXPERIMENTS ON DOPA

  • Replacing sparse MS COCO labels with DOPA reduced Hal. Rate from 11.8% to 9.4% and raised Cover Rate from 50.5% to 69.6%, with better AMBER/MMHal-Bench generalization.Both variants use the same SCAPO objective and differ in their annotation source.Source: Table 2 · p. 7, §5.3 · pp. 7–8
  • DOPA nearly doubles average annotated objects versus MS COCO annotations (5.13 vs. 2.73) and increases reward-bearing subsentence labels.The annotation-density analysis samples eight responses per training example and reports per-response averages.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 ANALYSIS OF SCAPO

  • Response-level Hal. Rate optimization cuts hallucinations to 4% but collapses Cover Rate, while Cap. Score optimization raises Hal. Rate to 17%; SCAPO expands coverage while controlling hallucinations.The object-level variant produces unstable increases in both metrics.Source: Figure 4 · p. 8, §5.4 · p. 8
  • Across four settings, GRPO-style methods trade facts for faithfulness; in detailed outputs, GRPO and DAPO yield 31.65 and 24.31 Avg. Facts versus 41.63 for Qwen2.5-VL, while SCAPO preserves both.FaithScore measures faithfulness, while Avg. Facts counts atomic facts per response.Source: Figure 5 · p. 9, §5.4 · p. 9

5.5 DESCRIPTION-AUGMENTED DISCRIMINATIVE INFERENCE

  • Description augmentation raises Qwen2.5-VL accuracy from 83.9% to 86.9% and F1 from 88.4% to 90.4%; the SCAPO backbone reaches 87.4% accuracy and 90.9% F1 without task-specific fine-tuning.The ablation progressively adds a generated description and then substitutes the SCAPO-aligned backbone.Source: Table 4 · p. 9, §5.5 · p. 9

6 CONCLUSION

The paper attributes the “saying less” shortcut to sparse rewards and coarse credit assignment. It presents DOPA and SCAPO as a framework that improves faithful, informative descriptions and transfers gains to discriminative tasks.

6 CONCLUSION

  • The conclusion attributes the “saying less” shortcut to sparse reward supervision and coarse response-level credit assignment.The paper frames these as the two bottlenecks addressed by its framework.Source: §6 · p. 9
  • DOPA provides a dense reward oracle, while SCAPO supplies precise subsentence-level credit assignment.Together, they form the paper’s dual-level fine-grained alignment framework.Source: §6 · pp. 9–10
  • Across hallucination benchmarks, the approach reduces unsupported content while expanding valid-object coverage, and its descriptions serve as auxiliary context for discriminative tasks.The conclusion describes these descriptions as faithful and informative.Source: §6 · p. 10

AI USE STATEMENT

The authors disclose using generative AI for dataset construction and routine assistance, while distinguishing those uses from hypothesis generation, method design, and scientific conclusions.

AI USE STATEMENT

  • GPT-5.5 drafted DOPA’s canonical object vocabulary, and Qwen3.6-27B generated initial and refined object-existence annotations.The paper identifies these as dataset-construction uses of generative AI.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI was not used to generate research hypotheses, design the method, or derive scientific conclusions.Source: AI USE STATEMENT · p. 10
  • AI-assisted text was reviewed, AI-assisted code was inspected and tested, and the authors take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper points to appendix materials supporting reproducibility, including method formalization, experimental configurations, and prompts.

REPRODUCIBILITY STATEMENT

A DOPA DATASET STATISTICS AND SPLIT DETAILS

DOPA supplies dense, exhaustive presence–absence annotations, while SCAPO uses those labels to assign object-claim rewards at the subsentence level. The appendix details dataset coverage, claim processing, optimization, and shared training configurations.

A DOPA DATASET STATISTICS AND SPLIT DETAILS

  • DOPA contains 5,000 training-side images annotated across 160 canonical labels, with 894 aliases and an average of 10.75 positive concepts per image.Its label frequencies are long-tailed: 76 labels (47.5%) appear in fewer than 100 images, while 14 (8.8%) appear in at least 1,000. Evaluation splits are kept separate, with independent annotations used only for evaluation.Source: A DOPA DATASET STATISTICS AND SPLIT… · p. 14, Table 5 · p. 14, Table 6 · p. 14

B.1 CLAIM SETS AND PROBLEM FORMULATION

  • SCAPO maps each subsentence’s object mentions to canonical labels and partitions in-vocabulary claims into supported and hallucinated sets using DOPA.Because every in-vocabulary image-label pair is present or absent, there is no unknown category; out-of-vocabulary entities are excluded from judgment and can send a subsentence to the no-objects branch.Source: §B.1 · pp. 14–15
  • A response-local history set separates novel from repeated supported claims, while the optimization target reduces hallucinated claims and discourages excessive repetition.The history is accumulated over supported claims and reset for each response, so the novelty bonus is response-specific.Source: §B.1 · p. 15

B.2 THE SCAPO OPTIMIZATION STEP

  • SCAPO uses deterministic vocabulary-and-alias lookup for dense per-subsentence rewards, decoupling rollout scoring latency from the judge model used to construct DOPA.On-policy baselines share the same applicable training configuration and differ in their advantage formulations.Source: §B.2 · p. 15, §C.1 · p. 15

C.2 EVALUATION METRIC DEFINITIONS

The appendix defines object-count metrics, describes how grounded descriptions are added to discriminative inference, and documents annotation and prompt choices. It also reports that SCAPO largely retains general multimodal benchmark performance.

C.2 EVALUATION METRIC DEFINITIONS

  • Hal. Rate measures faithfulness, Cover Rate measures informativeness, and both aggregate micro-averaged object counts; Caption Score combines them with a harmonic mean.The paper warns that low hallucination alone can reward saying less, so Cover Rate must be considered alongside Hal. Rate.Source: §C.2 · pp. 15–16

C.3 DESCRIPTION-AUGMENTED DISCRIMINATIVE INFERENCE

  • Description-augmented inference conditions a discriminative answer on both the original image and a SCAPO-generated description, using the same policy weights without task-specific fine-tuning.The description is auxiliary context, not a replacement for the image, preserving direct visual access to details it may omit.Source: §C.3 · p. 16

D.1 COARSE ANNOTATION PROMPT

  • The coarse verification prompt issues one binary visual-existence judgment for each image–label pair.Its instructions require the requested entity itself to be visibly identifiable rather than merely implied.Source: §D.1 · p. 17

D.2 LABEL-SPECIFIC VERIFICATION PROMPTS

  • Label-specific prompts clarify visual inclusion criteria, while joint verification contrasts confusing label pairs such as chair and bench.The prompts specify visibility requirements and distinctions from semantically adjacent categories.Source: §D.2 · p. 18

D.3 DESCRIPTION PROMPT pdesc

  • Training and generative evaluation use an unconstrained captioning instruction, whereas discriminative inference uses a prompt eliciting detailed, objectively visible content.This distinction reflects the description’s role as auxiliary context in the discriminative pipeline.Source: §D.3 · p. 20

D.4 DESCRIPTION-AUGMENTED DISCRIMINATIVE PROMPT

  • The discriminative prompt inserts the generated description while supplying the original image alongside the query.The image remains available as primary visual evidence.Source: §D.4 · p. 20

E.1 GENERAL CAPABILITY RETENTION

  • Six of seven general multimodal benchmark metrics remain stable or improve slightly after SCAPO alignment.The authors report that this avoids the alignment tax across the evaluated benchmarks.Source: Table 8 · p. 20, §E.1 · p. 20

E.2 ABLATION OF SCAPO COMPONENTS

The SCAPO reward-design ablations, an additional-backbone evaluation, and a qualitative case study examine complementary aspects of the method. The reported ablations favor the complete formulation, and the other evaluations describe transfer and a representative inference case.

E.2 ABLATION OF SCAPO COMPONENTS

  • The complete SCAPO formulation achieves the best overall score, while every reward-design ablation degrades at least one metric.The variants alter the hallucination penalty, repeated-claim reward, or no-object regularization while holding DOPA annotations and training configuration constant.Source: §E.2 · p. 21

E.3 GENERALIZATION ACROSS BACKBONES

  • SCAPO consistently enhances the Qwen3-VL backbone, supporting transfer of the approach beyond the primary backbone.The comparison is reported in Table 10.Source: §E.3 · p. 21

F CASE STUDY

  • In the representative small-background-people case, SCAPO’s description captures distant people and supports a correct query answer, unlike the baseline description that misses them.The baseline’s missing visual context is associated with a false negative in the subsequent prediction.Source: F CASE STUDY · p. 21

G LIMITATIONS

The framework’s current coverage is bounded by its predefined object vocabulary, and dense presence–absence annotation carries offline construction costs. The authors identify vocabulary expansion and model-assisted annotation as possible extensions.

G LIMITATIONS

  • Concepts outside the predefined vocabulary are not explicitly evaluated and therefore receive limited training signal.The authors suggest progressively expanding the vocabulary with newly observed model rollouts.Source: G LIMITATIONS · pp. 21–22
  • Dense presence–absence annotation adds offline verification costs, particularly for larger vocabularies or new visual domains.The overhead is confined to dataset construction rather than inference.Source: G LIMITATIONS · p. 22
  • Model-assisted annotation and iterative vocabulary expansion are proposed as ways to reduce annotation overhead and improve coverage.These are presented as future directions.Source: G LIMITATIONS · p. 22

Related papers

Loading 2609.35294v1…