PaperlayerView exact revision on arXiv

Source-linked AI summary

PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models

Jihyung Ko, Eunji Jung, Hyeongsub Kim, Ziseok Lee, Jae Won Cho, Sanghyun Jo, Kyungsu Kim

arXiv:2608.21819v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Image captioning needs precise, complete descriptions, but existing training-free methods mainly suppress likely hallucinated objects and struggle to recover omitted visible objects. PatchGate extracts prompt-free internal object evidence from a frozen VLM before decoding and uses it for two-sided logit calibration; on AMBER, it improves object-level reliability, including a 13.4% increase in coverage and 12.0% lower CHAIR.

  • Problem

    Image captioning requires avoiding unsupported object mentions while covering visible objects, but existing training-free methods are limited at recovering visible objects omitted during generation.

  • Method

    PatchGate uses VEX to construct a prompt-free, image-conditioned object inventory from internal patch representations, then VIED calibrates decoding logits to promote supported objects and suppress weakly supported mentions.

  • Results

    13.4% higher AMBER coverage and 12.0% lower CHAIR are reported for PatchGate, with gains across both object completeness and precision.

  • Takeaways & Limitations

    A frozen VLM’s own patch-level evidence can serve as a practical grounding signal for more reliable object verbalization without external detectors or fine-tuning.

  • Takeaways & Limitations

    PatchGate depends on the quality of VEX’s internal object inventory and does not directly address attribute, relation, counting, or higher-level scene hallucinations.

Abstract

from arXiv · show

Reliable image captioning in Vision-Language Models (VLMs) requires captions to be both precise and complete, avoiding unsupported object mentions while covering visible objects. Existing training-free methods primarily address the former requirement, suppressing unsupported object words by intervening on model-predicted mentions during generation. Because they operate only on objects the model is already likely to mention, visible objects omitted from the output remain difficult to recover. We propose PatchGate, a training-free framework that extracts prompt-free object evidence intrinsic to a frozen VLM before generation and uses it to narrow the gap between an intrinsic object set and final object mentions. In the first stage, Visual Evidence eXtraction (VEX) reads patch-level lexical evidence from the latter half of LM decoder layers and constructs an image-conditioned object set without any task prompt. In the second stage, Visual-Evidence Inclusion-Exclusion Decoding (VIED) uses this object evidence to calibrate decoding logits, promoting evidence-supported but under-verbalized objects and suppressing weakly supported but over-verbalized objects. On AMBER, PatchGate improves both sides of object-level reliability, increasing visible-object coverage from 49.4 to 56.0 (+13.4%) and reducing object hallucination by lowering CHAIR from 7.5 to 6.6 (-12.0%), without external detectors or fine-tuning and with one extra forward pass.

1 Introduction

Reliable VLM captions must be precise and complete, but generated captions can both hallucinate unsupported objects and omit visible ones. PatchGate addresses this mismatch by extracting prompt-free internal object evidence before generation and using it to calibrate decoding.

  • Caption reliability requires avoiding unsupported object mentions while covering visible objects in the image.
  • Existing training-free methods mainly suppress unsupported objects likely to appear during generation, limiting recovery of omitted visible objects.
  • PatchGate identifies an evidence-verbalization mismatch between prompt-free internal object evidence and the objects ultimately mentioned in captions.
  • VEX extracts patch-level lexical evidence from later decoder layers and constructs a prompt-free intrinsic object inventory before generation.
  • VIED calibrates decoding logits by promoting evidence-supported under-verbalized objects and suppressing weakly supported mentions.
  • 13.4% higher Cover and 12.0% lower CHAIR are reported for PatchGate on AMBER.

2 Related Work

Related work includes training-based and training-free approaches to improving VLM faithfulness, while internal representations offer another source of object evidence. PatchGate builds on these directions by using pre-generation patch evidence to address both hallucination and omission.

  • 2 Related Work: Training-based methods reduce hallucination through additional supervision or preference optimization but require model updates and extra training data.
  • 2 Related Work: Training-free methods use contrastive decoding, attention intervention, visual encoder correction, or external object guidance to mitigate hallucination.
  • 2 Related Work: Because many training-free methods act on objects likely to emerge during generation, they are limited in recovering visible objects that never surface.
  • 2 Related Work: Logit-lens analyses and VLM studies show that intermediate hidden states and patch representations can expose increasingly readable object information before final decoding.
  • 2 Related Work: PatchGate uses internal evidence comparatively rather than as absolute grounding alone, aligning pre-generation object evidence with final verbalization.

3 Method

PatchGate aligns a frozen VLM’s internal patch-level object evidence with caption generation through VEX and VIED. VEX builds a prompt-free inventory, while VIED promotes supported omissions and suppresses weakly supported object mentions.

  • Visual Evidence eXtraction (VEX): PatchGate extracts a prompt-free, image-conditioned object inventory from a frozen VLM before caption generation.VEX uses internal patch representations rather than prompts, external detectors, or fine-tuning.
  • Visual Evidence eXtraction (VEX): VEX reads top-1 lexical candidates by projecting patch hidden states into the output vocabulary across later decoder layers.For LLaVA-v1.5-7B, the default layer range is L = {22, . . . , 32}, with N = 576 visual patches.
  • Visual Evidence eXtraction (VEX): VEX scores each object by its strongest normalized evidence across selected patches and layers, preserving localized high-confidence candidates.This patch-confidence score supports both inventory selection and VIED calibration.
  • Visual Evidence eXtraction (VEX): VEX applies an image-adaptive cutoff, including a score floor and sorted-tail exclusion, to form the final intrinsic inventory.The cutoff uses the midpoint of the largest gap between sorted raw PCS values rather than a fixed top-k.
  • Visual-Evidence Inclusion-Exclusion Decoding (VIED): VIED calibrates decoding logits in two directions: ESI promotes supported inventory objects, while EDE suppresses weakly supported object words.ESI targets not-yet-generated inventory objects; EDE uses visual evidence rather than a hard inventory veto, and non-object tokens are unchanged.
  • Visual-Evidence Inclusion-Exclusion Decoding (VIED): EDE penalizes object words whose logits exceed a neutral median despite visual deficits, while stronger continuous evidence weakens suppression.This reduces reliance on the hard inventory boundary while targeting above-neutral object logits with weak visual evidence.

4 Experiments

Experiments on AMBER and POPE show that PatchGate improves object-level completeness and precision, diagnoses both omission and hallucination errors, and remains effective across backbones with modest computational overhead.

  • AMBER Results: PatchGate improves all four highlighted AMBER axes over the LLaVA-v1.5-7B baseline, unlike prior methods that trade off coverage, hallucination, precision, and recall.Cover rises from 49.4 to 56.0 (+13.4%), while CHAIR falls from 7.5 to 6.6 (−12.0%).
  • POPE Results: PatchGate improves POPE average accuracy, precision, and F1 while achieving second-best recall and stronger overall metric balance.Accuracy increases from 82.0 to 89.8 (+9.5%), precision from 88.5 to 90.5 (+2.3%), and F1 from 80.4 to 89.6 (+11.4%).
  • Qualitative Results: Qualitative results show two-sided correction: recovering omitted visible objects and suppressing hallucinated mentions in captioning, while correcting both types of POPE answer errors.These behaviors align with improvements in omission-related Cover and Recall and hallucination-related metrics.
  • Diagnostic Analysis: Pre-decoding VEX evidence identifies both hallucinated and omitted objects well above random baselines.AUROC reaches 0.883/0.890 versus 0.500, and AUPRC reaches 0.707/0.705 versus positive-rate baselines 0.163/0.233.
  • Inventory Ablation: VEX outperforms external object inventories under VIED while adding no parameters and only 0.3GB peak VRAM over the baseline.Cover increases from the best external result 51.3 to 56.0, and CHAIR decreases from 7.2 to 6.6.
  • Component Ablation: ESI primarily improves omission-related coverage, whereas EDE most directly suppresses hallucination; together they provide a balanced precision–coverage trade-off.ESI raises Cover from 49.4 to 54.8 (+10.9%), while EDE reduces CHAIR from 7.5 to 6.2 (−17.3%); the full model reaches Cover 56.0 and CHAIR 6.6.
  • Backbone Robustness: Across evaluated VLM scales and families, PatchGate improves Cover by +1.3% to +13.4% and reduces CHAIR by −6.8% to −12.3%.The results are reported across multiple backbones without fine-tuning or external detectors.
  • Efficiency: PatchGate increases runtime from 2.92s to 3.30s per image and peak VRAM from 14.6GB to 14.9GB on LLaVA-v1.5-7B.The method uses one additional VEX forward pass and lightweight VIED logit edits.

5 Conclusion

PatchGate uses a frozen VLM’s internal patch representations to extract a prompt-free object inventory before decoding, then applies inventory-conditioned correction to improve object reliability. Its vocabulary and adaptive selection procedures support evidence-based inclusion while retaining broad candidate coverage.

  • 5 Conclusion: PatchGate mitigates both object hallucinations and omissions in frozen VLMs through prompt-free evidence extraction and two-sided logit calibration.It promotes visually supported but under-verbalized objects and suppresses weakly supported mentions.
  • VEX: VEX constructs an intrinsic object inventory from patch-level lexical evidence in later decoder layers without task prompts.The inventory is image-conditioned and extracted before generation through an image-only forward pass.
  • VEX: The object vocabulary favors broad coverage, while downstream PCS scoring and adaptive thresholds select image-supported candidates for the final inventory.Candidates below an evidence floor are removed, and the largest eligible score gap determines the adaptive cutoff.
  • VIED: VIED adapts inventory use to the output space, directly editing object-token logits for captioning and guiding Yes/No logits for binary QA.Inventory-conditioned guidance can raise “Yes” for supported objects and shift toward “No” for unsupported objects.

A.2 PatchGate Algorithm

PatchGate’s algorithm first builds an image-only object inventory and then uses it during decoding to suppress unsupported mentions and recover omitted visible objects. Worked examples illustrate these operations for captioning and binary QA.

  • Stage 1 — VEX: VEX requires one additional image-only forward pass and no prompt or token decoding before constructing the object inventory.Task-specific prompts are introduced only during downstream inference.
  • Stage 2 — VIED: In captioning, VIED applies EDE to penalize weakly supported object logits and ESI to promote supported inventory objects during greedy decoding.The worked example reduces the cup logit from 6.8 to 4.3 and increases the omitted lamp logit from 16.1 to 17.2.
  • Binary QA: In binary QA, inventory-conditioned guidance corrects hallucinated affirmative answers by shifting the relative logits of “Yes” and “No”.For an absent ship, the inventory excludes the object and guidance yields the correct “No” answer.
  • Worked example: The captioning example selects an inventory of 24 objects after retaining strongly supported objects and excluding unsupported cup and couch mentions.The largest eligible interior gap occurs at θTTD = 0.57.

A.3 Experimental Details

Experiments evaluate PatchGate against frozen and training-free baselines on AMBER and POPE using official reliability metrics and deterministic inference settings. The evaluation also documents annotation sensitivities affecting Hal and Cog.

  • Benchmarks: PatchGate is evaluated on AMBER and POPE, covering generative captioning, object-existence QA, and broader attribute and relation questions.AMBER includes 1,004 free-form image-description samples and a discriminative binary QA task.
  • Metrics: Cover measures visible-object coverage, while CHAIR, Hal, and Cog measure different forms of unsupported object mentioning.Higher Cover is better; lower CHAIR, Hal, and Cog are better.
  • Metric limitations: Hal and Cog are sensitive to annotation inconsistencies, including cases where a visibly present object is placed in AMBER’s hallucinatory target set.The paper therefore treats Cover and CHAIR as primary complementary measures.
  • Metrics: AMBER and POPE use different positive-label conventions, so POPE Precision reflects resistance to hallucinated “Yes” answers while Recall reflects omission recovery.AMBER treats “No” as the positive label, whereas POPE treats “Yes” as positive.
  • Inference settings: PatchGate uses deterministic greedy decoding without sampling, temperature, top-p, beams, or a random seed.Experiments run on one NVIDIA RTX A6000 GPU with 48GB memory.

B.1 Additional AMBER Results

Across AMBER results, PatchGate improves complementary object-reliability dimensions rather than trading coverage against hallucination or QA performance. The gains remain consistent across backbones and produce the strongest aggregate improvement among evaluated methods.

  • Overall performance across backbones: PatchGate improves AMBER Cover from 49.4 to 56.0 (+13.4%) and reduces CHAIR from 7.5 to 6.6 (−12.0%) on LLaVA-v1.5-7B.The same trend appears on LLaVA-v1.5-13B and Qwen2.5-VL-7B.
  • Balanced improvement without metric trade-offs: PatchGate is the only evaluated method that simultaneously improves Cover, Precision, and Recall while reducing CHAIR relative to the frozen backbone.This indicates balanced gains across caption coverage, hallucination, and discriminative QA.
  • Aggregate improvement across primary metrics: 11.9 percentage points is PatchGate’s largest aggregate gain across Cover, CHAIR, Precision, and Recall, compared with 7.5 for OPERA.The analysis counts a CHAIR reduction as positive.

B.2 Additional POPE Results

PatchGate improves both precision and recall across all three POPE splits, addressing hallucinated and missed visible-object answers without trading one error type for the other.

  • PatchGate improves Precision by 2.6%, 1.1%, and 3.0% across the random, popular, and adversarial splits, respectively.
  • PatchGate improves Recall by 22.0%, 20.8%, and 19.3% across the random, popular, and adversarial splits, respectively.
  • PatchGate achieves the best Accuracy and F1 in every split.
  • PND has the highest Precision but Recall 7.6–9.6 points below PatchGate, while ProjectAway’s Precision is 8.4 and 12.3 points lower on popular and adversarial splits.

C Additional Qualitative Results

Additional AMBER and POPE examples show PatchGate correcting both unsupported object mentions and omissions, rather than reducing one error type by sacrificing the other.

  • PatchGate removes unsupported mentions while recovering visible content omitted by the frozen backbone, including sky, wooden floor, and bed frame.
  • PatchGate corrects hallucinated “Yes” answers on absent objects and incorrect “No” answers about visible content.
  • Suppression-oriented baselines can shorten captions, leave omissions untouched, and drop correctly mentioned objects.
  • Figure D marks unsupported object mentions in red and recovered visible objects in green.

D Details of Ablation Study

The ablations identify operating choices that balance omission recovery, hallucination suppression, linguistic quality, and computational cost on AMBER.

  • Effect of VIED hyperparameters: Increasing αesi from 2 to 16 raises Cover from 51.6 to 58.8, while CHAIR and Hal also increase; αesi = 8 is selected as a compromise.
  • Effect of VIED hyperparameters: Increasing γede from 0.25 to 1.0 reduces CHAIR from 6.8 to 6.5 and Hal from 36.4 to 32.4, but lowers Cover from 56.3 to 54.6.
  • Effect of VIED hyperparameters: At γede = 2.0, Cover decreases further and CHAIR rises to 7.0, so γede = 0.5 is used as a conservative operating point.
  • Binary-QA guidance strength: Increasing λQA from 0 to 0.7 improves Accuracy from 72.0 to 74.5 and F1 from 74.9 to 77.4, with smaller gains at larger values.
  • Computational efficiency: PatchGate uses 14.9 GB peak memory and 3.30 seconds inference time, remaining shorter than most competing approaches.
  • Computational efficiency: Relative to LLaVA-v1.5-7B, PatchGate adds 0.3 GB (2.1%) peak VRAM and 0.38 seconds (13.0%) inference time.
  • Evidence aggregation and adaptive object selection: Max-softmax aggregation better preserves localized object responses and separates supported objects from the low-evidence tail than mean aggregation.

P. ↑R. ↑F1 ↑|T| Cover ↑CHAIR ↓Hal ↓Cog ↓

The comparison examines internal and external object inventories as diagnostics and decoding inputs, showing that higher tag-level accuracy does not alone determine caption-level effectiveness or efficiency.

  • Mean aggregation retains about one object per image on average, yielding high tag Precision but very low Recall, low Cover, and high CHAIR.
  • Later-layer max aggregation increases tag Recall from 31.9 to 49.2 and tag F1 from 47.0 to 62.3 compared with single-layer max aggregation.
  • Restricting extraction to the later 33% of layers changes tag Recall from 49.6 to 49.2 while increasing tag Precision and F1 and reducing inventory size.
  • VEX achieves tag-level F1 of 62.3, only 2.6 points below GroundingDINO and 6.0–8.8 points below RAM++ variants.
  • External inventories remain viable, but higher tag-level accuracy alone does not guarantee more effective or efficient decoding intervention.
  • The diagnostic comparison evaluates hallucination on baseline-mentioned objects and omission on unmentioned objects using VEX and RAM++ signals.

E Limitations and Future Work

PatchGate’s current object-token inventory is strongest for object-centric tasks, while attribute and relation questions require structured evidence it does not yet capture. Coverage, fluency, and evaluation reliability also remain bounded by vocabulary, decoding, and annotation constraints.

  • Scope of the current inventory: PatchGate’s object-token inventory yields its largest gains in object-centric captioning and existence QA.Attribute and relation questions require structured evidence not explicitly captured by the current inventory.
  • Future work: Attribute and relation questions motivate extending the inventory to object–attribute pairs and object–relation–object triplets.Future decoding strategies would need to correspond to these structured evidence types.
  • Vocabulary coverage: The predefined single-token vocabulary may limit coverage of open-vocabulary or multi-token concepts.The paper motivates improved evidence calibration and integration with external visual signals.
  • Decoding side effects: Directly promoting object tokens can slightly affect local phrasing, perplexity, and repetition.Fluency-aware constraints are proposed to reduce this effect.
  • Evaluation reliability: Hal and Cog may penalize visually valid but unannotated objects, limiting evaluation reliability.More exhaustive annotations and human verification would provide a more reliable evaluation.
Loading 2608.21819v1…