Source-linked AI summary
EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation
Sihang Jia, Shuliang Liu, Songbo Yang, Xuming Hu
TL;DR
LVLMs often produce content unsupported by images, and visual evidence incorporated early in decoding becomes less influential later. EviAnchor preserves that evidence in regional anchors, routes it to the current decision state, and restores native computation; experiments across multiple benchmarks report improved visual grounding.
Problem
LVLMs frequently generate objects, attributes, or relationships unsupported by visual inputs, limiting their reliability in real-world applications.
Method
EviAnchor is a training-free, single-branch inference framework that stores regional visual evidence in REA slots, routes relevant anchors to the current generation state, and restores native Transformer computation.
Results
Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding and LVLM generation reliability.
Takeaways & Limitations
Preserving and reusing regional visual evidence across decoder layers improves visual grounding without updating model parameters.
Takeaways & Limitations
Some competing methods’ lower CHAIRi values are partly influenced by reduced output coverage when they produce empty responses, which substantially degrades recall.
Abstract
from arXiv · showhide
Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.
Introduction
LVLM hallucinations arise when generated content is unsupported by visual inputs, while visual evidence is strongest in early-to-middle decoder layers and weakens later. EviAnchor preserves and reconnects that evidence during generation through regional anchors and decision-conditioned routing.
- LVLMs frequently generate objects, attributes, or relationships unsupported by images, limiting their reliability in real-world applications.
- Layer-wise analyses show that visual evidence most strongly affects answer-side representations in early-to-middle layers and weakens rapidly later.Attention-path restriction and counterfactual analyses localize visual influence across the decoder.
- This weakening can leave visual evidence acquired earlier underutilized by higher-layer generation states, framing hallucination as insufficient cross-layer propagation.The paper illustrates this failure mode with later-layer visual-state ablations that leave hallucinated mentions nearly unchanged.
- EviAnchor preserves earlier visual evidence in spatially structured REA slots and reconnects relevant anchors to the current generation state.The framework is presented as an inference-time mitigation method rather than a parameter-updating approach.
- Decision-conditioned routing reduces excessive reliance on textual representations before native Transformer computation recomposes visual evidence, question semantics, and generation history.
- The framework is training-free, single-branch, and evaluated on discriminative and generative benchmarks.The contribution description states that it requires no external tools or multi-branch inference, while the evaluation passage mentions both benchmark types.
Related Work
Prior mitigation methods intervene either in output probabilities after multimodal representations form or directly in internal representations. EviAnchor instead uses regional anchors and routing to preserve and reuse visual evidence across decoder stages.
- Training-based mitigation requires additional data and optimization, motivating increased interest in inference-time methods for existing LVLMs.
- Output-space methods modify token probabilities through contrastive distributions, candidate penalties, or attention-based decoding interventions.These methods mainly adjust final predictions after multimodal representations have already formed.
- Its overview combines regional evidence anchors, decision-conditioned routing, and native Transformer recomposition across decoder stages.
- Representation-space methods modify hidden states or internal attention computation to influence the generation trajectory and visual reliance.Examples include VTI, ClearSight, and Nullu.
Methodology
EviAnchor preserves regional visual evidence across decoder stages and reroutes it to the current decision before resuming native Transformer computation. Its three-stage design separates evidence compensation from lexical realization while retaining spatial detail and linguistic coherence.
- Methodology: EviAnchor is a training-free framework that targets the weakened influence of early-layer visual evidence on later token predictions.It operates exclusively at inference time.
- Methodology: The inference process divides decoder layers into Lpool for visual acquisition, Lcomp for compensation, and Lnative for lexical prediction.Visual influence is strong in Lpool, weakens in Lcomp, and is organized into predictions in Lnative.
- Regional Evidence Anchor: REA slots partition visual tokens into spatially disjoint regions and progressively aggregate each region into a temporary spatially indexed representation.Each slot attends only to itself and the visual tokens in its corresponding region.
- Decision-Conditioned Evidence Routing: Decision-conditioned routing reduces the current decision state’s dependence on target-text tokens and transfers that attention mass to regionally preferred REA slots.The redistribution preserves a normalized attention distribution and lets retrieved regional evidence affect the next-token decision.
- Native Semantic Recomposition: After routing, Native Semantic Recomposition ends explicit interventions and lets the pretrained Transformer jointly integrate visual tokens, REA slots, question semantics, and generation history.This preserves the model’s native computation while separating evidence compensation from lexical realization.
Experimental Setup
The evaluation covers discriminative and open-ended hallucination benchmarks across three LVLM architectures and model scales, using greedy decoding and training-free baselines for comparison.
- Benchmarks: The experiments evaluate hallucinations across discriminative object verification and open-ended generation.The benchmark suite comprises POPE, MMHal-Bench, and CHAIR.
- Benchmarks: POPE uses 27,000 image–question pairs across Random, Popular, and Adversarial settings to evaluate object presence with accuracy, precision, recall, and F1.Questions are binary yes-or-no prompts.
- Benchmarks: MMHal-Bench contains 96 image–question pairs across eight categories and reports GPT-4-assigned average scores and hallucination rates.Scores range from 0 to 6, with scores below 3 treated as hallucinated.
- Benchmarks: CHAIR measures unsupported object mentions in descriptions using CHAIRs and CHAIRi, alongside object recall and average response length.Evaluation uses 500 COCO val2014 images.
- Models and Baselines: EviAnchor is applied to LLaVA-1.5-7B, InstructBLIP-Vicuna-7B, and Qwen3-VL-4B-Instruct with greedy decoding.The study compares EviAnchor with output-space and representation-space training-free methods.
Main Results
Across POPE, CHAIR, and MMHal-Bench, EviAnchor consistently improves visual grounding and reduces hallucination across three LVLM backbones. The gains preserve response coverage rather than relying on shorter outputs.
- POPE: EviAnchor achieves the highest POPE accuracy and F1 across all three backbones, improving object-existence verification.On Qwen3-VL, it reaches 90.93 accuracy and 91.41 F1.
- CHAIR: EviAnchor consistently reduces CHAIR sentence-level and instance-level object hallucinations relative to Origin.On LLaVA-1.5, CHAIRs falls from 58.20 to 48.80 and CHAIRi from 17.06 to 14.10 while recall remains 77.38.
- MMHal-Bench: EviAnchor achieves the best MMHal-Bench performance across all three backbones, raising scores and reducing hallucination rates.On Qwen3-VL, the score rises from 4.04 to 4.30 while hallucination rate falls from 40.6 to 29.2.
- Overall: Across discriminative and open-ended generation tasks, the results demonstrate improved visual grounding across model architectures.
Ablation Studies
Ablations show that regional anchoring, evidence routing, and native recomposition each contribute to EviAnchor’s performance. Larger evidence banks and longer routing can impose response-completeness trade-offs, motivating the default 3×3 configuration and moderated intervention.
- Module Ablation: Removing regional anchoring decreases POPE F1 by 1.33 points and increases CHAIRs from 48.80 to 53.60.This supports the benefit of compact regional evidence representations.
- Module Ablation: Removing evidence routing increases CHAIRs to 52.20, showing that preserved visual anchors alone do not ensure effective later utilization.
- Module Ablation: Extending routing to the final layer reduces recall from 77.38 to 71.80, highlighting the importance of native upper-layer recomposition.
- REA Slot Granularity: Increasing REA slots generally improves POPE performance, but excessively fine-grained banks introduce generation instability and recall degradation.With default relaxation, 16- and 25-slot banks produce 8 and 25 empty responses, respectively, partly affecting lower CHAIRi values.
- Evidence Aggregation: Figure 4 tracks REA-slot attention to corresponding image tokens against the decision state’s attention to original visual tokens.
- REA Slot Granularity: The 3×3 configuration is adopted as the default because it balances visual grounding and response completeness.Larger banks provide stronger visual constraints but require careful control of intervention intensity.
- Evidence Routing Length: Routing from 4 to 8 layers improves POPE F1 from 85.37 to 86.76 and lowers CHAIRs/CHAIRi from 53.20/15.35 to 48.80/14.11, whereas 12 layers generally degrades all three metrics.
Conclusion
EviAnchor addresses cross-layer visual evidence underuse with training-free, single-branch inference that preserves regional evidence, reconnects it to generation states, and restores native upper-layer computation. Results across multiple benchmarks validate this approach for improving LVLM generation reliability.
- Conclusion: EviAnchor preserves regional visual evidence in REA slots, reconnects it through decision-conditioned routing, and restores native upper-layer computation.
- Conclusion: Layer-wise analyses show that visual evidence is incorporated mainly in early-to-middle decoder layers while its direct influence weakens later.
- Conclusion: Experiments across multiple benchmarks validate cross-layer visual evidence preservation and reuse for improving LVLM generation reliability.