RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction

I. INTRODUCTION

• Relational hallucinations—incorrect spatial or interactive descriptions—remain largely unaddressed by existing correction methods.The introduction notes that object- and attribute-level errors have received more attention.Source: §I · p. 1
• MLLMs score 20–30 percentage points lower on spatial reasoning queries than object queries, and LLaVA v1 13B has accuracy+ 0.000 on MME position.The passage says this means it cannot answer both positive and negative variants correctly for any image in that dataset.Source: §I · p. 1
• RelCheck adds RelTR subject–predicate–object triples and GroundingDINO box-derived spatial predicates to Woodpecker’s object-level knowledge base.These form relational evidence layers alongside object-level claims.Source: §I · p. 1
• In the cat-and-hat example, scene-graph and box evidence distinguish wearing from holding, prompting the corrector to revise the caption.Object verification and attribute VQA confirm the entities and hat color but do not resolve their relationship.Source: §I · p. 1
• The three-layer knowledge base unifies object claims, learned relational triples, and deterministic geometric predicates under an explicit reliability ordering.The correction pipeline also uses layer-attributed JSON edits and an edit-distance gate for minimal, evidence-grounded revisions.Source: §I · p. 2
• RelCheck improves over a Woodpecker-style baseline across POPE and MME, with gains concentrated on subtasks requiring relational reasoning.A four-configuration ablation and McNemar tests indicate that both relational layers contribute independently.Source: §I · p. 2

A. Hallucinations in MLLMs

• CHAIR measures object hallucination in captions but addresses object presence rather than relational errors.The literature distinguishes object-, attribute-, and relational-level hallucinations.Source: A. Hallucinations in MLLMs · p. 2
• Models show a 20–30-point spatial-versus-object query gap, while SpatialVLM uses targeted training and RelCheck operates post-hoc without retraining.The Visual Spatial Reasoning benchmark covers topological, directional, and distance-based categories.Source: A. Hallucinations in MLLMs · p. 2

B. Post-Hoc Hallucination Correction

• Woodpecker builds object- and attribute-level claims from detections and VQA, but its knowledge base lacks inter-object relational structure.RelCheck fills this gap with scene-graph and geometric layers.Source: B. Post-Hoc Hallucination Correction · p. 2
• Self-feedback [20], LURE, and OPERA rely respectively on model critique, uncertainty signals, and decoding-time attention patterns, with the latter two requiring fine-tuning or internal access.LURE cannot be applied to closed-source MLLMs; OPERA cannot correct hallucinations post-hoc.Source: B. Post-Hoc Hallucination Correction · p. 2

C. Scene Graph Generation

• RelTR predicts subject–predicate–object triples, with predicate types spanning spatial, interaction, and possessive relationships.It is trained on Visual Genome and predicts triples end-to-end in a single forward pass.Source: C. Scene Graph Generation · p. 2
• No prior post-hoc hallucination correction pipeline had incorporated scene-graph output as a relational evidence layer.Existing pipelines relied on object detectors and VQA models without querying a dedicated relational model.Source: C. Scene Graph Generation · p. 2

A. Overview

• The image-derived knowledge base combines Woodpecker-style object and attribute claims, RelTR scene triples, and deterministic spatial predicates from bounding boxes.All three layers feed the KB-aware language-model corrector to produce grounded output.Source: A. Overview · pp. 2–3, Figure 1 · p. 3

B. Shared Claim Generation Pipeline (Stages 1–4)

• Stages 1–4 extract entities, formulate object and attribute questions, validate them visually, and convert the answers into declarative claims.GroundingDINO answers object-level questions, multimodal VQA handles attribute questions, and a pretrained QA-to-claim model organizes the resulting claims.Source: B. Shared Claim Generation Pipeline… · p. 3

C. Multi-Layer Visual Knowledge Base

• The Claim Layer is the shared Woodpecker-style knowledge base of object-count, per-entity, and scene-level attribute claims.It is identical between Woodpecker and RelCheck.Source: C. Multi-Layer Visual Knowledge Base · p. 3
• RelTR supplies Scene(I) as subject–predicate–object triples, with predicates drawn from a roughly 150-class vocabulary and filtered by a confidence threshold.RelTR is applied directly to the image without text conditioning.Source: C. Multi-Layer Visual Knowledge Base · p. 3
• Geom(I) compares detected-object bounding-box centroids to emit deterministic spatial predicates, suppressing nearly aligned pairs with a deadzone and emitting none when neither axis dominates.The illustrated predicates include left-of, above, and on/inside, with 5% tolerance for containment.Source: C. Multi-Layer Visual Knowledge Base · p. 3, Figure 2 · p. 3

• Woodpecker concatenates its claim-layer knowledge base with the MLLM response and asks a lightweight language model to correct the description.Its output is unrestricted free-form text.Source: D. Hallucination Correction · pp. 3–4
• RelCheck gives claim, scene-graph, and geometric evidence different reliability levels, treating geometry as reliable for separated 2D directions but not 3D relations such as in front of or behind.RelTR evidence may be incomplete when it does not find a relation.Source: D. Hallucination Correction · p. 4
• Decision rules resolve conflicts by trusting higher-confidence evidence, including claim-layer facts over contradictory scene-graph relations.The corrector is instructed to ignore geometric evidence when it alone contradicts an on/under/behind claim.Source: D. Hallucination Correction · p. 4
• RelCheck returns corrected text and edit-level evidence labels, then leaves the original response unchanged when edit distance falls outside 3–50 characters.The structured output also includes confidence and supports layer-attribution analysis; the edit gate enforces a minimal-edit principle.Source: D. Hallucination Correction · p. 4

E. Running Example

• In Figure 3, Woodpecker passes the MLLM’s left-side hallucination through unchanged, while RelCheck uses GroundingDINO bounding-box centroids to compute car right-of fire hydrant and flips Yes to No.All systems share LLaVA v1 13B and Stages 1–4.Source: E. Running Example · p. 4, Table I · p. 4

F. Answer Extraction

• A zero-shot language model converts each corrected free-form response into one yes/no token, using the same extractor for RawMLLM, Woodpecker, and RelCheck.Answers are cached for identical question–response pairs.Source: F. Answer Extraction · p. 4

A. Benchmarks

• POPE evaluates 1,500 COCO val2014 questions across Random, Popular, and Adversarial object-sampling splits using accuracy, precision, recall, and F1.Each split contains 500 questions.Source: A. Benchmarks · p. 4
• MME covers Existence, Count, Position, and Color with 60 yes/no questions per subtask, reporting accuracy, accuracy+, and a 0–200 score.Each subtask uses 30 images with two complementary questions per image.Source: A. Benchmarks · p. 4

B. Target MLLM and Implementation

• Experiments use LLaVA v1 13B with greedy decoding on one A100 GPU, and set GroundingDINO, RelTR, and spatial-deadzone thresholds.The thresholds are τobj = 0.35, text matching = 0.25, τrel = 0.30, and T = 0.08.Source: B. Target MLLM and Implementation · p. 4

C. Systems Compared

• RawMLLM is uncorrected LLaVA output; Woodpecker uses a lightweight corrector, while RelCheck uses all three knowledge-base layers and a higher-effort corrector.The systems share the same claim-generation pipeline and GroundingDINO backbone; RelCheck also uses structured JSON output and an edit-distance gate.Source: C. Systems Compared · p. 4
• In the position example, RelCheck’s geometric layer identifies the car as right of the hydrant and corrects the false answer, while Woodpecker leaves it unchanged.The RelCheck hyperparameters are held constant across experiments.Source: Figure 3 · p. 5, Table II · p. 5

D. Implementation Details

• Experiments use one NVIDIA A100 40GB GPU; LLaVA v1 13B runs in 8-bit quantization with greedy decoding and a 50-token limit.GroundingDINO and RelTR use thresholds of 0.35 and 0.30, respectively, with RelTR capped at 20 triples per image.Source: D. Implementation Details · p. 5
• Language-model stages use gpt-5.4-mini or gpt-5.4, while the QA-to-claim model runs locally; evaluating 240 MME samples across three systems takes about 1.5 hours.API responses are cached by prompt-image hash.Source: D. Implementation Details · p. 5

A. POPE Evaluation

• Raw LLaVA has a strong yes-bias: yes-rate rises from 0.759 to 0.860 across the random-to-adversarial splits, while adversarial precision falls to 0.571.The adversarial split selects negative objects that co-occur with present COCO objects, exploiting pretraining co-occurrence priors.Source: A. POPE Evaluation · pp. 5–6
• Corrections raise accuracy from 0.62–0.72 to 0.80–0.87 for Woodpecker and 0.81–0.88 for RelCheck, while precision rises from 0.57–0.65 to 0.81–0.94 and 0.83–0.95, respectively.The passage attributes the precision improvement to GroundingDINO grounding absent-object answers.Source: A. POPE Evaluation · p. 6
• RelCheck exceeds Woodpecker by +0.6/+1.4/+1.0 accuracy points on POPE’s random/popular/adversarial splits.The largest gain is on the popular split, where scene-graph triples can provide a corroborating object-presence signal.Source: A. POPE Evaluation · p. 6

B. MME Hallucination Evaluation

• 630.0 versus 585.0 is RelCheck’s total MME score over Woodpecker; Position rises to 135.0 from 103.3, with accuracy+ increasing from 0.367 to 0.600.The geometric layer supplies deterministic left/right predicates for the position questions, provided GroundingDINO detects the relevant objects.Source: Table IV · p. 6, B. MME Hallucination Evaluation · p. 6
• Existence scores 195.0 for RelCheck versus 185.0 for Woodpecker, with accuracy+ of 0.967 versus 0.900.The passage attributes the additional signal to objects appearing in RelTR triples as well as GroundingDINO detections.Source: B. MME Hallucination Evaluation · p. 6
• Count is tied at 146.7 because both systems rely on the same GroundingDINO detections and neither relational layer supplies counting information.The passage treats this as a control against unrelated system-level differences explaining other gains.Source: B. MME Hallucination Evaluation · p. 6
• Color improves by 3.3 points over Woodpecker despite equal accuracy of 0.833; RelCheck’s accuracy+ is 0.700 versus 0.667.The ablation reports a minor regression for SCENE alone before GEOM is added, with irrelevant relation evidence sometimes distracting the corrector on color questions.Source: B. MME Hallucination Evaluation · p. 6

C. Qualitative Analysis

• For a café sidewalk scene, RelCheck replaces “walking down the street” with “sitting at a café’s sidewalk,” correcting both action and scene context.RelTR predicts sitting and café-table triples, while the claim layer independently confirms the café-sidewalk setting; the example illustrates interaction relations missed by object detection or VQA alone.Source: C. Qualitative Analysis · p. 7

D. Ablation Study

• Position gains add +1.7 from the corrector, +16.7 from GEOM, and +13.3 from SCENE, supporting independent contributions from both relational layers.The passage reports an additive improvement across the configurations.Source: D. Ablation Study · p. 7
• ClaimOnly to RelCheckFull is significant (McNemar p = 0.025), while ClaimOnly is near Woodpecker at 580 versus 585 (p = 0.45).ClaimOnly uses the upgraded corrector with claims only; adding GEOM and then SCENE gives the two relational-layer configurations.Source: D. Ablation Study · p. 7
• GEOM produces non-empty evidence on 87% of Position samples (52/60) but rarely elsewhere, matching its activation pattern to the position gain.The passage uses this task-specific activation pattern to support the interpretation that GEOM drives position improvement rather than a general corrector effect.Source: D. Ablation Study · p. 7

E. Disagreement Analysis

• Among 35 disagreements, RelCheck wins 21 to Woodpecker’s 14, including a 13-to-8 RelCheck advantage on Position; Count and Color are tied.In eight Woodpecker wins, GEOM identifies the spatial predicate but the corrector does not apply it amid conflicting signals.Source: E. Disagreement Analysis · p. 7

VI. CONCLUSION

• RelCheck combines Woodpecker-style object claims, RelTR scene graphs, and deterministic geometry into a three-layer visual knowledge base for spatial and interactional relations.In Fig. 5, RelTR detects sitting while VQA confirms a café sidewalk, correcting “walking down the street.”Source: §VI · p. 7, Figure 5 · p. 8
• 630.0 vs. 585.0 total score on the MME hallucination benchmark favors RelCheck over the Woodpecker-style baseline, with 31.7 points concentrated on position.Accuracy+ on position improves from 0.367 to 0.600.Source: §VI · p. 7
• ClaimOnly ≈ Woodpecker (p = 0.45), while the full three-layer system transition is significant (p = 0.025); position gains decompose into +1.7 corrector, +16.7 GEOM, and +13.3 SCENE.Source: §VI · p. 7
• Undetected entities provide no GEOM or VQA evidence, and 2D geometry cannot assess 3D relations.Experiments cover only LLaVA v1 13B; proposed future work includes a monocular depth layer for 3D predicates and open-source alternatives to API-based components.Source: §VI · p. 7
• The authors conclude that training-free, plug-and-play relational grounding is a practical path for addressing relational hallucinations.Source: §VI · p. 7
• The authors connect accurate spatial grounding to medical imaging, robotics, and scientific literature mining, and describe structured geometric evidence as a reusable pattern.Source: §VI · pp. 7–8