Source-linked AI summary
3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding
Makanjuola Ogunleye, Eman Abdelrahman, Ismini Lourentzou
TL;DR
Embodied agents can produce scene-inconsistent hallucinations that undermine safe physical interaction, while existing inference-time methods largely target 2D settings. 3D-VCD contrasts original and distorted structured 3D scenes, consistently improving grounded reasoning across 3D-POPE and HEAL without retraining.
Problem
3D embodied agents hallucinate about objects and spatial scenes, creating unsafe downstream interactions, while training-based mitigation cannot cover real-world scene diversity.
Method
3D-VCD performs inference-time contrastive decoding by contrasting predictions from original and semantically or geometrically distorted 3D scene graphs.
Results
3D-VCD consistently improves grounded reasoning across 3D-POPE and HEAL without training, including reduced over-affirmation and lower object and state hallucination rates.
Takeaways & Limitations
Inference-time contrastive decoding over structured 3D representations offers a practical route to more reliable embodied agents.
Abstract
from arXiv · showhide
Large multimodal models are increasingly used as the reasoning core of embodied agents operating in 3D environments, yet they remain prone to hallucinations that can produce unsafe and ungrounded decisions. Existing inference-time hallucination mitigation methods largely target 2D vision-language settings and do not transfer to embodied 3D reasoning, where failures arise from object presence, spatial layout, and geometric grounding rather than pixel-level inconsistencies. We introduce 3D-VCD, the first inference-time visual contrastive decoding framework for hallucination mitigation in 3D embodied agents. 3D-VCD constructs a distorted 3D scene graph by applying semantic and geometric perturbations to object-centric representations, such as category substitutions and coordinate or extent corruption. By contrasting predictions under the original and distorted 3D contexts, our method suppresses tokens that are insensitive to grounded scene evidence and are therefore likely driven by language priors. We evaluate 3D-VCD on the 3D-POPE and HEAL benchmarks and show that it consistently improves grounded reasoning without any retraining, establishing inference-time contrastive decoding over structured 3D representations as an effective and practical route to more reliable embodied intelligence.
1. Introduction
3D embodied agents can produce plausible but scene-inconsistent outputs, creating safety and reliability risks, while existing hallucination mitigation largely targets 2D pixel-space settings. 3D-VCD addresses this gap with training-free contrastive decoding over structured 3D representations and improves grounded prediction across 3D-POPE splits.
- Embodied agents may affirm nonexistent objects or misidentify present ones, propagating hallucinations into unsafe actions and unreliable system behavior.
- Agents often rely on language priors when visual evidence is weak, ambiguous, or occluded, and no prior hallucination mitigation strategy specifically targets 3D embodied agents.
- Training-only mitigation is limited by the combinatorial diversity of real-world scenes, motivating inference-time methods that do not depend on deployment conditions appearing in training data.
- 2D mitigation methods contrast predictions under corrupted images, but do not transfer to multi-view, incomplete, geometry-rich embodied observations.
- 3D-VCD contrasts logits from original and semantically or geometrically distorted 3D representations to suppress predictions insensitive to 3D evidence.
- 10.9%–24.7% lower over-affirmation and 8.1%–35.8% higher accuracy relative to 3D-LLM were observed across all 3D-POPE splits.
2. Related Work
Prior 3D embodied models support spatial reasoning but remain vulnerable to hallucinations, especially under scene–task inconsistencies. Existing inference-time contrastive methods operate in 2D pixel space, leaving a gap that 3D-VCD addresses with structured scene-graph perturbations.
- 3D embodied models use scene graphs, volumetric features, or neural descriptors to support open-vocabulary reasoning, grounding, and object-centric understanding.
- A single hallucinated object can derail an embodied task, and HEAL reports frequent hallucinations under scene–task inconsistencies with reliance on language priors.
- Inference-time interventions avoid retraining, while 2D Visual Contrastive Decoding contrasts logits from original and corrupted images using blur, masking, or patch noise.
- Pixel-level contrastive decoding is insufficient for embodied agents because their hallucinations stem from 3D structure and spatial reasoning rather than only 2D semantic inconsistency.
3. Method
3D-VCD mitigates 3D hallucinations by contrasting MLLM predictions from original and deliberately distorted scene contexts. It operates on structured object-centric representations and supports both scene-graph and adversarial task distortions with lightweight inference overhead.
- 3D Visual Contrastive Decoding: 3D-VCD identifies hallucination-prone tokens by comparing predictions under the original and corrupted 3D context.Tokens that remain unsuppressed under corruption are treated as unsupported by grounded evidence and likely driven by language priors.
- Structured 3D Representation: The method represents each scene as an object-centric graph whose nodes encode semantic categories and benchmark-dependent geometric, state, or relational attributes.In 3D-POPE, attributes include centroids and spatial extents; HEAL uses discrete states and explicit relations.
- Graph-Space Distortions: 3D-VCD creates syntactically valid corrupted graphs by perturbing object labels, 3D centroids, spatial extents, or other geometric attributes.Semantic perturbations shuffle categories, while geometric perturbations add Gaussian noise to positions and extents.
- HEAL Adaptation: For HEAL, adversarial prompts serve as distorted contexts while the underlying scene representation remains fixed.The paired contexts include distractor injection, synonym substitution, and scene–task contradictions.
- Contrastive Decoding: Dual-context logit fusion penalizes predictions that remain highly probable under both original and distorted representations.The contrastive penalty is controlled by α ≥0 and relatively favors tokens supported by the true 3D scene.
- Efficiency: 3D-VCD adds one forward pass per query, uses key–value caching, and increases average query time from approximately 2 s to 2.5 s.The method introduces no trainable parameters or retraining and is described as suitable for interactive embodied reasoning.
4. Experiments
Experiments evaluate 3D-VCD on 3D-POPE and HEAL, testing object and state hallucinations, distortion robustness, and qualitative grounding. Across these evaluations, 3D-VCD improves benchmark performance and suppresses hallucination through semantic, geometric, structural, and mixed scene-graph perturbations.
- 3D-POPE results: 3D-VCD achieves the highest precision, accuracy, and F1-score across the Random, Popular, and Adversarial 3D-POPE settings.It also reduces over-affirmation, with the Random-split Yes-rate decreasing from 99.81% to 75.15%.
- HEAL results: 3.3× reduction in Qwen-14B-Instruct state hallucination lowers CHAIR–CS from 16.45% to 5.0% on HEAL.Object hallucination also decreases for Qwen-14B and Llama-3-8B under distraction injection.
- Qualitative results: Qualitative examples show 3D-VCD recovering a dining table missed by the baseline and removing a hallucinated microwave from HEAL symbolic goals.The HEAL example instead grounds all sweaters on the bed and performs the instruction-required state changes.
5. Inference Efficiency
3D-VCD scales smoothly with scene complexity, requiring only a modest runtime increase while preserving inference-time practicality.
- Inference time increases modestly as the number of scene objects grows.The runtime analysis reports approximately 3.8s for simple scenes, although the supplied passage truncates the larger-scene value.
- 3D-VCD operates purely at decoding time without additional training or auxiliary models.
- The method’s overhead remains minimal and comparable to standard 3D-LLM inference.
6. Conclusion
The paper concludes that 3D-VCD mitigates hallucinations through contrastive decoding over structured 3D representations. It improves grounding while remaining training-free, architecture-agnostic, and computationally lightweight.
- 3D-VCD uses a dual-scene contrastive process over structured 3D scene graphs.
- The method substantially reduces over-affirmation bias while improving factual precision and grounding reliability across all 3D-POPE categories.
- 3D-VCD is training-free, architecture-agnostic, and incurs minimal computational overhead.
- The paper identifies inference-time contrastive reasoning as an efficient mechanism for improving perception in embodied environments.
A. Implementation Details
3D-VCD operates on released 3D-LLM checkpoints without modifying model parameters, using object-centric scene-graph attributes to manipulate token logits at inference time.
- The implementation uses released 3D-LLM checkpoints from 3D-GRAND with a LLaMA-style causal decoder.
- 3D-VCD does not finetune or modify model parameters, instead operating purely at inference time.
- Each scene is serialized using object category, centroid (x, y, z), and extent (w, h, d) attributes extracted from dataset JSON.
- The framework contrastively manipulates token logits during decoding.
B. Additional Ablations
The ablations show that contrastive decoding requires moderate geometric distortion: insufficient noise weakens the signal, while excessive noise damages spatial grounding.
- Negative Transfer and Query Sensitivity: Minimal distortion ε = 0.01 produces higher Yes-rates (86.05%) and lower F1 scores (73.81%) than moderate distortions.
- Negative Transfer and Query Sensitivity: Moderate distortion ε = 0.05 achieves the highest random-split F1 (75.00%) and Accuracy (67.65%).
- Sensitivity to Geometric Perturbations: Excessive distortion ε = 0.45 degrades performance, indicating that structural coherence is necessary for meaningful contrastive signals.
- Sensitivity to Geometric Perturbations: Moderate perturbations disrupt over-specific coordinate cues while preserving high-level spatial layout needed for grounding.
C. Results on the HEAL Probing Set
On HEAL, 3D-VCD uses contrastive comparisons between adversarial and clean prompts to restore grounding under scene–task inconsistencies. It substantially reduces hallucination rates across challenging probes, including distractor, synonymous-object, and contradiction settings.
- 3D-VCD contrasts adversarial prompts with corresponding clean baseline prompts to test recovery of grounding under misleading textual cues.The evaluation includes adversarial probes such as Distractor Injection.
- Approximately 70%: 3D-VCD reduces State Hallucination Rate (CS) from 16.5% to 5.0% under Distractor Injection.This probe tests resilience to irrelevant text mentions.
- 1.0%: 3D-VCD achieves the lowest Object Hallucination Rate (CO) in the Scene-Object Synonymous probe among evaluated models.The result is reported for Qwen-14B and outperforms the Llama-3-8B baseline.
- 1.5%: 3D-VCD lowers Object Hallucination Rate (CO) under Scene-Task Contradiction, compared with 53.9% for the base model.The probe tests cases where the base model hallucinates objects to satisfy impossible goals.
- Across all challenging HEAL splits, the Qwen-14B-Instruct model shows enhanced grounding with the inference-time intervention.The reported effects include steering generation away from ungrounded textual cues and toward physical scene evidence.
D. Additional Qualitative Examples
Qualitative examples show baseline 3D-LLMs making incorrect object-presence judgments, while 3D-VCD produces answers aligned with scene evidence. The evaluation also requires both a correct binary decision and correct object-instance grounding.
- Baseline models hallucinate object presence in qualitative 3D-POPE and HEAL examples, including nonexistent beds and desks.The examples illustrate incorrect “yes” or “no” responses and reliance on linguistic priors rather than 3D evidence.
- A prediction is correct only when both the Yes/No decision and the referenced object identifier match the scene ground truth.Responses are parsed from the first grounded segment enclosed within <detailed grounding> tags.
- Reducing hallucinations improves grounding reliability but does not guarantee safe downstream planning or action under noise, distribution shift, or adversarial conditions.Additional safeguards such as uncertainty estimation, human oversight, and real-world testing remain necessary.
- 3D-VCD correctly rejects nonexistent bed and desk objects by contrasting logits under perturbed 3D scene graphs.The method suppresses hallucinated object activations and spurious category matches.
- The model input and output are illustrated in Figure 9 for object-presence queries over a 3D scene.The prompt requires object noun phrases to be linked explicitly to unique scene-object identifiers.