Where Hallucinations Live: A Cross-Architecture Circuit in VQ-Tokenized Vision-Language Models

1 Introduction

• VQ-tokenized models show extreme yes-response biases or collapse toward no, unlike continuous-projector VLMs.VILA-U and Liquid answer yes on roughly 78–81% of records, while Chameleon answers yes on 22% and Lumina-mGPT on 0.07%.Source: §1 · p. 1
• Existing decoding-time interventions target output logits, while whether they reduce open-ended hallucination remains a separate question.Prior mechanistic studies are typically single-model case studies, leaving cross-architecture circuit generality unresolved.Source: §1 · pp. 1–2
• Activation patching across 25 models identifies an early-layer attention-routing circuit in five unified-VQ VLMs spanning three LLM families.The circuit is absent in continuous-projector models; a CLIP+MLP-to-VQ+Linear swap installs it, while a matched-compute MLP control does not.Source: §1 · p. 2
• Only L0 ablation reduces open-ended object hallucination, although tuned DoLA wins on binary calibration.The introduction reports CHAIR_i falling from 10.3% to 7.1% under L0 ablation.Source: §1 · p. 2
• The proposed three-gate diagnostic combines residual-stream divergence, L0 routing concentration, and codebook off-manifold rate.The gates use thresholds tied to architectural properties measured before behavioral outcomes.Source: §1 · p. 2

2 Background and Related Work

• VQ tokenizers map visual patches to entries in a learned codebook, making visual tokens share a vocabulary with text.The passage identifies codebook collapse and snapping artifacts as recurring failure modes.Source: §2 · p. 2
• VCD, DoLA, ITI, and VTI intervene on logits at inference without modifying model weights or explaining why a model hallucinates.Mechanistic-interpretability studies address the underlying why, but are described as single-model case studies.Source: §2 · p. 2
• Activation patching localizes where behavior is causally encoded by replacing corrupted-input activations with clean-input activations one site at a time.The paper's diagnostic builds on this procedure.Source: §2 · p. 2

3.1 Mechanistic diagnostic: a three-gate chain

• Clean/corrupted visual embeddings are patched across token categories; per-model noise calibration targets 0.5 mean cosine similarity, and token dropout reproduces early divergence.Restoration scores of 0, 1, and >1 indicate no effect, full restoration, and overshoot; permuting visual-token embeddings produces near-zero divergence.Source: §3.1 · pp. 3–4
• A model passes only when residual divergence starts in layers [0, 8) and persists at ℓ≥12, early attention meets its tiered threshold, and codebook off-manifold rate reaches 80%.The gates are conjunctive because attention mass alone can occur without a pathological signal; the matched-compute MLP control and natural negatives illustrate distinct gate failures.Source: §3.1 · p. 4

3.2 Constructive induction procedure with matched-compute control

• A frozen LLaVA-1.6 LM and CLIP encoder receive either VQ+Linear or matched 2-layer GeLU MLP projectors, trained 2,000 steps on identical 50k CC3M data.Capacity sweeps test K={1024, 4096, 16384, 65536}; FSQ variants test circuit formation without codebook collapse.Source: §3.2 · pp. 4–5

3.3 Interventions and a sanity criterion

• Inference-time hooks test L0 and L1 attention-output ablations and VTI; a POPE sanity screen rejects gains caused by collapsed responding or lost class discrimination.An intervention is preserved only when the baseline-to-intervention within-class standard-deviation ratio stays below 3 and the across-class mean logit-gap difference stays above 0.5.Source: §3.3 · p. 5

3.4 Experimental setup

• Evaluation covers POPE, AMBER, and CHAIR; activation patching uses 500 paired NaturalBench/POPE records across natural VQ, continuous-projector, and induced model cohorts.The setup reports 300 NaturalBench image-swap pairs and 200 POPE Gaussian-noise pairs; behavioral intervals use 10,000 bootstrap resamples.Source: §3.4 · pp. 5–6

4.1 A shared early-layer routing circuit across VQ-tokenized VLMs

• All five natural unified-VQ models across three LLM families pass the diagnostic; the circuit is a layer-level routing channel, and codebook bias aligns with polarity only associationally.The held-out Lumina-mGPT passes all three gates despite a null L0 behavioral change.Source: §4.1 · p. 6

4.2 Causal identification: induction and asymmetry

• With identical training parameters, LLaVA-VQ reaches an 85.5% POPE yes-rate and falls to 14.1% under L0 ablation, while the matched-compute MLP control shows only a 5-point drop.The MLP control fails both halves of A1 but passes A2 in isolation.Source: §4.2 · p. 7, Table 2 · p. 7
• Capacity alone does not determine polarity: K = 4096 inverts to YES-suppressing while larger capacities are YES-promoting, and an FSQ configuration inverts without codebook collapse.The second FSQ configuration reproduces behavioral inversion but lacks a gate-detectable early divergence event.Source: §4.2 · pp. 7–8

4.3 The diagnostic generalizes across models

• The diagnostic rejects fifteen models for distinct mechanistic reasons, including UniTok’s lack of an early event, Emu3’s failure to propagate, and Janus-Pro’s lack of a probeable codebook.Qwen-VQ shows that codebook collapse alone is not sufficient to install the circuit.Source: §4.3 · p. 8

4.4 Mechanism-targeted intervention reduces hallucination in generation

• L0 and L1 interventions help on VILA-U, but the same L0 ablation is null on LLaVA-1.6 and catastrophic on Qwen2.5-VL; effects are specific to early routing layers.Layer-wise ablation helps at layers 0 and 1, harms at layers 2 and 4, and has no effect at layers 8 and 16.Source: §4.4 · p. 9
• On 500 COCO captions, L0 ablation cuts CHAIR_i from 10.3% to 7.1% (31% relative), while tuned DoLA barely changes CHAIR_i and tuned VCD worsens it.Tuned DoLA nevertheless wins on binary POPE and AMBER benchmarks.Source: §4.4 · p. 9, Table 5 · p. 10

Implications for VQ tokenizer design

• Codebook collapse is neither necessary nor sufficient for the L0 routing circuit, and collapse polarity does not determine whether a model over-claims or denies.The non-collapsing FSQ configuration installs the circuit with inverted polarity; the induced Qwen-VQ counterexample shows collapse alone is insufficient.Source: §5 · p. 10

Where the circuit lives

• The L0 routing pattern is already present in the backbone, while VQ pretraining installs the pathological signal content and residual divergence that it carries forward.A healthy Vicuna-7B with an MLP projector trained for 2000 steps shows L0 attention mass 25.52.Source: §5 · pp. 10–11

Binary vs. generative dissociation

• Only L0 ablation reduces open-ended object hallucination, whereas tuned DoLA wins on POPE/AMBER binary benchmarks.The discussion describes this dissociation as making the mechanism-targeted intervention nonredundant with mechanism-agnostic baselines.Source: §5 · p. 11

6 Conclusion

• The circuit is shared across VQ-tokenized VLMs, isolated through a single-variable construction with matched-compute control, and targeted by the only tested method that reduces open-ended object hallucination.The diagnostic distinguishes the circuit with three gates and requires no single-measure fit.Source: §6 · p. 11
• The findings reframe object hallucination in unified VLMs as a property of architecture and pretraining rather than decoding-time calibration.The conclusion points to VQ tokenizer design as a leverage point for future unified VLMs.Source: §6 · p. 11

Limitations

• The reverse-ablation result shows asymmetry at matched budget, not impossibility; longer fine-tuning with the LM unfrozen might reverse the circuit.The authors make only the weaker claim.Source: Limitations · p. 11
• The Section 4.1 polarity association is not causal; the K = 4096 inversion and Chameleon’s mildly YES-promoting L0 lack mechanistic explanations.The authors do not claim that intervening on codebook bias would flip polarity.Source: Limitations · p. 11
• Although circuit topology is shared, per-model VTI directions have pairwise cosine similarity ≤0.054 and must be re-derived for each architecture.Liquid and Anole are excluded as practical intervention models because their L0 ablation collapses to a degenerate emitter; they serve as gate-level evidence only.Source: Limitations · p. 11
• The distributed-head account of L0 routing rests on per-head mass distributions, so its head-level claim is structural rather than causal.Source: Limitations · p. 11
• The claim that codebook collapse is unnecessary rests on one FSQ model; a second configuration reproduces polarity inversion but fails A1 at initiation (0.379).The two lattices agree on behavioral consequence but differ in mechanism.Source: Limitations · p. 11
• Open-ended CHAIR evidence comes from VILA-U and one induced variant, but the variant’s CHAIRi reduction is not significant at image level.Thus, significant open-ended evidence rests on VILA-U alone.Source: Limitations · p. 11

A.1 Cohort taxonomy

• Table 6 reports architecture, σ-calibration, and three structural gate values; Table 7 reports behavioral results under L0 ablation.A2 uses L0 mass ≥15 for projector-amplified models and a [0, 8) window total ≥15 for unified-vocabulary models.Source: §A.1 · p. 14

• Table 8 reports per-model σ calibration values and achieved cosine similarities for the causal sweep.Source: §A.2 · p. 14

• CHAIR_i falls from 10.30 [8.80, 11.82] to 7.13 [5.04, 9.57], a difference of -3.17 [-5.37, -0.76] whose interval excludes zero.Open-ended captioning intervals resample 500 COCO images because CHAIR is scored per caption.Source: §A.3 · p. 14
• All 24 binary-benchmark paired-bootstrap 95% confidence intervals are at most 3.8 pp wide; AMBER intervals are at most 1.7 pp wide.The passage reports 14,216 AMBER records and 3,000 POPE records.Source: §A.3 · p. 14
• CHAIR_s falls by 18.20 [13.80, 22.80] and recall by 6.89 [4.86, 8.98] in open-ended captioning.Source: §A.3 · p. 14

A.4 Sweep grid

• Table 11 reports POPEadversarial and AMBER accuracy across 36 VCD configurations; the VILA-U baseline is 0.619 and 0.397, respectively.Winning configurations are bolded in the table.Source: §A.4 · p. 14

A.5 Training hyperparameters

• Induced models were trained for 2,000 steps on the same 50k-caption CC3M subset, with CLIP ViT-L/14 and Vicuna-7B frozen throughout.Training used batch size 8, gradient accumulation 4, and learning rate 2e-3; codes counts distinct codebook entries with positive EMA usage at the final checkpoint.Source: §A.5 · p. 14

A.6 Three-gate diagnostic: alternative single-measure A2 criteria

• A uniform L0 mass threshold of 15 makes three errors, while a uniform [0, 8) window-mass threshold makes two false positives.No single mass threshold reproduces the architecture-class verdicts; the two-tier rule follows the projector-amplified versus unified-vocabulary distinction.Source: §A.6 · p. 14
• Across 19 cohort models with measured gate values, the A2 partition is invariant from 8–15 and first changes at 18; the A3 partition is invariant from 60–95%.The A1 default is the only tested value that reproduces the partition: lowering it admits FSQ v2 and UniTok, while raising it drops the FSQ v1 positive.Source: §A.6 · p. 14

A.7 Open-ended generation: supporting analyses

• Hallucinated mentions fall from 310 to 192, a 38% reduction; 36 categories improve, 11 worsen, and 31 are unchanged.Orange falls 41 →11, chair 34 →8, bird 12 →1, and bear 10 →0, while dining table rises 20 →43 and sports ball 8 →25; the reported pattern includes residual shifts toward scene-generic objects.Source: §A.7 · pp. 14–15

A.8 Robustness to corruption type

• Token dropout reproduces the early divergence peak in both VQ tiers, while permuting visual-token embeddings produces near-zero divergence despite the largest tested perturbation magnitude.The passage interprets this as routing that pools visual-token content independently of order.Source: §A.8 · p. 15
• For VILA-U, both open-ended hallucination reductions and the accompanying recall cost are significant; for induced LLaVA-VQ, CHAIR_s and recall differences are significant, but CHAIR_i is not.The induced model's recall is approximately 11%, making its CHAIR_i difference suggestive only.Source: Table 10 · p. 17

A.7 Open-ended generation: supporting analyses

• Table 15 reports reduced hallucination at a moderate caption-informativeness cost, with METEOR improving.Source: Table 15 · p. 18