PaperlayerView exact revision on arXiv

Source-linked AI summary

ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

Jihae Jeong, Junha Choi, Hwanjo Yu

arXiv:2608.19075v1cs.CVcs.AIcs.CL

TL;DR

Large vision-language models can produce fluent text unsupported by the image, while existing decoding signals do not directly measure image support. ReWEIGH calibrates token-level ordinal visual evidence and applies a bounded decoding penalty, reducing hallucinations across diverse models while largely preserving multimodal utility.

  • Problem

    Existing decoding interventions do not directly measure how strongly the image supports each candidate token, and global references conflate token-specific evidence differences.

  • Method

    ReWEIGH aggregates within-position vocabulary ranks into ordinal visual evidence and compares each candidate against a token-specific reference to guide bounded, training-free decoding penalties.

  • Results

    Up to 21.3% lower CHAIRI across four 7B backbones, with largely preserved or improved descriptive and general multimodal performance and reductions extending to 32B models.

  • Takeaways & Limitations

    Token-calibrated ordinal readouts provide a practical interface between internal visual states and generation while balancing hallucination reduction with retained multimodal content.

  • Takeaways & Limitations

    ReWEIGH requires separate calibration and tuning for each backbone, so transfer across backbones without additional calibration or development data remains unestablished.

Abstract

from arXiv · show

Large vision-language models (LVLMs) often hallucinate, generating content that the input image does not support. Preventing such content during decoding calls for a candidate-specific measure of how strongly the image supports the token under consideration. The model's visual-token states offer a natural source of this evidence because projecting each state through the output head reveals which vocabulary items that position favors. These position-wise readouts cannot be pooled directly because their probability magnitudes are not comparable across visual positions. Vocabulary ranks provide a scale-invariant basis for pooling, but tokens still differ systematically in their typical rank-based evidence. We propose ReWEIGH, a training-free decoding intervention that aggregates these ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images. At inference, ReWEIGH caches the image evidence during prefill and applies a bounded penalty only to candidates that fall below their reference. On four 7B backbones, ReWEIGH reduces hallucinated object mentions by up to 21.3% while largely preserving or improving descriptive and general performance. With evidence cached, the average added latency is 1.33% per token, and the reductions extend across six architecture families to 32B parameters.

1 Introduction

LVLMs can produce fluent hallucinations when linguistic priors outweigh visual evidence, motivating decoding-time interventions. ReWEIGH calibrates token-specific ordinal visual evidence with a training-free, bounded intervention that reduces hallucinations while preserving performance with low overhead.

  • Motivation: LVLM hallucinations can conflict with or lack support from the input image, while fluent output may conceal these errors.This motivates interventions during decoding, before unsupported content is emitted.
  • Motivation: Existing decoding interventions trade computational cost against grounding specificity, with contrastive methods often requiring additional forward or decoding passes.Lighter alternatives use attention or uncertainty to control reliance on visual tokens.
  • Method: Rank aggregation provides a scale-invariant basis for pooling visual-token readouts, while token-specific baselines account for systematic differences in typical ordinal evidence.Probability magnitudes depend on position-specific output sharpness, and ranks remain invariant to strictly increasing transformations at each position.
  • Method: ReWEIGH is a training-free decoding intervention that calibrates candidate tokens from unlabeled images, caches image evidence during inference, and applies bounded penalties below stable references.Offline calibration aggregates visual readouts with dense mean reciprocal rank and estimates each candidate token’s typical evidence level.
  • Results: 21.3%: ReWEIGH reduces CHAIRI across four 7B backbones, extends reductions to 11 models from six architecture families, and adds 1.33% average latency per token with cached evidence.Descriptive and general multimodal performance is largely preserved or improved, with one-time offline calibration required per model.

2 Related Work

Prior training-free decoding methods mitigate LVLM hallucinations using auxiliary reference signals derived from altered inputs, instructions, descriptions, or vision-token selections. ReWEIGH is positioned instead as an internal-representation intervention based on vocabulary projections of visual-token states.

  • Training-free decoding methods: Training-free contrastive methods derive reference logits from altered visual inputs, instructions, self-generated descriptions, or selectively retained vision tokens.These approaches typically require an auxiliary decoding trajectory.
  • Internal-representation interventions: Internal-representation interventions project hidden states into language space through the output head to expose visual grounding signals.In LVLMs, vocabulary projection can help distinguish visually grounded objects from hallucinated ones.
  • Internal-representation interventions: ReVisiT selects one vision token per decoding step and uses its constrained probability projection to refine the output.

3 Two Obstacles to Decoding with Internal Visual Evidence

Decoding with internal visual evidence faces two obstacles: probability magnitudes vary across visual positions, and rank-based evidence still differs systematically across tokens. ReWEIGH addresses them with ordinal aggregation, token-specific references, and a safeguard for uncertain calibration.

  • 3.1 Aggregating Evidence Across Visual Positions: Probability magnitudes are not directly comparable across visual positions because position-wise distribution sharpness can make identical values reflect different support.A small probability may indicate weak support or result from position-specific distribution sharpness.
  • 3.1 Aggregating Evidence Across Visual Positions: 0.045 and 0.046 are the mean absolute suppression-strength changes under probability-readout rescaling for LLaVA-1.5-7B and Qwen2.5-VL-7B, respectively.The rescaling preserves vocabulary ordering, while rank-derived suppression strengths remain exactly unchanged.
  • 3.1 Aggregating Evidence Across Visual Positions: Ranks provide a transformation-invariant basis for pooling visual evidence, and DMRR emphasizes positions ranking a token near the top while compressing deep-rank differences.DMRR is a parameter-free order-only statistic requiring no truncation depth or fitted weights.
  • 3.2 Interpreting Evidence Relative to Candidates: Token-specific calibration is needed because some tokens routinely rank near the head of visual-position rankings while others have lower typical evidence levels.The calibration uses token-specific medians fitted on 500 unlabeled MS COCO images and evaluated on 4,969 disjoint images.
  • 3.2 Interpreting Evidence Relative to Candidates: ReWEIGH registers a token only when uncertainty in its estimated reference has little effect on the edit, abstaining otherwise.This stability safeguard reduces the risk of suppressing well-grounded candidates when references are estimated from few or highly variable observations.

4 ReWEIGH: Token-Calibrated Ordinal Suppression

ReWEIGH measures image support for tokens through rank-based visual evidence, registers reliable token-specific references from unlabeled calibration images, and applies bounded suppression during decoding. The method caches image evidence after prefill, avoiding additional model forward passes while intervening only on candidates with evidence deficits.

  • Measure: DMRR aggregates reciprocal vocabulary ranks across visual positions, avoiding dependence on comparable score magnitudes while emphasizing positions where a token ranks highly.The evidence is computed from within-position rankings, with low-ranked positions contributing less.
  • Register: Calibration records visual evidence only when tokens appear as plausible top-p candidates, preserving repeated candidate occurrences in each token’s evidence multiset.The calibration images are unlabeled and disjoint from evaluation data.
  • Register: Token-specific references estimate typical candidate evidence, while a shared scale normalizes evidence across tokens.The reference b(v) is token-specific, whereas b0 supplies a pooled normalization scale.
  • Register: Tokens are registered only when their nominal 95% order-statistic range exists and uncertainty satisfies Δe(v) < 0.5; otherwise, ReWEIGH abstains.The resulting frozen table stores b0 and references b(v) for registered vocabulary tokens.
  • Intervene: Intervene suppresses only registered candidates whose evidence falls below reference, clipping each logit reduction to a maximum penalty β while leaving other tokens unchanged.Candidate sets are recomputed before editing, whereas image evidence is computed once during prefill and cached throughout decoding.

5 Experiments

Across four 7B LVLMs and multiple hallucination, grounding, reliability, and capability benchmarks, ReWEIGH reduces hallucinations while preserving content and general performance. Ablations support ordinal, token-specific, image-specific, and conservative evidence handling, with low cached-evidence overhead.

  • Visual Grounding and Hallucination Mitigation: Across four 7B LVLMs, ReWEIGH reduces CHAIRI by 10.3%–21.3% while largely preserving or improving F1.The improvement extends to 11 models across six architecture families and 7B–32B sizes, reducing both CHAIRS and CHAIRI for every model.
  • Visual Grounding and Hallucination Mitigation: On AMBER generation, ReWEIGH lowers CHAIR and hallucination rate while improving AMBER score across all four backbones.It is the only compared method that never degrades a baseline metric by more than 5% on any backbone under Table 1’s shading rule.
  • Open-ended Reliability and General Multimodal Utility: ReWEIGH improves MMHal-Bench quality and MM-Vet accuracy across all four backbones while reducing or maintaining MMHal-Bench hallucination rate.The evaluation combines hallucination, recall-oriented, and general-capability measures to avoid rewarding mere suppression of content.
  • Ablations: 44.8 to 50.0: replacing DMRR with probability pooling increases CHAIRS under a matched mean edit budget, supporting the ordinal readout.Global or shuffled token references also degrade performance, isolating the importance of token-reference correspondence.
  • Ablations: Every mismatched-evidence substitution recovers less than half of the full method’s CHAIRS reduction and pushes F1 below base decoding.Rotating cached evidence across images while holding other components fixed shows that ReWEIGH uses evidence specific to the current image.
  • Efficiency: 1.33% latency: ReWEIGH adds this per-token overhead after evidence caching, with 2.40% end-to-end overhead including online DMRR computation.Peak allocated memory increases by 0.31% under the cached-evidence protocol.

6 Conclusion

ReWEIGH is a training-free decoding intervention that aggregates vocabulary ranks across visual positions and compares candidates with token-specific references. It uses bounded evidence-deficit penalties and cached prefill-time computation to reduce hallucination while preserving or improving multimodal utility across diverse models.

  • ReWEIGH aggregates vocabulary ranks across visual positions and compares each candidate with a reliably estimated token-specific reference.This provides the decoder’s candidate-specific visual-evidence measure.
  • A bounded, one-sided penalty converts evidence deficits into logit edits, while prefill-time computation enables generation to reuse cached evidence.
  • Across four 7B backbones, ReWEIGH reduces hallucination while preserving or improving descriptive and general multimodal utility.
  • The reductions extend to 11 models across six architecture families from 7B to 32B.

Limitations

ReWEIGH requires internal model access and separate per-backbone calibration, limiting applicability through closed APIs and leaving cross-backbone transfer unestablished. Its visual-only, English-calibrated references also leave factual correction and multilingual behavior unresolved.

  • Model access and calibration: ReWEIGH cannot operate through closed APIs because it requires visual-token hidden states, output normalization, and the vocabulary head.The current implementation also uses separate calibration tables, readout layers, and intervention strengths for each backbone, without establishing transfer across backbones.
  • Knowledge limitations: Because ReWEIGH relies only on visual representations, it can suppress unsupported candidates but cannot supply facts absent from the image or backbone.The evaluated benchmarks do not test errors requiring current or specialized knowledge.
  • Language coverage: Calibration and evaluation were conducted only in English, so multilingual behavior remains unknown amid differences in tokenization, morphology, and alignment.Token-specific vocabulary references may alter candidate frequencies and visual-evidence baselines across languages.

Ethical Considerations · A Experimental and Evaluation Details · A.1 Model Checkpoints

ReWEIGH may improve LVLM output reliability but does not establish safety for high-stakes use or eliminate hallucinations. Its deployment requires internal model access, offline calibration data, and attention to licensing, privacy, provenance, and calibration bias; the experiments document their exact checkpoints in Table 4.

  • Ethical Considerations: Reducing unsupported statements does not make LVLMs safe for high-stakes use, because hallucinations may persist or change form.Medical, legal, scientific, and public-safety claims require independent verification.
  • Ethical Considerations: Users should independently verify claims produced by the method in medical, legal, scientific, and public-safety settings.
  • Ethical Considerations: Deployments require internal model states and an offline calibration corpus, while respecting checkpoint and image licenses, privacy constraints, and provenance requirements.
  • Ethical Considerations: Calibration data can encode geographic, cultural, and frequency biases that token-specific references may reproduce by favoring common visual concepts over rare ones.
  • Ethical Considerations: The method reports abstention for unstable references and cautions that lower benchmark scores do not demonstrate universal factual reliability.
  • A.1 Model Checkpoints: The paper lists exact experimental model checkpoints together with their language backbones in Table 4.Abbreviated model names from the table are used consistently throughout the paper.

A.2 Calibration Data and Operating-Point Selection … B.1 Scope of the Diagnostic Analyses

The paper uses disjoint calibration, tuning, and evaluation data with fixed benchmark-specific protocols, then scopes diagnostic analyses to within-backbone evidence properties rather than universal detector claims or cross-model absolute comparisons.

  • A.2 Calibration Data and Operating-Point Selection: 500 MS COCO train2014 images seeded 4242 calibrate token DMRR observations, while a disjoint 500-image seed-42 subset selects operating points across 18 layer-strength configurations.Calibration uses probability prefixes reaching 0.9 mass, with at least two and at most 50 candidates; selection jointly inspects CHAIRS, CHAIRI, recall, and F1.
  • A.2 Calibration Data and Operating-Point Selection: The calibration-size sweep uses nested prefixes of a frozen seed-4242 sequence, with 700- and 1,000-image sets extending the same image-ID-sorted panel.The 500-image arm exactly reproduces deployment and recovers the deployed b0, all 2,580 token references, and observation counts.
  • A.3 Evaluation Protocols: Evaluation holds prompts, decoding limits, and evaluator contracts fixed across the base model, ReWEIGH, and comparison methods, using separate benchmark splits where specified.CHAIR uses a fixed disjoint 500-image COCO val2014 manifest, the exact detailed-description prompt, 512-token generation, and official object vocabulary and synonym mapping.
  • A.3.1 CHAIR: CHAIR reports caption- and mention-level hallucination rates, corpusmicro object recall, derived precision, F1, and caption length under its official object definitions.CHAIRS measures captions containing at least one hallucinated object mention, while CHAIRI measures hallucinated generated object mentions.
  • A.3.2 AMBER: AMBER evaluates its 1,004-example generative and 14,216-question discriminative splits with exact prompts, bounded generation, released noun-scoring resources, and strict yes/no parsing.Its reported metrics include CHAIR, Cover, Hal., Cog., accuracy, precision, recall, F1, and unparsed-response counts; AMBER’s quantities are not numerically equated with COCO CHAIRI.
  • A.4 Generation Settings / A.5 Hardware and Software: Base and ReWEIGH generations use batch size 1, eager attention, one beam, no sampling, seed 42, and checkpoint-native repetition penalties, whereas VCD samples instead of decoding greedily.Experiments run on NVIDIA A100-SXM4-80GB GPUs with approximately 250 total GPU-hours across specified Python, PyTorch, Transformers, and precision environments.
  • B Additional Diagnostic Analyses / B.1 Scope of the Diagnostic Analyses: The diagnostic analyses ask which visual-readout property pools across positions and which reference makes pooled evidence token-interpretable, without claiming universal rank superiority or standalone centering benefits.Analyses focus on LLaVA-1.5-7B and Qwen2.5-VL-7B at selected layers; scale-dispersion and token-identity-share measurements cover all four backbones, with effects reported within backbone rather than across absolute evidence scales.

B.2 Rank-Preserving Scale Analysis … C Method and Implementation Details

The paper validates rank-based evidence as scale-invariant, token-calibrated, and more effective for autoregressive hallucination mitigation than detection quality or output confidence alone. Across analyses, calibrated rank evidence better identifies where interventions should act while preserving explicit limitations on causal interpretation.

  • B.2 Rank-Preserving Scale Analysis: Rank-preserving rescaling leaves within-position vocabulary order unchanged while testing sensitivity to position-wise cardinal scale.The manipulation reassigns positive scale factors among visual positions while retaining observed readout directions, isolating cardinal-scale sensitivity.
  • B.2 Rank-Preserving Scale Analysis: 0 is the maximum rank change, while probability-derived suppression shifts by 0.0445 for LLaVA and 0.0462 for Qwen; rank-derived strengths remain unchanged.Scale equalization closes only 6.94% and 8.76% of the full probability-to-rank suppression gap.
  • B.2 Rank-Preserving Scale Analysis: At every selected readout layer, fixed probability thresholds vary in rank depth across positions by at least fourfold on 5,000 development images.Rank-10 spreads correspond to 13.0×–15.2× and 7.6×–7.9× ratios in the reported intervals.
  • B.3 Token-Conditioned Reference Analysis: 66.36% of DMRR variability reflects token differences, versus 33.64% reflecting image differences for the same token in the image-balanced LLaVA decomposition.This motivates token-specific evidence references rather than a single global reference.
  • B.3 Token-Conditioned Reference Analysis: Token-specific references are evaluated out of sample using 500 calibration images and 4,969 disjoint images, with held-out token medians scored by normalized gaps.Aggregates weight tokens either equally or by held-out candidate mass.
  • B.4 Detection Quality Does Not Predict Mitigation: 0.718 versus 0.679 AUROC favors probability over rank on LLaVA-1.5-7B, yet calibrated rank lowers CHAIRS from 50.0 to 44.8 against calibrated probability.Standalone detection ordering therefore does not predict end-to-end mitigation, and the comparison does not isolate calibration as a single causal mediator.
  • B.5 From Detection to Allocation: Rank-derived assignment yields higher hallucination-linked precursor switch selectivity than probability-derived assignment on LLaVA-1.5-7B under both magnitude donors.On Qwen2.5-VL-7B, contrasts are directionally consistent but intervals include zero; the advantage concentrates on structural tokens.
  • B.6 Output Confidence and the Evidence Deficit: 13.9% of high-confidence object-mention onsets are hallucinated on LLaVA-1.5-7B, and hallucinated onsets show positive median evidence deficits while grounded onsets show none.Qwen2.5-VL-7B reports 7.3% hallucinated onsets, with directionally consistent but interval-weakened separation.

C.1 Architecture-Specific Visual Positions … D.1 Full CHAIR and AMBER Results

ReWEIGH reads visual evidence from architecture-specific positions, calibrates token-specific references, and edits candidate tokens using cached evidence. The appendices specify candidate construction, stability registration, computational costs, and complete benchmark reporting.

  • C.1 Architecture-Specific Visual Positions: Visual positions are architecture-specific: patch embeddings for LLaVA, Q-Former queries for InstructBLIP, and merged visual embeddings for Qwen2.5-VL.Readouts use final output normalization and the native vocabulary head without training an intermediate translator or probe.
  • C.2 Candidate Construction and Reference Storage: Candidate sets use the shortest next-token probability prefix reaching cumulative mass 0.9, expanded to two tokens and capped at 50 when necessary.The same candidate rule applies during calibration and inference.
  • C.2 Candidate Construction and Reference Storage: Calibration stores token-specific DMRR observations and references only for registered tokens, while inference requires neither calibration images nor their hidden states.The stored table also contains the shared normalization unit b0 and registration mask; pooled b0 is not substituted for missing entries.
  • C.3 Reference-Stability Registration: Registered references form a nominal 95% order-statistic range, used to test downstream edit stability rather than claim independent-sample confidence coverage.Tokens are not registered when no valid order-statistic index exists, with an implied minimum sample size of six.
  • LLaVA-1.5-7B: The LLaVA-1.5-7B analysis evaluates reference variation in normalized edit space and prioritizes stability of the decoder-reaching quantity over raw-median coverage claims.The appendix contrasts this engineering criterion with minimum-count registration at the frozen operating point.
  • C.4 Complete Procedure: At inference, ReWEIGH computes DMRRI during prefill, caches suppression strengths for registered tokens, and edits only candidates with registered references at each decoding step.Calibration and inference both use the Appendix C.2 candidate rule and C.3 reference interval.
  • C.5 Computational Cost: 1.33% overhead is incurred over greedy decoding with cached evidence, versus 2.40% when DMRR is computed online during prefill.The cached evidence footprint is 375 KiB per image across three readout layers, while deployment reads only the selected layer’s slice.
  • D.1 Full CHAIR and AMBER Results: Tables 17 and 18 provide complete per-benchmark CHAIR and AMBER results, including omitted metrics and the LLaVA-NeXT-7B backbone.This appendix expands the main-paper summary with full benchmark reporting.

D.2 Ablation Definitions and Diagnostics · D.3 Design-Space and Data Robustness · D.4 Scaling across Model Sizes and Architectures

Ablations show that ReWEIGH’s image-specific evidence and bounded safeguards are necessary for reliable gains, while design choices remain robust across calibration data and model families. Scaling experiments show consistent hallucination reductions from 7B to 32B models.

  • D.2 Ablation Definitions and Diagnostics: D.2 Ablation Definitions and Diagnostics: Mismatched evidence fires at 87.8% versus 80.4% for the full method, while unbounded decoding causes 765 bound-escape events and repetition up to 122.The mismatched arm performs worse, and the unbounded arm’s apparent CHAIRS gain is coupled with severe degeneration and coverage loss.
  • D.2 Ablation Definitions and Diagnostics: D.2 Ablation Definitions and Diagnostics: Cyclic-shift, fixed-derangement, and image-blind evidence each recover less than half of the full CHAIRS reduction and push F1 below base decoding.These counterfactuals hold other components fixed, isolating the importance of image-specific visual evidence.
  • D.2 Ablation Definitions and Diagnostics: D.2 Ablation Definitions and Diagnostics: Removing abstention affects about 3.5% of candidate positions, shifts CHAIRS by 0.4 points and CHAIR F1 by 0.05 points, and raises discriminative F1 from 74.81 to 76.20.The F1 increase accompanies a shift toward the majority “no” label, so the safeguard is retained as protection against unstable references.
  • D.3 Design-Space and Data Robustness: D.3 Design-Space and Data Robustness: Reference-stability, n(v) ≥5, and n(v) ≥10 registration remain within 1.2 CHAIRS points on every backbone.Reference-stability avoids the AMBER F1 drop observed with n(v) ≥5 and adapts to normalized decoder edits without an external count parameter.
  • D.3 Design-Space and Data Robustness: D.3 Design-Space and Data Robustness: GQA calibration matches MS COCO on CHAIRS, while AMBER and pooled calibration remain well below base decoding.The method therefore does not require calibration on the evaluation distribution, although corpus shift can change the operating point.
  • D.4 Scaling across Model Sizes and Architectures: D.4 Scaling across Model Sizes and Architectures: ReWEIGH reduces both CHAIRS and CHAIRI for every tested model across six architecture families and sizes from 7B to 32B.The study evaluates 11 models on 500 CHAIR validation examples using reference-stability registration with the fixed b0 scale.
  • D.4 Scaling across Model Sizes and Architectures: D.4 Scaling across Model Sizes and Architectures: F1 improves for six of 11 models, while every remaining decrease stays within 0.7 points of base decoding and coincides with lower recall.Operating points are selected by model family before evaluation on a separate 500-image COCO val2014 subset.

E Qualitative Results and Error Analysis … (b) LLaVA-1.5-7B

The qualitative analyses show that ReWEIGH suppresses image-unsupported object mentions while preserving supported descriptions, whereas removing key safeguards can cause severe decoding degeneration. Recorded traces explain the intervention’s bounded, reference-calibrated behavior, and examples span CHAIR, AMBER, and multiple backbones.

  • E.1 Mechanism Case Study: At one LLaVA-1.5-7B decoding step, ReWEIGH fully suppresses an unsupported “cars” candidate, leaves sufficiently supported tokens unchanged, and applies an intermediate penalty to partially supported candidates.The unsupported candidate satisfies DMRRI(v) ≪ b(v) and reaches normalized deficit sI(v)=1.
  • E.1 Mechanism Case Study: In a complementary LLaVA-NeXT-7B trace, ReWEIGH replaces a suppressed hallucinated object with a grounded one.The values are read directly from recorded deployed-configuration traces.
  • E.2 Additional CHAIR Examples: Across CHAIR examples, ReWEIGH selectively removes contextual, repeated-object, and peripheral hallucinations while retaining or introducing matched objects and their surrounding relations.The examples compare base and ReWEIGH responses under the same image and prompt across different backbones.
  • E.3 AMBER Examples: In the AMBER example, ReWEIGH avoids a beach-setting term flagged by the annotations while preserving the image-grounded description.The candidate was mechanically flagged from the annotation vocabulary and manually verified against the image.
  • E.4 Failure Modes of Ablated Components: 122 maximum single-object repetitions occur after removing the clipping bound, as unbounded logit displacement drives a two-token loop until the length limit.This failure inflates caption length and introduces five unsupported object classes, whereas the full method adds none beyond the base model’s two flagged objects.
  • (a) LLaVA-1.5-7B: The full bounded method produces a coherent city description with vehicles, while the unbounded ablation degenerates into repeated “bus” and “car” tokens.Figure 11 identifies the repeated tokens in the abridged unbounded caption.
  • (b) LLaVA-1.5-7B: The global-reference ablation generates a detailed living-room description, illustrating a qualitative failure mode associated with removing token-specific references.The comparison is labeled “−Token reference (global reference)” and includes objects such as a couch, chair, remote, bottle, plant, vase, and book.
Loading 2608.19075v1…