PaperlayerView exact revision on arXiv

Source-linked AI summary

WIDE: Wildcard Inference with Dynamic Expansion for Cross-Modal Generative Retrieval

Teng Guo, Xin Wang, Jiayou Xu, Keying Zhou, Jifeng Shen, Haoxin Ruan

arXiv:2609.03554v1cs.CVcs.AI

TL;DR

Cross-modal generative retrieval struggles because sparse queries cannot determine all details encoded in dense visual identifiers, causing forced hallucination under trie-constrained decoding. WIDE calibrates uncertainty, replaces unreliable decisions with wildcard expansion, and re-ranks the expanded candidates. Experiments on M-BEIR and diverse cross-modal tasks show clear improvements over existing generative retrieval methods.

  • Problem

    Cross-modal generative retrieval faces information asymmetry between concise queries and dense candidates, leaving some identifier details unsupported by the query.

  • Method

    WIDE uses Adaptive Entropy Thresholding for uncertainty calibration, Asymmetry-aware Wildcard Decoding for dynamic expansion, and Blind-Spot Re-ranking with hybrid signals.

  • Results

    WIDE provides clear improvements over existing generative retrieval methods across diverse cross-modal retrieval tasks.

  • Takeaways & Limitations

    Explicitly handling uncertain identifier layers preserves alternative retrieval paths and effectively addresses cross-modal information asymmetry.

  • Takeaways & Limitations

    WIDE adds decoding and re-ranking overhead, partially relies on embedding-based semantic discrimination, and may activate wildcards unnecessarily for noisy or ambiguous queries.

Abstract

from arXiv · show

Generative retrieval has demonstrated significant success by unifying representation learning and search into a single sequence-to-sequence generation task. However, extending this paradigm to cross-modal retrieval reveals a critical challenge arising from the inherent information asymmetry across different modalities, such as the gap between concise text queries and dense visual candidates. This structural mismatch causes the autoregressive decoder to suffer from forced hallucination when generating identifiers via standard trie-constrained beam search, where the model is severely penalized for failing to guess fine-grained details absent from the query, allowing irrelevant candidates to hijack top rankings. To address this issue, we propose Wildcard Inference with Dynamic Expansion (WIDE). WIDE employs Adaptive Entropy Thresholding (AET) to calibrate layer-specific uncertainty boundaries offline. During the decoding generation phase, Asymmetry-aware Wildcard Decoding (AWD) detects semantic blind spots and emits wildcards instead of forced deterministic identifiers, dynamically expanding the search space without incurring log-probability penalties. Finally, Blind-Spot Re-ranking (BSR) evaluates the expanded candidate pool using a hybrid scoring mechanism that combines discrete generation confidence with continuous semantic similarity. Extensive experiments on the M-BEIR benchmark demonstrate that WIDE outperforms state-of-the-art generative retrieval methods, effectively suppressing forced hallucination while maintaining compact index structures.

1 Introduction

Cross-modal generative retrieval suffers from information asymmetry: sparse text queries cannot specify all details encoded in visual identifiers, causing forced hallucination during constrained decoding. WIDE addresses this by calibrating uncertainty, emitting wildcards at semantic blind spots, and re-ranking expanded candidates.

  • Motivation: Cross-modal retrieval exposes a mismatch between sparse text queries and dense visual candidates, disrupting autoregressive identifier generation.Visual details absent from the query can correspond to identifier layers that the decoder cannot reliably infer.
  • Motivation: Forced hallucination penalizes correct retrieval paths when trie-constrained decoding forces deterministic choices under uncertainty.Probability mass spreads across candidates, causing accumulated log-probability drops and allowing irrelevant candidates to rank higher.
  • WIDE: WIDE combines offline uncertainty calibration, wildcard decoding, and adaptive candidate evaluation to avoid penalizing uninformed identifier layers.AET defines layer-specific thresholds, AWD emits wildcards, and BSR evaluates the resulting candidate sets.
  • Results: Experiments on M-BEIR show strong performance relative to existing generative retrieval methods while keeping the expanded search space highly constrained.The analysis attributes this to wildcard inference with dynamic candidate expansion.
  • Results: WIDE achieves state-of-the-art generative retrieval performance across diverse cross-modal benchmarks while addressing information asymmetry.Experiments are reported as extensive and show that the framework effectively handles the identified failure mode.

2 Related Work

Related work establishes generative retrieval, cross-modal identifier learning, hybrid re-ranking, trie-constrained decoding, and entropy-based uncertainty estimation as foundations for WIDE. The paper positions WIDE as using uncertainty to guide structured candidate expansion rather than relying on unguided forced guesses.

  • Generative retrieval: Generative retrieval replaces conventional match-and-rank pipelines with direct identifier generation and has been extended from text to cross-modal settings.Cross-modal extensions must address representational differences between heterogeneous modalities.
  • Cross-modal retrieval: Multimodal generative retrieval methods learn modality-invariant semantic anchors to improve identifier generation across heterogeneous inputs.GENIUS is cited as a representative generalized multimodal framework.
  • Hybrid retrieval: Hybrid generative retrieval systems generate initial identifiers and refine them with additional similarity signals.These designs retain discrete-generation efficiency while adding complementary continuous representations.
  • Constrained decoding: Trie-constrained beam search guarantees valid registered identifiers but forces a specific decision even when the query lacks sufficient evidence.The prefix tree restricts generation to valid database paths while preserving the commitment problem.
  • Uncertainty estimation: Prior uncertainty studies support predictive entropy as a reliability signal, motivating its use to calibrate decoding interventions.WIDE turns output-distribution uncertainty into structured candidate expansion.

3 Preliminary

The preliminary formulation represents candidates with hierarchical discrete identifiers generated through trie-constrained autoregressive decoding. Cross-modal information asymmetry leaves some quantization levels weakly grounded, where forced token selection can penalize correct candidates and promote distractors.

  • Task formulation: Cross-modal retrieval uses multimodal queries and a heterogeneous candidate database, with each candidate mapped to a discrete identifier sequence.The retrieval objective maximizes conditional generation probability subject to valid identifier structure.
  • Residual quantization: Residual quantization compresses dense candidate embeddings into hierarchical codebook identifiers whose shallow levels capture dominant components and deeper levels encode finer details.The quantized representation is formed by sequentially selecting code vectors across residual levels.
  • Information asymmetry: Text queries provide uneven semantic coverage across quantization levels because they specify selected attributes while omitting many visual details.This varying coverage disrupts subsequent identifier generation.
  • Trie-constrained decoding: Autoregressive inference explores a prefix trie and restricts each step to admissible child nodes, evaluating hypotheses by cumulative log-probability.The constraint ensures outputs correspond to registered candidates.
  • Forced hallucination: When query evidence is insufficient, the decoder becomes less concentrated and relies more on prefix-dependent statistical priors than query-specific evidence.Standard decoding still forces a specific token, creating forced hallucination under cross-modal asymmetry.
  • Forced hallucination: Forced token selection can incur a penalty approaching −log |T_k| under near-uniform uncertainty, displacing correct candidates with irrelevant distractors.The resulting failure motivates interventions that quantify query coverage, express uncertainty, and evaluate candidate sets adaptively.

4 Methodology

WIDE integrates offline uncertainty calibration, wildcard-aware decoding, and hybrid re-ranking into generative retrieval. The method preserves reliable identifier constraints while bypassing uncertain levels and dynamically expanding candidate search.

  • Framework overview: WIDE combines AET, AWD, and BSR as sequential modules within the generative retrieval inference pipeline.AET calibrates uncertainty, AWD modifies decoding at blind spots, and BSR produces final retrieval rankings.
  • Adaptive Entropy Thresholding: AET estimates level-specific predictive entropy offline to provide uncertainty references that account for differing admissible trie sizes across RQ levels.The thresholds are computed under teacher forcing using the true identifier prefix.
  • Asymmetry-aware Wildcard Decoding: AWD emits a wildcard when real-time entropy exceeds its threshold, recording the level as a blind spot without adding a log-probability penalty.When uncertainty is below the threshold, AWD retains standard token scoring; otherwise, it preserves the beam while marking the uncertain level.
  • Asymmetry-aware Wildcard Decoding: Wildcard-triggered trie expansion activates valid child branches, while confident generations at later levels resume pruning and retain reliable identifier constraints.Candidates match the beam’s confident codes at non-wildcard levels while wildcard levels remain unconstrained.
  • Blind-Spot Re-ranking: BSR ranks the expanded candidate pool by combining normalized discrete generation confidence with continuous embedding similarity.Its interpolation weight increases with the source beam’s blind-spot proportion, shifting emphasis toward semantic similarity for less-specific queries.

5 Experiments

The experiments evaluate WIDE’s ability to address semantic blind spots under cross-modal asymmetry against embedding-based and generative retrieval methods.

  • Evaluation focus: Experiments benchmark WIDE against state-of-the-art embedding-based and generative retrieval paradigms for cross-modal semantic blind spots.The evaluation focuses on whether WIDE resolves uncertainty arising from cross-modal asymmetry.

5.1 Experimental Setup

The evaluation uses the broad M-BEIR benchmark and compares WIDE with a GENIUS-based generative retrieval baseline under a specified training configuration.

  • Datasets and metrics: M-BEIR aggregates 10 datasets and 5.6 million candidate items across eight multimodal retrieval tasks.The benchmark includes MS-COCO, VisualNews, Fashion200K, NIGHTS, FashionIQ, CIRR, WebQA, OVEN, InfoSeek, and EDIS.
  • Baseline: The primary generative baseline follows GENIUS with residual quantization and autoregressive decoding, using fine-tuned CLIP variants with score-level fusion.The baseline is described as a directly analogous two-stage architecture.
  • Training strategy: Training uses AdamW with batch size 256, peak learning rate 1 × 10^-4, cosine decay, 20 RQ epochs, and 30 decoder epochs.The RQ module and autoregressive decoder are trained sequentially.

5.2 Overall Retrieval Performance

WIDE achieves state-of-the-art generative retrieval performance across almost all tasks, with notable gains on asymmetric reasoning and image retrieval benchmarks. It also narrows the gap with embedding-based methods while remaining competitive with a much larger multimodal model.

  • Comparison with embedding-based retrieval: WIDE achieves comparable performance to BLIP-FF and CLIP-SF on NIGHTS and EDIS without exhaustive similarity computation over the entire database.The authors present this as reducing the performance gap between generative and embedding-based retrieval while preserving a unified prefix trie.
  • Comparison with multimodal foundation models: WIDE remains competitive with U-MARVEL despite using a lightweight T5-small decoder instead of a multimodal foundation model with billions of parameters.U-MARVEL achieves superior benchmark performance but has a much larger inference computational footprint.

5.3 Ablation Study

The ablation study evaluates WIDE by incrementally adding wildcard decoding, adaptive entropy thresholds, and BSR re-ranking. Each component improves retrieval, with BSR providing further gains after candidate expansion.

  • Experimental Setup: The study evaluates incremental integration of wildcard decoding, adaptive thresholding, and re-ranking against a standard trie-constrained beam-search baseline.The baseline applies no uncertainty intervention.
  • Component Ablations: Wildcard decoding improves retrieval by 1.4–2.6 percentage points across MS-COCO, VisualNews, WebQA, and FashionIQ.Replacing uncertain identifier generations with wildcards preserves relevant paths that trie-constrained decoding might otherwise prune.
  • Component Ablations: Adaptive thresholding adds 0.3, 1.9, 2.0, and 0.8 percentage points on MS-COCO, VisualNews, WebQA, and FashionIQ, respectively.Level-specific thresholds more accurately identify intermediate semantic blind spots than a fixed threshold.
  • Component Ablations: BSR further improves performance by 2.6, 3.5, 3.2, and 2.8 percentage points on MS-COCO, VisualNews, WebQA, and FashionIQ, respectively.It combines constrained-level generation confidence with continuous embedding similarity to distinguish expanded candidates.

5.4 Analysis

The analysis examines how WIDE allocates uncertainty intervention and expands candidates. Wildcards are triggered progressively across deeper layers while preserving strong identifier constraints and improving qualitative retrieval rankings.

  • Adaptive Uncertainty Intervention: Deeper quantization layers exhibit greater entropy than shallow layers, reflecting higher uncertainty when generating identifiers.Shallow levels align more closely with text-query semantics, whereas deeper levels encode finer visual details.
  • Adaptive Uncertainty Intervention: Wildcard trigger ratios rise smoothly across layers rather than concentrating in particular architectural levels.The trajectory reflects activation based on the gap between real-time predictive entropy and the adaptive threshold.
  • Candidate Expansion: Wildcard decoding reduces the candidate space by 99.76% relative to the database and activates at an average of 1.97 identifier levels.Remaining identifier levels continue constraining candidate matching after uncertain levels are relaxed.
  • Candidate Expansion: WebQA has an average expansion size of 276.8 but a P95 of 2.0, indicating a skewed distribution dominated by small candidate sets.Similar skewed behavior appears on CIRR, VisualNews, and EDIS.
  • Qualitative Retrieval Analysis: WIDE uses hybrid re-ranking and wildcarded identifiers to recover missed ground truths, refine rankings, and mitigate visually similar distractors.Figure 5 compares baseline and WIDE top-5 results, with wildcarded identifiers marking replaced levels.

6 Limitations

WIDE improves retrieval accuracy but introduces additional decoding and re-ranking overhead. Its entropy-based intervention can also activate unnecessarily when uncertainty reflects noisy or ambiguous queries or insufficient decoder confidence.

  • Computational and Modeling Constraints: WIDE incurs additional decoding and re-ranking overhead compared with standard trie-constrained beam search.The overhead remains despite restricted wildcard activation.
  • Computational and Modeling Constraints: Entropy deviations may trigger unnecessary wildcard activation when high uncertainty comes from noisy or ambiguous queries or insufficient decoder confidence.The method therefore does not uniquely attribute every high-uncertainty state to a semantic blind spot.

7 Conclusion

WIDE addresses forced hallucination in cross-modal discrete identifier generation by relaxing unreliable decoding decisions and preserving alternative retrieval paths. Its hybrid re-ranking combines generative evidence with continuous semantic similarity, and experiments report clear improvements across diverse M-BEIR tasks.

  • Conclusion: WIDE identifies forced hallucination as a failure mode caused by information asymmetry in cross-modal discrete identifier generation.The framework responds by relaxing unreliable decoding decisions and preserving alternative retrieval paths.
  • Conclusion: WIDE combines reliable generative evidence with continuous semantic similarity to distinguish candidates expanded through wildcard inference.This hybrid re-ranking strategy retains the discrete retrieval formulation while adding continuous semantic comparison.
  • Conclusion: Experiments on the M-BEIR benchmark show clear improvements over existing generative retrieval methods across diverse cross-modal retrieval tasks.The conclusion attributes the result to explicitly handling uncertainty while retaining discrete retrieval.
Loading 2609.03554v1…