PaperlayerView exact revision on arXiv

Source-linked AI summary

SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions

Zirong Chen, Fuda Ye, Kuan Zhang, Enjun Du, Junfu Pu, Xinlei Wang, Xinyu Zuo, Lisheng Duan, Jin Ma, Yongqi Zhang

arXiv:2608.29607v1cs.CVcs.IR

TL;DR

Snap-and-ask retrieval must identify an intended entity from a potentially corrupted image and short or mistyped question, while existing benchmarks do not isolate paired robustness. The paper introduces SnapBench, a controlled paired benchmark and evaluates multimodal retrievers, finding image-side fragility, text underuse, coarse-text drag, and non-additive joint corruption effects; MOOR tests adaptive modality calibration.

  • Problem

    Existing benchmarks do not evaluate short-question, user-captured-image entity retrieval while keeping the target entity, gallery, and labels fixed across clean and corrupted inputs.

  • Method

    SnapBench pairs entity-centric queries with fixed galleries and controlled image-side, text-side, and joint corruptions, while MOOR adaptively reweights modality paths from retrieval-score reliability.

  • Results

    Image artifacts cause the largest failures; text artifacts have weaker effects in joint retrieval, while coarse-text drag and non-additive joint artifacts expose multimodal robustness failures.

  • Takeaways & Limitations

    Robust snap-and-ask retrieval requires evaluating paired multimodal corruption and calibrating modality contributions rather than relying only on clean accuracy or fixed fusion.

  • Takeaways & Limitations

    SnapBench uses programmatically generated corruptions and prioritizes controllable, annotated coverage over in-the-wild distributional fidelity and web-scale size.

Abstract

from arXiv · show

Mobile AI acts as a visual oracle, empowering users to snap a picture of something and ask for information. Snap-and-ask retrieval is now one of the most common entry points for mobile AI, yet photos are often blurry, while text questions may be short or mistyped. Existing benchmarks only test on clean inputs or do not isolate paired robustness in snap-and-ask retrieval. Therefore, we introduce SnapBench, the first paired benchmark for robust snap-and-ask multimodal retrieval, spanning 1,145 queries, 9,085 gallery items under 53 controlled corruption conditions with human annotations. We evaluate 16 multimodal retrievers, covering dual-tower encoders and embedding-based VLMs. Results show that image corruptions substantially degrade retrieval, while text corruptions mainly affect text-only retrieval and have limited impact on joint retrieval. Clean image-only retrieval often outperforms joint retrieval, indicating the coarse-text drag and the lack of cross-modal fallback under noisy inputs. SnapBench provides a controlled testbed for evaluating robust retrieval in snap-and-ask scenarios. We further propose MOOR (Modality-anchored, Outlier-aware, Optimal Reweighting), a simple adaptive fusion approach, highlighting the need for reliability-aware modality calibration in snap-and-ask retrieval.

1 Introduction

Snap-and-ask retrieval targets a specific entity from a snapped image and short question, but existing benchmarks do not isolate paired robustness to mobile artifacts. SnapBench addresses this gap with controlled paired evaluation and identifies modality-specific failure modes plus adaptive fusion.

  • Motivation: Snap-and-ask retrieval ranks the intended entity above same-category hard negatives using a snapped image and short question.The question may express only coarse intent, leaving the image responsible for discriminative evidence.
  • Benchmark gap: Existing benchmarks do not keep the target entity, gallery, and labels fixed across clean and corrupted inputs.Consequently, they cannot isolate how a mobile artifact changes retrieval for the same intent.
  • SnapBench: SnapBench contains 1,145 queries, 9,085 gallery items, and 53 controlled corruption conditions with human annotations.The paired design preserves retrieval intent, positives, and gallery while varying only observed input conditions.
  • Findings: Image artifacts cause the largest retrieval degradation, while text artifacts mainly affect text-only retrieval and have limited impact on joint retrieval.Joint systems often underuse text, and coarse questions can dilute strong visual rankings through coarse-text drag.
  • Findings: Joint image-text artifacts are non-additive, so single-modality robustness does not reliably predict robustness when both modalities are corrupted.The benchmark therefore tests paired and jointly degraded inputs rather than relying only on clean accuracy or separate stress tests.
  • Adaptive fusion: MOOR estimates modality reliability from retrieval scores and adaptively reweights modality paths without changing the retrieval architecture.Its gains over fixed fusion motivate reliability-aware modality calibration.

2 Related Work

Prior retrieval and robustness benchmarks evaluate related capabilities but generally miss entity retrieval from short questions grounded in user-captured images and paired multimodal corruption effects. SnapBench distinguishes itself by fixing the retrieval instance across conditions and corrupting both modalities jointly.

  • Image-text retrieval: MS COCO, Flickr30K, and Conceptual Captions evaluate cross-modal alignment through paired captions or alt text, not short underspecified questions about captured entities.Their task formulation emphasizes caption-style alignment rather than entity identification from snap-and-ask inputs.
  • Visual question answering: VizWiz and InfoSeek capture noisy images and underspecified questions but primarily evaluate answer generation rather than retrieving an intended gallery entity.Robustness benchmarks also typically change the test instance when applying corruptions.
  • Robustness benchmarks: SnapBench fixes the target entity, gallery, and labels across clean and corrupted conditions, so score deltas measure corruption impact directly.ImageNet-C and TextAttack instead use a new test instance per corruption, conflating label difficulty with corruption sensitivity.
  • Robustness benchmarks: Only SnapBench among the compared benchmarks corrupts both modalities jointly, enabling analysis of super-additive interactions.This distinguishes paired multimodal robustness from independent image- or text-side perturbation tests.
  • Reranking and fusion: MOOR adaptively reweights native similarity paths, whereas EviRank adds structured relevance evidence for image-candidate reranking.The methods address related but distinct post-encoding retrieval problems.

3 SnapBench

SnapBench constructs entity-centric retrieval queries with hard negatives, applies deterministic image and text artifacts only to queries, and builds human-annotated galleries. Its controlled design preserves the retrieval intent while varying mobile interaction conditions.

  • Task design: SnapBench uses coarse questions and same-category hard negatives so retrieval requires identifying the intended entity rather than matching a broad category.The gallery and labels remain fixed across clean and perturbed inputs.
  • Data construction: The construction pipeline begins with manually screened images containing visually identifiable primary entities and excludes severe clutter, low visibility, and leakage-prone overlays.An audit of 200 sampled images achieved 96% three-way agreement among raters.
  • Data construction: Gallery images are sampled with strict non-overlap from queries and paired with entity-centric captions describing primary entities and salient visual attributes.Entries lacking identifiable entities, informative captions, or uniqueness are removed.
  • Artifact simulation: Deterministic corruption operators modify only the query, leaving gallery items unchanged so each variant represents the same snap-and-ask intent.The pipeline models snap-side capture/interface distortion and ask-side text noise.
  • Artifact simulation: Image artifacts use ADD, REMOVE, DEGRADE, and TRANSFORM primitives, with 15 operators at three severity levels producing 45 conditions.Examples include overlays, cropping, blur, compression, lighting changes, rotation, and lens distortion.
  • Artifact simulation: Text artifacts use eight deterministic rule-based operators across character, word, and sentence granularities.They simulate typing errors and changes to entity or category cues in user questions.
  • Annotation: Items receive 0–3 scores on visual similarity, entity relevance, caption accuracy, and query-intent relevance; fitness ≥2 marks positives and fitness =1 marks hard negatives.The final benchmark contains 5,462 positives and 4,005 hard negatives with ten annotators and a 95% final audit pass rate.

4 Experiments

Experiments evaluate 16 multimodal retrievers across five retrieval modes and controlled clean, text, image, and joint corruption conditions. Image artifacts cause the largest degradation, coarse text can hurt joint retrieval, and joint corruption produces super-additive failures.

  • Setup: 16 multimodal retrieval models are evaluated across five query-to-gallery modes, including joint, image-only, text-only, and cross-modal retrieval.Dual-encoder joint scores use late fusion of image and text similarities.
  • Main Results: Mean R@1 falls from 63.2 to 57.9 under image corruption, while character-, word-, and sentence-level text corruption remains within 1 point of clean performance.Text-corruption means are 63.5, 63.7, and 63.9 R@1, respectively.
  • Image Artifacts: Image corruption severity drives mean R@1 drops of 0.8 points at s1, 3.8 at s2, and 11.2 at s3.At s3, remove is most damaging at 16.3 R@1 points, followed by degrade at 12.5, add at 10.9, and transform at 5.2.
  • Image Artifacts: Some image operators show cliff behavior, with little degradation before collapsing at s3; low light, mosaic, watermark, downscaling, and cropping are strongest examples.Figure 3 groups seven operators as cliff-like and eight as gradual.
  • Text Artifacts: Joint retrieval underperforms image-only retrieval for most models, with a mean coarse-text drag of −11.21 R@1 across all 16 models.Coarse questions elevate related same-category candidates and dilute visual ranking.
  • Joint Artifacts: Joint corruption produces a +7.6-point interaction gap: observed joint drops average 13.6 versus 6.0 for the naive additive prediction.The paired regime is hardest for fixed fusion because both query signals degrade simultaneously.

5 Adaptive Modality Fusion with MOOR

MOOR is a training-free adaptive fusion adapter that reweights modality score paths using frozen-encoder retrieval statistics. It improves over fixed fusion across clean and corrupted settings, supporting query-dependent modality calibration.

  • MOOR: MOOR reweights four gallery score paths per query using only frozen encoder outputs, without learned parameters or architecture changes.The paths are sII, sTT, sIT, and sTI; sII anchors the ranking.
  • MOOR: Text-involving paths are gated by Pearson correlation with the image-only anchor and scaled by score variance before normalized weighted fusion.The gating suppresses unreliable and redundant paths.
  • Evaluation: MOOR improves over fixed fusion across clean, text-corrupted, and image-corrupted settings for both dual-encoder and VLM-embedding models.The evaluation changes only the fusion method while retaining the same setup.
  • Evaluation: MOOR gains generally increase with image-corruption severity but diminish when artifacts remove the target entity itself.This establishes a practical boundary for adaptive fusion when visual evidence is destroyed.
  • Why MOOR Generalizes: Across all 16 models, per-query optimal fusion weights have standard deviations of 0.086–0.138 on a [0, 1] scale, indicating dispersed modality reliability.MOOR uses score statistics rather than model internals, supporting architecture-independent operation.

6 Conclusion

SnapBench benchmarks entity-centric snap-and-ask retrieval with unchanged targets, galleries, and annotations across clean and perturbed inputs. Results expose image fragility, coarse-text drag, and non-additive joint failures, while MOOR indicates the value of modality calibration.

  • 6 Conclusion: SnapBench preserves the same target, gallery, and annotations across clean and perturbed inputs for paired robustness evaluation.The benchmark focuses on users providing a snapped image and a text question.
  • 6 Conclusion: Evaluation of 16 retrieval models finds that image artifacts cause the largest failures, especially when they obscure or remove entity-level evidence.Text artifacts have weaker effects in joint retrieval.
  • 6 Conclusion: Adding a text question can reduce accuracy relative to image-only retrieval, demonstrating coarse-text drag in snap-and-ask retrieval.Joint image–text artifacts also interact non-additively.
  • 6 Conclusion: MOOR provides a lightweight diagnostic baseline whose gains suggest that modality calibration is important for future snap-and-ask retrieval systems.The conclusion frames calibration as a direction supported by the benchmark results.

Limitations

The paper’s findings are scoped to snap-and-ask entity retrieval, and SnapBench is a controlled benchmark rather than an in-the-wild deployment substitute. Its main limitations concern synthetic corruption distributions, benchmark scale, and MOOR’s in-distribution behavior.

  • The findings apply to snap-and-ask entity retrieval rather than mobile AI broadly.
  • SnapBench uses programmatically generated corruptions, favoring factorial controllability and reproducibility over fidelity to real-world corruption distributions.The joint distribution of real corruptions may differ from the benchmark’s marginal distributions.
  • The benchmark contains 1,145 queries and 9,085 gallery items, prioritizing dense annotations and hard-negative coverage over web-scale size.
  • MOOR is most effective when modality quality varies across queries and provides no advantage when both modalities are uniformly informative or uninformative.In those cases, MOOR reverts to fixed-fusion behavior by construction.

Ethical Considerations

The benchmark is constructed from a private image pool sourced from publicly accessible web material, with filtering and safety screening. Internal pilot analysis used de-identified data and reported only aggregate statistics.

  • The private image pool was collected from publicly accessible web sources and processed with copyright filtering, deduplication, and content safety screening.The full source pool is not released; the released benchmark contains only evaluation items, annotations, and corruptions.
  • The pilot analysis used de-identified data under the production service’s User Agreement and Privacy Policy for internal algorithm-optimization purposes.
  • Results were reported only as aggregate statistics, with no personally identifying information released.Four percent of images were reviewed solely to calibrate automated detection thresholds, with no content extracted or retained beyond that purpose.

A Pilot Analysis of Real User Inputs

The pilot analysis tests whether SnapBench’s controlled corruption taxonomy reflects real mobile uploads. Signal-level checks found frequent single- and dual-modality issues, and a separate real-upload study supported transfer of the reported patterns while preserving the benchmark’s scope boundary.

  • Real-upload validation: The analysis independently checked image artifacts using blur, low-light, overexposure or glare, and compression statistics.Thresholds were calibrated on 200 manually labeled images and selected for high precision.
  • Real-upload validation: Text questions were flagged for high out-of-vocabulary rates, fewer than two content tokens, or lacking an entity-bearing noun.These rule-based checks targeted observable input-quality issues without model-dependent semantic judgments.
  • Real-upload validation: 71% of 5,000 uploads exhibited at least one signal-level issue, while 35% exhibited issues on both modalities.These distributions motivated single-side and joint corruption conditions in SnapBench.
  • Real-upload validation: A separate study of 200 fresh real uploads supported transfer of all three synthetic-corruption predictions to production conditions.The study did not claim that real corruption joint distributions match the factorial benchmark design.
  • Evaluation design: The paired protocol compares clean and corrupted variants of the same query, entity, gallery, and labels using ΔR@k.This isolates corruption impact from differences in baseline visual quality, provided the primary entity remains identifiable.
  • Query construction: The query pipeline generated coarse entity labels and short questions, then applied rule-based filters, semantic checks, repair passes, and human spot-checking.The semantic check required entity alignment, coarse labels, well-formed questions, no identifying names, and entity-tag presence in the query.
  • Query construction: The 1,145 released queries came from 1,150 image-grounded candidates, yielding approximately 99.6% after filtering and repair.
  • Gallery construction: The 9,085-item shared gallery was built from top-30 retrieved candidates per query, excluding query images and near-duplicates.

B.2.3 Annotator Training and Quality Control

Annotation quality was managed through calibration and rolling audits, while the shared-gallery design creates a potential unjudged-positive concern. The paired evaluation protocol and empirical bound limit the effect of that concern on reported conclusions.

  • Annotator Training and Quality Control: Ten annotators completed 3,000-item calibration rounds and required at least 90% accuracy before live annotation.About 34,000 query–candidate pairs were annotated, with 20–30% of submissions re-checked during rolling audits.
  • Annotator Training and Quality Control: The final audit pass rate was 95%, providing an upper bound on residual label noise in the annotation pipeline.
  • Relevance-Judgment Completeness: Because relevance judgments are per-query over a shared gallery, an unannotated gallery item could theoretically be an unjudged positive.Top-30 candidates from strong retrievers are exhaustively annotated for each query, prioritizing the highest-risk region.
  • Relevance-Judgment Completeness: The paired clean-versus-corrupted label set makes any hypothetical missing positive cancel equally in both conditions.Consequently, the reported paired differences are unaffected by such unjudged relations under the protocol.
  • Relevance-Judgment Completeness: Under a worst-case assumption over 2,000 checked pairs, the absolute R@1 impact is bounded by ≤0.22 percentage points, while reported ΔR@1 impact is exactly zero.
  • MOOR Motivation: MOOR is motivated by per-query variation in optimal fusion weights, whose means cluster near 0.5 while standard deviations range from 0.086 to 0.138.This motivates adaptive rather than globally fixed modality weighting.

C.1.3 Step-by-Step Derivation

MOOR builds adaptive multimodal fusion by whitening embeddings, evaluating four modality paths, gating auxiliaries by rank consistency, and weighting them by score variance. Across SnapBench evaluations, this design improves robustness by shifting reliance toward text when image evidence degrades, with the Bell gate and adaptive weighting outperforming fixed alternatives.

  • Gallery-side whitening: MOOR standardizes modality embeddings before similarity computation to reduce anisotropic scale biases and hubness.Whitening is skipped for VLM2Vec-Full and VLM2Vec-V2 because it introduces numerical distortions.
  • Four similarity paths: Four query-gallery paths cover image-image, text-text, image-text, and text-image affinities, with sII as the primary path and the others as text-involving auxiliaries.Dual-encoder joint scores use late fusion, while cross-modal paths use the opposite modality’s gallery statistics for comparable inner products.
  • Bell-gated rank consistency: MOOR gates auxiliary paths by rank consistency with the primary image ranking, suppressing paths that are redundant, conflicting, or unsupported by signal.The Bell gate is highest near r = 0.5 and collapses as correlation approaches 0, 1, or negative values.
  • Variance-weighted fusion: Variance-weighted fusion downweights paths whose score vectors are flat over the gallery, then combines the gated paths into a normalized weighted score.This prevents rank-consistent paths from contributing when they provide little discriminative signal.
  • Adaptive modality calibration: As image corruption increases, MOOR raises the text-path weight share, while clean high correlation suppresses redundant text paths without explicit corruption detection.For sTI, r ≈0.8 gives g ≈0.016 under clean conditions, whereas r ≈0.5 gives g ≈0.063 after image degradation; Table 11 reports a monotonic increase in text-path share.
  • Evaluation and ablations: MOOR consistently beats fixed-fusion baselines, with CLIP gaining +6.0 R@1 on average and the Bell gate dominating six alternatives across 16 models and 53 corruption conditions.MOOR closes 40–60% of the oracle gap on most models; No Gate recovers only ∼50% of the Bell improvement, while MOOR beats wh-Img on 16/16 models under image corruption by mean +0.60pp.

D.1 Per-Model Implementation Notes

The evaluated systems use varied image and text encoders, with joint retrieval combining image- and text-similarity vectors. Results expose broad sensitivity to corruption, coarse-text drag, and heterogeneous domain difficulty.

  • Text robustness: Character-level text corruption can change many tokens while causing negligible or mildly positive mean joint R@1 change.Sentence-level semantic replacement is the text condition that meaningfully degrades retrieval.
  • Text robustness: Word-level operators can cause substantial R@1 drops despite near-zero token change rates, revealing a token-level diagnostic blind spot.This is described as a “sneaky” failure mode.
  • Implementation: Joint retrieval averages image-image and text-text similarity vectors for the evaluated embedding models.BLIP-ITM4 uses its contrastive embedding output rather than the matching score.
  • Fusion behavior: All 16 models exhibit negative clean coarse-text drag, ranging from −3.50 to −23.93 R@1 points.Fixed fusion therefore over-weights coarse text across architectures.
  • Image robustness: At severe image corruption, REMOVE is most damaging with a 26.8-point R@1 drop, while TRANSFORM is mildest because it preserves entity content.REMOVE also pushes the correct entity down 7.84 ranking positions on average.

E.3.3 Answerability vs. Robustness at High Severity

High-severity corruption separates genuine robustness failures from answerability limits. Human recognizability and severity trends show that some operators destroy target evidence, whereas others preserve identifiable targets while still exposing retrieval weaknesses.

  • Regime distinction: A severe corruption may either preserve recognizable target evidence or remove it entirely, creating distinct robustness and answerability regimes.The benchmark uses a human-recognizability check to distinguish these cases.
  • Cliff-type operators: At severity 3, cliff-type operators leave targets identifiable in only 10% of images, with 89% of their R@1 drop attributed to answerability limits.These operators include MOSAIC, WATERMARK, LOW LIGHT, DOWNSCALE, and CROPPING.
  • Gradual-type operators: At severity 3, gradual-type operators keep 95% of targets identifiable, so their retrieval losses reflect genuine robustness failures.The listed gradual operators include COMPRESSION, MOTION BLUR, and ROTATION.
  • Severity interpretation: A smooth decline from s1 to s3 indicates genuine robustness failure, whereas a sharp late-stage drop signals information collapse.This severity pattern provides a way to distinguish the two regimes.
  • Image operators: Image examples show progressively obscured entity cues as severity increases across the 15 image operators.The cases are grouped by the four image primitives.
  • Text operators: Sentence-level text replacement removes the category keyword and steers retrieval toward generic visual matches, triggering Semantic Drift.Character-level edits instead attack the entity token directly.
Loading 2608.29607v1…