PaperlayerView exact revision on arXiv

Source-linked AI summary

UNMATCH: Selective Unbalanced Token-Patch Matching for Forensic Image-Claim Verification

Xinjin Li, Lian Lian, Yuanzhe Yang, Yudi Xia, Calvin Chang Liu, Yeyun Xu, Yu Ma, Jinghan Cao, Yuruo Gong

arXiv:2609.31766v1cs.CV

TL;DR

Image-claim verification must distinguish topical plausibility from whether an image supports its specific claim. UNMATCH summarizes local support in both directions and across scales; on the Snopes subset of Fauxtography, it outperforms the MCOT adaptation on both reported metrics.

  • Problem

    Global compatibility scores and flexible interaction modules can obscure local image-claim mismatches, while the task requires testing whether predictions depend on the pairing.

    Source: §2.3 · p. 3, §3.1 · p. 3

  • Method

    UNMATCH summarizes frozen token-patch affinities in both directions at three image scales and fuses the compact descriptor with frozen global semantics.

    Source: §5 · p. 6

  • Results

    69.82 ± 1.47 Macro-F1 and 71.05 ± 1.20 balanced accuracy; UNMATCH outperforms the text-only, bidirectional cross-attention, and MCOT adaptation comparisons on both metrics.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The matched-reassigned intervention raises mean false-pair probability from .386 to .612, lowers coverage from .432 to .329, and increases discrepancy from .607 to .720.

    Source: §5 · pp. 6–7

    The detector estimates its prediction from the image and claim alone; group identifiers are used exclusively by the data splitter.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Contextual image misuse pairs an image with a misleading claim. We study image-claim correspondence in fact-checked pairs containing out-of-context reuse, visual manipulation, or both. Existing pair-based detectors often compress the two modalities into a global compatibility score or learn a highly flexible interaction module, which can obscure a decisive local mismatch. We introduce directional multiscale coverage, a compact representation that summarizes local image-claim affinity in both directions and at three spatial scales. At each scale, each direction is summarized by its mean, lower quartile, and two thresholded support ratios; the signed difference between directional means completes a nine-dimensional scale descriptor. Concatenating the three scales yields a compact local representation for a lightweight global-local classifier. Under leakage-aware three-fold, three-seed evaluation on the Snopes subset of the Fauxtography benchmark, UNMATCH achieves 69.82 Macro-F1 and 71.05 balanced accuracy, exceeding the MCOT adaptation by 2.60 and 2.16 points. A matched-reassigned intervention shows that breaking the observed pairing lowers coverage and increases both discrepancy and false-pair probability.

1 Introduction

Image-claim verification must detect local mismatches that global similarity can miss, despite pair-level labels and risks from language or visual regularities. UNMATCH exposes directional, multiscale local support in a compact representation and evaluates it with controlled comparisons and pairing intervention.

1 Introduction

  • Image-based misinformation includes out-of-context reuse, visual manipulation, or both, and requires image-language reasoning distinct from pixel-level authenticity assessment.The study targets semantic consistency in fact-checked image-claim pairs.Source: §1 · p. 1
  • Pair-level detection is difficult because global similarity can conceal a critical mismatch, token-region explanations lack supervision, and models may exploit unimodal regularities.The introduction motivates representations that expose local cross-modal support and evaluations that test dependence on pairing.Source: §1 · p. 1
  • UNMATCH represents local image-claim support in both directions and across three scales, preserving asymmetric evidence in a fixed 27-dimensional descriptor.Each scale contributes four statistics per direction and a signed mean gap; the descriptor is fused with frozen global pair features.Source: §1 · pp. 1–2
  • The evaluation uses leakage-aware grouped outer cross-validation, keeping related articles, claims, and duplicate images within partitions and assigning each pair one out-of-fold prediction per seed.Comparisons include text-only, cross-attention, an MCOT adaptation, representation controls, and a matched-reassigned intervention.Source: §1 · p. 2
  • The study evaluates 838 Snopes pairs with leakage-aware three-fold, three-seed cross-validation and examines pairing sensitivity alongside predictive gains.Its contributions include comparisons against global-only, shuffled, and fixed global-projection representations.Source: §1 · p. 2

2 Related Work

Contextual image misuse is a correspondence problem: topical overlap does not establish that an image supports its specific claim. UNMATCH instead makes local support explicit in both directions and across spatial scales, within a pair-only representation.

2.1 Contextual image misuse and pair-only verification

  • Contextual misuse benchmarks show that topical overlap is insufficient: an image must support the specific statement, while retrieval-based systems can recover chronology, provenance, and background knowledge beyond the pair.UNMATCH operates at the pair-only boundary and extracts correspondence evidence from the input image and claim.Source: §2.1 · p. 2
  • The evaluation compares text-only prediction under group-disjoint folds and intervenes on observed image-claim assignments to address unimodal regularities in benchmark data.VERITE is cited for showing that such regularities can make models appear successful without using the image-text relation.Source: §2.1 · p. 2

2.2 Global vision–language alignment and fine-grained interaction

  • Global vision-language alignment captures dominant scene and topic but removes the location and distribution of support; SigLIP token and patch states therefore serve as affinity features, not calibrated grounding probabilities.FILIP's bidirectional token-wise maximum similarity inspires UNMATCH's operator, while UNMATCH uses frozen SigLIP hidden states and summarizes affinity distributions.Source: §2.2 · pp. 2–3
  • Late-interaction methods preserve local matches, but their learned representation spaces are expressly trained through fine-grained alignment objectives, unlike UNMATCH's frozen SigLIP features.FILIP and FineLIP illustrate this line of work.Source: §2.2 · pp. 2–3

2.3 Directional multiscale coverage

  • UNMATCH addresses gaps in granularity, direction, and capacity: global alignment suppresses local disagreement, averaged maxima lose support-distribution shape, and learned fusion leaves evidence organization to the classifier.Its design retains unsupported claim elements, unexplained visual regions, and evidence at different spatial extents.Source: §2.3 · p. 3
  • UNMATCH preserves directional coverage distributions across native, intermediate, and coarse grids in a 27-dimensional descriptor, without learned grounding or scale-specific alignment.Its mean affinity, lower-tail affinity, support mass, and signed directional gap retain complementary evidence before classification.Source: §2.3 · p. 3

3 Method

UNMATCH makes local image–claim correspondence explicit through deterministic, bidirectional coverage at three spatial scales, then combines that evidence with global pair features. A compact classifier uses the fused representation to assess pair compatibility.

3.1 Problem formulation and model overview

  • The classifier combines global pair semantics with directional coverage, making local correspondence explicit before task-specific prediction.Global features capture broad pair content, while coverage tests support across claim elements, image regions, and scales.Source: §3.1 · p. 3, §3.4 · p. 5

3.2 Frozen global and multiscale local representation

  • Frozen vision–language features provide global embeddings and local text-token and image-patch states for correspondence analysis.The image representation uses native 14×14 patches and pooled 7×7 and 2×2 grids.Source: §3.2 · p. 3

3.3 Directional multiscale coverage

  • At each of three grids, directional affinity distributions are summarized by their mean, lower quartile, two threshold support masses, and signed mean gap in a 27-dimensional descriptor.The grids provide native, intermediate, and coarse spatial support.Source: §3.2 · p. 3, §3.3 · p. 4

3.4 Global–local prediction and mechanism diagnostics

  • The controls isolate coverage’s contribution, while held-out image reassignment tests pairing sensitivity through coverage, discrepancy, and false-pair probability.Reassignment changes image-to-claim pairing without retraining the model and prohibits assignments within the same leakage group.Source: §3.4 · p. 5

4 Experiments and Results

The leakage-aware evaluation compares UNMATCH with pair-only baselines and representation controls, then tests pairing sensitivity by reassignment. UNMATCH leads both baseline metrics, while the controls and intervention examine the role of observed correspondence.

4.1 Experiment Setup

  • Evaluation uses stratified group-disjoint three-fold cross-validation with three seeds, one out-of-fold prediction per pair per seed, and a fixed 0.5 decision threshold.All methods use frozen SigLIP representations and only the input image and claim; Macro-F1 is primary under class imbalance, with balanced accuracy as a complementary measure.Source: §4.1 · p. 5

4.2 Comparison with Baselines

  • 69.82 ± 1.47 Macro-F1 and 71.05 ± 1.20 balanced accuracy; UNMATCH beats bidirectional cross-attention by 5.23 and 4.55 points and MCOT by 2.60 and 2.16 points, respectively.These comparisons use the same partitions and pair-only observation setting.Source: §4.2 · p. 6

4.3 Representation Controls

  • Removing the local descriptor yields 65.95 Macro-F1 and 67.35 balanced accuracy, while shuffling local correspondence lowers the full model by 2.97 and 3.06 points, respectively.Global-only and fixed global-projection controls also trail the full model by 1.91/1.73 and 2.45/2.47 points on Macro-F1/balanced accuracy, respectively.Source: §4.3 · p. 6

4.4 Pairing-Sensitivity Analysis

  • Reassigning images among held-out true pairs raises mean false-pair probability from .386 to .612, lowers coverage from .432 to .329, and increases discrepancy from .607 to .720.The corresponding AUC values are .811 for false-pair probability, .789 for negative coverage, and .804 for discrepancy; pairs are reassigned without retraining or changing the held-out image and claim sets.Source: §4.4 · p. 6

5 Conclusion

UNMATCH makes local image–claim support explicit through a fixed, directional multiscale descriptor, separating representation design from classifier capacity. On the Snopes subset of Fauxtography, it outperforms the comparison methods, and matched–reassigned results show sensitivity to image–claim pairing.

5 Conclusion

  • UNMATCH converts frozen token–patch affinities into directional multiscale coverage, preserving unsupported claim elements, unexplained image regions, and scale-dependent changes suppressed by a single global score.Its matching is selective and unbalanced: each token or patch retains its strongest counterpart, with both query directions computed independently and without a one-to-one constraint.Source: §5 · p. 6
  • The fixed descriptor and shared feature cache separate representation design from classifier capacity, making performance changes attributable to representation choices rather than a larger trainable interaction module.Controls test whether pairing, direction, and multiscale organization carry information beyond the global representation.Source: §5 · p. 6
  • Reassigning images among held-out true pairs raises false-pair probability from .386 to .612, lowers coverage from .432 to .329, and increases discrepancy from .607 to .720.The intervention preserves the held-out image and claim sets while changing their assignment; the measured shifts show that the descriptor and final prediction respond to the image–claim relationship.Source: §5 · pp. 6–7, Table 3 · p. 7
  • 69.82 ± 1.47 Macro-F1 and 71.05 ± 1.20 balanced accuracy: UNMATCH outperforms the text-only, cross-attention, and MCOT adaptation comparisons on both metrics.These results use the shared leakage-aware three-fold, three-seed protocol on the Snopes subset of Fauxtography.Source: §5 · p. 6, Table 1 · p. 7
  • UNMATCH offers a lightweight alternative to flexible cross-modal fusion and can serve as a pair-level component in pipelines that also use retrieval, provenance, or external evidence.The paper presents selective unbalanced token–patch matching as a practical formulation for contextual image–claim verification.Source: §5 · p. 7

Related papers

Loading 2609.31766v1…