PaperlayerView exact revision on arXiv

Source-linked AI summary

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

Subhadeep Roy, Christoph Leiter, Steffen Eger

arXiv:2610.16441v1cs.CVcs.AIcs.CL

TL;DR

Image–text evaluation can obscure whether specific semantic details are supported, and natural-language rationales can be difficult to verify against visual evidence. VisExMEM evaluates localized atomic claims and derives a global score from their support. Across eight benchmarks, it leads compared open-source evaluators on the composite, while AGITA-Bench enables global and claim-level human evaluation.

  • Problem

    Global image–text scores can hide local semantic errors, and rationales may not show whether the visual evidence and decisions behind them are correct.

    Source: §1 · p. 1

  • Method

    VisExMEM decomposes captions into atomic claims, localizes visual evidence, scores claim support, averages those scores globally, and generates explanations from the same evidence and decisions.

    Source: §1 · pp. 1–2

  • Results

    Across eight benchmarks, VisExMEM-9B has the highest composite among compared open-source evaluators, exceeding VQAScore by an 11.7-point composite difference.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The results show that compact open evaluators can combine strong alignment judgment with interpretable evidence and reproducible local inference.

    Source: §7 · p. 9

    VisExMEM remains limited by fine-grained verification and evidence-localization errors, costs more inference than lightweight embedding metrics, and targets semantic alignment rather than image quality.

    Source: §7 · p. 9, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explainable metrics provide rationales, but the visual evidence supporting those rationales can still be difficult to verify. We introduce VisExMEM, a reference-free evaluator that decomposes captions into atomic claims, localizes visual evidence, verifies each claim, and averages claim-level support into a global score. Explanations use the same evidence and decisions. We train Qwen3.5-based 4B and 9B models on 100K samples with multi-task SFT, counterfactual refinement, and evidence-aware GDPO. Across eight benchmarks, VisExMEM-9B achieves the highest descriptive composite among open-source evaluators in our comparison, exceeding VQAScore by 11.7 points while remaining competitive with GPT-5.6 Sol. We also introduce AGITA-Bench, containing 600 human-annotated image-text pairs and 8.7K atomic claims for global and claim-level evaluation. For untuned Qwen3.5 and GPT-5.6 Sol controls, decomposed prompting raises the same composite by 3.6-5.1 points relative to holistic prompting. Human evaluation shows that VisExMEM explanations outperform prior explainable metrics and approach GPT-5.6 Sol.

1 INTRODUCTION

Global image–text scores can miss local semantic errors, while fluent rationales may not reveal whether the evidence and decisions behind a score are correct. VisExMEM scores localized atomic claims and introduces AGITA-Bench to evaluate both global and claim-level agreement with human judgments.

1 INTRODUCTION

  • Global image–text scores can miss local errors in counts, attributes, actions, relations, and subject–object roles.These fine-grained and compositional errors motivate evaluation beyond a single global score.Source: §1 · p. 1
  • A strong score or fluent rationale does not establish that the metric used the right visual evidence or responded to the specific constraint in error.Global score correlation alone does not show whether a metric responds to a particular visual or linguistic constraint.Source: §1 · p. 1
  • VisExMEM localizes evidence for atomic claims, maps each claim’s Yes/No log-probability margin to support, and averages support scores into a global score.Explanations are generated afterward from the same evidence and fixed claim decisions, so they do not determine the score.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • AGITA-Bench contains 600 human-annotated cases across I2T and T2I settings and supports evaluation of global ratings and claim-level support judgments.Cases span easy, medium, and hard compositional settings.Source: §1 · p. 2
  • Training uses multi-task SFT, counterfactual refinement, and evidence-aware GDPO to establish structured evaluation, sensitivity to minimal semantic changes, and reliance on localized evidence.Counterfactual refinement changes a semantic constraint while keeping remaining content fixed.Source: §1 · p. 2

2 RELATED WORK

Prior image–text metrics range from whole-pair scoring to decomposed semantic checks, while explainable and grounding-oriented work explores how to expose evaluation criteria and evidence. VisExMEM evaluates atomic claims with localized evidence, and AGITA-Bench pairs global ratings with claim-level judgments.

2 RELATED WORK

  • Reference-free evaluation includes whole-pair embedding and preference metrics, while TIFA, DSG, and T2I-FineEval decompose text or image content into smaller semantic units.VQAScore uses generative VQA likelihood; VisExMEM follows the fine-grained setting by evaluating atomic caption claims.Source: §2 · p. 2
  • FLEUR and VIEScore provide natural-language rationales, while EXPERT and HarmonicEval expose structured criteria; REVEALER performs element-level alignment through grounding, reasoning, and scoring.VisExMEM instead uses localized visual evidence to score atomic claims, then generates explanations from the same evidence and fixed decisions.Source: §2 · pp. 2–3
  • Existing diagnostic benchmarks probe compositionality, fine-grained faithfulness, counting, and preference, while AGITA-Bench pairs a global rating with claim-level support judgments for each case.This pairing allows overall agreement and individual claim decisions to be evaluated together.Source: §2 · p. 3

3 VISEXMEM

VISEXMEM evaluates image–caption alignment by grounding atomic-claim verification in localized visual evidence, then averaging claim support into a global score. Its three-stage training teaches structured evaluation, sensitivity to semantic changes, and dependence on visual evidence.

3.1 EVIDENCE-GROUNDED VERIFICATION AND SCORING

  • Each caption is decomposed into typed atomic claims, and GroundingDINO localizes relevant regions while the full image preserves scene context.Claim types include objects, attributes, actions, relations, counts, and negation.Source: §3.1 · p. 3
  • A Yes/No log-probability margin is mapped to each claim’s support score, and their arithmetic mean forms the global score.Explanations are generated from fixed claim records and cannot alter the score.Source: §3.1 · p. 3

3.2 TRAINING

  • Training data are converted into typed claims and localized evidence cues, yielding 40K SFT examples, a 20K counterfactual pool, and 40K Stage 3 examples.The Stage 3 examples are split equally between semantic-verification and causal-evidence supervision.Source: §3.2 · pp. 3–4
  • Multi-task SFT supervises decomposition, claim verification, evidence selection, and explanation generation; later stages replay earlier supervision.The three-stage procedure adds counterfactual refinement for minimal semantic changes and evidence-aware GDPO for verification tied to localized evidence.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Counterfactual refinement contrasts supported claims with minimally changed contradicted counterparts, training positive and negative verification margins.This is supervised margin training, not reinforcement learning; decomposition and verification examples are replayed.Source: §3.2 · p. 4
  • Evidence-aware GDPO uses semantic correctness for its group-relative policy signal and a separate causal objective to increase dependence on selected visual evidence.The causal objective compares selected evidence with matched random regions and uses a 7:2:1 GDPO:counterfactual:SFT update mixture.Source: §3.2 · pp. 4–5

4 AGITA-BENCH

AGITA-Bench is a 600-pair benchmark for evaluating human agreement with image–text judgments at both global and atomic-claim levels. It spans image-to-text and text-to-image settings and three compositional difficulty levels.

4 AGITA-BENCH

  • The benchmark contains 300 real-image I2T pairs and 300 T2I pairs, balanced across 200 easy, 200 medium, and 200 hard cases by semantic complexity.It contains 8,735 claims, and is excluded from training, calibration, and model selection.Source: §4 · p. 5
  • Annotators provide global-alignment ratings from 1–5 and atomic-support ratings from 0–4; mean scores are 3.64/5 and 3.35/4, respectively.They judged 93.35% of claims reasonable as written; mean pairwise quadratic-weighted κ was .602 for global alignment and .564 for atomic support.Source: §4 · p. 5
  • Figure 3 summarizes global-alignment and atomic-support distributions by direction and difficulty, with means above the bars, alongside decomposition quality.I2T ratings remain relatively high across complexity levels, whereas T2I global and atomic ratings shift lower as complexity increases.Source: §4 · p. 5

5 EXPERIMENTS

The experiments compare image–text evaluators across eight established benchmarks, with additional protocols for AGITA-Bench and out-of-distribution evaluation. They also assess explanation quality and grounding using human ratings.

5 EXPERIMENTS

  • Evaluation uses benchmark-specific measures including group scores, pairwise and preference accuracy, rank correlation, counting accuracy, and AGITA-Bench Spearman’s ρ.The paper also evaluates out-of-distribution generalization on VisPlotBench, CulturalFrames, and ProtoBias.Source: §5 · p. 6
  • The metric comparison includes embedding-, preference-, explainability-, structured-, and VQA-based evaluators, plus controlled holistic and decomposed prompting conditions.REVEALER and T2I-FineEval are omitted from the controlled benchmark comparison; the omission is discussed in Appendix A.5.2.Source: §5 · pp. 6–7
  • The main-results table compares prior alignment metrics, controlled VLM-as-a-Judge baselines, and VISEXMEM across eight established benchmarks, with higher scores indicating better performance.The reported average is a descriptive composite of different benchmark statistics.Source: Table 1 · p. 7, §5 · p. 7
  • Human raters assess explanation correctness, visual grounding, clarity, usefulness, and unsupported factual content, plus localization accuracy and sufficiency where applicable.Ratings use a 1–5 scale.Source: §5 · p. 7

6 RESULTS & ANALYSIS

VISEXMEM-9B leads open-source evaluators on the eight-benchmark composite, while structured evaluation improves results across tested backbones. Human evaluations, T2I results, and training ablations further characterize its performance and limitations.

Comparison with Existing Metrics

  • 11.7 points is VISEXMEM-9B’s composite advantage over VQAScore; it has the highest composite among open-source evaluators in the comparison.Its composite is 72.2, versus 74.0 for GPT-5.6 Sol (Hol.) and 77.6 for GPT-5.6 Sol (Dec.).Source: §6 · p. 7

Structured Evaluation Helps Across Backbones

  • +5.0, +5.1, and +3.6 points are the average gains from structured evaluation for Qwen3.5-4B, Qwen3.5-9B, and GPT-5.6 Sol, respectively.Gains vary by benchmark; the controlled comparisons show improvement across all three backbones.Source: §6 · p. 8

T2IScoreScore Performance

  • On Real, VISEXMEM-9B achieves the highest ordering score while maintaining strong separation; both variants remain in the high-ordering, high-separation region across All, Synth, Nat, and Real.Figure 5 plots ordering (rankm) against separation (sepm), with higher values better on both axes.Source: §6 · p. 8, Figure 5 · p. 8

Fine-Grained Human Alignment on AGITA-Bench

  • 54.7 is VISEXMEM-9B’s strongest open-source global average on AGITA-Bench; its atomic average is 44.9, compared with 64.7 for GPT-5.6 Sol (Dec.).The Hard-versus-Medium correlation pattern should not be read as Hard being easier: in T2I, Hard has more claims (15.72 vs. 8.58) and lower human alignment (2.99 vs. 3.49).Source: §6 · p. 8, Table 2 · p. 8

Explanation Quality Analysis

  • 4.75 diagnostic usefulness and 10.3% unsupported content are VISEXMEM-9B’s explanation ratings, close to GPT-5.6 Sol’s 4.85 and 8.1%.Its usefulness exceeds EXPERT (3.65) and VIEScore (2.13), with less unsupported content than either; localized evidence also rates highly for accuracy and sufficiency.Source: §6 · p. 9, Figure 6 · p. 9

Training-Stage Ablation

  • +8.8/+9.0 points are the average gains from counterfactual refinement over SFT for 4B/9B; joint GDPO then adds +3.3/+4.5 and performs best on all five benchmarks at both scales.Verification-GRPO and Causal-GRPO alone provide smaller, less consistent gains.Source: Table 3 · p. 8, §6 · p. 9

Efficiency, Cost, and Reproducibility

  • 8.9 s per image-text pair is VISEXMEM-9B’s inference time including explanation generation, compared with about 30 s for GPT-5.6 Sol.The eight benchmarks and AGITA-Bench cost approximately $800 to evaluate with GPT-5.6 Sol; the authors describe VISEXMEM-9B as locally deployable, with fixed checkpoints and deterministic aggregation.Source: §6 · p. 9

7 CONCLUSION, LIMITATIONS, AND FUTURE WORK

The paper concludes that compact, open evaluators can combine alignment judgment with interpretable evidence and reproducible local inference. It also bounds this conclusion by verification and localization errors, benchmark scope, and possible biases in learned components and data.

Conclusion

  • VISEXMEM derives global scores from localized atomic verification, and AGITA-Bench evaluates global and fine-grained human agreement.The authors report that structured decomposed prompting improves average performance across tested backbones, with task-specific training further strengthening semantic-error sensitivity and grounded explanations.Source: §7 · p. 9

Limitations and Future Work

  • Errors in fine-grained verification and evidence localization remain, and multi-stage inference costs more than lightweight embedding-based metrics.The prompting experiments do not exhaust alternatives, and AGITA-Bench focuses on compositional semantic alignment rather than perceptual, aesthetic, or domain-specific quality.Source: §7 · p. 9

Scope and Responsible Use

  • VISEXMEM targets semantic image-text alignment, not image quality, social acceptability, or correctness of culturally or demographically sensitive representations.Its behavior may reflect biases and representation gaps in backbone, grounding, teacher models, and training data; out-of-distribution conclusions are limited to evaluated benchmarks, and consequential uses should include independent human review.Source: ETHICS STATEMENT · p. 10

Evaluation Scope and Data

  • Results are interpreted within the scope of the benchmarks and datasets used, whose collection, licensing, cultural coverage, and representation can affect evaluation.Human annotation covers AGITA-Bench and the explanation and evidence evaluations; explanation-study identities were hidden and outputs randomized to reduce presentation bias.Source: ETHICS STATEMENT · p. 10

A.1 TRAINING DATA CONSTRUCTION

Training data combine complementary sources of image-text supervision, which are converted into typed claims, correctness decisions, and visual evidence. Counterfactuals and staged processed examples support evaluation, while data isolation separates final benchmarks from training and model selection.

Data sources

  • The corpus combines dense region annotations, grounded captions, spatial relations, counts, long-form descriptions, human judgments, and contrastive image pairs.These sources are selected for complementary coverage of semantic skills required by VISEXMEM.Source: §A.1 · p. 17

Atomic claim construction and verification

  • Captions become typed, visually verifiable claims in six categories: object, attribute, action, relation, count, and negation.Explicit dataset structure is converted programmatically; Qwen3.6-27B is an offline decomposition teacher for free-form sources, without overwriting reliable source annotations. Supported or contradicted labels come preferentially from native annotations or human judgments; low-confidence or unresolved outputs are discarded.Source: §A.1 · pp. 17–18

Counterfactual supervision

  • Counterfactuals pair descriptions with independently generated images or change one semantic constraint while preserving a natural image.The text-side changes target object, attribute, action, relation, count, or negation, creating minimal pairs with an explicitly known semantic change.Source: §A.1 · p. 18

Visual and causal evidence

  • Evidence comes from native annotations when available; otherwise, GroundingDINO localizes already-labeled content and does not supply correctness labels.Evidence varies by claim type, and pure absence claims receive no fabricated region; eligible regions are paired with random same-image controls for Stage 3 causal-evidence training.Source: §A.1 · p. 18

Explanation supervision

  • Explanation targets are generated only after claims, decisions, and evidence are fixed, so explanation teachers do not redefine correctness or infer new boxes.This separation prevents explanation learning from modifying the supervision used to derive the numerical score.Source: §A.1 · p. 18

Processed training examples

  • Processed examples show Stage 1 task-specific SFT views, Stage 2 paired supported and counterfactual claims, and Stage 3 semantic and causal-evidence examples.Stage 3 compares selected evidence with a matched random region using multiple image views and replays earlier SFT and counterfactual examples.Source: §A.1 · p. 18

Data isolation

  • Final evaluation benchmarks are excluded from training and model selection, and training and development groups are separated before optimization.Only the training split of EvalMuse is used for training.Source: §A.1 · p. 18

A.2.1 MODELS AND PARAMETER-EFFICIENT TRAINING

The models use LoRA-based parameter-efficient training, and the appendix illustrates processed examples and task-specific target formats. The examples include a one-attribute counterfactual and evidence selection over candidate regions.

Models and parameter-efficient training

  • Qwen3.5-4B and Qwen3.5-9B are trained with LoRA at rank 16, scaling factor α = 32, and dropout 0.05.LoRA is applied to q, k, v, o, gate, up, and down projections; training uses bfloat16 and disables Qwen3.5 thinking mode.Source: §A.2.1 · p. 18

Counterfactual example

  • A Stage 2 example changes “yellow fire hydrant” to “green fire hydrant” while keeping the image pixels identical.The example uses a Yes/No decision-margin loss without a generated target text.Source: §A.2.1 · p. 19

Processed examples and target formats

  • During Stage 1 SFT, candidate bounding boxes are inputs to evidence selection, not prediction targets; Figure 7 shows representative processed training examples from the three VisExMEM training stages.Source: Figure 7 · p. 19, Table 5 · p. 20

A.2.2 VERIFICATION PROMPT AND DECISION SCORING

Verification uses a shared prompt template for counterfactual refinement and Stage 3, returning binary decisions that support logit-based scoring and policy optimization.

A.2.2 VERIFICATION PROMPT AND DECISION SCORING

  • Both counterfactual refinement and Stage 3 use the same prompt template, which requests exactly one JSON yes/no decision for an image-caption claim.Stage 3 treats outputs without either valid decision as invalid during policy optimization.Source: §A.2.2 · p. 20
  • Counterfactual refinement scores each supported/counterfactual claim pair using the verification margin from yes/no decision logits after their shared output prefix.It performs no free-form generation.Source: §A.2.2 · p. 20
  • Stage 3 semantic optimization samples eight decision completions, while causal-evidence training holds the textual prompt fixed and changes only the image view.This design makes differences between causal views arise from the image intervention rather than changed textual instructions.Source: §A.2.2 · p. 20

A.2.3 MULTI-TASK SFT

This section documents VisExMEM’s training, evaluation, benchmark, and annotation protocols, alongside results and analyses including out-of-distribution evaluation, ablations, human-study agreement, and worked examples of image support and decomposition quality.

A.2.3 MULTI-TASK SFT

  • Stage 1 moves from task-specific decomposition, verification, and evidence-selection curricula to 40K joint examples, sampling decomposition, verification, evidence selection, and explanation at an 8:9:4:3 schedule.The task-specific views contain 13.5K decomposition, 15K verification, and 7K evidence-selection examples; the joint stage includes 4.5K explanation examples.Source: §A.2.3 · p. 20

A.2.4 COUNTERFACTUAL REFINEMENT

  • Counterfactual refinement starts from the promoted Stage 1 checkpoint, uses 20K supported/counterfactual pairs, and interleaves counterfactual, verification replay, and decomposition replay updates at 8:1:1.Its three objective terms separate supported from counterfactual claims, preserve sufficient positive support, and push the counterfactual below the negative target.Source: §A.2.4 · p. 21

A.2.5 EVIDENCE-AWARE GDPO

  • Evidence-aware GDPO jointly trains semantic correctness through group-relative rollouts and causal evidence through a direct auxiliary objective, avoiding attenuation of shared image-intervention signals by within-group centering.The causal objective uses five image views—intact, selected-evidence and random-control removals, and selected-evidence and random-control-only views—to compare selected evidence with a matched control.Source: §A.2.5 · pp. 21–22

A.3 GRPO ABLATIONS

  • The GRPO ablations branch from the Stage 2 checkpoint: Verification-GRPO optimizes semantic decisions, then Causal-GRPO adds a selected-versus-random evidence bonus, unlike the final joint direct causal objective.Because the causal signal is shared across rollouts of an input, within-group centering largely removes it; direct optimization preserves it while learning semantic correctness jointly.Source: §A.3 · pp. 23–24

A.4.1 OUT-OF-DISTRIBUTION GENERALIZATION

  • VISEXMEM-9B reaches the highest VisPlotBench Group score (78.6 vs. 74.5 for the strongest 9B Qwen control) and leads CulturalFrames (ρ = .311; PAccwp = 65.8).On ProtoBias, it scores 73.5/71.5/81.0 accuracy and .254/.205/.286 κ on Animals/Demography/Objects; PROTOSCORE has higher category accuracy, while VISEXMEM-9B has stronger κ on Animals and Objects.Source: §A.4.1 · p. 24, Table 7 · p. 25

A.4.2 REPLAY ABLATION

  • Replay raises the final GDPO average from 68.2 without replay to 73.3, reversing the CF-stage result, where no replay scores 70.7 versus 69.0.The replay trajectory outperforms no replay on all five benchmarks after GDPO, with the largest final differences on Pick-a-Pic (+11.1) and EqBen (+5.3); this compares complete trajectories, not replay isolated to Stage 3.Source: Table 8 · p. 25, §A.4.2 · pp. 25–26

A.4.3 AGGREGATION ABLATION

  • The arithmetic mean outperforms geometric, harmonic, and minimum aggregation on all five evaluated benchmarks, supporting equal weighting of visually verifiable claims.It requires no learned or benchmark-specific aggregation parameters.Source: Table 9 · p. 26, §A.4.3 · p. 26

A.5 BENCHMARK EVALUATION PROTOCOLS

  • Evaluation uses a fixed inference procedure with no benchmark-test tuning; Winoground and EqBen report Text, Image, and Group matching, with Group requiring both conditions.TIFA160 uses 800 pairs and GenAI-Bench a 527-prompt coreset; both report tie-calibrated pairwise accuracy against human ratings.Source: §A.5 · pp. 26–27
  • Evaluation protocols measure CountBench top-1 counting accuracy, TS2 ordering and separation across semantic-error partitions, and AGITA-Bench global and atomic agreement.TS2 reports rank_m and sep_m, while AGITA-Bench correlations are averaged unweighted across Easy, Medium, and Hard subsets.Source: §A.5 · pp. 27–28

A.5.1 VLM-AS-A-JUDGE EVALUATION

  • For each fixed backbone, Hol. directly scores image-caption alignment from 0.05 to 0.95 without decomposition, whereas Dec. verifies localized atomic claims and produces a VLM-generated overall score in [0, 1].The controls use no VISEXMEM-specific fine-tuning, learned evidence-conditioned verifier, or external grounding cues.Source: §A.5.1 · p. 28

A.5.2 BASELINE COVERAGE AND REPRODUCIBILITY

  • REVEALER and T2I-FineEval results are omitted because their published configurations could not be reproduced faithfully under the controlled evaluation setup.REVEALER lacked a usable public implementation and checkpoint, while T2I-FineEval required its GPT-4 decomposition stage; original-paper numbers were also excluded because protocols differed.Source: §A.5.2 · p. 29

A.6.1 BENCHMARK CONSTRUCTION

  • AGITA-Bench contains 600 image-text pairs and 8,735 atomic claims, balanced across I2T/T2I and easy, medium, and hard semantic-complexity cases.The benchmark is excluded from training, calibration, and model selection; difficulty is based on claim complexity, not alignment severity.Source: §A.6.1 · p. 32

A.6.2 HUMAN ANNOTATION PROTOCOL

  • AGITA-Bench separately annotates atomic image support on a 0–4 scale, global alignment on a 1–5 scale, and whether each decomposition is reasonable.Annotators can mark claims cannot judge; the benchmark has human score labels, not human bounding-box annotations, and shared-pair annotations support agreement analysis.Source: §A.6.2 · p. 32

A.6.3 ANNOTATION INTERFACE, GUIDELINES AND EXAMPLES

  • The annotation interface records decomposition quality and claim-level support alongside the image and text, plus one global alignment rating per sample.Annotators can consult a guide and worked examples spanning high, partial, and weak alignment.Source: §A.6.3 · p. 32

Example 1: High alignment High alignment

A.7 EXPLANATION AND EVIDENCE ANNOTATION

  • Mean pairwise weighted κ was .59 across explanation-quality criteria and .58 for localized evidence; Unsupported Content κ was .42.Agreement also reached 89.5% within one scale point for explanation ratings and 86.9% for evidence ratings.Source: §A.7 · p. 41

Related papers

Loading 2610.16441v1…