PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

Sungguk Cha, Mintae Kim, Youngsub Han, Byoung-Ki Jeon, Sangyeob Lee

arXiv:2609.34480v1cs.CV

TL;DR

Visual question answering must answer correctly when evidence is sufficient and abstain when it is not. This paper connects complete-question evaluation with evidence for labels across charts and scenes; the highest observed complete task success rates are 57.0% on PlotQA, 43.5% on CLEVR, and 33.7% on GQA.

  • Problem

    Visual answering must handle evidence changes by preserving answers under irrelevant edits, changing answers under relevant edits, and abstaining when essential evidence is removed.

    Source: §1 · p. 1

  • Method

    The benchmark combines complete-question scoring with checked chart witnesses, program-derived scene labels, and a photographic residual-cue audit.

    Source: §1 · pp. 1–2

  • Results

    57.0% on PlotQA, 43.5% on CLEVR, and 33.7% on GQA are the highest observed complete task success rates across six configurations and 72,000 responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Complete task scores and evidence for each response obligation are needed because correct answerability decisions can coexist with incorrect supported answers.

    Source: §8 · p. 9, §7 · p. 9

    Image-level certification covers fixed-axis chart lookup under declared program, world, and renderer semantics; photographic certification must account for residual visual cues.

    Source: §7 · p. 9, §6.2 · p. 9

Abstract

from arXiv · show

Reliable visual question answering requires correct answers when evidence is sufficient and abstention when it is not. We introduce a benchmark that connects complete-question evaluation with explicit evidence for its labels across PlotQA charts, CLEVR rendered scenes, and GQA photographs. Each question groups original and edited images, presented independently; success requires every supported answer and every required abstention to be correct. For chart missing-information labels, executable witnesses establish that admissible complete charts give different answers but identical pixels after masking. Scene labels follow source programs and edits, with a residual-cue analysis for photographs. Across 72,000 responses from six model configurations, the highest observed complete task success rates are 57.0%, 43.5%, and 33.7%, respectively. On charts, the strongest configuration achieves 96.2% per-view decision accuracy, yet 265 of its 835 groups with every decision correct still contain incorrect answers. Evaluating supported answers and necessary abstentions together exposes failures that answerability decisions alone conceal.

1 INTRODUCTION

The paper evaluates whether visual answers track changing evidence, requiring correct answers when supported and abstention when evidence is insufficient. It combines complete-question scoring with evidence for labels across charts and scenes, and reports the highest complete-group success rates across three domains.

Motivation

  • Visual answering must preserve answers under irrelevant edits, revise them under relevant edits, and abstain when essential evidence is removed.Always refusing fails the first two obligations, while always answering fails the third.Source: §1 · p. 1

Controlled benchmark construction

  • Across PlotQA, CLEVR, and GQA, 3,000 question groups vary chart values and visibility, scene attributes and occlusion, or masks on and outside program dependencies.The 12,000 views are presented independently, and each group succeeds only when every answer and abstention is correct.Source: §1 · pp. 1–2

Evaluation and label evidence

  • The benchmark pairs complete-question scoring with explicit label evidence: 1,000 checked chart ambiguity witnesses, program-derived scene targets, and a photographic residual-cue audit.Complete success requires every supported answer and required abstention to be correct.Source: §1 · p. 2

Empirical diagnosis

  • 57.0% on PlotQA, 43.5% on CLEVR, and 33.7% on GQA are the highest observed complete-group success rates across six configurations and 72,000 responses.State-level errors and per-view versus group comparisons expose distinct failure patterns.Source: §1 · p. 2

2 RELATED WORK

Prior work studies visual answering, abstention, and consistency under changed evidence, but across separate protocols and domains. This paper combines these principles across three domains and requires all supported answers and abstentions within a question group.

Visual questions and benchmark coverage

Abstention and evidence sufficiency

Controlled changes and joint success

  • Controlled-edit and joint-context benchmarks study answer invariance, answer changes, linguistic consistency, and related visual contexts; this work combines these principles across three domains.Causal VQA, VQA-Rephrasings, HallusionBench, DynaMath, and MuirBench cover these related settings (Agarwal et al., 2020; Shah et al., 2019; Guan et al., 2024).Source: §2 · pp. 2–3

Structured reasoning and label evidence

  • Structured-reasoning benchmarks diagnose scene-query validity and generalization, while this work keeps scene questions fixed as evidence changes and makes chart-answer disagreement and observation equality explicit premises for abstention.The chart-witness framing specializes possible-world semantics to rendered observations (Imieli´nski & Lipski, 1984; Console et al., 2022).Source: §2 · p. 3

3 BENCHMARK CONSTRUCTION AND LABEL EVIDENCE

The benchmark keeps questions fixed while changing image evidence, with source-specific response obligations and label evidence. Chart missing-information labels are grounded in admissible complete worlds that yield different answers but identical observations, and audits check the witness conditions.

3.1 RESPONSE OBLIGATIONS ACROSS THREE SOURCES

  • Each source contributes 1,000 groups with fixed questions and varied images; charts and CLEVR include answer-preserving, answer-changing, and missing-evidence states, while GQA uses controls and missing-evidence masks.Charts also include invalid-reference targets; models see each view independently, and group success requires answers on supported states and abstention on negative states.Source: §3.1 · p. 3, §3 · p. 3, Table 2 · p. 4

3.2 FROM COMPATIBLE WORLDS TO A NEGATIVE LABEL

  • For charts, an image lacks identifying information when admissible complete worlds that render to the same image yield at least two defined answers; a pair witnesses ambiguity, but failed witness search does not establish uniqueness.The guarantee assumes the answer program and declared worlds, visibility operation, and renderer capture the intended task.Source: §3.2 · p. 5
  • The chart checker verifies admissible and renderable worlds, differing lookup answers, and pixel equality with the saved observation; the fixed-axis setup uses finite rational values and masks the queried x location.Axes are exogenous, and the mask position is independent of the hidden y value.Source: §3.2 · p. 5

3.3 AUDITING THE WITNESS CONDITIONS

  • In 1,024 synthetic four-cell cases, 414 had one answer across completions and 610 had multiple answers; hidden-value use alone falsely rejected 132 constant-answer cases, while endpoint-only comparisons missed 90 ambiguous cases.Enumeration and the Z3 solver agreed on all 822 ambiguous and 634 constant cases across the synthetic and recursive-program audits.Source: §3.3 · p. 5
  • Complete-world checks found 785 out-of-axis values in 503 initial chart-witness groups despite observation checks passing all 5,000 views; matched witness pairs satisfy both conditions for all 1,000 groups.These pairs preserve evaluated images, labels, and model scores and cover all 5,000 chart views.Source: §3.3 · pp. 5–6

4 EVALUATION PROTOCOL

The protocol tests whether models answer supported inputs and abstain when evidence is insufficient, while scoring related image-question views as a complete task. It defines decision and answer criteria, counts invalid outputs as failures, and uses group-level bootstrap intervals for uncertainty.

Inputs and configurations

  • Each call presents one image and question without state identifiers or other group members; six configurations each receive 12,000 views, for 72,000 responses.The chart split separates source images and groups, holds out a rendering family, and excludes 46,000 training views.Source: §4 · p. 6

Complete task success

  • A group succeeds only when every supported answer and required abstention is correct; Jk measures this complete task success.Chart answers use exact rational equality after numeric parsing; scene answers use rational equality or normalized exact-string matching.Source: §4 · p. 6

Decision diagnostics

  • Decision diagnostics separate complete-group decision success from per-view decision failures and distinguish supported-state rejection from negative-state acceptance.With EU pooling negative states, Ebal weights supported and negative classes equally; always-answerable and always-unanswerable policies have Ebal = 50% and Bk = Jk = 0.Source: §4 · p. 6
  • Jk ≤ Bk, and Bk − Jk is the share of groups with correct decisions but incorrect supported answers.Invalid responses count as failures in the full-denominator scores.Source: §4 · p. 6

Denominators and uncertainty

  • Descriptive 95% percentile intervals use 10,000 bootstrap draws of 1,000 groups per source, keeping all views of a question together.Paired comparisons use the same draws; group partitions, class balancing, common-state comparisons, and paired diagnostics are post-hoc analyses.Source: §4 · p. 6

5 MODEL BEHAVIOR ON VERIFIED CHART QUESTIONS

On verified chart questions, correct answerability decisions do not guarantee correct answers, and missing-information errors differ from invalid-reference errors. Complete task success therefore falls below decision success, with the gap driven by incorrect supported answers after correct decisions.

5.1 ACCEPTING INPUTS WITH CERTIFIED MISSING EVIDENCE

  • 113 of 1,000 chart groups have a valid Qwen 3.8 Flash Next answerability acceptance despite certified missing evidence, where admissible complete charts yield different answers but identical observed pixels.Table 3 reports decisions and answers on the same 1,000 chart groups covered by witness checks.Source: §5.1 · p. 6, Table 3 · p. 7
  • False acceptance of certified missing inputs ranges from 11.3% to 61.2% across configurations.GLM 5.3 Flash rejects 15/3,000 supported views but accepts 612/1,000 missing views; high willingness to answer supported inputs coexists with accepting insufficient evidence.Source: §5.1 · p. 7
  • Gemma 4 26B A4B fails on 427 missing-information inputs but only 32 invalid-reference inputs, so pooling the negative states obscures which obligation fails.Reporting EM and EI separately reveals this difference.Source: §5.1 · p. 7

5.2 FROM CORRECT DECISIONS TO CORRECT ANSWERS

  • 57.0% of chart groups achieve complete task success for Qwen 3.8 Flash Next, while 265 of 835 groups with every decision correct still contain an incorrect supported answer.Its per-view decision accuracy is 96.2%, but complete task success is 57.0%.Source: §5.2 · p. 7, Table 4 · p. 8
  • Qwen 3.8 Flash Next gives 2,473 correct answers, rejects 49, and returns 478 incorrect numerical values across 3,000 supported chart views.All submitted supported answers parse as numbers, but the 478 errors fail the exact-value criterion.Source: §5.2 · p. 7
  • Gemma 4 31B records J5 = 55.6% and GLM 5.3 Flash 18.2%; the Qwen 3.8 Flash Next–Gemma gap narrows from 16.8 points in complete decision success to 1.4 points in complete task success.Source: §5.2 · p. 8

6 COMPLETE TASK SUCCESS ON CLEVR AND GQA

Complete task success is highest for Qwen 3.8 Flash Next on both scene cohorts, but error patterns differ across supported rejection, missing-state acceptance, residual photographic cues, and grouped model comparisons.

6 COMPLETE TASK SUCCESS ON CLEVR AND GQA

  • 43.5% on CLEVR and 33.7% on GQA are Qwen 3.8 Flash Next’s highest observed complete task success rates on the scene cohorts.On CLEVR, 546 groups fail an answerability decision and 19 more fail a supported answer despite correct decisions; on GQA, the corresponding counts are 601 and 62.Source: §6 · p. 8

6.1 SUPPORTED REJECTION AND MISSING-STATE ACCEPTANCE

  • 3.80% chart decision failure contrasts with 14.98% on CLEVR and 25.67% on GQA for Qwen 3.8 Flash Next; on charts, 265 groups fail supported answers despite correct decisions, versus 165 decision errors.On CLEVR M, Qwen has 478 false acceptances and one invalid response (47.9% failure); indiscriminately increasing abstention cannot resolve errors that coexist with rejection of supported scenes.Source: §6.1 · pp. 8–9, §6 · p. 8

6.2 RESIDUAL CUES IN PHOTOGRAPHIC INPUTS

  • Excluding 376 GQA groups whose original answers are spatial terms leaves 624 groups, but the fixed-label analysis neither counts label errors nor certifies the remaining photographs.Qwen 3.8 Flash Next fails on 489/1,872 views (26.12%) and GLM 5.3 Flash on 431/1,872 (23.02%); the stratum measures sensitivity to a specified residual cue.Source: §6.2 · p. 9

6.3 THE EVALUATION UNIT CHANGES MODEL COMPARISONS

  • 33.7% versus 24.5% complete task success favors Qwen over GLM on GQA, while GLM has fewer decision failures and more supported views answered correctly.Qwen makes every decision correctly in more groups (399 versus 305), illustrating how fewer individual errors can coexist with failures on more questions.Source: §6.3 · p. 9

7 DISCUSSION AND LIMITATIONS

The discussion treats label evidence and group scoring as complementary: evidence justifies required responses, while group scoring checks whether all obligations for a question are met. Verification scope differs across charts, rendered scenes, and photographs.

7 DISCUSSION AND LIMITATIONS

  • Chart witnesses justify required abstentions, while decision–answer decomposition exposes failures even when every answerability decision is correct.The GQA comparison also shows that fewer individual errors can coexist with failures on more questions.Source: §7 · p. 9
  • Image-level guarantees cover fixed-axis chart lookup under declared program, world, and renderer semantics; CLEVR and GQA extend behavioral evaluation under source-derived labels.Recursive-program experiments validate answer-level logic, while photographic certification requires accounting for residual visual cues.Source: §7 · p. 9
  • Appearance-matched and text-only controls would isolate reliance on edit appearance and nonvisual information.The discussion identifies these controls as ways to extend evaluation to richer visual tasks.Source: §7 · p. 9

8 CONCLUSION

The conclusion connects complete-question evaluation with explicit label evidence to assess visual answering as evidence changes. Its results expose errors despite correct answerability decisions and comparisons that shift with the evaluation unit.

8 CONCLUSION

  • The conclusion links complete-question evaluation with explicit evidence for response obligations across visual answering as evidence changes.Executable chart witnesses establish required abstentions, while program-derived scene labels extend evaluation across rendered and photographic inputs.Source: §8 · p. 9
  • Incorrect answers remain within groups whose answerability decisions are entirely correct, and model comparisons change with the evaluation unit.The conclusion states that reliable visual answering requires correct answers to supported questions and abstention when evidence is insufficient.Source: §8 · p. 9
  • Evaluating this behavior requires complete task scores and evidence for each response obligation.Source: §8 · p. 9

ETHICS STATEMENT

The ethics statement describes the study’s data sources and information collection.

ETHICS STATEMENT

A EVALUATION INPUTS AND MODEL CONFIGURATIONS

The evaluation presents each image and fixed question independently, with answerability defined by whether the image supports one answer and response validity checked separately.

A EVALUATION INPUTS AND MODEL CONFIGURATIONS

  • Each input pairs one image with a fixed question, and the prompt defines answerable and unanswerable using evidence available in that image alone.Difficult reading can still be answerable; models are instructed not to infer missing information from trends, world knowledge, or other images.Source: A EVALUATION INPUTS AND MODEL… · p. 18
  • A valid response must include a Boolean answerability decision, an answer or null, and a nonempty reason; invalid outputs count as failures.Unanswerable decisions require a null answer, while an answerable decision with a null answer can pass decision scoring but fails answer correctness.Source: A EVALUATION INPUTS AND MODEL… · p. 18
  • CLEVR uses official scene assets, while GQA interventions mask recorded object boxes in source photographs; all six configurations use vLLM with temperature zero and a 512-token cap.Qwen 3.8 27B uses the specified FP8 checkpoint.Source: A EVALUATION INPUTS AND MODEL… · p. 18, Table 5 · p. 18

B FULL METRICS AND SENSITIVITY ANALYSES

This section reports full and diagnostic scoring, GQA residual-cue analyses, and checks of the benchmark’s chart ambiguity witnesses. Across these analyses, decision accuracy and complete task success capture different errors and can yield different model rankings.

B FULL METRICS AND SENSITIVITY ANALYSES

  • 86.1%, 47.4%, and 39.9% are Qwen 3.8 Flash Next’s BF SM scores on PlotQA, CLEVR, and GQA, while 265 of 835 chart groups with every decision correct still contain a wrong supported answer.The common-state comparison preserves each source’s questions and intervention mechanism; supported chart views also include 478 numerical mismatches among 3,000 views.Source: B FULL METRICS AND SENSITIVITY ANALYSES · p. 19
  • Answer correctness uses fixed exact-match rules, while complete task success requires all supported answers and required abstentions in a group to be correct.The full metrics distinguish conditional disagreement, invalid outputs, class-specific failures, and post-hoc balanced or common-state comparisons.Source: B FULL METRICS AND SENSITIVITY ANALYSES · p. 19, Table 6 · p. 19, Table 7 · p. 19
  • Descriptive 95% intervals use 10,000 paired group-bootstrap draws per source, keeping all views of each selected question group together.The group partitions, class balancing, common-state comparisons, and paired diagnostics are post-hoc analyses.Source: B FULL METRICS AND SENSITIVITY ANALYSES · pp. 18, 20

C GQA RESIDUAL CUES AND EXAMPLE CONSTRUCTION

  • Excluding 376 location-answer GQA groups changes Qwen 3.8 Flash Next’s failure from 770/3,000 to 489/1,872 (26.12%) and GLM 5.3 Flash’s from 741/3,000 to 431/1,872 (23.02%).The location stratum is a sensitivity analysis; GQA’s photographic labels are source-derived, and equality of complete-world observations remains unverified.Source: C GQA RESIDUAL CUES AND EXAMPLE… · p. 20, Table 9 · p. 20

C.1 ILLUSTRATIVE INPUTS

  • Figure 2’s CLEVR example changes an unrelated cylinder in S, changes the referent’s color in C, and occludes it in M; its GQA example retains the racket’s lower-frame position after masking.All six configurations abstain on the illustrated GQA example, while the quantitative analysis uses all 1,000 groups.Source: §C.1 · pp. 20–21

D CONSTRUCTION AND WITNESS VERIFICATION

  • All 1,000 chart witness pairs pass checks for admissibility, renderability, different answers, and identical masked pixels; the checker rejects all 100 designed faults.The construction scan compiled 5,895 candidates from eligible PlotQA questions, and 4,000 of 4,463 attempted constructions succeeded.Source: D CONSTRUCTION AND WITNESS VERIFICATION · p. 21
  • In finite-program tests, hidden-cell reference falsely labels 132 constant-answer cases as missing, while endpoint-only checking misses 90 ambiguous cases; enumeration and Z3 agree on 1,456 cases.The recursive-tree extension contributes 212 ambiguous and 220 constant cases, and both constructors produce accepted witnesses for all 822 ambiguous cases.Source: D CONSTRUCTION AND WITNESS VERIFICATION · p. 21

E GQA RANKING DIAGNOSTICS

  • At λ = 1/2, Qwen 3.8 Flash Next has 30.20% failure and GLM 5.3 Flash 34.55%, reversing their point-estimate ranking as class weights change while predictions remain fixed.Their GQA per-view failures are 770/3,000 (25.67%) and 741/3,000 (24.70%), respectively; the paired difference interval includes zero.Source: §E.1 · p. 22, Figure 5 · p. 22, E GQA RANKING DIAGNOSTICS · p. 22

E.2 EXACT ACCOUNTING OF GROUP ERRORS

The section explains why per-view failures and complete-group success can rank models differently, then verifies the group accounting and numerical results. It also places the evaluation design in the context of targeted diagnostics and explicit reporting conditions.

E.2 EXACT ACCOUNTING OF GROUP ERRORS

  • Grouped success depends on error concentration as well as total errors: per-view counts include repeated failures, whereas 1 − Bk counts each affected question group once.This identity separates each group’s first failure from additional failures in the same group.Source: §E.2 · pp. 22–23
  • Qwen 3.8 Flash Next has 770 failures, including 169 repeated failures, while GLM 5.3 Flash has 741, including 46; the additional repeated failures explain their opposing per-view and group-score rankings.Qwen has 399 groups with zero failures versus GLM’s 305; Table 12’s group counts recover per-view failure totals by weighting each group by its failures.Source: §E.2 · p. 23, Table 12 · p. 23
  • Qwen 3.8 Flash Next has higher GQA complete task success than GLM 5.3 Flash, 33.7% versus 24.5%, while GLM answers more supported views correctly, 77.65% versus 69.85%.The evaluation objective determines which behavior an aggregate score rewards.Source: §E.2 · p. 23

G NUMERICAL VERIFICATION

  • Numerical verification recomputes scores and paired analyses from all 72,000 final-response records and checks question identities, invalid-response handling, and group-score consistency.All 1,000 chart witnesses match evaluated groups, and the chart decomposition identities hold for all six configurations.Source: G NUMERICAL VERIFICATION · pp. 23–24, F EVALUATION COHORT VERIFICATION AND… · p. 23
  • The frozen release includes evaluated views, 1,000 bounded chart proofs and their checker, scene-edit evidence, and scoring projections, but excludes the broader training and development resource.GQA photographs and related source materials are obtained separately and reconstructed using the supplied instructions.Source: G NUMERICAL VERIFICATION · p. 24

H EXTENDED CONTEXT FOR THE EVALUATION DESIGN

  • The evaluation’s targeted diagnostics distinguish supported-answer errors from decisions to answer insufficient evidence, alongside explicit state definitions, label evidence, denominators, and scoring rules.These reporting choices frame complete-question success as distinct from aggregate accuracy.Source: H EXTENDED CONTEXT FOR THE EVALUATION… · pp. 24–25

Related papers

Loading 2609.34480v1…