PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correct Answers: How Compositional Cues Shape Visual Judgments

Xiaoyu Ma, Zheyuan Zhang, Kaiwen Shi, Yanfang Ye

arXiv:2610.28843v1cs.CVcs.CL

TL;DR

Visual question answering requires assigning attributes and relation roles to the correct objects, but overall accuracy does not reveal which prompt labels help or how they interact. The paper compares fixed-content labels, auxiliary role hints, and hint or task interventions; results show model- and condition-dependent label effects, with swapped hints accounting for much of Qwen3's preserving-hint advantage.

  • Problem

    Overall prompt gains do not reveal which explicit labels affect visual judgments or how label effects interact.

    Source: §1 · p. 1

  • Method

    The paper uses fixed-content label comparisons, no-hint and field-order controls for auxiliary hints, and tests hint rewriting and relation selection.

    Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 5

  • Results

    Across ten models and six tasks, label effects differ by model; Qwen3's preserving-over-swapped hint advantage mainly reflects swapped-hint harm, while clearer wording does not consistently improve visual judgments.

    Source: §1 · p. 2, §10 · p. 8

  • Takeaways & Limitations

    Observed error patterns can guide model- and task-specific label choices and whether to revise hint wording or change the judgment task, rather than adding structure by default.

    Source: §9 · p. 8, §10 · p. 8

    RQ3 changes task instructions and answer meanings together, so their separate contributions remain unresolved.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Visual question answering requires models to assign attributes and relations to the correct objects, a problem known as visual binding. Structured prompts make these assignments explicit through labeled fields. Yet an improvement from the full prompt does not reveal which labels help or how they interact. First, we isolate field-label effects across ten models and six tasks by holding the image and claim fixed. These comparisons reveal differences that overall accuracy can hide. With a relation label present, adding object-role labels helps Qwen3 but hurts Qwen2 on Spatial. Second, we compare auxiliary hints that preserve or swap the question's role assignments against a no-hint baseline. Across two image sets, Qwen3's advantage with preserving hints mainly reflects errors caused by swapped hints. Preserving hints remain near baseline. Field order further affects the harm. Third, we test clearer hint wording and direct selection of the relation shown in the image. Clearer hints improve text matching but do not consistently improve visual judgments. In contrast, relation selection partly reduces errors for some hints in Qwen3 and GPT-4.1-mini.

1 Introduction

The paper studies how explicit labels and auxiliary role hints shape visual binding, beyond what aggregate accuracy reveals. It compares fixed-content labels, preserving and swapped hints, and interventions that rewrite hints or change the judgment task.

1 Introduction

  • Structured-prompt gains do not isolate label effects because content, organization, and reasoning can change together.Visual binding requires assigning attributes and relation roles to the correct objects, not merely recognizing scene elements.Source: §1 · p. 1
  • The paper asks whether preserving-hint advantages reflect useful guidance, swapped-hint harm, or both, using no-hint baselines and field-order controls.A preserving hint adds no new visual evidence about the claim's truth.Source: §1 · p. 2
  • The intervention study tests whether clearer hint wording or direct relation selection can reduce hint-induced errors without retraining.The motivation distinguishes improved role interpretation from changing the judgment task when errors persist despite correct interpretation.Source: §1 · p. 2
  • Across ten models and six tasks, fixed-content comparisons expose label effects hidden by overall accuracy; on Spatial, object-role labels with a relation label help Qwen3 but hurt Qwen2.For auxiliary hints, Qwen3's preserving-over-swapped advantage mainly reflects swapped-hint harm; preserving hints remain near baseline, and canonical order amplifies the harm.Source: §1 · p. 2
  • The paper proposes fixed-content comparisons to separate responses to object and value errors and identify label-component effects and interactions.The introduction also proposes no-hint baselines, field-order controls, and paired-answer analysis to distinguish hint guidance from added errors.Source: §1 · p. 2

2 Related Work

Related work covers compositional evaluation, structured prompting, prompt sensitivity, and consistency-based answer verification. The paper positions its comparisons as isolating role-label effects and diagnosing hint-related influences.

Visual Binding and Compositionality

  • Compositional evaluations distinguish recognizing scene elements from recovering their assignments, while structured prompting represents objects, attributes, and relations explicitly.The paper holds claim content and field order fixed to isolate the role labels' effects on visual judgments.Source: §2 · p. 2

Prompt Sensitivity and Textual Interference

  • Prompt studies examine format variation and rephrasing sensitivity, while VLM studies report that language priors and conflicting text can override visual evidence.The paper compares hints with no-hint baselines and varies order to distinguish role-preserving advantage from swapped-hint harm and presentation effects.Source: §2 · p. 2

Consistency-Based Answer Verification

  • Consistency across related questions has been used for visual-answer checking; this paper compares role-explicit and ordinary checks at the same query budget.The stated application is ranking errors in original answers for potential human review.Source: §2 · p. 2

3 Preliminaries

The preliminaries define visual binding through assignments of claim positions to image objects and distinguish wrong-object from wrong-value alternatives. They also specify how field labels and order are manipulated while preserving label–value assignments.

3 Preliminaries

  • A claim's binding assignment identifies which image object has a proposed attribute value or occupies each relation role.The image determines whether the claim under that assignment is true.Source: §3 · pp. 2–3
  • Wrong-object alternatives change the object assignment while retaining the attribute and value; wrong-value alternatives change the value while retaining the object and attribute.An alternative is retained as a negative only when the change makes the claim false.Source: §3 · p. 3
  • Replacing a field label such as subject with a generic name removes the explicit role label but preserves its value, and reordering whole fields preserves label–value assignments.Figure 2 depicts label conditions and hint types across field orders; its names and answer options are illustrative.Source: §3 · p. 3, Figure 2 · p. 3

4 Methodology

The methodology uses matched prompt interventions to isolate the effects of field labels and auxiliary role hints while keeping visual claims fixed. It also tests hint wording and relation selection as interventions on hint interpretation and the judgment task.

4.1 Role-Label Interventions

  • For attributes, the contrast compares explicit-versus-generic changes in correct-answer NLL across wrong-object and wrong-value alternatives.A negative value means labels reduce NLL more, or increase it less, for wrong-object judgments.Source: §4.1 · pp. 3–4
  • Label comparisons hold participants, relations, and field order fixed while varying object-role and relation labels across four conditions.The conditions are Generic, Object-role only, Relation only, and Both.Source: §4.1 · p. 4

4.2 Auxiliary-Hint Interventions

  • Auxiliary hints preserve or swap participant values while keeping the original question unchanged, and are compared with a no-hint baseline across field orders.The preserving-over-swapped gap is decomposed into preserving-hint benefit and swapped-hint loss relative to no hint.Source: §4.2 · p. 4

4.3 Hint Rewriting and Task Reformulation

  • Hint rewriting compares field and sentence forms, while verification and relation selection keep image, claim, and hint fixed but change instructions and answer meanings together.Selection’s contrast is adjusted for the possibility that selection is easier even without a hint.Source: §4.3 · p. 5

5 Experimental Setup

The experiments evaluate ten models across attribute, relation, and hint tasks using image sets with fixed claims or paired spatial judgments. Paired image- or pair-level bootstrap intervals quantify uncertainty for the reported comparisons.

5 Experimental Setup

  • The study evaluates ten models across Material, Size, Color, Spatial, Holding, and Attached-to tasks, with 96 images each for the first three, 48 opposite-truth pairs for Spatial, and eight images each for Holding and Attached-to.Attribute images use GQA/Visual Genome annotations; Spatial pairs hold the left/right claim fixed across opposite-truth images.Source: §5 · p. 5
  • External-48 uses 48 previously used images disjoint from Spatial, while Qwen3 and mini answer opposing claims with varied answer mappings and question positions.RQ3 uses two models and includes a fixed follow-up of 24 images in 12 source pairs, with four claims and two answer mappings per image.Source: §5 · p. 5
  • Confidence intervals resample images or pairs with prompts together; RQ3 reports descriptive paired 95% bootstrap intervals over 12 follow-up source pairs.The original Spatial hint study designates Qwen3’s DC > 0 and H > 0 as primary tests with Holm correction.Source: §5 · p. 5
  • Table 3 compares each hint with the unhinted baseline on 48 Spatial pairs, with paired 95% intervals and Qwen3’s hint benefit and harm decomposition highlighted.NVIDIA’s gains partly reflect fewer invalid-format answers.Source: Table 3 · p. 6
  • Figure 3 plots accuracy changes from generic field names across six model–task comparisons, with 95% image-bootstrap intervals over 96 images per task.Appendix E.1 gives condition accuracies.Source: Figure 3 · p. 6

6 RQ1: Effects of Role Labels

Role-label effects vary by model and task, and the Spatial component ablation shows that adding object-role labels to a relation-labeled prompt can help one model while hurting another. Across attribute results, score readouts distinguish error types, while some accuracy interactions reflect recovery from component losses rather than gains over baseline.

6.1 Cross-Task Label Sensitivity

  • Attribute label effects differ across models and readouts: LLaVA and Qwen3 have contrast estimates of +.057 and −.072 despite similar Color accuracy, while mini’s NLL and generated-answer contrasts favor wrong-object judgments across tasks.Only LLaVA’s interval excludes zero; GPT-4o’s Material and Color readouts favor different error types, with accuracy-contrast intervals spanning zero.Source: §6.1 · p. 6
  • On the eight-image Holding set, mini’s generated-answer interaction is +7.81 pp despite identical Both and Generic accuracy, because each component alone lowers accuracy.The positive interaction reflects recovery from component losses, with no gain over Generic.Source: §6.1 · p. 6

6.2 Component Effects

  • 4.69 pp for Qwen3 and −3.39 pp for Qwen2: adding object-role labels to Relation only has opposite Spatial accuracy effects, with both paired intervals excluding zero.The cross-model difference is 8.07 pp [4.17, 12.11].Source: §6.2 · p. 6
  • Useful relation-label components depend on the model, and Qwen3’s best field-based condition remains below its ordinary-language reference.A separate plausible-participant cohort has mean accuracy above both blind controls, but intended-minus-swapped interaction intervals span zero for both relations.Source: §6.2 · p. 6

7 RQ2: Benefits and Harms of Hints

No-hint comparisons show whether role hints help or harm, while order and model affect their impact. For Qwen3, the preserving-over-swapped advantage mainly reflects swapped-hint harm, not a clear preserving-hint gain.

7.1 Help–Harm Decomposition

  • Qwen3’s 4.95 pp preserving-over-swapped gap [1.43, 8.72] mainly reflects swapped-hint harm: preserving changes accuracy by −.72 pp and swapped hints by −5.66 pp versus no hint.The comparisons cover role hints alongside answering the original question alone.Source: §7.1 · pp. 6–7
  • Preserving-hint effects vary across models, ranging from gains for GPT-4.1-mini to small losses for Qwen2; NVIDIA’s gains also reflect fewer invalid-format answers.NVIDIA’s larger gains combine improved format compliance with changes in answer choice.Source: §7.1 · p. 7

7.2 Content–Order Interactions

  • Canonical order makes Qwen3’s swapped hint more disruptive, with an 8.33 pp content-by-order interaction [3.91, 13.15]; mini’s interaction is −4.43 pp [−7.94, −1.30].Order effects depend on both assignment and model; token counts match across hint variants in all five observable tokenizers.Source: §7.2 · p. 7, Figure 4 · p. 7

7.3 Replication and Error Patterns

  • On External-48, Qwen3 again shows swapped-hint harm without a clear preserving gain, and its order interaction remains positive; mini’s interaction changes sign across cohorts.Qwen3’s pattern also holds under both field-order weighting schemes.Source: §7.3 · p. 7
  • For Qwen3, canonical swapped hints raise opposing claim pairs answered No to both from 19 to 49 of 96, mainly by rejecting true claims rather than reversing both judgments.Question-last presentation lowers mini’s canonical swapped-hint accuracy, while Qwen3’s interval spans zero; repeating the question shows no clear harm reduction.Source: §7.3 · p. 7

8 RQ3: Mitigating Hint-Induced Errors

The study tests whether clearer hint wording or changing verification into relation selection can reduce hint-induced errors. Clearer wording improves text matching but does not reliably improve visual judgments; relation selection partly recovers some disrupted judgments.

8.1 Meaning-Preserving Hint Rewriting

  • Sentence hints let both models answer all 32 text-only judgments correctly, but clearer role matching does not consistently improve visual judgments.For Qwen3, sentence rewriting raises verification accuracy by 14.58 pp for B right of A but lowers it by 21.35 pp for B left of A; mini’s changes vary across hints and cohorts.Source: §8.1 · p. 7

8.2 Verification versus Selection

  • For A right of B in field form, selection’s adjusted advantage is 9.90 pp [.52, 19.27] for Qwen3 and 15.62 pp [5.21, 25.00] for mini.These contrasts adjust for the no-hint task difference, and remain positive for both models after omitting any one source pair.Source: §8.2 · pp. 7–8, Figure 5 · p. 8
  • Relation selection can partly recover judgments disrupted by role hints, but the tested condition remains below unhinted selection accuracy.Qwen3 corrects 26 judgments without new errors; mini corrects 55 and introduces 22 errors, with answers more sensitive to answer mapping.Source: §8.2 · p. 8

9 Discussion

The discussion links prompt-component choices and task changes to the error patterns they address. It distinguishes correcting judgments from identifying unreliable answers for possible human review.

9 Discussion

  • Choose labels for a specific model and task rather than adding structure by default; observed error patterns can guide whether to revise hint wording or change the judgment task.The discussion notes that omitting the hint still yields higher accuracy in the main recovery condition.Source: §9 · p. 8
  • Text-only matching tests hint interpretation, while visual evaluation tests which errors an intervention corrects or introduces.If role interpretation improves but visual errors persist, task reformulation offers another route to test on the same inputs.Source: §9 · p. 8
  • Role-explicit checks improve error ranking for GPT-4.1-mini and GPT-4o, but show no clear gain for the other four models.This answer-checking application ranks original answers without revising them and could prioritize predictions for human review.Source: §9 · p. 8

10 Conclusion

The conclusion connects role labels and auxiliary hints to visual-judgment errors, then links those diagnoses to intervention outcomes. It favors selecting cues and judgment tasks based on observed error patterns rather than adding structure by default.

10 Conclusion

  • Matched comparisons show that labels can affect object and value errors differently, even when ordinary accuracy is similar; useful relation labels depend on the model and other labels present.No-hint comparisons show that swapped-hint harm can create a preserving-over-swapped advantage without a preserving-hint gain.Source: §10 · p. 8
  • Clearer hint wording improves text matching but not consistently visual judgments, while relation selection partly recovers disrupted judgments with benefits varying across hints and models.The studies connect prompt components, error patterns, and intervention outcomes to guide cue and task choices.Source: §10 · p. 8

Limitations

The findings are limited by curated English-language cohorts, previously used image sets, and unresolved methodological distinctions. They characterize behavioral responses rather than internal model mechanisms.

Limitations

  • Curated English-language cohorts limit generalization, and small association sets yield wide intervals.Model-specific readouts also limit comparisons of score magnitudes.Source: Limitations · p. 9
  • Both RQ2 image sets were used previously; External-48 also changes question pairing.RQ3 covers two models, with 12 source pairs in its fixed follow-up.Source: Limitations · p. 9
  • Without a length-matched neutral hint, no-hint comparisons cannot isolate role semantics from added-text effects.RQ3 also changes task instructions and answer meanings together, leaving their separate contributions unresolved.Source: Limitations · p. 9
  • The findings characterize behavioral responses to prompt interventions, rather than internal model mechanisms.Source: Limitations · p. 9

Ethical Considerations

This section addresses responsible data release and documents the role-hint evaluation setup and interpretive safeguards. It covers license and privacy constraints, evaluation cohorts and procedures, and factors that qualify interpretation of hint-related effects.

Ethical Considerations

  • The planned release uses source identifiers, links, attribution, and permitted annotations, not original images or private annotated renderings.This approach is intended to respect source licenses and privacy.Source: Ethical Considerations · p. 9

G.2 Spatial prediction and paired inference

  • Qwen3’s predictive estimates remained negative on External-48 and Spatial-96, whereas Qwen2’s estimates were near zero on both cohorts.External-48 estimates were −1.623 [−3.139, −.334] for Qwen3 and +.001 [−.007, .009] for Qwen2; Spatial-96 estimates were −1.222 [−2.049, −.387] and −.002 [−.007, .002], respectively.Source: §G.2 · p. 18

G.3 Association prediction and token-length sensitivity

  • Association-prediction role effects remained uncertain: adjusted p-values were 1.0, the joint sign-only criterion was unmet, and matched token counts prevented isolating a length effect.Separate local-only improvements were −.002941 [−.005859, −.000739] for Holding and −.002514 [−.005549, −.000345] for Attached-to; the pooled role intervals were wide.Source: §G.3 · p. 19

H Ordinary-Query and Nonlinear Predictive Controls

  • At matched query budgets, Qwen3’s fourth label condition added predictive information, but separately worded ordinary-query alternatives were stronger for this target.Qwen3’s four-minus-three differences were −1.924 [−3.568, −.601] and −1.355 [−2.048, −.695]; ordinary-query alternatives were nevertheless substantially stronger.Source: H Ordinary-Query and Nonlinear… · p. 19, Table 9 · p. 20

I.1 Average order effects

  • Across local models, preserving hints showed little average help while swapped hints caused harm; blind gray-carrier and image-free inputs yielded 50% accuracy and zero both-correct outcomes for five models.Hinted variants had identical plain-text token counts, while no-hint prompts were shorter; NVIDIA’s accuracy gains could not be attributed solely to better visual decisions because hinting also improved format validity.Source: §I.1 · p. 20, §I.3 · p. 21, §I.4 · p. 21

I.2 Observed cells and cue effects

  • Qwen3’s preserving hints stayed near baseline, while swapped hints drove the pair-both-correct gap: DC = 10.94 pp [4.17, 18.23] and H = −.91 pp [−5.34, 3.78].The Holm-adjusted p-value was .0112 for DC > 0 and .739 for H > 0.Source: §I.2 · p. 21, Table 11 · p. 22

J.2 Question position and repetition

  • On External-48, moving the question last did not reduce canonical swapped-hint harm, and swapped hints mainly shifted Qwen3 and mini response pairs from both correct to No/No.Qwen3’s primary First content-by-order interaction test had Holm-adjusted p = .0017, while the test of reduced swapped-hint harm under Last had p = .871; Qwen3 had 29 both-correct-to-No/No transitions and mini had nine, with none in reverse.Source: §J.1 · p. 22, §J.2 · pp. 22–23, §J.3 · p. 23

J.4 Field-order weighting

  • Changing field-order weights changed effect sizes but preserved Qwen3’s help–harm interpretation in both cohorts; mini’s swapped-hint change stayed positive on Spatial and negative on External-48.This supplementary weighting analysis was specified after the original results.Source: §J.4 · p. 23

K.1 Hint Meanings and Prompts

  • The hint-rewriting setup treats C and E as claim-equivalent, S and F as equivalent opposite relations, and compares claim verification with direct relation selection.Each relation has canonical-field and complete-sentence forms, plus a no-hint condition.Source: K Hint Rewriting and Response Tasks · p. 23, §K.1 · pp. 23–24
  • Role hints specify a relation through answer choices or a sentence, while the no-hint line is “(none)”; both answer mappings are tested, and relation selection retains the original claim.For example, the image-based choices ask whether the cup is left or right of the bowl, while the sentence hint says the bowl is left of the cup; task instructions and answer meanings change together.Source: §K.1 · p. 24

K.2 Cohorts and Evaluation

  • Sentence hints yielded 32/32 text-only accuracy for both Qwen3 and GPT-4.1-mini, versus canonical-field scores of 19/32 and 28/32.Other field orders scored 20/32 and 19/32 for Qwen3, and 18/32 and 17/32 for GPT-4.1-mini; field and sentence outputs came from separate batches.Source: §K.2 · p. 24
  • The visual studies used 24 images in 12 source pairs for rewriting, 12 images in six pairs for the pilot, and 24 further images in 12 disjoint pairs for follow-up.Each image supplied four claims crossing object order and relation direction, with two true and two false claims; each claim used both answer mappings, and the follow-up retained the pilot’s full design.Source: §K.2 · p. 24
  • Visual evaluations used Qwen3-VL-8B-Instruct and GPT-4.1-mini-2025-04-14 at temperature zero, accepting only complete A/B answers and counting invalid responses as errors.The initial rewriting and task-pilot studies each required 1,728 requests per model; follow-up required 3,456, with 192 judgments per task–hint cell, contemporaneous no-hint baselines, and interleaved matched blocks. Follow-up intervals were descriptive 95% bootstrap intervals from 20,000 paired draws over 12 source pairs, keeping claims, mappings, and conditions together.Source: §K.2 · p. 24

K.3 Complete Results and Error Trade-offs

The follow-up finds shared recovery from relation-reversed field hints, alongside model- and hint-dependent effects of relation selection and hint wording. Answer-checking results vary with comparison design, and factual histories shift support toward crossed candidates.

K.3 Complete Results and Error Trade-offs; K.4 Source-pair influence

  • Relation-reversed field hints provide the clearest shared recovery, and their positive contrasts persist across every source-pair omission for both models.The contrasts range from 7.95 to 11.93 pp for Qwen3 and from 13.64 to 19.32 pp for mini; these ranges describe pair influence, not confidence intervals.Source: §K.3 · p. 24, §K.4 · p. 26, Table 20 · p. 27

K.3 Complete Results and Error Trade-offs

  • In the fixed follow-up, Qwen3 corrects 26 true-claim judgments with no added errors; mini corrects 55 but adds 22 errors.Correctly answering both opposing claims rises from 2 to 28 of 96 for Qwen3 and from 32 to 69 for mini.Source: §K.3 · pp. 24–25, Table 17 · p. 25
  • In the rewriting cohorts, Qwen3’s E sentences improve while S sentences fall, whereas mini’s F-sentence effect reverses direction in follow-up verification.For Qwen3, E improves by 10.42 pp initially and 14.58 pp in follow-up verification, while S falls by 21.88 and 21.35 pp; in the initial cohort, greater agreement between equivalent hints also includes more jointly wrong judgments.Source: §K.3 · p. 25, Table 19 · p. 26

L Answer Checking: Matched and Order-Only Comparisons; L.1 Fixed answers, scoring, and evaluation

  • Answer checking ranks fixed original answers by disagreement with three auxiliary checks; NVIDIA’s ranking also reflects 61 invalid originals, not visual errors alone.Each strategy uses four queries, and the score uses disagreement with the initial answer rather than ground truth.Source: §L.1 · pp. 26–27

L.2 Matched wording and answer-label changes

  • In matched-budget checking, GPT comparisons favor role cues, while Qwen2, LLaVA, and NVIDIA intervals include zero.Qwen3’s mean disadvantage is smaller than in the order-only comparison and its interval includes zero.Source: Table 22 · p. 28, §L.2 · p. 28

L.3 Order-only sensitivity comparison; L.4 Errors in Stable Responses

  • For Qwen3, swapped-hint consistency has worse secondary AURC than ordinary checking: +5.74 [0.74, 10.89] on the ×100 scale.The order-only comparison varies field order in role arms, while ordinary checks vary wording and answer labels.Source: §L.4 · p. 28, §L.3 · p. 28

L.5 Query and Computational Costs

  • Each checking strategy uses four queries, but ordinary and role-preserving strategies differ in mean per-answer input-token totals, including 1151/1192 for Qwen3.Local-model timings are unavailable, and tokenizers and image accounting differ.Source: §L.5 · p. 28

M Attribute Judgments with Factual Histories

  • Across four models, TERM histories produce a larger shift toward the crossed candidate than FACT histories.The mean probability-scale interactions are .0649 [.0584, .0714] for FACT and .1455 [.1366, .1544] for TERM; attention, prompt length, and attribute recency differ between them, leaving several explanations for the larger TERM effect.Source: M Attribute Judgments with Factual… · p. 29
Loading 2610.28843v1…