PaperlayerView exact revision on arXiv

Source-linked AI summary

What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck

Yiliu Xu, Xinjin Li, Yu Ma

arXiv:2610.19611v1cs.CV

TL;DR

Aggregate accuracy obscures where multi-image reasoning pipelines fail. The paper uses matched factorial Oracle interventions to test verified schema and ordered target-appearance information while retaining the original images and question. Ordered instance information produced the larger strict-accuracy rescue, localizing the dominant recoverable deficit to upstream instance recovery.

  • Problem

    End-to-end accuracy collapses multi-image pipeline stages into one outcome, obscuring whether failures arise in schema acquisition, ordered target recovery, or downstream lookup and composition.

    Source: §I · p. 1

  • Method

    A within-item 2 × 2 factorial Oracle design supplies a verified schema transcript, verified ordered target appearances, both, or neither while keeping the images and question fixed.

    Source: §I · p. 1, §III · p. 2

  • Results

    Supplying verified ordered instance information increased strict exact accuracy by 54.6 percentage points, and Oracle-B exceeded Oracle-A by 60.6 points across the fixed model panel.

    Source: A. Primary Oracle Effects · p. 5

  • Takeaways & Limitations

    The results localize the dominant recoverable deficit to ordered instance recovery rather than downstream schema use and composition.

    Source: §I · p. 1, §VIII · p. 8

    The study uses controllable synthetic images, so whether the same rescue profile holds for natural images, documentlike layouts, video frames, and realistic multimodal workflows remains to be tested.

    Source: §VII · p. 8

Abstract

from arXiv · show

Modern vision-language models can accept multi-image contexts, but aggregate accuracy rarely reveals where a multi-image reasoning pipeline fails. We study a controlled schema-application task in which a model must locate two target characters in a 16-image sequence, identify their visual attributes, consult an item-specific visual codebook, and compose two ordered nonce code names. We introduce a within-item 2-by-2 factorial Oracle design: Oracle-A supplies a verified textual schema, Oracle-B supplies a verified ordered target-appearance representation, and both interventions retain the original images and final question. Across 72 new items and 1,188 analyzed model responses, Oracle-B increased strict exact accuracy by 54.6 percentage points (95% confidence interval: 46.3 to 63.0) and outperformed Oracle-A by 60.6 points (Holm-adjusted p-value no greater than .0002). An auxiliary retrieval diagnostic reached only 49.1% ordered-appearance accuracy even after removing code-name generation. Providing both Oracle sources yielded 100% accuracy across all models. These results localize the dominant recoverable deficit to upstream ordered instance recovery rather than primarily to downstream schema use and composition.

I. INTRODUCTION

The paper addresses how to localize failures hidden by aggregate multi-image accuracy, using matched Oracle interventions to test which intermediate information restores performance. Across three commercial VLMs, ordered instance information produced a much larger rescue than schema transcription.

I. INTRODUCTION

  • End-to-end accuracy cannot distinguish whether failures arise in visual-schema acquisition, ordered target-instance recovery, or downstream lookup and composition.The related-work discussion positions the study alongside multi-image evaluation, long-context evidence use, and Oracle-based failure diagnosis.Source: §I · p. 1, §II · p. 1
  • The within-item 2 × 2 Oracle design keeps the 16 images and question fixed while supplying a verified schema transcript, ordered target appearances, both, or neither.The 16-image task is a controlled testbed for the diagnostic design, not the contribution by itself.Source: §I · p. 1
  • 54.6-point strict rescue came from supplying ordered instance information, while complete schema transcription barely changed semantic accuracy across three commercial VLMs.The factorial design and auxiliary retrieval controls support locating the dominant recoverable deficit in ordered instance recovery rather than downstream schema use and composition.Source: §I · p. 1

A. Multi-Image VLM Evaluation

Prior work measures multi-image capabilities, long-context evidence use, and intermediate failure diagnosis. This study pairs two independently repaired information sources within each item while retaining the same image context, question, and answer.

A. Multi-Image VLM Evaluation

  • Multi-image benchmarks measure end-to-end capability, whereas this study intervenes on intermediate information to identify where a controlled failure becomes recoverable.The cited work includes general multimodal benchmarks, MIRB and MuirBench [2],, MANTIS [4], and OSEF [20].Source: A. Multi-Image VLM Evaluation · pp. 1–2

B. Long-Context Retrieval and Position Sensitivity

  • Long-context and multimodal retrieval studies examine non-uniform evidence use and position sensitivity; this task additionally requires composing two relevant sources through an item-specific rule.The paper balances evidence order and absolute position because input order can change reasoning even when semantics are preserved –, [42].Source: B. Long-Context Retrieval and Position… · p. 2

C. Oracle and Decomposition Diagnostics

  • Unlike prior Oracle and decomposition diagnostics, this design independently supplies two intermediate information sources within each item while retaining the same 16-image context, question, and answer.Related approaches include separating extraction, perception, and integration failures in scientific VQA [12], distinguishing recognition from cross-property reasoning [13], and selectively repairing intermediate VQA steps.Source: C. Oracle and Decomposition Diagnostics · p. 2

III. FACTORIAL ORACLE DESIGN AND TASK INSTANTIATION

The paper defines a matched factorial Oracle evaluation and instantiates it as a 16-image task with separate schema and target-instance evidence. Each item requires ordered target recovery followed by visual-codebook lookup and code-name composition.

III. FACTORIAL ORACLE DESIGN AND TASK INSTANTIATION

  • The factorial Oracle design independently repairs two candidate intermediate representations while holding the original context and question fixed; it diagnoses recoverable deficits rather than providing a new solver.The task instantiates this evaluation design in a 16-image setting.Source: §III · p. 2

A. Task Pipeline

  • Each item requires locating two characters by their objects, identifying their color and head shape, preserving target order, retrieving codes, and composing two ordered names.Fig. 1 depicts the matched item, the two information paths, and the factorial intervention points.Source: A. Task Pipeline · p. 2, Figure 1 · p. 3

B. Visual Schema A

  • Evidence A is an item-specific visual codebook mapping five colors and five head shapes to nonce tokens, with a COLOR CODE - HEAD-SHAPE CODE composition rule.Ten distinct two-letter tokens are sampled without replacement for each item, and mappings are regenerated so tokens have no stable meaning across items.Source: B. Visual Schema A · p. 2

C. Instance Evidence B and Distractors

  • Evidence B contains two target characters that differ on both color and head shape, alongside 14 distractor images; target order and evidence placement are counterbalanced.The figure describes one schema image, one dual-target image, and 14 fixed distractors; A/B order and slot pair are counterbalanced.Source: C. Instance Evidence B and Distractors · p. 2, Figure 1 · p. 3

D. Main Conditions

The main conditions selectively provide verified schema information, ordered target appearances, both, or neither, while retaining the same images and question. Auxiliary diagnostics separate schema use from ordered-appearance recovery.

D. Main Conditions

  • Within each item, the ordered image list, question, expected answer, response instruction, and model settings stay fixed; only verified-information fields change.Oracle-A supplies the full color and shape mappings and composition rule but leaves target localization and recognition to the model.Source: D. Main Conditions · p. 3
  • Oracle-B supplies target appearances in the requested order but no schema token or final answer, leaving schema reading, code retrieval, and ordered name composition to the model.This intervention repairs target localization, character selection, attribute recognition, binding, and target-order preservation together.Source: D. Main Conditions · p. 3

E. Auxiliary Controls

  • Auxiliary controls include a text-only ceiling, Retrieval-A for schema use, and Retrieval-B for direct ordered-appearance recovery on the same balanced 36-item subset.These diagnostics are distinct from the factorial test and are intended to show what the main rescue pattern comprises.Source: E. Auxiliary Controls · p. 3, Table II · p. 4, Table III · p. 4

IV. EXPERIMENTAL DESIGN AND EVALUATION PROTOCOL

The protocol uses a balanced 72-item testbed and fixed model settings to compare Oracle conditions under controlled presentation and inference. Randomized runs and item-cluster analysis support paired diagnostic contrasts.

A. Dataset and Balance

  • The 72-item dataset balances appearances across 144 target roles, with each target-appearance pair unique.The same balanced 36-item subset supports all auxiliary controls.Source: A. Dataset and Balance · p. 4

B. Models and Inference Settings

  • Three fixed models provide a same-family OpenAI comparison and cross-provider replication.The evaluated panel includes two OpenAI models and Claude Sonnet 5.Source: B. Models and Inference Settings · p. 4
  • Each observation is an independent API request with no conversation history, using fixed nondeliberative configurations rather than each model’s maximum attainable performance.Anthropic extended thinking is disabled; the prompt requests a specified final-answer wrapper and no chain-of-thought.Source: B. Models and Inference Settings · p. 4
  • The analysis covers 1,188 responses: 288 main and 108 auxiliary observations per model.The counts correspond to 72 items across four main conditions and 36 items across three auxiliary controls.Source: B. Models and Inference Settings · p. 4

C. Run Order

  • Main-condition runs were randomized in blocks so every item appeared once before any item recurred, with no adjacent requests sharing an item.The same schedule was used across models; Retrieval-B used a separate randomized order.Source: C. Run Order · p. 4

D. Outcomes

The study prioritizes strict exact accuracy for both ordered code names, then estimates paired Oracle effects across a fixed three-model panel. Item-cluster inference and Holm correction assess the primary contrasts.

D. Outcomes

  • Strict dual-target exact accuracy requires both code names, target order, code order, and the requested answer wrapper to be correct.The study also reports semantic accuracy without the wrapper, target-level and atom-level accuracy, order reversals, and retrieval partial credit.Source: D. Outcomes · p. 4

E. Statistical Analysis

  • The primary panel estimand averages ordered-instance rescue equally across the three fixed models, comparing Oracle-B with Full.The defined model-level rescue is p01,m − p00,m.Source: E. Statistical Analysis · p. 4
  • The study estimates 95% confidence intervals with 10,000 item-cluster bootstrap draws and tests primary contrasts with item-cluster sign-flip tests and Holm correction.Bootstrap draws retain all model and condition observations for sampled items.Source: E. Statistical Analysis · p. 4
  • Secondary contrasts quantify schema, instance, and joint rescue, factorial interaction, and residual error; robustness analyses compare semantic and strict scoring and stratify by evidence center and order.All contrasts are reported on the probability scale.Source: E. Statistical Analysis · p. 4

V. RESULTS

The primary factorial results show a large advantage for supplying ordered target appearances over supplying schema text alone. Across models, the combined repair reaches ceiling, while semantic scoring confirms the pattern despite format failures.

A. Primary Oracle Effects

  • 54.6 percentage points: Oracle-B increased strict exact accuracy across the fixed model panel; it exceeded Oracle-A by 60.6 points.The respective 95% item-cluster confidence intervals were 46.3–63.0 and 52.8–68.1; both contrasts remained significant after Holm correction at pHolm ≤.0002.Source: Table IV · p. 5, A. Primary Oracle Effects · p. 5
  • Oracle-B rescue was +36.1 points for GPT-4.1, +52.8 for Claude Sonnet 5, and +75.0 for GPT-4.1-mini.The direction was consistent across models, and absolute rescue was larger for the weaker unassisted model.Source: A. Primary Oracle Effects · p. 5

B. Factorial Profiles and Residual Error

  • Oracle-B raised every model from full-task accuracy of 20.8%–63.9% to at least 95.8%, while Oracle-A+B reached 100% for all models.Across models, strict Oracle-A rescue was −6.0 points, joint rescue +57.4 points, conditional schema rescue after Oracle-B +2.8 points, and interaction +8.8 points.Source: B. Factorial Profiles and Residual Error · p. 5
  • Semantic-only scoring preserved the Oracle-B advantage: rescue was +52.3 points, and Oracle-B exceeded Oracle-A by +51.9 points.The strict Oracle-A drop was mostly a formatting artifact; semantically, schema text alone changed accuracy by +0.5 points.Source: B. Factorial Profiles and Residual Error · p. 5

C. Within-Item Rescue Transitions

Within-item transitions and representation controls reinforce the asymmetry: Oracle-B recovers many Full failures, while ordered-appearance retrieval remains difficult. Its advantage persists across evidence positions and orders.

C. Within-Item Rescue Transitions

  • Oracle-B rescued 119 Full failures, lost only one Full success, and left five pairs wrong in both conditions; Oracle-A rescued 14 and lost 27 successes.Under semantic scoring, Oracle-B rescued 113 and lost none, while Oracle-A rescued 19 and lost 18.Source: C. Within-Item Rescue Transitions · p. 5

D. Representation-Level Diagnostics

  • Retrieval-A accuracy was 97.2% for GPT-4.1, 100.0% for Claude Sonnet 5, and 91.7% for GPT-4.1-mini once ordered appearances were specified.Oracle-A+B text-only accuracy was 36/36 for every model on the auxiliary subset.Source: D. Representation-Level Diagnostics · p. 5
  • 49.1%: fixed-panel mean exact accuracy for ordered-appearance Retrieval-B, even without code-name generation.Model-level accuracy was 52.8% for GPT-4.1, 55.6% for Claude Sonnet 5, and 38.9% for GPT-4.1-mini.Source: D. Representation-Level Diagnostics · p. 5
  • Retrieval-B errors were often partial: target-level accuracy was 64.4% and color/shape atom accuracy was 72.9%, versus 49.1% for the complete ordered pair.No response exactly reversed two otherwise correct appearances; Full semantic accuracy was 77.4% when Retrieval-B was correct and 23.6% when it was incorrect.Source: D. Representation-Level Diagnostics · p. 5

E. Evidence Position and Order

  • Oracle-B rescue ranged from +51.4 to +58.3 points across evidence centers and was positive in both evidence orders.Rescue was +63.0 points when B preceded A and +46.3 when A preceded B; Oracle-B exceeded Oracle-A in every stratum.Source: E. Evidence Position and Order · p. 5

A. Where the Task Breaks

In the 16-image task, supplying ordered target appearances substantially improves accuracy, whereas supplying the schema alone barely changes it. Retrieval and transition analyses point to ordered instance recovery as the main recoverable deficit, within a setting whose fixed sequence length limits what the comparison can establish.

A. Where the Task Breaks

  • A +54.6-point strict-accuracy rescue followed Oracle-B, while Oracle-A changed semantic accuracy by only +0.5 points; both sources together brought all three models to 100%.Supplying ordered target appearances also raised semantic accuracy by 52.3 points.Source: Figure 2 · p. 6, A. Where the Task Breaks · p. 6
  • Oracle-B supplied colors and shapes, not schema tokens or code names, leaving codebook reading, four lookups, name composition, and order preservation to the model.Those downstream operations were not the limiting step in this experiment.Source: A. Where the Task Breaks · p. 6

B. Converging Evidence for an Instance-Information Bottleneck

  • Retrieval-B recovered complete ordered appearances on 49.1% of observations, versus 72.9% atom-level accuracy; Retrieval-A semantic accuracy was 91.7–100% once appearances were provided.Full semantic accuracy was 77.4% when Retrieval-B succeeded and 23.6% when it failed.Source: B. Converging Evidence for an… · pp. 6–7, Figure 4 · p. 7
  • Oracle-B rescued 119 strict-exact failures and lost one success, whereas Oracle-A rescued 14 failures and lost 27 successes.Fig. 3 partitions the same model–item pairs into rescued, lost, persistently wrong, and persistently correct outcomes.Source: B. Converging Evidence for an… · p. 6, Figure 3 · p. 7
  • Because sequence length was fixed at 16 images, the experiment cannot separate Evidence B’s intrinsic difficulty from the burden added by distractors.The conclusion is bounded to this 16-image schema-application setting.Source: B. Converging Evidence for an… · p. 7

C. Implications for Multi-Image Evaluation

The matched Oracle design complements broad multi-image benchmarks by testing recoverability at predefined intermediate boundaries. Its condition pattern was consistent across three models, and it suggests that upstream retrieval or verification may be a more useful system-design target than additional symbolic scaffolding when errors occur upstream.

C. Implications for Multi-Image Evaluation

  • The matched 2 × 2 Oracle contrast measures recoverability at predefined intermediate boundaries while preserving visual context, complementing task-level multi-image benchmarks.It estimates independent and joint repairs rather than proposing a new solver.Source: C. Implications for Multi-Image… · p. 7
  • When the dominant error occurs upstream, improving visual retrieval, external grounding, or intermediate-state verification may be more effective than adding symbolic reasoning scaffolds.This is a system-design implication stated within the paper’s identified failure setting.Source: C. Implications for Multi-Image… · p. 7
  • Ordered instance recovery is a candidate signal for targeted verification or escalation in multimodal systems that route uncertain states from edge to cloud.The cited CE-Agent example routes uncertain multi-step states to a cloud model and distills successful offloaded trajectories back to the edge.Source: C. Implications for Multi-Image… · p. 8
  • Across three models, Oracle-A produced little semantic gain, Oracle-B produced a large rescue, and Oracle-A+B reached 100%, despite substantial differences in unassisted accuracy.GPT-4.1 had the highest Full accuracy and smallest absolute rescue; GPT-4.1-mini had the lowest Full accuracy and largest rescue.Source: C. Implications for Multi-Image… · p. 8

VII. LIMITATIONS

The study’s conclusions are bounded by a controlled task, two coarse intermediate representations, and a fixed model panel. Synthetic inputs, bundled Oracle-B repairs, text timing, and strict exact scoring further constrain interpretation; within that scope, ordered instance information produced the largest recovery.

VII. LIMITATIONS

  • The design localizes a recoverable bottleneck only within one controlled task, two coarse intermediate representations, and a fixed model panel.Source: §VII · p. 8
  • Synthetic images make visual attributes and nonce codes controllable, but leave transfer to natural images, documentlike layouts, video frames, and realistic workflows untested.Source: §VII · p. 8
  • Because Oracle treatments arrive as text immediately before the question, their effects combine semantic correction with textual accessibility and recency.Matched irrelevant text or delayed Oracle information could separate these components more directly.Source: §VII · p. 8
  • Oracle-B repairs localization, character selection, attribute recognition, binding, and target-order preservation together, so the study does not isolate the smallest failing subroutine.Retrieval-B suggests errors accumulate across local selection and binding decisions, but finer interventions are needed to identify which upstream subprocesses dominate.Source: §VII · p. 8
  • Strict exact accuracy mixes answer correctness with instruction following, especially for Claude Sonnet 5 and GPT-4.1-mini in Oracle-A.The semantic-only analysis keeps the main contrast intact: schema text alone barely helps, while ordered target appearances produce a large rescue.Source: §VII · p. 8
  • The model panel is small and proprietary, hosted versions change, and the fixed 16-image context does not map effects of sequence length, distractor count, or visual similarity.Source: §VII · p. 8

VIII. CONCLUSION

  • Verified ordered instance information rescued strict accuracy by 54.6 percentage points, and Oracle-B exceeded Oracle-A by 60.6 points across the model panel.The effect appeared in all three models, survived semantic-only scoring, and both Oracles together reached 100% accuracy; Retrieval-B recovered complete ordered appearances on only 49.1% of observations.Source: §VIII · p. 8

Related papers

Loading 2610.19611v1…