Source-linked AI summary
Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning
Chunxi Luo, Qishuai Ding, Xingchen Peng, Hengwei Ye, Ke Zhang, Wenxin Lian, Tianyi Lu, Chunlong Yu, Zhenya Lin
TL;DR
Spatial answers depend on both the specified reference frame and visual evidence, dimensions that item accuracy alone cannot assess across matched changes. FrameEvidence tests frame and evidence interventions with joint scoring. Prompting and adaptation improve item accuracy without consistent joint gains, while images most often benefit MSS–addition pairs.
Problem
Spatial reasoning evaluation must test whether models identify the language-specified frame and whether partial views jointly support an answer, since item accuracy alone does not establish correctness across matched inputs.
Source: §1 · p. 1Method
Results
Prompting and task-specific adaptation improve item accuracy without consistent joint-success gains, while images most often benefit MSS–addition pairs.
Source: §8 · p. 9Takeaways & Limitations
Paired analyses distinguish isolated correct answers from success across matched changes and identify incorrect frame switches, changes in initial MSS coverage, and execution instability.
Source: §8 · p. 9The benchmark tests static, Chinese, four-option spatial reasoning on held-out groups, which may share scenes across splits; metadata and image-ordinal changes can also influence responses.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctness persists when either changes. We introduce Frame Evidence, a benchmark of 320 matched triplets and 960 Chinese multiple-choice questions from captured and rendered scenes. One family compares three grounded reference frames over fixed images. The other pairs image deletion annotated as insufficient with answer-preserving addition. Joint metrics require correct answers across all three frames and paired evidence variants. Across 20 configurations on 288 held-out questions, GPT three-shot reaches 50.00\% item accuracy, but the best three-frame group accuracy is 5/48 (10.42\%), reached by GPT two-shot. Few-shot prompting and task-specific adaptation improve item accuracy without consistently improving joint correctness. Input controls show where images support matched success, while repeated executions reveal answer instability beneath similar scores.
1 Introduction
Spatial reasoning requires identifying the language-specified reference frame and checking whether partial views jointly support an answer, needs that item accuracy alone does not capture. FrameEvidence tests matched changes to frames and visual evidence, finding that improved item accuracy does not necessarily bring joint success.
1 Introduction
- Spatial answers depend on the language-specified reference frame, and partial views may need to jointly support the description.An arrangement that is left from a camera may be in front in another object's frame.Source: §1 · p. 1
- Item accuracy alone cannot establish whether correctness persists across matched inputs, where the correct answer may need to change or remain fixed.A directionally matching answer may still fail to reflect the specified frame.Source: §1 · p. 1
- FrameEvidence pairs three frame queries over fixed images with deletion-to-insufficiency and answer-preserving hard-image addition protocols.FQ1 joint scoring requires three correct frame answers; FQ2 scores a correct MSS answer together with a correct deletion or addition answer.Source: §1 · p. 1, Figure 1 · p. 2
- On 288 held-out questions, the strongest item-level setting answers 50% correctly but solves only 3 of 48 frame groups; images improve MSS–addition success in 11 of 12 open-weight configurations, item accuracy in six.Source: §1 · pp. 1–2
2 Related Work
Related benchmarks address spatial relations, reference-frame ambiguity, multi-view reasoning, and evidence dependence. FrameEvidence distinguishes its evaluation by requiring joint correctness across matched frame prompts or evidence interventions.
2 Related Work
- FrameEvidence's frame protocol measures joint correctness across three multi-view frame prompts, distinguishing a correct switch from an answer change alone.The cited comparison describes COMFORT as measuring probabilistic consistency on matched synthetic scenes with explicit frames.Source: §2 · p. 2
- Prior work spans spatial relations and reference-frame ambiguity, cross-view and scale variation, and subject-centered left–right judgments.The passage names COMFORT, FoREST, ViewSpatial-Bench, SCOPE, All-Angles, CityCube, and MirrorQA among these lines of work.Source: §2 · p. 2
- Multi-image benchmarks assess general reasoning and controlled failure cases, while spatial integration work focuses on cross-view evidence.The cited passage names MIRB, ReMI, MMIU, MuirBench, MIMIC, MMSI-Bench, CrossView Suite, and MultiView-Bench.Source: §2 · p. 2
- Evidence-centered diagnostics measure sufficiency, paired unanswerability, view selection, and sensitivity to evidence changes, while SSMNBench adds and removes views of minimal necessary sets.These measures characterize supporting views and average sensitivity; paired correctness on shared, correctly answered MSSs separates initial coverage from later intervention errors.Source: §2 · p. 2
3 Problem Formulation
FrameEvidence formalizes spatial multiple-choice items and two matched intervention families. Its group-level framing tests whether answers are jointly correct across frame changes or evidence variants.
3 Problem Formulation
- Each evaluation item contains ordered images, a Chinese question, and a four-option answer set, with models returning one option.Spatial answers use axial or quadrant directions; insufficient information is included for FQ2, while metric reconstruction, vertical relations, and motion are excluded.Source: §3 · pp. 2–3
- The protocol comparison table summarizes coverage of multi-image, matched, frame-switching, minimal-evidence, and joint evaluation dimensions.Its legend distinguishes explicit, partial, and unspecified coverage, and labels deletion-to-insufficiency and hard addition.Source: §3 · p. 3, Table 1 · p. 3
- FQ1 holds images and entity relation fixed while querying camera, object, and scene frames, with at least two distinct gold answers per group.The anchor and its forward or backward convention are specified; repeating one answer cannot solve the group.Source: §3 · p. 3
- FQ2 defines an MSS as subset-minimal evidence for a spatial answer, then pairs required-view deletion labeled insufficient with a hard, answer-preserving addition.Subset-minimality does not require minimum cardinality, and deletion can change image-ordinal bindings.Source: §3 · p. 3
4 FrameEvidence Construction
FrameEvidence combines captured and rendered static scenes into matched groups with controlled questions and review. Its splits keep matched variants together but permit scene overlap, so evaluation concerns held-out groups rather than necessarily unseen scenes.
4.1 Scene Acquisition and Group Assembly
- The benchmark combines 126 captured-image and 194 rendered-image groups of static layouts across tabletop, large indoor, and outdoor public-space scenes.Construction requires unambiguous identity matches, unchanged layouts, grounded frame anchors, and evidence combined across views.Source: §4.1 · p. 3
4.2 Annotation and Quality Control
- Controlled Chinese questions specify entities and reference frames, while each group receives a separate reviewer approval covering scenes, bindings, directions, MSS necessity, and answer preservation.Answer letters are balanced at 240 overall and 40 per condition.Source: §4.2 · pp. 3–4
4.3 Split and Validation Scope
- The split assigns 192/32/96 complete groups to training, development, and test, keeping matched variants and identical images together while allowing scenes to overlap across splits.The evaluation therefore concerns held-out groups, not necessarily unseen scenes; the reported human reference is one participant's sampled blind-answer summary.Source: §4.3 · p. 4
5 FrameEvidence Analysis
The analysis tests whether image-count and metadata cues predict FQ2 answers and paired success, then examines scene overlap, deletion-driven reference changes, and annotation boundaries. Controls achieve modest item accuracy, while several factors qualify how paired outcomes should be interpreted.
Composition and balance
- The benchmark balances relation ontologies and nearly equal scene strata, with 160 questions per condition.The ten most frequent labels account for 374 of 960 group-level occurrences, while 116 further types retain their source names.Source: §5 · p. 4
Evidence structure and non-visual controls
- 37.50% accuracy from a train-fitted count-plus-schema rule exceeds 33.33% always-insufficient, but NTS and DIS are only 6.25% and 2.08%.The figure covers FQ2 image counts and held-out metadata controls; test accuracy uses 144 items and NTS/DIS use 48 groups.Source: §5 · p. 4, Figure 3 · p. 5
Scene overlap and sensitivity
- Across the full test, rendered inputs yield higher DIS in 16/20 configurations and higher accuracy in 14, while scene/template overlap complicates subset interpretation.Exact images stay within one split, but 81 test groups share inferred clusters with training; the 15 groups in clusters absent from train and dev differ in composition from the full test.Source: §5 · p. 4
Ordinal references after deletion
- Deletion removes the original referent in 36/48 groups and shifts it in six; current-order ordinals resolve to another image in 38 cases or out of range in four.Both InternVL checkpoints’ joint deletion successes occur only when the referent is removed, so their NTS scores combine evidence removal and reference changes.Source: §5 · p. 4, Table 3 · p. 5
Annotation boundary
- Camera headings occur in 30/48 MSS prompts with non-camera tags, which define annotation strata rather than distinct frame conditions.Source: §5 · p. 4
6 Experiments
Across matched frame and evidence evaluations, item accuracy often diverges from success across all variants. Images more often improve addition-pair success than item accuracy, while adaptation and repeated runs reveal further limits on stable joint correctness.
6.1 Evaluation Setup
- The evaluation compares 20 configurations on 288 questions, using zero- to three-shot prompts with nested training-group demonstrations.Queries preserve image order and request one letter; invalid LLaVA-OneVision responses count as errors, while other main responses are valid.Source: §6.1 · p. 5
6.2 Item Accuracy and Matched Success
- 50.00% accuracy is GPT three-shot’s peak, but GPT two-shot leads FQ1-GEM and NTS at 10.42% and 22.92%, respectively.MiniCPM-V-4.5 matches three-shot’s 47.92% DIS at 32.99% accuracy; item accuracy and joint scores identify different leaders.Source: §6.2 · p. 5, Table 4 · p. 6
- NTS falls below the independent-rearrangement reference in 19/20 configurations, while DIS exceeds it in all 20.An image-free majority control shows the same asymmetry: repeated answers can succeed on answer-preserving addition but miss deletion’s insufficiency label.Source: §6.2 · p. 5
6.3 Input Controls and Adaptation
- Images raise DIS for 11/12 open-weight configurations but raise item accuracy for only six.Qwen2.5-VL-7B gains 43.8 DIS points despite lower accuracy; GPT gains 8.7 accuracy points over text-only. Using all images instead of the first raises GPT DIS from 8.33% to 35.42%.Source: §6.3 · pp. 5–6, Table 5 · p. 6
- Across three seeds, Qwen reaches 36.00±1.40% accuracy, yet every seed solves 6/48 addition pairs versus 22/48 for its control.For both primary LoRA–Base pairs, DIS intervals remain below zero under exact-image, scene and union clustering.Source: §6.3 · p. 6
6.4 Option-Order Robustness
- GPT leads Circular at 18.75%, while Qwen LoRA reaches 16.20±0.53% Circular accuracy despite greater consistency.InternVL’s higher Vanilla accuracy after adaptation accompanies lower Circular accuracy and consistency.Source: §6.4 · pp. 6–7, Table 6 · p. 7
- A repeated GPT run changes 46.53% of answers despite 45.14% accuracy versus 45.49% at rotation zero.Two-run joint correctness is 28.47% at fixed order and 28.12–31.25% across rotations, so Circular scores also reflect execution variability.Source: §6.4 · p. 7, Table 7 · p. 7
7 Stratified Results and Failure Analysis
The failure analyses show that answer changes do not necessarily improve frame correctness, and that paired outcomes depend on both initial evidence coverage and responses to interventions. Prompt-related item-level agreement likewise does not ensure matched deletion success.
Answer changes and frame groups
- GPT three-shot changes 94 frame pairs but answers both correctly on only 27; its complete triplets fall from five to three between two and three shots.Qwen3-VL-8B repeats a direction on 115/144 pairs, and pairwise accuracy can miss failure on the third frame.Source: §7 · p. 7
Insufficiency responses
- Qwen’s insufficiency recall rises from 6.25% to 79.17%, while precision shifts from 42.86% to 41.30%.The additional deletion hits come with more false insufficiency responses on answerable MSS and addition inputs; joint scoring requires success on both sides of the intervention.Source: §7 · p. 8, Table 8 · p. 8
MSS coverage and intervention outcomes
- Qwen loses fourteen DIS successes through changed MSS coverage and one among shared groups; InternVL loses three through each route.Qwen’s decline occurs mainly before addition, whereas InternVL also fails to retain initially correct answers.Source: §7 · p. 8
Prompt effects across executions
- Label agreement is 51.39% under original wording and 52.78% under stable IDs, yet NTS falls from 8/24 to 4/24 group–block observations.At fixed wording, 14–15/36 answers change; the small item-level gain does not carry over to matched deletion answers.Source: §7 · p. 8
8 Conclusion
FrameEvidence evaluates spatial reasoning under matched changes in reference frames and visual evidence. Its diagnostics distinguish isolated correct answers from correctness across matched changes.
8 Conclusion
- FrameEvidence tests spatial reasoning under matched changes in reference frames and visual evidence.Its joint metrics distinguish isolated correct answers from success across matched changes.Source: §8 · p. 9
- Prompting and task-specific adaptation improve item accuracy without consistently improving joint success, while images most often benefit MSS–addition pairs.These findings summarize the benchmark’s supported scope rather than establishing that every prompting or adaptation setting behaves the same way.Source: §8 · p. 9
- Paired analyses identify incorrect frame switches, changes in initial MSS coverage and execution instability.Together, these diagnostics separate isolated correct answers from success across matched changes.Source: §8 · p. 9
Limitations
The evaluation is limited to static, Chinese, four-option spatial reasoning on held-out groups that may share scenes with training data. Results also depend on tested settings and have limits in label validation and human-reference sampling.
Limitations
- Scene overlap across splits means the 96 held-out groups do not necessarily test generalization to unseen scenes.The benchmark evaluates static, Chinese, four-option spatial reasoning.Source: Limitations · p. 9
- Metadata cues and image-ordinal changes may influence responses alongside visual evidence, while NTS measures agreement with released deletion labels rather than independently validating them.Production review does not independently validate response-level labels or every-member MSS necessity.Source: Limitations · p. 9
- Comparisons reflect tested decoding settings, one demonstration pool, and provider-reported gateway identities; the human reference is one participant’s sampled summary without individual answers or a documented sample size.Three-seed deviations describe training variation, and Circular uses within-experiment controls.Source: Limitations · p. 9
Ethical Considerations
The released images came from an external data provider, and the authors describe language-model assistance in preparing the paper and its materials. The authors retain responsibility for evidence, citations, and privacy checks.
Ethical Considerations
- Captured and rendered images were supplied by an external data provider with permission for public release.The passage specifies permission for public release.Source: Ethical Considerations · p. 9
- Language-model assistance supported editing, scripting, and simulated review.The passage identifies these as forms of assistance.Source: Ethical Considerations · p. 9
- The authors remain responsible for checking the evidence, citations, and privacy of released materials.Responsibility for these checks remains with the authors.Source: Ethical Considerations · p. 9
A Implementation and Scoring Details
This section documents scoring and release checks and examines joint outcomes across interventions, metadata controls, and held-out groups. It also tests whether LoRA–Base outcome differences on FQ2 groups persist under alternative cluster-based uncertainty estimates, with NTS and DIS findings and a cluster-neutralization check.
A Implementation and Scoring Details
- The parser never guesses invalid answers: they count as false negatives for their gold class, and ten empty API failures among 5,760 main-table records remain incorrect.Reconstructed answers and point estimates match frozen results. The release preserves all 960 semantic gold answers and option-meaning sets, although raw fields differ in 917 questions; all 1,120 reviewed images match.Source: A Implementation and Scoring Details · pp. 11–12
B Release Composition
- Each of the six release conditions contains 160 questions, and object-label occurrences are counted once per matched group across 126 distinct labels.The “Other” category aggregates labels outside the ten most frequent types.Source: B Release Composition · p. 14, Figure 6 · p. 15
D.1 Geometry-associated differences
- Axial relations favor DIS across most configurations, while NTS has no median gap; granularity is more consistently associated with addition than deletion-label success.These contrasts are descriptive because strata contain different instances and share scene-dependent groups; annotation tags are not independently validated frame categories, and 30/48 MSS prompts with non-camera tags mention camera headings.Source: D Stratified Results and Paired… · p. 14, Table 15 · p. 15, §D.1 · p. 15
D.2 Unconditional response distributions
- Qwen Base-to-LoRA spatial-to-insufficiency switches numbered 28, 35, and 22 for MSS, deletion, and addition, with no reverse switches.InternVL had 10, 14, and nine such switches, plus one reverse switch in MSS; paired answers indicate a broader shift toward insufficiency across all three conditions.Source: §D.2 · p. 15
D.3 Common initially correct groups
- On common MSS-correct groups, deletion had four gains and no losses for Qwen, versus one gain and no losses for InternVL; addition lost one and three successes, respectively.The common sets are small and selected by observed outcomes, so these paired changes describe subsets rather than causal adaptation effects.Source: §D.3 · pp. 15–16, Table 21 · p. 18
D.5 Uncertainty
- The paired-contrast intervals condition on the held-out test and exact-image grouping; they do not capture provider nondeterminism, demonstration-selection variance, or all scene reuse.A zero-width interval at zero indicates no observed group variation, not zero population performance.Source: §D.5 · p. 18
D.4 Fixed-marginal joint-score diagnostic
- NTS joint success fell below its fixed-marginal reference in 19 primary configurations, while DIS exceeded it in all 20; FQ1-GEM showed no uniform direction.Across eight ordinary multi-seed configurations, FQ1-GEM and NTS gaps were negative and DIS gaps positive; configurations share test groups and are not independent replications.Source: §D.4 · p. 18, Table 22 · p. 19, Table 23 · p. 19
- Training-majority letter B alone produced NTS of 0.00% versus a 5.73% reference and DIS of 22.92% versus a 5.25% reference.The same negative-NTS/positive-DIS pattern can therefore arise without images; the reference preserves variant accuracy while rearranging successes across groups, and is descriptive because groups are heterogeneous.Source: §D.4 · p. 18, Table 24 · p. 19
D.6 Clustering sensitivity
- The analysis bootstraps 20,000 draws over 29 exact-image, 14 inferred-scene, or 13 union components, resampling clusters as blocks and normalizing replicates by sampled group count.Original point estimates divide by 48; the union joins groups connected by either relation, including indirect paths.Source: §D.6 · p. 19, Table 26 · p. 20
- Both NTS intervals include zero, while both DIS intervals remain negative under all three cluster partitions; InternVL’s unchanged NTS conceals two newly successful and two newly failed groups.The sensitivity analysis covers exact-image, inferred-scene, and union partitions.Source: §D.6 · p. 20, Table 26 · p. 20
- Across inferred clusters, Qwen’s NTS range remains positive and both DIS ranges remain negative after neutralizing each cluster’s net contribution while holding the denominator fixed.The adjustment computes 100(D − dc)/48 for a cluster contribution dc and overall total D.Source: §D.6 · p. 20
E Deletion and Image-Reference Audit
The audit finds that image deletion can also change what an ordinal image reference denotes, while repeated prompting comparisons show answer instability beneath similar aggregate agreement. Across the additional sensitivity analyses, item-level changes do not consistently carry over to joint correctness.
E Deletion and Image-Reference Audit; E.1 Original Inputs and Image-Order Traces
- 36 removed referents were all the original first image; 38 ordinals pointed to another retained image, and four were out of range.Group 00392 illustrates deletion of the referenced image, so the unchanged ordinal points to a different image.Source: E Deletion and Image-Reference Audit · pp. 20–21
E Deletion and Image-Reference Audit
- Qwen2.5-VL-7B joint deletion successes rose from one to five among 36 removed-anchor groups, while all four InternVL3-8B check-point successes were in that stratum.All four checkpoints failed jointly on the six retained-but-shifted groups; the six stable-ordinal or no-ordinal groups were too few to establish the pattern independently.Source: Table 27 · p. 21, E Deletion and Image-Reference Audit · p. 21
- 37/72 original-wording responses and 38/72 stable-ID responses matched labels, but their 1.39-point difference had an interval spanning zero.Across repeated blocks, 14/36 original-wording answers and 15/36 stable-ID answers changed; the initial 25-point decline did not recur.Source: Table 28 · p. 21, E Deletion and Image-Reference Audit · p. 22
F.1 Sensitivity to Scene/Template Overlap
- Qwen’s primary LoRA reduced DIS from 17/40 to 4/40 in overlap groups and from 4/8 to 2/8 elsewhere; these strata are descriptive, not causal.InternVL’s unchanged overall NTS combines opposing subgroup changes, and Qwen’s non-overlapping DIS decline comprises two gains and four losses.Source: §F.1 · pp. 22–24
F.2 Semantic Changes Across FQ1 Pairs
- GPT two-shot and three-shot each changed 94 frame pairs, with 26 and 27 jointly correct, yet fully correct triplets fell from five to three.All test gold frame pairs require a semantic change, so false-change rates on unchanged gold could not be estimated.Source: §F.2 · p. 24
F.3 Train-Only Text Shortcut Baseline
- The train-fitted text baseline answered 69 of 288 test items correctly and solved five MSS–addition pairs, all among groups where its MSS answer was correct.Addition agreement can persist even for a weak text-only classifier.Source: §F.3 · pp. 24–25
G.1 Captured and rendered inputs
- Rendered inputs yielded higher accuracy in 14/20 configurations and higher DIS in 16, but source contrasts also combine scene and question differences.Qwen2.5-VL-32B favored captured inputs on both endpoints, unlike its 7B Base.Source: §G.1 · p. 25
G.2 Paired Base executions
- Qwen changed seven answers for one net additional correct item, while InternVL changed eleven for two; joint counts remained unchanged for both.The paired runs use the same 288 questions.Source: §G.2 · p. 25, Table 34 · p. 26