PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Parts, Reasoning about Worlds: Visual Inference under Partial Observation

Wei Wang, Wenqiao Zhang, Yutong Lin, Jun Xiao, Yueting Zhuang

arXiv:2609.32456v1cs.CVcs.AI

TL;DR

Partial visual observations can leave several complete worlds possible, making it necessary to distinguish exact conclusions from supported bounds and unresolved claims. WorldScope combines possible-world semantics, data and benchmarks, and WorldFlow’s subset evidence representations. On WorldScope-Bench, WorldFlow reaches 64.34% exact accuracy, 24.88 points above same-backbone QA-only training.

  • Problem

    Occluded objects and unseen regions can leave multiple complete worlds compatible with the visual evidence, so the strongest shared conclusion may be a bound or an unresolved claim rather than an exact answer.

    Source: §1 · p. 1

  • Method

    WorldScope combines question–answer data, certified counterworld supervision, and benchmarks with WorldFlow representations that track compatible worlds as observations change.

    Source: §1 · p. 3, §6 · p. 20

  • Results

    64.34% exact accuracy, 24.88 points above same-backbone QA-only training on WorldScope-Bench.

    Source: §6 · p. 20

  • Takeaways & Limitations

    WorldScope studies which conclusions models can reliably draw from partial observations and how those conclusions change as visual evidence is acquired, removed, or composed.

    Source: §1 · pp. 1, 3

    WorldScope is limited to controlled tabletop scenes with 38 target categories, support-surface queries, rendered HSSD interiors, and finite camera families; structure-disjoint testing does not establish transfer to arbitrary photographs or embodied environments.

    Source: §6 · p. 20

Abstract

from arXiv · show

World modeling under partial observation requires reasoning about the complete worlds that remain compatible with limited visual evidence. Occluded objects and unseen regions can leave several world states possible; additional views can exclude alternatives and strengthen the conclusions supported by the observations. We introduce WorldScope to study this process through possible-world semantics, evidence-grounded data, and learned visual representations. WorldScope-1.2M provides 1.2 million English question-answer pairs spanning eight world properties, ten task interfaces, and three observation protocols. Its answers encode confirmed facts, supported bounds, and unresolved possibilities. Complementary supervision comprises 4,800 certified counterworld groups with equivalent base observations and different hidden object configurations and query answers. These groups provide physical witnesses of ambiguity and training-only labels for world compatibility and view-induced exclusions. We propose WorldFlow, which composes cross-view entity evidence and support-surface coverage into an image-subset evidence lattice. Counterworld compatibility and transition objectives train subset representations to reflect how new observations constrain possible worlds. A shared answer generator uses these representations to predict the strongest supported conclusion. WorldScope-Bench evaluates claim judgments and evidence-dependent conclusions as views are selected, combined, removed, or ordered. On its 5,000-question test set, WorldFlow reaches 64.34% exact accuracy, improving over the same backbone trained on QA alone by 24.88 percentage points. It retains 50.43% accuracy on the 3,000 questions from structure-disjoint scenes.

1 Introduction

Partial visual evidence can leave several complete worlds possible, so sound reasoning must preserve uncertainty while stating the strongest conclusion they jointly support. WorldScope addresses this with data, evaluation, and a model designed to track how observations constrain those worlds.

1 Introduction

  • A partial scene can support a lower bound without supporting an exact count, because unseen regions may contain additional objects.The target is the strongest conclusion shared by all complete configurations compatible with the observations.Source: §1 · p. 1
  • Existing visual question-answering and synthetic-scene datasets generally provide one answer without labeling hidden configurations or distinguishing exact answers, bounds, and unresolved conclusions.Existing benchmarks also do not systematically test evidence boundaries or answer updates as observations change.Source: §1 · p. 2
  • WorldScope-1.2M provides 1.2 million question–answer pairs, while certified counterworld groups add visual compatibility supervision without a second language-question distribution.The dataset spans eight world properties, ten task interfaces, and three observation protocols.Source: §1 · p. 3
  • WorldScope-Bench tests whether conclusions stay within the evidence as images are added, removed, or combined across ten task interfaces.Its 5,000 balanced questions include 3,000 from three HSSD structures unseen during training.Source: §1 · p. 3
  • WorldFlow combines per-image evidence, cross-view entity fusion, support-surface coverage, and image-subset encoding to represent how observations constrain possible worlds.Certified counterworlds supervise the contraction of compatible worlds as observations are added.Source: §1 · p. 3

2 Related Work

Prior work provides multi-image interfaces, cross-view identity methods, incomplete-evidence benchmarks, and partial-observability frameworks, but the cited work leaves open how to track conclusions across compatible complete worlds as visual evidence changes.

2.1 Multi-Image Reasoning

  • Multi-image systems combine image content, but final-answer supervision does not establish which view adds evidence, whether a subset suffices, or whether claims exceed visible support.Spatial and multi-view methods target spatial relations or scene information; evidence sufficiency also requires tracking informative, repeated, and irrelevant views.Source: §2.1 · pp. 3–4

2.2 Cross-View Identity

  • Cross-view identity and object-centric representations help match repeated entities, but do not represent what unseen regions may still contain or evidence relations across arbitrary image subsets.Correspondence alone cannot determine whether an additional entity remains outside matched regions or behind an occluder.Source: §2.2 · p. 4

2.3 Incomplete Evidence

  • Existing entailment, diagnostic, and abstention benchmarks do not certify which alternative complete scenes remain consistent with partial observations or distinguish evidence-induced indeterminacy from model uncertainty.A further question is whether an unseen support region could contain an instance that changes a count or existence claim.Source: §2.3 · pp. 4–5

2.4 Partial Observability

  • Belief-state and learned world-model frameworks address sequential interaction, prediction, or control, while active-perception methods leave open assessing supplied or candidate observations before an action.These frameworks do not directly address which conclusions hold across all complete worlds compatible with current visual observations.Source: §2.4 · p. 5

3 The WorldScope Dataset and Benchmark

WorldScope formalizes partial-observation reasoning as determining which conclusions hold across all complete worlds compatible with the observed images. It builds evidence-grounded data and a benchmark around that semantics, with object coverage, view operations, and certified counterworlds supporting the study.

3.1 Problem Formulation

  • A proposition is true if every complete world compatible with the observations satisfies it, false if none does, and non-identifiable otherwise.Adding a view can contract the compatible-world set, while a repeated or irrelevant view can leave it unchanged.Source: §3.1 · p. 5

3.2 Scene and Observation Construction

  • The scene pipeline renders physical layouts once, then extracts evidence and generates answers across image subsets; privileged depth, semantic labels, and identities are used offline, not as model inputs.The held-out pool uses five additional HSSD structures, with structural scene identity as the split unit.Source: §3.2 · p. 7

3.3 Evidence-Grounded Answer Generation

  • Confirmed object identities set the count lower bound, while category- and size-dependent feasible placements in hidden space determine whether an additional instance remains possible.The answer generator reports supported intervals and evaluates propositions over them without substituting the complete-scene count when unseen space remains.Source: §3.3 · pp. 7–8

3.4 WorldScope-1.2M

  • WorldScope-1.2M comprises 1.2 million English question–answer pairs from 805,955 reasoning instances, with ten task interfaces receiving 120,000 questions each.Eight base-query families select the property to infer, while task interfaces determine how observations are inspected or changed.Source: §3.4 · p. 8

3.5 Certified Counterworld Supervision

  • Across 4,800 groups, shared base images can correspond to physically rendered completions agreeing on RGB, depth, and semantics yet disagreeing on answers; 3,491 have two completion answers and 1,309 have three.The groups produce 446,400 image-subset compatibility states and 1,080,000 one-view transitions from 14,400 complete worlds. The three completions are counterexamples, not an exhaustive enumeration of all possible worlds.Source: §3.5 · p. 10

3.6 WorldScope-Bench

  • WorldScope-Bench contains 5,000 visually grounded questions, including 3,000 from structures unseen during training, and scores conclusions as views are added, removed, combined, or ordered.Balanced task quotas make the overall question accuracy equal the macro-average across ten tasks.Source: §3.6 · pp. 10–11

3.7 Quality Control and Human Review

  • Automatic checks reject invalid scenes and evidence transitions, while human review covers rendered scenes, a stratified sample of training questions, and every benchmark candidate.After benchmark review, 97.2% of candidates remain in the released evaluation pool.Source: §3.7 · p. 11

4 Methodology

WorldFlow builds image-subset representations from confirmed entities and unresolved support-surface coverage, then uses them to produce conclusions supported by the observations. Its shared lattice supports varied image-selection tasks, while training combines visual grounding, world compatibility, and answer supervision.

4.1 Overview

  • WorldFlow combines confirmed entities with latent evidence about unexcluded support regions for each image subset, then uses the resulting lattice to support conclusions from partial observations.All task interfaces use the shared representation; the model receives no task identifier or task-specific answer head.Source: §4.1 · pp. 11–12

4.2 Observation Encoding

  • Each image’s Qwen patch features are resampled into 64 evidence tokens, with public image labels added to preserve image-role associations.Labels contain no task number or teacher annotation.Source: §4.2 · p. 12

4.3 Observed Evidence

  • Sixteen cross-view slots represent stable entity candidates, with matching and subset consistency preventing repeated views of one object from inflating confirmed counts.These slots support count lower bounds but cannot rule out additional objects in unseen regions.Source: §4.3 · pp. 12–13

4.4 Latent Evidence

  • The latent branch predicts support-surface coverage and combines views to estimate which placement regions remain unresolved.Its coverage grid is not an image-space visibility map; counterworld supervision adds global compatibility information during training only.Source: §4.4 · p. 13

4.5 Evidence Lattice

  • Four subset tokens summarize entity slots, coverage, and selected-view evidence; lattice nodes support single-view, multi-view, addition, removal, set, and sequence tasks.The same encoders are used across lattice nodes, which differ in selected images.Source: §4.5 · pp. 13–14

4.6 Answer Generation

  • Qwen receives projected subset tokens alongside the original image features and question, and generates native answer strings from RGB images, public labels, and the question.Complete-world teachers, compatibility heads, answers, scene metadata, and privileged sensor arrays are absent at inference.Source: §4.6 · p. 14

4.7 Learning

  • Training proceeds through scene grounding, world-compatibility learning, and joint QA updates, with counterworld objectives teaching subset states about compatible worlds and view-induced exclusions.Counterworld supervision shapes the subset encoder without overwriting grounded entity and coverage perception; the teacher and compatibility head are discarded after training.Source: §4.7 · pp. 14–15
  • Answer-token QA loss trains Qwen alongside auxiliary evidence objectives, while the student receives only RGB images, image labels, and the question.Scene records supervise entity, coverage, and subset targets; compatibility records supply certified worlds and one-view transitions.Source: §4.7 · pp. 14–15

5 Experiments

Experiments evaluate WorldFlow against QA-only and other models, test its architecture and training curriculum, and examine performance on structure-disjoint scenes. Across ten tasks, WorldFlow improves over QA-only; ablations and qualitative diagnostics examine evidence composition and updates.

5.1 Experimental Setup

  • The test set has 5,000 questions, including 3,000 from structures excluded from training; models share the same backbone, images, labels, and answer contracts.Exact accuracy is reported for ten equally sized tasks, and ablations vary architecture components or curriculum stages.Source: §5.1 · p. 15

5.2 Main Comparison

  • WorldFlow improves over QA-only on all ten tasks in the complete 5,000-question test set.Figure 4(a) compares WorldFlow with the same-backbone QA-only and native models and the strongest external result on each task.Source: Figure 4 · p. 16

5.3 Architecture Ablations

  • On unseen structures, the evidence lattice raises accuracy to 45.97%, and counterworld supervision raises it further to 50.43%.Adding entity or coverage evidence alone remains close to QA-only; removing entity evidence costs 7.03 points and removing coverage costs 8.00.Source: §5.3 · p. 16, Table 4 · p. 18

5.4 Training Curriculum Ablations

  • Removing Stage 1 lowers unseen accuracy by 11.36 points, while omitting both warmups lowers it by 14.23 points.With both warmups retained, QA-only Stage 3 and removing Stage 2 leave deficits of 2.90 and 2.73 points, respectively.Source: §5.4 · p. 17, Table 5 · p. 18

5.5 Generalization and Remaining Difficulty

  • Unseen-structure accuracy is 50.43% versus 85.20% on seen structures; minimal sufficient set, necessity/redundancy, and maximal identifiable answer remain hardest.Their unseen accuracies are 32.33%, 33.33%, and 34.67%, respectively; each requires an exact set or several mutually consistent fields.Source: §5.5 · pp. 17, 19

5.6 Qualitative Evidence Updates

  • As added views expose certified blind regions, predicted mean coverage rises and compatibility logits exclude alternative certified worlds.Across three examples, mean coverage rises from 62% to 92%, 73% to 96%, and 26% to 93%; these are explanatory cases, while aggregate results and ablations provide the broader evidence.Source: §5.6 · p. 19, Figure 5 · p. 19

6 Conclusion

WorldScope links partial visual evidence to compatible possible worlds and the strongest supported conclusions, with WorldFlow combining cross-view evidence and surface coverage. Its results show gains over QA-only training, while the paper bounds its claims to controlled tabletop scenes and identifies directions for extending the framework.

6 Conclusion

  • The benchmark isolates partial-observation reasoning in controlled rendered tabletop scenes, not arbitrary photographs or embodied environments.Its 38 target categories, support-surface queries, and finite camera families exclude deformable objects, articulated interactions, dynamic events, outdoor scenes, and real sensor artifacts.Source: §6 · p. 20
  • Three certified worlds witness ambiguity but do not exhaust physically possible completions, and WorldFlow’s 4×4 coverage map can discard fine geometry and object-specific placements.The denser data executor uses that information to construct gold evidence.Source: §6 · p. 20
  • Large-model results come from single runs selected on a fixed development set, and the seen–unseen structure gap signals a generalization challenge.Exact-set scoring gives no credit when a prediction omits even one view; the gap is especially marked on minimal sets and maximal identifiable answers.Source: §6 · p. 20
  • 64.34% exact accuracy exceeds same-backbone QA-only training by 24.88 points.The conclusion attributes the largest ablation gains to image-subset comparison and evidence-boundary reasoning.Source: §6 · p. 20
  • WorldScope combines 1.2 million questions, certified counterworld supervision, and a benchmark of ten ways to judge and compose partial evidence.WorldFlow combines cross-view entity evidence and physical-surface coverage in image-subset representations.Source: §6 · p. 20
  • Future work proposes certifying worlds from real multi-view captures, adaptive surface representations, richer possible-world supervision, and camera selection based on answer-set contraction.The proposed extensions include relations, object state, and temporal events.Source: §6 · p. 20

Related papers

Loading 2609.32456v1…