PaperlayerView exact revision on arXiv

Source-linked AI summary

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

Hao Wang, Tao Yu, Liuzhou Zhang, HeXin Wang, Haopeng Jin, Yuxuan Zhou, Xinming Wang, Hongzhu Yi, Xinye Li, Yuanlei Wang, Ping Nie, Yan Huang, Yuxuan Zhang, Pengfei Zhou, Yanyan Zou, Wei Yang

arXiv:2609.35052v2cs.CVcs.AI

TL;DR

Existing protocols do not fully isolate whether video world models preserve objects from their initial observation. OPIS evaluates a fixed input inventory with object-level Presence, Identity, and Structure measures. Across evaluated models, object accounting is stronger than identity fidelity, which declines as the reference inventory grows.

  • Problem

    Existing frame-, video-, history-, and revisit-based protocols do not fully isolate preservation of objects observed in the input.

    Source: §1 · p. 3

  • Method

    OPIS evaluates a fixed initial-object reference using association, explicit visibility reasoning, and Presence, Identity, and Structure measures.

    Source: §1 · pp. 3–4

  • Results

    Across eight models, Presence ranges from 85.63 to 93.49 while strict Identity ranges from 28.12 to 36.88, and average Identity falls from 40.22 at at most 20 objects to 23.11 above 40.

    Source: §1 · p. 3, §5.2 · p. 10

  • Takeaways & Limitations

    The eight-model study reveals a substantial difference between object accounting and identity fidelity, supporting input-grounded object evaluation alongside broader video-quality measures.

    Source: §6 · p. 12

    OPIS measures observable input-object preservation, not internal memory mechanisms, physical causality, or complete world-model competence; its monocular reference is not complete 3D ground truth.

    Source: B. Limitations · p. 18

Abstract

from arXiv · show

Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's true memory capability. To address this, we introduce OPIS, an input-grounded benchmark that strictly anchors the assessment to a fixed set of object instances from the initial observation for evaluating multi-object memory in video world models. The OPIS dataset comprises 500 cases across real-world, embodied-robotic, and game-world domains, providing dense object-level annotations for 12,672 rigid, articulated, and deformable instances. Our object-centric evaluator combines association and explicit visibility reasoning to hierarchically measure Object (O) Presence (P), Identity (I), and Structure (S), utilizing static or dynamic evaluation tracks based on object kinematics. Across eight image-to-video or camera-conditioned world models, our proposed OPIS scores range from 48.65 to 56.01. As the reference inventory grows from less than 20 to more than 40 objects, the Presence, Identity, and Structure scores show an overall decline, with the average Identity score falling from 40.22 to 23.11. The results demonstrate that preserving the particular object instances in the input is considerably harder than generating plausible visual elements.

1. Introduction

Existing evaluation protocols do not fully isolate whether video world models preserve the particular objects in their initial observation. OPIS anchors evaluation to those input instances and finds that identity preservation remains substantially harder than object accounting.

1. Introduction

  • Existing protocols use frame quality, reference videos, generated histories, or selected revisits, but do not fully isolate preservation of input objects.The paper notes that these approaches can miss input-object anchors, penalize valid trajectory differences, inherit drift, or cover only selected frames and viewpoints.Source: §1 · p. 3
  • OPIS evaluates fixed input instances through Presence, Identity, and Structure, using one-to-one association, explicit visibility reasoning, and kinematics-based static or dynamic Structure tracks.Its references include instance masks, appearance descriptors, mobility labels, and auxiliary geometry.Source: §1 · p. 3
  • The benchmark contains 500 cases and 12,672 rigid, articulated, and deformable instances across real-world, embodied-robotic, and game-world domains.The instances comprise 8,432 rigid, 2,046 articulated, and 2,194 deformable objects.Source: §1 · p. 3
  • Presence ranges from 85.63 to 93.49, while strict Identity ranges from 28.12 to 36.88 across eight evaluated models.The comparison shows that accounting for observable objects is easier than preserving the particular instances established in the input.Source: §1 · p. 3
  • Average Identity falls from 40.22 for scenes with at most 20 reference instances to 23.11 for scenes with more than 40.The results identify increasing addressable-object count as a setting in which identity preservation is more difficult.Source: §1 · p. 3

2. Related Work

Prior work evaluates generated frames, videos, histories, revisits, and object-level consistency, but these approaches provide different forms of evidence. OPIS combines a fixed input inventory with object association, visibility reasoning, and Presence, Identity, and Structure measures.

2.1. Evaluation References and Memory

  • Prior evaluation protocols compare frame quality, reference videos, generated histories, or selected revisit views, each with different limits for testing input-object preservation.Video-reference comparisons may penalize valid continuations, generated histories may inherit drift, and revisit tests cover selected frames and viewpoints.Source: §2.1 · p. 4

2.2. Object-Level Evidence

  • Object-level evaluations already cover composition, human identity, entity consistency, visibility handling, and rigidity, but do not integrate a fixed input inventory with explicit missing-versus-unobservable reasoning.OPIS combines that inventory with one-to-one association, explicit visibility and unknown states, and object-level Presence, Identity, and Structure.Source: §2.2 · p. 4
  • Figure 2 presents OPIS as a 500-case dataset spanning three domains, ten subcategories, and rigid, articulated, and deformable objects.Its construction pipeline includes image selection, noun-phrase cleaning, instance annotation, geometry and task construction, and human review.Source: Figure 2 · p. 5

3. OPIS Dataset

OPIS evaluates observable preservation of objects from a fixed initial image, without requiring a target continuation or access to a model’s internal memory. Its dataset combines varied image sources and annotations, with human review of the resulting references and tasks.

3.1. Task and Scope

  • Each case pairs an initial image and instruction with a fixed evaluator-side reference, while the generator receives only the image and instruction.The reference is constructed before generation and is never updated with generated content, so evaluation does not require a target continuation or access to internal memory.Source: §3.1 · p. 5
  • OPIS instructions expose memory under changing viewpoints, motion, and partial visibility while allowing valid changes in camera pose, object motion, articulation, and deformation.They encourage re-observation, including cases where objects leave view and later reappear, but do not prescribe a future trajectory.Source: §3.1 · p. 5

3.2. Data Coverage and Sources

  • The dataset contains 500 cases across three domains and ten subcategories, with 12,672 addressable rigid, articulated, and deformable instances.Each subcategory contains 50 cases; the sources include real-world imagery, game screenshots, simulator-rendered scenes, and generated images.Source: §3.2 · pp. 5–6

3.3. Data Construction

  • Construction combines VLM-assisted image selection and annotation, concept-guided instance segmentation, and final human review.The pipeline cleans noun phrases, removes overlapping duplicate masks, extracts appearance embeddings for identity matching, and builds a geometric scaffold from the initial image.Source: §3.3 · p. 6

4. OPIS Evaluation

OPIS anchors memory evaluation to the fixed input inventory, establishing which objects are associated and observable before separately scoring Presence, Identity, and Structure. It evaluates frames and videos by failed-instance counts rather than failure proportions.

4.1. Association and Visibility

  • One-to-one association matches generated observations to the fixed input inventory using appearance, attributes, category, and geometric evidence, while visibility reasoning separates unobservable objects from absent ones.Static instances use projected input geometry and visibility checks; independently moving objects are assessed through direct observations.Source: §4.1 · p. 7

4.2. Presence and Identity

  • Presence zeros a frame for a confirmed missing or extra object, while Identity zeros it when any valid appearance score falls below its acceptance threshold.Confirmed occluded, out-of-view, or too-small instances are excluded from Presence; missing objects contribute to Presence but not Identity.Source: §4.2 · pp. 7–8

4.3. Structure Across Static and Dynamic Objects

  • Mobility labels route stationary rigid instances to static geometry, while articulated and other dynamic instances use structural evidence suited to their kinematics.The static track measures projective preservation from input geometry and image correspondences; dynamic evaluation uses landmark proportions or localized structural claims.Source: §4.3 · p. 8

4.4. Aggregation and Evidence Coverage

  • Case OPIS is a weighted P/I/S composite, macro-averaged across cases, subcategories, and domains; coverage is reported separately because low Structure may reflect failure or insufficient evidence.Structural coverage weighting prevents a few assessable frames from representing an entire rollout.Source: §4.4 · p. 9

5. Experiments

Experiments evaluate eight world models with OPIS and find that overall scores conceal weaker identity fidelity and structural evidence constraints. Diagnostics show performance varies with object load and scene domain, while evaluator validation and ablations test the scoring choices.

5.1. Experimental Setup

  • The setup evaluates eight systems across image-to-video and camera-conditioned interfaces, using approximately 10-second rollouts and sampling every eight frames.The implementation specifies segmentation, appearance and geometric evidence, dynamic-structure assessment, thresholds, and fixed P/I/S weights.Source: §5.1 · p. 9

5.2. Main Results

  • 56.01 and 55.57 are the highest overall OPIS point estimates for Seedance 2.0 and H3-Max-Turbo; Echo-WM-Flash leads camera-conditioned models at 51.42.Table 2 reports formal OPIS results with a 95% stratified case-bootstrap interval.Source: §5.2 · p. 10, Table 2 · p. 10
  • Presence ranges from 85.63 to 93.49, while strict Identity ranges from 28.12 to 36.88, exposing a gap between object accounting and preservation of particular input instances.Gemini-Omni-Flash has the highest Presence (93.49), but its OPIS is 50.43; neither Presence nor Identity alone predicts the overall ranking.Source: §5.2 · p. 10
  • Seedance 2.0 leads coverage-adjusted Structure at 60.08 and structural frame coverage at 81.53%, while coverage ranges from 67.42% to 81.53% across models.Structure is coverage-adjusted, so reporting it with coverage distinguishes broad evidence from scores based on fewer assessable frames.Source: §5.2 · p. 10

5.3. Object-Centric Diagnostics

  • Average strict Identity falls from 40.22 with at most 20 reference objects to 23.11 with more than 40, while threshold-failure frames rise from 44.24% to 67.55%.Domain performance also varies: seven of eight systems favor game worlds, while Matrix-Game performs best on embodied scenes.Source: §5.3 · p. 11

5.4. Evaluator Validation and Ablations

  • Removing the structural threshold raises scores by 5.05–8.60 points, removing the coverage multiplier raises them by 3.62–9.61, and using a minimum lowers them by 3.71–5.24.These variants recompute scores from the same archived object measurements.Source: §5.4 · p. 11
  • Structural-claim repeat agreement is 90.5%; manual-review agreement is 92.7% for association, 90.7% for visibility, 86.0% for identity, and 83.3% for structure.Structure has the lowest agreement in the audit.Source: §5.4 · p. 11

6. Conclusion

OPIS evaluates multi-object memory against a fixed initial observation through Presence, Identity, and Structure. The paper reports its scope, implementation and ethical considerations, and limitations on what the benchmark can measure.

6. Conclusion

  • OPIS evaluates multi-object memory against a fixed initial observation through Presence, Identity, and Structure, with strict frame scoring and separate coverage diagnostics.The eight-model study finds a difference between object accounting and identity fidelity, as well as sensitivity to structural coverage.Source: §6 · p. 12

AI Use Statement

  • Generative AI assisted with manuscript editing, dataset construction, and evaluation, while authors retained responsibility for scientific claims, definitions, citations, final statistics, and release decisions.The authors reviewed AI-assisted outputs and verified dataset and evaluator components and reported statistics.Source: AI Use Statement · p. 12

Ethics Statement

  • The benchmark uses licensed or attributed real-world, simulator, and game assets, plus a limited set of AI-generated images; its results may reflect biases in several data and evaluation sources.The paper reports no human-subject experiments, participant interaction, private data, or sensitive personal information.Source: Ethics Statement · p. 12

Reproducibility Statement

  • The evaluation code and configuration information are available publicly, and the dataset is available subject to applicable licenses.The reproducibility materials describe the evaluator, thresholds, aggregation, sampling, and bootstrap analysis.Source: Reproducibility Statement · p. 12

B. Limitations

  • OPIS measures observable input-object preservation, not internal memory mechanisms, physical causality, or complete world-model competence.Its monocular reference estimates visible geometry rather than complete 3D ground truth; some models' generation-length limits also leave long-rollout behavior unevaluated.Source: B. Limitations · p. 18

C. Dataset Composition

The dataset composition spans static-geometry and dynamic-structure evaluation across its annotated object instances.

C. Dataset Composition

  • Of 12,672 instances across 500 cases, 6,128 are assigned to static-geometry evaluation and 6,544 to dynamic-structure evaluation.The counts are reported in Table 3.Source: C. Dataset Composition · p. 18

D. Detailed Results and Reproducibility

The supplementary material provides numeric tables, confidence-interval procedures, temporal diagnostics, and generation and evaluator configuration details. The temporal analysis identifies identity preservation as the most fragile memory component as generation proceeds.

D. Detailed Results and Reproducibility

  • The detailed results section supplies numeric tables for main-text figures and per-subcategory scores, while the aggregation table caption defines the tested scoring variants.The table captions specify multi-object inventory-size comparisons and aggregation variants, including removing the structural threshold or coverage multiplier and replacing means with minima.Source: D. Detailed Results and Reproducibility · p. 18, Table 4 · p. 19, Table 5 · p. 19
  • Confidence intervals for the main results use 2,000 stratified case-level bootstrap replicates, resampling cases within each of ten subcategories and retaining the aggregation hierarchy.The procedure preserves intended domain weights while propagating case-level variation.Source: D. Detailed Results and Reproducibility · p. 18

E. Temporal Memory Diagnostics

  • Strict Identity drops from 43.13 at 1–4 seconds to 31.60 at 4–7 seconds, then reaches 33.24 at 7–10 seconds; Structure is more stable across these intervals.Structural frame coverage changes from 83.79% to 69.89% and then 76.27%.Source: E. Temporal Memory Diagnostics · p. 19

F.1. Generation and evaluator configurations.

  • Generation configuration records output duration, sampling stride, effective sampling rates, temporal timestamps, evaluator thresholds, and per-run manifests.All models use an eight-frame stride after the first second, and duration is measured from the output file.Source: §F.1 · p. 19

F.2. AI-Generated Dataset Images and VLM Quality Control

OPIS supplements collected scenes with a small set of AI-generated initial images, curated to provide visually usable, object-rich inputs for image-to-video evaluation. Generation prompts and review criteria aim to retain concrete foreground instances while excluding unsuitable imagery and recording provenance.

F.2. AI-Generated Dataset Images and VLM Quality Control

  • 83 AI-generated initial images supplement the collected scenes, spanning four real-world subcategories.The set contains 21 home indoor, 30 natural outdoor, 17 public indoor, and 15 urban outdoor cases.Source: §F.2 · p. 19
  • Prompts request coherent 16:9 scenes with at least 12 distinct foreground instances, varied depth and overlap, and surfaces useful for subsequent motion.They also exclude text, UI, malformed or duplicated objects, heavy blur, and scenes dominated by background scenery.Source: §F.2 · pp. 20–21
  • VLM review cleans each candidate inventory, lists distinct visible foreground instances, and assigns a quality score and flags.Multiple instances of one category remain separate; the review removes background scenery and interface elements.Source: §F.2 · pp. 21–22
  • Candidates are retained only with at least 10 foreground instances, a quality score of 4/5 or higher, at least 5 usable noun phrases, and no quality flags.Per-case provenance records generation and review details; release requires separate verification of usage and licensing conditions.Source: §F.2 · p. 23

G. Evaluator Validity

Evaluator checks compare repeated structural judgments and evaluator decisions with manual review. The scoring rationale treats a frame-level component as failed when any evaluable instance has a confirmed failure, and the case study reports selected examples with visible preservation failures.

G. Evaluator Validity

  • 529 of 600 manual-audit decisions agree (88.2%), while structural-claim judgments agree in 181 of 200 repeat pairs (90.5%); structure has the most manual-audit disagreements.The repeatability result measures judge stability, not correctness against manual review.Source: G. Evaluator Validity · p. 24

H. Rationale for OPIS scoring.

  • A single confirmed object-memory failure invalidates its frame-level component, so lower scores in denser scenes indicate difficulty preserving the complete observable inventory, not necessarily lower per-instance fidelity.The rationale favors an absolute failure criterion over the proportion of failed instances.Source: H. Rationale for OPIS scoring · pp. 24–25

I. Case Study

  • The case study retains examples with visible preservation failures and excludes low-scoring cases attributable only to incorrect evaluator association; Table 14 reports their case-level scores.The eight examples span seven dataset subcategories.Source: §I · p. 25, Table 14 · p. 25

I.1. Four detailed visual comparisons

Four visual comparisons illustrate case-level differences between reference and generated frames, with strict Presence, Identity, Structure, and OPIS scores reported for each case. The examples include both image-to-video and camera-conditioned models.

I.1. Four detailed visual comparisons

  • For Seedance 2.0 on home_indoor_0020, P=100.0, I=2.5, S=52.4, and OPIS=41.9; the dining-room composition remains visible, but chandelier branches, chairs, and tabletop objects differ.The comparison is at frame 56 (2.33 seconds).Source: §I.1 · p. 25, Figure 8 · p. 27
  • For Wan 3.0 on home_indoor_0006, P=94.1, I=4.2, S=14.7, and OPIS=26.4; the reference cat, coffee table, and sofa area are absent at frame 294.The dining table and bookcase remain the dominant foreground structures.Source: §I.1 · p. 25, Figure 9 · p. 28
  • For LingBot-World 2.0 on robot_lab_setup_0029, P=100.0, I=4.9, S=60.8, and OPIS=46.3; after the return interval, the original tabletop arrangement is not restored.At frame 152, a large dark object fills the foreground and remaining items differ in color and shape.Source: §I.1 · p. 25, Figure 10 · p. 28
  • For Matrix-Game 3.5 on Realistic_style_3D_world_0001, P=84.6, I=73.6, S=66.2, and OPIS=72.8; facial and armor details differ, and left-side figures are replaced.At frame 152, the standing figure and sacks are replaced by a blue bag and a different container arrangement.Source: §I.1 · p. 25, Figure 11 · p. 28

Related papers

Loading 2609.35052v2…