World Observer: Joint Actor-Observer Generation for Persistent World Modeling

• Existing world models exhibit frame-locked, lost, frozen, and impostor states when objects leave the actor’s view.These failures respectively involve objects sticking to the image frame, disappearing, stopping their dynamics, or returning with incorrect evolution.Source: §1 · p. 2
• Memory, generative priors, and state extrapolation infer unseen evolution rather than observe it, with longer intervals or complex dynamics increasing uncertainty and drift.Source: §1 · p. 3
• World Observer jointly generates a perspective actor and one or more 360° panoramic observers, keeping selected regions represented as they evolve outside the actor’s view.Objects covered by an observer can reappear with updated states, mitigating the four described failure modes.Source: §1 · p. 3
• Warping the initial panorama supplies geometric correspondences across views, while high-resolution perspective references in the Observer Sink restore fine appearance detail.Source: §1 · p. 3
• Observers can be placed independently of the actor and controlled to guide out-of-view evolution; evaluation introduces world-space metrics and a real-and-synthetic benchmark.The metrics assess ground-truth motion agreement, self-consistency, and valid leave-and-return cases.Source: §1 · p. 3
• World Observer significantly improves out-of-view dynamics while remaining competitive in visual fidelity, camera control, and 3D adherence.Source: §1 · p. 3

• Training combines real panoramic video for real-world appearance with synthetic simulation for decoupled and multi-observer configurations rarely found in real captures.Source: §2 · p. 3
• Figure 3 contrasts real-world moving-observer examples above synthetic two-fixed-observer examples, with colored boxes marking cross-view correspondences and diagrams showing camera configurations and trajectories.Source: Figure 3 · p. 4
• Real sequences are stabilized panoramas paired with synchronized perspective actor videos rendered along randomized trajectories; both views share a panorama, so the observers are co-located.Per-frame depth is estimated for the geometry-aware condition.Source: §2 · p. 4
• CARLA supports independently placed observers, including several synchronized viewpoints that provide complementary coverage.Source: §2 · p. 4

3 METHOD

• A shared pretrained DiT jointly generates actor and panoramic observer streams in 3D-VAE latent space, conditioned on separate prompts and camera trajectories.Long videos are generated autoregressively in T-frame chunks, using tail latents from the previous chunk as history.Source: §3 · p. 4

3.1 JOINT ACTOR-OBSERVER WORLD MODELING

• Matched actor-observer latents share temporal positions, letting re-entering-object queries attend to the observer’s same-time out-of-view state.Separate prompts make the observer both evolving memory and a control channel for unseen events.Source: §3.1 · p. 5, Figure 5 · p. 5

3.2 PANORAMIC OBSERVER MODELING

• Warping the shared panorama and metric-scale depth into each stream’s viewpoint gives actor and observer views geometric correspondence; the observer runs below actor resolution.Coarse structure, object locations, motion, and visibility changes suffice for tracking out-of-view dynamics, keeping 360° coverage tractable.Source: §3.2 · p. 6
• Four fixed high-resolution perspective crops form an Observer Sink that preserves initial appearance while the observer stream captures temporal evolution, supporting detailed re-entry.The sink latents remain fixed and are appended so generated tokens can attend to them.Source: §3.2 · p. 6

3.3 FLEXIBLE OBSERVER PLACEMENT

• Observers can move independently of the actor, while multiple panoramic streams cover spatially separated or mutually occluded regions.For actor-view warping, the nearest observer panorama is selected as the actor moves; observer streams continue evolving jointly across their coverage regions.Source: §3.3 · pp. 6–7

4.1 OUT-OF-VIEW DYNAMICS METRIC

• The evaluation lifts segmented objects into 3D using metric-scale depth to measure their positions and out-of-view displacement in world space.The resulting displacement has comparable direction and magnitude across methods.Source: §4.1 · p. 7
• OOV-F measures valid exit-and-re-entry patterns, while OOV-Dgt and OOV-Dself compare out-of-view motion with prompt-ground-truth dynamics and pre-exit motion, respectively.OOV-Dgt is more sensitive to frozen motion; OOV-Dself is more sensitive to impostor motion.Source: §4.1 · p. 7

4.2 IMPLEMENTATION DETAILS

• The held-out evaluation uses 200 real and synthetic sequences and fits each full 77-frame sequence in one chunk, without memory retrieval.This setup isolates out-of-view maintenance from limited context or missing retrieval.Source: §4.2 · p. 8

4.3 RESULTS

• Qualitative results show that observers preserve evolving states outside the actor view, support prompt-driven events, and maintain consistency during long-horizon and multi-observer generation.Observers placed at selected locations cover regions occluded from the actor so objects can re-enter consistently.Source: §4.3 · pp. 8–9
• World Observer achieves the strongest out-of-view dynamics while remaining favorable in visual and temporal fidelity, camera-following accuracy, and 3D adherence.Adding observers further raises OOV-F while both OOV-D metrics remain competitive.Source: Table 1 · p. 9, §4.3 · p. 9

4.4 ABLATION STUDY

• Joint actor-observer generation combines strong out-of-view scores with stable OOV-F and strong camera control, 3D adherence, and visual fidelity.Without the actor, panoramic generation has strong OOV scores but weaker camera control and visual fidelity; without the observer, re-entering objects tend toward inconsistent states.Source: §4.4 · p. 9
• Removing the Observer Sink lowers OOV-Dself and also degrades 3D adherence, FID, and FVD.The Sink supplies shared high-resolution perspective references that ground appearance.Source: §4.4 · p. 9

5 CONCLUSION

• World Observer decouples observing from acting, jointly generating a perspective actor and panoramic observers that maintain world state beyond the actor’s view.The paper also introduces panorama grounding, an Observer Sink, and world-space metrics and benchmarks on real and synthetic scenes.Source: §5 · p. 10

A.1 VIDEO WORLD MODELS

• Actor-centric world models and approaches using priors, retrieved observations, or stored states lack direct evidence of how dynamic content evolves after it leaves the actor view.The paper contrasts these methods with jointly generating a panoramic observer that explicitly represents evolving out-of-view content.Source: §A.1 · p. 16

A.2 PANORAMA WORLD MODELS

• Prior panorama methods use panoramas as intermediate 3D representations or direct outputs, while World Observer lets observers cover selected regions independently of the actor’s view.The perspective actor remains the primary output, and the observer need not carry its fidelity.Source: §A.2 · p. 16

A.3 OUT-OF-VIEW STATE AND DYNAMICS

• Existing out-of-view benchmarks use VLM judgments of naturalness, while prior methods use memory, generative priors, or explicit state advancement to handle hidden dynamics.The paper notes limits including plausible but incorrect states, dependence on previously observed evidence, and requirements for structured entity-level modeling.Source: §A.3 · p. 16

B.1.1 ADDITIONAL TRAINING DETAILS

• Training uses teacher forcing, mixes real panoramic videos and synthetic CARLA videos at 60% and 40%, and trains the multi-observer stage on synthetic data only.History-frame noise is injected with 50% probability; each chunk conditions on H=2 history latents.Source: §B.1.1 · p. 17

B.1.2 ADDITIONAL MODEL DETAILS

• The model concatenates noisy target latents, warped-video encodings, a validity mask, view embeddings, and extra warp streams before patch embedding, with camera ray maps added afterward.Perspective and panorama tokens use distinct view embeddings; learnable observer-index vectors distinguish observers of the same type. The Observer Sink contributes four high-resolution perspective crops.Source: §B.1.2 · p. 17

B.2.1 PANORAMA VIDEO PROCESSING

• Panorama processing stabilizes raw 360° footage and estimates full-panorama depth by combining perspective-view predictions into a shared-scale map.Frames are gravity-aligned; overlapping 110-FOV views cover every direction with at least two views before their depth predictions are fused.Source: §B.2.1 · p. 18

B.2.2 TEXT CAPTIONING

• Captions use four perspective crops of each panorama and describe scene dynamics, including objects entering, leaving, or changing outside a perspective camera’s current field of view.Each clip receives a panorama caption and a perspective caption for each trajectory.Source: §B.2.2 · p. 18

B.2.3 TRAJECTORY CONSTRUCTION

• Six temporally aligned perspective trajectories comprise two static and four moving views, with sampled yaw, pitch, and field of view held fixed where specified.The moving paths include right and left rotations and paths that reverse direction.Source: §B.2.3 · pp. 18–19

B.3 SYNTHETIC DATA PROCESSING

• CARLA scenes vary towns, traffic, weather, and lighting, while metric depth and camera parameters support geometric referencing.Panoramas are stitched from six synchronized RGB-D cameras; one moving actor and two static observers capture the same world at the same simulation time.Source: §B.3 · p. 19

C.1 BENCHMARK CONSTRUCTION

• Benchmark clips deliberately rotate away from and back to the starting heading, with 1–3 manually curated objects that clearly exit and re-enter.Candidate scenes contain fewer than 10 dynamic objects; multi-observer captures use panoramas from the actor’s start and final positions.Source: §C.1 · p. 19

C.2 EVALUATION METRICS

• Generation quality is evaluated for visual and temporal fidelity, image quality, camera following, and static-scene 3D adherence.The evaluation uses FID and FVD, VBench, RotErr and TransErr, and masked PSNR and LPIPS, respectively.Source: §C.2 · pp. 19–20

C.3 OUT-OF-VIEW DYNAMICS METRICS

• Objects are tracked in 3D from segmentation masks and median depth; displacement compares their positions just before exit and after re-entry.Displacement is undefined unless an object actually exits and returns; all methods use the same segmentation and depth pipeline.Source: §C.3 · p. 20, Figure 8 · p. 20
• OOV-F reports the fraction of objects following the ground-truth exit-and-re-entry pattern, alongside OOV-D scores conditioned on valid returns.This prevents dropped or frame-locked objects from leaving a smaller, easier subset for displacement evaluation.Source: §C.3 · p. 20
• OOV-Dgt and OOV-Dself score out-of-view motion against ground-truth and pre-exit-motion references, respectively, exposing different kinds of inconsistency.OOV-Dgt penalizes frozen, under- or over-shot, and wrong-direction motion; OOV-Dself surfaces motion inconsistent with the object's prior trajectory.Source: §C.3 · p. 21

C.4 COMPARISON WITH EXISTING OUT-OF-VIEW METRICS

• VLM judges answered “Yes” for both a stopped object and a car whose reappearing motion broke from its prior trajectory, whereas 3D displacement metrics quantify these failures.OOV-Dgt exposes the frozen case and OOV-Dself the impostor case.Source: §C.4 · p. 21

C.5 BASELINE ADAPTATION

• Baseline camera trajectories are converted to the representations expected by each model while preserving the underlying motion; HyDRA instead uses its native out-of-view reconstruction protocol.HyDRA splits sequences at the trajectory apex and evaluates only the valid target interval.Source: §C.5 · p. 22

D.1 ADDITIONAL QUALITATIVE RESULTS

• Across diverse real and synthetic scenes, World Observer preserves target identity and state evolution through out-of-view intervals, unlike models showing lost, frozen, or impostor states.The comparison includes the Multi variant on synthetic scenes.Source: §D.1 · pp. 22–23, Figure 10 · p. 23
• The single- and multi-observer settings maintain target state throughout unseen intervals, with the multi-observer variant remaining consistent across regions watched by different observers.Source: §D.1 · pp. 22–23

D.2 ADDITIONAL QUALITATIVE ABLATION RESULTS

• Joint generation lets the actor produce in-view dynamics while the observer tracks out-of-view dynamics, unlike retrieval of isolated past frames.Without joint generation, actor history and the Observer Sink cannot maintain how the state evolves between observations; OOV-Dself drops markedly.Source: §D.2 · p. 23
• Removing the observer loses object consistency at re-entry, while removing the actor and generating only from the panorama reduces visual fidelity and introduces distortion.The ablations show disappearance or changed state without the observer, and reduced fidelity without the actor.Source: §D.2 · p. 23, Figure 11 · p. 24
• The Observer Sink supplies high-resolution perspective references that preserve fine appearance during large viewpoint changes and aggressive camera rotation.Without it, fine appearance degrades as rotation grows and errors accumulate over rollout.Source: §D.2 · pp. 23–24
• Decoupled actor and observer prompts place events in the surrounding world rather than hallucinating them in the actor's current view, and avoid duplicate subjects from conflicting stream outputs.The examples show an event appearing only when it enters the actor's view and a unified prompt producing a duplicate Santa.Source: §D.2 · p. 24

E.1 LIMITATIONS

• World Observer currently requires panoramic observations as conditioning inputs, restricting it to settings where such views are available.The authors suggest outpainting perspective observations into panoramas as a possible way to relax this requirement.Source: §E.1 · p. 24
• Without the observer, out-of-view object dynamics are not maintained; without the Observer Sink, fine appearance degrades under large viewpoint changes.In the reported 360° camera sweep, the full model preserves dynamic objects and appearance details upon re-entry.Source: Figure 12 · p. 25
• The observer budget is limited, so observing the entire world remains difficult.Flexible placement can allocate the budget to regions of interest, preserving their dynamics independently of the actor's movement.Source: §E.1 · p. 25

E.2 SOCIETAL IMPACT

• Persistent representation of selected regions may benefit embodied navigation, interactive simulation, and planning when relevant world parts are not currently observed.The passage ties these applications to flexible, multi-observer configurations that allocate a limited observation budget without requiring the actor to revisit those regions.Source: §E.2 · p. 25
• Decoupled prompting produces a consistent event across actor and observer streams, unlike actor-concentrated or unified prompting in the illustrated ablation.Actor-concentrated prompting misplaces off-view events in the actor view; unified prompting can duplicate subjects in the observer.Source: Figure 13 · p. 26