PaperlayerView exact revision on arXiv

Source-linked AI summary

Captain Safari: A World Engine with Pose-Aligned 3D Memory

Yu-Cheng Chou, Xingrui Wang, Yitong Li, Jiahao Wang, Hanting Liu, Cihang Xie, Alan Yuille, Junfei Xiao

arXiv:2511.22815v2cs.CV

TL;DR

Video world models struggle to preserve 3D coherence and follow aggressive camera paths through complex outdoor scenes. Captain Safari retrieves pose-aligned cues from persistent world memory, and its OpenSafari results show gains across video quality, 3D consistency, and trajectory following.

  • Problem

    Video world models struggle with long-range 3D coherence, aggressive camera trajectories, and complex outdoor layouts.

    Source: §1 · p. 2

  • Method

    Captain Safari maintains persistent world state and retrieves pose-aligned scene cues to condition video generation along target camera paths.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Captain Safari improves video quality, 3D consistency, and trajectory alignment on OpenSafari, and receives 67% of votes in five-way human comparisons.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Captain Safari and OpenSafari establish a benchmark for controllable video generation with verified-pose, in-the-wild drone footage.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

World engines aim to synthesize long, 3D-consistent videos that support interactive exploration of a scene under user-controlled camera motion. However, existing systems struggle under aggressive 6-DoF trajectories and complex outdoor layouts: they lose long-range geometric coherence, deviate from the target path, or collapse into overly conservative motion. To this end, we introduce Captain Safari, a pose-conditioned world engine that generates videos by retrieving from a persistent world memory. Given a camera path, our method maintains a dynamic local memory and uses a retriever to fetch pose-aligned world tokens, which then condition video generation along the trajectory. This design enables the model to maintain stable 3D structure while accurately executing challenging camera maneuvers. To evaluate this setting, we curate OpenSafari, a new in-the-wild FPV dataset containing high-dynamic drone videos with verified camera trajectories, constructed through a multi-stage geometric and kinematic validation pipeline. Across video quality, 3D consistency, and trajectory following, Captain Safari substantially outperforms state-of-the-art camera-controlled generators. It reduces MEt3R from 0.3703 to 0.3690, improves AUC@30 from 0.181 to 0.200, and yields substantially lower FVD than all camera-controlled baselines. More importantly, in a 50-participant, 5-way human study where annotators select the best result among five anonymized models, 67.6% of preferences favor our method across all axes. Our results demonstrate that pose-conditioned world memory is a powerful mechanism for long-horizon, controllable video generation and provide OpenSafari as a challenging new benchmark for future world-engine research.

1. Introduction

Existing video generators struggle to maintain long-range 3D consistency and follow aggressive camera paths in complex outdoor scenes. Captain Safari uses pose-aware world memory, and OpenSafari provides verified, high-dynamic FPV trajectories for evaluation; the method outperforms camera-controlled generators across the reported evaluation axes.

1. Introduction

  • The paper positions Captain Safari as a general, diverse, and interactive alternative to traditional game engines and feed-forward video generators.Traditional engines require heavy manual authoring and computation, while feed-forward generators lack persistent world state.Source: §1 · pp. 1–2
  • Existing systems struggle with long-horizon coherence, aggressive 6-DoF camera maneuvers, and diverse outdoor layouts.Under fast motion and sharp turns, pose-conditioned methods may either dampen requested motion or follow it at the cost of distortions and structural drift.Source: §1 · p. 2
  • Captain Safari maintains persistent world state and retrieves pose-aligned scene cues to guide aggressive camera motion while preserving 3D structure.The retrieval mechanism selects and aggregates informative cues rather than propagating the full long-term state.Source: §1 · p. 2
  • OpenSafari is an in-the-wild FPV drone dataset with verified camera poses, cluttered outdoor scenes, and rapid 6-DoF motion.Flights weave around buildings and vegetation across uneven terrain, with large parallax and sharp viewpoint changes.Source: §1 · p. 2
  • Captain Safari improves video quality, 3D consistency, and trajectory alignment on OpenSafari, with a 67% human preference rate in five-way comparisons.The authors also report clear gains in 3D consistency and tracking under complex maneuvers while maintaining strong perceptual quality.Source: §1 · p. 2

2. Related Work

Prior work adds geometric reasoning, memory, or explicit camera control to generative world models, but the approaches described here use clip-bound or heavy scene representations. Captain Safari instead couples long-horizon camera control with persistent pose-indexed memory.

2.1. 3D-Consistent World Models

  • Figure 2 depicts Captain Safari retrieving pose-aligned tokens from local world memory to condition generation along a user-specified trajectory.The overview describes this conditioning as preserving a stable 3D layout.Source: Figure 2 · p. 3
  • Earlier 3D-consistent world models use geometric supervision, spatio-temporal or adaptive memory, and open-world scene representations.The paper contrasts these approaches' implicit clip-bound memories or heavy point clouds with its compact, fixed-cost decoupled retriever.Source: §2.1 · pp. 2–3

2.2. Camera Controlled Video Generation

  • Camera-control methods use explicit trajectory conditions, path-alignment losses, geometric priors, or shared scene representations, while Captain Safari uses persistent pose-indexed memory across trajectories.The cited prior approaches improve control or consistency within clips or across paths, but typically build one-off 3D scenes.Source: §2.2 · p. 3

3. Captain Safari

Captain Safari represents the world with a local memory and retrieves scene tokens according to target camera poses. The retrieved tokens condition a DiT generator to produce clips along the prescribed trajectory.

3. Captain Safari

  • Captain Safari combines implicit world memory with pose-conditioned retrieval to guide a DiT generator along arbitrary camera trajectories.Source: §3 · p. 3

3.1. Implicit Memory of World Geometry

  • For each target clip, a bounded local window of shared memory keeps conditioning temporally nearby, while overlapping windows couple neighboring clips to the same world.Source: §3.1 · p. 4

3.2. Memory Retrieval and Conditioning

  • Pose and memory tokens are jointly embedded, and target-pose queries use cross-attention to retrieve relevant tokens rather than relying on temporal proximity.Source: §3.2 · p. 4
  • Retrieved clip-level world tokens enter DiT hidden space and are reused in memory cross-attention across layers to shape denoising.Source: §3.2 · pp. 4–5

4. OpenSafari

OpenSafari targets real-world FPV drone flights with aggressive motion and complex outdoor scenes, pairing them with camera trajectories checked for geometric and kinematic plausibility. The resulting dataset is designed to challenge camera-controlled video generation.

4.2. Camera Trajectory Reconstruction

  • Camera trajectories are estimated at 4 fps, then checked using SfM statistics, geometric verification, and kinematic tests for implausible motion.Sparse localized errors can be fixed by interpolating camera centers and SLERP rotations, followed by revalidation; trajectories with dense or persistent failures are discarded.Source: §4.2 · pp. 5–6
  • The benchmark table caption reports Captain Safari ranking first in 3D consistency and trajectory following, with competitive video quality and a slight quality trade-off versus the no-memory variant.The human-preference table caption reports 67% of votes for Captain Safari across criteria.Source: Table 1 · p. 6, Table 2 · p. 6

4.1. Video Data Curation

  • OpenSafari pairs in-the-wild FPV drone videos with verified poses, emphasizing aggressive 6-DoF motion, strong parallax, and complex outdoor layouts.Clips are retained for stable, coherent motion rather than near-static views.Source: §4.1 · p. 5, §4.2 · p. 6

5. Experiments

Experiments evaluate Captain Safari on video quality, 3D consistency, and trajectory following. It leads the reported comparisons across these axes, while ablations highlight the role and trade-offs of pose-conditioned memory.

5.1. Implementation Details

  • Training warms up the pose-conditioned memory retriever before joint retriever–DiT training, using LoRA updates for the DiT.The setup uses 11,481 final training clips and a 787-clip non-overlapping test set; generated clips are 5 seconds at 24 fps with a 5-second memory window.Source: §5.1 · p. 6

5.2. Benchmark

  • Evaluation measures video quality with FVD and LPIPS, 3D consistency with MEt3R and reconstruction rate, and trajectory following with AUC and pose cosine similarity.The benchmark compares three camera-controllable generators and includes 50 participants selecting among five generated videos per case across three criteria.Source: §5.2 · pp. 6–7

5.3. Generation Quality; 5.4. 3D Consistency; 5.5. Trajectory Following

  • FVD 1023.46 vs. 1387.75, MEt3R 0.3690 vs. 0.3703, and AUC@30 0.200 are reported for video quality, 3D consistency, and trajectory following.The passages compare FVD with the SOTA baseline, MEt3R with the strongest baseline, and report the highest AUC@30 among competing models.Source: §5.3 · p. 7, §5.4 · p. 7, §5.5 · p. 8

5.3. Generation Quality

  • In the human study, 67.40% of participants preferred Captain Safari over competing methods; qualitative comparisons describe realistic scene dynamics and natural-looking drone videos.The reported quality comparison gives LPIPS scores of 0.512 for Captain Safari and 0.513 for the SOTA baseline.Source: §5.3 · p. 7

5.4. 3D Consistency

  • Captain Safari lowers MEt3R by 0.0013 and raises reconstruction rate by 0.045 versus the strongest baseline, while visualizations show structures remaining stable across viewpoint changes.The human study reports 65.60% preference for Captain Safari on 3D consistency.Source: §5.4 · p. 7

5.5. Trajectory Following

  • Captain Safari achieves the highest AUC@30 (0.200) and AUC@15 (0.068), and the best cosine similarity (0.563); 69.00% of participants favor it for path following.Figure 5 shows its predicted trajectory closely aligned with ground truth, while the ablated variant deviates and Real-CamI2V advances only slightly.Source: §5.5 · p. 8

5.6. Ablation Study

  • Removing memory slightly improves FVD and LPIPS but degrades 3D consistency and trajectory following; memory retrieval is decoupled from DiT denoising so its computational overhead scales constantly.The ablation describes memory as a geometry prior that favors 3D stability over appearance optimization.Source: §5.6 · p. 8

6. Conclusion

The conclusion presents Captain Safari and OpenSafari as a benchmark and system for long-range, 3D-consistent video generation under complex FPV trajectories. It identifies real-time engines and autoregressive generation as future directions.

6. Conclusion

  • Captain Safari uses world memory for long-range, 3D-consistent video generation under complex FPV trajectories, and OpenSafari supplies in-the-wild drone videos with verified camera poses.The conclusion reports improved 3D consistency and trajectory accuracy while maintaining strong visual fidelity.Source: §6 · p. 8
  • Future work will explore real-time world engines and autoregressive generation that continuously builds global memory from generated frames.The authors hope the method and dataset encourage research on persistent world models and long-horizon controllable video generation.Source: §6 · p. 8

7. Additional Ablations on Memory Retrieval

The additional ablation compares pose-aligned cross-attention retrieval with a nearest-pose memory baseline. Under aggressive FPV motion, the single-pose variant performs worse than both Captain Safari and the no-memory DiT baseline.

7. Additional Ablations on Memory Retrieval

  • Nearest-pose retrieval selects the memory feature from the past frame whose camera pose is closest to the query, instead of aggregating pose-aligned tokens with cross-attention.The comparison tests whether a single pose match suffices for the clip’s viewpoint span.Source: §7 · p. 12
  • The nearest-pose variant performs worse across all metrics than Captain Safari and even the no-memory DiT baseline under aggressive FPV motion.The passage attributes this to one pose often being unaligned with the clip’s full viewpoint span, misguiding denoising.Source: §7 · p. 12, Table 3 · p. 12

8. Complexity and Scalability Analysis

Captain Safari decouples memory retrieval from iterative denoising, keeping the denoising cost of longer memory essentially constant. Its fixed-size pose-aligned world tokens limit the repeated attention cost compared with approaches that attend to the full memory.

8. Complexity and Scalability Analysis

  • Full-memory cross-attention scales linearly with memory length, while early concatenation scales quadratically.Captain Safari instead retrieves once and cross-attends to a compact, fixed-size token set during denoising.Source: §8 · p. 12
  • Table 4 shows memory-length scaling affects only one-time retrieval, while denoising overhead remains essentially constant because the retrieved world tokens have fixed size.Retrieval costs O(|Mlocal|), and denoising adds O(ST |wt|) cross-attention; |wt| is controlled by the number of learnable queries M.Source: Table 4 · p. 12, §8 · p. 12

9. Dynamic Scene Generation and Additional Qualitative Results

Captain Safari combines an implicit memory and trajectory formulation with diffusion priors to model dynamic content while maintaining stable 3D environments. The paper also points to supplementary demonstrations of moving elements and challenging outdoor motion.

9. Dynamic Scene Generation and Additional Qualitative Results

  • The generator’s diffusion priors allow Captain Safari to synthesize dynamic content beyond rigid 3D rendering, alongside the geometric prior for stable backgrounds.The paper reports plausible moving vehicles and ocean waves within stable 3D environments.Source: §9 · p. 12
  • Supplementary demonstrations include aggressive 6-DoF motion, strong parallax, and complex outdoor layouts.The additional qualitative results include high-resolution video comparisons.Source: §9 · p. 12

10. Data Usage and Ethics

OpenSafari is distributed as an open-source data curation pipeline rather than as redistributed raw videos. Researchers can use its scripts to retrieve and reproduce the geometry-annotated dataset for non-commercial academic research.

10. Data Usage and Ethics

  • OpenSafari does not redistribute raw videos to comply with platform policies and respect copyright ownership.The release provides downloading, preprocessing, semantic-filtering, and camera-reconstruction scripts instead.Source: §10 · p. 12
  • The provided scripts let researchers fetch original URLs and automatically reproduce the verified, geometry-annotated dataset.The pipeline is open source.Source: §10 · p. 12
  • Dataset reproduction through the pipeline is designated for non-commercial academic research purposes.Source: §10 · p. 12

Related papers

Loading 2511.22815v2…