PaperlayerView exact revision on arXiv

Source-linked AI summary

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy

arXiv:2609.04196v1cs.CV

TL;DR

Existing approaches do not provide a single model that jointly handles physical understanding, spatial simulation, and 3D generation and reconstruction with explicit physical grounding. Puffin-World addresses this gap with native physics, geometry, and appearance states, an Omni-Camera condition, and physics propagation; it reports strong performance across perception, simulation, and world modeling while scaling through Puffin-16M.

  • Problem

    Existing world models focus mainly on appearance, while unified multimodal models generally provide only 2D semantics rather than jointly modeling physical state, geometry, and appearance.

  • Method

    Puffin-World jointly models native physics, geometry, and appearance states using Omni-Camera conditions and propagates absolute physical knowledge across future relative controls.

  • Results

    Puffin-World supports physical-world perception, free-viewpoint simulation, and 3D generation with reconstruction, achieving the best median error and AUC at 5° across four public camera-to-world benchmarks.

  • Takeaways & Limitations

    The unified paradigm supports physically grounded world modeling and closed-loop applications including mimic and self-calibrated world exploration.

  • Takeaways & Limitations

    The paper identifies extending the framework to dynamic scenes, longer horizons, and richer physical states beyond gravity and latitude as future directions.

Abstract

from arXiv · show

We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.

1. Introduction

Puffin-World addresses the need for a unified model that jointly represents physical state, geometry, and appearance for perception, simulation, generation, and reconstruction. Its native world states, Omni-Camera representation, physics propagation, and scaled data support physically consistent multimodal world modeling.

  • Motivation: Existing world models primarily predict appearance, while unified multimodal models generally address only 2D semantics.The paper identifies the absence of a single model unifying physical understanding, geometry, appearance, and multimodal generation.
  • Approach: Puffin-World jointly models physics, geometry, and appearance as native 3D world states without external offline modules.Physics grounds observations in the absolute world, geometry represents 3D structure, and appearance represents visual content.
  • Approach: The Omni-Camera combines gravity-anchored absolute orientation with relative ray-based geometry for single-view and cross-view control.It provides a dense action signal for physics-grounded text-to-image simulation and image-to-3D scene generation.
  • Approach: Physics propagation transfers absolute spatial knowledge across future relative controls, maintaining stable and gravity-consistent appearance during complex motions.This mechanism anchors future frames to the physical state inferred from the reference view.
  • Results: Puffin-World achieves the best median error and AUC at 5° across four public camera-to-world benchmarks.It also supports camera-controllable spatial simulation, joint geometry reconstruction, and closed-loop applications such as mimic and self-calibrated exploration.
  • Data and release: Puffin-16M contains 15 million vision-language-camera triplets and 1 million challenging-motion trajectories.The authors also release the code, models, and datasets, and annotate approximately 44.5 million images across 28 public datasets.

2. Related Work

Related work has separately advanced camera understanding, unified multimodal modeling, and controllable 3D world generation. Puffin-World is positioned as a unified framework that anchors perception, simulation, and reconstruction in absolute physical coordinates while jointly modeling physics, geometry, and appearance.

  • Camera-to-World Understanding: Camera-to-world methods include direct parameter regression and dense geometric representations such as distortion maps, displacement fields, camera rays, perspective fields, and incidence fields.Dense pixel-wise representations provide spatially grounded supervision and are generally more robust than direct global regression.
  • Camera-to-World Understanding: Puffin reframes camera-to-world understanding as language modeling, but its limited model scale and training data constrain performance and scalability.The related-work discussion places Puffin below specialized methods in this setting.
  • Unified Multimodal Models: Unified multimodal models combine visual understanding and generation within one network using discrete or continuous visual tokens or diffusion decoders.The cited approaches integrate modalities architecturally, but the surrounding discussion distinguishes their primarily 2D semantic focus.
  • 3D World Models: Generative world models simulate future observations, while video and multi-view diffusion frameworks increasingly condition generation on camera information.These developments improve visual fidelity and controllability but motivate a broader unified world model.
  • Synthesis: Puffin-World addresses the remaining gap by jointly representing physics, geometry, and appearance, anchoring generation to an absolute physical frame, and integrating perception, simulation, and reconstruction.This positions the model as a unified alternative to separate advances across the three related-work areas.

3. Method

Puffin-World unifies physics, geometry, and appearance modeling with a camera representation that combines absolute physical grounding and relative spatial geometry. Its unified generation process propagates physical orientation across trajectories and jointly models appearance with depth for stable, multi-view 3D world modeling.

  • Native 3D World States: Puffin-World jointly models physics, geometry, and appearance as native 3D world states within one multimodal framework.Physics captures gravity and latitude, geometry represents depth, and appearance represents RGB observations.
  • Unified Camera Representation: The Omni-Camera representation combines gravity-aware absolute orientation with ray-based relative geometry for physical grounding and continuous spatial control.Its absolute component uses pixel-wise up-vectors and latitude angles, while its relative component uses ray origins and directions.
  • Unified Camera Representation: Puffin-World fuses camera conditions directly into diffusion latents through pixel-aligned additive injection, supporting versatile tasks with negligible parameter overhead.The fusion module maps the Omni-Camera representation and role mask into the input latent space before patch embedding.
  • Trajectory Anchoring: Physics propagation combines relative ray motion with gravity-aligned absolute orientation so trajectory views share a coherent world frame while retaining individual motions.Propagated gravity directions render absolute camera representations that are concatenated with relative ray maps.
  • Joint Appearance–Geometry Modeling: Appearance and geometry are generated jointly under shared camera and view-axis conditions, while separate modality losses and a scheduled geometry weight protect visual quality.The geometry weight ramps linearly from 0 to ωmax over the first τ0 iterations, preventing early depth gradients from perturbing shared layers.
  • Multi-Task Synergy: The unified framework supports closed-loop mimic and self-calibrated world exploration that jointly involve perception, reasoning, and generation.Self-calibrated exploration detects and corrects gravity misalignment through predicted actions and imagined target observations.

4. Dataset

Puffin-16M scales aligned vision-language-camera data with diverse resolutions, aspect ratios, and challenging camera trajectories. It also enriches spatial coverage and physical and geometric cues beyond prior datasets.

  • Puffin-16M Overview: Puffin-16M comprises 15 million vision-language-camera triplets with diverse resolutions and aspect ratios, plus 1 million trajectories with challenging rotational camera motions.The subsets are named Puffin-Cam-15M and Puffin-Traj-1M.
  • Puffin-Cam-15M: 15 million perspective images are generated from panoramas using varied camera intrinsics and extrinsics, including roll and pitch sampled from [−45°, 45°] and vertical FoV from [20°, 105°].The virtual pinhole-camera process adaptively determines crop counts by source-panorama resolution.
  • Puffin-Traj-1M: Puffin-Traj-1M addresses limited rotational diversity by providing one million trajectories with broad camera-motion coverage for arbitrary-viewpoint exploration.Existing datasets typically restrict roll to [−5°, 5°] and pitch to [−10°, 10°].
  • Puffin-Traj-1M: Its trajectories support continuous look-down, look-up, clockwise, counterclockwise, and full 360° surrounding-view exploration.Yaw is sampled from [0°, 360°), while roll and pitch are sampled from [−45°, 45°].
  • Dataset Enrichment: The dataset expands spatial and physical coverage while addressing missing absolute orientation, which otherwise leaves gravity direction, horizon location, and camera orientation ambiguous.The authors also enrich public datasets with physical and geometric cues and provide dense geometry where available.

5. Experiments

Experiments evaluate Puffin-World across camera understanding, camera-controllable generation, 3D world modeling, physics propagation, and closed-loop applications. Results show strong camera accuracy, spatial consistency, perceptual fidelity, and multi-task behavior across diverse settings.

  • Camera-to-World Understanding Results: Puffin-World outperforms competing camera-to-world methods on all median-error metrics and most AUC metrics across four public benchmarks.The benchmarks are Stanford2D3D, MegaDepth, TartanAir, and LaMAR; AnyCalib is included as a specialized FoV-estimation reference.
  • Camera-Controllable Generation Results: Puffin-World substantially outperforms general-purpose multimodal generators across camera-controllable generation metrics while preserving visual quality and scene diversity.The evaluation uses 600 text–camera specification pairs spanning varied scenes, viewpoints, poses, fields of view, and aspect ratios.
  • 3D World Modeling Results: Puffin-World handles long trajectories, extreme rotations, and compound motions while preserving semantic content and coherent spatial structure across viewpoints.On the challenging Puffin-Traj-Bench, it achieves the best PSNR and LPIPS, competitive SSIM, and substantially better camera control across roll, pitch, and FoV.
  • Ablation Study: Physics propagation consistently improves visual fidelity and physical grounding, with the largest gains on roll and pitch trajectories coupled to gravity.It stabilizes horizons, improves scene uprightness, and produces camera states that better follow intended motion; yaw shows smaller appearance gains.
  • Applications: Puffin-World supports mimic and self-calibrated world exploration as closed-loop applications combining perception, reasoning, and generation.These capabilities demonstrate the framework’s use of multi-task synergy without auxiliary perception, calibration, or generation modules.

6. Conclusion

The conclusion presents Puffin-World as a unified framework for physically grounded perception, simulation, generation, and reconstruction. It identifies future extensions to broader dynamics, longer horizons, and richer physical representations.

  • 6. Conclusion: Puffin-World jointly models physics, geometry, and appearance to perceive, simulate, generate, and reconstruct the physical world.Its Omni-Camera representation and physics propagation support single-view control, multi-view synthesis, and challenging camera trajectories.
  • 6. Conclusion: Future work includes extending Puffin-World to dynamic scenes, longer horizons, and physical states beyond gravity and latitude.These directions define the current scope boundary identified in the conclusion.

A. Appendix

The appendix supplements the main paper with dataset, annotation, training, geometric-supervision, and qualitative-result details.

  • A. Appendix: The appendix details released and used datasets, camera annotations, additional Puffin-16M physical labels, camera statistics, training procedures, and qualitative results.It also describes dense geometric supervision constructed for real-world datasets.

A.1.1. Annotating Public Datasets

Puffin-World annotates public datasets with absolute camera properties to broaden physical-world supervision and analyze camera diversity. The resulting collection spans many datasets and scene distributions, including distinctive orientation biases.

  • A.1.1. Annotating Public Datasets: Puffin-World annotates 28 public datasets by predicting roll, pitch, and vertical FoV for each image.The annotations address the lack of accurate absolute camera parameters anchored to the real world.
  • A.1.1. Annotating Public Datasets: Camera distributions vary substantially: EgoObjects is downward-biased, DeepFashion has low parameter variation, and TartanAir has the largest rotational diversity.These distributions reflect capture conditions and identify data useful for rarely observed camera orientations.
  • A.1.1. Annotating Public Datasets: The annotated collection covers approximately 44.5 million images across diverse scene distributions.Figure 8 summarizes the public datasets included in the annotation effort.
  • A.1.1. Annotating Public Datasets: Absolute-camera subsets align per-frame predictions with camera trajectories to support consistency analysis and gravity-grounded world modeling.The appendix also reports distributions of roll, pitch, and FoV across existing public 3D or sequential datasets.

A.1.2. Camera Height of Puffin-16M

Puffin-16M augments precise camera annotations with coarse, physically meaningful camera-height labels inherited by rendered perspective views. These annotations support richer spatial reasoning and viewpoint-aware generation.

  • Each original panoramic image receives a camera-height annotation before perspective rendering.The annotation process uses Qwen3-VL-32B.
  • Camera height is categorized into five levels: underwater, low-position, eye-level, high-position, and aerial shots.The labels describe the observer’s vertical position in the scene.
  • The resulting height label is inherited by every perspective view rendered from its panorama.This complements precise camera parameters with a high-level physical cue.
  • The complementary annotations facilitate spatial reasoning, viewpoint-aware generation, and camera-grounded physical world studies.

A.2. Training Details

Puffin-World combines public sequential and 3D datasets with Puffin-16M to supervise native 3D world states across training stages. The training materials also refine sparse depth labels and use camera-height prompts and physics-propagation ablations to support unified modeling.

  • Dataset mixture: Public sequential and 3D datasets provide complementary supervision across diverse environments, trajectory lengths, motion patterns, and depth availability.The mixture progressively supports single-view understanding and generation, cross-view modeling, and 3D world modeling.
  • Dataset mixture: Puffin-16M combines Puffin-Cam-15M language-image-camera triplets with Puffin-Traj-1M rotational trajectories to broaden camera and motion supervision.Puffin-Cam-15M is reused across stages, while Puffin-Traj-1M extends motion diversity beyond conventional capture datasets.
  • Depth supervision: Dense depth labels are created by predicting depth with DA3 and aligning it to available measurements, preserving metric scale while filling missing regions.This addresses sensor holes in ScanNet and sparse multi-view stereo reconstruction in DL3DV.
  • Camera-height supervision: Camera-height captioning uses five labels—underwater, low-position, eye-level, high-position, and aerial—under a strictly horizontal camera-angle assumption.Classification uses object positions relative to the horizon together with perspective, ground coverage, and object-scale cues.
  • Depth supervision: Puffin-World depth predictions have sharper object boundaries and fewer artifacts while maintaining spatial layouts consistent with the input images.The refined dense annotations provide more complete geometric supervision for unified RGB–depth training.
  • Ablation: The physics-propagation ablation compares generation results with and without propagation using error up-vectors and error values.The comparison is visualized per sample in Figure 12.

A.3. More Results

Additional results evaluate Puffin-World across diverse datasets, camera configurations, and world-generation settings. The qualitative studies emphasize generalization, controllable views, motion following, and coherent native 3D world reconstruction.

  • Physical world-state estimation: Puffin-World generalizes absolute camera-parameter estimation across diverse datasets and challenging camera configurations.The estimated parameters are converted into latitude maps and gravity fields for visualization.
  • Free-viewpoint simulation: Puffin-World enables free-viewpoint simulation that follows specified camera orientation and field of view while preserving realistic appearance and coherent layouts.The results cover indoor and outdoor spatial configurations and maintain semantic consistency with the input condition.
  • 3D world modeling: Across rotational controls, long trajectories, compound actions, and text-driven generation, the model follows prescribed camera motions while preserving semantic and spatial coherence.Figure 15 separates roll, pitch, and yaw controls and compares single-pass with recursive paths.
  • 3D world modeling: Additional 3D-world results jointly visualize appearance, geometry, and physics states alongside reconstructed 3D scenes.The native states comprise images, depth, gravity fields, and latitude maps.
Loading 2609.04196v1…