PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation

Kang Liao, Size Wu, Zhonghua Wu, Linyi Jin, Chao Wang, Yikai Wang, Fei Wang, Wei Li, Chen Change Loy

arXiv:2510.08673v2cs.CV

TL;DR

Camera-centric understanding and generation are usually studied separately, and numerical camera parameters pose a modality gap for large multimodal models. Puffin unifies the tasks with camera-aware reasoning and generation; experiments report that it outperforms specialized models, with instruction tuning extending it to cross-view tasks.

  • Problem

    Camera-centric understanding and generation are commonly treated as separate problems, while numerical camera parameters can be ignored or misinterpreted by LMMs and geometric details can collapse into coarse representations.

    Source: §1 · p. 2

  • Method

    Puffin jointly performs camera-centric understanding and generation, using spatially grounded cues, photographic terminology, and geometric reasoning to treat camera as a language-aligned modality.

    Source: §1 · p. 2

  • Results

    Puffin outperforms specialized models for camera-centric understanding or generation and can be adapted to diverse downstream applications.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Instruction tuning extends Puffin to cross-view spatial imagination, world exploration, and photographic guidance.

    Source: §3.3 · p. 6

    Training at 512 × 512 currently restricts image generation to a single scale, and central cropping and resizing for camera understanding may discard valid content, particularly for images with strongly non-square aspect ratios.

    Source: §5.6 · p. 15

Abstract

from arXiv · show

Camera-centric understanding and generation are two cornerstones of spatial intelligence, yet they are typically studied in isolation. We present Puffin, a unified camera-centric multimodal model that extends spatial awareness along the camera dimension. Puffin integrates language regression and diffusion-based generation to interpret and create scenes from arbitrary viewpoints. To bridge the modality gap between cameras and vision-language, we introduce a novel paradigm that treats camera as language, enabling thinking with camera. This guides the model to align spatially grounded visual cues with photographic terminology while reasoning across geometric context. Puffin is trained on Puffin-4M, a large-scale dataset of 4 million vision-language-camera triplets. We incorporate both global camera parameters and pixel-wise camera maps, yielding flexible and reliable spatial generation. Experiments demonstrate Puffin superior performance over specialized models for camera-centric generation and understanding. With instruction tuning, Puffin generalizes to diverse cross-view tasks such as spatial imagination, world exploration, and photography guidance. We will release the code, models, dataset pipeline, and benchmark to advance multimodal spatial intelligence research.

1 INTRODUCTION

Camera-centric understanding and generation are complementary but have typically been studied separately, while naïvely extending LMMs leaves a gap between numerical camera parameters and language. Puffin unifies the tasks through camera-aware reasoning and a dataset of 4 million vision-language-camera triplets.

1 INTRODUCTION

  • Camera geometry supports spatial structure recovery and navigation, while camera parameters also control spatial content generation from different viewpoints.The paper identifies these as complementary objectives that have typically been studied in isolation.Source: §1 · p. 2
  • Puffin combines autoregressive and diffusion modeling to jointly perform camera-centric understanding and generation, treating cameras as a first-class modality.Its “thinking with camera” approach aligns spatially grounded visual cues with photographic terminology while reasoning over geometric context.Source: §1 · pp. 2–3
  • The paper focuses on single-view calibration and text-to-image controllable generation, with cross-view tasks added through instruction tuning.Examples include spatial imagination and world exploration.Source: §1 · p. 2
  • Puffin outperforms specialized models for camera-centric understanding or generation and adapts to diverse downstream applications.The reported applications include cross-view tasks such as spatial imagination and world exploration.Source: §1 · p. 3
  • Puffin-4M contains 4 million vision-language-camera triplets spanning diverse indoor and outdoor scenes.The dataset includes precise camera parameters, descriptive captions, pixel-wise camera maps, spatial reasoning annotations, and cross-view and aesthetic images.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses visual understanding, image generation, and camera geometry through separate approaches. The paper positions Puffin as a camera-centric framework for joining understanding and controllable generation beyond simplified camera assumptions.

2 RELATED WORK

  • Large multimodal models excel at high-level semantic tasks but remain limited in capturing image intrinsics such as depth and geometry.The passage identifies geometry-aware prior knowledge as important for preserving structural detail beyond semantics.Source: §2 · p. 3
  • Unified multimodal models combine visual understanding and generation, but existing approaches assume simplified cameras such as fixed front views or predefined fields of view.The paper introduces a camera-centric framework for joint camera understanding and controllable generation.Source: §2 · p. 3
  • Camera-geometry methods use intermediate representations such as geometric structures, semantic features, and dense geometry fields, but these can emphasize low- and mid-level patterns over holistic spatial concepts.The paper explores interpreting the camera as language as an alternative to pursuing better representations.Source: §2 · p. 4

3 CAMERA-CENTRIC UNIFIED MULTIMODAL MODEL

Puffin combines camera-centric understanding and generation, representing camera geometry through language-aligned reasoning and camera-conditioned generation. Its instruction tuning extends the framework to cross-view tasks, while staged training aligns its components and adds spatial reasoning.

3 CAMERA-CENTRIC UNIFIED MULTIMODAL MODEL

  • Puffin unifies camera understanding and generation using a geometry-aligned vision encoder, an LLM, and a diffusion model conditioned through a connector.Discrete camera tokens represent numerical parameters, while continuous camera latents from pixel-wise maps provide fine-grained spatial control.Source: §3 · p. 4

3.1 CAMERA UNDERSTANDING

  • Camera understanding is formulated as question answering that generates a concise description or spatial reasoning together with estimated roll, pitch, and field of view.A geometry-aligned encoder distilled from semantic and vision-centric teachers is aligned with an LLM through progressive unfreezing and joint fine-tuning.Source: §3.1 · pp. 4, 6
  • Thinking with camera links spatially grounded cues to photographic terminology across geometric context to support structured camera-parameter prediction.The cues include vertical regularities relevant to pitch and spatial composition relevant to field of view; the method decouples roll, pitch, and field of view across context.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 CAMERA-CONTROLLABLE GENERATION

  • For controllable generation, textual reasoning connects camera parameters to likely visual cues, while pixel-wise camera maps provide local geometric context alongside discrete global camera tokens.A connector maps LLM representations into diffusion conditioning signals, supporting control of spatial layout and viewpoint.Source: §3.2 · p. 6

3.3 INSTRUCTION TUNING

  • With minor modifications, Puffin extends from single-view tasks to spatial imagination, world exploration, and photographic guidance.World exploration conditions on a target-view camera map and an encoded initial view; text descriptions may be dropped to support text-conditioned and text-free generation.Source: §3.3 · p. 6

3.4 TRAINING RECIPE

  • Training progresses from component alignment to joint supervised fine-tuning, then adds spatial-reasoning captions and instruction tuning for cross-view tasks.The stages train understanding and generation, add reasoning that grounds visual cues and translates camera values into photographic terms, and finally tune on spatial imagination, world exploration, and photographic guidance.Source: §3.4 · pp. 6–7
  • Puffin-4M comprises 4 million vision-language-camera triplets across scenarios and camera configurations.The dataset overview marks sample images with different colors for different camera-configuration variants.Source: Figure 4 · p. 8

4 DATASET CONSTRUCTION

Puffin-4M is constructed by turning panoramas into perspective views with camera parameters and pixel-wise maps, then adding scene, spatial-reasoning, and cross-view captions. Its photographic-guidance labels identify camera offsets toward views rated more aesthetically by an LMM.

Panoramic data collection and preprocessing

  • Approximately 200K panoramas from public datasets, online platforms, and Google Street View are corrected for geometric distortions and alignment with gravity.Google Street View data spans 12 cities across Asia, Europe, and North America; some panoramas reach 10K resolution.Source: §4 · p. 8

Perspective image generation

  • Using an ideal pinhole camera model, the method synthesizes perspective crops by uniformly sampling roll and pitch in [−45°, 45°] and vertical FoV in [20°, 105°]; crop count adapts to panorama resolution.Source: §4 · p. 8

Dataset comparison

  • Puffin-4M is larger than prior camera-focused datasets and adds spatial-reasoning captions and cross-view image pairs.Table 2 compares datasets for camera understanding and camera-controllable generation.Source: Table 2 · p. 9

Perspective image generation

  • Each perspective image’s camera parameters are converted into a pixel-wise Perspective Field map annotating every pixel with an up-vector and latitude angle.The associated definitions describe each pixel as corresponding to a light ray originating from a 3D point.Source: §4 · p. 9

Scene and spatial reasoning captions

  • Spatial-reasoning captions map numerical camera parameters to photographic terms, then use those terms to prompt relevant visual concepts from an LMM.Qwen2.5-VL-32B-Instruct generates the final captions; Figure 5’s P2T label denotes parameter-to-term mapping.Source: §4 · p. 9, Figure 5 · p. 9

Cross-view extension

  • Cross-view pairs use a zero-angle initial pose and a randomly sampled target pose, with yaw sampled from [0°, 360°) alongside roll and pitch.The target view is captioned with Qwen2.5-VL-7B-Instruct.Source: §4 · p. 10

Photographic guidance

  • Photographic-guidance labels are pitch and yaw offsets from an initial view to the highest-scoring neighboring view, rated using four aesthetic criteria.The criteria are viewpoint creativity, subject emphasis, compositional balance, and spatial harmonization; Qwen2.5-VL-32B-Instruct assigns scores through voting.Source: §4 · p. 10

Prompt design

  • Scene captions are first generated in three to four sentences, then shortened into vivid, visually rich descriptions for text-to-image generation.The refinement prompt requests one or two sentences capturing essential scene elements and atmosphere.Source: §4 · p. 10

5 EXPERIMENTS

Puffin is evaluated on camera understanding and controllable generation, then examined through architectural, representation, and joint-training ablations. The experiments also demonstrate cross-view applications while identifying limits in image scale and spatial evaluation.

5.1 IMPLEMENTATION DETAILS

  • The implementation uses a shared text tokenizer for serialized camera parameters and reuses the image tokenizer for three-channel camera maps.Images and camera maps are set to 512 × 512; specialized camera-map tokenization is left for future work.Source: §5.1 · p. 10

5.2 EVALUATIONS ON CAMERA UNDERSTANDING

  • Puffin outperforms baselines on MegaDepth and Puffin-Und and achieves comparable results on TartanAir and LaMAR, with robustness in tilted poses and scenes with limited geometric features.Central cropping and resizing for non-square images can discard valid content, particularly at large aspect-ratio deviations; the discussion also identifies pitch and FoV as more challenging to estimate than roll.Source: §5.2 · pp. 11–12, Table 3 · p. 11

5.3 EVALUATIONS ON CAMERA-CONTROLLABLE GENERATION

  • Puffin outperforms existing multimodal models by a large margin across all controllable-generation metrics, while generated camera parameters show a reversed roll-versus-pitch difficulty relative to camera understanding.Prior methods’ generated roll values fail to align with target ground truth; the authors attribute this in part to aesthetic preferences for near-level shots and roll’s effect on perceived gravity.Source: §5.3 · pp. 13–14

5.4 ABLATION STUDIES

  • Pairing an LLM with a geometry-aligned vision encoder and staged joint optimization outperforms direct VLM fine-tuning and a vision-only network on all camera-understanding metrics.The authors report that direct fine-tuning of current VLMs underperforms the vision-only network.Source: §5.4 · p. 14, Table 5 · p. 14
  • Thinking with camera improves camera understanding, especially pitch and FoV prediction, and uses inferred spatial cues to guide controllable generation.For generation, the model uses scene descriptions and target parameters as a semantic planning stage; the figure evaluates thinking mode and camera-map encoding.Source: §5.4 · p. 14, Figure 8 · p. 14
  • Pixel-wise camera maps encode local orientation and spatial-displacement clues for precise viewpoint control, while unified understanding-and-generation training improves understanding and can benefit challenging FoV simulation.Without camera-map conditions, challenging configurations may produce geometric distortions or inverted spatial illusions; joint training adds low-level appearance supervision.Source: §5.4 · p. 15, Figure 9 · p. 15

5.5 APPLICATIONS

  • Puffin supports 3D object insertion through camera prediction and extends via instruction tuning to spatial imagination, world exploration, and photographic guidance.The reported world-exploration views show spatial consistency across viewpoints using 3D reconstructions.Source: §5.5 · p. 15, Figure 10 · p. 16

5.6 LIMITATION AND FUTURE WORK

  • Generation is currently restricted to one scale by the fixed 512 × 512 training resolution, and calibration-based evaluation can be ambiguous for subtle spatial differences.The authors identify multi-scale data and stronger evaluators and benchmarks for geometric consistency as future directions; they also aim to extend cross-view capability and support camera-centric video.Source: §5.6 · pp. 15–16

6 CONCLUSION

Puffin unifies camera-centric understanding and generation by treating camera as language and reasoning over shared camera geometry. The conclusion presents this shared representation as a basis for more holistic spatial reasoning and versatile applications.

6 CONCLUSION

  • Puffin jointly performs camera-centric understanding and generation across arbitrary viewpoints, addressing two tasks commonly explored in isolation.The paper characterizes understanding as decoding world geometry and generation as encoding it into controllable, perceptually consistent visual content.Source: §6 · p. 16
  • Treating camera as language and using thinking with camera are presented as ways to bridge the modality gap in unified models.The conclusion contrasts this approach with prior unified models restricted to oversimplified front-view assumptions.Source: §6 · p. 16
  • The authors argue that a shared representation of camera geometry supports more holistic spatial reasoning and versatile applications.Source: §6 · p. 16

APPENDIX

Table A1 maps numerical ranges of roll, pitch, and FoV to professional photographic terms for alignment with LMM prior knowledge.

APPENDIX

  • The mapping covers roll, pitch, and FoV, linking their numerical ranges to professional photographic terms.The table is described as aligning camera parameters with the prior-knowledge space of LMMs.Source: Table A1 · p. 23
  • The table’s scope is parameter-to-term mapping rather than a reported performance evaluation.Source: Table A1 · p. 23
  • The listed terms provide a photographic-language representation of camera parameters.Source: Table A1 · p. 23

A1.1 CAMERA PARAMETERS TO PHOTOGRAPHIC TERMS

The paper quantizes camera parameters into photographic terms and combines them with scene descriptions to express global spatial arrangements in language.

A1.1 CAMERA PARAMETERS TO PHOTOGRAPHIC TERMS

A1.2 PROMPT DESIGN FOR MULTIMODAL TASKS

The prompts encode camera parameters alongside image descriptions and tailor instructions to understanding, generation, and downstream tasks. Dataset-caption prompts separately target spatial reasoning and photographic aesthetics.

A1.2 PROMPT DESIGN FOR MULTIMODAL TASKS

  • Text-to-image prompts combine an image description with numerical camera parameters, while understanding prompts ask for image description, spatial reasoning, and estimates of roll, pitch, and field of view.Cross-view generation specifies target camera parameters; its caption is randomly omitted during instruction tuning with probability 0.5.Source: A1.2 PROMPT DESIGN FOR MULTIMODAL TASKS · p. 23
  • Cross-view prompts pair an initial-view image with a target description and camera parameters, supporting both text-conditioned and text-free generation.The target parameters are roll, pitch, yaw, and field of view, all in radians.Source: A1.2 PROMPT DESIGN FOR MULTIMODAL TASKS · p. 23
  • Dataset-captioning examples use distinct prompts for reasoning captions and photographic-aesthetic captions, with results produced by LMMs.The figure presents each input image, its captioning prompt template, and the resulting caption.Source: Figure A1 · p. 24
  • Spatial-imagination prompts ask for a target-view description from the initial view and yaw deviation; photographic-guidance prompts request camera estimates and target-view yaw and pitch predictions.The photographic-guidance instruction specifies high photographic aesthetics.Source: A1.2 PROMPT DESIGN FOR MULTIMODAL TASKS · p. 24

A2.1 CAMERA UNDERSTANDING

The appendix reports additional camera-understanding and controllable-generation evaluations, analyzes model and data scaling, and illustrates interactive and downstream capabilities.

A2.1 CAMERA UNDERSTANDING

  • Stanford2D3D results are reported for reference only, because training images rendered from its source panoramas differ from test samples; they are excluded from the main evaluation.Additional camera-understanding visualizations include spatial reasoning cues, horizon lines, AIGC images, and real-world photographs.Source: A2.1 CAMERA UNDERSTANDING · pp. 24–25, Table A2 · p. 25, Figure A4 · p. 28

A2.2 CAMERA-CONTROLLABLE GENERATION

A2.3 ANALYSIS

  • Puffin exceeds specialized models in size but unifies understanding and generation, simplifying deployment, and remains more affordable in parameter count and FLOPs than Bagel and Qwen-Image.Table A3 compares model sizes for specialized understanding and generation models and Puffin.Source: A2.3 ANALYSIS · p. 25, Table A3 · p. 25
  • Puffin is more robust across the data distribution, with consistent gains across intrinsics and extrinsics that remain stable under challenging camera configurations.The appendix visualizes error bars for representative baselines and Puffin in Figure A2(a).Source: A2.3 ANALYSIS · p. 25, Figure A2 · p. 26
  • Retrained GeoCalib on Puffin-4M slightly underperforms the original model trained on 40K images; the authors attribute this to its roughly 29M-parameter capacity underfitting broad data distributions.They contrast this lightweight CNN-like architecture with Puffin’s high-capacity LLM backbone, which they say needs sufficiently large and diverse data.Source: A2.3 ANALYSIS · pp. 25–26, Table A4 · p. 26
  • Puffin’s performance improves consistently as Puffin-4M scales; roll benefits earlier, while pitch and field of view benefit more from larger-scale data.The authors suggest further data scaling could bring additional gains, especially for pitch and field of view.Source: A2.3 ANALYSIS · pp. 26–27, Figure A2 · p. 26
  • Puffin supports coherent multi-turn generation-to-understanding and understanding-to-generation dialogues, using prior reasoning for cross-view generation and its generated images for camera estimates.The appendix reports these interleaved interactions without task-specific switching or separate models.Source: A2.3 ANALYSIS · p. 27, Figure A3 · p. 27

A2.4 DOWNSTREAM APPLICATIONS

Loading 2510.08673v2…