PaperlayerView exact revision on arXiv

Source-linked AI summary

PreciseCam: Precise Camera Control for Text-to-Image Generation

Edurne Bernal-Berdun, Ana Serrano, Belen Masia, Matheus Gadelha, Yannick Hold-Geoffroy, Xin Sun, Diego Gutierrez

arXiv:2501.12910v1cs.CVcs.AIcs.LG

TL;DR

Text-to-image models offer limited precise camera control, despite camera choices shaping how images convey meaning. PreciseCam conditions generation on four camera parameters through a per-pixel representation, and experiments report precise, consistent control that exceeds prompt engineering approaches.

  • Problem

    Text-to-image models make precise camera parameters difficult to control, limiting the ability to use camera language to convey ideas or emotions.

    Source: §1 · pp. 1–2

  • Method

    PreciseCam guides text-to-image generation with four user-provided parameters—roll, pitch, vertical FoV, and distortion—encoded in a per-pixel PF-US representation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Experiments demonstrate precise, consistent camera control that far exceeds current approaches, including prompt engineering.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The paper shows proof-of-concept applications in background generation for object compositing and video generation.

    Source: §5 · p. 8

    PreciseCam may struggle with extreme roll rotations, and conflicts between prompt descriptions and specified camera views may produce incoherent outputs.

    Source: §5 · p. 8

Abstract

from arXiv · show

Images as an artistic medium often rely on specific camera angles and lens distortions to convey ideas or emotions; however, such precise control is missing in current text-to-image models. We propose an efficient and general solution that allows precise control over the camera when generating both photographic and artistic images. Unlike prior methods that rely on predefined shots, we rely solely on four simple extrinsic and intrinsic camera parameters, removing the need for pre-existing geometry, reference 3D objects, and multi-view data. We also present a novel dataset with more than 57,000 images, along with their text prompts and ground-truth camera parameters. Our evaluation shows precise camera control in text-to-image generation, surpassing traditional prompt engineering approaches. Our data, model, and code are publicly available at https://graphics.unizar.es/projects/PreciseCam2024.

1. Introduction

Precise camera control matters because camera choices shape how an image communicates, yet text-to-image models offer limited control through prompts. PreciseCam uses four camera parameters and a per-pixel representation to guide generation, and introduces a dataset for training and evaluation.

1. Introduction

  • Text-to-image models often produce flat camera angles, limiting artists’ and designers’ ability to use images to convey emotion, sensations, or messages.Source: §1 · p. 2
  • Camera parameters are difficult to specify through prompts, leaving trial-and-error prompt engineering as a coarse, expertise-demanding control method.Source: §1 · p. 2
  • PreciseCam uses roll, pitch, vertical FoV, and distortion to form a per-pixel PF-US representation that guides image generation from a user prompt.The representation describes each pixel’s up-vector and distance to the horizon, while image content remains determined by the text prompt.Source: §1 · p. 2
  • The dataset contains 57,380 single-view RGB images with ground-truth camera parameters and prompt descriptions, spanning diverse scenes and camera parameters.Source: §1 · p. 2
  • The method is primarily designed for images but also demonstrates limited absolute camera control for video and can set a precise initial camera position.The paper also states that its code, data, and model are publicly available.Source: §1 · p. 2

2. Related Work

Related work includes increasingly controllable image generation, but precise camera control remains limited. Existing approaches use prompts, 3D representations, or spatial guidance, while PreciseCam incorporates fine-grained camera controls into diffusion adapters.

2. Related Work

  • Earlier conditional-generation methods expanded control through color, hand-crafted terms, and localized prompts, but did not support granular structural guidance such as edges or depth maps.Source: §2 · p. 2
  • Prior camera-control methods include a NeRF-based approach requiring multi-view object images and Continuous 3D Words, which supports some camera control but can struggle with complex prompts and styles.The cited methods are Kumari et al. [27] and Cheng et al. [11], respectively.Source: §2 · p. 3
  • Depth-, edge-, normal-, and segmentation-guided methods do not directly control the camera; PreciseCam adds fine-grained camera controls to adapters such as ControlNet [52].Source: §2 · p. 3
  • Relative-camera video methods adjust movement from the first frame but lack control over the initial camera position, which PreciseCam can provide.Source: §2 · p. 3

3. Our Approach

Precise camera control is specified with four user-selected parameters, represented as per-pixel perspective-field maps and learned through ControlNet conditioning. Training data pairs 360° image crops with camera parameters and generated prompts.

3. Our Approach

  • The framework combines a text prompt with four camera parameters, using diffusion generation and a ControlNet-based module trained on images with ground-truth camera views.Users specify roll, pitch, vertical FoV, and distortion ξ.Source: §3 · p. 3

3.1. Precise Camera View Representation

  • PF-US represents each pixel with an up-vector and latitude, encoding camera effects from roll, pitch, vertical FoV, and distortion without a heavy 3D representation.The Unified Spherical model supports camera models beyond pinhole, including wide-angle and fisheye views; yaw is excluded because an image horizon does not define left or right.Source: §3.1 · pp. 3–4

3.2. Learning Camera View Control

  • ControlNet learns PF-US conditioning while SDXL stays frozen; at inference, injecting only its middle-block output improves generation consistency without harming condition adherence.The module duplicates SDXL encoder and middle-block layers and is initialized with the same weights.Source: §3.2 · p. 4

3.3. Dataset Generation

  • Existing PF data is limited to mostly urban outdoor scenes and a narrow camera-parameter range, while estimates from an existing dataset lack training-suitable accuracy.In particular, the data has little distortion and avoids large vertical FoVs.Source: §3.3 · p. 4
  • The authors create ground-truth PF-US data by sampling camera parameters and cropping corresponding regions from 360° images.The stated sampling ranges are roll and pitch from −90° to 90°, vertical FoV from 15° to 140°, and distortion ξ from 0 to 1.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
  • Six 360° image datasets supply diverse indoor and outdoor scenes, including natural and urban settings.The sources include 360-SOD, CVRG-Pano, F-360iSOD, Poly Haven HDRIs, Sitzmann et al., and 360cities.Source: §3.3 · p. 5
  • BLIP-2 generates prompts for the resulting images, yielding a dataset of 57,380 RGB images with prompts and ground-truth PF-US parameters.The authors state that occasional inaccurate prompts do not affect training because only the conditioning module, not the text-to-image model, is trained.Source: §3.3 · p. 5

4. Results

PreciseCam follows requested extrinsic and intrinsic camera variations and controls views in realistic and artistic images while maintaining prompt adherence comparable to SDXL. The analysis and applications show consistent conditioning, a residual-injection trade-off, and uses for perspective-matched backgrounds and controlled video.

4. Results

  • Generated images follow variations in roll, pitch, vertical field of view, and distortion while the other camera parameters remain fixed.Larger vFoV increases latitude values in PF-US maps, whereas ξ mainly affects the up vectors.Source: §4 · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6
  • PreciseCam controls camera views in realistic and artistic styles, while achieving prompt-adherence scores comparable to prompt-engineered SDXL across 2,940 images per method.SDXL struggles with complex camera prompts, and Firefly’s preset tags do not cover all views or styles.Source: §4 · p. 6, Figure 7 · p. 7

4.1. Model analysis

  • Camera conditioning remains consistent when the input noise varies while prompts and PF-US maps stay fixed.The paper reports this stability test with examples in the supplementary material.Source: §4.1 · p. 7
  • Middle-block residuals preserve camera adherence and SDXL image quality, whereas adding residuals at other levels can introduce distortions and harm semantic integrity.The model therefore uses only middle-level residuals at inference as a trade-off between camera control and image quality.Source: §4.1 · p. 7, Figure 9 · p. 8

4.2. Applications

  • PreciseCam can generate backgrounds aligned with an object’s perspective, helping the object appear naturally embedded in a coherent scene.The comparison shows SDXL lacking perspective consistency while PreciseCam’s generated background aligns with the object.Source: §4.2 · pp. 7–8, Figure 10 · p. 8
  • Video camera views can be specified frame by frame, or a controlled initial view can anchor a sequence guided by relative camera-control methods.A custom-trained adapter applies the four camera parameters to Stable Video Diffusion for framewise control.Source: §4.2 · p. 8, Figure 11 · p. 8
  • PreciseCam can combine camera-view control with depth, Canny-edge, and pose ControlNets to control subject positioning and scene structure simultaneously.The paper presents this as extended control with multiple ControlNets.Source: §4.2 · p. 8

5. Conclusion

PreciseCam provides precise, consistent camera control in text-to-image generation, including for paintings and artistic images. The authors also show proof-of-concept applications beyond creative expression, while noting limitations with extreme roll and conflicts between prompts and camera views.

5. Conclusion

  • The experiments show precise, consistent camera control that far exceeds current approaches, including prompt engineering, and works with paintings despite their absence from the dataset.The authors also show proof-of-concept background generation for object compositing and video generation.Source: §5 · p. 8
  • Extreme roll may leave objects or people vertically oriented, while conflicts between the prompt and specified camera view may produce semantically incoherent outputs.The figure illustrates the vertical-orientation issue with a man on a boat.Source: §5 · p. 8, Figure 12 · p. 9

A. Training Details

The framework is trained as a ControlNet initialized from SDXL weights, using encoded PF-US camera maps and text prompts. Training uses 70,000 steps on eight GPUs.

A. Training Details

  • PF-US maps encode up-vector coordinates in the R and G channels and latitude in the B channel, and serve as RGB inputs for ControlNet training.The up-vector coordinates are scaled from [-1, 1] to [0, 255], and latitude from [-90, 90] to [0, 255].Source: A. Training Details · p. 10
  • ControlNet is initialized with SDXL weights and trained with Adam at a 10^-6 learning rate, total batch size 32, 1024x1024 resolution, and 16-bit precision.The Adam settings are β1 = 0.9, β2 = 0.999, and weight decay 10^-2.Source: A. Training Details · p. 10
  • Training runs for 70,000 steps, replaces half of text prompts with empty strings, and takes four days on eight NVIDIA RTX A100 GPUs.Training uses the Accelerate framework.Source: A. Training Details · p. 10

B. Dataset Details

The dataset is built from cropped 360° images to provide diverse content and ground-truth camera parameters across a wide range of views. It contains 57,380 images paired with descriptive prompts and PF-US conditions.

B. Dataset Details

  • Existing PF datasets lack broad content and camera-parameter coverage, while PF estimators are too imprecise for training and may omit ξ distortion.Estimator noise could compromise training, as illustrated by incorrect PF map estimates.Source: B. Dataset Details · p. 10, Figure 13 · p. 10
  • The dataset instead uses 360° images to crop patches with ground-truth PF-US maps, enabling sampling across camera parameters including ξ distortion.Six source datasets provide indoor and outdoor scenes across natural and urban settings.Source: B. Dataset Details · p. 10
  • Each 360° image yields 24 patches: four crops from each of six regions, varying camera settings while reusing the same image content.The sampling varies yaw, pitch, vFoV, ξ, and roll, including small and large vFoV and low and high ξ settings.Source: B. Dataset Details · pp. 10–11
  • 57,380 RGB images have ground-truth PF-US conditions spanning roll and pitch from −90° to 90°, vFoV from 15° to 140°, and ξ from 0 to 1.BLIP-2 generates a descriptive prompt for each image.Source: B. Dataset Details · p. 11

C. Additional Results

PreciseCam retains prompt adherence comparable to baseline SDXL scores while providing camera control. The supplementary results show outputs for varying camera parameters, and the prompt-engineered baseline uses verbal instructions for roll, pitch, and vFoV but cannot reproduce distortion effects.

C. Additional Results

  • Additional results show generated images grouped by prompt, with one camera parameter varied per row while the other settings remain fixed.The first row in each tab displays the corresponding PF-US camera settings.Source: C. Additional Results · p. 11

D. Prompt Engineering for Baseline SDXL

  • PreciseCam achieves comparable CLIP and BLIP scores to baseline SDXL while maintaining prompt adherence across 2,940 generated images.The baseline used prompt engineering to encourage the desired camera views.Source: D. Prompt Engineering for Baseline SDXL · p. 11
  • The SDXL baseline prompt scheme specifies roll, pitch, and vFoV with textual shot descriptions, but cannot replicate distortion effects.The scheme maps roll sign to left or right Dutch angles, pitch sign to high or low angles, and vFoV ranges to shot descriptions.Source: D. Prompt Engineering for Baseline SDXL · p. 11

E. Consistent Camera for Input Variations

PreciseCam maintains the specified camera view across changes in input noise and prompt, while allowing generated content to vary with the prompt.

E. Consistent Camera for Input Variations

F. Compatibility with Multiple ControlNets

PreciseCam works with pose and depth ControlNets, adding camera-view control while preserving pose control and supporting coherent background perspective.

F. Compatibility with Multiple ControlNets

G. Video Demo

The supplementary video demonstrates slider-based camera adjustment, preview, and image generation; accompanying examples illustrate camera consistency and compatibility with other controls.

G. Video Demo

  • The supplementary video lets users adjust camera parameters with sliders, preview the desired view, and generate an image from a prompt.Source: G. Video Demo · p. 11
  • Different input noises yield different images while the specified camera parameters, represented as PF-US maps, remain in use.Source: Figure 14 · p. 12
  • Prompt variations produce different images while maintaining the camera view represented by the PF-US map.Source: Figure 15 · p. 12
  • PreciseCam combines camera control with pose or depth ControlNets, including perspective-consistent backgrounds in the depth example.The depth example shows a changed perspective of the red house when camera control is included.Source: Figure 16 · p. 12

Related papers

Loading 2501.12910v1…