A Perceptual Measure for Deep Single Image Camera Calibration

1. Introduction

• Many single-image calibration approaches depend on vanishing lines, coplanar circles, or repeated textures, so they do not apply when those features are absent.The paper frames camera calibration as a prerequisite for tasks including 3D reconstruction, image metrology, and photographic editing.Source: §1 · p. 1
• The CNN improves on recent focal-length and horizon-estimation results and outperforms previous methods on the perceptual measure.The paper also demonstrates calibration-based image retrieval and compositing, including 3D object insertion.Source: §1 · p. 1
• A deep convolutional network directly estimates focal length, pitch, and roll from a single image, using automatically extracted panorama samples for training.The authors also analyze learned features against those used by predefined feature-based calibration methods.Source: §1 · p. 1
• A large-scale user study asks people to judge the realism of composites made with ground-truth or randomly perturbed camera parameters, informing a perceptual calibration-error measure.The motivation is that exact calibration may not always be necessary for perceptual realism.Source: §1 · p. 1

2. Related Work

• On SUN360 examples, Upright performs well with sharp human-made objects, whereas deep learned methods are more robust in organic scenes.The figure's final column shows failures where the scene is underrepresented in training or vanishing-line cues are scarce.Source: Figure 1 · p. 2
• After-the-fact calibration methods often detect geometric objects or vanishing points, while lighting-based methods can fail when complex scenes require semantic reasoning to reject misleading cues.CNN approaches had also been used to estimate field of view and horizon lines.Source: §2 · p. 2
• Prior perceptual studies examined color sensitivity, photo-manipulation artifacts, street-level rendering distortion, super-resolution, image captioning, and video alignment.These works provide context for studying human sensitivity to camera-calibration errors.Source: §2 · p. 2
• The proposed work jointly estimates field of view and horizon line with a CNN, studies human sensitivity to calibration errors, and compares learned features with vanishing-line cues.The paper situates the approach among methods that estimate field of view or horizon lines from single images.Source: §2 · p. 2

3. Geometric camera model

• Under the pinhole model, a 3D point is projected to pixel coordinates using camera intrinsics and extrinsics; the simplified intrinsics assume square pixels, no skew, and a centered principal point.The projection matrix is diagonal with focal length in pixels on the first two diagonal entries.Source: §3 · pp. 2–3
• The model estimates vertical field of view rather than focal length directly, since focal length has no direct interpretation in the image.Here, h denotes image height.Source: §3 · p. 3
• Rotation is restricted to pitch and roll because an arbitrary image provides no natural reference for yaw; the horizon line represents these angles.The horizon midpoint is its intersection with the image's vertical axis, and roll is measured relative to horizontal.Source: §3 · p. 3
• In the image-units representation, the image top and bottom have coordinates 1 and −1; camera calibration refers to vertical field of view hθ, pitch bp, and roll ψ from the simplified geometric camera model.Source: §3 · p. 3

4.1. Dataset

• The dataset contains 399,728 panorama-derived image and calibration pairs, split into training, validation, and test sets with no panorama shared across splits.Seven rectified images are extracted from each SUN360 panorama using randomly sampled pinhole-camera parameters.Source: §4.1 · p. 3
• The horizon is represented by slope and offset, parameterized by roll and the horizon line’s perpendicular distance from the image center.Source: §4.1 · p. 3

4.2. Architecture

• The model estimates horizon angle, horizon offset, and vertical field of view from a single image using three DenseNet output heads.Each head predicts a discretized probability distribution; the model is trained with the summed Kullback-Leibler divergence.Source: §4.2 · p. 3, §4 · p. 3
• Figures 2 and 3 plot estimation errors by parameter-value bin for pitch and roll on HLW and SUN360, and field of view on SUN360.Figure 3 pairs a box-percentile plot with a cumulative distribution function.Source: Figure 2 · p. 4, Figure 3 · p. 4

• Our method outperforms prior methods in most horizon-line cases, except for large HLW roll errors, and significantly outperforms Upright across SUN360 field-of-view parameters.Source: §5 · p. 4
• Guided backpropagation highlights vanishing-line edges, ignores unhelpful sharp edges, and tends toward sky-land boundaries when no clear vanishing line is present.The authors suggest these focus maps could help geometric methods select appropriate edges.Source: §5 · p. 4

6.1. Dataset generation

• Study images came from 530 panoramas (79 indoor, 451 outdoor), yielding 10,638 realistic-parameter images; paired renders altered camera parameters while matching object location and size.Source: §6.1 · pp. 4–5

6.2. Perceptual evaluation

• The forced-choice study measured preference for ground-truth composites, with 50% denoting confusion; 376 workers supplied 124,740 accepted judgments, averaging 11 annotators per image.Source: §6.2 · p. 5, §6.3 · p. 6, Figure 5 · p. 6

6.3. Study results

• Object and background effects were mostly absent, but the sphere and bunny obscured ground-truth detection except at field-of-view changes of at least 30°; size, scene category and brightness showed no differences.Source: §6.3 · pp. 5–6
• Users missed field-of-view errors from 15° over to 50° under ground truth; pitch tolerance reached 0.2 rescaled units, while roll errors of 12° or more were almost systematically detected.Source: §6.3 · p. 6
• Sensitivity to roll error rises with absolute roll, whereas pitch-error sensitivity shows little relation to horizon position and field-of-view sensitivity stays constant across tested values.Source: §6.3 · p. 6, Figure 6 · p. 7

6.4. CNN evaluation on human perception

• Our method’s third quartile remains below 20% human sensitivity across the parameter range and is systematically lower than every compared method.Source: §6.4 · p. 6, Figure 8 · p. 7

7. Applications

• For retrieval, estimated horizon-line intersections rank images in a random 10,000-image Places2 subset by L2 distance from the query.Source: §7 · p. 8
• The estimated camera parameters support realistic object transfer between images without requiring an object of known height.Source: §7 · p. 8
• For virtual insertion, users select an image insertion point and camera height; automatic insertion assumes a local flat plane aligned with the horizon.Source: §7 · p. 8, Figure 10 · p. 8

8. Discussion

• Humans were not always sensitive to large calibration errors, especially pronounced roll or underestimated field of view, in virtual-object insertion.The study examined sensitivity to pitch, roll, and field of view.Source: §8 · p. 8
• The CNN achieves state-of-the-art performance, learns semantically meaningful vanishing lines, and supports object insertion, image retrieval, and compositing.The learned cues parallel geometrically based auto-calibration techniques.Source: §8 · p. 8
• The authors use the user-study results to define a human-perception-based distance function for comparing the CNN with previous methods.Source: §8 · p. 8
• The horizon-line parameterization cannot represent extreme pitch angles such as looking straight down; proposed extensions include perceptual-loss training and combining CNN robustness with geometric-method accuracy.The authors also suggest using photorealistic objects throughout the study to better model human perception.Source: §8 · p. 8