Unified Camera Positional Encoding for Controlled Video Generation

Motivation

• UCPE addresses limited generalization from pinhole assumptions by encoding complete camera geometry across heterogeneous intrinsics, distortions, and viewpoints.Existing absolute and relative methods either depend on world-frame configurations or remain restricted to pinhole projection.Source: §1 · pp. 2–3

Relative Ray Encoding

• Relative Ray Encoding represents each token as a viewing ray in local coordinates, enabling attention to reason directly in ray space across camera lenses.This reformulates relative-camera encoding into geometry-consistent relative-ray encoding.Source: §1 · p. 2

Absolute Orientation Encoding

• Absolute Orientation Encoding resolves the ambiguous initial rotation frame by explicitly controlling pitch and roll relative to a gravity-aligned up direction.This enables precise and repeatable control over the initial camera orientation in camera-controlled text-to-video generation.Source: §1 · p. 2

Unified Camera Positional Encoding

• UCPE injects 6-DoF poses, intrinsics, and lens distortions into Transformer attention through a lightweight spatial adapter using less than 1% additional trainable parameters.The framework is designed to integrate with pretrained video Diffusion Transformers while preserving their visual fidelity.Source: §1 · p. 3

Evaluation

• Experiments on a large dataset spanning camera motions, intrinsics, and distortion profiles show improved camera controllability and visual fidelity across varied camera configurations.The dataset provides a benchmark for systematic training and evaluation of camera-controllable generation.Source: §1 · p. 3

Video Diffusion Models

• Pretrained video diffusion models achieve strong realism and temporal coherence, yet remain largely agnostic to the camera geometry forming visual observations.Existing conditioning commonly guides motion and appearance through text, images, depth, edges, or optical flow.Source: §2 · p. 3

Camera-Controlled Generation

• 3D-representation approaches improve viewpoint control and consistency but depend on estimated depth, point clouds, meshes, or Gaussians that can limit motion diversity and degrade under large movements.Their performance also depends on the quality of reconstructed scene representations.Source: §2 · p. 3
• Direct camera-pose conditioning supports camera-aware video generation but typically lacks absolute pitch and roll control and overlooks intrinsics and lens distortion.These omissions hinder generalization across diverse camera configurations and projection models.Source: §2 · p. 3

Camera Encoding

• Absolute ray-map and Pl¨ucker encodings incorporate camera properties but depend on a predefined world frame, limiting cross-scene generalization.Relative encodings remove the fixed-reference requirement by modeling pairwise geometric transformations within attention.Source: §2 · pp. 3–4
• UCPE extends relative camera encoding beyond perspective cameras by jointly representing poses, intrinsics, and distortion within an attention-compatible geometry-consistent formulation.This enables viewpoint control across different projection types without relying only on extrinsic parameters.Source: §2 · pp. 3–4

3.1. Preliminaries

• Camera ray mapping converts pixels into 3D ray origins and directions, supporting diverse projection models including central and non-central cameras.The formulation can represent lens parameters and distortion rather than assuming a single pinhole projection.Source: §3.1 · p. 4
• Existing absolute and relative camera encodings remain frame-sensitive or assume image-wide linear projection, limiting generalization to distorted and wide-field cameras.Relative methods improve coordinate invariance, but camera-level transforms ignore spatially varying intra-image projection geometry.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.2. Unified Camera Positional Encoding

• Relative Ray Encoding assigns each token a local ray coordinate system and transforms world coordinates into ray space for geometry-consistent attention across heterogeneous lenses.Each token uses its ray origin and direction to construct a local ray-to-world transform, whose inverse operates as the attention-level geometric operator.Source: §3.2 · p. 5
• Lat-Up augments relative ray geometry with gravity-aligned latitude and up-vector cues, enabling explicit control of absolute pitch and roll.The encoding supplies global orientation context while retaining distortion awareness for wide-angle lenses.Source: §3.2 · p. 5

3.3. Spatial Attention Adapter for UCPE Injection

• A parallel, compressed spatial attention adapter preserves pretrained self-attention and fuses camera-aware features through a zero-initialized projection, requiring minimal added parameters.The adapter reduces token dimension and attention heads, while zero initialization leaves the pretrained model unchanged at initialization.Source: Figure 3 · p. 5, §3.3 · p. 6
• UCPE combines ray-space geometry with RoPE image-space positions through a block-diagonal attention operator, optionally adding Lat-Up features as token bias.The hybrid operator allocates feature subspaces to ray transformations and RoPE, while Lat-Up provides absolute orientation when enabled.Source: §3.3 · pp. 5–6

4.2. Comparison with Previous Methods

• On the synthesized dataset, UCPE follows target trajectories and lens distortions more faithfully than Wan CameraCtrl and ReCamMaster.Wan CameraCtrl deviates in motion, whereas ReCamMaster fails to reproduce the intended distortion.Source: Figure 4 · p. 6, §4.2 · p. 7
• On RealEstate10K, UCPE produces sharper, more detailed videos with better camera-motion adherence, while competing methods show artifacts, framing problems, reduced motion, or distortion.The comparison includes CameraCtrl, AC3D, Wan CameraCtrl, and ReCamMaster.Source: Figure 5 · p. 7, §4.2 · p. 8
• UCPE outperforms all baselines across lens, orientation, and pose control while maintaining strong video quality with 90% fewer parameters than ReCamMaster.The comparison covers both absolute-orientation settings on the synthesized dataset.Source: §4.2 · p. 7, Table 1 · p. 8
• Without RealEstate10K fine-tuning, UCPE achieves the lowest rotation, translation, and motion errors and higher Q-Align than CameraCtrl and AC3D.It also outperforms all baselines in relative camera pose control on unseen trajectories and prompts.Source: §4.2 · p. 7, Table 2 · p. 8

4.3. Ablation Study

• A 1/8 token-projection compression ratio provides the best trade-off between adapter efficiency and performance, while Pre-Attn and Post-Attn variants degrade control and video quality.Replacing relative ray encoding with PRoPE or GTA also weakens lens, relative-pose, and video-quality performance because they cannot model distortion variation across tokens.Source: §4.3 · p. 8

• UCPE jointly models poses, intrinsics, and lens distortions through relative ray encoding and absolute orientation cues.It is integrated into video Diffusion Transformers through a lightweight spatial attention adapter.Source: §5 · p. 8
• Experiments on a camera-diverse dataset show gains in lens controllability, orientation accuracy, and pose fidelity with under 1% additional parameters.The paper presents UCPE as a potential general encoding for multi-view, video, and 3D tasks.Source: §5 · p. 8

• The supplementary material covers dataset construction, absolute orientation encoding, evaluation metrics, implementation details, robustness, limitations, future work, and a demo video.Source: Supplementary Material · p. 12

A. Dataset Construction

• The pipeline starts from 360° panoramas and synthesizes virtual-camera videos with varied intrinsics, distortions, and rotations.Its three stages curate panoramas, transfer realistic camera motions, and generate camera-diverse videos.Source: A. Dataset Construction · p. 12

A.1. Coordinate Conventions

• The supplementary coordinate convention uses x right, y downward, and z forward, with world-up defined consistently for projections, ray mappings, and Up-map construction.Source: §A.1 · p. 12

A.2. 360° Panoramic Video Curation

• Gravity-aligned 360° footage supplies a consistent up vector, enabling virtual cameras to inherit absolute pitch and yaw; quality filtering removes cuts, artifacts, watermarks, and unstable clips.The curation process yields roughly 400k temporally consistent clips and approximately 300k clips with valid pose estimates before further filtering.Source: §A.2 · pp. 12–13

A.3. Emulating Realistic Camera Rotations

• Realistic rotations are transferred from perspective videos by matching translation trajectories, aligning candidates to gravity-aware panoramas, and filtering unstable motion.CameraBench supplies motion-diverse trajectories, while ViPE, GeoCalib, and Umeyama fitting support extraction and alignment.Source: §A.3 · p. 13
• Diversity constraints limit repeated panoramas and trajectories while balancing semantic categories and motion magnitudes, producing 12k panorama–rotation-sequence pairs.The constraints reduce long-tail distributions in point-of-interest categories and motion, although they do not guarantee perfect uniformity.Source: §A.3 · p. 13

Camera Motion Augmentation

• 48k unique camera motions arise from consistent yaw, yaw-pitch, and smooth yaw-pitch-roll augmentations applied to transferred trajectories.The augmented rotations use specified perturbation ranges and define a virtual camera rotation for each panoramic frame.Source: §A.4 · p. 16

Unified Camera Model

• The Unified Camera Model represents cameras from pinhole to extreme fisheye using one distortion parameter ξ, with ξ=0 reducing to pinhole projection.Larger ξ values introduce stronger ray bending for wide-angle and fisheye lenses.Source: §A.4 · p. 16

Panoramic-to-UCM Projection

• Each UCM pixel is mapped to a central-camera ray using FoV-parameterized intrinsics and distortion, then projected through augmented rotations onto the panorama.The sampled panorama coordinates produce UCM-rendered frames consistent with the selected intrinsics, distortion, and virtual pose.Source: §A.4 · pp. 16–17

Camera Sampling

• Camera sampling spans pinhole, wide-angle, fisheye, and extreme-fisheye categories across broad xFoV and ξ intervals.The sampled ranges cover xFoV from 90° to 200° and distortion levels from 0.0 to 2.3.Source: §A.4 · p. 16

Virtual Camera Pose Composition

• Virtual camera poses compose the original equirectangular camera-to-world pose with augmented rotation while preserving the original trajectory translation.The resulting pose is paired with sampled xFoV and ξ for camera-aware training.Source: §A.4 · p. 17

Dataset Output

• The pipeline generates approximately 48k 81-frame videos at 480×832 resolution and 16 fps, with 272 clips reserved for evaluation.Videos are captioned using a detailed-description prompt, and the representation remains compatible with camera models beyond UCM.Source: §A.4 · p. 17

Camera Motion Direction Distribution

• The synthetic dataset covers a more balanced range of translation directions, whereas RealEstate10K is dominated by forward translations.The comparison uses first-to-last-frame translation directions after matching clips to 81 frames.Source: §A.5 · p. 17

Field of View Distribution

• The dataset spans a 60°–110° FoV range, while RealEstate10K is concentrated near 60°.The synthetic data also includes broad UCM distortion levels and multiple lens types, unlike the primarily distortion-free pinhole cameras in RealEstate10K.Source: §A.5 · p. 18

Ray Construction

• Each token’s pixel center is mapped through FoV-parameterized UCM intrinsics to a unit camera-frame ray, then rotated into the world frame.This establishes the ray used to construct the orientation cue.Source: B. Details of Absolute Orientation… · p. 18

World-Up Perturbation

• The Up-map perturbation uses the tangent axis d_t × u^wld to rotate each viewing ray infinitesimally toward the world-up direction.The normalized cross product supplies the Rodrigues rotation axis.Source: B. Details of Absolute Orientation… · p. 18

Projected Displacement

• The perturbed ray is transformed back to the camera frame and projected with the UCM operator to obtain its displaced image position.The original and rotated pixel coordinates are compared after projection.Source: B. Details of Absolute Orientation… · p. 18

Up Map Definition

• The Up map normalizes the displacement vector [Δu_t, Δv_t] by its Euclidean magnitude at each token.This yields a two-dimensional direction encoding for the local image-plane response.Source: B. Details of Absolute Orientation… · p. 18

Encoding Properties

• The resulting Up-map pattern varies with absolute pitch and roll, scene structures, and lens-specific projection warping.It therefore combines orientation, semantic, and distortion-sensitive visual cues.Source: B. Details of Absolute Orientation… · p. 18

Relative Camera Pose Control

• Generated frames are rectified to pinhole images before trajectory estimation because pose estimators are not robust to strong distortion and wide-FoV warping.Rectification uses the ground-truth UCM ray-pixel operators and caps effective horizontal FoV at 100°.Source: C. Evaluation Metrics · pp. 18–19
• Relative pose metrics compare ground-truth and estimated trajectories after normalizing both camera-to-world poses by their respective initial poses.This removes global alignment and evaluates relative motion.Source: C. Evaluation Metrics · p. 19

Relative Camera Pose Metrics

• Rotation Error, Translation Error, and Camera Motion Consistency quantify angular deviation, translation differences, and flattened relative-pose differences.The metrics evaluate how closely generated trajectories follow target camera motion.Source: C. Evaluation Metrics · p. 19

Evaluation Protocol

• All methods are evaluated at the same 16 uniformly sampled timestamps to avoid bias from differing video lengths and frame rates.The compared outputs include methods producing 16, 48, and 81 frames at different frame rates.Source: C. Evaluation Metrics · p. 19

Absolute Orientation and Lens Control

• GeoCalib estimates per-frame pitch, roll, vertical FoV, and radial distortion coefficients for absolute orientation and lens-control evaluation.Errors are computed against parameters estimated from the original UCM frames.Source: C. Evaluation Metrics · p. 19
• Pitch, roll, FoV, and distortion errors jointly measure controllability over absolute camera orientation, lens intrinsics, and distortion.The evaluation uses absolute parameter differences for these quantities.Source: C. Evaluation Metrics · p. 19

Our Method

• Wan2.1-T2V-1.3B is fine-tuned on a synthesized dataset with original weights frozen while only proposed attention adapters are trained.Training uses AdamW for 10k steps with batch size 8 on eight NVIDIA A800 GPUs, completing in roughly one day.Source: D. Implementation Details · p. 19

ReCamMaster

• ReCamMaster injects normalized field-of-view, distortion, and 12-dimensional pose parameters into every Transformer block after broadcasting them across spatial tokens.The adapted implementation downsamples per-frame parameters to match the token sequence and projects them into token dimension before self-attention.Source: D. Implementation Details · p. 19, Figure D.1 · p. 20

Wan CameraCtrl

• Wan CameraCtrl converts Plücker-encoded rays into token-shaped features through a convolutional adapter, generalized here from pinhole to UCM cameras.The convolutional and residual layers are zero-initialized, and the diffusion Transformer and adapter are fine-tuned under the official training recipe.Source: D. Implementation Details · p. 20

UCPE Spatial Adapters Ablation

• UCPE ablations compare spatial attention adapters placed before, after, or parallel to the original self-attention, with all variants sharing a 1/8 compression ratio.The parallel branch is the main configuration, and the variants are trained under the same setting.Source: D. Implementation Details · p. 20, Figure D.1 · p. 20

Inference Latency

• UCPE generates 81 frames at 480×832 in 184 seconds, comparable to ReCamMaster at 179 seconds and Wan CameraCtrl at 177 seconds on an NVIDIA A800.The comparison uses the same settings and reports minimal computational overhead relative to the base model and other camera-conditioned adapters.Source: D. Implementation Details · p. 20, Table D.1 · p. 20

E. Robustness, Generalization, Limitations, and Future Works

• UCPE successfully enforces a requested fisheye geometry when the text prompt specifies a contradictory flat telephoto view.The tested fisheye uses xFoV = 180° and distortion ξ = 2.0, showing that camera conditioning can override conflicting textual appearance cues.Source: E. Robustness, Generalization,… · p. 20, Figure E.1 · p. 21
• UCPE generalizes to the unseen Brown-Conrady camera model by substituting its ray-mapping function at inference without additional training.The model synthesizes realistic barrel distortion for the tested OpenCV lens-calibration setting.Source: E. Robustness, Generalization,… · p. 21, Figure E.2 · p. 21
• UCPE fails to synthesize full 360° panoramas correctly with equirectangular projection, an extreme setting absent from training.The authors expect targeted fine-tuning on panoramic datasets to improve this case.Source: E. Robustness, Generalization,… · p. 21, Figure E.3 · p. 21
• UCPE currently models pose, intrinsics, and distortion but not richer camera attributes such as zoom, focus, or depth-of-field, and relies on camera poses during training.Future work includes adding these controls and reducing reliance on accurate pose supervision through geometric losses.Source: E. Robustness, Generalization,… · p. 21
• UCPE also extends to constrained Image-to-Video tasks, where rigid input-image pixels anchor lens and initial orientation so Relative Ray Encoding can control subsequent camera motion without Absolute Orientation Encoding.The authors demonstrate feasibility by fine-tuning Wan2.1-I2V-14B and identify Video-to-Video exploration as future work.Source: E. Robustness, Generalization,… · p. 21, Figure E.4 · p. 21

F. Supplementary Video

• The supplementary video demonstrates UCPE components, intrinsic, distortion, extrinsic, and orientation control, plus applications and comparisons on synthetic and RealEstate10K data.It includes examples from the large-scale synthetic dataset and visual comparisons with baseline methods.Source: F. Supplementary Video · p. 21