Generative Cinematographer: Composing Camera and Object Motion in 3D

1 Introduction

• 2D drags can correspond to several 3D motions or mix object motion with viewpoint change, and reliable 3D motion annotations are scarce.Synthetic data provide annotations but differ visually from real footage, while per-pixel conditioning leaves artists working in image space.Source: §1 · pp. 1, 3
• Figure 1 contrasts fixed-camera object motion with combined camera and object motion across examples including a rider, plant, turtle, and jumping girl and dog.Orange marks camera paths and poses; other colors mark foreground trajectories.Source: Figure 1 · p. 2
• GenCine lifts a single image into a shared 3D scene where artists specify camera paths and move selected foreground regions with local motion handles.Multiple handles control different subject parts independently, approximating non-rigid motion without a skeleton, simulator, or category-specific prior.Source: §1 · pp. 3–4
• For real training videos, estimated geometry and camera motion lift tracked foreground points into 3D, while synthetic videos provide ground-truth geometry and trajectories.Each clip pairs its first frame as the reference image with the video and controls describing its observed motion.Source: §1 · p. 3
• Its guidance maps project background and foreground points into each camera view, preserve handle identity with fixed colors, and encode their world-space positions.Foreground and background XYZ maps share a world frame and scale, distinguishing viewpoint changes from changes in an object's world position.Source: §1 · p. 3
• A lightweight side branch and LoRA adapters train the diffusion model to generate paired video from the first frame, text, and encoded guidance while base weights stay frozen.Training varies handle number and placement to expose the model to sparse, partial authoring controls.Source: §1 · p. 3

2 Related Work

• Classical virtual cinematography systems compose shots with a camera in a fully modeled 3D scene, whereas GenCine uses a scaffold lifted from one image.Related approaches include cinematographic idioms, declarative camera planning, and procedural camera control.Source: §2 · p. 4
• Video-diffusion camera-control methods condition on poses, render geometry-derived caches, or jointly generate frames with point maps, camera trajectories, and flow.Examples include MotionCtrl (Wang et al., 2024), CameraCtrl (He et al., 2025a;b), GEN3C (Ren et al., 2025), and WorldReel (Fang et al., 2025).Source: §2 · p. 4
• Object-motion methods use image-space trajectories, entities, masks, boxes, predicted motion, or entity-level 6DoF trajectories.Examples include DragNUWA (Yin et al., 2023), Motion Prompting (Geng et al., 2024), and 3DTrajMaster.Source: §2 · p. 4
• Joint camera-and-object methods use geometry-aware guidance, Gaussian trajectories, persistent tracks, or canonical trajectories; GenCine maps persistent local-handle identities and positions into conditioning maps.GenCine's artist-placed 3D handles share the camera's world frame.Source: §2 · p. 4

3 Method

• GenCine projects background and handle points into each view, preserves their world-space positions and handle identities in guidance maps, and trains the model on paired videos and controls.The same map construction carries recovered motion at training time and artist-authored controls at inference.Source: §3 · p. 4, Figure 3 · p. 5

3.1 Scene Scaffold and Motion Authoring

• A depth estimate lifts the input image into a colored point cloud, which a dynamic mask partitions into static background and potentially mobile foreground for 3D motion authoring.Artists assign trajectories to selected foreground regions, each of which moves its points as a rigid group; the camera poses are specified in the same world frame.Source: §3.1 · p. 5

3.2 Motion Representation: Correspondence Maps and Rigid Handles

• Multiple independently controlled rigid handles approximate non-rigid motion without dense deformation fields, object templates, skeletons, or category-specific motion priors.The guidance maps pair foreground handle identity with current world-space XYZ positions, while background XYZ supplies a static scene reference.Source: §3.2 · pp. 5, 7

3.3 Architecture

• The system freezes Wan’s VAE and base diffusion weights while training a geometric-guidance side branch, convolutional projections, and LoRA adapters on paired videos and controls.The side branch receives encoded maps and validity masks, while the main branch receives the reference frame and foreground identity sequence.Source: §3.3 · p. 7

3.4 Sampling Motion Handles for Training

• For real clips, estimated geometry, camera motion, and tracked foreground trajectories yield approximate controls; synthetic clips provide ground-truth geometry and trajectories.Unreliable tracks and handles are filtered, handle counts and placements vary, and clips without valid foreground trajectories use background-only conditioning.Source: §3.4 · p. 7

4 Experiments

• Training uses 45k clips, including about 5k synthetic clips, and evaluation covers 75 DAVIS and SpatialVid examples across two motion categories.The categories are Camera Shooting and Object Interactions.Source: §4 · pp. 7–8

4.1 Main Results

• GenCine has the lowest translation error and R-LPIPS in both categories, but ranks second in rotation error behind Go-with-the-Track.On Camera Shooting it leads PSNR (16.64) and R-PSNR (17.16), and ties Wan-Move and Go-with-the-Track for lowest LPIPS (0.36 at reported precision).Source: Table 1 · p. 8, §4.1 · p. 8

4 Experiments

• Baselines receive different control inputs: some use dense ground-truth 2D trajectories, while VerseCrafter and SymphoMotion use camera and coarse object control; GenCine uses sparse 3D handles.GenCine's target camera trajectory is derived from the same reference motion.Source: §4 · p. 8

4.1 Main Results

• In artist-authored examples, GenCine coordinates the bus turn with camera pullback and articulates connected robotic-arm segments from sparse guidance.The comparisons also report less tunnel distortion than Wan-Move and VerseCrafter; Go-with-the-Track leaves most of the arm static, while VerseCrafter introduces extraneous boxes.Source: §4.1 · p. 8, Figure 5 · p. 9

4.2 Ablation Study

• Foreground XYZ and identity maps alone show background drift, background XYZ alone gives no explicit object-motion control, and full guidance better preserves the static scene while guiding the boat.The caption describes the three rows as foreground-only, background-only, and full guidance.Source: Figure 6 · p. 10
• Removing Fbg raises R-LPIPS from 0.184 to 0.235 and lowers R-PSNR from 17.16 to 16.20.The qualitative fixed-camera boat example shows background drift with foreground-only guidance, but differing saved camera exports limit attribution to background guidance alone.Source: §4.2 · p. 10
• Removing Ffg yields R-LPIPS 0.223 and R-PSNR 15.26, while sside = 2 improves R-LPIPS from 0.189 to 0.184 relative to sside = 1.The default configuration has the lowest R-LPIPS and highest background consistency and imaging quality among tested variants.Source: Table 2 · p. 10, §4.2 · p. 10

5 Discussion

• The single-image scaffold leaves occluded regions unmodeled, so the generator must synthesize unseen surfaces under large viewpoint changes.Geometry completion is proposed as a possible extension.Source: §5 · p. 11
• Local rigid handles suit articulated motion better than fracture, tearing, or fluid dynamics; many handles can be cumbersome for distributed deformation.Independently authored handles also impose no physical constraints or causal interactions between objects.Source: §5 · p. 11

AI Use Statement

• Generative AI was used for language polishing during manuscript preparation.Source: AI Use Statement · p. 12
• Qwen2.5-VL-72B generated missing training captions in Section 4.Source: AI Use Statement · p. 12
• The authors take responsibility for all content.Source: AI Use Statement · p. 12

Ethics Statement

• Generated videos should be identified as synthetic to reduce misleading use.Source: Ethics Statement · p. 12
• Privacy should be respected when using generated videos.Source: Ethics Statement · p. 12
• Applicable data and model licenses should be respected.Source: Ethics Statement · p. 12

• Sections 3 and 4 describe the method, datasets, training setup, and evaluation.Source: Reproducibility Statement · p. 12
• Supplementary Sections C–G provide implementation details.Source: Reproducibility Statement · p. 12
• The supplement also covers failure cases and limitations in Section H.Source: Reproducibility Statement · p. 12

• The project website hosts videos.The passage gives the URL generative-cinematographer.github.io.Source: Supplementary Material · p. 16
• The supplement begins with additional qualitative comparisons, then reviews latent video diffusion and details the scene scaffold, guidance maps, architecture, training, and inference.Source: Supplementary Material · p. 16
• It concludes with failure cases and limitations in Section H.Source: Supplementary Material · p. 16

A Additional Qualitative Comparisons

• In the teapot edit, GenCine better preserves the intended world-space orientation and scene geometry than Wan-Move, whose motion is geometrically inconsistent.Source: A Additional Qualitative Comparisons · p. 16, Figure 7 · p. 16
• Across hiking, camel, and dog examples, GenCine follows foreground motion while preserving scene context; its dog poses more closely match the reference than baseline outputs.Source: A Additional Qualitative Comparisons · p. 16, Figure 8 · p. 17

B Latent Video Diffusion Background

• The Wan backbone uses latent video diffusion with a pretrained VAE; its flow-matching objective operates in compressed space, and the VAE compresses time and space while expanding channels.For Wan2.1, the stated factors are ft = 4, fs = 8, and C = 16.Source: B Latent Video Diffusion Background · p. 18

C Scene Scaffold Details

• The initial camera frame defines the canonical world frame, and per-frame intrinsics remain constant unless the artist authors a zoom.The first-frame depth estimate and dynamic mask support the scaffold; invalid-depth pixels are excluded from the initial point cloud, and later dynamic-mask frames are padded with ones rather than predicted.Source: C Scene Scaffold Details · p. 18
• The dynamic mask partitions the colored point cloud into static background and potentially mobile foreground, while fixed artist-selected handle points move rigidly as a group.Independently moving handles provide a piecewise-rigid approximation to non-rigid motion; more handles allow finer local control.Source: C Scene Scaffold Details · p. 18

D Guidance Map Construction Details

• Background and foreground XYZ maps share one per-clip normalization derived from first-frame geometry; distant depth outliers are clipped before scale estimation, and extreme-scale clips are excluded from training.Source: D Guidance Map Construction Details · p. 19
• Fixed handle colors preserve identity across frames, while projected Gaussian footprints encode spatial weights and select the visible handle by depth ordering.Handles with fewer than five points are filtered, and covariance receives a positive diagonal floor for stability.Source: D Guidance Map Construction Details · p. 19
• Validity masks distinguish invalid zero-valued guidance from valid encodings; background-hole refinement accepts a proposed hole only when a local surface fit yields a camera-ray intersection.The refinement does not reconstruct surfaces occluded in the input image.Source: D Guidance Map Construction Details · p. 19

E Architecture Details

• GenCine uses a frozen Wan2.1-Fun-V1.1-14B-Control backbone with an eight-block side branch operating at half spatial resolution, inter-branch projections, and LoRA adapters.The side branch has about one-fifth of the main branch’s parameter count; LoRA adapters have rank 64, and the default downsampling uses one quarter as many spatial tokens.Source: E Architecture Details · pp. 19–20

F Training Details

• The main branch receives the noisy video latent and foreground identity sequence, while the side branch receives encoded guidance maps and validity masks; training uses 45k clips, including about 5k synthetic clips with ground-truth geometry and trajectories.Real clips use estimated geometry and motion; clips are processed at 480 × 832 resolution.Source: E Architecture Details · pp. 20–21, F Training Details · p. 21, §F.1 · p. 21

F.1 Sampling Motion Handles for Training

• Training handles are sampled from observed foreground motion, with distinct supports encouraged by rejecting centers inside accepted support balls; evolving local tracks can change support as trajectories become invalid or reappear.A handle is retained only if its valid-frame ratio satisfies the stated threshold; frames with fewer than nmin points are invalid.Source: §F.1 · pp. 21–22

G Inference Details

• In Blender, artists select 3D foreground handles and author their trajectories with a camera path; with no foreground handles, the model still receives the reference image, text, and background XYZ guidance.When no camera motion is authored, the camera remains at its initial pose.Source: G Inference Details · p. 22

H Failure Cases and Limitations

• Depth errors on reflective or transparent surfaces and thin structures can propagate into the background XYZ map, while occluded surfaces remain unmodeled and must be synthesized as they appear.These limitations can cause geometric inconsistencies during viewpoint changes.Source: H Failure Cases and Limitations · p. 22
• Rigid handles can approximate articulated motion, but fracture, tearing, and fluid dynamics remain difficult; finer control requires more handles and authoring effort.Independently authored handles also impose neither physical constraints nor causal interactions between objects.Source: H Failure Cases and Limitations · p. 22
• Points moved outside the first-frame scene bounding box can produce foreground XYZ values outside [0, 1]^3; authoring within the scene extent keeps guidance values in the RGB range.The per-axis scale is exposed in the UI as a visual cue.Source: H Failure Cases and Limitations · p. 23