Plenoptic Video Generation

1. Introduction

• Prior methods such as ReCamMaster and TrajectoryCrafter primarily succeed in single-view settings and struggle with consistent spatio-temporal hallucinations across views.Stochastic generation and limited long-range spatial memory can yield geometric misalignment and view desynchronization.Source: §1 · p. 2
• PlenopticDreamer generates views autoregressively, conditioning each generation on previously generated video-camera pairs retrieved by 3D field-of-view co-visibility.This design is intended to synchronize hallucinations across time and viewpoints while preserving scene geometry and motion dynamics.Source: §1 · p. 2
• Progressive context-scaling, self-conditioned training, and long-video conditioning target stable learning, reduced error accumulation, and longer generated sequences.Context scaling gradually increases conditioning videos; self-conditioning fine-tunes on the model’s own synthesized outputs.Source: §1 · p. 2
• PlenopticDreamer achieves state-of-the-art video re-rendering on the Basic and Agibot benchmarks, including view synchronization, camera accuracy, and visual fidelity.It supports third-person-to-third-person and head-view-to-gripper-view transformations.Source: §1 · p. 2

2. Related Work

• Existing memory approaches are grouped as frame-level, latent-level, 3D-level, and network-level memory, with different stored representations and retrieval strategies.Examples include key historical frames, hierarchical tokens, reconstructed 3D structures, and test-time-updated network weights.Source: §2 · p. 2
• The reviewed approaches do not integrate memory mechanisms for long-term spatio-temporal coherence across multiple views.PlenopticDreamer is presented as a memory-based generative video-to-video re-rendering framework.Source: §2 · p. 2
• PlenopticDreamer retrieves video-camera pairs from a memory bank using 3D field-of-view-based selection to condition target-video generation.The framework also preserves some preceding generated frames as clean inputs for long-video generation.Source: Figure 2 · p. 3

3.1. Preliminary and Problem Definition

• Given a source video and target camera trajectories, the task synthesizes target videos that preserve source content and maintain cross-view spatio-temporal consistency, especially in hallucinated regions.The formulation uses a standard pinhole camera model with zero horizontal and vertical skew.Source: §3.1 · p. 3

3.2. Injecting Conditions into Video DiT

• The autoregressive model conditions each next video on previously generated video–camera pairs, avoiding a single-shot context that becomes computationally prohibitive as view count or resolution grows.The native enlarged-context approach can become prone to out-of-memory failures as the number of views or video resolution increases.Source: §3.2 · p. 4
• 3D FOV retrieval selects prior videos by spatial co-visibility across frames, while camera raymap tokens are added to video tokens before self-attention.Video-level similarity averages frame-wise similarities; when retrieved videos exceed context capacity, divide-and-conquer inference aims to cover diverse viewpoints while minimizing overlap.Source: §3.2 · p. 4, Figure 3 · p. 4
• For videos beyond the temporal window, overlapping chunks reuse frames from the previous chunk to preserve temporal continuity.The overlap consists of frames from the latter portion of the previous chunk and is included as additional conditioning.Source: §3.2 · p. 4

3.3. Training Strategy

• Progressive training gradually increases context size after training begins with a small context, stabilizing convergence; self-conditioned training replaces ground-truth conditions with generated outputs to improve robustness to imperfect inputs.The paper describes self-conditioning as a second training round using model-generated outputs from training-set inputs.Source: §3.3 · p. 5

4.1. Experiment Setting

• The Basic benchmark uses in-the-wild videos; Agibot tests head-to-hand and hand-to-hand transformations using synchronized head- and gripper-view data.The Agibot benchmark contains 200 test videos.Source: §4.2 · p. 5, §4.3 · p. 6

4.2. Experiment on Basic Benchmark

• On Basic, PlenopticDreamer outperforms all baselines in view synchronization across shots while maintaining high-fidelity visuals and camera accuracy.Qualitatively, it preserves consistent hallucinations under large-angle viewpoint changes, unlike ReCamMaster and TrajectoryCrafter.Source: Table 1 · p. 6, §4.2 · p. 6, Figure 4 · p. 6

4.3. Experiment on Agibot Benchmark

• On Agibot, PlenopticDreamer generates temporally consistent left- and right-gripper views from head-view videos across manipulation tasks.The comparison reports that retrained ReCamMaster* fails to maintain view synchronization and high visual quality.Source: §4.3 · p. 6, Figure 5 · p. 7, Table 2 · p. 7

4.4. Ablation Study

• Ablations examine how progressive training, self-conditioning, and video retrieval relate to camera accuracy, robustness, and synchronization.Removing progressive training shifts TransErr from 0.54 to 0.63; removing self-conditioning worsens FVD and IQ and yields artifacts, while random retrieval reduces synchronization. Increasing retrieved contexts from 4 to 6 improves consistency, with diminishing gains beyond that.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5. Application

• Long-video conditioning supports temporally consistent generation and spatial coherence across adjacent chunks under a leftward-rotation trajectory.The figure highlights seamless chunk transitions at their boundaries.Source: §4.5 · p. 7, Figure 7 · p. 8

• PlenopticDreamer combines autoregressive generation with spatio-temporal memory, progressive context-scaling, and self-conditioning to support consistent generation and reduce long-range error accumulation.Source: §5 · p. 8
• The focal-length example varies from 18mm to 100mm under a zoom-in camera trajectory to simulate differing depth-of-field effects.Source: Figure 8 · p. 8
• Remaining failures include occasional over-exposure and distortion in long-shot videos, and degraded performance in complex human motions such as dancing.The passage suggests pretraining data biases as a likely explanation for the complex-motion degradation.Source: Figure 8 · p. 8

• The retrieval setting uses k = 4 to balance context visibility and computation, with video consistency as the selection criterion.Source: §A.1 · p. 9
• Long-video inference starts with a 93-frame chunk and continues with 71-frame chunks, using 6 overlapped latent frames and a 0.45 conditioning ratio during training.Source: §A.1 · p. 9
• The synthetic pipeline uses clean ground-truth context for noisy pseudo-ground-truth generation, and long-shot autoregressive synthesis showed no clear performance gains.The N-shot comparison is evaluated on the Basic benchmark.Source: §A.1 · p. 9, Table R2 · p. 9
• In the image-matching figure, red points mark matched pixel correspondences across the input images.Source: Figure S1 · p. 9

A.2. Evaluation

• FVD uses StyleGAN-V features from 49 resized frames per interval and indirectly measures video similarity, including for substantial camera motion.For low-FOV-overlap videos, evaluation selects an alternative with a similar camera trajectory.Source: §A.2 · p. 9
• TransErr compares predicted and ground-truth camera translations after aligning estimated camera scale with the input cameras.The camera estimates come from ViPE or VGGT.Source: §A.2 · p. 9
• RotErr compares predicted and ground-truth rotation matrices using their trace and is reported in radians.The trace is denoted tr(·).Source: §A.2 · p. 9
• Mat. Pix. counts pixels whose confidence exceeds τ; evaluation sets τ = 0.5 and averages the resized 432×768 frames.The Basic benchmark’s sequential 12-shot trajectory is used for synchronization calculations on video pairs listed in Table R3.Source: §A.2 · p. 10, Table R3 · p. 10

B. More Qualitative Results

• Basic benchmark qualitative results are shown in Fig. S3, with comparative visualizations in Fig. S4.These are additional results beyond the main presentation.Source: B. More Qualitative Results · p. 10
• Agibot benchmark results are shown in Fig. S5, with corresponding comparisons in Fig. S6.Source: B. More Qualitative Results · p. 10
• Extended long-video generation results are shown in Fig. S7.Source: B. More Qualitative Results · p. 10
• The focal-length effect is illustrated in Fig. S8.Source: B. More Qualitative Results · p. 10