PaperlayerView exact revision on arXiv

Source-linked AI summary

PartiCam: Camera Controlled Video Generation with Reward Guidance

Amine Ouasfi, Runjia Li, Junlin Han, Eric Marchand, Philip H. S. Torr, Adnane Boukhayma

arXiv:2609.39504v1cs.CVcs.AI

TL;DR

Precisely controlling camera trajectories in generated video remains challenging, while training-based approaches face data, computation, and backbone constraints. PartiCam combines global SMC exploration with local reward-guided particle-filtered restarts, and evaluations report state-of-the-art reshooting performance with improved trajectory adherence, visual quality, and temporal coherence.

  • Problem

    Video models often lack fine-grained control, and training-based camera control can be costly and vulnerable to OOD generalization failures.

    Source: §2 · p. 2

  • Method

    PartiCam combines global Sequential Monte Carlo trajectory exploration with local reward-guided particle-filtered restarts for training-free camera control.

    Source: §8 · p. 10

  • Results

    PartiCam reports state-of-the-art performance on video-reshooting benchmarks across image and video settings, with improved camera-trajectory adherence, visual quality, and temporal coherence.

    Source: §2 · p. 3, §8 · p. 11

  • Takeaways & Limitations

    A reliable test-time procedure can generate paired camera-annotated videos for distillation back into model weights.

    Source: §2 · p. 2

    Maintaining multiple candidate trajectories can require a larger denoising budget, and effectiveness depends on task-specific reward-function tuning.

    Source: §7 · p. 10

Abstract

from arXiv · show

We present PartiCam, a training-free Particle filtering rooted method for improved Camera controlled video generation. Generating videos that follow a precisely specified camera trajectory remains challenging for large video diffusion models. Training-free approaches are backbone-agnostic and avoid the need to construct large camera-annotated datasets by steering pretrained models toward the desired camera motion at test time. This enables the generation of camera-controlled video data that can subsequently be used to train camera-conditioned video diffusion models. Existing sampling-based guidance approaches often suffer from unstable trajectories: they either explore too broadly and fail to respect the target camera motion or collapse early and lose visual diversity over time. We introduce a global-local refinement framework for diffusion reward guidance, enabling accurate and consistent camera control during video generation. Our method builds on Sequential Monte-Carlo (SMC) guidance, but introduces a local refinement stage based on particle filtered resampling. Experiments show large improvements in camera trajectory adherence, reduced drift, and better visual quality, without requiring model retraining.

2. Introduction

PartiCam combines global SMC exploration with local reward-guided PF-Restart to address drift and instability in camera-controlled video generation. It reports state-of-the-art video-reshooting performance across image and video settings.

2. Introduction

  • Training-based camera control can require scarce paired supervision and substantial computation, while backbone-specific control modules risk obsolescence as video models evolve.Training-free methods are largely backbone-agnostic and can inherit improvements in newer base models without additional training.Source: §2 · p. 2
  • Reliable test-time procedures can generate paired camera-annotated videos whose samples can be distilled back into model weights.The paper relates this process to rejection-sampling fine-tuning.Source: §2 · p. 2
  • Score modulation steers a single denoising trajectory using warped input views, but early errors can compound into persistent camera drift.Increasing the number of inference steps does not necessarily correct this drift.Source: §2 · p. 2
  • Unrestricted stochastic restarts can discard useful denoising structure, so PF-Restart scores candidate continuations by camera alignment and retains the best.The local procedure supports differentiable and non-differentiable reward functions, but alone lacks broader population exploration.Source: §2 · p. 2
  • Global SMC jointly propagates, reweights, and resamples candidate trajectories, while local PF-Restart refines high-reward particles to mitigate drift and preserve diversity.The framework is also applicable on top of trained camera-control baselines, where it mitigates some OOD generalization failures.Source: §2 · pp. 2–3
  • PartiCam achieves state-of-the-art video-reshooting performance across image and video settings, outperforming popular training-free and training-based camera-control methods.The evaluation covers reshooting from images and videos for static and dynamic scenes.Source: §2 · p. 3

3. Related Work

Related work places PartiCam among training-free inference-time guidance methods, stochastic restart strategies, and reward-aligned SMC approaches. It also contrasts camera-control approaches based on explicit 3D reconstruction or learned multi-view diffusion.

Controllable Diffusion Models

  • Inference-time diffusion guidance includes noise or attention manipulation, gradient-based guidance, and particle filtering; PartiCam uses reward-weighted SMC without costly gradient backpropagation.The related-work discussion notes that attention methods can be difficult to adapt to complex 3D constraints, while gradient-based guidance can be expensive and high variance.Source: §3 · p. 3

Stochastic Sampling and Restart Mechanisms

  • Prior restart methods use local annealing but lack principled candidate weighting; PF-Restart formalizes restart selection through reward-weighted resampling [14].Other cited approaches include predictor-corrector samplers, partial noise cycles, and learnable forward-backward passes [13] [14].Source: §3 · p. 3

Inference-Time Alignment and Sequential Monte Carlo

  • Reward-aligned SMC methods include twisted proposals, adaptive resampling, and reward-aware particle initialization; PartiCam extends global SMC weighting to video-latent camera alignment [77].The related-work discussion distinguishes training-based aligners, which require parameter updates, as orthogonal to this approach [6] [25].Source: §3 · p. 3

Camera-Controlled Generation

  • Reconstruction-based 3D methods struggle with large viewpoint extrapolation or disocclusions, while multi-view diffusion often requires costly training data and can have limited temporal consistency [2] [8] [9] [10] [12] [24] [27] [37] [43] [44] [48] [52] [72].The discussion also describes dynamic reconstruction using canonical 3D and deformation fields [46].Source: §3 · pp. 3–4

4. Method

PartiCam steers latent video trajectories with global SMC exploration and local particle-filtered restarts, targeting camera paths and desired visual properties.

4.1. Problem Setting

  • The sampler seeks latent trajectories that follow a target camera path while rewarding background consistency, motion smoothness, and aesthetic quality.Source: §4.1 · p. 4

4.2. Preliminaries: Score Modulation for Camera Control

  • Score modulation steers denoising with warped input views, but early trajectory errors compound and unrestricted stochastic restarts can degrade generation quality.Source: §4.2 · p. 4

4.3. Global Trajectory Exploration via SMC

  • SMC propagates weighted trajectories toward reward-weighted diffusion paths, using expected-future-reward potentials and resampling when effective sample size falls below a threshold.Source: §4.3 · pp. 4–5

4.4. Local Refinement via Guided Restarts

  • At selected timesteps, each particle generates Nr noising-denoising candidates; reward-guided local resampling explores its neighborhood while approximately preserving the diffusion marginal.Source: §4.4 · p. 5

5. Implementation Details

The implementation separates camera correction from aesthetic exploration and reports inference time below NVS-SOLVER while preserving trajectory accuracy and visual quality.

5. Implementation Details

  • Local PF-Restart corrects camera trajectories through reward-guided restarts, while global SMC periodically resamples reward-weighted trajectories to preserve diversity.Source: §5 · p. 6
  • Local refinement uses camera reward, whereas global SMC uses aesthetic reward; ablations indicate this split offers a trade-off between aesthetic quality and camera accuracy.Source: §5 · p. 6
  • With Nr=2, K=2, and 100 diffusion iterations, local refinement runs through iteration 32 and global SMC through iteration 40, updating every four iterations; local refinement repeats four times per update.Source: §5 · p. 6
  • Inference time stays below NVS-SOLVER while preserving trajectory accuracy and visual quality.Source: §5 · p. 6

6. Experiments

Experiments evaluate camera control and visual quality across static and dynamic scenes. PartiCam is reported to improve trajectory adherence and generated-video quality, with ablations examining its global and local components and reward choices.

6. Experiments

  • Reward guidance outperforms other methods in static-scene camera accuracy and visual quality, with better global trajectories and occluded-region structures.Table 1 covers single-image novel-view synthesis on static scenes; lower values are better for all metrics.Source: Table 1 · p. 6, §6 · p. 6

6.1. Camera Control in Dynamic Scenes

  • On dynamic scenes, PartiCam achieves lower ATE, RPE-T, and RPE-R, while generated videos show sharper structures and more consistent geometry.The authors also report fewer motion artifacts and better handling of disocclusions and dynamic regions.Source: §6.1 · pp. 6–7, Table 2 · p. 7

6.2. Ablation Studies

  • Across SVD and CogVideo backbones, PartiCam consistently improves on NVS-Solver; applying it to TrajectoryCrafter further improves camera accuracy with comparable visual quality.TrajectoryCrafter is a CogVideo finetune designed for camera control.Source: Table 3 · p. 7, §6.2 · p. 10
  • Unguided restart improves NVS-Solver’s ATE from 1.56 to 1.09 and FID from 133.96 to 117.07, but its per-scene effects vary widely.Without reward-guided resampling, noise injection makes generation unstable.Source: §6.2 · pp. 7–8
  • With K = 2 and Nr = 2, global SMC plus PF-Restart reaches RPE-T 0.09, RPE-R 0.09, ATE 0.37, and FID 103.82; Nr = 4 does not monotonically improve results.The reported configuration uses aesthetic reward globally and camera reward locally; the authors report 6× lower translation error and 4× lower ATE than NVS-Solver.Source: §6.2 · p. 8, Figure 5 · p. 9
  • Camera reward in both stages yields ATE 0.28, while aesthetic reward globally and camera reward locally offers a visual-quality and trajectory-adherence trade-off.The camera-global variant has FID 103.06 versus 102.51 for the aesthetic-global variant, and RPE-R 0.11 versus 0.14; relative to the K = 2, Nr = 2 default, its RPE-R is 0.11 versus 0.09.Source: §6.2 · p. 9

6.3. Mitigating limitations of TrajectoryCrafter[82]

  • The authors show inference-time improvements over TrajectoryCrafter on harder test trajectories and over ReCamMaster under challenging trajectories.The ReCamMaster comparison is described as qualitative.Source: §6.3 · p. 10, §6.4 · p. 10, Figure 6 · p. 10

7. Limitations

The method trades additional denoising budget for improved generation quality and control adherence, while its effectiveness depends on reward design.

7. Limitations

  • Maintaining multiple candidate trajectories can require a larger denoising budget.Reward guidance can also accelerate convergence by steering sampling toward better solutions early.Source: §7 · p. 10
  • The framework’s effectiveness depends on reward-function design, which may require task-specific tuning.Source: §7 · p. 10
  • The authors characterize the approach as a reasonable trade-off when generation quality and control adherence are priorities.Source: §7 · p. 10

8. Conclusion

PartiCam combines global SMC exploration with local particle-filtered restarts for reward-guided camera control. It produces videos that better follow desired camera trajectories while maintaining visual quality and temporal coherence.

8. Conclusion

  • PartiCam combines global Sequential Monte Carlo exploration with local particle-filtered restarts to balance exploration and refinement during denoising.Source: §8 · p. 10
  • Figure 7 compares estimated camera trajectories from PartiCam and TrajectoryCrafter with ground truth in two views.Source: Figure 7 · p. 11
  • Videos generated with the method better adhere to the desired camera trajectory while maintaining visual quality and temporal coherence.Source: §8 · p. 11
  • The framework offers reward-guided diffusion steering without gradient backpropagation or model retraining, and may extend to other controlled-generation tasks.Source: §8 · p. 11

10. Algorithm

The paper points to Algorithm 1 as a summary of its proposed method.

10. Algorithm

11. Qualitative results

Figures 8 and 9 provide additional qualitative comparisons for monocular-video reshooting of dynamic scenes, complementing Figure 4 with additional frames.

11. Qualitative results

Loading 2609.39504v1…