PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeCHROMA: Generative Chroma Keying from Unaligned Background References

Jiayi Wu, Tianfu Wang, Rui Zhao, Florian Schiffers, Zhimin Chen, Cornelia Fermuller, Yezhou Yang, Christopher Metzler, Yiannis Aloimonos

arXiv:2610.14445v1cs.CV

TL;DR

Conventional chroma keying and recent generative methods impose alignment, editing, or scene-conditioning constraints on compositing approved foreground performances. FreeCHROMA adapts a pretrained video diffusion model to generate and harmonize video from unaligned background references, and outperforms recent baselines across key compositing criteria. Its outputs target conditional plausibility and reference fidelity, not reconstruction of a unique unseen background trajectory.

  • Problem

    Existing compositing workflows require coordinated plates and harmonization, while generative approaches retain alignment, keyframe, or reference-fidelity constraints.

    Source: §1 · pp. 1–2

  • Method

    FreeCHROMA jointly generates a background and harmonizes it with a preserved foreground using one to three unaligned references, trained through pseudo-pairs constructed from natural videos.

    Source: §1 · p. 2

  • Results

    Across diverse interactions and camera motions, FreeCHROMA outperforms state-of-the-art baselines in foreground preservation, background reference fidelity, and harmonization.

    Source: §1 · p. 3

  • Takeaways & Limitations

    FreeCHROMA supports generative compositing without pre-aligned plates, edited keyframes, or a separate relighting stage.

    Source: §5 · p. 9

    The output is not unique: the objective is conditional plausibility and reference fidelity rather than reconstruction of a particular unseen background trajectory.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In film production, chroma keying composites an approved foreground performance into a new environment, but conventional workflows require coordinated background plates and dedicated harmonization. Recent generative methods often require pre-aligned backgrounds, focus on background-conditioned foreground relighting, or represent the environment only through an edited first frame. These restrictions limit background selection, scene composition, and camera motion while adding preparation or editing effort. We present FreeCHROMA, which adapts a pretrained video diffusion model in a parameter-efficient manner for generative video compositing from one to three unaligned background references. The model integrates the foreground into the referenced environment, preserving foreground identity, motion, and fine visual details while generating a spatiotemporally coherent background. Foreground-background harmonization occurs within the same generation process, without a separate relighting stage. Optional text instructions help disambiguate complex spatial relationships and camera behavior. To train the model, we construct pseudo-paired data from readily available real monocular videos through foreground-background separation and augmentation with existing foundation models. The original videos serve as training targets, while automated processing and quality verification enable scalable data construction. Across reference fidelity, foreground preservation, scene integration, harmonization, and overall video quality, FreeCHROMA consistently outperforms recent compositing and conditional video-generation baselines, with particularly clear gains under camera motion. FreeCHROMA relaxes the constraints of existing tools to enable end-to-end generative chroma keying from unaligned inputs, with exploratory experiments demonstrating its potential in robotics.

1 INTRODUCTION

FreeCHROMA preserves an approved foreground performance while generating and harmonizing an environment from unaligned references. It targets production workflows constrained by plate alignment, fixed backgrounds, and edited keyframes, using scalable pseudo-paired supervision from natural videos.

1 INTRODUCTION

  • Conventional compositing requires background plates aligned in viewpoint, framing, camera motion, perspective, and illumination, while recent systems retain pre-alignment or edited-keyframe constraints.Layer composition uses a fixed background, and newly revealed regions in keyframe editing can drift or become inconsistent.Source: §1 · pp. 1–2
  • Mid-shot action with tracking, panning, or orbiting cameras requires controllable environments as new regions enter view while approved choreography remains unchanged.Such shots need more than relighting an approximately independent foreground layer because foreground–background spatial relationships matter.Source: §1 · pp. 1–2
  • FreeCHROMA uses a masked foreground video, one to three unaligned environment images, and optional text to jointly infer composition, background evolution, and harmonization.The references need not match any output frame in camera pose, crop, or layout; text helps resolve ambiguous compositions.Source: §1 · p. 2
  • Pseudo-pairs from ordinary videos provide training supervision by reconstructing original videos from automatically extracted foregrounds and clean, unaligned background references.The pipeline perturbs foreground illumination and uses foreground removal and viewpoint transformation of selected keyframes, without green-screen capture, 3D annotations, or a separate relighting network.Source: §1 · p. 2
  • Across diverse foreground–background interactions and camera motions, FreeCHROMA outperforms state-of-the-art baselines in foreground preservation, background reference fidelity, and harmonization.The model integrates background synthesis and video-level harmonization without an explicit or post-hoc relighting stage.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

Prior work addresses foreground extraction, relighting, and conditional video editing, but these approaches often assume an established or pre-aligned composition. FreeCHROMA instead treats unaligned background images as persistent references for jointly inferring the environment and its relationship to the foreground.

2 BACKGROUND AND RELATED WORK

  • Matting and keying recover foreground layers, while harmonization and relighting adjust appearance within an established composition; this setting also requires inferring composition and camera-dependent background evolution.The cited work includes Smith & Blinn (1996), Lin et al. (2021), Cong et al. (2020), and Zhang et al. (2025).Source: §2 · p. 3
  • Video relighting methods coordinate foreground appearance and illumination over time, but often treat backgrounds as light or conditioning signals rather than structured environments for joint inference.RelightVid (Fang et al., 2025), Light-A-Video (Zhou et al., 2025), Lumen (Zeng et al., 2025), IllumiCraft (Lin et al., 2025), AnyPortal, and FlowPortal (Gao et al., 2025; 2026b) exemplify this line; PAI-Studio (Gao et al., 2026a) associates references with designated keyframes.Source: §2 · p. 3
  • Keyframe-guided editing delegates composition to edited frames, so newly revealed regions may be underconstrained and drift from the intended scene.ID-V2V (Xu et al., 2026) propagates a scene from a stylized first frame and optional keyframes; FreeCHROMA instead keeps standalone, unaligned references available throughout generation.Source: §2 · pp. 3–4

3 METHOD

FreeCHROMA formulates chroma keying as preserving a foreground performance while generating a plausible, reference-consistent environment without requiring aligned background frames. It builds supervision from natural videos and jointly composes and harmonizes the output through a parameter-efficient video diffusion model.

3.1 PROBLEM FORMULATION

  • The output preserves the foreground video’s semantic and temporal content while generating and harmonizing a background consistent with one to three environment images, without pixel correspondence or camera-pose alignment.The goal is conditional plausibility and reference fidelity, not reconstruction of a particular unseen background trajectory.Source: §3.1 · p. 4

3.2 SCALABLE PSEUDO-PAIRS FROM NATURAL VIDEO

  • Natural-video pseudo-pairs use extracted foregrounds and transformed background references as conditions, with the untouched source video as the temporally coherent training target.The target already has mutually consistent foreground, background, camera motion, and illumination.Source: §3.2 · pp. 4–5
  • Foreground illumination augmentation perturbs only the conditioning foreground, preventing exact copying from being optimal and providing an explicit harmonization signal from the original-video target.The bounded, temporally shared transform combines photometric changes, low-frequency spatial variation, and optional normal-guided directional shading.Source: §3.2 · p. 5

3.3 JOINT COMPOSITION AND HARMONIZATION

  • FreeCHROMA concatenates noisy-target, foreground, and mask latents through an expanded input projection, while prepended reference tokens remain available across denoising blocks and output frames.The frozen backbone is adapted by training added foreground and mask weights, the background-reference convolution, and DiT LoRA adapters; composition and harmonization occur in one denoising process.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

4 EXPERIMENTAL RESULTS

FreeCHROMA is evaluated on video quality, compositing, camera-trajectory consistency, augmentation ablations, and exploratory robotics applications. It outperforms most baselines on overall video-quality metrics and leads on all four task-specific compositing criteria.

4.1 EVALUATION PROTOCOL

  • Evaluation covers 500 foreground videos paired with two reference sets each, with shared pairs across methods and metrics for video quality, compositing, and camera trajectories.The foreground videos come from HumanVid-Real and OpenVidHD; baselines receive inputs adapted to their supported conditioning modalities.Source: §4.1 · pp. 6–7

4.2 GENERAL VIDEO QUALITY

  • FreeCHROMA outperforms most baselines across evaluated video-quality metrics, suggesting it retains the pretrained backbone’s general video-generation quality while adapting it to compositing.Table 1 reports video quality on the evaluation benchmark, using VBench except for multi-view consistency, which uses VBench-2.0.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 FOREGROUND PRESERVATION AND SCENE INTEGRATION

  • FreeCHROMA achieves the highest vision-language-model score on foreground preservation, edge quality, foreground–background fusion, and illumination harmony.Qualitative comparisons also report preserved foreground appearance, motion, and fine details, alongside coherent spatial geometry and lighting.Source: §4.3 · p. 7, Table 2 · p. 7, Figure 4 · p. 8

4.4 CAMERA-TRAJECTORY CONSISTENCY

  • FreeCHROMA has the lowest global and local trajectory errors and a relative path length closest to 1.0, indicating more faithful preservation of source camera motion.Trajectory comparisons use the original source video from which each foreground input was extracted.Source: Table 3 · p. 8, §4.4 · p. 9

4.5 ABLATION STUDY

  • Removing foreground illumination perturbation lowers fusion and illumination-harmony scores, while removing background viewpoint augmentation weakens camera-motion agreement.The full model performs best across all ablation metrics.Source: §4.5 · p. 9, Table 4 · p. 9

4.6 DOWNSTREAM ROBOTICS APPLICATIONS

  • Without robot-specific training, selected RoboCasa and DROID examples show coherent motion and plausible interactions, suggesting potential for realistic manipulation videos and rough visual world modeling.The experiments retain selected scene elements and motion while generating the rest; examples include successful and failed outcomes.Source: §4.6 · p. 9

5 CONCLUSION AND LIMITATIONS

FreeCHROMA preserves an existing foreground performance while jointly learning composition, background evolution, and harmonization from unaligned references. Its stated boundaries are short-clip scene memory and precise consistency across independently generated shots.

Conclusion

  • FreeCHROMA jointly learns composition, background evolution, and foreground–background harmonization from one to three unaligned reference images while preserving the existing foreground performance.Its scalable pseudo-pair pipeline uses natural videos, with parameter-efficient adaptation of a pretrained video diffusion transformer.Source: §5 · p. 9

Limitations

  • References provide context for short clips but do not supply persistent scene memory for long-form generation.Source: §5 · p. 9
  • Independently generated shots lack precise cross-shot foreground–background consistency beyond coarse textual guidance.Persistent environment representations and explicit cross-shot spatial constraints are proposed future directions.Source: §5 · p. 9

A QUALITATIVE ABLATION

The qualitative ablation shows that foreground illumination perturbation and background viewpoint augmentation address complementary failures. Together, they support foreground relighting and scene geometry consistent with the foreground camera when references are unaligned.

A QUALITATIVE ABLATION

  • Without foreground illumination perturbation, stage lighting, color casts, and exposure carry over instead of adapting to the target environment.During training, the foreground control therefore always carries the target video's illumination, allowing the model to learn that output lighting can be read from its input.Source: A QUALITATIVE ABLATION · p. 12
  • Without background viewpoint augmentation, the model treats references as aligned layers and forces an unaligned reference's viewpoint into the output.The full model instead synthesizes a background layout consistent with the foreground camera.Source: A QUALITATIVE ABLATION · p. 12, Figure 5 · p. 13
  • Without illumination perturbation, the model tends to copy source lighting; without viewpoint augmentation, it forces the unaligned reference viewpoint into an inconsistent scene.The full model instead re-lights the foreground and uses the reference as an appearance cue; both ablations use the same data and schedule as the full model, with only one component removed.Source: A QUALITATIVE ABLATION · p. 12, Figure 5 · p. 13

B DOWNSTREAM ROBOTICS APPLICATIONS

The same generation pipeline is explored for robotics without robot-specific training. Foreground sequences preserve robot actions while references and task prompts guide generated environments and interactions.

B DOWNSTREAM ROBOTICS APPLICATIONS

  • The robotics pipeline uses a masked foreground sequence to specify scene elements and motions to preserve, with references and task prompts guiding generation.The exploration uses the same generation pipeline without robot-specific training.Source: B DOWNSTREAM ROBOTICS APPLICATIONS · p. 13
  • For sim-to-real transfer, FreeCHROMA re-renders RoboCasa robot-object interactions into referenced environments while preserving arm motion and object interaction.A single background reference specifies the target environment, and a task prompt guides the interaction.Source: B DOWNSTREAM ROBOTICS APPLICATIONS · p. 13, Figure 6 · p. 14
  • In visual world modeling, the model retains robot action from RoboCasa or DROID sequences and generates surrounding scenes and objects' responses.RoboCasa robot bodies are rendered, while DROID robot bodies are segmented; these inputs exclude interacting objects.Source: B DOWNSTREAM ROBOTICS APPLICATIONS · p. 14

C VLM EVALUATION PROTOCOL

The evaluation uses Qwen3-VL-8B-Instruct to rate generated videos on scene integration and subject preservation. It samples frames, applies separate rating procedures, and aggregates repeated scores across 1,000 generated videos.

C VLM EVALUATION PROTOCOL

  • Qwen3-VL-8B-Instruct rates background fit, spatial plausibility, and outline cleanliness from 12 evenly spaced frames of each generated video.The criteria cover lighting and shadows, believable size and position, and clean outlines around difficult regions such as hair, fingers, and motion-blurred edges.Source: C VLM EVALUATION PROTOCOL · pp. 14–15
  • Subject preservation is rated separately by comparing 12 frames from the foreground-only original with 12 from the generated result.The assessment checks identity, shape, clothing, pose, and movement, while explicitly allowing lighting and color to change; the target background reference is not shown.Source: C VLM EVALUATION PROTOCOL · p. 15
  • Each aspect receives an integer score from 1 to 10; the protocol takes the median of three repeat ratings and averages scores over 1,000 generated videos.The 1,000 videos comprise 500 foreground videos, each paired with two distinct background conditions.Source: C VLM EVALUATION PROTOCOL · p. 15

D DETAILS OF DATA CONSTRUCTION PIPELINE

The data-construction pipeline creates pseudo-pairs from natural videos, augmenting foreground appearance and background references while retaining the original video as the target. Automated selection, extraction, and reference processing filter clips and create unaligned environmental cues.

D DETAILS OF DATA CONSTRUCTION PIPELINE

  • Pseudo-pairs augment foreground appearance and background references while retaining the original video as the training target.The pipeline uses natural videos and automated processing described in the data-construction section.Source: D DETAILS OF DATA CONSTRUCTION PIPELINE · p. 15
  • Video selection retains short clips with clear foregrounds, visible background, and no shot transitions, then rejects unstable masks, severe blur, or excessive foreground occupancy.Claude Opus 4.6 identifies foreground entities and checks visual quality; SAM 3.1 segments and tracks subjects and interacting objects for a joint matte.Source: D DETAILS OF DATA CONSTRUCTION PIPELINE · p. 15
  • NormalCrafter estimates video surface normals aligned with foreground frames for the illumination augmentation.Source: D DETAILS OF DATA CONSTRUCTION PIPELINE · p. 15
  • Background references come from one to three source keyframes, with foreground entities removed and viewpoint and crop changed to create unaligned references.ViPE camera trajectories guide keyframe selection; a vision-language model rejects candidates that alter scene identity or introduce conspicuous artifacts.Source: D DETAILS OF DATA CONSTRUCTION PIPELINE · p. 15

D.1 FOREGROUND ILLUMINATION AUGMENTATION

Foreground illumination is perturbed so the model cannot simply copy the source lighting, while the unchanged natural video remains the harmonized target.

D.1 FOREGROUND ILLUMINATION AUGMENTATION

  • The foreground condition is perturbed while the target video remains unchanged, requiring the model to preserve performance while recovering a compatible appearance.The original matte support is restored after the transformations, and multiplication by the support restores the black background.Source: §D.1 · p. 16
  • Photometric augmentation varies clip-wide appearance in approximately linear light, with s sampled from U(0.3, 1.4) and mild draws from U(0, 0.1) for 5% of clips.White balance, gamma, and saturation use sc = 0.35s; the remaining operators use s, with random parameters shared across the clip to avoid artificial flicker.Source: §D.1 · p. 16, Table 5 · p. 16
  • A fixed directional-ramp and radial-vignette field is applied with probability 0.6, with its combined gain clipped to [0.2, 2.0].The ramp direction is uniform on [0, 2π), its amplitude lies in [0, 0.5s], and vignette strength lies in [0, 0.4s].Source: §D.1 · p. 16
  • When frame-aligned normals are available, sampled directional shading uses clip-fixed light parameters; unavailable normals fall back to photometric augmentation.The gain parameters are sampled as a ∼ U(0.4, 0.8) and k ∼ U(0.3, 0.9), and normalization separates directional shading from global exposure.Source: §D.1 · p. 16

E DATASETS AND IMPLEMENTATION DETAILS

The implementation builds a filtered and rebalanced training set from real video clips, then fine-tunes a pretrained video model with LoRA and conditioned foreground and background inputs.

E DATASETS AND IMPLEMENTATION DETAILS

  • Filtering and motion/person/object rebalancing reduce an initial 453K clips to 35K, with 34.1K used for training and 900 held out for validation and testing.Coarse filtering retained 45K clips based on foreground occupancy, background clarity, temporal continuity, and absence of subtitles, logos, or black borders.Source: E DATASETS AND IMPLEMENTATION DETAILS · p. 16
  • The model starts from Wan2.2-Fun-A14B-Control and separately fine-tunes its high- and low-noise experts, using the checkpoint saved at step 4,500 for each.Training clips contain 81 frames at 720×1280 resolution, with a global batch size of eight.Source: E DATASETS AND IMPLEMENTATION DETAILS · pp. 16–17
  • LoRA is applied to attention query, key, and value projections and feed-forward linear layers, while the VAE, text encoder, and remaining backbone weights are frozen.The input projection trains only foreground and mask condition weights, and the background-reference convolution is also trained.Source: E DATASETS AND IMPLEMENTATION DETAILS · p. 17
  • Each example supplies a masked foreground video, binary mask sequence, and up to three background references, with text and background conditioning dropout during training.Text dropout is 10%; background conditioning uses 10% whole-set and 10% per-reference dropout, retaining at least one valid view when the set is kept; inference uses 25 steps and guidance scale 4.5.Source: E DATASETS AND IMPLEMENTATION DETAILS · p. 17

Related papers

Loading 2610.14445v1…