PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2LIVE: Text-Driven Layered Image and Video Editing

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, Tali Dekel

arXiv:2204.02491v2cs.CV

TL;DR

Text2LIVE addresses zero-shot, localized text-guided editing of existing objects in real-world images and videos without a pre-trained generator or user masks. It generates CLIP-guided RGBA edit layers from a single input and composites them over the original, demonstrating semantic edits across varied objects, scenes, textures, and effects. The paper also identifies prompt sensitivity and dependence on accurate neural-atlas representations as limitations.

  • Problem

    Text2LIVE addresses localized semantic editing of existing objects in real-world images and videos, where controlling localization, preserving original content, and maintaining visual quality is challenging.

  • Method

    Text2LIVE trains a generator through internal augmented image-text examples and CLIP losses to produce an RGBA edit layer composited over the input.

  • Results

    The method demonstrates localized, semantic edits on high-resolution images and videos, including object textures and complex semi-transparent effects across varied objects and scenes.

  • Takeaways & Limitations

    Layered editing and internal learning support automatic text-guided appearance manipulation while preserving the original input without a pre-trained generator.

  • Takeaways & Limitations

    Image edits can follow CLIP’s biased solutions and vary with similar prompts, while video editing is restricted to examples where the pre-trained NLA representation works well.

Abstract

from arXiv · show

We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with visual effects (e.g., smoke, fire) in a semantically meaningful manner. We train a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. We demonstrate localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.

1 Introduction

Text2LIVE targets zero-shot, localized semantic editing of existing objects in real-world images and videos using simple text prompts. It generates layered edits from a single input, preserving original content while supporting textures and complex effects.

  • 1 Introduction: Text2LIVE uses simple text prompts to specify both the target appearance and the object or region to edit.The method can automatically locate a cake region and synthesize an “oreo cake” texture that combines with the original image.
  • 1 Introduction: The method generates an RGBA edit layer and composites it over the input instead of directly generating the edited image.Text-driven losses applied to the layer guide edit content and localization.
  • 1 Introduction: An internal learning approach trains a generator on diverse augmented image-text examples derived from a single input.The approach is presented as strong regularization for generating complex textures and semi-transparent effects.
  • 1 Introduction: The framework extends to video by decomposing complex motion and scenes into 2D atlases that represent foreground objects or backgrounds throughout the video.Edits applied to a unified atlas can support consistent text-guided video editing.
  • 1 Introduction: The paper demonstrates localized semantic edits across textures and semi-transparent effects on varied objects and scenes.Examples include object textures, smoke, fire, and other complex effects in images and videos.

2 Related Work

Prior text-guided editing methods commonly rely on domain-specific generators, global control, or user masks. Text2LIVE instead combines CLIP-guided internal learning with layered representations to support localized edits in images and consistent edits in videos.

  • 2 Related Work: Conditional GAN approaches support text-guided editing but remain restricted in image domains and text expressiveness.Methods such as ManiGAN edit object appearance while preserving content, but are trained for specific domains.
  • 2 Related Work: Pre-trained GAN methods require GAN inversion for real-image editing, while their applicable image domains remain limited.StyleCLIP and StyleGAN-NADA manipulate latent codes or output domains of StyleGAN2.
  • 2 Related Work: Diffusion-based CLIP generation is globally controlled and is not designed for localized edits applied only to selected objects.Region-based diffusion editing uses user-provided masks for inpainting.
  • 2 Related Work: Test-time CLIP optimization avoids a pre-trained generator, but related methods such as CLIPStyler focus on global stylization rather than localized object edits.Text2LIVE shares the use of CLIP guidance while targeting local semantic changes.
  • 2 Related Work: For video editing, prior approaches use keyframes, optical flow, or layered video representations, whereas Text2LIVE uses a neural atlas as a renderer for text-guided edits.The atlas represents video content across frames in a unified 2D form.

3 Text-Guided Layered Image and Video Editing

Text2LIVE performs localized, semantic appearance edits by generating an RGBA layer over the input, using CLIP-guided losses to control target appearance, structure preservation, and edit localization. It trains separately for each image or video from augmented internal examples, while bootstrapping opacity from text relevancy.

  • Layered editing: The generator produces an RGBA edit layer that is composited over the source image, rather than directly generating the edited image.The layer consists of color C and opacity α, with Io = α · C + (1 − α) · Is.
  • Objective function: The objective combines composition, screen, structure, and sparsity losses to match the target text while preserving content and limiting edit extent.The loss weights λg, λs, and λr control the relative contributions of the auxiliary terms.
  • Objective function: Composition loss drives the composite image toward the target prompt using CLIP cosine and directional distances.The directional term compares the change from source to output with the change from region-of-interest text to target text in CLIP space.
  • Objective function: Screen loss directly supervises the edit layer by compositing it over green and matching a text template such as “fire over a green screen.”This provides intuitive supervision for effects such as semi-transparent fire while separating the desired effect from the original image content.
  • Objective function: Structure loss preserves spatial layout, shape, and perceived semantics through self-similarity matrices computed from CLIP’s deepest spatial tokens.It minimizes the Frobenius distance between the source and edited self-similarity matrices.
  • Localization and training: Text-driven relevancy bootstrapping initializes opacity, while internal augmented examples and annealed supervision refine noisy maps into clean localized mattes.The generator is trained from scratch for each input using crops, color jitter, flips, and text-template augmentations.

4 Results

The method is evaluated on diverse real-world images and videos, with qualitative and human studies showing localized, high-quality edits and benefits from its design choices. Limitations include CLIP's bias toward particular visual solutions and dependence on accurate video atlas representations.

  • The evaluation covers 35 high-resolution images and seven DAVIS videos, with edits spanning object textures, smoke, fire, and clouds.
  • Our method automatically locates existing objects and produces high-quality textures that naturally combine with original image content.Compared methods either generate new masked content, edit globally, or lose fidelity to the input or target text.
  • 2,400 user judgments over 19 video-text combinations support the complete model, while frame-based edits are temporally inconsistent and atlas-based edits are often blurry.
  • Ablation Study: Relevancy-based bootstrapping refines rough relevancy maps into accurate object mattes, improving localization.
  • Ablation Study: Removing sparsity causes global color shifts, while removing structure prevents full preservation of the mango shape.
  • Limitations: CLIP bias can produce unwanted new objects or prompt-dependent edit variations, although more specific text can sometimes steer the desired result.
  • Limitations: Video editing is restricted to cases where the pretrained NLA model accurately represents the original video, because atlas artifacts can propagate into edited results.

5 Conclusion

The paper frames zero-shot text-guided editing of existing objects in real-world images and videos as a new problem setting. Text-driven edit layers address localization, content preservation, and visual quality without a pretrained generator, and the framework extends to consistent video editing.

  • The proposed setting targets semantic, localized editing of existing objects within real-world images and videos.
  • Text-driven edit layers address edit localization, preservation of original content, and visual quality without using a pretrained generator in the loop.
  • The image framework is adapted with minimal changes to perform consistent text-guided video editing.

A Implementation Details

The appendix provides implementation details for the model architecture and training regime.

  • The appendix provides implementation details for the architecture and training regime.

A.1 Generator Network Architecture

The generator uses a seven-layer U-Net with skip-like feature concatenation and a sigmoid output layer for the final RGBA representation.

  • The generator is based on a U-Net with a seven-layer encoder and symmetrical decoder.
  • Each layer uses 3×3 convolutions followed by BatchNorm and LeakyReLU, with intermediate channel dimension 128.

A.2 Internal Dataset (Sec. 3.1)

The method constructs an internal training dataset by augmenting the source image and target text prompt at each training step.

  • The augmented source-image and text pairs form multiple internal training examples derived from the single input.
  • Each training step applies a random set of image augmentations to the source image Is and augments target text T.
  • Text augmentation uses predefined templates while preserving the prompt’s semantics and varying its resulting CLIP embedding.

Image augmentations

Image augmentations vary spatial geometry, orientation, and appearance, while text templates provide semantically consistent prompt variation during training.

  • Image augmentations: Random spatial crops use 0.85 and 0.95 of the image size in the image and video frameworks, respectively.
  • Image augmentations: Random horizontal flipping is applied with probability p=0.5, alongside jittering of brightness, contrast, saturation, and hue.
  • Text augmentations: Target text is composed with one randomly selected template from a predefined list of 14 semantics-preserving templates.
  • Text augmentations: Multiple prompts describing the same edit can be supplied as additional text augmentations.

A.3 Training Details

Training uses a pretrained ViT-B/32 CLIP model for losses, periodically includes the unaugmented input, and has reported image and video training times.

  • Implementation: The framework uses pretrained ViT-B/32 CLIP features to establish the losses and optimizes the full objective with specified relative weights.The weights are λg = 1, λs = 2 for images and 3 for videos, λr = 5 · 10^-2 for images and 5 · 10^-4 for videos, and γ = 2.
  • Implementation: Every 75 iterations, the unaugmented source image and target text are added to the training batch.
  • Runtime: An input image of size 512×512 takes approximately 9 minutes for 1000 iterations on one NVIDIA RTX 6000 GPU.
  • Runtime: One 70-frame video layer at 432 × 768 takes approximately 60 minutes for 3000 iterations on one NVIDIA RTX 8000 GPU.

A.4 Video Framework

The video framework trains generators on atlas crops mapped from sampled video locations, applies losses to reconstructed edited frames, and uses separate generators for foreground and background edits.

  • Atlas preprocessing: The framework renders and crops a discretized atlas to focus training on regions occupied by the video content.The atlas is rendered at 2000×2000 px, then cropped using the bounding box of mapped video locations; foreground cropping retains pixels with opacity above 0.95.
  • Training: Each training iteration samples a random frame, spatial crop, and three-frame temporal segment with frame offsets m∈{-k, 0, k}, where k=2.
  • Training: Sampled video locations are mapped to UV atlas coordinates using a pretrained mapping network, and the atlas crop is the minimal crop containing them.
  • Training: Atlas crops and target text are augmented to create internal training data, while losses are applied after mapping the atlas edit layer back to video frames.
  • Inference: At inference, the entire atlas is fed to the generator, and separate generators are trained for foreground and background edits.
Loading 2204.02491v2…