PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Visual Editing with a Generatively Pre-Trained Artist

Andrew Brown, Cheng-Yang Fu, Omkar Parkhi, Tamara L. Berg, Andrea Vedaldi

arXiv:2205.01668v1cs.CV

TL;DR

Targeted visual editing aims to change a source-image region according to a driver image, but suitable example edits for training are scarce. E2EVE learns the conditional edit distribution using self-supervised examples synthesized from unedited images. The authors report that it significantly outperforms prior work across several datasets, with edits that are more natural and responsive to the driver.

  • Problem

    Targeted visual editing needs to combine a source image with a driver image in a specified region, but suitable training examples of the desired edits are scarce.

    Source: §1 · p. 2

  • Method

    E2EVE learns the conditional edit distribution end-to-end using self-supervised training examples synthesized from unedited images.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across several datasets, E2EVE significantly outperforms prior work, producing edits that are more natural and more dependent on the driver image.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The choice of augmentation controls how much driver information is retained, balancing natural-looking edits against faithfulness to the driver.

    Source: §3.2 · p. 6

    The model can propose unreasonable edits because it lacks a full understanding of image semantics, and its unsupervised, data-driven nature may produce unwanted biases.

    Source: §5 · p. 14

Abstract

from arXiv · show

We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change. Differently from prior works, we solve this problem by learning a conditional probability distribution of the edits, end-to-end. Training such a model requires addressing a fundamental technical challenge: the lack of example edits for training. To this end, we propose a self-supervised approach that simulates edits by augmenting off-the-shelf images in a target domain. The benefits are remarkable: implemented as a state-of-the-art auto-regressive transformer, our approach is simple, sidesteps difficulties with previous methods based on GAN-like priors, obtains significantly better edits, and is efficient. Furthermore, we show that different blending effects can be learned by an intuitive control of the augmentation process, with no other changes required to the model architecture. We demonstrate the superiority of this approach across several datasets in extensive quantitative and qualitative experiments, including human studies, significantly outperforming prior work.

1 Introduction

The paper frames targeted visual editing as changing a specified source-image region using a driver image while keeping the surrounding context as unchanged as possible. It learns this edit distribution directly, using self-supervised training data to address the shortage of example edits.

1 Introduction

  • The edit should look natural, preserve the source outside region R, and resemble the driver within R.The driver image gives visual rather than textual guidance for the desired change.Source: §1 · p. 2
  • Training the conditional edit model is difficult because suitable source, driver, region, and edited-image quadruplets are scarce.Prior approaches often learn an unconditional image prior instead, but can show weak dependence on the driver or produce artifacts.Source: §1 · p. 2
  • E2EVE learns the edit distribution end-to-end from automatically synthesized training quadruplets, designed to make source and driver images independent enough for use at inference.The training data is generated self-supervised from unedited images without manual intervention.Source: §1 · p. 2
  • Across several datasets, experiments report higher-quality edits that are more driver-dependent and natural-looking than prior GAN- and attention-based work.The authors also report easier implementation and tuning, and direct sampling without some prior methods’ expensive pre- or post-processing.Source: §1 · p. 3

2 Related Work

Prior work addresses targeted editing, global manipulation, inpainting, and image composition through distinct approaches with different control and alignment constraints. E2EVE instead targets end-to-end, image-driven editing in a specified region.

2 Related Work

  • Targeted image-editing approaches include spatial object manipulation, closed-set object addition or removal, and CLIP-based text-driven manipulation.The cited GAN-based approaches may require difficult GAN inversion, are not trained end-to-end, or offer limited fine-grained shape and texture control.Source: §2 · p. 3
  • Global image manipulation edits entire images, while inpainting conditions generated regions on surrounding context rather than a driver image.The paper distinguishes its task from both by conditioning on a driver image and targeting a region.Source: §2 · p. 3
  • Image-composition methods can perform worse when the driver is not sufficiently aligned with the source, an alignment that usually requires manual intervention.The authors report their end-to-end model performs better in this setting.Source: §2 · p. 3
  • The paper distinguishes its target from image-based out-painting, which it describes as fixing output tokens and using paired training data from the same image.Targeted editing adds the challenge of preserving context while mixing images with very different statistics.Source: §2 · p. 4

3 Method

E2EVE learns a conditional distribution for editing a specified region using a source image and a driver image. It constructs self-supervised training examples from unedited images, then uses a two-stage autoregressive model to generate edits.

3 Method

  • E2EVE models edited images conditioned on the source, driver, and binary edit-region mask, sampling an output from that conditional distribution.The output should blend the images naturally while retaining the source outside the edited region.Source: §3 · p. 4

3.1 End-to-End Conditional Generation

  • Direct conditional training uses negative log-likelihood on edit quadruplets, but assembling a sufficiently large set of human-created examples would be difficult.Once trained, the model directly samples an edited image; the central challenge is constructing the training set from available unedited images.Source: §3.1 · p. 5

3.2 Synthesizing a Dataset of Meaningful Edits

  • Self-supervision treats an unedited image as the target edit, masks a random region to form the source, and transforms that region to form the driver.The random transformation must remove enough information to avoid simple pasting without removing the driver information needed for a meaningful edit.Source: §3.2 · pp. 5–6
  • Block and free-form edits approximate unpaired source and driver inputs by reducing their correlation; free-form edits use segmentation masks while block edits use square regions.Free-form construction can reduce correlation further, but requires additional machinery such as a segmentation network.Source: §3.2 · pp. 6–7, Figure 4 · p. 7
  • Changing the augmentations controls what the editor transfers from the driver, such as style or color, and what it ignores, such as spatial arrangement.Augmentations should remove information that is not intended to transfer while preserving the desired edit cues.Source: §3.2 · p. 7

3.3 Two-Stage Conditional Auto-Regressive Image Generation

  • A discrete autoencoder compresses images into tokens, and a transformer models the autoregressive output distribution conditioned on source and driver tokens.The first stage uses VQ-GAN with 16-fold image compression; the second predicts output tokens sequentially from prior output tokens and conditioning information.Source: §3.3 · pp. 7–8
  • The model encodes the edit region through the hole in the masked source, allowing pixel-accurate masks without a separate region token.During inference, output tokens are sampled iteratively; causal masking ensures each prediction depends only on earlier tokens.Source: §3.3 · p. 8

4 Experiments

E2EVE is evaluated on block and free-form edits using naturalness, locality, and faithfulness measures, alongside human judgments and qualitative comparisons. Ablations show how crop and augmentation choices trade off these metrics and affect sample diversity and likelihood.

4 Experiments

  • Evaluation uses FID for naturalness, L1 distance outside the edit region for locality, and retrieval against distractor images for faithfulness.The evaluation triplets pair a source image and edit region with a same-location crop from another image, weakly constraining source-driver compatibility; experiments cover Dresses-7m, LSUN Bedrooms, and FFHQ.Source: §4 · p. 9

4.1 Quantitative Evaluation

  • E2EVE scores best in naturalness and first or second among non-trivial baselines in locality, while ranking second only to copy-paste in faithfulness on block edits.Copy-paste is more faithful but much less natural; inpainting baselines are natural but weakly faithful.Source: §4.1 · pp. 10–11, Table 1 · p. 11
  • Human assessors preferred E2EVE over EdiBERT as more natural 83.2% of the time and more faithful 80.5% of the time.The study used 256 Dresses-7m edit samples, each rated by three assessors.Source: §4.1 · p. 11
  • For free-form edits, E2EVE outperforms prior methods in naturalness, faithfulness, and locality, while EdiBERT produces blocky artifacts on irregular regions.The evaluation uses semantic segmentation to define the edit region and compares E2EVE with EdiBERT on Dresses-7m.Source: §4.1 · p. 11

4.2 Qualitative Evaluation

  • Qualitative comparisons show natural, faithful edits across faces, clothing, and bedrooms, with cohesive and varied outputs even for out-of-domain drivers.Figure 5 includes block and free-form edits; Figure 6 shows three outputs for the same source using weather or nature drivers.Source: Figure 5 · p. 12, §4.2 · p. 13, Figure 6 · p. 13

4.3 Ablations

  • Ablations expose trade-offs: reducing α increases naturalness but weakens faithfulness, while position augmentation improves naturalness and size augmentation improves it further at some faithfulness cost.The final model has unchanged locality, more diverse samples than the preceding variant, and minimized NLL; removing driver-similarity filtering lowers faithfulness for both E2EVE and EdiBERT, while E2EVE still beats prior work on that metric. Using one rather than two VQ-GANs also reduces faithfulness because small-driver details are harder to reconstruct.Source: §4.3 · pp. 13–14, Table 2 · p. 14

5 Conclusions, Limitations and Future work

E2EVE uses self-supervision from an unlabeled collection of natural images to train a targeted visual editor without manual supervision. The authors also identify limits in semantic understanding and possible unwanted biases, and propose extending the approach to 3D scenes and video.

Conclusions

  • Self-supervised training from unlabeled natural images supports a conditional image generator that responds to diverse user inputs and significantly outperforms prior work qualitatively and quantitatively.The method requires no manual supervision.Source: §5 · p. 14

Limitations

  • The data-generation technique may be difficult to extend to text-based edits, and limited semantic understanding can yield unreasonable edits or unwanted biases.The authors describe the model as unsupervised and data-driven.Source: §5 · p. 14

Future work

  • Future work includes extending E2EVE from images to spatial editing in 3D scenes and spatio-temporal editing in video.Source: §5 · p. 14

A Additional Qualitative Results

Supplementary examples show E2EVE making natural, driver-faithful, localized edits across FFHQ and Dresses-7m, while also revealing occasional unreasonable generations under specific conditions.

Dresses-7m — block edits

  • Dresses-7m block edits remain natural, localized, and driver-faithful while showing diverse clothing content, including patterns and structures absent from training images.The source and driver images shown were not depicted in Dresses-7m, yet the model blends their content cohesively.Source: A Additional Qualitative Results · p. 17

Dresses-7m — free-form edits

  • Free-form edits produce diverse, driver-faithful clothing structures, including T-shirt edits despite training only on dresses.The source preserves only the clothing outline, while generated waist and neckline structures vary.Source: A Additional Qualitative Results · p. 17, Figure 2 · p. 20

Unreasonable generations

  • A small number of Dresses-7m block-edit outputs are unreasonable, including residual masked regions, unnatural faces, and edits from non-aligned inputs.Residual holes occur when the region reaches the bottom rows; unnatural faces arise when the region includes the top edge; a legs-to-dress edit illustrates severe misalignment.Source: A Additional Qualitative Results · pp. 17–18, Figure 3 · p. 21

FFHQ — comparisons

  • Prior-work comparisons on FFHQ show E2EVE balancing naturalness, driver faithfulness, and locality, whereas prior methods struggle to achieve all three.The comparison is qualitative; the passage points to Table 1 for its quantitative demonstration.Source: A Additional Qualitative Results · p. 18, Figure 4 · p. 21

FFHQ — E2EVE

  • For FFHQ, using the same driver-and-region pairs across varied source images probes whether the model can repeat edits across sources.The accompanying qualitative results also show three distinct driver-and-region pairs for each source.Source: A Additional Qualitative Results · p. 18, Figure 5 · p. 22
  • On FFHQ, edits retain specific driver details across varied sources, while adapting hair color and texture to fit the surrounding image.Examples include glasses and hairstyles, including hair and beards.Source: A Additional Qualitative Results · p. 18

B Additional Method Details

This section supplements the main paper with implementation details on VQ-GAN training and evaluation-triplet construction.

B Additional Method Details

  • The supplementary section adds details on training the VQ-GANs used by the method and on forming evaluation data triplets.These topics supplement Sections 3.3 and 4 of the main paper, respectively.Source: B Additional Method Details · p. 23

B.1 VQ-GAN Training

The supplementary material specifies the VQ-GAN loss, evaluation-triplet construction, and baseline input preparation.

B.1 VQ-GAN Training

  • VQ-GAN training combines reconstruction, codebook-pull, and commitment losses, using stop-gradient and a straight-through gradient estimator for quantization.The commitment term encourages encoder commitment to an embedding and prevents the output from growing arbitrarily.Source: §B.1 · p. 23

B.2 Constructing Evaluation Data Triplets

  • Evaluation triplets pair a source and edit region with a driver crop from another image at matching spatial coordinates, and make the edit region larger than the driver.The larger region is intended to increase the chance of a feasible edit when datasets lack strict alignment, except FFHQ.Source: §B.2 · p. 23

C Additional Baseline Details

  • The supplementary material outlines baseline input construction, hyperparameters, code sources, and analyses of selected implementation choices.It identifies sections C.1–C.3 as covering these details.Source: C Additional Baseline Details · p. 23

C.1 Constructing Inputs for Baseline methods

  • Baselines receive composited images with the driver pasted at varied positions inside the edit region; the remaining hole is inpainted with source content.This avoids leaving an unnatural hole in the composite, which would disadvantage methods trained on natural images; the resulting input also shows more source content.Source: §C.1 · p. 24

C.2 Baseline Implementation Details

The baseline implementation details describe code and hyperparameter choices, including deviations from recommended settings for EdiBERT and In-Domain.

Baseline implementation choices

  • Where possible, the authors use official baseline code and recommended default hyperparameters, with selected deviations analyzed separately.The supplementary table describes design-choice comparisons and marks the settings reported in the main paper.Source: §C.2 · p. 24, Table 1 · p. 25

EdiBERT

  • For EdiBERT, one optimization epoch is reported instead of the authors’ recommended two, based on an empirical balance between metrics.The implementation also dilates the mask by one token, periodically collages with the input, and uses spiral sampling.Source: §C.2 · p. 24

In-Domain and GAN inversion

  • For In-Domain, removing the domain-regularized optimization stage empirically gives a better balance between metrics; GAN inversion uses recommended StyleGAN2-Ada settings.In-Domain is implemented as GAN inversion followed by domain-regularized optimization before the reported removal of that second stage.Source: §C.2 · p. 25

C.3 Quantitative Analysis of Baseline Design Choices

The baseline analyses show that optimization and regularization choices trade off naturalness, faithfulness, and locality. The additional results section also examines computational efficiency and sampling techniques.

C.3 Quantitative Analysis of Baseline Design Choices

  • More EdiBERT optimization epochs improve naturalness but sharply reduce faithfulness; E2EVE remains superior on both metrics across the tested versions.The paper reports one optimization epoch for the main comparison as the most competitive balance between metrics.Source: §C.3 · p. 26
  • In-Domain regularization improves faithfulness but substantially reduces locality and naturalness, with FFHQ locality at 0.321 and source recognizability also impaired.The paper reports unregularized In-Domain metrics in the main paper because that version’s editing can be considered local.Source: §C.3 · p. 26

D Additional Quantitative Results

D.1 Computational Efficiency of E2EVE

E2EVE samples edits directly from its trained model, achieving higher throughput than some GAN-based approaches and throughput comparable to EdiBERT. Its speed is competitive despite sampling every output token, while fast In-Domain inference comes with worse editing metrics.

D.1 Computational Efficiency of E2EVE

  • E2EVE exceeds EdiBERT’s throughput when EdiBERT uses more than one optimization epoch, despite E2EVE sampling every output token and EdiBERT sampling only edit-region tokens.E2EVE’s 24-layer backbone is smaller than EdiBERT’s 32-layer backbone, contributing to comparable throughput.Source: §D.1 · pp. 26–27
  • E2EVE achieves higher throughput than GAN inv and regularized In-Domain, and comparable throughput to EdiBERT; Table 2 reports generated images per second.Throughput is measured on an NVIDIA A100 with batch size 1 and averaged over multiple samples.Source: §D.1 · p. 26, Table 2 · p. 27
  • Unregularized In-Domain inference is fast, but its generated samples are significantly worse than E2EVE across naturalness, faithfulness, and locality.Source: §D.1 · p. 27

D.2 Effect of Sampling Methods

Sampling choices affect edit quality: deterministic sampling sharply reduces naturalness and faithfulness, while other tested top-p and top-k settings are broadly robust. Greedy sampling favors locality, whereas sampling from more tokens can introduce non-local edits.

D.2 Effect of Sampling Methods

  • Top-k sampling restricts each step to the k highest-probability tokens, while top-p sampling retains tokens up to a cumulative-probability threshold.Top-p 0 and top-k 1 are deterministic; top-p 1 and top-k 1024 consider every token.Source: §D.2 · p. 27, Figure 6 · p. 28
  • Deterministic sampling sharply reduces naturalness and faithfulness, while performance across other tested top-p and top-k values remains fairly consistent across metrics and datasets.The paper attributes the robustness to peaked token distributions conditioned on the editing inputs.Source: §D.2 · pp. 27–28
  • Locality is best with deterministic sampling and degrades as more tokens are considered, because long-tail samples can introduce non-local edits.The authors conjecture that the most probable token outside the edit region often corresponds to the source-image token at that location.Source: §D.2 · p. 28

Related papers

Loading 2205.01668v1…