Rewriting a Deep Generative Model

David Bau; Steven Liu; Tongzhou Wang; Jun-Yan Zhu; Antonio Torralba

Source-linked AI summary

Rewriting a Deep Generative Model

David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba

arXiv:2007.15646v1Jul 30, 2020cs.CVcs.GRcs.LG

TL;DR

The paper asks how specific semantic and physical rules encoded in deep generative models can be changed. It proposes rewriting a layer as a linear associative memory, demonstrates interactive rule edits and proof-of-concept applications, and reports advantages over fine-tuning and edit-transfer methods. The approach enables novel combinations such as images of trees growing from towers, while standard parameter updates can overfit new examples without good generalization.

Problem
The paper addresses how to manipulate specific semantic and physical rules encoded in pretrained deep generative models.
Method
The method treats layer representations as key-value associative memories and optimizes a constrained update to add or edit one rule while preserving existing relationships.
Results
The method produces object additions, undesired-pattern removal, and global lighting changes, and quantitative experiments on several datasets outperform fine-tuning and image-based edit-transfer methods on photorealism and desired effects.
Takeaways & Limitations
Rewriting existing generative models can combine learned rules into novel, endlessly synthesizable image distributions, including tree-and-tower compositions not seen during training.
Takeaways & Limitations
Standard unconstrained parameter updates can quickly overfit new examples without producing a generalized rule.

Abstract

from arXiv · show

A deep generative model such as a GAN learns to model a rich set of semantic and physical rules about the target distribution, but up to now, it has been obscure how such rules are encoded in the network, or how a rule could be changed. In this paper, we introduce a new problem setting: manipulation of specific rules encoded by a deep generative model. To address the problem, we propose a formulation in which the desired rule is changed by manipulating a layer of a deep network as a linear associative memory. We derive an algorithm for modifying one entry of the associative memory, and we demonstrate that several interesting structural rules can be located and modified within the layers of state-of-the-art generative models. We present a user interface to enable users to interactively change the rules of a generative model to achieve desired effects, and we show several proof-of-concept applications. Finally, results on multiple datasets demonstrate the advantage of our method against standard fine-tuning methods and edit transfer algorithms.

1 Introduction

The paper introduces model rewriting: editing a pretrained generative model’s semantic and physical rules so it can synthesize unbounded new images under modified rules. The method targets specific internal relationships, preserving existing semantics while enabling effects such as removing patterns, adding objects, and replacing definitions.

Model rewriting edits a generative model’s semantic and physical rules rather than individual input images.This enables the altered model to synthesize an unbounded number of new images following the modified rules.
Editing the generator can remove watermarks, add crowds, or replace tower tops with treetops across whole categories of synthesized images.The StyleGANv2 church-scene example applies each modification globally, producing samples from an endless distribution.
Rewriting models could make editing large collections, videos, or virtual worlds less tedious by transferring similar edits across many outputs.The paper motivates model-level authoring as an alternative to editing individual images or frames.
The approach stores latent rules as key-value relationships in hidden features and edits one associative-memory rule while preserving other semantic relationships.The method directly measures and manipulates the model’s internal structure without requiring new training data.
The method supports visual effects including new object arrangements, systematic pattern removal, and global changes to physical-light modeling.A user can provide a single example or a small handful of examples, and the paper demonstrates an interactive interface for novice users.

2 Related Work

The paper distinguishes model rewriting from image manipulation, edit transfer, interactive training, and transfer learning. Its focus is changing structural rules inside a pretrained generator so the altered network can produce countless images under those rules.

Unlike conventional deep image manipulation, model rewriting changes the generator’s structural rules instead of synthesizing or editing a single image.The altered network can produce countless new images following the modified rules.
Edit-transfer methods typically propagate pixel edits across aligned instances or adjacent video frames, whereas this method targets context-sensitive rules beyond pixel correspondences.The paper identifies alignment and restricted edit types as limitations of prior transfer methods.
Interactive machine-learning systems usually help users optimize a fixed objective, while model rewriting supports novel creative modeling tasks with a pretrained model.The distinction is between assisting optimization and creating networks with new rules beyond training-data patterns.
Transfer learning adapts learned models to unseen tasks, domains, or settings, whereas model rewriting creates new generative rules within an existing network.The related-work discussion places model rewriting alongside, but distinct from, transfer learning and fine-tuning.

3 Method

The method rewrites a pretrained generator by changing a targeted layer as an associative memory, optimizing a specific rule while preserving existing relationships. It derives a constrained, rank-one update for linear layers and restricts nonlinear optimization to the corresponding update subspace.

3.1 Objective: Changing a Rule with Minimal Collateral Damage: The method optimizes updated weights to match user-specified feature changes while minimizing interference with the generator’s existing behavior.Rather than modifying every parameter, it updates weights at one layer and measures distance in that layer’s output feature space.
3.1 Objective: Changing a Rule with Minimal Collateral Damage: Standard joint optimization can overfit target examples instead of producing a generalized rule within the generator.The many degrees of freedom in the full parameter set allow the generator to match new examples without generalizing the requested change.
3.2 Viewing a Convolutional Layer as an Associative Memory: A convolutional layer can be viewed as an associative memory that maps single-location feature keys to output-feature patterns, without changing the layer’s computation.The convolutional tensor is reshaped into a matrix, with each key k mapped to a value v through v = Wk.
3.2 Viewing a Convolutional Layer as an Associative Memory: For nonorthogonal keys, the pretrained memory minimizes retrieval error using the normal equations or the equivalent pseudoinverse solution.Keys and values are collected into matrices K and V, yielding W0KKT = VKT or W0 = VK+.
3.3 Updating W to Insert a New Value: Overwriting one key with a new value becomes a constrained least-squares problem that preserves the best fit to previously stored values.The update adds a rank-one matrix Λ(C^-1k*)^T, where C is the second-moment matrix of the keys.
3.3 Updating W to Insert a New Value: The update direction depends on key statistics and the targeted key, while the target value determines only the row-change magnitudes.For nonlinear layers, the method retains this rank-one direction and optimizes within its subspace rather than solving for a closed-form weight update.

4 User Interface

The Copy-Paste-Context interface lets users define a model rule by copying an object, pasting it into a target image, and selecting context regions to control generalization. The resulting edit changes the generative model, so the rule applies across newly generated images rather than only one image.

Copy-Paste-Context: The interface uses three steps—Copy, Paste, and Context—to specify and generalize a new model rule.Copy selects the source object, Paste defines the target K∗→V∗ relationship, and Context supplies regions used to determine the update direction.
Model-level editing: The edit is applied to the model rather than a specific image, so newly generated images acquire the modified rule.The interface example makes hats appear on horse heads across different generated horses and poses.
Copy-Paste-Context: Copy and Paste associate target-image features K∗ with copied layer values V∗ that determine the new appearance.The copied region is downsampled to the selected layer’s resolution before becoming the target value V∗.
Copy-Paste-Context: Selecting several context regions determines the update direction d and improves generalization of the edit.In Figure 3, four examples provide context for a single direction d.
Model-level editing: Applying one rule change to a StyleGANv2 model requires about eight seconds on a single Titan GTX GPU.The reported runtime concerns one rule change on the StyleGANv2 model.

5 Results

The method produces model-level edits that add or replace objects, remove undesired watermark features, and invert a contextual light-reflection rule. Across these settings, it generally yields realistic, targeted changes, although it is less aggressive than baselines for introducing smiles.

Experimental settings: The experiments evaluate three effects: adding or replacing objects, removing watermark features, and inverting a physical relationship between windows and reflections.The object edits include domes-to-spires, domes-to-trees, and closed-lips-to-open-mouth smiles.
Quantitative evaluation: The smiling edit measures smile efficacy, undesired changes outside the mouth, and realism across 10,000 generated images and 1,000 pairwise judgments.Smile efficacy uses an attribute classifier, while masked-image LPIPS measures changes outside the mouth region.
Quantitative evaluation: The dome edits measure target-category pixel changes, non-dome LPIPS changes, and realism across 10,000 generated images and 1,000 pairwise judgments.A segmentation model determines how many dome pixels change to the target category.
Results: The method produces more realistic and more narrowly targeted outputs than baseline methods overall.For smiles it is less aggressive at introducing smiles, whereas for dome edits it is more effective at executing the change.
Removing undesired features: A rank-one update removes both middle and bottom watermark text cleanly while introducing few changes to the main image content.The method uses ten annotated images for context and one successfully removed watermark as the optimization target.
Changing contextual rules: A single-layer update inverts the learned relationship between windows and reflections: adding windows reduces reflections, and vice versa.The experiment uses 15 paired scenes and constrains the change to rank one with projected gradient descent and SVD.

6 Discussion

The paper argues that rewriting pretrained generative models can create image distributions and combinations absent from training data. It also positions model rewriting as a promising way to use learned structure across future generative domains.

6 Discussion: Rewriting existing networks can synthesize image distributions containing combinations that were absent from the training data.The paper connects trees and towers through rewritten rules to generate images of trees sprouting from towers.
6 Discussion: Model rewriting may become useful beyond images as sophisticated generative models expand to language and audio.The discussion names GPT-3 and WaveNet as examples of generative models in other domains.
6 Discussion: The paper acknowledges support from DARPA, NSF, and NVIDIA hardware donations.

A Additional Editing Examples

Additional examples show that the method applies low-rank changes to StyleGAN v2 models, while the nonlinear setting requires optimization rather than direct matrix inversion. The examples illustrate edits that alter rules across generated images.

A Additional Editing Examples: Figures 7–12 demonstrate varied effects produced by single low-rank changes to StyleGAN v2 models.The examples include adding hats or longer tails, removing windows, reducing tree occlusion, and removing earrings or glasses.
A Additional Editing Examples: The linear formulation can solve for the updated weights and associative-memory entry simultaneously through a block matrix system.
A Additional Editing Examples: Because neural-network layers are nonlinear, the practical method finds Λ by optimizing the objective in Equation 17 instead of inverting the linear system.

C Implementation details

The implementation evaluates model rewriting across multiple generative architectures, datasets, edits, perceptual metrics, efficacy measures, and human realism comparisons. The examples include both realistic edits and a deliberately fantastical transformation excluded from realism testing.

C Implementation details: Editing sessions consist of user-drawn copy-paste masks and optional context regions, with benchmark sessions distributed in the source code.
C Implementation details: Additional examples show edits including longer horse tails, removed church windows, and reduced tree occlusion around buildings.
C Implementation details: The experiments use Progressive GAN and StyleGAN v2 models trained on LSUN kitchens, churches, horses, and FFHQ faces.Progressive GAN has 18.3 million parameters and 15 convolutional layers; StyleGAN v2 has 30 million parameters and 14 convolutional layers, or 17 for higher-resolution faces.
C Implementation details: LPIPS measures undesired perceptual differences by comparing unedited and edited images outside the intended edit region.The intended change is masked using semantic segmentation networks for faces and churches.
C Implementation details: Pretrained classifiers quantify edit efficacy for attributes such as smiling faces and dome-to-building or dome-to-tree changes.
C Implementation details: Human realism studies compare 500 image pairs per baseline method, yielding 1000 comparative judgments, but exclude the intentionally unrealistic domes-to-trees edit.

D Rank Reduction for DS

The rank-reduction procedure compresses user context selections into a lower-dimensional subspace that preserves informative directions under a Gaussian information model. Related variants identify dominant directions or axis-aligned units for the editing task.

D Rank Reduction for DS: The method transforms a user context selection K into a constraint subspace DS with dimensionality S much smaller than the number of feature samples T.
D Rank Reduction for DS: The rank-reduction objective treats context selection as lossy compression and measures its information using cross-entropy relative to a feature distribution P.
D Rank Reduction for DS: Removing earrings generalizes from one example to multiple earring types appearing in different poses.
D Rank Reduction for DS: The selected S-dimensional subspace preserves the most informative directions while discarding R = N − S uninformative directions.
D Rank Reduction for DS: The resulting subspace is described as the maximally informative low-dimensional representation of the user’s context selection.
D Rank Reduction for DS: An axis-aligned Gaussian variant selects unit directions with the largest expected scores and applies them to identify units relevant to watermarks.

F Experiment Details and Results

The experiments compare model rewriting with baselines on StyleGANv2 edits and specify the optimization settings used for each model variant.

Table 2 compares the proposed method with multiple baselines on two edits to a StyleGANv2 LSUN church model.
Both edits modify the generator’s 7th convolutional layer using Adam with learning rate 0.05 and 2001 gradient iterations.The update is projected to a low-rank change every 10 iterations and after optimization.
The domes →trees edit uses a rank 1 change, whereas domes →spires uses a rank 10 change.These settings are also the defaults used in the user interface and video demonstrations.
The StyleGANv2 FFHQ edit modifies the generator’s 9th convolutional layer with the same optimizer, learning rate, iteration count, and projection schedule.
The all-weights fine-tuning baseline uses Adam for 2001 iterations with a learning rate of 10^-4.

G Reflection Experiment Details

The reflection experiment reverses a context-sensitive lighting rule by optimizing a constrained layer-weight update from paired generated images and masked reflected-light regions.

The experiment targets a rank-one reversal of an abstract window-lighting rule.
The GAN generates 15 images with windows activated or deactivated, using window-correlated units identified by dissection.
A user masks illuminated regions away from windows where reflected light differs between paired images.
Optimization changes one layer so reflected light becomes dark when a window is present and bright when no window is present.An SVD reduction constrains the update to rank one every 10 iterations.
The selected layer is the one with the lowest loss under a rank-one change, which is layer 6 in this experiment.

H Selecting a Layer for Editing

The paper treats suitable convolutional layers as associative memories when local feature patches can be edited with limited dependence on neighboring locations. FID-based comparisons identify layers where isolated patches resemble patches rendered in full context.

A convolutional layer can be viewed either as combining neighboring information or as memorizing independent feature mappings.
The adopted model treats a layer as an associative memory mapping local feature vectors in one layer to local feature-vector patches in the next.This view assumes neighboring locations are sufficiently disentangled, a property observed in ProgressiveGAN and StyleGAN representations.
FID compares isolated rendered patches with equally sized crops from full generated images; lower FID indicates less dependence on neighboring patches.Higher FID indicates greater neighboring dependence.
StyleGANv2 layers 6–11 and Progressive GAN layers 4 and higher are identified as most appropriate for associative-memory editing.The featuremap-to-convolution indexing differs between the two architectures because StyleGANv2’s first featuremap layer is fixed.
Figures 14–15 visualize isolated patches versus full images for StyleGANv2, while Figures 17–18 provide the corresponding Progressive GAN comparisons.