Source-linked AI summary
Deep Feature Interpolation for Image Content Changes
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, Kilian Weinberger
TL;DR
Image transformation methods often rely on specialized generative architectures, raising whether many tasks truly require that complexity. DFI instead interpolates attributes in a pre-trained convolutional feature space, and the authors report strong results across tasks, including high-resolution transformations, while identifying alignment and reference-similarity constraints.
Problem
Many image-transformation tasks have been approached with specialized generative architectures, motivating the question of whether simpler methods can solve them.
Method
DFI transforms images by linearly interpolating pre-trained convolutional features, using differences between mean features of images with and without an attribute.
Results
DFI performs surprisingly well across several transformation tasks, often outperforming substantially more involved state-of-the-art methods and supporting high-resolution images.
Takeaways & Limitations
DFI provides a simple competitive baseline for certain image transformations on aligned data and helps identify tasks where generative models remain advantageous.
Takeaways & Limitations
DFI requires aligned data and reference images otherwise similar to the target, and it struggles with large masked regions, shape changes, and rotation transformations.
Abstract
from arXiv · showhide
We propose Deep Feature Interpolation (DFI), a new data-driven baseline for automatic high-resolution image transformation. As the name suggests, it relies only on simple linear interpolation of deep convolutional features from pre-trained convnets. We show that despite its simplicity, DFI can perform high-level semantic transformations like "make older/younger", "make bespectacled", "add smile", among others, surprisingly well - sometimes even matching or outperforming the state-of-the-art. This is particularly unexpected as DFI requires no specialized network architecture or even any deep network to be trained for these tasks. DFI therefore can be used as a new baseline to evaluate more complex algorithms and provides a practical answer to the question of which image transformation tasks are still challenging in the rise of deep learning.
1. Introduction
DFI argues that many challenging image transformations can be achieved by linearly interpolating features in a suitable pre-trained convolutional representation. It offers a simple, high-resolution approach that often performs well, while depending on aligned and otherwise similar reference images.
- Motivation: Recent image-transformation methods use supervised conditional or adversarial generators, but many such tasks may instead be solvable through linear interpolation in deep feature space.This challenges the need for specialized architectures for a broad set of transformations.
- Motivation: Deep convolutional networks map images into representations where classes can be linearly separable, motivating interpolation between images with and without a target attribute.The proposed attribute direction is formed from differences between representative feature means.
- Method: DFI averages convolutional features from images with and without an attribute, then uses their difference to transform a target image.The method is called Deep Feature Interpolation and is illustrated on a 390 × 504 facial image.
- Limitations: DFI works best when images are aligned and when attribute-reference images are otherwise similar to the target.For example, reference images may need matching demographic or visual characteristics.
- Contribution: DFI does not require convnet retraining, task specialization, or specialized deep architectures, and it can process higher-resolution images than many alternatives.The authors report that it is often faster and more versatile than prior work.
2. Related Work
DFI is positioned against feature-based generative methods, latent-space editing, and complex face-transfer pipelines. Its distinguishing claim is that discriminatively trained feature spaces can support simpler, broader image-content transformations.
- Feature-based generative methods: The closest compared methods use deep feature spaces for data-driven attribute transformations but rely on specially trained generative auto-encoders.DFI differs fundamentally by not learning a specialized image-transformation generator.
- Content editing: Unlike latent-space editing methods, DFI edits existing image content while seeking to preserve photo-realism and unrelated content.The comparison distinguishes content editing from reconstructing imagery or transferring artistic style.
- Face-transfer pipelines: DFI is less complicated than face puppetry pipelines that combine fiducial detection, 3D reconstruction, and optical flow, and it applies beyond faces.The authors mention product images of shoes as another domain.
- Positioning: The authors present DFI as surprisingly powerful and effective, while acknowledging that it does not cover every case addressed by prior techniques.They call for further investigation into why the simple approach works so well.
3. Deep Feature Interpolation
DFI transforms images by estimating an attribute direction from averaged deep features, moving a test image along that direction, and reconstructing the result in pixel space. The procedure uses a pretrained convnet and gradient-based reverse mapping, with alignment and smoothness considerations for practical effectiveness.
- Inputs and representation: DFI assumes a test image, target images with the desired attribute, source images without it, and a pretrained convnet that provides deep feature representations.The representation is formed from concatenated convnet activations, such as those from VGG trained on ImageNet.
- Attribute direction: The method maps target and source images into deep feature space, averages each set, and subtracts the source mean from the target mean to obtain the attribute vector w.This difference is intended to isolate the targeted transformation while suppressing unrelated visual features through averaging.
- Feature transformation: DFI moves the test representation from φ(x) to φ(x) + αw, thereby applying a controllable displacement along the estimated attribute direction.The scalar α controls the displacement in deep feature space.
- Effectiveness: Although DFI uses a simple procedure, the paper reports that it can be surprisingly effective for image transformations.The effectiveness claim is presented as a result of the procedure described in this section.
- Neighbor selection: The attribute vector can be estimated from selected target neighbors, chosen using matching attributes when labels exist or cosine distance in deep feature space otherwise.The method defines K nearest neighbors of the target set relative to the test image representation.
- Reverse mapping: The transformed feature is reverse-mapped by finding a pixel-space image z whose deep representation approximates φ(x) + αw, using gradient descent because VGG has no closed-form inverse.The reconstruction includes a Total Variation regularizer that encourages smooth transitions between neighboring pixels; experiments use L-BFGS with λVβ = 0.001 and β = 2.
4. Experimental Results
DFI is evaluated across face-attribute editing, high-resolution aging and facial-hair changes, and attribute-free inpainting. Results show strong identity and attribute preservation in many cases, while alignment, coverage, and parameter choices remain important constraints.
- Changing Face Attributes: DFI uses LFW attribute annotations to test six face modifications on 38 male test images, with 100 nearest neighbors forming each source and target set.The tested attributes are SENIOR, MOUTH SLIGHTLY OPEN, EYES OPEN, SMILING, MOUSTACHE, and EYEGLASSES.
- Changing Face Attributes: In Figure 3, DFI often adds the requested facial attributes while preserving identity better than AEGAN.The comparison evaluates target-attribute expression down columns and identity preservation across rows.
- Changing Face Attributes: The perceptual study compares DFI and AEGAN through forced-choice judgments on six attribute changes across 38 test images.Workers received a tutorial and selected the image best expressing the target attribute while preserving the original identity; the study collected an average of 29.6 judgments.
- High Resolution Aging and Facial Hair: DFI processes high-resolution face images for aging and facial-hair editing, using a 100,000-image database, attribute classifiers, and test-time alignment.The paper reports megapixel-face results for aging and adding facial hair on three faces.
- Inpainting Without Attributes: For inpainting, DFI creates masked-to-unmasked source-target pairs and selects K =100 nearest neighbors without requiring supervised attributes.On faces, filled regions match skin tone, wrinkles, gender, and pose; on shoes, they match color and style, but some results show ghosting or fail to produce eyeglasses.
- Inpainting Without Attributes: DFI performs worse than disCVAE when half of the image is missing, despite producing plausible results when central face regions are missing.Shoe silhouette ghosting is attributed to imperfect dataset alignment.
- Varying the free parameters: Increasing β strengthens transformations but can cause ghosting at low values or unnatural reconstructions at high values, while K trades off variety against subclass mixing.The authors choose β and K that work across images and tasks rather than per case.
5. Discussion
The discussion presents DFI as a simple, effective baseline for many aligned image transformations, while emphasizing requirements, computational costs, and tasks where it fails. It also argues that DFI’s results challenge assumptions about the necessity of specialized generative architectures and motivate higher-resolution transformation research.
- Requirements and limitations: DFI works best with aligned images and source examples otherwise similar to the target.The method uses differences of means to cancel features unrelated to the desired attribute.
- High-resolution transformations: DFI can edit megapixel faces for aging and facial-hair addition, although high resolution makes artifacts more noticeable.The paper reports effectiveness on these tasks across three different faces.
- Time and space complexity: DFI’s resource use is linear, but a 1280 × 960 image requires 4 GB of GPU memory and 5 minutes to reconstruct.A 200 × 200 image takes 20 seconds to process, and test-time processing is slower than a trained model.
- Generative versus discriminative networks: The authors argue that discriminatively trained VGG-19 features can support content editing comparably to an autoencoder latent space.They suggest ImageNet’s richer recognition data may contribute, reporting that VGG-19 outperformed VGG-Face on face attribute changes.
- Linear interpolation as a baseline: DFI provides a simple baseline for testing whether image-editing tasks require complex generative machinery.Tasks easily solved by DFI may be too simple to showcase generative models.
- Hard tasks: The method has difficulty with half-image inpainting and cannot perform shape or rotation transformations because those tasks require aligned data.The paper also notes that generative models outperform DFI on many problems.
6. Conclusion
DFI performs surprisingly well across varied image-transformation tasks despite its simplicity, often producing higher-quality images than existing state-of-the-art methods.
- DFI produces high-quality images across varied transformations, including aging, adding facial hair, and inpainting.
- In many cases, DFI achieves higher image quality than existing state-of-the-art methods.
- DFI is proposed as a highly competitive baseline for certain image transformations on aligned data.
- The authors hope DFI’s performance will spur methods that outperform it.