Source-linked AI summary
GANalyze: Toward Visual Definitions of Cognitive Image Properties
Lore Goetschalckx, Alex Andonian, Aude Oliva, Phillip Isola
TL;DR
GANalyze tackles the lack of concrete visual definitions for cognitive image properties by navigating GAN latent spaces under an assessor’s guidance. It surfaces fine-grained memorability correlates, behavioral experiments support a causal effect on human memory, and the framework also applies to aesthetics and emotional valence.
Problem
Memorability, aesthetics, and emotional valence lack concrete visual definitions, while existing memorability predictors provide limited fine-grained visual explanations.
Method
GANalyze learns to navigate a GAN’s latent space so generated images move toward higher or lower assessor-predicted values of a target property.
Results
The visualizations surface overlooked memorability correlates, behavioral experiments show the manipulations affect human memory, and the framework extends to aesthetics and emotional valence.
Takeaways & Limitations
GANalyze offers visual definitions of otherwise hard-to-define cognitive image properties and can explore any property modeled as a differentiable image function.
Takeaways & Limitations
The method currently modifies only GAN-generated seed images with known latent encodings, not user-supplied real images.
Abstract
from arXiv · showhide
We introduce a framework that uses Generative Adversarial Networks (GANs) to study cognitive properties like memorability, aesthetics, and emotional valence. These attributes are of interest because we do not have a concrete visual definition of what they entail. What does it look like for a dog to be more or less memorable? GANs allow us to generate a manifold of natural-looking images with fine-grained differences in their visual attributes. By navigating this manifold in directions that increase memorability, we can visualize what it looks like for a particular generated image to become more or less memorable. The resulting ``visual definitions" surface image properties (like ``object size") that may underlie memorability. Through behavioral experiments, we verify that our method indeed discovers image manipulations that causally affect human memory performance. We further demonstrate that the same framework can be used to analyze image aesthetics and emotional valence. Visit the GANalyze website at http://ganalyze.csail.mit.edu/.
1 Introduction
GANalyze addresses the lack of fine-grained visual definitions for cognitive image properties by navigating GAN-generated images toward higher or lower predicted values. It surfaces candidate visual correlates of memorability, validates their behavioral effect, and generalizes the framework to aesthetics and emotional valence.
- 1 Introduction: GANalyze uses GANs to study visual features underlying memorability, aesthetics, and emotional valence, which lack concrete visual definitions.Prior memorability predictors provide predictions but not fine-grained visual explanations.
- 1 Introduction: The method navigates GAN latent space to generate fine-grained image sequences with increasing or decreasing predicted memorability.Sequences from diverse seed images form a visual definition of memorability.
- 1 Introduction: The visualizations surface overlooked memorability correlates, including object size, circularity, and colorfulness.Class-conditional BigGAN changes can remain orthogonal to object class while producing substantial predicted memorability differences.
- 1 Introduction: Behavioral experiments found that moving toward images predicted as more memorable produced images that were measurably more memorable to people.This tests whether the learned manipulations affect human memory rather than only the assessor’s predictions.
- 1 Introduction: The framework provides visual definitions for image properties that are measurable but difficult to define verbally.The paper also demonstrates applications to aesthetics and emotional valence.
2 Model
The model transforms GAN latent vectors along a learned direction so an assessor’s predicted image property changes by a controllable amount. A pretrained generator and assessor support training, testing, and substitution for other properties.
- 2 Model: A Transformer moves latent vector z along learned direction θ before a Generator produces the modified image.The input also includes a class vector y, allowing class-conditional generation.
- 2 Model: The training objective minimizes mean squared error between the transformed image’s assessor score and the seed score shifted by α.The scalar α specifies the desired increase or decrease in the property value.
- 2 Model: When α = 0, the Transformer performs a null operation and the generated image remains unchanged.This provides the original image as the zero-change condition.
- 2 Model: The implementation uses BigGAN as Generator and MemNet as memorability Assessor, while either module can be substituted.The paper notes that alternative assessors support properties such as aesthetics.
- 2 Model: Training samples 400K latent vectors with α uniformly drawn from -0.5 to 0.5 and randomly chosen classes.The test set contains 1.5K five-image sets, totaling 7.5K images, with α values from -0.2 to 0.2.
3 Experiments
The experiments show that GANalyze can steer generated images along memorability-related dimensions while preserving perceptual realness, and that these changes affect human memory performance. Visualizations reveal candidate factors including object size, centering, shape, simplicity, eyes, and color.
- 3.1 Model validation: β = 0.68, 95%CI = [0.66, 0.70], p < 0.001: MemNet scores increased significantly as α increased.This confirms that the model learned to navigate the latent space toward higher or lower predicted memorability.
- 3.2 Emerging factors: Increasing α commonly enlarged and centered subjects, promoted rounder or squarer shapes, and simplified cluttered scenes.These candidate factors emerged from qualitative visualization of the generated image sequences.
- 3.2 Emerging factors: Positive α often produced brighter, more colorful images and sometimes more pronounced eyes, while negative α produced darker, duller images.The model also sometimes preferred red hues when multiple realistic object colors were available.
- 3.2 Emerging factors: The emerging factors were quantified using pixel statistics, color metrics, entropy, and segmentation-mask measurements of size, centeredness, and squareness.Factor scores were normalized and averaged per α condition for the model-results analysis.
- 3.3 Realness: FID remained below 40 across all α conditions, indicating that memorability modifications were not explained by making images generally unrealistic.Human fakeness-discrimination performance also did not differ significantly with α (β = −0.08, 95%CI = [−0.33, 0.18], p = 0.55).
- 3.4.1 Emerging factors: Emerging factors generally had positive associations with memory hits, but no single factor fit the data as well as α.Because the factors were intercorrelated and correlational, the individual factors were difficult to interpret as causal drivers.
4 Other properties
GANalyze generalizes beyond memorability by substituting the Assessor module to visualize aesthetics and emotional valence. The resulting models increase or decrease their respective predicted scores across generated image sequences.
- 4 Other properties: Replacing the Assessor module lets GANalyze analyze image aesthetics and emotional valence in addition to memorability.Aesthetics uses AestheticsNet, while emotional valence uses EmoNet trained on the Cornell Emotion6 Image Database.
- 4 Other properties: 0.72: AestheticsNet scores increased significantly with α, showing that the model can increase or decrease an image’s predicted aesthetic score.The regression reported β = 0.72, 95%CI = [0.70, 0.74], p < 0.001.
- 4 Other properties: Figure 9 visualizes generated image sequences moving toward higher or lower aesthetics and emotional valence scores from a shared seed image.The middle columns contain the original generated seeds, and the images’ Assessor scores appear in their top-left corners.
- 4 Other properties: 0.44: EmoNet scores increased significantly with α, demonstrating controllable changes along the emotional valence dimension.The regression reported β = 0.44, 95%CI = [0.43, 0.45], p < 0.001.
- 4 Other properties: The aesthetics model changes depth of field, color palette, and lighting, while decreasing emotional valence often averts the subject’s gaze from the camera.These observations suggest the respective Assessors are sensitive to those visual factors.
5 Conclusion
GANalyze uses GAN-based manipulation to visualize otherwise difficult-to-define cognitive image properties and validates memorability changes with a behavioral experiment. Its current operation is limited to GAN-generated seed images rather than user-supplied real images.
- 5 Conclusion: GANalyze visualizes what another model has learned about high-level image properties, producing candidate visual features associated with memorability.The framework can generalize to aesthetics or emotional valence by replacing the Assessor module.
- 5 Conclusion: A behavioral human memory experiment validated that the model’s manipulations successfully made GAN images more or less memorable.
- 5 Conclusion: GANalyze currently modifies only GAN images with known latent-space encodings, not user-supplied real images.An encoder network could make modification of real images possible in future applications.