PaperlayerView exact revision on arXiv

Source-linked AI summary

The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement

William Peebles, John Peebles, Jun-Yan Zhu, Alexei Efros, Antonio Torralba

arXiv:2008.10599v1cs.CVcs.GRcs.LGcs.NE

TL;DR

The paper asks what it means to disentangle a function. The Hessian Penalty minimizes off-diagonal Hessian entries to encourage independent output changes when a single input component is perturbed. Across several datasets, it yields smoother, more disentangled outputs, shrinks overparameterized latent spaces, and uncovers interpretable BigGAN directions.

  • Problem

    The paper asks what it means to disentangle a function.

    Source: §1 · p. 1

  • Method

    The Hessian Penalty minimizes off-diagonal Hessian entries to encourage independent output changes when a single input component is perturbed.

    Source: §1 · p. 1

  • Results

    Across several datasets, it yields smoother, more disentangled outputs, shrinks overparameterized latent spaces, and uncovers interpretable BigGAN directions.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    On synthetic CLEVR datasets, the method can uncover known variation factors while shrinking an overparameterized latent space.

    Source: §5 · p. 14

    The weak prior cannot guarantee perfect disentanglement; on CLEVR-Complex, one component controls both objects' colors despite their independence in the dataset.

    Source: §5 · p. 14

Abstract

from arXiv · show

Existing disentanglement methods for deep generative models rely on hand-picked priors and complex encoder-based architectures. In this paper, we propose the Hessian Penalty, a simple regularization term that encourages the Hessian of a generative model with respect to its input to be diagonal. We introduce a model-agnostic, unbiased stochastic approximation of this term based on Hutchinson's estimator to compute it efficiently during training. Our method can be applied to a wide range of deep generators with just a few lines of code. We show that training with the Hessian Penalty often causes axis-aligned disentanglement to emerge in latent space when applied to ProGAN on several datasets. Additionally, we use our regularization term to identify interpretable directions in BigGAN's latent space in an unsupervised fashion. Finally, we provide empirical evidence that the Hessian Penalty encourages substantial shrinkage when applied to over-parameterized latent spaces.

1 Introduction

Many recent disentanglement methods are specific to their architectures and dataset variations. The paper proposes a model-agnostic Hessian Penalty and reports smoother, more disentangled outputs alongside latent-space shrinkage.

1 Introduction

  • Many recent disentanglement efforts are specific to the network architecture and dataset variations, motivating a simple, general alternative.The introduction contrasts these efforts with the proposed method.Source: §1 · p. 1
  • The Hessian Penalty minimizes off-diagonal Hessian entries so changes from perturbing one input component are independent of other components.The method estimates the Hessian using finite differences, is model-agnostic, and requires no auxiliary encoder.Source: §1 · p. 1
  • Across several datasets, the Hessian Penalty makes generative outputs smoother and more disentangled, and induces latent-space shrinkage compared with baseline models.The experiments apply it to BigGAN and ProGAN on ImageNet, Zappos50K, and CLEVR.Source: §1 · p. 2

2 Related Work

Related work regularizes derivatives in deep networks and pursues disentanglement through several VAE and GAN strategies. The paper also connects its Hessian prior to ICA and latent-space direction discovery.

Derivative-based regularization

  • Derivative-based regularizers target robustness, Jacobian properties, mode collapse, disentanglement, or discriminator gradients in deep networks.Examples include Hessian-eigenvalue regularization, Jacobian regularizers [36] [43] [38], and the GAN gradient penalty.Source: §2 · p. 2

Disentanglement in generative models

  • Generative-model disentanglement work spans VAE extensions and GAN approaches using mutual information, latent swapping and mixing, or architectural and 3D priors.Other work identifies disentangled channels in GAN activations and uses them for semantic edits.Source: §2 · p. 2

Latent directions and ICA

  • Latent-space control includes semantic vector arithmetic, user-provided transformations [18], and unsupervised direction discovery with a classifier.The Hessian Penalty is framed as a prior on mixing functions that biases them toward a diagonal Hessian, connecting it to ICA.Source: §2 · pp. 2–3

3 Method

The method penalizes off-diagonal Hessian terms to encourage input components to affect output changes independently, extending this idea to vector-valued functions and intermediate feature spaces. It uses stochastic directional derivatives and finite differences to make the penalty practical, then applies it to GAN generators.

3.1 Formulation

  • Minimizing squared off-diagonal Hessian terms encourages each input component to change the output independently of the other components.The formulation applies to scalar-valued functions and is motivated by the effect of one input perturbation on the output.Source: §3.1 · p. 3

3.2 Generalization to Vector-valued Functions

  • For vector-valued functions, the penalty applies the scalar formulation to each output component's Hessian and takes the maximum across components.The authors report that this maximum can impose stronger regularization than a mean in some instances, but alternatives were not thoroughly explored.Source: §3.2 · p. 4

3.3 The Hessian Penalty in Practice

  • An unbiased stochastic estimate uses the empirical variance of directional second derivatives from Rademacher vectors, typically sampling just two vectors.When the estimator equals the squared off-diagonal formulation, minimizing it is equivalent to minimizing the sum of squared off-diagonal Hessian entries; the result is attributed to Hutchinson.Source: §3.3 · pp. 4–5
  • A second-order central finite difference estimates the directional derivative, with ϵ = 0.1 in practice, and supports functions whose analytic Hessians are zero.The authors note this includes piece-wise linear neural networks; the PyTorch implementation is about seven lines and can be inserted into most codebases.Source: §3.3 · p. 5
  • The penalty can be applied with respect to the initial input or to intermediate feature spaces and downstream generator activations.In most experiments, the authors optimize penalties on several intermediate activations with respect to the initial input to achieve a stronger regularization effect.Source: §3.3 · p. 5

3.4 Applications in Deep Generative Models

  • In the GAN application, the discriminator objective stays unchanged while the generator loss combines the standard adversarial loss with a weighted Hessian Penalty.The weight λ balances the two terms.Source: §3.4 · pp. 5–6
  • Fine-tuning a pretrained ProGAN with the Hessian Penalty often works as well as or better than training from scratch, and disentangles the edge-to-shoe transition at z6 = 0.After fine-tuning, positive z6 values change shoe style while preserving structure; the baseline transition is inconsistent.Source: §3.4 · p. 6, Figure 2 · p. 6

4 Experiments

The experiments test the Hessian Penalty on ProGAN and BigGAN, finding disentangled latent factors, component shrinkage, and interpretable BigGAN directions. They also examine smoothness and whether the penalty strengthens the Hessian’s diagonal in practice.

4.1 ProGAN with Hessian Penalty

  • On CLEVR-Simple, the method uncovers all major factors of variation; on CLEVR-Complex, fine-tuned z11 mostly controls the left object’s vertical position, with residual changes.In CLEVR-Complex, the method separates object control better than the baseline; its z7 controls both objects’ colors, which are independent factors.Source: CLEVR · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 9

4.2 Overparameterized Latent Spaces

  • On CLEVR-1FOV, the Hessian Penalty effectively deactivates all but one latent component, while on CLEVR-Simple six of 12 components have virtually no output control.Activeness is the mean variance of G(z) as one component changes while the others remain fixed.Source: §4.2 · p. 8, Figure 5 · p. 10
  • With an underparameterized three-dimensional latent space on four-factor CLEVR-Simple, fine-tuning makes each component control the output more smoothly, though full disentanglement is impossible.The paper assesses smoothness quantitatively in the following section.Source: §4.2 · p. 9

4.3 Quantitative Evaluation of Disentanglement

  • The Hessian Penalty attains substantially better Perceptual Path Lengths than other methods across datasets, though image quality sometimes degrades, especially early in training or fine-tuning.Table 1 compares PPL and FID for ProGAN-based methods; the supplied passages do not give its cell values.Source: §4.3 · pp. 10–11, Table 1 · p. 11

4.4 Unsupervised Learning of Directions in BigGAN

  • The learned Hessian Penalty directions are not fully disentangled: extreme Golden Retriever zooms can substantially change the background.The learned directions are orthonormal, with edits shown over the specified η ranges.Source: Figure 6 · p. 12
  • On ImageNet-conditioned BigGAN, the method discovers rotation, zooming, and colorization directions; its rotation direction preserves alignment better than z11 and the Voynov–Babenko direction.Some native BigGAN components control lighting, color filtering, and background removal better than the learned directions.Source: §4.4 · p. 12, Figure 6 · p. 12

4.5 What is the Hessian Penalty Actually Doing?

  • Across the tested ProGAN experiments, the penalty strengthens the diagonal of pixel-level Hessians under both reported measures, except on underparameterized CLEVR-U.Pixel Hessians are estimated using second-order finite differences; the training penalty is applied to intermediate activations rather than pixels directly.Source: §4.5 · p. 13, Table 2 · p. 14

5 Discussion

The discussion summarizes the Hessian Penalty’s applications to ProGAN and BigGAN, while noting that its weak prior has important failure modes and can affect image quality.

5 Discussion

  • The Hessian Penalty uncovers factors in ProGAN and BigGAN and can shrink overparameterized latent spaces.Examples include edges↔shoes in Edges+Shoes, known factors in synthetic CLEVR, and rotation, colorization, and zooming in BigGAN.Source: §5 · p. 14
  • As a weak prior, the Hessian Penalty cannot be expected to produce perfect disentanglement; on CLEVR-Complex, one component controls both objects’ colors.Fine-tuning can also degrade image quality, and applying the penalty only at early layers can yield a degenerate solution without disentanglement or latent-space shrinkage.Source: §5 · p. 14

A Implementation Details

The appendix introduces experiment-specific implementation details in the following subsections.

A Implementation Details

  • The appendix’s implementation details are organized by experiment in the sections that follow.The passage introduces those details but does not specify their contents.Source: A Implementation Details · p. 18

A.1 ProGAN Experiments

For ProGAN experiments, the appendix describes a gradual Hessian Penalty warm-up for fine-tuning, reports early image-quality and metric changes, and lists implementation and baseline settings.

A.1 ProGAN Experiments

  • Without a warm-up, the additional regularization can sometimes cause the generator to collapse; training from scratch does not require this warm-up.For fine-tuning, the loss weight is ramped over T gradient steps; T = 10^5 or T = 10^6 tended to work across datasets.Source: §A.1 · p. 18
  • At the start of fine-tuning, image quality and FID commonly worsen while PPL decreases, usually followed by a quick recovery.Increasing the warm-up duration decreases the FID and PPL spikes.Source: §A.1 · p. 18
  • The experiments regularize the first 10 convolutions after pixel normalization, use ϵ = 0.1 and k = 2 Rademacher vectors, and retain pretrained discriminator weights during fine-tuning.Source: §A.1 · p. 18
  • The penalty weight is λ = 0.1 for Edges+Shoes, CLEVR-Simple, and CLEVR-Complex, but λ = 0.025 for CLEVR-1FOV and CLEVR-U to limit early image-quality degradation.The InfoGAN baseline adds a discriminator head trained with mean squared error at weight 1.0 and, for CLEVR, regularizes only the first c latent components.Source: §A.1 · p. 18

A.2 BigGAN Experiments

The appendix describes how BigGAN directions are learned with the Hessian Penalty and notes a potential degenerate solution when regularizing intermediate activations after linear operations.

A.2 BigGAN Experiments

  • BigGAN directions are learned by optimizing the Hessian Penalty on G(z + ηAwi) with respect to wi, using sampled η and a one-hot wi.The direction matrix A is kept orthogonal with Gram–Schmidt at each forward pass; intermediate-activation regularization yielded no significant difference from pixel-space regularization.Source: §A.2 · pp. 18–19
  • The BigGAN experiments use a mean reduction for the Hessian Penalty, a choice the authors did not ablate, and Adam with default parameters at learning rate 0.01.Source: §A.2 · p. 19

A.3 Avoiding Degenerate Solutions

  • Applying the penalty after a linear operation in intermediate activations can reduce filter norms without disentangling; applying it after normalization layers helps avoid this potential degeneracy.This issue is specific to intermediate activations; regularizing immediately after a linear function is fine when it is the network output.Source: §A.3 · p. 19

B Proof of Theorem 1

The proof uses a finite sample of Rademacher vectors to estimate a variance and argues that mismatched index pairs have zero covariance because their products are independent.

B Proof of Theorem 1

  • L_H is defined as Var(x⊺Hx)/2, and the proof says to estimate this variance with finitely many Rademacher vectors.It then sets out to prove that L_H is an unbiased estimator of P.Source: B Proof of Theorem 1 · p. 19
  • When the index pairs differ, the products x_i x_j and x_k x_ℓ are independent, so their covariance is zero.The proof explains that the second product contains a random variable absent from the first, which randomizes its sign independently.Source: B Proof of Theorem 1 · p. 20
  • The proof’s independence argument relies on the entries of x being independent and taking values ±1.An extra variable in the second product makes its sign independent of the first product’s value.Source: B Proof of Theorem 1 · p. 20

C Visualizing Pixel Hessians

The section visualizes pixel-level Hessian matrices for sampled images across datasets and methods, including baseline and Hessian-Penalty ProGAN variants.

C Visualizing Pixel Hessians

  • For each dataset, the visualization samples six images per method and displays the Hessians of the pixels with the largest Hessian Penalty in each image.Each Hessian is computed for one scalar output pixel using second-order finite differences.Source: C Visualizing Pixel Hessians · p. 20
  • The 12 × 12 Hessian visualizations cover CLEVR-1FOV, CLEVR-Simple, Edges+Shoes, and CLEVR-Complex, comparing baseline ProGAN and InfoGAN with Hessian-Penalty training variants.The captions identify training from scratch and fine-tuning with the Hessian Penalty among the compared methods.Source: Figure 8 · p. 21, Figure 9 · p. 21, Figure 10 · p. 22
  • CLEVR-U is shown separately with 3×3 Hessians, comparing baseline ProGAN, InfoGAN, and Hessian-Penalty training variants.The caption includes training from scratch and fine-tuning with the Hessian Penalty.Source: Figure 12 · p. 23

D Hessian Penalty Heatmaps

The section overlays pixel-space Hessian-Penalty values on images and includes a visualization for Edges+Shoes models.

D Hessian Penalty Heatmaps

  • The heatmaps overlay per-pixel Hessian-Penalty values on each image.The values are computed in pixel space.Source: D Hessian Penalty Heatmaps · p. 23
  • The visualized quantity is the Hessian Penalty at each pixel, rather than a latent-space value.The passage specifies that these penalties are computed in pixel space.Source: D Hessian Penalty Heatmaps · p. 23
  • The per-pixel Hessian-Penalty visualization covers Edges+Shoes models.The figure caption identifies the displayed values as per-pixel Hessian Penalties.Source: Figure 13 · p. 23

Related papers

Loading 2008.10599v1…