PaperlayerView exact revision on arXiv

Source-linked AI summary

Equivariant Finite Normalizing Flows

Avishek Joey Bose, Marcus Brubaker, Ivan Kobyzev

arXiv:2110.08649v2cs.LGcs.AI

TL;DR

Equivariant continuous flows face numerical challenges that complicate their application to higher-dimensional generative modeling. This paper develops finite equivariant flows, proves existence and representational results, and reports experiments highlighting EFNF benefits.

  • Problem

    Building equivariant finite layers is challenging because equivariance and invertibility can be incompatible, and the resulting flow may not be universal.

    Source: §1 · p. 2

  • Method

    The paper proves equivariant-map existence under compactness conditions and builds finite equivariant flows from G-residual and G-coupling layers.

    Source: §1 · p. 2

  • Results

    The paper proves compact-case existence and exact G-residual representation of any G-equivariant diffeomorphism on R^n, and reports experiments highlighting benefits on synthetic and image datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    These results lay foundations for modeling invariant densities with equivariant finite flows.

    Source: §7 · p. 11

    G-coupling requires permutation representations, restricting its group to finite G and excluding compact groups such as SO(2).

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Generative modeling seeks to uncover the underlying factors that give rise to observed data that can often be modeled as the natural symmetries that manifest themselves through invariances and equivariances to certain transformation laws. However, current approaches to representing these symmetries are couched in the formalism of continuous normalizing flows that require the construction of equivariant vector fields -- inhibiting their simple application to conventional higher dimensional generative modelling domains like natural images. In this paper, we focus on building equivariant normalizing flows using discrete layers. We first theoretically prove the existence of an equivariant map for compact groups whose actions are on compact spaces. We further introduce three new equivariant flows: $G$-Residual Flows, $G$-Coupling Flows, and $G$-Inverse Autoregressive Flows that elevate classical Residual, Coupling, and Inverse Autoregressive Flows with equivariant maps to a prescribed group $G$. Our construction of $G$-Residual Flows are also universal, in the sense that we prove an $G$-equivariant diffeomorphism can be exactly mapped by a $G$-residual flow. Finally, we complement our theoretical insights with demonstrative experiments -- for the first time -- on image datasets like CIFAR-10 and show $G$-Equivariant Finite Normalizing flows lead to increased data efficiency, faster convergence, and improved likelihood estimates.

1 Introduction

Symmetries provide useful inductive biases, but continuous equivariant flows pose practical challenges for higher-dimensional generative modeling. This paper develops finite equivariant flows and establishes theoretical results about their existence and representational power.

1 Introduction

  • Equivariant architectures encode symmetries as inductive biases, which are associated with parameter efficiency, faster convergence, and data-independent generalization in supervised tasks.The paper motivates generative modeling by these benefits of equivariant models.Source: §1 · p. 1
  • Continuous equivariant flows require equivariant vector fields and can involve many ODE evaluations, noisy backward gradients, and dimension-dependent trace-estimator variance.These numerical challenges complicate application to higher-dimensional image density estimation.Source: §1 · pp. 1–2
  • G-residual and G-affine coupling layers build invertible equivariant finite flows, with coupling constructions yielding equivariant RealNVP and Inverse Autoregressive Flow variants.The paper frames these layers as finite-flow alternatives to continuous normalizing flows.Source: §1 · p. 2
  • For compact groups acting on compact spaces, the paper proves the existence of an equivariant diffeomorphism between invariant measures.The introduction states this result for compact groups and the compact spaces on which they act.Source: §1 · p. 2
  • Any G-equivariant diffeomorphism on R^n can be exactly represented by a G-residual flow operating on an augmented space.The paper contrasts this universality with G-coupling flows, which cannot be applied to all input data types.Source: §1 · p. 2
  • Experiments cover synthetic invariant densities, Rotation MNIST, and augmented CIFAR10, and highlight benefits of equivariant finite normalizing flows.The introduction identifies both synthetic and image datasets as experimental settings.Source: §1 · p. 2

2 Background and Preliminaries

The background defines equivariance and steerable convolutions, then reviews normalizing flows and the finite-flow architectures used as building blocks. These include affine coupling, autoregressive, and residual approaches.

2.1 Equivariance and Linear Steerability

  • A map is G-equivariant when it respects the group action, and an image can be treated as a function transformed by [g · f](x) = f(g^-1x).A steerable convolution is G-equivariant, with its kernel built using irreducible representations of G.Source: §2.1 · pp. 2–3

2.2 Normalizing Flows

  • A normalizing flow transforms a base-distribution sample through a diffeomorphism whose Jacobian determinant and inverse are efficiently computable.The density after transformation follows from the chain rule.Source: §2.2 · p. 3
  • Discrete flows compose finitely many elementary diffeomorphisms, and this paper focuses on affine flows for their simplicity and expressiveness.Affine coupling flows are one such construction.Source: §2.2 · p. 3
  • RealNVP coupling transforms one input partition using a function of the other, while the inverse remains a coupling layer.The paper denotes the full coupling operation as the concatenation of an identity operation on the first partition and a coupling operation on the second.Source: §2.2 · p. 3
  • Inverse Autoregressive Flow transforms the input over feature indices, producing a lower-triangular Jacobian computable in linear time.Each index depends on all previous indices.Source: §2.2 · p. 3
  • Residual normalizing flows use residual networks as an Euler discretization of an ODE, with invertibility ensured when each residual function has Lipschitz constant below one.The inverse can be obtained by fixed-point iteration, and the Jacobian determinant can be estimated efficiently.Source: §2.2 · pp. 3–4

3 Existence of Equivariant Diffeomorphisms

This section establishes conditions for equivariant maps to push one invariant density to another. It proves existence under compactness assumptions and sketches an equivariant modification of Moser’s trick.

3 Existence of Equivariant Diffeomorphisms

  • If a flow map is G-equivariant and its base density is G-invariant, the resulting density is also G-invariant.An invariant base distribution, such as a standard normal under rotations about the origin and reflections, supports this construction.Source: §3 · p. 4
  • Theorem 2 guarantees a G-equivariant diffeomorphism pushing one invariant volume form to another for a compact group acting smoothly on a connected, compact, orientable, closed manifold.The volume forms represent the given orientation.Source: §3 · p. 4
  • The proof sketch adapts Moser’s trick by connecting the volume forms through a segment and choosing an invariant form η.The invariant time-dependent vector field generates an isotopy whose integration yields the required equivariant diffeomorphism.Source: §3 · p. 4
  • The existence result supports modeling invariant densities with equivariant diffeomorphisms, and the paper states that G-residual flows can represent such maps exactly on an augmented space.The exact representation claim is made for the G-residual flow in the later representational-capabilities discussion.Source: §3 · pp. 4–5

4 Constructing G-Equivariant Normalizing Flows

The section constructs discrete equivariant flows using representations suited to geometric data, then establishes their conditions and representational limits. It also describes equivariant linear maps and a matrix-exponential route to invertibility.

4.1 Exploiting Geometric Structure

  • Steerable-CNN representations treat R^2 as a base space and image channels as fields, expressing layer equivariance through constrained convolution kernels.Discretizing a planar circle into four quadrants makes its quarter-turn actions specific C4 permutations, not arbitrary permutations of the quadrants.Source: §4.1 · pp. 5–6

4.2 G-Residual Normalizing Flows

  • A residual layer is G-equivariant when its residual map is G-equivariant, and the Lipschitz condition Lip(h) < 1 makes the layer invertible.The composition of such layers remains equivariant.Source: §4.2 · p. 6

4.3 Representation Capabilities of G-Residual Flows

  • Any G-equivariant Lipschitz diffeomorphism on R^n can be exactly represented by a G-residual flow on an augmented R^2n space.The G-action is extended trivially to the padding; the theorem establishes representability, not ease of optimization under the required constraints.Source: §4.3 · p. 7

4.4 G-Coupling Normalizing Flows

  • For coupling flows, equivariance follows when s and t are G-equivariant and the representation is a completely reducible diagonal permutation representation.The representation must use permutation matrices; consequently, G must be finite, excluding SO(2), though finite subgroups of E(2) are applicable.Source: §4.4 · pp. 7–8

4.5 G-Inverse Autoregressive Flows

  • A G-equivariant IAF results when its k blocks use a completely reducible diagonal permutation representation and its scale and translation functions are G-equivariant.The representation has n = k · d, with a d × d permutation matrix for each block.Source: §4.5 · p. 8

4.6 Invertible Equivariant Linear Maps

  • Equivariant linear maps between induced representations can be expressed using constrained convolution-like kernels, while exponentiating an equivariant kernel yields an invertible equivariant map.The matrix-exponential construction uses a numerically approximated truncated power series.Source: §4.6 · p. 9

5 Related Work

Earlier equivariant-flow work spans continuous normalizing flows, Hamiltonian methods, coupling transforms for permutation-invariant data, and approaches for Euclidean and manifold symmetries.

5 Related Work

  • Continuous normalizing flows and Hamiltonian methods were concurrent early approaches to symmetry-aware flows; the latter uses Leapfrog integration for finite-time invertible solutions.The Hamiltonian approach is described as a specific continuous-time instantiation of the proposed G-coupling flow.Source: §5 · p. 9
  • Coupling-based flows also address permutation-invariant sets, including point-cloud datasets, while equivariant flows have been constructed for E(n).The cited set-flow models are described as instantiations of the G-coupling layer; the E(n) approach requires global equivariance and suits domains such as molecular dynamics.Source: §5 · pp. 9–10
  • Other work applies flow models to lattice gauge theory and SU(N), and constructs equivariant manifold flows for isometry groups.The manifold-flow work extends Euclidean-space equivariance results.Source: §5 · p. 10

6 Experiments

Experiments compare equivariant finite flows with non-equivariant flows trained using group augmentation, across toy distributions and image datasets. Results show parameter-efficiency gains for residual flows and faster convergence with lower average group test bits per dimension for coupling flows on augmented CIFAR-10.

6 Experiments

  • The image evaluation covered Rotation MNIST and CIFAR-10, using non-equivariant RealNVP-based coupling and residual flows trained with random group augmentation as baselines.CIFAR-10 was augmented because it has reflection symmetry but not rotation symmetry.Source: §6 · p. 10, §6.2 · p. 10

6.1 Synthetic Experiments

  • On toy data, equivariant residual flows better respected symmetries and produced sharper density transitions; D16 reduced smearing on Gaussian modes and better respected spiral symmetry.The toy experiments used 2k parameters per model and compared equivariant and non-equivariant residual flows.Source: §6.1 · p. 10, Figure 2 · p. 10

6.2 Image Datasets

  • Generated samples illustrate D8 rotations and reflections, C16 rotations, and translation symmetry through digits splicing across image boundaries.The translation group was implemented by treating inputs as a torus with circular padding.Source: Figure 3 · p. 10
  • All tested G-coupling flows converged faster and reached lower average group test bits/dim than non-equivariant coupling flows on augmented CIFAR-10.Increasing the constraint from C4 to C16 also improved test bits/dim in the reported ablations.Source: Figure 4 · p. 11, §6.2 · p. 11
  • G-residual flows used 227% and 195% greater parameter efficiency on Rotation MNIST and CIFAR-10, respectively, while producing slightly worse bits/dim.Table 1 reports bits per dimension and parameter counts; CIFAR-10 was augmented with random group transformations for comparison.Source: Table 1 · p. 11, §6.2 · p. 11

7 Conclusion

The paper establishes a theoretical basis for equivariant finite flows and introduces equivariant coupling and residual layers. Experiments report parameter efficiency for residual flows and faster convergence with lower average test bits per dimension for coupling flows, alongside limitations in applicability and scope.

7 Conclusion

  • Equivariant Finite Normalizing Flows use finite layers, including G-coupling and G-residual layers, to equip classical flows with equivariance to a prescribed group.The paper presents these layers as building blocks for equivariant diffeomorphisms on Euclidean spaces.Source: §7 · p. 11
  • The existence proof establishes an equivariant map for compact groups acting on compact spaces, while non-compact symmetric domains remain a direction for future work.The conclusion identifies the latter as an open setting for proving existence.Source: §7 · pp. 11–12
  • G-residual flows provide significant parameter efficiency but incur a small density-estimation performance drop on images.The conclusion contrasts this with G-coupling flows, which achieve faster convergence and lower average test bits/dim on augmented CIFAR-10.Source: §7 · p. 12
  • G-coupling flows are limited to domains with no more than a single channel, while their reported CIFAR-10 results use augmented data.The paper gives RGB images as an example beyond the stated channel limitation.Source: §7 · p. 12

A.1 Proof of Theorem 2.

The proof adapts Moser’s trick to construct an equivariant diffeomorphism between invariant volume forms. It interpolates between the forms, obtains an invariant vector field, and integrates that field on the compact manifold.

A.1 Proof of Theorem 2

  • The proof adapts Moser’s trick and uses the one-dimensional top cohomology group to express one volume form as the other plus an exact form.The proof states ν = µ + dη for an (n−1)-form η.Source: §A.1 · p. 15
  • Averaging η with the Haar measure of compact G makes the form G-invariant without changing the construction's required relation.The proof relies on the invariance of µ and ν and the naturality of the group action.Source: §A.1 · p. 15
  • The proof interpolates between µ and ν using µ_t = µ + t dη, then seeks an isotopy carrying the initial volume form to the interpolated form.Setting t = 1 yields the desired endpoint; compactness permits the isotopy to be generated by a time-dependent vector field.Source: §A.1 · p. 15
  • Solving pointwise for the vector field and integrating it on compact M yields a diffeomorphism; invariance of the forms makes the field and resulting diffeomorphism G-equivariant.The proof obtains a unique smooth vector field because µ_t is non-degenerate.Source: §A.1 · p. 15

B Representation Capabilities of G-Residual Layer

The construction represents any Lipschitz continuous diffeomorphism on R^n by a Residual flow on the padded space R^{2n}. When the diffeomorphism is G-equivariant, the constructed flow is also G-equivariant under the extended action.

B Representation Capabilities of G-Residual Layer

  • Any Lipschitz continuous diffeomorphism φ: R^n → R^n can be represented exactly on padded inputs by a Residual flow ψ: R^{2n} → R^{2n}, with ψ([x, 0]) = [φ(x), 0].The proof composes i-ResNets to obtain ψ.Source: B Representation Capabilities of… · pp. 15–16
  • The construction uses a sequence of i-ResNets whose residual parts have Lipschitz constant less than one, then composes them into a Residual flow.The proof defines δ(x) = φ(x) − x and uses the resulting padded-space maps in the composition.Source: B Representation Capabilities of… · pp. 15–16
  • If φ is G-equivariant, the constructed Residual flow is G-equivariant under the extended G-action on the padded space.The proof applies equivariance to ψ(g · [x, 0]) and obtains [φ(g · x), 0].Source: B Representation Capabilities of… · p. 16
Loading 2110.08649v2…