PaperlayerView exact revision on arXiv

Source-linked AI summary

SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds

Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, Nam Soo Kim

arXiv:2006.04604v4cs.CVcs.LG

TL;DR

Flow models struggle to estimate data distributions on lower-dimensional manifolds because their transformations require matching data and target dimensions. SoftFlow trains on conditional distributions of noise-perturbed data; experiments report high-quality manifold samples and state-of-the-art point-cloud generation. The authors identify noise-distribution selection and architecture as opportunities for further improvement.

  • Problem

    Normalizing flows are not suitable for estimating distributions on lower-dimensional manifolds when data and target dimensions differ.

    Source: §3 · p. 3

  • Method

    SoftFlow perturbs input data with noise from randomly selected distributions and estimates the conditional distribution of the perturbed data.

    Source: §4 · p. 4

  • Results

    SoftFlow captures manifold structure and produces high-quality samples, while SoftPointFlow describes object details more exactly and achieves state-of-the-art point-cloud generation performance.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Applied to 3D point clouds, SoftPointFlow addresses the difficulty of forming thin structures such as chair legs or wing tips.

    Source: §5 · p. 6, §5.1 · p. 7

    The framework could be further improved by identifying more useful noise distributions or finding architectures that leverage noise parameters efficiently.

    Source: §6 · p. 9

Abstract

from arXiv · show

Flow-based generative models are composed of invertible transformations between two random variables of the same dimension. Therefore, flow-based models cannot be adequately trained if the dimension of the data distribution does not match that of the underlying target distribution. In this paper, we propose SoftFlow, a probabilistic framework for training normalizing flows on manifolds. To sidestep the dimension mismatch problem, SoftFlow estimates a conditional distribution of the perturbed input data instead of learning the data distribution directly. We experimentally show that SoftFlow can capture the innate structure of the manifold data and generate high-quality samples unlike the conventional flow-based models. Furthermore, we apply the proposed framework to 3D point clouds to alleviate the difficulty of forming thin structures for flow-based models. The proposed model for 3D point clouds, namely SoftPointFlow, can estimate the distribution of various shapes more accurately and achieves state-of-the-art performance in point cloud generation.

1 Introduction

Flow-based models offer exact likelihood evaluation, efficient sampling, and structured latent spaces, but this paper targets their difficulty modeling manifold data. SoftFlow addresses that gap by estimating conditional distributions of perturbed data.

1 Introduction

  • Flow-based models are valued for exact log-likelihood evaluation, efficient synthetic-data generation, and well-structured latent spaces.The passage links these properties to modeling complex dependencies, real-time sampling, and latent spaces useful for downstream tasks or interpolation.Source: §1 · p. 1
  • Flow++ uses variational dequantization to provide a more natural and smoother density approximation for discrete data.The passage also introduces continuous normalizing flows, whose architectures are less restricted because their objective can use Hutchinson’s estimator.Source: §1 · p. 1
  • SoftFlow perturbs data with noise sampled from different distributions and estimates the conditional distribution of the perturbed data to train flows on manifolds.The authors report that it captures manifold structure and generates high-quality samples; SoftPointFlow applies the framework to 3D point clouds and achieves state-of-the-art generation performance.Source: §1 · p. 2

2 Flow-based generative model

Normalizing flows transform between latent and data distributions through invertible mappings and change-of-variables likelihoods. Continuous normalizing flows extend this formulation to time-varying latent variables, but their ODE solvers can make training slow.

2 Flow-based generative model

  • A normalizing flow maps a simple latent distribution to a complex data distribution through invertible transformations, with log-likelihood expressed using the change-of-variables theorem.The passage introduces transformations from z to x and describes z as the latent variable used to express the likelihood.Source: §2 · p. 2
  • Continuous normalizing flows use time-varying latent variables and an instantaneous change-of-variables formula for their log density.Their flexible architecture and efficient trace estimation come with substantial ODE-solver cost; one cited MNIST training run took five days on six GPUs.Source: §2 · p. 2

3 Normalizing flow on manifolds

Conventional normalizing flows struggle when data lie on a lower-dimensional manifold because their transformations cannot expand that data to fill a higher-dimensional target distribution. Directly formulating manifold likelihoods also brings inversion, determinant, and dimensionality challenges.

3 Normalizing flow on manifolds

  • When 1D manifold data lie in 2D space, FFJORD’s transformed latent distribution differs from its 2D target, illustrating the dimension-mismatch problem.The paper attributes the limitation to flow transformations being homeomorphisms and notes that the manifold data cannot expand to the target’s full 2D shape.Source: §3 · p. 3
  • A map from a lower-dimensional subset to a higher-dimensional manifold requires a manifold-volume change-of-variables formulation rather than the ordinary equal-dimension formula.The passage defines F from A ⊂ R^m to M ⊂ R^n for m < n and introduces the n × m Jacobian DF and its transpose.Source: §3 · p. 3
  • Figure 1 contrasts flow training on 2D data with training on 1D manifold data.Figure 2 illustrates a function mapping a subset of R^m to a manifold in R^n.Source: Figure 1 · p. 3, Figure 2 · p. 3
  • Directly designing flows from the manifold likelihood is difficult because inversion may fail, the Jacobian determinant can cost O(m^3), and the manifold dimension must be known.Without prior knowledge of m, the authors say a heuristic search may be needed.Source: §3 · p. 3

4 SoftFlow

SoftFlow trains normalizing flows on noise-perturbed data conditioned on noise parameters to address the dimension mismatch for manifold data. On artificial distributions, it generated samples that followed the data more closely than Glow and FFJORD.

4 SoftFlow

  • SoftFlow adds sampled noise to each input and trains a flow conditioned on the noise parameter, so perturbed data spans the full data-space dimensions.Training across distributions with different volumes teaches the flow the relation between distribution shape and noise parameters.Source: §4 · p. 4
  • After training, setting c_sp to a small value or zero yields realistic samples from the learned flow.The method is described as implementable within existing flow-based frameworks.Source: §4 · p. 4
  • The framework also extends to continuous normalizing flows by conditioning their dynamics on the noise parameter.The stated dynamics map from the latent variable to the perturbed input.Source: §4 · p. 4

4.1 Experiments on artificial data

  • SoftFlow generated high-quality samples that followed the artificial manifold distributions, while Glow and FFJORD produced poorer fits, especially for circles.The comparison covers five distributions, with c_sp set to zero for SoftFlow sampling.Source: Figure 4 · p. 5, §4.1 · p. 5
  • Varying c_sp from 0 to 0.1 produced samples faithful to different intended distributions and suggested potential for unseen or plausible synthetic distributions.Figure 5 shows examples sampled at different c_sp values.Source: §4.1 · p. 5, Figure 5 · p. 5

5 SoftPointFlow

SoftPointFlow applies SoftFlow to PointFlow to address difficulty modeling thin structures in 3D point clouds. It produces detailed point-cloud samples, is competitive with PointFlow on two categories, and shows robustness to latent-variance changes.

5 SoftPointFlow

  • SoftPointFlow models hierarchical distributions of shape and points, using PriorFlow for the shape latent and DecoderFlow for point likelihood conditioned on shape and noise.The shape latent is encoded from the point set; the paper replaces PointFlow's continuous normalizing flow networks with discrete flows because of slow convergence and to test the framework on discrete flows.Source: §5 · p. 6

5.1 Experiments on point clouds

  • SoftPointFlow captured fine object details, including thin structures such as chair legs and wing tips, while PointFlow produced blurry samples that failed to form them.Figure 7 shows reconstructed samples of seen and unseen data, as well as synthetic samples.Source: §5.1 · p. 7, Figure 7 · p. 7
  • Across varying latent variances, SoftPointFlow preserved the chair's X-shaped structure and whole form, whereas PointFlow missed the X shape or concentrated samples at the center.The authors conclude that SoftPointFlow learns more desirable features from manifold data and is robust to latent-variable variance.Source: §5.1 · p. 8
  • SoftPointFlow achieved significantly better 1-NNA results than GAN-based models in all categories, was competitive with PointFlow on airplane and chair, and scored slightly lower on car.The authors note that thin structures are less prevalent in the car data set; Table 1 reports 1-NNA, for which lower is better.Source: Table 1 · p. 8, §5.1 · p. 8

6 Conclusion

The conclusion presents SoftFlow as a framework for manifold normalizing flows and applies it to point-cloud generation. It identifies noise-distribution selection and efficient use of noise parameters as directions for improvement.

6 Conclusion

  • SoftFlow is introduced as a probabilistic framework for training normalizing flows on manifolds, capturing manifold structure and producing high-quality samples where current flow-based models cannot.Source: §6 · p. 8
  • Applied to point clouds, SoftPointFlow describes object details more exactly and achieves state-of-the-art performance in point-cloud generation.Source: §6 · pp. 8–9
  • The authors identify theoretical selection of a useful noise distribution and architectures that leverage noise parameters efficiently as possible improvements.Source: §6 · p. 9

Broader Impact

The broader-impact passage anticipates applications of the framework and cautions that improperly configured training data could produce biased samples. The appendix code includes constructions for 2-sines and target distributions.

Broader Impact

  • The proposed model could produce biased samples if the training set is not properly set up.Source: Broader Impact · p. 9

A Code for the 2-sines and target distributions

B.1 Reconstruction of seen data

Figure 10 presents reconstructed point clouds of seen data.

B.1 Reconstruction of seen data

B.2 Reconstruction of unseen data

Figure 11 presents reconstructed point clouds of unseen data, and Figure 12 presents synthetic point clouds generated by SoftPointFlow.

B.2 Reconstruction of unseen data

B.3 Generation

  • Figure 12 shows synthetic point clouds generated by SoftPointFlow.Source: Figure 12 · p. 14

C.1 Reconstruction of seen data

Figure 13 presents reconstructed point clouds of seen data.

C.1 Reconstruction of seen data

C.2 Reconstruction of unseen data

Figure 14 presents reconstructed point clouds of unseen data, and Figure 15 presents synthetic point clouds generated by SoftPointFlow.

C.2 Reconstruction of unseen data

C.3 Generation

  • Figure 15 shows synthetic point clouds generated by SoftPointFlow.Source: Figure 15 · p. 17
Loading 2006.04604v4…