PaperlayerView exact revision on arXiv

Source-linked AI summary

Bernoulli Flow Models: Self-Consistent Generative Modeling for Binary Data

Hao Mo, Liying Yang, Shumin Yao, Xinxing Yu, Ajian Liu, Xudong Mao, Yanyan Liang

arXiv:2610.11362v1cs.LGcs.CV

TL;DR

Binary diffusion models often require many sequential evaluations, and reducing them without additional training is difficult when one-step transitions cannot represent cross-step updates. BFM defines a continuous Bernoulli path with analytical posteriors over arbitrary intervals. On LSUN Churches, it achieves FID 9.22 at 16 sampling steps versus BLD's 204.10, and the paper reports competitive performance under standard inference.

  • Problem

    Binary generative models typically require hundreds or thousands of sequential evaluations, while training-free low-NFE sampling is difficult for models based on fixed one-step discrete transitions.

    Source: §1 · pp. 1–2

  • Method

    BFM defines a continuous Bernoulli probability path and derives closed-form marginal, transition, and posterior distributions for arbitrary intervals.

    Source: §1 · p. 2

  • Results

    FID 9.22 for BFM versus 204.10 for BLD at 16 inference steps on LSUN Churches, with BFM reported as competitive under standard full-step inference.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Changing the inference grid requires re-evaluating BFM's analytical posterior on new intervals rather than approximating skipped discrete steps.

    Source: §1 · p. 2, §5 · p. 9

    The framework assumes a uniform Bernoulli B(0.5) prior representing binary white noise.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Binary diffusion models typically require a large number of function evaluations (NFEs) to generate high-quality samples, making practical inference computationally expensive. Reducing NFEs while preserving sample quality without distillation or additional training remains a significant challenge. Existing binary diffusion models define a discrete one-step forward path and then derive the reverse posterior. In low-NFE settings requiring cross-step sampling, they approximate the true multi-step likelihood with a single-step likelihood transition, which severely degrades sample quality. To address this fundamental limitation and decouple the generative dynamics from fixed discrete time steps, we propose Bernoulli Flow Models (BFM). Rather than relying on sequential one-step Markov diffusion chains, BFM defines a unified continuous global Bernoulli probability flow path between data distributions and pure noise, from which we derive analytical closed-form posterior transitions over arbitrary time intervals. Consequently, reducing the inference NFE is no longer an approximation based on skipping discrete steps; it only requires re-evaluating the analytical posterior over a new time grid. This eliminates the structural training-inference mismatch inherent to discrete chains and yields self-consistent low-NFE sampling. Experiments show that BFM is highly robust to aggressive NFE reduction. On LSUN Churches 256x256, a BFM trained with 256 steps achieves an FID of 9.22 using only 16 sampling steps, whereas the state-of-the-art discrete baseline degrades to 204.10. BFM also remains competitive with continuous and discrete generative baselines under standard full-step inference. These results establish BFM as a theoretically rigorous, self-consistent, and practically effective framework for fast binary data generation.

1 Introduction

Binary diffusion often needs hundreds or thousands of sequential evaluations, while reducing evaluations without additional training is difficult because discrete one-step posteriors do not describe enlarged cross-step transitions. Bernoulli Flow Models define a continuous Bernoulli path and analytical arbitrary-interval posteriors, supporting self-consistent low-NFE sampling.

1 Introduction

  • Binary generative modeling spans image modeling, binarized latent representations, and statistical-physics systems with two-state configurations.Existing models typically require hundreds or thousands of sequential evaluations for high-fidelity samples.Source: §1 · p. 1
  • Coarsening a trained binary diffusion model's inference grid requires cross-step posteriors; reusing one-step transitions no longer describes the same generative process.The example given is reducing a 256-step model to 16 sampling steps.Source: §1 · p. 2
  • BFM defines a continuous Bernoulli probability path and derives closed-form marginal, transition, and posterior distributions over arbitrary time intervals.The path connects data and pure noise.Source: §1 · p. 2
  • When inference NFE is reduced after training, BFM recomputes its analytical posterior on the new intervals instead of approximating large jumps with local one-step rules.This approach is presented as self-consistent low-NFE sampling.Source: §1 · p. 2
  • FID 9.22 for BFM versus 204.10 for BLD at 16 inference steps from the same 256-step training setup on LSUN Churches.The reported BFM result uses the linear path scheduler.Source: §1 · p. 2

2 Related Work

Related work spans binary diffusion and flow approaches, training-free acceleration for continuous diffusion, and broader discrete or categorical generative models. The paper distinguishes its focus on arbitrary-interval Bernoulli posteriors for low-NFE inference from these lines of work.

Binary and Bit-Level Flow/Diffusion Models

  • Binary and bit-level methods have studied representation, training objectives, and binary-space generative quality, while self-consistent low-NFE inference from a fixed checkpoint remains underexplored.Examples include Analog Bits [4], Binary Latent Diffusion [32], and Binary Diffusion Probabilistic Models [18].Source: §2 · p. 3

Fast Sampling and Training-Free Acceleration

  • Continuous-domain training-free samplers evaluate reverse dynamics on coarser grids, but binary models need valid cross-step posterior updates over enlarged intervals.Binary diffusion cannot directly rely on continuous-domain ODE or SDE solvers when its reverse process uses discrete Bernoulli transitions.Source: §2 · p. 3

Relation to General Discrete Diffusion and Flow Models

  • General discrete diffusion and flow models provide broad foundations, whereas this work specializes in Bernoulli variables and closed-form arbitrary-interval posteriors.The stated target is self-consistent low-NFE inference from a single pretrained checkpoint.Source: §2 · p. 3

3 Bernoulli Flow Models

Bernoulli Flow Models define a continuous probability path between binary data and a uniform Bernoulli prior, then derive analytical distributions along that path. Their closed-form cross-step posterior allows sampling on a changed time grid by reparameterizing the path.

3.1 Notation and Problem Formulation

  • The Bernoulli path interpolates in parameter space from target data to a prior of uniform binary noise, with intermediate states following that path.The time parameter runs from data to prior.Source: §3.1 · p. 3

3.2 Bernoulli Probability Flow Dynamics

  • Assuming a uniform Bernoulli prior, BFM derives marginal, transition, and posterior distributions along the path and trains a model to predict uncorrupted data with binary cross-entropy.The model’s prediction is aligned with the analytical posterior path.Source: §3.2 · pp. 3–4

3.3 Self-Consistent Cross-Step Sampling

  • For any continuous cross-step interval, BFM has a closed-form posterior; changing the NFE budget reparameterizes the time path instead of approximating skipped transitions.The framework calls this self-consistency under cross-step sampling.Source: §3.3 · pp. 4–5

3.4 Model Training and Sampling

  • Training predicts clean data with a neural network; sampling starts from uniform binary noise and iteratively draws reverse states, with temperature controlling sample diversity.The paper describes training and sampling as requiring only minimal modification of existing binary diffusion architectures.Source: §3.4 · p. 5

4 Experiments

The experiments test BFM’s low-NFE stability, standard image-generation performance, binarized-MNIST results, and sensitivity to probability-path scheduling. Across these settings, the reported results show robust low-NFE sampling and competitive full-step performance.

4 Experiments

  • The evaluation covers training-free NFE reduction, standard latent-image sampling, binarized MNIST, and probability-path scheduler ablations.The experiments also include Ising-system modeling in Appendix C.Source: §4 · p. 6

4.1 Experimental Setup

  • Experiments use the official BLD codebase and identical backbones and hyperparameters for a fair BFM comparison.Unless otherwise specified, metrics are computed from 50K generated samples; the setup uses two RTX 4090 GPUs and temperature κ = 0.9.Source: §4.1 · p. 6

4.2 Self-Consistent Low-NFE Sampling

  • At 16 inference steps, BFM reaches FID 9.22 while BLD degrades to 204.10 on LSUN Churches 256 × 256.Both use 100K-iteration checkpoints trained with 256 sampling steps; BFM also preserves scene structure as NFE falls, while BLD becomes blurry and unstable. BFM reaches lower FID throughout 100K–400K training iterations on LSUN Bedrooms.Source: §4.2 · pp. 6–7, Figure 1 · p. 6, Figure 2 · p. 7

4.3 Standard Sampling Performance on High-Dimensional Image Generation

  • In the matched 800K-iteration comparison, BFM outperforms reproduced BLD across LSUN Bedrooms, LSUN Churches, and FFHQ.Reported examples are FID 5.86 vs. 6.22 on LSUN Bedrooms and 10.87 vs. 11.46 on FFHQ.Source: §4.3 · p. 7, Table 1 · p. 8

4.4 Performance on Binarized MNIST

  • BFM achieves the best evaluated FID on binarized MNIST, with a score of 0.42.Pixels greater than 0 are set to 1 and the others to 0.Source: §4.4 · pp. 8–9, Figure 4 · p. 8

4.5 Ablation Study on Bernoulli Probability Path Scheduling Strategies

  • BFM has lower FID than BLD under all three tested probability paths, and the linear scheduler gives BFM its best overall result, 9.22.BFM’s FID varies somewhat more across schedulers than BLD’s, but its overall variation remains small; the linear schedule is used as the default in subsequent experiments.Source: §4.5 · p. 9, Table 2 · p. 9

5 Discussion

The discussion presents BFM as a continuous Bernoulli-path framework with analytical posterior transitions over arbitrary intervals. It reports robust training-free low-NFE sampling and competitive standard-inference performance, while identifying broader domain validation as future work.

5 Discussion

  • BFM defines a continuous global Bernoulli path and derives analytical posterior transitions over arbitrary time intervals.Changing NFE re-evaluates the posterior on new intervals rather than approximating skipped one-step transitions.Source: §5 · p. 9
  • The evaluation is limited to representative image-generation and scientific binary benchmarks, leaving broader domain-specific validation for future work.Source: §5 · p. 9
  • The authors plan to combine BFM with distillation or consistency-style training for very-few-step or one-step generation.Source: §5 · p. 9

A Proofs of Bernoulli Probability Flow Dynamics

The appendix proves the Bernoulli probability-flow dynamics by deriving the marginal, forward transition, and one-step posterior distributions. It also introduces a symmetric binary combination operator used in the derivations.

A.1 Mathematical Preliminaries

  • The symmetric binary combination operator is introduced to simplify the binary probability space.The function g is defined on [0, 1] × [0, 1].Source: §A.1 · p. 12

A.2 Marginal Distribution Dynamics

  • Marginalizing the uniform Bernoulli prior gives the exact marginal distribution from a data sample to an arbitrary intermediate state.The proof evaluates the marginal for both possible clean binary states and expresses it using the symmetric form.Source: §A.2 · p. 12

A.3 One-Step Forward Transition Dynamics

  • Under a Markovian transition with error rate γ_tτ, the forward transition probability is uniquely specified by the marginal dynamics.The derivation uses the law of total probability to aggregate paths from the previous state.Source: §A.3 · pp. 12–13

A.4 Analytical One-Step Posterior Derivation

  • Bayes’ theorem yields an exact closed-form posterior for the previous state conditioned on the current state and source data.The posterior is simplified using the symmetric binary combination operator.Source: §A.4 · p. 13

A.5 Exact Cross-Step Posterior (Proof of Theorem 1)

The proof extends the Bernoulli posterior from one-step updates to arbitrary time intervals. Continuous-time transition parameters make the cross-step posterior available in closed form.

A.5 Exact Cross-Step Posterior (Proof of Theorem 1)

  • For any continuous times t_τ−k and t_τ with k > 1, the cross-step posterior has an exact closed-form solution.The derived posterior conditions on the current state and source data, using parameters for the interval between the two times.Source: §A.5 · pp. 13–14
  • The proof generalizes the transition error rate to the chosen interval and applies Bayes’ theorem to derive the cross-step posterior.The continuous definitions of α and γ allow the algebraic relationships to hold for arbitrary temporal strides.Source: §A.5 · p. 14
  • The exact cross-step posterior theoretically guarantees structural consistency under low-NFE inference.It is obtained by applying the analytical posterior to the new time interval.Source: §A.5 · p. 14

A.6 Connecting the Analytical Posterior to the Practical Sampler

This section connects the clean-data-conditioned oracle posterior to the learned reverse sampler. It explains how training uses that target and how the practical sampler parameterizes its reverse transition.

A.6 Connecting the Analytical Posterior to the Practical Sampler

  • Generation starts from the Bernoulli prior and repeatedly applies learned reverse transitions to obtain a clean sample.The forward process transforms clean data toward Bernoulli noise, with analytically specified transitions that have no learnable parameters.Source: §A.6 · p. 14
  • The oracle posterior is a tractable ground-truth reverse transition during training, when the clean sample is known, but is unavailable during generation.Likelihood training makes the learned reverse transition approximate this posterior through denoising-matching terms.Source: §A.6 · p. 15
  • The practical reverse transition is parameterized by predicting the clean sample X_t0, which recovers the oracle posterior when that prediction is exact.The network takes the current noisy state and time as input; the clean state is not assumed available during generation.Source: §A.6 · pp. 15–16

B Visualization of Different Temperatures of BFM Generated Samples

The visualizations show unconditional 256 × 256 samples across three datasets. For each figure, sampling temperature κ increases from left to right.

B Visualization of Different Temperatures of BFM Generated Samples

  • Figure 6 shows unconditional 256 × 256 LSUN Bedrooms samples, with κ increasing from 0.5 to 1.0 from left to right.Source: Figure 6 · p. 17
  • Figure 7 shows unconditional 256 × 256 LSUN Churches samples, with κ increasing from 0.5 to 1.0 from left to right.Source: Figure 7 · p. 17
  • Figure 8 shows unconditional 256 × 256 FFHQ samples, with κ increasing from 0.5 to 1.0 from left to right.Source: Figure 8 · p. 18

C Beyond Image Synthesis: Ising Model Generation

BFM generates Ising configurations that visually match Wolff-cluster references and captures the system’s magnetization across temperatures. The evaluation demonstrates its use beyond image generation on binary scientific data.

D Discussion on Cross-step Sampling Consistency of BFM

BFM supports cross-step sampling by recalculating its posterior for the new time intervals, rather than approximating a multi-step transition with local rules. The section also gives implementation details for the experiments.

D Discussion on Cross-step Sampling Consistency of BFM

  • BLD’s official implementation uses only the last-step noise parameter β_{t−m} for its cross-step evidence transition, rather than the exact interval transition.For K sampling evaluations from N training steps, the section describes cross-step stride as m = N/K.Source: D Discussion on Cross-step Sampling… · p. 19
  • BFM’s cross-step posterior recalculates γ_{tτ} from the new interval endpoints without approximation, supporting sampling at arbitrary NFE.The time-grid interpolation is reparameterized from the training steps to the sampling NFE.Source: D Discussion on Cross-step Sampling… · pp. 19–20

E Additional Implementation Details

  • For image tasks, BFM uses a pre-trained binary autoencoder and the BLD Transformer backbone, while Binarized MNIST and Ising are modeled directly in native binary spaces.The section also notes minor sequence-length and positional-embedding adaptations for Binarized MNIST and Ising.Source: E Additional Implementation Details · p. 20, Table 3 · p. 20

E.1 Compute Resources and Inference Efficiency

Under single-sample, 64-step inference, all evaluated tasks use less than 1.1 GB of peak GPU memory. Image-256 has higher latency and memory use than the native binary-space tasks.

  • The benchmark measures single-sample latency and peak GPU memory after five warm-up runs, averaging 20 runs with CUDA synchronization.Image-256 includes binary-latent sampling and BAE decoding; Binarized MNIST and Ising are sampled directly in their native binary spaces.Source: §E.1 · pp. 20–21, Table 4 · p. 21
  • Less than 1.1 GB of peak GPU memory is used across all tasks in the single-sample, 64-step inference benchmark.Source: Table 4 · p. 21, §E.1 · p. 21
  • Image-256 has slightly higher latency and memory use than Binarized MNIST and Ising because it uses a higher-dimensional latent space and invokes the BAE decoder.Source: §E.1 · p. 21
Loading 2610.11362v1…