PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Estimation with Discrete Diffusion

Alberto Foresti, Giulio Franzese, Pietro Michiardi

arXiv:2610.23404v1cs.LGcs.ITmath.IT

TL;DR

High-dimensional discrete data remain difficult for information-theoretic estimation, and common embedding-based approaches use estimators designed for continuous distributions. INFO-SEDD uses CTMC-based discrete diffusion to estimate information measures. The authors report that it outperforms embedding-based alternatives for MI estimation and demonstrate applications including summarization model selection and DNA motif detection.

  • Problem

    Few methods estimate information-theoretic quantities for high-dimensional discrete data, while embedding-based workarounds may not capture discreteness and can require application-specific embeddings.

    Source: §1 · p. 2

  • Method

    INFO-SEDD uses CTMC-based discrete diffusion and score functions to estimate information measures, including MI and entropy.

    Source: §1 · p. 2

  • Results

    INFO-SEDD outperforms alternatives for MI estimation in synthetic and real-data experiments, including text summarization and genomics applications.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors show MI can serve as a proxy for text-summarization model selection and that accurate MI estimates can help detect motif locations in DNA data.

    Source: §1 · p. 2, §5 · p. 10

    The general KL estimator is not scalable for practical use, so the paper introduces a practical method to address this limitation.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the "embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.

1 INTRODUCTION

Estimating information-theoretic quantities for high-dimensional discrete data remains challenging, while embedding-based workarounds may fail to capture discreteness and require application-specific choices. INFO-SEDD uses CTMCs to estimate information measures, and the reported experiments find it outperforms alternatives for MI estimation.

1 INTRODUCTION

  • Some lower-bound estimators require sample counts that grow exponentially with ground-truth MI, challenging their use in high-MI settings.The paper situates this concern within a broader landscape of classical, variational, and neural estimation approaches.Source: §1 · p. 1
  • Few estimators address high-dimensional discrete distributions, and classical discrete estimators lose accuracy as data dimensionality grows.The introduction names DNA or peptide sequencing, text summarization, and neuroscience as potential applications.Source: §1 · p. 2
  • Embedding discrete data and applying neural estimators for continuous distributions is a common workaround, but may miss discreteness and require application-specific embeddings.The authors report that MINDE, a strong continuous-domain estimator, struggles on discrete data.Source: §1 · p. 2
  • INFO-SEDD applies CTMC-based diffusion estimation to discrete data, using score functions to compute measures including mutual information and entropy.The method extends the CTMC framework via Dynkin’s lemma and is designed to use one parametric model for MI across arbitrary variable subsets.Source: §1 · p. 2
  • INFO-SEDD outperforms alternatives for MI estimation across synthetic and real-data experiments, with applications to text-summary model selection and DNA motif localization.The summarization estimates align with human evaluation metrics; the genomics experiments evaluate consistency and detect motif locations.Source: §1 · p. 2

2 METHODOLOGY

The methodology connects CTMC time reversal and parametric score approximations to estimating KL divergence between discrete distributions. It then uses this framework to derive information measures, while noting that the general KL estimator is not practical at scale.

2.2 KL DIVERGENCES VIA CTMCS

  • The methodology adapts CTMCs to estimate KL divergence between discrete distributions by comparing processes with the same generator but different initial conditions.The section develops the formulation using Dynkin’s formula and parametric score approximations; the general estimator is not scalable for practical use.Source: §2 · p. 2, §2.2 · pp. 3–4

2 METHODOLOGY

  • The section frames CTMC time-reversal properties and parametric approximations as the foundation for adapting these processes to divergence estimation.The KL formulation compares two processes that share a generator but start from different distributions.Source: §2 · p. 2, §2.2 · p. 3

2.1 PRELIMINARIES

  • A CTMC can converge to a reference distribution independent of its initial distribution, and its time-reversed process is also a CTMC.The paper uses parametric score functions to approximate otherwise inaccessible time-dependent probabilities.Source: §2.1 · p. 3

3 OUR APPROACH: INFO-SEDD

INFO-SEDD formulates MI estimation through KL divergence and uses structured CTMC perturbations to share a score model across joint and marginal distributions. Its theoretical analysis bounds estimation error and identifies a finite-horizon bias that decays as the absorbing-state probability approaches one.

3 OUR APPROACH: INFO-SEDD

  • INFO-SEDD estimates MI through either the KL divergence between the joint distribution and product of marginals or conditional and marginal distributions.These are the joint and conditional approaches, INFO-SEDD-J and INFO-SEDD-C.Source: §3 · p. 4
  • Structured sequence decomposition enables sparse rate matrices that modify one subcomponent at a time, reducing computation for high-dimensional state spaces.The full state space grows as |χ|^D for D subcomponents, while the sparse construction limits transitions to sequences differing in one component.Source: §3 · p. 4
  • A carefully chosen transition matrix lets one score model trained on the joint distribution provide marginal scores, avoiding separate models for the joint and product of marginals.The absorbing matrix restricts subcomponents to transitions into an absorbing state, enabling marginal-score computation from the joint-trained model.Source: §3 · pp. 4–5
  • Under bounded-score and neural-approximation assumptions, the estimator’s error combines a score-error term and finite-horizon bias that vanishes exponentially as absorbing-state probability approaches one.The authors describe INFO-SEDD as consistent up to this exponentially decaying bias and contrast it with exponential variance from standard importance sampling.Source: §3 · p. 5
  • The KL framework also yields INFO-SEDD-H, an estimator for entropy.The paper derives entropy estimation by expressing entropy in terms of KL divergence.Source: §3 · p. 5

4 EXPERIMENTS

Experiments evaluate INFO-SEDD on synthetic benchmarks and text and DNA applications, with additional Ising entropy results. The reported findings emphasize accurate estimation and usefulness for downstream tasks.

4.1 SYNTHETIC EXPERIMENTS

  • INFO-SEDD consistently outperforms synthetic-benchmark competitors, especially at high dimensionality and MI, and remains robust across support sizes and with 10^3 samples.Even when competitors estimate correctly, they can require more epochs to converge.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 APPLICATION: TEXT SUMMARIZATION

  • Both INFO-SEDD variants best match the expected linear MI trend across ρ; variational methods underestimate high MI, while HD-DIME fails below ρ=0.5.The benchmark varies the probability of pairing BART summaries with their source rather than a random text.Source: §4.2 · pp. 6–7, Figure 1 · p. 7
  • MI correlates most with consistency, then fluency and relevance, but consistency saturates while MI does not; raters also penalize hallucinations MI misses.INFO-SEDD-C achieves a comparable correlation with consistency without elaborate embedding models.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 APPLICATION: GENOMICS

  • INFO-SEDD-C most closely matches classifier-reference MI on HUMAN VS. WORM, outperforming competitors; its label-focused conditional scores are easier to optimize than joint sequence-label scores.The test randomizes labels with probability ρ and approximates reference MI using classifier accuracy.Source: §4.3 · p. 8, Figure 4 · p. 9
  • INFO-SEDD-J’s masked-window MI profile locates the Arabidopsis TATA-BOX and assesses correlated motifs without interference; subset MI avoids retraining for each window.The method masks sequence outside a sliding window and evaluates the window’s MI with promoter labels.Source: §4.3 · p. 9

5 CONCLUSION

INFO-SEDD targets scalable, accurate estimation of information-theoretic metrics for high-dimensional discrete data without ad-hoc preprocessing. The conclusion highlights its applications to text summarization and genetics, and its flexibility to other masked discrete diffusion models.

5 CONCLUSION

  • INFO-SEDD is presented as scalable, consistent, and accurate for high-dimensional discrete distributions, while leveraging pretrained models without ad-hoc procedures.The authors also report flexibility across masked discrete diffusion models, including MD4 and LLADA.Source: §5 · p. 10
  • MI estimates support text-summarization model selection and have potential to enable discoveries using discrete sequencing data.The authors also point to possible extensions to mixed continuous/discrete data through Generator Matching.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the shared codebase and hardware basis for performance comparisons. It also directs readers to experiment setup and dataset details by section.

REPRODUCIBILITY STATEMENT

  • Performance comparisons use the same codebase and machine, differing only in MI-estimation methodology and loss functions.Code, dataset-building utilities, and experiment instructions are provided at the project repository.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The statement directs readers to separate sections for text-model setup and for synthetic, text, and DNA dataset details.It points to Sections C.2.2, C.1.1, C.2.1, and C.4.1.Source: REPRODUCIBILITY STATEMENT · p. 11

A MATHEMATICAL PROOFS OF THE MAIN METHOD

The appendix combines proof steps, algorithm listings, and benchmark details for INFO-SEDD. Its empirical results report faster convergence, memory advantages, and robustness as support size increases, while noting sample-complexity constraints.

A MATHEMATICAL PROOFS OF THE MAIN METHOD

  • The proof simplifies terms using element-wise division and Dynkin's formula, and the entropy appendix supplies derivation steps.The appendix also indicates that mutual information can be estimated without modifying the score network.Source: §A.1 · p. 19, §A.2 · p. 21, §A.3 · p. 21

B ALGORITHM PSEUDO-CODE

  • The appendix provides pseudocode variants for estimating joint MI, conditional MI, and entropy.The listings identify INFO-SEDD-J, INFO-SEDD-C, and INFO-SEDD-H, respectively.Source: B ALGORITHM PSEUDO-CODE · p. 22

C.1.1 DATASET CONSTRUCTION

  • The synthetic datasets use user-defined MI and support sizes, with added independent noise and Cantor pairing used to expand support while preserving MI.The described construction uses the additivity of MI for independent random variables and isomorphisms that preserve it.Source: §C.1.1 · p. 23

C.1.2 MODEL AND TRAINING SETUP

  • Across benchmark methods, training uses an approximately 130k-parameter MLP with skip connections; the discrete score model follows the absorb configuration.At inference, each method uses its last valid validation-step estimate to avoid NaN values.Source: §C.1.2 · p. 23

C.1.4 LOW DIMENSIONAL BENCHMARK

  • On the low-dimensional synthetic benchmark, all estimators besides MINDE do a good job predicting the right MI, and Table 5 shows that no clear winner emerges among them.Source: §C.1.4 · p. 24, Table 5 · p. 24

C.1.5 SAMPLE COMPLEXITY BENCHMARK

  • INFO-SEDD's sample complexity depends on the score network, and practitioners must balance network capacity against the number of training samples.The authors defer theoretical guarantees for score networks and discrete diffusion models to future work.Source: §C.1.5 · pp. 24, 26

C.1.3 STUDY ON TRAINING CONVERGENCE

  • INFO-SEDD converges faster than the benchmarked methods; it also uses less memory than all competitors and runs faster than variational competitors.The convergence comparison is shown in Figure 6; runtime and memory were benchmarked on an NVIDIA A100-SXM4-80GB GPU.Source: §C.1.3 · p. 23, Figure 6 · p. 25

C.1.6 ABLATION STUDY ON SUPPORT DIMENSION

  • In the support-dimension ablation, INFO-SEDD is the only estimator reported as robust to increasing support size.The experiment fixes D=10 and MI=0.5.Source: §C.1.6 · p. 26, Table 7 · p. 26

C.1.7 EXTREME DIMENSIONALITY EXPERIMENTS

The appendix evaluates extreme-dimensional synthetic settings and describes textual consistency-test setup and results. INFO-SEDD remains accurate at D=1024 when trained with sufficient data and capacity.

C.1.7 EXTREME DIMENSIONALITY EXPERIMENTS

  • At D=256 and MI=0.5, INFO-SEDD provides a competitive, well-calibrated estimate while competing methods remain relatively stable.The experiment also compares against GAN-DIME, HD-DIME, and SMILE.Source: §C.1.7 · p. 26, Table 8 · p. 26
  • At D=1024 and ground-truth MI=0.5, competing methods collapse to zero while INFO-SEDD remains accurate and close to ground truth.The discussion qualifies this scalability result as requiring sufficient data and capacity.Source: §C.1.7 · pp. 26–27, Table 9 · p. 26

C.2.1 DATASET AND GROUND TRUTH CONSTRUCTION

  • Text consistency tests mix BART summaries paired with their source text at probability ρ and random texts otherwise, with MI expected to grow linearly under the stated assumption.The assumption is that MI substantially exceeds log 2.Source: §C.2.1 · p. 28

C.2.2 MODEL SETUP

  • The consistency-test numerical results are reported in nats, and the text experiments use MDLM-small trained on OpenWebText.The model's score can be recovered from its output; MINDE instead uses a variance-preserving SDE on token embeddings.Source: §C.2.3 · p. 28, §C.2.2 · p. 28

C.2.4 DETAILED RESULTS FOR HUMAN METRICS CORRELATION EXPERIMENTS

Across the experiments, INFO-SEDD estimates information measures in discrete-data settings, including text summarization and Ising models. The reported analyses compare its estimates with human judgments and other metrics, test DNA-label consistency, and assess entropy estimation.

C.2.4 DETAILED RESULTS FOR HUMAN METRICS CORRELATION EXPERIMENTS

  • Human consistency and fluency scores rise with MI only until they saturate at 5.0; MI avoids this ceiling, and INFO-SEDD needs no human annotations for model selection.The GP mean follows an increasing trend for consistency and fluency until human scores saturate.Source: §C.2.4 · pp. 29–30, Figure 7 · p. 30, Figure 8 · p. 31

C.3 COMPARISON OF MI ESTIMATES WITH INFO-SEDD AND OTHER TEXT SUMMARIZATION

  • INFO-SEDD-J generally correlates better with other summarization metrics, while INFO-SEDD-C correlates better with human metrics and best with consistency among those metrics.The comparisons use Pearson and Kendall Tau correlation heatmaps.Source: §C.3 · pp. 31–32, Figure 10 · p. 32, Figure 9 · p. 32

C.4.1 DATASET AND GROUND TRUTH CONSTRUCTION

  • The DNA consistency test constructs labels by retaining the original label with probability 1−ρ and otherwise sampling an independent label; its conditional-probability approximation assumes near-certain classification.The authors choose ε using the best HUMAN VS WORM classifier accuracy reported by Schiff et al. (2024).Source: §C.4.1 · p. 33

C.4.2 MODEL SETUP

  • For motif selection, the experiments use INFO-SEDD-J because masking DNA-sequence elements is required, whereas INFO-SEDD-C does not mask during training.The setup uses a pretrained Caduceus backbone and makes no architectural variations to INFO-SEDD.Source: §C.4.2 · pp. 33–34

C.4.3 NUMERICAL RESULTS FOR THE CONSISTENCY TESTS

  • Table 13 reports mutual-information estimates for the DNA consistency tests in nats.The supplied passage identifies the table but does not state its numerical values.Source: §C.4.3 · p. 34, Table 13 · p. 34

D.1 ISING MODEL ENTROPY ESTIMATION

  • INFO-SEDD accurately estimates large KL divergences in the Ising experiments, particularly at low temperatures, where variational estimators struggle with limited samples.The evaluation used a 20 × 20 lattice and 10,000 samples per temperature from 1.0K to 4.0K.Source: §D.1 · p. 36, Figure 11 · p. 36

D.2 COMPARISON WITH PLUG-IN ESTIMATORS

The plug-in estimators examined are unreliable in high-MI or high-dimensional regimes. The theoretical appendix states assumptions for INFO-SEDD’s analysis, including sparse CTMC transitions and bounded score functions and networks.

D.2 COMPARISON WITH PLUG-IN ESTIMATORS

  • Taylor-approximation and nearest-neighbor plug-in estimators are not robust in high-MI or high-dimensional regimes, while INFO-SEDD remains stable and accurate across both scenarios.The Gao et al. (2017) estimator is reasonably accurate at moderate MI but deteriorates as dimensionality and dependence increase, and overestimates MI in an extreme-dimensional, low-MI setting.Source: §D.2 · pp. 36–37

E THEORETICAL PROPERTIES OF INFO-SEDD

  • The theoretical-properties section introduces assumptions and an error-bound analysis for INFO-SEDD’s KL estimate.The supplied passage outlines these topics but does not state the bound itself.Source: E THEORETICAL PROPERTIES OF INFO-SEDD · p. 38

E.1 ASSUMPTIONS AND SETTING

  • The theoretical analysis assumes sparse CTMC transitions that change one component at a time, considers Hamming-distance-one sample pairs, and imposes boundedness conditions on score functions and networks.A positive lower bound avoids degeneracies and ensures well-defined log terms in the loss.Source: §E.1 · p. 38

E.2 ERROR ANALYSIS OF INFO-SEDD

The appendix decomposes INFO-SEDD’s KL error into score-network and finite-horizon terms, then analyzes consistency and truncation bias. Under score-network convergence, the estimator converges to the true KL minus truncation bias, which can be reduced by increasing the horizon.

E.2 ERROR ANALYSIS OF INFO-SEDD

  • The KL error’s score-network component increases linearly with score error at fixed T, while the other term approaches zero exponentially as the absorbing-state probability approaches 1.The bound’s second term does not depend on factors exponential in the KL divergence.Source: §E.2 · p. 41

E.3 DISCUSSION ON CONSISTENCY

  • With enough samples and model capacity for score-network convergence, the final estimator converges in probability to the true KL minus truncation bias.The error decomposition includes a score-network term and a CTMC- and data-dependent term that better score networks cannot reduce.Source: §E.3 · p. 42

E.4 DISCUSSION ON BIAS

  • INFO-SEDD is unbiased for α, but retains truncation bias; increasing T reduces that bias exponentially while the other error term grows only linearly with the horizon parameter.The passage describes reducing truncation bias without affecting the other error term’s linear dependence on the horizon parameter.Source: §E.4 · pp. 42–43

Related papers

Loading 2610.23404v1…