Predicting integers from continuous parameters

Bas Maat; Peter Bloem

Source-linked AI summary

Predicting integers from continuous parameters

Bas Maat, Peter Bloem

arXiv:2602.10751v3Feb 11, 2026cs.LG

TL;DR

The paper asks whether integer-valued labels can be modeled directly with discrete distributions while retaining continuous parameters for neural-network optimization. It evaluates existing and novel distributions across tabular, sequential, and image tasks, finding Dalap strongest overall and Bitwise effective for highly dispersed data. The results support discrete probabilistic modeling when valid integer probability mass is needed, while identifying computational and distribution-specific limitations.

Problem
The paper investigates whether integer or bounded-integer labels can be modeled with genuine discrete distributions instead of continuous relaxations, while supporting neural-network backpropagation.
Method
The study evaluates several existing and novel discrete distributions with continuous parameters across tabular regression, sequential prediction, and image-generation tasks.
Results
Dalap offers the strongest overall performance across the evaluated settings, while Bitwise and Dnormal may be preferable for highly dispersed data.
Takeaways & Limitations
When a discrete probabilistic model is necessary, Dalap provides the preferred overall balance of distributional quality and prediction accuracy in the evaluated settings.
Takeaways & Limitations
Image conclusions are limited by computational constraints, including no exhaustive search over mixture components and image experiments using a single random seed.

Abstract

from arXiv · show

We study the problem of predicting numeric labels that are constrained to the integers or to a subrange of the integers. For example, the number of up-votes on social media posts, or the number of bicycles available at a public rental station. While it is possible to model these as continuous values, and to apply traditional regression, this approach changes the underlying distribution on the labels from discrete to continuous. Discrete distributions have certain benefits, which leads us to the question whether such integer labels can be modeled directly by a discrete distribution, whose parameters are predicted from the features of a given instance. Moreover, we focus on the use case of output distributions of neural networks, which adds the requirement that the parameters of the distribution be continuous so that backpropagation and gradient descent may be used to learn the weights of the network. We investigate several options for such distributions, some existing and some novel, and test them on a range of tasks, including tabular learning, sequential prediction and image generation. We find that overall the best performance comes from two distributions: Bitwise, which represents the target integer in bits and places a Bernoulli distribution on each, and a discrete analogue of the Laplace distribution, which uses a distribution with exponentially decaying tails around a continuous mean.

1 Introduction

The paper asks whether integer-valued regression targets can be modeled directly with discrete distributions whose continuous parameters support neural-network training. It develops and evaluates candidate distributions across tabular, sequential, and image tasks, finding strongest overall performance from Dalap and Bitwise.

Integer regression targets include signed values, counts, and bounded integer ranges such as RGB channel values.
Continuous relaxation treats labels as continuous during training and rounds predictions at inference, but can perform poorly for pixel prediction.
Direct categorical modeling assigns probabilities to allowed labels but becomes impractical for large or unbounded ranges and ignores target ordinality.
Neural-network distributions require continuous parameters, well-behaved gradients, and the ability to concentrate probability near an arbitrary integer.
The study introduces three novel approaches alongside three existing baselines and evaluates them on tabular regression, MIDI prediction, and autoregressive image generation.
Dalap outperforms existing distributions in most negative-log-probability comparisons, while continuous squared-error regression remains a strong point-prediction baseline.

2 Related Work

Related work covers discrete count and integer distributions, neural-network-specific requirements, and approaches formed by discretizing continuous distributions or constructing discrete analogues. The paper emphasizes that integer regression with potentially negative labels is less represented in the literature.

Count-data regression commonly uses Poisson or negative binomial distributions for non-negative or positive integer labels.
Neural-network modeling requires continuous distribution parameters and prioritizes differentiable optimization rather than closed-form parameter solutions.
Poisson models impose equidispersion because their mean and variance are equal, while extensions address overdispersion and underdispersion.
The paper includes the Discrete Weibull as a baseline because it is a discrete distribution with continuous parameters.
Integer regression with potentially negative labels is less recognized, possibly because continuous relaxation creates fewer problems in those domains.
The paper distinguishes discretizing continuous distributions from constructing discrete analogues that preserve selected properties of continuous distributions.

3 Methods

The paper reviews existing discrete distributions and introduces Dalap, Danorm, and Bitwise for integer-valued neural-network regression. These distributions use continuous parameters while preserving discrete outputs, with analyses of boundedness, losses, means, variances, and concentration.

Distribution choices: The section presents discretized Normal, discretized Laplace, Weibull, Dalap, Danorm, and Bitwise as distributions for integer-valued regression.The first three are existing distributions; the latter three are partly novel approaches.
Existing distributions: Discretized Normal and Laplace assign integer probability mass by rounding continuous random-variable samples.The discretized Normal uses a rounding function and computes masses from the Normal cumulative distribution; the Laplace version follows the same procedure.
Dalap: Dalap uses p(n) ∝ γ^|n−µ| with continuous µ, and its bounded versions condition the unconstrained distribution on an allowed integer subset.The construction modifies the partition function so probability changes continuously as µ crosses integer boundaries.
Dalap: Dalap's negative log-probability is linear in |µ−n|, with uncertainty controlled by a multiplicative factor involving log 1/γ.This makes minimizing the loss equivalent to minimizing absolute distance, tempered by uncertainty.
Danorm: Danorm provides a discrete distribution with a squared-distance loss, but its partition function lacks a closed-form solution; its variance nevertheless has the same γ → 0 limit pattern.Its expectation converges to the nearest integer, with the same midpoint exception.
Bitwise: Bitwise models an integer through independently sampled Bernoulli bits, and its variance can approach 0 around any supported value.The method interprets binary integer representations as a probability distribution with real-valued parameters.

4 Experiments

The experiments evaluate integer-valued distributions across tabular, sequential, and image settings, comparing discrete candidates with continuous relaxation where applicable. The study also examines practical training choices and exclusions across bounded, unbounded, and one-way-bounded targets.

Experimental scope: The evaluation spans tabular regression, MIDI next-note tick prediction, and image modeling across bounded, unbounded, and one-way-bounded targets.Datasets include Bicycles, Upvotes, Migration, MAESTRO, MNIST, FashionMNIST, and CIFAR10.
Evaluation criteria: The experiments compare six candidates using negative log-loss when discrete distributions are required and RMSE when continuous relaxation is also considered.These are the two main evaluation questions stated for all experiments.
Evaluation protocol: Hyperparameters are selected using validation-set log loss, and final results are reported on the test set.Tabular and MAESTRO experiments sweep six learning rates and test mixture sizes k ∈ {1, 2, 4, 8}.
Modeling setup: For tabular datasets, an MLP uses one output for squared error, 32 outputs for Bitwise, and two outputs for the other distributions.The network has a hidden size of 128, with valid parameter mappings detailed separately.
4.2 MAESTRO: In MAESTRO, the study predicts the duration in ticks of the next note from preceding MIDI events rather than modeling complete music generation.The MIDI representation includes Event, Data1, Data2, Instrument, Channel, and Tick fields; Tick uses a 32-bit binary representation concatenated with its log value.
4.3 PixelCNN++: Dweib and Danorm are excluded from PixelCNN experiments because of numerical instability and computational cost, with Danorm requiring over 90GB of VRAM.Categorical mixtures are also excluded because parameter costs scale with support size and mixture count.

5 Results

Across tabular and image experiments, Dalap generally provides the strongest likelihood performance, while the best distribution varies with dataset characteristics and evaluation metric. Bitwise is particularly effective on highly dispersed Migration data, whereas continuous squared-error regression remains a strong RMSE baseline.

Count data regression: Dalap achieves the lowest bits on Bicycles (6.78±0.02), Upvotes (6.74±0.01), and MAESTRO (5.00 ± 0.00) without mixtures.Poisson has much worse bits on applicable datasets, despite sometimes competitive RMSE.
Count data regression: 22.9 ± 1.0 bits makes Bitwise the best non-mixture result on Migration, while its mixture result is also best at 18.0 ± 0.0.Dalap is unstable on this extremely dispersed dataset, with 8 of 10 seeds converging to low bits and 2 diverging.
Count data regression: Bitwise’s strong likelihood performance on highly dispersed data comes with substantially increased RMSE across datasets because rare higher-order-bit assignments can create large outliers.The practical importance of this trade-off depends on the deployment use case.
Count data regression: Continuous squared-error regression has the best RMSE on three of four datasets without mixtures, but discrete mixtures outperform it on some datasets.When valid discrete log-probabilities are required, Dalap is reported to offer the best balance of bits and RMSE.
Image modeling: Dalap achieves the best non-mixture bits per dimension on MNIST (0.61) and FashionMNIST (1.23), and remains competitive on CIFAR10 at 3.11.In mixture models, it achieves the best CIFAR10 result at 3.0206 bits per dimension and matches or exceeds Dlogistic on the reported comparisons.
Image modeling: Image-generation results depend on the task: Dlogistic dominates mixture reconstruction FID, whereas Dalap is competitive or better for sampling while maintaining stronger likelihood scores.The paper therefore presents distribution choice as dependent on application requirements.

6 Conclusion

The paper finds that genuinely discrete distributions can model integer labels competitively across tabular, sequential, and image tasks, with Dalap offering the strongest overall performance when discrete probability modeling is required. Results also show that mixture components help challenging data, while computational constraints limit some image conclusions.

Findings: Dalap generally provides the strongest negative log-likelihood performance across the evaluated tabular, sequential, and image settings.It is competitive with continuous relaxations while retaining valid probability mass functions on integers.
Findings: Continuous squared-error regression remained a strong point-prediction baseline, whereas Danorm often had worse negative log-probability and was impractical for image experiments.Danorm’s numerical partition function made high-dimensional use costly.
Findings: Mixture components improved both bits and RMSE for most distributions across most datasets.On Migration, increasing Dalap to K = 8 resolved divergence observed in 2 of 10 seeds.
Findings: Bitwise achieved the best bits on Migration with mixtures, but uncertainty in higher-order bits produced high variance and RMSE.Dnormal and Bitwise can therefore outperform Dalap on highly dispersed targets.
Limitations: Image experiments used one random seed and did not exhaustively search mixture counts because PixelCNN training required substantial computational resources.These constraints limit conclusions about image-result variability and possible hyperparameter improvements.

A.1 Exploratory analysis of the target distributions

The datasets span bounded and unbounded integer targets with markedly different dispersion, skewness, kurtosis, and tail behavior. Exploratory summaries and logarithmic visualizations make these distributional differences explicit for interpreting model performance.

Distributional characteristics: Upvotes and Migration are extremely overdispersed and heavy tailed, while Bicycles is mildly overdispersed.MAESTRO has moderate dispersion, consistent with the relative success of the Poisson baseline there.
Distributional characteristics: MNIST, FashionMNIST, and CIFAR10 pixels occupy the bounded support {0, . . . , 255}.MNIST and FashionMNIST concentrate near 0 and 255; CIFAR10 is broader but still bimodal.
Summary statistics: The appendix summarizes each target’s support, sample size, range, mean, variance, dispersion index, skewness, and excess kurtosis.DI is defined as σ2/µ.
Visual analysis: Figure 6 uses a logarithmic count axis because heavy-tailed datasets place several orders of magnitude more mass near the mode than in the tail.Its horizontal axis is clipped to the central 98 percent, with the full range reported in panel annotations.
Visual analysis: Figure 7 plots the dispersion index σ2/µ for each dataset on a logarithmic scale, distinguishing tabular and image datasets by color.Tabular datasets are blue and image datasets are red.

A.2 Generated image examples

The generated-image figures compare seeded completion with unconditional random sampling for five discrete distribution models, using ten mixture components where indicated. Each figure organizes conditioned samples above unconditional samples.

Dalap: Dalap mixture samples are shown with seeded conditioning in the top row and unconditional random sampling in the bottom row.The model uses K=10 mixture components.
Dlogistic: Dlogistic mixture samples use the same top-row seeded and bottom-row unconditional layout with K=10.This provides a direct visual comparison between conditional completion and free sampling.
Bitwise: Bitwise mixture samples likewise separate seeded sampling above from unconditional random sampling below.The figure specifies K=10.
Dnormal: Dnormal mixture samples are arranged as seeded top-row images and unconditional bottom-row images, with K=10.The layout matches the other mixture-model figures.
Laplace: Laplace mixture samples use the same seeded-versus-unconditional arrangement and K=10 setting.The top portion of each image supplies the conditioning signal.

A.3 Terms of the Dalap loss

The Dalap-loss analysis examines how certainty and distance from the target interact when the continuous mean moves around integer values. Although one penalty term can favor non-integer means in isolation, the complete loss accounts for both distance and certainty.

Whole-loss behavior: The complete Dalap loss balances the distance-related cost against the penalty term associated with certainty.The first term promotes low γ, while exponentially low target probability penalizes large |µ − n|.
Whole-loss behavior: Figure 13’s apparent reward for non-integer µ as γ approaches 0 disappears when the whole loss is considered.Moving µ to a non-integer value still incurs a small loss increase in the analyzed case.
Implementation: The implementation constrains γ to remain above ϵ to avoid numerical instability.This activation-level floor is an implementation safeguard.
Integer displacement: For s = 0, the loss difference is 0; for s = 1/2, its limit is log 2.The analysis concludes that increasing distance from the target always carries a cost.
Integer displacement: When moving away from an integer reduces target distance, the distance improvement can outweigh the cost of leaving an integer mean.The difference function remains below 0 for s > 0 in the stated case.

A.4 Parameter activations

Network outputs are transformed into valid distribution parameters using support-specific activation functions. Dispersion limits are manually tuned by distribution and experiment type.

Parameter mapping: Raw network outputs x1 and x2 are mapped to valid location and scale parameter ranges.x1 denotes the location output, while x2 denotes the scale output.
Dispersion control: γmax controls maximum dispersion for Dalap and Danorm, trading wider tails against numerical stability.Values near 1 allow wider distributions but may become unstable; lower values constrain the tails.
Experiment settings: γmax is set to 1.0 for tabular and MAESTRO experiments, 0.9 for image experiments, and smax is 1.0 for discretized distributions.These hyperparameters were selected through manual tuning.

A.5 Numerical stability

The implementation addresses numerical instability in bounded and discrete distributions through support-aware activations, clamping, and bounded parameterizations.

Bounded Dalap: The bounded Dalap partition clamps µ to [l, u] before computation and uses truncated-geometric forms when µ lies outside the interval.The simplified forms apply when the activation permits out-of-range values or numerical deviations occur.
Activation functions: Table 9 specifies support-specific activations that map raw outputs into valid parameter ranges.For bounded support, sigmoid-based transformations constrain values to [l, u], while other distributions use distribution-specific transformations.
Logarithmic stability: The partition denominator is clamped to at least ϵ before taking its logarithm, preventing log(0).This safeguard directly targets invalid logarithms during numerical evaluation.

A.5.1 Discrete Weibull implementation

The discrete Weibull implementation computes probabilities and gradients with explicit safeguards for infinities, sign errors, precision loss, and undefined gradient values. Bitwise training additionally emphasizes higher-order bits according to their effect on decoded integers.

Forward pass: The Dweib forward pass computes negative-power terms from alpha, beta, and the target before taking their difference.The implementation uses eps to stabilize inputs and identifies cases where large targets or small alpha produce -inf.
Forward stability: Double -inf terms are mapped to a zero-probability difference, while positive differences are clamped to zero before stable expm1 evaluation.These fixes prevent NaNs from inf - inf and from logarithms of negative values.
Backward pass: The backward pass derives alpha and beta gradients from saved forward-pass quantities and replaces NaN gradient components with 0.0.The returned gradients are multiplied by the incoming log-probability gradient.
Gradient helper: The helper tet(x) computes x · exp(x) and returns 0 when x = -inf, supporting stable gradient calculations.This convention avoids propagating undefined values from negative infinity.
Bitwise weighting: During Bitwise training, bit position i receives a binary cross-entropy weight of 2^i because higher-order bit errors change decoded integers more.Evaluation weights all bits equally to recover the standard log-likelihood.