Source-linked AI summary
Tensor network representations of discrete maximum entropy distributions via mean polytopes
Alex Goessmann, Martin Eigel
TL;DR
Maximum-entropy distributions are well characterized by exponential families only when their mean parameters lie in the relative interior of the mean polytope. This paper uses Computation-Activation Networks and polytope faces to characterize distributions for arbitrary mean parameters, including boundary cases. The characterization gives a unique maximum-entropy distribution for every feasible mean parameter, with a tensor-network representation for means on representable faces.
Problem
Exponential-family maximum-entropy distributions cover mean parameters in the relative interior of the mean polytope, leaving the general boundary case to be characterized.
Source: §4 · p. 11Method
The paper uses Computation-Activation Networks and the face containing a mean parameter to characterize maximum-entropy distributions beyond the relative interior.
Source: §4 · p. 11, §4.1 · pp. 11–12Results
Every feasible mean parameter has a unique maximum-entropy distribution, and distributions for means on a face are in the network family when that face is representable with respect to the hypergraph.
Source: §4.1 · p. 12Takeaways & Limitations
The characterization extends to arbitrary feasible mean parameters, including those on lower-dimensional faces of the mean polytope.
Source: §4 · p. 11, §4.1 · pp. 11–12The representing weight vector need not be unique unless the exponential family is minimal; exponential families on refined base measures are not minimal.
Source: §3.2 · p. 11
Abstract
from arXiv · showhide
We present tensor network representations for discrete maximum entropy distributions under expectation constraints. To this end, we introduce Computation-Activation Networks (CompActNets), a tensor network architecture that subsumes exponential families. By leveraging the geometry of the convex polytope of realizable expectation vectors, we represent any maximum entropy distribution in the same architecture. We exploit the fact that proper faces of this polytope correspond to the boundary closure of exponential families, which restricts the distribution's support. We then derive explicit representations for the support within the CompActNet architecture. The proposed framework suggests tensor network ranks as complexity measures for faces. Finally, a case study on Boolean statistics links the geometry of 0/1-polytopes directly to propositional formulas.
1 Introduction
The introduction motivates maximum-entropy modeling as a learning regularizer and tensor networks as a common language for general maximum-entropy distributions and logical support. It previews a framework based on mean-polytope geometry, with a Boolean-statistics connection to propositional logic.
1 Introduction
- Maximum entropy selects a least-structured distribution consistent with observed characteristics and is dual to convex maximum likelihood.The introduction also motivates linking probabilistic models, which typically focus on maximal support, with logic's focus on worlds consistent with a knowledge base.Source: §1 · p. 1
1.1 Tensor notation
- Tensor notation assigns an index variable and dimension to each axis, while decorating tensor networks place variables on nodes and tensors on hyperedges.Contracting a network while keeping selected nodes open yields a tensor over their variables.Source: §1.1 · p. 2
1.2 Maximum entropy problem
- A statistic maps states to real vectors, and its mean parameter is encoded as a contraction using a coordinate-selection variable.The introduction defines the statistic's coordinate restrictions and the mean-parameter coordinates.Source: §1.2 · pp. 2–3
- The maximum-entropy setup constrains distributions to match a statistic's mean parameter and restricts their support to the base measure's support.The entropy is defined relative to the base measure and depends on its choice.Source: §1.2 · pp. 2–3
1.3 Computation-Activation Networks
- Computation-Activation Networks are introduced as tensor networks representing solutions to maximum-entropy problems, with statistic-image enumeration used to define their encoding.The paper notes that networks for statistics composed of connective functions can be constructed as in Goessmann et al. (2026).Source: §1.3 · p. 3
1.4 Contributions
- The contributions use tensor-network representations to characterize generic maximum-entropy distributions from mean-polytope position and connect Boolean-statistic polytopes with propositional logic.The paper also constructs statistics for arbitrary polytopes.Source: §1.4 · p. 3
1.5 Structure of the paper
- The paper proceeds from mean polytopes and their tensor-network representations to exponential families, generic maximum-entropy distributions, and Boolean-statistic connections to propositional logic.Source: §1.5 · p. 3
2 The mean polytope
The mean polytope contains the realizable mean parameters, and its faces organize restricted distributions and their tensor-network representations.
2 The mean polytope
- The mean polytope contains all mean parameters and is the convex hull of the vectors obtained from the statistic and base measure.Source: §2 · p. 4
2.1 Faces
- Restricting distributions to a face measure restricts the mean polytope to that face, and representability by face measures is equivalent to the mean parameter lying on a face.Source: §2.1 · p. 4
- A face measure is the indicator of the preimage of the face under the statistic’s selection encoding.Source: §2.1 · p. 5
2.2 Representation in the CP format
- Any face is representable by a CP graph with a hidden rank, and the stated CP rank bound is tight for the vertex and maximal-face examples.The elementary format is the rank-1 restriction of the CP format.Source: §2.2 · p. 7
2.3 Examples of elementarily representable faces
- All faces of a hypercube and a simplex are representable by elementary activation tensors.The simplex’s faces correspond to subsets and are standard simplices; the hypercube’s faces are described by fixing selected coordinates.Source: §2.3 · p. 8
2.4 Partition into relative interiors of faces
- Each mean parameter belongs to the relative interior of a unique face of the mean polytope.Source: §2.4 · p. 8
3 Exponential families
Exponential-family members have elementary Computation-Activation Network representations, and they are maximum-entropy distributions for their mean constraints. Such a member is the unique maximum-entropy solution exactly when its mean parameter lies in the relative interior of the mean polytope.
3 Exponential families
- Exponential families are a class of maximum-entropy distributions, but not every maximum-entropy distribution is an exponential-family member.The section develops their representation as Computation-Activation Networks before treating the general case.Source: §3 · p. 9
3.1 Representation as elementary Computation-Activation Networks
- Every exponential-family member can be represented as a Computation-Activation Network with elementary activations.The construction uses activation cores for the statistic coordinates.Source: §3.1 · p. 9
3.2 Maximum entropy distributions
- Exponential-family members maximize entropy among distributions with the same mean vector under the specified statistic and base measure.The result applies to any member of the family, while the representing weight vector is unique only for minimal families.Source: §3.2 · pp. 10–11
- A maximum-entropy solution is an exponential-family member exactly when its mean parameter lies in the relative interior of the mean polytope.The distribution is unique, although its representing weight vector need not be.Source: §3.2 · p. 11
4 Generic maximum entropy distributions
The paper extends the maximum-entropy characterization to every feasible mean parameter by associating it with a face of the mean polytope. The resulting distributions use face-refined base measures, with tensor-network representation available when the face is representable by the chosen hypergraph.
4 Generic maximum entropy distributions
- The characterization extends beyond exponential-family members, whose mean parameters lie in the relative interior rather than on lower-dimensional faces.The general case uses tensor representations of those faces to characterize maximum-entropy distributions.Source: §4.1 · p. 11, §4 · p. 11
4.1 Main result
- For any feasible mean parameter, the unique maximum-entropy distribution is determined using its associated face and a backward map for the refined base measure.The maximum-entropy problem is feasible exactly when the mean parameter belongs to the mean polytope.Source: §4.1 · p. 12
4.2 Family of maximum entropy distributions
- Maximum-entropy distributions decompose into a face-dependent, possibly non-elementary Boolean tensor and an elementary tensor determined by the weight vector.All distributions for means in a face's relative interior belong to the specified tensor-network family when that face is representable by the hypergraph.Source: §4.1 · p. 12, §4.2 · p. 13
5 Boolean statistics
This section studies Boolean-statistics polytopes and relates their face geometry to CompActNet support representations, propositional logic, and codeword constraints. It characterizes elementary face representations and constructs statistics for arbitrary polytopes with Boolean vertices, including an equivalence between constructed Boolean-statistics polytopes.
5.1 Characterization of elementary representable faces
- For Boolean statistics, a face is representable by an elementary tensor exactly when it is cube-like, and every distribution in Λt,EL,ν is a maximum entropy distribution.The maximum-entropy claim is with respect to the statistic, base measure, and the distribution's own mean parameter.Source: §5.1 · pp. 14–15
5.2 Interpretation by propositional formulas
- A Boolean vector is a vertex of the mean polytope exactly when the conjunction associated with that vector is satisfiable.For atomic formulas, the mean polytope is a hypercube; the section also relates the hypercube case to the absence of entailment or contradiction among formulas.Source: §5.2 · pp. 15–16
- Minterm statistics yield a standard simplex, and a Boolean statistic has a standard-simplex mean polytope exactly when each target-coordinate restriction contradicts all others.The simplex is cube-like, so its faces admit elementary representations.Source: §5.2 · p. 16
- The formulas X0 ∧ X1 and X0 ⇒ X1 produce a non-cube-like face whose activation core is non-elementary; other non-vertex faces have elementary cores.The exceptional face is the convex hull of (0,0) and (1,1), and its maximum-entropy distributions use a CP-rank 2 hard activation core.Source: §5.2 · pp. 16–17
5.3 Faces activated in the TT format
- A TT construction represents a diagonal face using Boolean hidden variables of dimension 2, which is minimal because every nontrivial partition matrification has rank 2.The hidden variables indicate whether a coordinate flip has occurred.Source: §5.3 · pp. 18–19
5.4 Construction of Boolean statistics to polytopes
- Every polytope with Boolean vertices in R^p can be realized as the mean polytope of a Boolean statistic constructed from propositional formulas.The construction assigns formulas coordinate by coordinate using tautologies, contradictions, or atoms according to the polytope's coordinate restrictions.Source: §5.4 · p. 19
- The constructed Boolean-statistics and associated 0/1 polytopes have coincident vertices and are equivalent, with the defining equation established by induction for arbitrary ℓ.The argument’s ℓ = p − 1 case gives the stated claim.Source: §5.4 · p. 19
5.5 Examples in statistical physics and coding theory
- In the Ising statistic, the only cube-like facet of the four vertices has parametrization (U, y_U) = ({2}, 0); separately, some maximum-entropy distributions in the Ising statistic lack a representation by an elementary CompActNet.Source: §5.5 · p. 20
- The example’s length-3 codewords satisfy vanishing parity, and LDPC codes generalize this construction through parity constraints; the resulting polytopes are cycle polytopes (Gallager (1963); Grötschel et al. (1993)).Source: §5.5 · pp. 20–21
- For the codeword polytope, all facets are non-cube-like, whereas its other faces are cube-like.Source: §5.5 · p. 21
- Each codeword-polytope facet is defined by Hamming distance one from a non-codeword cube vertex (Wainwright and Jordan (2008)); rank-2 TT activation tensors compute these distances.Each tensor core contributes one mode’s Hamming-distance term.Source: §5.5 · p. 21
6 Outlook
The paper characterizes all discrete maximum-entropy distributions as CompActNets by refining the base measure according to the mean-polytope face. It connects representation choices to learning and points to tensor-format and face complexity as directions for further study.
6 Outlook
- Refining the base measure by the mean-polytope face lets CompActNets represent all discrete maximum-entropy distributions.Exponential families cover mean parameters in the polytope's relative interior; the face-based refinement handles parameters on particular faces.Source: §6 · p. 21
- A poor learning architecture can cause numerical instabilities through the absence of an optimum, while the right tensor format for the refined base measure guarantees one.The discussion connects this representation issue to maximum-likelihood learning, which is dual to maximum-entropy problems.Source: §6 · p. 21
- Further work could relate tensor-format complexity to facet complexity and study face complexity in randomly drawn polytopes, especially at large scale.The authors identify the asymptotic limit of large statistics as a setting of interest.Source: §6 · p. 21