Source-linked AI summary
Equivalence of restricted Boltzmann machines and tensor network states
Jing Chen, Song Cheng, Haidong Xie, Lei Wang, Tao Xiang
TL;DR
The paper addresses the gap between rigorous analysis of neural-network expressive power and deep learning applications by constructing a general correspondence between RBMs and tensor network states. It develops transformations in both directions, uses entanglement entropy to quantify RBM expressibility, and finds that RBMs can represent quantum states more compactly than TNS.
Problem
Rigorously quantifying neural-network expressive power remains difficult despite deep learning’s broad applications and connections to physical principles.
Method
The paper constructs a general RBM–TNS correspondence, including algorithms for RBM-to-TNS conversion and necessary and sufficient conditions for TNS-to-RBM conversion.
Results
The correspondence enables entanglement-entropy bounds to quantify RBM expressibility and shows that RBMs can represent quantum states more compactly than TNS.
Takeaways & Limitations
TNS methods can inform machine-learning analysis and architecture design, while deep-learning methods and RBM representations may benefit quantum many-body simulations.
Abstract
from arXiv · showhide
The restricted Boltzmann machine (RBM) is one of the fundamental building blocks of deep learning. RBM finds wide applications in dimensional reduction, feature extraction, and recommender systems via modeling the probability distributions of a variety of input data including natural images, speech signals, and customer ratings, etc. We build a bridge between RBM and tensor network states (TNS) widely used in quantum many-body physics research. We devise efficient algorithms to translate an RBM into the commonly used TNS. Conversely, we give sufficient and necessary conditions to determine whether a TNS can be transformed into an RBM of given architectures. Revealing these general and constructive connections can cross-fertilize both deep learning and quantum many-body physics. Notably, by exploiting the entanglement entropy bound of TNS, we can rigorously quantify the expressive power of RBM on complex data sets. Insights into TNS and its entanglement capacity can guide the design of more powerful deep learning architectures. On the other hand, RBM can represent quantum many-body states with fewer parameters compared to TNS, which may allow more efficient classical simulations.
I. INTRODUCTION
The introduction frames a practical gap in quantifying neural-network expressivity and proposes a constructive bridge between RBMs and tensor network states to address it. This connection enables entanglement-based analysis of RBM expressivity and information exchange between deep learning and quantum physics.
- RBM background: RBMs model visible-variable distributions and support feature extraction, dimensionality reduction, discriminative tasks, and generative tasks.They use interconnected visible and hidden binary variables, with hidden units generating effective interactions among visible units.
- Motivation: Existing universal-approximation results require exponentially large resources and offer limited practical guidance for complex physical or industrial datasets.The introduction presents this as a limitation of using such theorems to quantify expressive power in practice.
- Tensor network background: TNS, including MPS, represent multivariable functions through tensor contractions whose capacity increases with virtual bond dimension.MPS uses three-index tensors, and its virtual bond dimension is the matrix dimension associated with each physical variable.
- Tensor network background: TNS efficiently represent many physically relevant states because their entanglement entropy follows an area law and remains relatively low.The area law relates entanglement entropy to the boundary size separating subsystems.
- Contribution: The paper establishes a general, constructive RBM–TNS connection and uses TNS entanglement bounds to quantify RBM expressivity across quantum, statistical-physics, and industrial datasets.It also derives necessary and sufficient conditions for transforming a TNS into an RBM with a specified structure.
II. TNS REPRESENTATION OF RBM
This section constructs an MPS representation of an RBM by converting units and couplings into tensors, partitioning the network, and contracting internal indices. Long-range connections are factorized across cuts, making bond dimensions depend on the number of crossed connections.
- Direct mapping: The RBM is first converted into a tensor network by treating visible units as physical variables and hidden units as virtual variables.Diagonal tensors encode vertex terms, while 2 × 2 matrices encode connection weights on bonds.
- Direct mapping: The tensor network is partitioned into nv pieces, each containing one visible unit, and internal variables are contracted to form local MPS tensors.Hidden-unit assignments to pieces may be arbitrary; external connections become virtual MPS bonds.
- Direct mapping: Long-range connections are split into products of two 2 × 2 matrices and absorbed into local tensors along the path of the connection.For the example, factoring M (41) into P and Q separates one long-range bond into two short ones.
- Bond dimension: D = 2^n, where n is the number of connections cut at an MPS bipartition.Each long-range connection crossing a vertical cut doubles the associated bond dimension.
- Representation properties: The resulting MPS is not unique because hidden-unit geometry can be rearranged, and the local tensors may contain redundant degrees of freedom.Canonical transformations can remove this redundancy, while the mapping itself applies to arbitrarily dense RBMs despite being illustrated with sparse connections.
B. Optimal mapping of an RBM to an MPS
The optimal mapping improves on the direct construction by using conditional independence to choose the smallest virtual separators between successive visible-variable groups. This yields an MPS with optimal bond dimensions and can be applied to general undirected graphical models.
- Optimal mapping: The direct mapping can overestimate bond dimensions, so the optimal method searches for the smallest separator Z satisfying X ⊥ Y | Z.The separator variables become virtual-bond degrees of freedom after translation.
- Optimal mapping: The algorithm constructs MPS tensors from left to right, maintaining the smallest virtual bond dimension at each step.At each visible unit, X contains the current visible variable and the previous virtual degrees of freedom, while Y contains remaining visible variables.
- Tensor construction: Hidden units disconnected from the remaining variables after conditioning are traced out and incorporated into the current tensor.This removes hidden variables that no longer need to remain on the virtual bond.
- Worked construction: The worked construction selects separators such as {h1, h2} and {v3, v4}, with each separator determining the corresponding right virtual bond.The separator may contain visible and hidden units, depending on which set minimizes its size.
- Outcome: Each original RBM connection is considered once, allowing all six MPS tensors and the optimal bond dimension to be obtained even without numerical execution.The same conditional-independence method also applies to general undirected graphical models.
C. Implication of the RBM to MPS mapping
The RBM-to-MPS mapping turns interface degrees of freedom into virtual bonds, enabling entanglement-based bounds on RBM expressiveness. These bounds distinguish sparse and dense connectivity and can yield more compact representations than TNS for highly entangled states.
- Entanglement and expressive power: Entanglement entropy measures RBM information content and is bounded for MPS by the logarithm of the bond dimension, ln D.This connects MPS bond dimensions directly to the correlations an RBM representation can capture.
- Entanglement and expressive power: Fixing the interface variables Y1 factorizes the RBM into X and Y2, giving the MPS bond dimension D = 2^|Y1|.The interface contains visible units directly linked to X.
- Optimal bond dimensions: Using the smaller interface from X1 or Y1 bounds the entanglement entropy by Smax = min(|X1|, |Y1|) ln 2.The corresponding construction produces tighter bond-dimension bounds than the simpler MPS mapping.
- Optimal bond dimensions: The optimal mapping represents the minimal separating degrees of freedom, whether visible or hidden, as MPS virtual bonds.Algorithm 2 constructs an MPS with optimal smallest bond dimensions from RBM weights and biases.
- Implications: RBM architecture alone yields rigorous expressive-power bounds, which can be estimated efficiently and potentially tightened by MPS canonization.Canonization removes unnecessary degrees of freedom and can give precise entanglement entropies.
- Implications: Sparse RBMs obey an entanglement area law, whereas dense RBMs can reach volume-law entanglement with polynomially many parameters.Representing the same highly entangled state with an MPS or PEPS may require exponentially many parameters.
- Extensions: The RBM-to-TNS mapping extends beyond bipartite RBMs to general Boltzmann machines and to PEPS for data arranged on two-dimensional grids.The PEPS extension is particularly relevant when RBM inputs, such as image pixels, lie on a two-dimensional array.
III. RBM REPRESENTATION OF TNS: SUFFICIENT AND NECESSARY CONDITIONS
The paper gives constructive conditions for representing a TNS as an RBM with a specified architecture. The mapping requires solving tensor-factorization equations, whose feasibility depends on the architecture and tensor ranks.
- Reverse mapping: A TNS-to-RBM mapping is formulated for a fixed RBM architecture, avoiding the exponentially large resources of unrestricted universal approximation.The paper provides a constructive approach and discusses sufficient conditions for the mapping.
- Reverse mapping: For a six-site MPS example, Eq. (13) produces 64 linear equations for 40 tensor elements, so a unique solution is required.The hidden layer has four units, yielding one four-index tensor and three three-index tensors.
- Reverse mapping: If Eq. (13) has no solution, increasing hidden units or connections can enlarge the parameter space and alter the system from overdetermined to underdetermined.The architecture must be changed when the original equations are unsolvable.
- Tensor decomposition: Each tensor must then be decomposed into an RBM with one hidden unit, corresponding to a rank-2 tensor decomposition for binary hidden variables.For a three-index tensor, the decomposition has seven parameters constrained by eight equations and is solvable only in special cases.
- Tensor decomposition: Tensor rank can obstruct the mapping: if the rank exceeds two, binary hidden units cannot satisfy the required decomposition without enlarging their basis dimension.A rank-2 decomposition always exists for a 2×2×2 tensor over the complex field, whereas arbitrary tensor rank is difficult to determine.
- Tensor decomposition: The necessary and sufficient condition for mapping an MPS to an RBM is that both Eq. (13) and Eq. (14) have unique solutions.Adding hidden units and connections can provide enough parameters for both equations to admit solutions.
- Applications and boundary cases: A factorization test fixes a sequence of visible units and checks whether the TNS becomes a product state; this excludes short-range RBM representations of the AKLT state.The obstruction is the AKLT state's hidden string order, although it has a D = 2 MPS representation.
IV. EXAMPLE: RBM REPRESENTATION OF THE TORIC CODE GROUND STATES
The paper constructs RBM representations of all four toric-code ground states from PEPS, identifies their topological sectors, and explains gauge-equivalent transformations of the resulting networks.
- RBM construction: The PEPS-to-RBM construction identifies bond-centered I3 tensors as visible units and vertex-centered I4 tensors as hidden units.The interaction matrix U is decomposed to obtain the RBM parameters.
- RBM construction: The resulting RBM uses only nearest-neighbor visible-hidden connections, with each hidden unit coupled to four visible units.Tracing out the hidden units yields the toric-code wavefunction.
- Ground-state representation: The RBM represents equal-weight superpositions of closed loops whose visible variables sum to an even value at every vertex.This representation is simpler than an earlier construction using hidden units at both vertices and plaquette centers.
- Topological sectors: The four ground states are distinguished by the ±1 eigenvalues of two commuting Wilson-loop operators, defining four topological sectors.The displayed RBM belongs to the (+,+) sector.
- Topological sectors: Changing visible biases from a = 0 to a = iπ along winding Wilson-loop paths transforms the RBM between topological sectors.The paths need not be straight, but they must wind around the torus.
- Gauge invariance: Applying products of A+ operators can change visible biases around a contractible loop while preserving the topological sector through gauge equivalence.The resulting wavefunction remains in the same sector because each A+ operator conserves it.
A. Optimizing RBM using tensor-network methods
Tensor-network transformations provide a route to simplify redundant RBM parametrizations by reducing tensor bond dimensions and mapping the optimized network back to an RBM.
- Redundancy and simplification: Different RBM parameters can represent equivalent functions, so tensor-network methods can remove redundant degrees of freedom.This establishes tensor-network simplification as an optimization tool for RBMs.
- One-dimensional optimization: In one dimension, an RBM is mapped to an MPS, canonicalized by discarding zero singular vectors, and then mapped back to an optimized RBM.Canonicalization can reduce local bond dimensions and partially fix the MPS gauge.
- One-dimensional optimization: The cluster-state RBM’s bond dimension decreases from D = 4 to D = 2 after canonical transformation.The optimized RBM correspondingly reduces each hidden unit’s connections from three visible units to two neighboring visible units.
- Higher-dimensional optimization: Higher-dimensional RBMs can be simplified through PEPS or other tensor networks using higher-order or pairwise singular-value decompositions.These decompositions reduce, or partially reduce when redundancies exist, the bond degrees of freedom.
- Representation nonuniqueness: The toric-code example shows that distinct PEPS and RBM representations of the same ground state are not always connected by local internal-bond gauge transformations.The obstruction is attributed to the ground state being a non-injective Z2 spin liquid state.
- Computational handling: Huge intermediate tensor-network bond dimensions can be managed by dynamic truncation or by simplifying overlapping system pieces separately.Both approaches avoid storing the full large tensors during translation.
B. TNS representation of the shift-invariant RBM and its entanglement capacity
A shift-invariant RBM can be converted into an MPS, while translational replication increases entanglement capacity without increasing the number of variational parameters.
- Shift-invariant construction: A shift-invariant RBM is formed as a product of translated copies of an individual RBM factor.With nh hidden units per factor, the full construction contains nvnh hidden variables.
- MPS representation: Each RBM factor can be expressed as an MPS, and assembling the translated factors yields a single MPS representation.The straightforward assembled bond dimension is D = (DRBM)^nv.
- MPS representation: A direct mapping using the minimal interface region gives the enlarged shift-invariant RBM an MPS bond dimension D = 2^nv/2 at the center.This structure-specific mapping can be tighter than multiplying the bond dimensions of all factors.
- Entanglement capacity: The shift-invariant operation drastically increases the wavefunction’s entanglement capability without increasing the number of variational parameters.The construction can also be generalized to other variational wavefunctions and symmetries such as rotation or inversion.
- Design principles: Effective RBM designs favor global or long-range connectivity, parameter sharing, and multiple hidden units coupled to the same visible units.Sparse long-range connections can provide large bond dimensions without requiring dense connectivity.
C. An entanglement perspective to unsupervised learning
The paper transfers entanglement concepts to unsupervised learning, using dataset entanglement entropy to quantify learning difficulty and guide RBM connectivity design.
- Dataset complexity: Dataset entanglement entropy is introduced to quantify the resources required for RBMs to model probability distributions.This offers a more practically useful perspective than universal-approximation results requiring exponentially large resources.
- Dataset complexity: A probability amplitude Ψ(v) = √P(v) enables reduced density matrices and entanglement entropy to be defined for real datasets.The resulting quantity captures dataset complexity in a way analogous to classical information-theoretic measures.
- Connectivity implications: Natural-image datasets are expected to have relatively small entanglement entropy because pixel correlations are typically dominated by short-range interactions.This observation motivates connectivity designs that need not be fully dense.
- Connectivity implications: An RBM can retain good performance after 80% of its connections are randomly removed, while sparse small-world RBMs can perform well compared with densely connected RBMs.These reported findings support using dataset entanglement structure when designing neural-network connectivity.
- Transfer of methods: The RBM–TNS connection permits entanglement-entropy upper bounds to be estimated from tensor-network bond dimensions and supports applying quantum-physics techniques to machine learning.Entanglement entropy can also characterize the difficulty of learning when tensor networks directly model datasets.
D. Entanglement advantage of deep Boltzmann machines over the shallow ones
The RBM–TNS correspondence shows that deep Boltzmann machines can have greater entanglement capacity than shallow RBMs with comparable parameter counts. This provides a way to compare architectures and assess whether distributions can be represented.
- Architecture comparison: The RBM–TNS mapping extends to unrestricted Boltzmann machines and helps explain the advantage of deep Boltzmann machines over shallow RBMs.The correspondence applies to architectures without the RBM’s bipartite restriction.
- Architecture comparison: With nv visible units, nh = 3nv hidden units, and 9nv connections, the compared DBM has bond dimension DDBM = 24 versus DRBM = 22.Figure 11 compares the two architectures using the same visible-unit, hidden-unit, and connection counts.
- Entanglement capacity: A second comparison reports DDBM = 16 and DRBM = 4 for architectures using the same amount of parameters, giving the DBM larger entanglement entropy.The DBM’s multilayer hidden structure mediates longer-ranged effective connections.
- Dataset example: For the 4 × 4 Bars and Stripes dataset, exact entanglement entropy is approximately 1.80, exceeding ln 4, so the shown RBM cannot capture it while the shown DBM can.The dataset’s wave function is the equal superposition of 30 valid configurations.
- Implications: The TNS mapping offers a way to analyze and compare expressive power across Boltzmann-machine architectures through entanglement capacity.Deep hidden units can mediate longer-ranged effective connections among visible units.
- Implications: TNS entanglement theory provides bounds for quantifying RBM expressive power and can support resource estimates and removal of redundant degrees of freedom.The correspondence also connects methods across deep learning and quantum physics.
- Outlook: The paper identifies connections between deep architectures and multilayer TNS as an avenue for future study and neural-network design.The outlook specifically mentions tree tensor networks and multiscale entanglement renormalization ansatz.
Appendix A: A sufficient condition for RBM representation of MPS/PEPS and examples
The appendix gives sufficient conditions for representing certain MPS and PEPS as RBMs and constructs examples for Ising models and cluster states. The constructions place hidden units on lattice bonds and can remain exact at criticality.
- General condition: A sufficient condition is given for MPS or PEPS to have an RBM representation, covering examples including toric-code, Ising, and cluster-state constructions.The condition applies to many physically interesting thermal states and quantum wavefunctions.
- MPS construction: For an MPS, each local tensor must have the specified form involving two 2 × 2 matrices L and R.The product R and L can be replaced by coupling to an RBM hidden unit.
- MPS construction: The decomposition may be chosen arbitrarily; the appendix uses a symmetric form for simplicity.
- Ising example: The one-dimensional Ising partition function is represented as an MPS, then converted into RBM parameters by combining the MPS construction with the stated transformation equations.The Ising variables are converted to binary visible variables using vi = (si+1)/2.
- PEPS construction: The construction generalizes to two dimensions by representing the partition function as a PEPS and introducing one hidden unit for each lattice bond.The two-dimensional construction adjusts the external-field contribution because each site is shared by four bonds.
- Ising example: A sparse RBM with nh = nv hidden units in one dimension, or nh = 2nv in two dimensions, exactly reproduces the Ising thermal distribution independently of coupling strength, including criticality.The hidden units act as auxiliary fields that decouple interactions.
- Cluster-state example: The cluster-state construction couples hidden units to physical degrees of freedom on lattice bonds and is simpler than an earlier construction requiring three hidden units per visible unit.The simplification comes from using the cluster state’s canonical MPS.
- Parameterization: Table I lists one possible, nonunique set of RBM parameters for the statistical Ising model and cluster state.Each hidden unit interacts with the two visible units connected by a bond.
Appendix B: General equivalence between Boltzmann machines and TNS
The appendix establishes a general correspondence between Boltzmann machines and tensor-network states, including mappings in both directions. Rank structure determines when binary hidden units suffice, while higher ranks require multistate hidden units.
- General correspondence: The RBM–TNS correspondence generalizes to Boltzmann machines with hidden-layer interconnections and direct connections within visible or hidden units.Deep Boltzmann machines and unrestricted Boltzmann machines are included in this broader framework.
- BM to TNS: A Boltzmann-machine function can be written as a tensor network using edge tensors for couplings and diagonal vertex tensors for biases.Visible-unit tensors additionally include a dimension for external degrees of freedom.
- TNS to BM: Any TNS built only from rank-2 tensors can be directly mapped to a binary Boltzmann machine through rank decomposition.The mapping produces connection weights and bias contributions from the decomposed tensors.
- Rank conditions: The decomposition rank is the minimal number of terms required, with examples r = 1 for vectors and r equal to the smaller matrix dimension for matrices.For tensors of order n ≥ 3, the rank need not be bounded by any individual dimension di.
- Rank conditions: Binary hidden units require r = 2 and di = 2 for every tensor index in the stated direct mapping condition.
- TNS to BM: For larger tensor ranks, CP decomposition yields Boltzmann machines whose hidden units have multiple states rather than being restricted to binary values.